[==========] Running 8 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 6 tests from LeaderRebalancerTest
[ RUN      ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260709 12:19:13.493274 30503 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.201.254:36891
I20260709 12:19:13.495164 30503 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260709 12:19:13.496379 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:13.515597 30512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:13.515427 30510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:13.517076 30503 server_base.cc:1061] running on GCE node
W20260709 12:19:13.515413 30509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:14.751178 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:14.751477 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:14.751641 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599554751624 us; error 0 us; skew 500 ppm
I20260709 12:19:14.752491 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:14.759567 30503 webserver.cc:533] Webserver started at http://127.29.201.254:36677/ using document root <none> and password file <none>
I20260709 12:19:14.760537 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:14.760754 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:14.761176 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:14.767535 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/master-0-root/instance:
uuid: "fcb51a0848cb4dbdbc6cd9256992e573"
format_stamp: "Formatted at 2026-07-09 12:19:14 on dist-test-slave-qgf5"
I20260709 12:19:14.776363 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:19:14.782128 30518 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:14.783638 30503 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.000s
I20260709 12:19:14.784015 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/master-0-root
uuid: "fcb51a0848cb4dbdbc6cd9256992e573"
format_stamp: "Formatted at 2026-07-09 12:19:14 on dist-test-slave-qgf5"
I20260709 12:19:14.784338 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:14.846674 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:14.848315 30503 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260709 12:19:14.848793 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:14.925161 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.254:36891
I20260709 12:19:14.925248 30569 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.254:36891 every 8 connection(s)
I20260709 12:19:14.933877 30570 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:14.957227 30570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573: Bootstrap starting.
I20260709 12:19:14.963574 30570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:14.965530 30570 log.cc:826] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573: Log is configured to *not* fsync() on all Append() calls
I20260709 12:19:14.970567 30570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573: No bootstrap required, opened a new log
I20260709 12:19:14.992146 30570 raft_consensus.cc:359] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb51a0848cb4dbdbc6cd9256992e573" member_type: VOTER }
I20260709 12:19:14.992754 30570 raft_consensus.cc:385] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:14.993000 30570 raft_consensus.cc:740] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcb51a0848cb4dbdbc6cd9256992e573, State: Initialized, Role: FOLLOWER
I20260709 12:19:14.994019 30570 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb51a0848cb4dbdbc6cd9256992e573" member_type: VOTER }
I20260709 12:19:14.994562 30570 raft_consensus.cc:399] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:19:14.994856 30570 raft_consensus.cc:493] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:19:14.995157 30570 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:15.000232 30570 raft_consensus.cc:515] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb51a0848cb4dbdbc6cd9256992e573" member_type: VOTER }
I20260709 12:19:15.001199 30570 leader_election.cc:304] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcb51a0848cb4dbdbc6cd9256992e573; no voters: 
I20260709 12:19:15.003077 30570 leader_election.cc:290] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:19:15.003449 30573 raft_consensus.cc:2804] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:15.006352 30573 raft_consensus.cc:697] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [term 1 LEADER]: Becoming Leader. State: Replica: fcb51a0848cb4dbdbc6cd9256992e573, State: Running, Role: LEADER
I20260709 12:19:15.007402 30573 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb51a0848cb4dbdbc6cd9256992e573" member_type: VOTER }
I20260709 12:19:15.008744 30570 sys_catalog.cc:565] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:19:15.020541 30575 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcb51a0848cb4dbdbc6cd9256992e573. Latest consensus state: current_term: 1 leader_uuid: "fcb51a0848cb4dbdbc6cd9256992e573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb51a0848cb4dbdbc6cd9256992e573" member_type: VOTER } }
I20260709 12:19:15.021752 30575 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [sys.catalog]: This master's current role is: LEADER
I20260709 12:19:15.023236 30574 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcb51a0848cb4dbdbc6cd9256992e573" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcb51a0848cb4dbdbc6cd9256992e573" member_type: VOTER } }
I20260709 12:19:15.024013 30574 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [sys.catalog]: This master's current role is: LEADER
I20260709 12:19:15.036288 30584 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:19:15.042733 30503 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:19:15.046916 30584 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:19:15.064482 30584 catalog_manager.cc:1383] Generated new cluster ID: 578a0a04013b4fbfb3e7f3b21b528400
I20260709 12:19:15.064795 30584 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:19:15.081581 30584 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:19:15.083891 30584 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:19:15.101680 30584 catalog_manager.cc:6070] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573: Generated new TSK 0
I20260709 12:19:15.102854 30584 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:19:15.112852 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:15.123299 30592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:15.124725 30593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:15.129478 30595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:15.130506 30503 server_base.cc:1061] running on GCE node
I20260709 12:19:15.131433 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:15.131641 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:15.131870 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599555131846 us; error 0 us; skew 500 ppm
I20260709 12:19:15.132529 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:15.137513 30503 webserver.cc:533] Webserver started at http://127.29.201.193:36893/ using document root <none> and password file <none>
I20260709 12:19:15.138160 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:15.138391 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:15.138764 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:15.140007 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-0-root/instance:
uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
format_stamp: "Formatted at 2026-07-09 12:19:15 on dist-test-slave-qgf5"
I20260709 12:19:15.146450 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.001s
I20260709 12:19:15.151525 30600 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:15.152503 30503 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.002s	sys 0.002s
I20260709 12:19:15.152840 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-0-root
uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
format_stamp: "Formatted at 2026-07-09 12:19:15 on dist-test-slave-qgf5"
I20260709 12:19:15.153195 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:15.170238 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:15.171808 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:15.174924 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:15.180171 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:15.180478 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:15.180825 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:15.180999 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:15.295670 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.193:43449
I20260709 12:19:15.295778 30662 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.193:43449 every 8 connection(s)
I20260709 12:19:15.303721 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:15.319334 30668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:15.319428 30667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:15.326534 30670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:15.327270 30503 server_base.cc:1061] running on GCE node
I20260709 12:19:15.328328 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:15.328577 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:15.328807 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599555328772 us; error 0 us; skew 500 ppm
I20260709 12:19:15.329458 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:15.332887 30503 webserver.cc:533] Webserver started at http://127.29.201.194:38949/ using document root <none> and password file <none>
I20260709 12:19:15.333689 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:15.333999 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:15.334666 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:15.336704 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-1-root/instance:
uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
format_stamp: "Formatted at 2026-07-09 12:19:15 on dist-test-slave-qgf5"
I20260709 12:19:15.341084 30663 heartbeater.cc:344] Connected to a master server at 127.29.201.254:36891
I20260709 12:19:15.341660 30663 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:15.343015 30663 heartbeater.cc:507] Master 127.29.201.254:36891 requested a full tablet report, sending...
I20260709 12:19:15.346184 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.009s	user 0.009s	sys 0.000s
I20260709 12:19:15.347014 30535 ts_manager.cc:194] Registered new tserver with Master: e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449)
I20260709 12:19:15.349310 30535 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42466
I20260709 12:19:15.352702 30675 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:15.353787 30503 fs_manager.cc:730] Time spent opening block manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:19:15.354172 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-1-root
uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
format_stamp: "Formatted at 2026-07-09 12:19:15 on dist-test-slave-qgf5"
I20260709 12:19:15.354571 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:15.388767 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:15.390693 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:15.393599 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:15.397634 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:15.397868 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:15.398198 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:15.398365 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:15.512488 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.194:41949
I20260709 12:19:15.513006 30737 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.194:41949 every 8 connection(s)
I20260709 12:19:15.519150 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:15.528620 30741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:15.530912 30742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:15.540884 30738 heartbeater.cc:344] Connected to a master server at 127.29.201.254:36891
I20260709 12:19:15.541324 30738 heartbeater.cc:461] Registering TS with master...
W20260709 12:19:15.541769 30744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:15.542315 30738 heartbeater.cc:507] Master 127.29.201.254:36891 requested a full tablet report, sending...
I20260709 12:19:15.544292 30503 server_base.cc:1061] running on GCE node
I20260709 12:19:15.544903 30535 ts_manager.cc:194] Registered new tserver with Master: b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949)
I20260709 12:19:15.545486 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:15.545753 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:15.545966 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599555545947 us; error 0 us; skew 500 ppm
I20260709 12:19:15.546772 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:15.546778 30535 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42480
I20260709 12:19:15.550375 30503 webserver.cc:533] Webserver started at http://127.29.201.195:36501/ using document root <none> and password file <none>
I20260709 12:19:15.550946 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:15.551167 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:15.551483 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:15.552590 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-2-root/instance:
uuid: "07da37de9b1e4baca15164fe424a8fdb"
format_stamp: "Formatted at 2026-07-09 12:19:15 on dist-test-slave-qgf5"
I20260709 12:19:15.557746 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.006s	sys 0.000s
I20260709 12:19:15.561674 30749 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:15.562611 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:19:15.563046 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-2-root
uuid: "07da37de9b1e4baca15164fe424a8fdb"
format_stamp: "Formatted at 2026-07-09 12:19:15 on dist-test-slave-qgf5"
I20260709 12:19:15.563480 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1783599553313367-30503-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:15.583030 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:15.585009 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:15.588049 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:15.592483 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:15.592850 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:15.593192 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:15.593458 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:15.692368 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.195:35831
I20260709 12:19:15.692461 30811 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.195:35831 every 8 connection(s)
I20260709 12:19:15.711846 30812 heartbeater.cc:344] Connected to a master server at 127.29.201.254:36891
I20260709 12:19:15.712387 30812 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:15.713413 30812 heartbeater.cc:507] Master 127.29.201.254:36891 requested a full tablet report, sending...
I20260709 12:19:15.715991 30535 ts_manager.cc:194] Registered new tserver with Master: 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:15.716877 30503 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015167401s
I20260709 12:19:15.717686 30535 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42492
I20260709 12:19:15.771971 30535 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42508:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:19:15.776239 30535 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:19:15.885382 30628 tablet_service.cc:1511] Processing CreateTablet for tablet eec3ea46dfb242f4b4ce66a97604aceb (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:19:15.886605 30627 tablet_service.cc:1511] Processing CreateTablet for tablet 535f09def3bd494687aa4a3edeab61d6 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:19:15.890363 30626 tablet_service.cc:1511] Processing CreateTablet for tablet 4884fe1576a44b67862ba32ea082366a (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:19:15.894992 30628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec3ea46dfb242f4b4ce66a97604aceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.897197 30627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 535f09def3bd494687aa4a3edeab61d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.909435 30701 tablet_service.cc:1511] Processing CreateTablet for tablet 4884fe1576a44b67862ba32ea082366a (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:19:15.908138 30703 tablet_service.cc:1511] Processing CreateTablet for tablet eec3ea46dfb242f4b4ce66a97604aceb (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:19:15.912747 30626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4884fe1576a44b67862ba32ea082366a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.923808 30702 tablet_service.cc:1511] Processing CreateTablet for tablet 535f09def3bd494687aa4a3edeab61d6 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:19:15.927420 30777 tablet_service.cc:1511] Processing CreateTablet for tablet eec3ea46dfb242f4b4ce66a97604aceb (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:19:15.927953 30625 tablet_service.cc:1511] Processing CreateTablet for tablet a19c6e5d902745869661b7ee64f329e8 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:19:15.928397 30701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4884fe1576a44b67862ba32ea082366a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.932873 30777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec3ea46dfb242f4b4ce66a97604aceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.932606 30702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 535f09def3bd494687aa4a3edeab61d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.933792 30625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a19c6e5d902745869661b7ee64f329e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.933483 30700 tablet_service.cc:1511] Processing CreateTablet for tablet a19c6e5d902745869661b7ee64f329e8 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:19:15.938484 30775 tablet_service.cc:1511] Processing CreateTablet for tablet 4884fe1576a44b67862ba32ea082366a (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:19:15.936256 30776 tablet_service.cc:1511] Processing CreateTablet for tablet 535f09def3bd494687aa4a3edeab61d6 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:19:15.940956 30775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4884fe1576a44b67862ba32ea082366a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.942773 30703 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec3ea46dfb242f4b4ce66a97604aceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.945439 30776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 535f09def3bd494687aa4a3edeab61d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.947530 30624 tablet_service.cc:1511] Processing CreateTablet for tablet 5e82899b14fc4d10a58ca1f42c33e4f7 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:19:15.947551 30700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a19c6e5d902745869661b7ee64f329e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.957685 30624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e82899b14fc4d10a58ca1f42c33e4f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.959815 30774 tablet_service.cc:1511] Processing CreateTablet for tablet a19c6e5d902745869661b7ee64f329e8 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:19:15.961702 30774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a19c6e5d902745869661b7ee64f329e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:15.990496 30699 tablet_service.cc:1511] Processing CreateTablet for tablet 5e82899b14fc4d10a58ca1f42c33e4f7 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:19:15.992110 30699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e82899b14fc4d10a58ca1f42c33e4f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.021471 30773 tablet_service.cc:1511] Processing CreateTablet for tablet 5e82899b14fc4d10a58ca1f42c33e4f7 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:19:16.024936 30773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e82899b14fc4d10a58ca1f42c33e4f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.037493 30623 tablet_service.cc:1511] Processing CreateTablet for tablet 75f2a084d518442e8241188f2d30b856 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
W20260709 12:19:16.039543 30582 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:19:16.043578 30701 tablet_service.cc:1511] Processing CreateTablet for tablet 75f2a084d518442e8241188f2d30b856 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:19:16.045217 30623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75f2a084d518442e8241188f2d30b856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.061420 30701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75f2a084d518442e8241188f2d30b856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.098606 30698 tablet_service.cc:1511] Processing CreateTablet for tablet 385b71fa8aea425f94dba57ebffc17fe (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:19:16.102579 30622 tablet_service.cc:1511] Processing CreateTablet for tablet 385b71fa8aea425f94dba57ebffc17fe (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:19:16.106380 30830 tablet_bootstrap.cc:492] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe: Bootstrap starting.
I20260709 12:19:16.107795 30772 tablet_service.cc:1511] Processing CreateTablet for tablet 75f2a084d518442e8241188f2d30b856 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:19:16.115350 30698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385b71fa8aea425f94dba57ebffc17fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.111982 30772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75f2a084d518442e8241188f2d30b856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.113030 30622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385b71fa8aea425f94dba57ebffc17fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.118762 30831 tablet_bootstrap.cc:492] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7: Bootstrap starting.
I20260709 12:19:16.123463 30771 tablet_service.cc:1511] Processing CreateTablet for tablet 385b71fa8aea425f94dba57ebffc17fe (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:19:16.130406 30771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 385b71fa8aea425f94dba57ebffc17fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.159616 30626 tablet_service.cc:1511] Processing CreateTablet for tablet 498b463cdd8b414c8a752d8c093c49ac (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:19:16.163513 30626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 498b463cdd8b414c8a752d8c093c49ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.173164 30702 tablet_service.cc:1511] Processing CreateTablet for tablet 498b463cdd8b414c8a752d8c093c49ac (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:19:16.177147 30702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 498b463cdd8b414c8a752d8c093c49ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.176106 30774 tablet_service.cc:1511] Processing CreateTablet for tablet 498b463cdd8b414c8a752d8c093c49ac (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:19:16.181267 30774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 498b463cdd8b414c8a752d8c093c49ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.185537 30832 tablet_bootstrap.cc:492] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb: Bootstrap starting.
I20260709 12:19:16.190204 30624 tablet_service.cc:1511] Processing CreateTablet for tablet e7b159647196412382fcbeea841f3d93 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:19:16.191665 30624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7b159647196412382fcbeea841f3d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.226909 30831 tablet_bootstrap.cc:654] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.229216 30832 tablet_bootstrap.cc:654] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.232429 30830 tablet_bootstrap.cc:654] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.237566 30774 tablet_service.cc:1511] Processing CreateTablet for tablet e7b159647196412382fcbeea841f3d93 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:19:16.239123 30774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7b159647196412382fcbeea841f3d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.247520 30702 tablet_service.cc:1511] Processing CreateTablet for tablet e7b159647196412382fcbeea841f3d93 (DEFAULT_TABLE table=test-workload [id=7cb904c51d734848b7816367fabfaee8]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:19:16.248910 30702 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7b159647196412382fcbeea841f3d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:16.277587 30830 tablet_bootstrap.cc:492] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe: No bootstrap required, opened a new log
I20260709 12:19:16.278160 30832 tablet_bootstrap.cc:492] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb: No bootstrap required, opened a new log
I20260709 12:19:16.278394 30830 ts_tablet_manager.cc:1403] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent bootstrapping tablet: real 0.174s	user 0.013s	sys 0.029s
I20260709 12:19:16.278697 30832 ts_tablet_manager.cc:1403] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb: Time spent bootstrapping tablet: real 0.094s	user 0.007s	sys 0.011s
I20260709 12:19:16.285111 30832 raft_consensus.cc:359] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.288105 30832 raft_consensus.cc:385] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.288694 30832 raft_consensus.cc:740] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.289731 30830 raft_consensus.cc:359] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.289835 30832 consensus_queue.cc:260] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.288457 30831 tablet_bootstrap.cc:492] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7: No bootstrap required, opened a new log
I20260709 12:19:16.290699 30830 raft_consensus.cc:385] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.291152 30830 raft_consensus.cc:740] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.291256 30831 ts_tablet_manager.cc:1403] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent bootstrapping tablet: real 0.184s	user 0.015s	sys 0.018s
I20260709 12:19:16.292248 30830 consensus_queue.cc:260] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.296386 30831 raft_consensus.cc:359] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.297235 30831 raft_consensus.cc:385] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.297591 30831 raft_consensus.cc:740] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.298580 30831 consensus_queue.cc:260] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.301452 30832 ts_tablet_manager.cc:1434] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb: Time spent starting tablet: real 0.022s	user 0.004s	sys 0.010s
I20260709 12:19:16.303047 30812 heartbeater.cc:499] Master 127.29.201.254:36891 was elected leader, sending a full tablet report...
I20260709 12:19:16.322360 30832 tablet_bootstrap.cc:492] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb: Bootstrap starting.
I20260709 12:19:16.334143 30738 heartbeater.cc:499] Master 127.29.201.254:36891 was elected leader, sending a full tablet report...
I20260709 12:19:16.340745 30830 ts_tablet_manager.cc:1434] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent starting tablet: real 0.061s	user 0.012s	sys 0.036s
I20260709 12:19:16.341958 30830 tablet_bootstrap.cc:492] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe: Bootstrap starting.
I20260709 12:19:16.343549 30663 heartbeater.cc:499] Master 127.29.201.254:36891 was elected leader, sending a full tablet report...
I20260709 12:19:16.345196 30832 tablet_bootstrap.cc:654] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.349100 30830 tablet_bootstrap.cc:654] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.362392 30830 tablet_bootstrap.cc:492] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe: No bootstrap required, opened a new log
I20260709 12:19:16.363185 30830 ts_tablet_manager.cc:1403] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent bootstrapping tablet: real 0.021s	user 0.016s	sys 0.003s
I20260709 12:19:16.367442 30832 tablet_bootstrap.cc:492] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb: No bootstrap required, opened a new log
I20260709 12:19:16.367991 30832 ts_tablet_manager.cc:1403] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb: Time spent bootstrapping tablet: real 0.046s	user 0.020s	sys 0.001s
I20260709 12:19:16.367663 30830 raft_consensus.cc:359] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.368414 30830 raft_consensus.cc:385] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.368772 30830 raft_consensus.cc:740] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.369736 30830 consensus_queue.cc:260] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.371495 30832 raft_consensus.cc:359] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.372023 30830 ts_tablet_manager.cc:1434] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.004s
I20260709 12:19:16.372273 30832 raft_consensus.cc:385] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.372615 30832 raft_consensus.cc:740] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.373026 30830 tablet_bootstrap.cc:492] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe: Bootstrap starting.
I20260709 12:19:16.374022 30831 ts_tablet_manager.cc:1434] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent starting tablet: real 0.082s	user 0.013s	sys 0.029s
I20260709 12:19:16.373617 30832 consensus_queue.cc:260] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.375243 30831 tablet_bootstrap.cc:492] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7: Bootstrap starting.
I20260709 12:19:16.380216 30832 ts_tablet_manager.cc:1434] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.001s
I20260709 12:19:16.380816 30830 tablet_bootstrap.cc:654] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.381317 30832 tablet_bootstrap.cc:492] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb: Bootstrap starting.
I20260709 12:19:16.395767 30830 tablet_bootstrap.cc:492] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe: No bootstrap required, opened a new log
I20260709 12:19:16.396329 30830 ts_tablet_manager.cc:1403] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent bootstrapping tablet: real 0.024s	user 0.013s	sys 0.005s
I20260709 12:19:16.399948 30830 raft_consensus.cc:359] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.401247 30830 raft_consensus.cc:385] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.401607 30830 raft_consensus.cc:740] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.403179 30832 tablet_bootstrap.cc:654] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.402757 30830 consensus_queue.cc:260] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.403244 30831 tablet_bootstrap.cc:654] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.415237 30831 tablet_bootstrap.cc:492] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7: No bootstrap required, opened a new log
I20260709 12:19:16.415148 30830 ts_tablet_manager.cc:1434] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.001s
I20260709 12:19:16.416687 30830 tablet_bootstrap.cc:492] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe: Bootstrap starting.
I20260709 12:19:16.422051 30831 ts_tablet_manager.cc:1403] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent bootstrapping tablet: real 0.047s	user 0.011s	sys 0.008s
I20260709 12:19:16.425578 30830 tablet_bootstrap.cc:654] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.429577 30832 tablet_bootstrap.cc:492] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb: No bootstrap required, opened a new log
I20260709 12:19:16.429328 30831 raft_consensus.cc:359] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.430109 30832 ts_tablet_manager.cc:1403] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb: Time spent bootstrapping tablet: real 0.049s	user 0.008s	sys 0.022s
I20260709 12:19:16.430182 30831 raft_consensus.cc:385] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.430568 30831 raft_consensus.cc:740] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.431566 30831 consensus_queue.cc:260] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.433328 30832 raft_consensus.cc:359] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.434092 30831 ts_tablet_manager.cc:1434] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.002s
I20260709 12:19:16.434147 30832 raft_consensus.cc:385] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.434548 30832 raft_consensus.cc:740] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.435489 30832 consensus_queue.cc:260] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.442023 30838 raft_consensus.cc:493] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.442570 30838 raft_consensus.cc:515] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.444955 30832 ts_tablet_manager.cc:1434] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.004s
I20260709 12:19:16.454033 30832 tablet_bootstrap.cc:492] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb: Bootstrap starting.
I20260709 12:19:16.455029 30831 tablet_bootstrap.cc:492] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7: Bootstrap starting.
I20260709 12:19:16.457005 30838 leader_election.cc:290] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
W20260709 12:19:16.458165 30813 tablet.cc:2406] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:19:16.472059 30830 tablet_bootstrap.cc:492] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe: No bootstrap required, opened a new log
I20260709 12:19:16.472745 30830 ts_tablet_manager.cc:1403] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent bootstrapping tablet: real 0.056s	user 0.023s	sys 0.025s
I20260709 12:19:16.473874 30831 tablet_bootstrap.cc:654] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.476150 30830 raft_consensus.cc:359] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.477090 30830 raft_consensus.cc:385] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.477500 30830 raft_consensus.cc:740] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.478422 30830 consensus_queue.cc:260] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.479770 30832 tablet_bootstrap.cc:654] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.481300 30830 ts_tablet_manager.cc:1434] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:19:16.482512 30830 tablet_bootstrap.cc:492] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe: Bootstrap starting.
I20260709 12:19:16.490542 30830 tablet_bootstrap.cc:654] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.494831 30837 raft_consensus.cc:493] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.495460 30837 raft_consensus.cc:515] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.498257 30837 leader_election.cc:290] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:16.503229 30831 tablet_bootstrap.cc:492] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7: No bootstrap required, opened a new log
I20260709 12:19:16.507741 30831 ts_tablet_manager.cc:1403] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent bootstrapping tablet: real 0.073s	user 0.010s	sys 0.006s
I20260709 12:19:16.523113 30831 raft_consensus.cc:359] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.529780 30831 raft_consensus.cc:385] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.529294 30832 tablet_bootstrap.cc:492] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb: No bootstrap required, opened a new log
I20260709 12:19:16.531081 30832 ts_tablet_manager.cc:1403] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb: Time spent bootstrapping tablet: real 0.077s	user 0.022s	sys 0.009s
I20260709 12:19:16.530320 30831 raft_consensus.cc:740] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.535710 30831 consensus_queue.cc:260] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.539458 30831 ts_tablet_manager.cc:1434] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent starting tablet: real 0.031s	user 0.009s	sys 0.000s
I20260709 12:19:16.540884 30831 tablet_bootstrap.cc:492] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7: Bootstrap starting.
I20260709 12:19:16.551450 30831 tablet_bootstrap.cc:654] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.566740 30832 raft_consensus.cc:359] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.567937 30832 raft_consensus.cc:385] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.568346 30832 raft_consensus.cc:740] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.569391 30832 consensus_queue.cc:260] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.573597 30832 ts_tablet_manager.cc:1434] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb: Time spent starting tablet: real 0.042s	user 0.007s	sys 0.003s
I20260709 12:19:16.575489 30830 tablet_bootstrap.cc:492] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe: No bootstrap required, opened a new log
I20260709 12:19:16.576526 30787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb" is_pre_election: true
I20260709 12:19:16.577534 30787 raft_consensus.cc:2468] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 0.
I20260709 12:19:16.581012 30830 ts_tablet_manager.cc:1403] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent bootstrapping tablet: real 0.099s	user 0.011s	sys 0.015s
I20260709 12:19:16.582901 30604 leader_election.cc:304] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:16.583853 30838 raft_consensus.cc:2804] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:16.584226 30838 raft_consensus.cc:493] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.584604 30838 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.586035 30837 raft_consensus.cc:493] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.587067 30837 raft_consensus.cc:515] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.589867 30836 raft_consensus.cc:493] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.585426 30832 tablet_bootstrap.cc:492] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb: Bootstrap starting.
I20260709 12:19:16.590477 30836 raft_consensus.cc:515] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.597635 30830 raft_consensus.cc:359] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.599053 30837 leader_election.cc:290] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:16.603083 30836 leader_election.cc:290] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949)
I20260709 12:19:16.600663 30830 raft_consensus.cc:385] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.604414 30837 raft_consensus.cc:493] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.605950 30830 raft_consensus.cc:740] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.608114 30837 raft_consensus.cc:515] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.621747 30837 leader_election.cc:290] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:16.620921 30830 consensus_queue.cc:260] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.648895 30713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" is_pre_election: true
I20260709 12:19:16.651968 30830 ts_tablet_manager.cc:1434] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent starting tablet: real 0.071s	user 0.005s	sys 0.006s
I20260709 12:19:16.652035 30713 raft_consensus.cc:2468] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 0.
I20260709 12:19:16.654985 30830 tablet_bootstrap.cc:492] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe: Bootstrap starting.
I20260709 12:19:16.659123 30831 tablet_bootstrap.cc:492] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7: No bootstrap required, opened a new log
I20260709 12:19:16.659703 30831 ts_tablet_manager.cc:1403] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent bootstrapping tablet: real 0.119s	user 0.016s	sys 0.005s
I20260709 12:19:16.661353 30832 tablet_bootstrap.cc:654] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.662678 30838 raft_consensus.cc:515] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.665548 30787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb" is_pre_election: true
I20260709 12:19:16.666487 30787 raft_consensus.cc:2468] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 0.
I20260709 12:19:16.668603 30785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb" is_pre_election: true
I20260709 12:19:16.666465 30831 raft_consensus.cc:359] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.669632 30785 raft_consensus.cc:2468] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 0.
I20260709 12:19:16.670140 30831 raft_consensus.cc:385] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.668179 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb" is_pre_election: true
I20260709 12:19:16.672515 30786 raft_consensus.cc:2468] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 0.
I20260709 12:19:16.680130 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" is_pre_election: true
I20260709 12:19:16.671348 30831 raft_consensus.cc:740] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.687181 30638 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" is_pre_election: true
I20260709 12:19:16.687804 30713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:16.691277 30679 leader_election.cc:304] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe; no voters: 
I20260709 12:19:16.687951 30838 leader_election.cc:290] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 election: Requested vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:16.692178 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:16.691624 30636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" is_pre_election: true
I20260709 12:19:16.695226 30837 raft_consensus.cc:2804] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:16.697606 30679 leader_election.cc:304] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe; no voters: 
I20260709 12:19:16.702992 30837 raft_consensus.cc:493] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.703516 30837 raft_consensus.cc:3060] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.703672 30679 leader_election.cc:304] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe; no voters: 
I20260709 12:19:16.698791 30786 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.705065 30636 raft_consensus.cc:2393] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b2657dfca3a04182a3c74dbd5c3e72fe in current term 1: Already voted for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in this term.
I20260709 12:19:16.689822 30637 raft_consensus.cc:2468] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 0.
I20260709 12:19:16.708612 30830 tablet_bootstrap.cc:654] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.696347 30713 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:19:16.711171 30678 leader_election.cc:343] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:16.694770 30831 consensus_queue.cc:260] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.716897 30861 raft_consensus.cc:493] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:16.724242 30831 ts_tablet_manager.cc:1434] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent starting tablet: real 0.064s	user 0.007s	sys 0.003s
I20260709 12:19:16.725451 30831 tablet_bootstrap.cc:492] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7: Bootstrap starting.
I20260709 12:19:16.725998 30832 tablet_bootstrap.cc:492] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb: No bootstrap required, opened a new log
I20260709 12:19:16.726495 30832 ts_tablet_manager.cc:1403] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb: Time spent bootstrapping tablet: real 0.152s	user 0.013s	sys 0.025s
I20260709 12:19:16.728907 30832 raft_consensus.cc:359] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.729503 30832 raft_consensus.cc:385] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.729746 30832 raft_consensus.cc:740] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.717585 30861 raft_consensus.cc:515] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.740530 30863 raft_consensus.cc:2804] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:16.741052 30863 raft_consensus.cc:493] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.741385 30863 raft_consensus.cc:3060] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.748031 30712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4884fe1576a44b67862ba32ea082366a" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" is_pre_election: true
I20260709 12:19:16.749378 30712 raft_consensus.cc:2468] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 0.
I20260709 12:19:16.731060 30832 consensus_queue.cc:260] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.757344 30751 leader_election.cc:304] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe; no voters: 
I20260709 12:19:16.757881 30786 raft_consensus.cc:2468] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 1.
I20260709 12:19:16.758695 30836 raft_consensus.cc:2804] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:16.759007 30836 raft_consensus.cc:493] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.759248 30785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec3ea46dfb242f4b4ce66a97604aceb" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb" is_pre_election: true
I20260709 12:19:16.759778 30836 raft_consensus.cc:3060] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.759918 30604 leader_election.cc:304] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:16.760335 30785 raft_consensus.cc:2468] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 0.
I20260709 12:19:16.755126 30861 leader_election.cc:290] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:16.766614 30712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec3ea46dfb242f4b4ce66a97604aceb" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" is_pre_election: true
I20260709 12:19:16.767527 30712 raft_consensus.cc:2468] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 0.
I20260709 12:19:16.763708 30838 raft_consensus.cc:2804] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:16.770335 30838 raft_consensus.cc:697] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Becoming Leader. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:16.770340 30602 leader_election.cc:304] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2657dfca3a04182a3c74dbd5c3e72fe, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:16.771647 30838 consensus_queue.cc:237] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.755888 30837 raft_consensus.cc:515] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.784171 30636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:16.785286 30785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:16.785856 30785 raft_consensus.cc:3060] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:19:16.786592 30678 leader_election.cc:343] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:16.779536 30861 raft_consensus.cc:2804] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:16.784778 30837 leader_election.cc:290] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:16.795213 30713 raft_consensus.cc:2468] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 1.
I20260709 12:19:16.795753 30831 tablet_bootstrap.cc:654] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.796468 30857 raft_consensus.cc:2764] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:19:16.800940 30832 ts_tablet_manager.cc:1434] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb: Time spent starting tablet: real 0.074s	user 0.007s	sys 0.001s
I20260709 12:19:16.800580 30636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4884fe1576a44b67862ba32ea082366a" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" is_pre_election: true
I20260709 12:19:16.801859 30636 raft_consensus.cc:2468] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 0.
I20260709 12:19:16.789814 30861 raft_consensus.cc:493] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:16.803972 30861 raft_consensus.cc:3060] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.816365 30836 raft_consensus.cc:515] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.817684 30832 tablet_bootstrap.cc:492] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb: Bootstrap starting.
W20260709 12:19:16.822952 30602 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:41949, user_credentials={real_user=slave}} blocked reactor thread for 53309.4us
I20260709 12:19:16.826707 30861 raft_consensus.cc:515] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.828271 30636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4884fe1576a44b67862ba32ea082366a" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:16.828965 30636 raft_consensus.cc:3060] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.829010 30861 leader_election.cc:290] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 election: Requested vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:16.832679 30713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec3ea46dfb242f4b4ce66a97604aceb" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:16.833379 30713 raft_consensus.cc:3060] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.835922 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec3ea46dfb242f4b4ce66a97604aceb" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:16.836880 30786 raft_consensus.cc:3060] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.831858 30535 catalog_manager.cc:5697] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:16.846724 30713 raft_consensus.cc:2468] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 1.
I20260709 12:19:16.848165 30602 leader_election.cc:304] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2657dfca3a04182a3c74dbd5c3e72fe, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:16.849159 30861 raft_consensus.cc:2804] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:16.849673 30861 raft_consensus.cc:697] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Becoming Leader. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:16.850021 30786 raft_consensus.cc:2468] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 1.
I20260709 12:19:16.850554 30861 consensus_queue.cc:237] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.850770 30863 raft_consensus.cc:515] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.852712 30836 leader_election.cc:290] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 1 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949)
I20260709 12:19:16.856495 30830 tablet_bootstrap.cc:492] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe: No bootstrap required, opened a new log
I20260709 12:19:16.856990 30830 ts_tablet_manager.cc:1403] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent bootstrapping tablet: real 0.202s	user 0.012s	sys 0.015s
I20260709 12:19:16.860766 30863 leader_election.cc:290] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:16.861032 30713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4884fe1576a44b67862ba32ea082366a" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:16.861795 30713 raft_consensus.cc:3060] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.862452 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:16.863209 30637 raft_consensus.cc:3060] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.861457 30785 raft_consensus.cc:2468] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 1.
I20260709 12:19:16.854497 30636 raft_consensus.cc:2468] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 1.
I20260709 12:19:16.863935 30830 raft_consensus.cc:359] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.867578 30830 raft_consensus.cc:385] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.868081 30830 raft_consensus.cc:740] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.865061 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:16.870885 30786 raft_consensus.cc:3060] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:16.871274 30679 leader_election.cc:304] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe; no voters: e56eff8cb9af4916a3a0cb1f1077f3c7
I20260709 12:19:16.872954 30863 raft_consensus.cc:2804] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:16.873445 30863 raft_consensus.cc:697] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Becoming Leader. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:16.874261 30830 consensus_queue.cc:260] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.875272 30863 consensus_queue.cc:237] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.877317 30637 raft_consensus.cc:2468] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 1.
I20260709 12:19:16.877902 30830 ts_tablet_manager.cc:1434] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent starting tablet: real 0.021s	user 0.000s	sys 0.009s
I20260709 12:19:16.873992 30752 leader_election.cc:304] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:16.878556 30678 leader_election.cc:304] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2657dfca3a04182a3c74dbd5c3e72fe, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:16.879737 30836 raft_consensus.cc:2804] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:16.880973 30857 raft_consensus.cc:2804] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:16.882429 30857 raft_consensus.cc:697] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Becoming Leader. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:16.888275 30830 tablet_bootstrap.cc:492] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe: Bootstrap starting.
I20260709 12:19:16.896345 30831 tablet_bootstrap.cc:492] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7: No bootstrap required, opened a new log
I20260709 12:19:16.897400 30831 ts_tablet_manager.cc:1403] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent bootstrapping tablet: real 0.172s	user 0.024s	sys 0.013s
I20260709 12:19:16.893821 30832 tablet_bootstrap.cc:654] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.888770 30857 consensus_queue.cc:237] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.907647 30836 raft_consensus.cc:697] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 1 LEADER]: Becoming Leader. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Running, Role: LEADER
I20260709 12:19:16.913568 30836 consensus_queue.cc:237] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.928056 30713 raft_consensus.cc:2468] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 1.
I20260709 12:19:16.946365 30831 raft_consensus.cc:359] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.951683 30831 raft_consensus.cc:385] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.952057 30831 raft_consensus.cc:740] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.952937 30831 consensus_queue.cc:260] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.965396 30830 tablet_bootstrap.cc:654] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.965711 30831 ts_tablet_manager.cc:1434] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent starting tablet: real 0.068s	user 0.012s	sys 0.015s
I20260709 12:19:16.963845 30786 raft_consensus.cc:2468] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 1.
I20260709 12:19:16.966475 30531 catalog_manager.cc:5697] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195). New cstate: current_term: 1 leader_uuid: "07da37de9b1e4baca15164fe424a8fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:16.962031 30532 catalog_manager.cc:5697] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe reported cstate change: term changed from 0 to 1, leader changed from <none> to b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194). New cstate: current_term: 1 leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:16.975404 30831 tablet_bootstrap.cc:492] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7: Bootstrap starting.
I20260709 12:19:16.977192 30832 tablet_bootstrap.cc:492] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb: No bootstrap required, opened a new log
I20260709 12:19:16.977784 30832 ts_tablet_manager.cc:1403] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb: Time spent bootstrapping tablet: real 0.160s	user 0.015s	sys 0.003s
I20260709 12:19:16.981909 30832 raft_consensus.cc:359] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.982997 30832 raft_consensus.cc:385] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:16.983364 30832 raft_consensus.cc:740] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Initialized, Role: FOLLOWER
I20260709 12:19:16.984349 30832 consensus_queue.cc:260] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:16.987246 30832 ts_tablet_manager.cc:1434] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.007s
I20260709 12:19:16.988294 30832 tablet_bootstrap.cc:492] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb: Bootstrap starting.
I20260709 12:19:16.996148 30832 tablet_bootstrap.cc:654] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:16.996148 30831 tablet_bootstrap.cc:654] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:17.007058 30831 tablet_bootstrap.cc:492] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7: No bootstrap required, opened a new log
I20260709 12:19:17.007630 30831 ts_tablet_manager.cc:1403] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent bootstrapping tablet: real 0.035s	user 0.007s	sys 0.006s
I20260709 12:19:17.007841 30535 catalog_manager.cc:5697] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:17.011224 30831 raft_consensus.cc:359] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.012111 30831 raft_consensus.cc:385] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:17.012457 30831 raft_consensus.cc:740] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Initialized, Role: FOLLOWER
I20260709 12:19:17.013289 30830 tablet_bootstrap.cc:492] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe: No bootstrap required, opened a new log
I20260709 12:19:17.014008 30830 ts_tablet_manager.cc:1403] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent bootstrapping tablet: real 0.133s	user 0.009s	sys 0.010s
I20260709 12:19:17.013332 30831 consensus_queue.cc:260] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.016582 30831 ts_tablet_manager.cc:1434] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 12:19:17.017694 30831 tablet_bootstrap.cc:492] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7: Bootstrap starting.
I20260709 12:19:17.017423 30830 raft_consensus.cc:359] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.018306 30830 raft_consensus.cc:385] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:17.018719 30830 raft_consensus.cc:740] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Initialized, Role: FOLLOWER
I20260709 12:19:17.019986 30830 consensus_queue.cc:260] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.019937 30532 catalog_manager.cc:5697] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe reported cstate change: term changed from 0 to 1, leader changed from <none> to b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194). New cstate: current_term: 1 leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:17.023530 30830 ts_tablet_manager.cc:1434] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.006s
I20260709 12:19:17.024782 30830 tablet_bootstrap.cc:492] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe: Bootstrap starting.
I20260709 12:19:17.026142 30831 tablet_bootstrap.cc:654] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:17.034199 30830 tablet_bootstrap.cc:654] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:17.046950 30832 tablet_bootstrap.cc:492] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb: No bootstrap required, opened a new log
I20260709 12:19:17.047528 30832 ts_tablet_manager.cc:1403] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb: Time spent bootstrapping tablet: real 0.059s	user 0.010s	sys 0.004s
I20260709 12:19:17.049463 30831 tablet_bootstrap.cc:492] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7: No bootstrap required, opened a new log
I20260709 12:19:17.049978 30831 ts_tablet_manager.cc:1403] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent bootstrapping tablet: real 0.033s	user 0.019s	sys 0.009s
I20260709 12:19:17.052191 30830 tablet_bootstrap.cc:492] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe: No bootstrap required, opened a new log
I20260709 12:19:17.051791 30832 raft_consensus.cc:359] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.052958 30830 ts_tablet_manager.cc:1403] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent bootstrapping tablet: real 0.028s	user 0.010s	sys 0.011s
I20260709 12:19:17.052982 30832 raft_consensus.cc:385] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:17.053714 30832 raft_consensus.cc:740] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Initialized, Role: FOLLOWER
I20260709 12:19:17.054160 30831 raft_consensus.cc:359] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.055357 30831 raft_consensus.cc:385] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:17.055837 30831 raft_consensus.cc:740] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Initialized, Role: FOLLOWER
I20260709 12:19:17.055613 30832 consensus_queue.cc:260] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.056895 30831 consensus_queue.cc:260] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.057485 30830 raft_consensus.cc:359] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.058525 30830 raft_consensus.cc:385] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:17.058951 30830 raft_consensus.cc:740] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Initialized, Role: FOLLOWER
I20260709 12:19:17.059917 30830 consensus_queue.cc:260] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.062866 30831 ts_tablet_manager.cc:1434] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent starting tablet: real 0.013s	user 0.010s	sys 0.000s
I20260709 12:19:17.064020 30831 tablet_bootstrap.cc:492] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7: Bootstrap starting.
I20260709 12:19:17.064970 30830 ts_tablet_manager.cc:1434] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:19:17.066071 30830 tablet_bootstrap.cc:492] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe: Bootstrap starting.
I20260709 12:19:17.071640 30831 tablet_bootstrap.cc:654] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:17.073421 30830 tablet_bootstrap.cc:654] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:17.080124 30832 ts_tablet_manager.cc:1434] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb: Time spent starting tablet: real 0.032s	user 0.010s	sys 0.000s
I20260709 12:19:17.084102 30832 tablet_bootstrap.cc:492] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb: Bootstrap starting.
I20260709 12:19:17.092166 30861 raft_consensus.cc:493] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.092715 30832 tablet_bootstrap.cc:654] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:17.092826 30861 raft_consensus.cc:515] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.096347 30713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75f2a084d518442e8241188f2d30b856" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" is_pre_election: true
I20260709 12:19:17.097124 30713 raft_consensus.cc:2468] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 0.
I20260709 12:19:17.098583 30602 leader_election.cc:304] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2657dfca3a04182a3c74dbd5c3e72fe, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:17.099558 30838 raft_consensus.cc:2804] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:17.099970 30838 raft_consensus.cc:493] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:17.100369 30838 raft_consensus.cc:3060] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:17.100956 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75f2a084d518442e8241188f2d30b856" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb" is_pre_election: true
W20260709 12:19:17.102562 30604 leader_election.cc:343] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:17.103258 30861 leader_election.cc:290] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:17.106441 30832 tablet_bootstrap.cc:492] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb: No bootstrap required, opened a new log
I20260709 12:19:17.108693 30832 ts_tablet_manager.cc:1403] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb: Time spent bootstrapping tablet: real 0.025s	user 0.014s	sys 0.006s
W20260709 12:19:17.110917 30582 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:19:17.113945 30838 raft_consensus.cc:515] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.117465 30838 leader_election.cc:290] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 election: Requested vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:17.118903 30713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75f2a084d518442e8241188f2d30b856" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:17.119612 30713 raft_consensus.cc:3060] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:17.112015 30832 raft_consensus.cc:359] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.120296 30832 raft_consensus.cc:385] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:17.120616 30832 raft_consensus.cc:740] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Initialized, Role: FOLLOWER
I20260709 12:19:17.122454 30832 consensus_queue.cc:260] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.123091 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75f2a084d518442e8241188f2d30b856" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:17.123754 30786 raft_consensus.cc:3060] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:17.125559 30832 ts_tablet_manager.cc:1434] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 12:19:17.126668 30832 tablet_bootstrap.cc:492] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb: Bootstrap starting.
I20260709 12:19:17.130517 30786 raft_consensus.cc:2468] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 1.
I20260709 12:19:17.131912 30713 raft_consensus.cc:2468] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 1.
I20260709 12:19:17.135023 30602 leader_election.cc:304] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2657dfca3a04182a3c74dbd5c3e72fe, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:17.138787 30838 raft_consensus.cc:2804] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:17.139447 30838 raft_consensus.cc:697] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Becoming Leader. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:17.140403 30838 consensus_queue.cc:237] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.139582 30832 tablet_bootstrap.cc:654] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:17.146728 30831 tablet_bootstrap.cc:492] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7: No bootstrap required, opened a new log
I20260709 12:19:17.147267 30831 ts_tablet_manager.cc:1403] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent bootstrapping tablet: real 0.083s	user 0.024s	sys 0.020s
I20260709 12:19:17.147847 30861 raft_consensus.cc:493] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.148545 30861 raft_consensus.cc:515] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.150076 30831 raft_consensus.cc:359] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.151038 30831 raft_consensus.cc:385] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:17.151345 30831 raft_consensus.cc:740] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Initialized, Role: FOLLOWER
I20260709 12:19:17.152221 30831 consensus_queue.cc:260] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.154541 30713 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" is_pre_election: true
I20260709 12:19:17.154497 30831 ts_tablet_manager.cc:1434] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:19:17.156198 30831 tablet_bootstrap.cc:492] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7: Bootstrap starting.
I20260709 12:19:17.157018 30830 tablet_bootstrap.cc:492] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe: No bootstrap required, opened a new log
I20260709 12:19:17.157765 30830 ts_tablet_manager.cc:1403] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent bootstrapping tablet: real 0.092s	user 0.013s	sys 0.008s
I20260709 12:19:17.157886 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb" is_pre_election: true
I20260709 12:19:17.156986 30861 leader_election.cc:290] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:17.158879 30786 raft_consensus.cc:2393] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in current term 1: Already voted for candidate b2657dfca3a04182a3c74dbd5c3e72fe in this term.
I20260709 12:19:17.160094 30604 leader_election.cc:304] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe
I20260709 12:19:17.161188 30861 raft_consensus.cc:3060] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:17.161047 30830 raft_consensus.cc:359] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.161937 30830 raft_consensus.cc:385] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:17.162281 30830 raft_consensus.cc:740] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Initialized, Role: FOLLOWER
I20260709 12:19:17.162504 30831 tablet_bootstrap.cc:654] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:17.165513 30830 consensus_queue.cc:260] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.169420 30861 raft_consensus.cc:2749] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:17.175956 30830 ts_tablet_manager.cc:1434] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe: Time spent starting tablet: real 0.018s	user 0.003s	sys 0.003s
I20260709 12:19:17.185511 30832 tablet_bootstrap.cc:492] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb: No bootstrap required, opened a new log
I20260709 12:19:17.185954 30831 tablet_bootstrap.cc:492] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7: No bootstrap required, opened a new log
I20260709 12:19:17.186146 30832 ts_tablet_manager.cc:1403] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb: Time spent bootstrapping tablet: real 0.060s	user 0.020s	sys 0.004s
I20260709 12:19:17.186681 30831 ts_tablet_manager.cc:1403] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent bootstrapping tablet: real 0.031s	user 0.010s	sys 0.005s
I20260709 12:19:17.185909 30531 catalog_manager.cc:5697] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:17.189711 30831 raft_consensus.cc:359] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.189601 30832 raft_consensus.cc:359] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.190439 30831 raft_consensus.cc:385] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:17.190560 30832 raft_consensus.cc:385] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:17.190811 30831 raft_consensus.cc:740] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Initialized, Role: FOLLOWER
I20260709 12:19:17.190886 30832 raft_consensus.cc:740] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Initialized, Role: FOLLOWER
I20260709 12:19:17.191576 30831 consensus_queue.cc:260] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.191653 30832 consensus_queue.cc:260] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.193730 30831 ts_tablet_manager.cc:1434] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:19:17.196521 30832 ts_tablet_manager.cc:1434] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:19:17.205981 30838 consensus_queue.cc:1048] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:17.232622 30861 consensus_queue.cc:1048] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:17.261797 30836 raft_consensus.cc:493] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.262326 30836 raft_consensus.cc:515] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.264350 30836 leader_election.cc:290] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949)
I20260709 12:19:17.264294 30861 consensus_queue.cc:1048] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:17.266539 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "498b463cdd8b414c8a752d8c093c49ac" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" is_pre_election: true
I20260709 12:19:17.267793 30637 raft_consensus.cc:2468] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 0.
I20260709 12:19:17.266556 30863 raft_consensus.cc:493] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.267998 30857 consensus_queue.cc:1048] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:17.268497 30863 raft_consensus.cc:515] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.268904 30712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "498b463cdd8b414c8a752d8c093c49ac" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" is_pre_election: true
I20260709 12:19:17.269558 30712 raft_consensus.cc:2468] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 0.
I20260709 12:19:17.270529 30863 leader_election.cc:290] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:17.271452 30636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7b159647196412382fcbeea841f3d93" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" is_pre_election: true
I20260709 12:19:17.271672 30751 leader_election.cc:304] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe; no voters: 
I20260709 12:19:17.274031 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7b159647196412382fcbeea841f3d93" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb" is_pre_election: true
I20260709 12:19:17.272203 30636 raft_consensus.cc:2468] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 0.
I20260709 12:19:17.275848 30786 raft_consensus.cc:2468] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 0.
I20260709 12:19:17.276712 30836 raft_consensus.cc:2804] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:17.276546 30678 leader_election.cc:304] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2657dfca3a04182a3c74dbd5c3e72fe, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:17.277094 30836 raft_consensus.cc:493] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:17.277549 30836 raft_consensus.cc:3060] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:17.278187 30863 raft_consensus.cc:2804] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:17.278738 30863 raft_consensus.cc:493] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:17.279101 30863 raft_consensus.cc:3060] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:17.285959 30836 raft_consensus.cc:515] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.289481 30636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "498b463cdd8b414c8a752d8c093c49ac" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:17.290146 30636 raft_consensus.cc:3060] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:17.294142 30861 raft_consensus.cc:493] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:17.294754 30861 raft_consensus.cc:515] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.298475 30712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "385b71fa8aea425f94dba57ebffc17fe" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" is_pre_election: true
I20260709 12:19:17.299196 30712 raft_consensus.cc:2468] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 0.
I20260709 12:19:17.300251 30602 leader_election.cc:304] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2657dfca3a04182a3c74dbd5c3e72fe, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:17.301306 30865 raft_consensus.cc:2804] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:17.301694 30865 raft_consensus.cc:493] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:17.302062 30865 raft_consensus.cc:3060] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:17.307080 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "385b71fa8aea425f94dba57ebffc17fe" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb" is_pre_election: true
I20260709 12:19:17.307842 30786 raft_consensus.cc:2468] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 0.
I20260709 12:19:17.310233 30861 leader_election.cc:290] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:17.310961 30837 consensus_queue.cc:1048] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:17.285959 30863 raft_consensus.cc:515] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.323148 30836 leader_election.cc:290] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 1 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949)
I20260709 12:19:17.321465 30863 leader_election.cc:290] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:17.323804 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7b159647196412382fcbeea841f3d93" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:17.319062 30636 raft_consensus.cc:2468] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 1.
I20260709 12:19:17.324669 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7b159647196412382fcbeea841f3d93" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:17.322116 30865 raft_consensus.cc:515] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.326798 30637 raft_consensus.cc:3060] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:17.350908 30712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "385b71fa8aea425f94dba57ebffc17fe" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:17.351850 30712 raft_consensus.cc:3060] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:17.332299 30752 leader_election.cc:304] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:17.356177 30870 raft_consensus.cc:2804] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:17.364126 30865 leader_election.cc:290] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 election: Requested vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:17.329486 30711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "498b463cdd8b414c8a752d8c093c49ac" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:17.367508 30711 raft_consensus.cc:3060] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:17.371423 30870 raft_consensus.cc:697] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 1 LEADER]: Becoming Leader. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Running, Role: LEADER
I20260709 12:19:17.372658 30870 consensus_queue.cc:237] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.378119 30785 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "385b71fa8aea425f94dba57ebffc17fe" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:17.378921 30785 raft_consensus.cc:3060] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:17.388767 30863 consensus_queue.cc:1048] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:17.390815 30865 consensus_queue.cc:1048] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:17.395962 30711 raft_consensus.cc:2468] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 1.
I20260709 12:19:17.398180 30712 raft_consensus.cc:2468] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 1.
I20260709 12:19:17.399595 30602 leader_election.cc:304] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2657dfca3a04182a3c74dbd5c3e72fe, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:17.400594 30865 raft_consensus.cc:2804] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:17.401263 30865 raft_consensus.cc:697] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Becoming Leader. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:17.402345 30865 consensus_queue.cc:237] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.328662 30786 raft_consensus.cc:3060] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:17.329777 30836 consensus_queue.cc:1048] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.027s
I20260709 12:19:17.386348 30785 raft_consensus.cc:2468] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 1.
I20260709 12:19:17.421991 30863 consensus_queue.cc:1048] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:17.424783 30637 raft_consensus.cc:2468] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 1.
I20260709 12:19:17.426453 30678 leader_election.cc:304] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2657dfca3a04182a3c74dbd5c3e72fe, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:17.427613 30863 raft_consensus.cc:2804] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:17.428140 30863 raft_consensus.cc:697] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Becoming Leader. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:17.429148 30863 consensus_queue.cc:237] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.435001 30534 catalog_manager.cc:5697] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:17.437948 30786 raft_consensus.cc:2468] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 1.
I20260709 12:19:17.447842 30535 catalog_manager.cc:5697] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe reported cstate change: term changed from 0 to 1, leader changed from <none> to b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194). New cstate: current_term: 1 leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:17.469779 30535 catalog_manager.cc:5697] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb reported cstate change: term changed from 0 to 1, leader changed from <none> to 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195). New cstate: current_term: 1 leader_uuid: "07da37de9b1e4baca15164fe424a8fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:17.471695 30870 consensus_queue.cc:1048] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:17.533882 30865 consensus_queue.cc:1048] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:17.547264 30861 consensus_queue.cc:1048] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:17.762130 30870 consensus_queue.cc:1048] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:17.778422 30870 consensus_queue.cc:1048] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:17.814842 30838 consensus_queue.cc:1048] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:17.817399 30711 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
mode: GRACEFUL
new_leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
 from {username='slave'} at 127.0.0.1:42408
I20260709 12:19:17.818153 30711 raft_consensus.cc:606] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Received request to transfer leadership to TS e56eff8cb9af4916a3a0cb1f1077f3c7
I20260709 12:19:17.831032 30838 consensus_queue.cc:1048] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:17.851975 30786 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4884fe1576a44b67862ba32ea082366a"
dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
mode: GRACEFUL
new_leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
 from {username='slave'} at 127.0.0.1:51866
I20260709 12:19:17.852686 30786 raft_consensus.cc:606] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 1 LEADER]: Received request to transfer leadership to TS e56eff8cb9af4916a3a0cb1f1077f3c7
I20260709 12:19:17.862036 30712 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
mode: GRACEFUL
new_leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
 from {username='slave'} at 127.0.0.1:42408
I20260709 12:19:17.862777 30712 raft_consensus.cc:606] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Received request to transfer leadership to TS e56eff8cb9af4916a3a0cb1f1077f3c7
I20260709 12:19:17.867756 30837 raft_consensus.cc:993] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe: : Instructing follower e56eff8cb9af4916a3a0cb1f1077f3c7 to start an election
I20260709 12:19:17.868251 30863 raft_consensus.cc:1081] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Signalling peer e56eff8cb9af4916a3a0cb1f1077f3c7 to start an election
I20260709 12:19:17.869935 30637 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8"
dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
 from {username='slave'} at 127.0.0.1:38382
I20260709 12:19:17.870581 30637 raft_consensus.cc:493] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:17.871042 30637 raft_consensus.cc:3060] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:17.878733 30637 raft_consensus.cc:515] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.881994 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:17.882848 30786 raft_consensus.cc:3060] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:17.881954 30712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:17.884269 30712 raft_consensus.cc:3055] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:17.884660 30712 raft_consensus.cc:740] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:17.885620 30712 consensus_queue.cc:260] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.886801 30637 leader_election.cc:290] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 2 election: Requested vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:17.886934 30712 raft_consensus.cc:3060] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:17.887538 30636 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7"
dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
mode: GRACEFUL
new_leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:38402
I20260709 12:19:17.888257 30636 raft_consensus.cc:606] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Received request to transfer leadership to TS b2657dfca3a04182a3c74dbd5c3e72fe
I20260709 12:19:17.891091 30786 raft_consensus.cc:2468] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 2.
I20260709 12:19:17.892396 30604 leader_election.cc:304] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:17.893469 30838 raft_consensus.cc:2804] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:17.894003 30838 raft_consensus.cc:697] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Becoming Leader. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:17.894959 30838 consensus_queue.cc:237] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.895591 30712 raft_consensus.cc:2468] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 2.
I20260709 12:19:17.899585 30637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "75f2a084d518442e8241188f2d30b856"
dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
mode: GRACEFUL
new_leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:38402
I20260709 12:19:17.900224 30637 raft_consensus.cc:606] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Received request to transfer leadership to TS b2657dfca3a04182a3c74dbd5c3e72fe
I20260709 12:19:17.903620 30535 catalog_manager.cc:5697] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 reported cstate change: term changed from 1 to 2, leader changed from b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194) to e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193). New cstate: current_term: 2 leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:17.907167 30637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "385b71fa8aea425f94dba57ebffc17fe"
dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
mode: GRACEFUL
new_leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:38402
I20260709 12:19:17.907895 30637 raft_consensus.cc:606] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Received request to transfer leadership to TS b2657dfca3a04182a3c74dbd5c3e72fe
I20260709 12:19:17.915609 30786 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "498b463cdd8b414c8a752d8c093c49ac"
dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
mode: GRACEFUL
new_leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:51866
I20260709 12:19:17.916242 30786 raft_consensus.cc:606] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 1 LEADER]: Received request to transfer leadership to TS b2657dfca3a04182a3c74dbd5c3e72fe
I20260709 12:19:17.922533 30712 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e7b159647196412382fcbeea841f3d93"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
mode: GRACEFUL
new_leader_uuid: "07da37de9b1e4baca15164fe424a8fdb"
 from {username='slave'} at 127.0.0.1:42408
I20260709 12:19:17.923259 30712 raft_consensus.cc:606] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Received request to transfer leadership to TS 07da37de9b1e4baca15164fe424a8fdb
I20260709 12:19:17.940224 30863 consensus_queue.cc:1048] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:17.945082 30897 raft_consensus.cc:993] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb: : Instructing follower e56eff8cb9af4916a3a0cb1f1077f3c7 to start an election
I20260709 12:19:17.945582 30897 raft_consensus.cc:1081] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 1 LEADER]: Signalling peer e56eff8cb9af4916a3a0cb1f1077f3c7 to start an election
I20260709 12:19:17.947373 30636 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4884fe1576a44b67862ba32ea082366a"
dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
 from {username='slave'} at 127.0.0.1:38398
I20260709 12:19:17.947960 30636 raft_consensus.cc:493] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:17.948330 30636 raft_consensus.cc:3060] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:17.961948 30636 raft_consensus.cc:515] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.962263 30857 consensus_queue.cc:1048] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:17.967677 30712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4884fe1576a44b67862ba32ea082366a" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:17.968585 30712 raft_consensus.cc:3060] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:17.967677 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4884fe1576a44b67862ba32ea082366a" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:17.970590 30786 raft_consensus.cc:3055] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:17.970970 30786 raft_consensus.cc:740] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Running, Role: LEADER
I20260709 12:19:17.971793 30786 consensus_queue.cc:260] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.972895 30786 raft_consensus.cc:3060] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:17.977671 30712 raft_consensus.cc:2468] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 2.
I20260709 12:19:17.979000 30602 leader_election.cc:304] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2657dfca3a04182a3c74dbd5c3e72fe, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:17.979805 30838 raft_consensus.cc:2804] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:17.980286 30838 raft_consensus.cc:697] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Becoming Leader. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:17.981122 30838 consensus_queue.cc:237] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.984634 30857 raft_consensus.cc:993] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe: : Instructing follower 07da37de9b1e4baca15164fe424a8fdb to start an election
I20260709 12:19:17.985107 30857 raft_consensus.cc:1081] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Signalling peer 07da37de9b1e4baca15164fe424a8fdb to start an election
I20260709 12:19:17.987056 30787 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e7b159647196412382fcbeea841f3d93"
dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
 from {username='slave'} at 127.0.0.1:51864
I20260709 12:19:17.987587 30636 leader_election.cc:290] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 2 election: Requested vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:17.987671 30787 raft_consensus.cc:493] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:17.988546 30787 raft_consensus.cc:3060] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:17.991420 30786 raft_consensus.cc:2468] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 2.
I20260709 12:19:17.995849 30787 raft_consensus.cc:515] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:17.998922 30636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7b159647196412382fcbeea841f3d93" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:17.998557 30535 catalog_manager.cc:5697] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 reported cstate change: term changed from 1 to 2, leader changed from 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195) to e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193). New cstate: current_term: 2 leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:17.999521 30636 raft_consensus.cc:3060] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.001842 30787 leader_election.cc:290] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 2 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949)
I20260709 12:19:18.003098 30712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e7b159647196412382fcbeea841f3d93" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:18.003690 30712 raft_consensus.cc:3055] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:18.004016 30712 raft_consensus.cc:740] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:18.004788 30712 consensus_queue.cc:260] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.006042 30712 raft_consensus.cc:3060] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.009443 30636 raft_consensus.cc:2468] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 2.
I20260709 12:19:18.010838 30752 leader_election.cc:304] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:18.011698 30897 raft_consensus.cc:2804] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:18.012212 30897 raft_consensus.cc:697] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [term 2 LEADER]: Becoming Leader. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Running, Role: LEADER
I20260709 12:19:18.013055 30897 consensus_queue.cc:237] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.015051 30712 raft_consensus.cc:2468] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 2.
I20260709 12:19:18.020779 30535 catalog_manager.cc:5697] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb reported cstate change: term changed from 1 to 2, leader changed from b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194) to 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195). New cstate: current_term: 2 leader_uuid: "07da37de9b1e4baca15164fe424a8fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:18.159337 30861 raft_consensus.cc:993] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7: : Instructing follower b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:18.159909 30861 raft_consensus.cc:1081] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Signalling peer b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:18.161700 30711 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "75f2a084d518442e8241188f2d30b856"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:42388
I20260709 12:19:18.162317 30711 raft_consensus.cc:493] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:18.162719 30711 raft_consensus.cc:3060] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.162796 30861 raft_consensus.cc:993] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7: : Instructing follower b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:18.163280 30861 raft_consensus.cc:1081] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Signalling peer b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:18.165282 30712 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:42388
I20260709 12:19:18.165928 30712 raft_consensus.cc:493] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:18.166422 30712 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.170672 30711 raft_consensus.cc:515] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.173614 30711 leader_election.cc:290] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 2 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:18.174243 30787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75f2a084d518442e8241188f2d30b856" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:18.174190 30636 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75f2a084d518442e8241188f2d30b856" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:18.175030 30787 raft_consensus.cc:3060] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.175318 30636 raft_consensus.cc:3055] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:18.175874 30636 raft_consensus.cc:740] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:18.175302 30712 raft_consensus.cc:515] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.176731 30636 consensus_queue.cc:260] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.178000 30636 raft_consensus.cc:3060] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.179034 30712 leader_election.cc:290] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 2 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:18.178794 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:18.180027 30637 raft_consensus.cc:3055] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:18.180370 30637 raft_consensus.cc:740] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:18.180270 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:18.180919 30786 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.181258 30637 consensus_queue.cc:260] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.182379 30637 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.186554 30636 raft_consensus.cc:2468] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 2.
I20260709 12:19:18.187920 30786 raft_consensus.cc:2468] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 2.
I20260709 12:19:18.188128 30678 leader_election.cc:304] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2657dfca3a04182a3c74dbd5c3e72fe, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:18.188980 30857 raft_consensus.cc:2804] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:18.189507 30857 raft_consensus.cc:697] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Becoming Leader. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:18.189636 30679 leader_election.cc:304] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe; no voters: 
I20260709 12:19:18.190709 30837 raft_consensus.cc:2804] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:18.190424 30857 consensus_queue.cc:237] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.191201 30837 raft_consensus.cc:697] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Becoming Leader. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:18.192036 30837 consensus_queue.cc:237] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.193840 30787 raft_consensus.cc:2468] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 2.
I20260709 12:19:18.196810 30637 raft_consensus.cc:2468] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 2.
I20260709 12:19:18.199702 30535 catalog_manager.cc:5697] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe reported cstate change: term changed from 1 to 2, leader changed from e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193) to b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194). New cstate: current_term: 2 leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:18.211227 30870 raft_consensus.cc:993] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb: : Instructing follower b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:18.211758 30897 raft_consensus.cc:1081] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 1 LEADER]: Signalling peer b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:18.213477 30712 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "498b463cdd8b414c8a752d8c093c49ac"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:42392
I20260709 12:19:18.214025 30712 raft_consensus.cc:493] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:18.214363 30712 raft_consensus.cc:3060] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.215124 30535 catalog_manager.cc:5697] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe reported cstate change: term changed from 1 to 2, leader changed from e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193) to b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194). New cstate: current_term: 2 leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:18.220271 30712 raft_consensus.cc:515] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.222710 30712 leader_election.cc:290] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 2 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:18.224956 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "498b463cdd8b414c8a752d8c093c49ac" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:18.225769 30637 raft_consensus.cc:3060] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.225705 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "498b463cdd8b414c8a752d8c093c49ac" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:18.226652 30786 raft_consensus.cc:3055] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:18.227051 30786 raft_consensus.cc:740] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Running, Role: LEADER
I20260709 12:19:18.227918 30786 consensus_queue.cc:260] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.229061 30786 raft_consensus.cc:3060] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.234717 30786 raft_consensus.cc:2468] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 2.
I20260709 12:19:18.235885 30679 leader_election.cc:304] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe; no voters: 
I20260709 12:19:18.236651 30857 raft_consensus.cc:2804] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:18.237154 30857 raft_consensus.cc:697] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Becoming Leader. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:18.237969 30857 consensus_queue.cc:237] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.240965 30637 raft_consensus.cc:2468] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 2.
I20260709 12:19:18.245554 30535 catalog_manager.cc:5697] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe reported cstate change: term changed from 1 to 2, leader changed from 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195) to b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194). New cstate: current_term: 2 leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:18.283828 30712 raft_consensus.cc:1275] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Refusing update from remote peer e56eff8cb9af4916a3a0cb1f1077f3c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.285353 30838 consensus_queue.cc:1048] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:18.293411 30786 raft_consensus.cc:1275] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Refusing update from remote peer e56eff8cb9af4916a3a0cb1f1077f3c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.295152 30865 consensus_queue.cc:1048] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:18.301005 30857 raft_consensus.cc:993] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe: : Instructing follower e56eff8cb9af4916a3a0cb1f1077f3c7 to start an election
I20260709 12:19:18.301513 30857 raft_consensus.cc:1081] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Signalling peer e56eff8cb9af4916a3a0cb1f1077f3c7 to start an election
I20260709 12:19:18.303452 30637 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6"
dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
 from {username='slave'} at 127.0.0.1:38382
I20260709 12:19:18.304044 30637 raft_consensus.cc:493] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:18.304435 30637 raft_consensus.cc:3060] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.312620 30637 raft_consensus.cc:515] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.314913 30637 leader_election.cc:290] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 2 election: Requested vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:18.315793 30712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:18.316028 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:18.316684 30786 raft_consensus.cc:3060] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.316598 30712 raft_consensus.cc:3055] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:18.319581 30712 raft_consensus.cc:740] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:18.320622 30712 consensus_queue.cc:260] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.321943 30712 raft_consensus.cc:3060] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.326747 30786 raft_consensus.cc:2468] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 2.
I20260709 12:19:18.328068 30604 leader_election.cc:304] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:18.328948 30865 raft_consensus.cc:2804] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:18.329471 30865 raft_consensus.cc:697] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Becoming Leader. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:18.329582 30712 raft_consensus.cc:2468] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 2.
I20260709 12:19:18.330374 30865 consensus_queue.cc:237] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.339282 30531 catalog_manager.cc:5697] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 reported cstate change: term changed from 1 to 2, leader changed from b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194) to e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193). New cstate: current_term: 2 leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:18.378969 30861 raft_consensus.cc:993] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7: : Instructing follower b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:18.379736 30865 raft_consensus.cc:1081] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Signalling peer b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:18.382344 30712 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "385b71fa8aea425f94dba57ebffc17fe"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:42388
I20260709 12:19:18.383206 30712 raft_consensus.cc:493] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:18.383670 30712 raft_consensus.cc:3060] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.393676 30712 raft_consensus.cc:515] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.396795 30712 leader_election.cc:290] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 2 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:18.397562 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "385b71fa8aea425f94dba57ebffc17fe" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:18.397956 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "385b71fa8aea425f94dba57ebffc17fe" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:18.398213 30637 raft_consensus.cc:3055] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:18.398658 30637 raft_consensus.cc:740] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:18.398622 30786 raft_consensus.cc:3060] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.399690 30637 consensus_queue.cc:260] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.401105 30637 raft_consensus.cc:3060] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:18.406800 30786 raft_consensus.cc:2468] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 2.
I20260709 12:19:18.408149 30679 leader_election.cc:304] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe; no voters: 
I20260709 12:19:18.408464 30637 raft_consensus.cc:2468] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 2.
I20260709 12:19:18.409194 30837 raft_consensus.cc:2804] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:18.409766 30837 raft_consensus.cc:697] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Becoming Leader. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:18.410686 30837 consensus_queue.cc:237] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:18.418725 30531 catalog_manager.cc:5697] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe reported cstate change: term changed from 1 to 2, leader changed from e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193) to b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194). New cstate: current_term: 2 leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:18.483270 30712 raft_consensus.cc:1275] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Refusing update from remote peer e56eff8cb9af4916a3a0cb1f1077f3c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.484539 30861 consensus_queue.cc:1048] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:18.491334 30711 raft_consensus.cc:1275] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Refusing update from remote peer 07da37de9b1e4baca15164fe424a8fdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.492873 30897 consensus_queue.cc:1048] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:18.497789 30786 raft_consensus.cc:1275] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Refusing update from remote peer e56eff8cb9af4916a3a0cb1f1077f3c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.500069 30838 consensus_queue.cc:1048] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:18.503876 30637 raft_consensus.cc:1275] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Refusing update from remote peer 07da37de9b1e4baca15164fe424a8fdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.505692 30919 consensus_queue.cc:1048] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:18.649019 30637 raft_consensus.cc:1275] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.650533 30837 consensus_queue.cc:1048] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:18.659647 30786 raft_consensus.cc:1275] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.663125 30922 consensus_queue.cc:1048] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:18.703886 30637 raft_consensus.cc:1275] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.705336 30922 consensus_queue.cc:1048] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:18.714576 30786 raft_consensus.cc:1275] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.716197 30922 consensus_queue.cc:1048] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:18.766053 30786 raft_consensus.cc:1275] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Refusing update from remote peer e56eff8cb9af4916a3a0cb1f1077f3c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.767395 30838 consensus_queue.cc:1048] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:18.776279 30712 raft_consensus.cc:1275] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Refusing update from remote peer e56eff8cb9af4916a3a0cb1f1077f3c7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.777952 30865 consensus_queue.cc:1048] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:18.836450 30637 raft_consensus.cc:1275] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.837970 30922 consensus_queue.cc:1048] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:18.848238 30786 raft_consensus.cc:1275] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.850263 30922 consensus_queue.cc:1048] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:19:18.949887 30637 raft_consensus.cc:1275] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.951373 30922 consensus_queue.cc:1048] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:18.960080 30786 raft_consensus.cc:1275] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:18.961814 30857 consensus_queue.cc:1048] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:20.954250 30503 auto_leader_rebalancer-test.cc:294] The leader distribution is 
I20260709 12:19:20.955271 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:20.978117 30712 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
mode: GRACEFUL
new_leader_uuid: "07da37de9b1e4baca15164fe424a8fdb"
 from {username='slave'} at 127.0.0.1:42414
I20260709 12:19:20.978686 30712 raft_consensus.cc:606] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Received request to transfer leadership to TS 07da37de9b1e4baca15164fe424a8fdb
I20260709 12:19:20.991433 30637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eec3ea46dfb242f4b4ce66a97604aceb"
dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
mode: GRACEFUL
new_leader_uuid: "07da37de9b1e4baca15164fe424a8fdb"
 from {username='slave'} at 127.0.0.1:38412
I20260709 12:19:20.992002 30637 raft_consensus.cc:606] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Received request to transfer leadership to TS 07da37de9b1e4baca15164fe424a8fdb
I20260709 12:19:20.993124 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:19:20.993628 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:21.155061 30936 raft_consensus.cc:993] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe: : Instructing follower 07da37de9b1e4baca15164fe424a8fdb to start an election
I20260709 12:19:21.155613 30933 raft_consensus.cc:1081] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Signalling peer 07da37de9b1e4baca15164fe424a8fdb to start an election
I20260709 12:19:21.157255 30786 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7"
dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
 from {username='slave'} at 127.0.0.1:51864
I20260709 12:19:21.157737 30786 raft_consensus.cc:493] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:21.158017 30786 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:21.162274 30786 raft_consensus.cc:515] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:21.163992 30786 leader_election.cc:290] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 3 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949)
I20260709 12:19:21.164917 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:21.165191 30712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:21.165563 30637 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:21.165688 30712 raft_consensus.cc:3055] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:21.166023 30712 raft_consensus.cc:740] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:21.166919 30712 consensus_queue.cc:260] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:21.168107 30712 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:21.172821 30637 raft_consensus.cc:2468] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 3.
I20260709 12:19:21.174274 30752 leader_election.cc:304] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:21.174368 30712 raft_consensus.cc:2468] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 3.
I20260709 12:19:21.175232 30935 raft_consensus.cc:2804] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:21.176596 30935 raft_consensus.cc:697] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 3 LEADER]: Becoming Leader. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Running, Role: LEADER
I20260709 12:19:21.177416 30935 consensus_queue.cc:237] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:21.184332 30534 catalog_manager.cc:5697] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb reported cstate change: term changed from 2 to 3, leader changed from b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194) to 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195). New cstate: current_term: 3 leader_uuid: "07da37de9b1e4baca15164fe424a8fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:21.362032 30932 raft_consensus.cc:993] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7: : Instructing follower 07da37de9b1e4baca15164fe424a8fdb to start an election
I20260709 12:19:21.362480 30932 raft_consensus.cc:1081] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Signalling peer 07da37de9b1e4baca15164fe424a8fdb to start an election
I20260709 12:19:21.364087 30786 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eec3ea46dfb242f4b4ce66a97604aceb"
dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
 from {username='slave'} at 127.0.0.1:51862
I20260709 12:19:21.364655 30786 raft_consensus.cc:493] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:21.364948 30786 raft_consensus.cc:3060] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:21.369477 30786 raft_consensus.cc:515] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:21.371088 30786 leader_election.cc:290] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 2 election: Requested vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449)
I20260709 12:19:21.371956 30712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec3ea46dfb242f4b4ce66a97604aceb" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:21.372129 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec3ea46dfb242f4b4ce66a97604aceb" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:21.372536 30712 raft_consensus.cc:3060] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:21.372637 30637 raft_consensus.cc:3055] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:21.372984 30637 raft_consensus.cc:740] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:21.373771 30637 consensus_queue.cc:260] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:21.375001 30637 raft_consensus.cc:3060] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:21.379226 30712 raft_consensus.cc:2468] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 2.
I20260709 12:19:21.381031 30751 leader_election.cc:304] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe; no voters: 
I20260709 12:19:21.381887 30935 raft_consensus.cc:2804] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:21.382350 30935 raft_consensus.cc:697] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 2 LEADER]: Becoming Leader. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Running, Role: LEADER
I20260709 12:19:21.382440 30637 raft_consensus.cc:2468] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 2.
I20260709 12:19:21.383203 30935 consensus_queue.cc:237] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:21.390027 30534 catalog_manager.cc:5697] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb reported cstate change: term changed from 1 to 2, leader changed from e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193) to 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195). New cstate: current_term: 2 leader_uuid: "07da37de9b1e4baca15164fe424a8fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:21.648125 30712 raft_consensus.cc:1275] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Refusing update from remote peer 07da37de9b1e4baca15164fe424a8fdb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:21.649516 30935 consensus_queue.cc:1048] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:21.664696 30637 raft_consensus.cc:1275] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Refusing update from remote peer 07da37de9b1e4baca15164fe424a8fdb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:21.666092 30951 consensus_queue.cc:1048] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:21.830209 30712 raft_consensus.cc:1275] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Refusing update from remote peer 07da37de9b1e4baca15164fe424a8fdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:21.831676 30951 consensus_queue.cc:1048] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:21.841435 30637 raft_consensus.cc:1275] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Refusing update from remote peer 07da37de9b1e4baca15164fe424a8fdb: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:21.843730 30940 consensus_queue.cc:1048] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:21.994678 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:21.999797 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:22.000254 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:23.001331 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:23.007768 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:23.008368 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:24.009965 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:24.016211 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:24.016614 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:25.017688 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:25.022783 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:25.023219 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:26.024430 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:26.029925 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:26.030453 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:27.031594 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:27.036857 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:27.037374 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:28.038393 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:28.043496 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:28.043946 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:29.044955 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:29.049975 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:29.050422 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:30.051398 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:30.057415 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:30.057940 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:31.087759 30503 auto_leader_rebalancer-test.cc:313] The leader distribution is 
I20260709 12:19:31.092520 30786 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eec3ea46dfb242f4b4ce66a97604aceb"
dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
mode: GRACEFUL
new_leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:51866
I20260709 12:19:31.093165 30786 raft_consensus.cc:606] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 2 LEADER]: Received request to transfer leadership to TS b2657dfca3a04182a3c74dbd5c3e72fe
I20260709 12:19:31.098361 30637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6"
dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
mode: GRACEFUL
new_leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:38402
I20260709 12:19:31.098901 30637 raft_consensus.cc:606] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Received request to transfer leadership to TS b2657dfca3a04182a3c74dbd5c3e72fe
I20260709 12:19:31.104038 30637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4884fe1576a44b67862ba32ea082366a"
dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
mode: GRACEFUL
new_leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:38402
I20260709 12:19:31.104518 30637 raft_consensus.cc:606] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Received request to transfer leadership to TS b2657dfca3a04182a3c74dbd5c3e72fe
I20260709 12:19:31.109259 30637 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8"
dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
mode: GRACEFUL
new_leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:38402
I20260709 12:19:31.109735 30637 raft_consensus.cc:606] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Received request to transfer leadership to TS b2657dfca3a04182a3c74dbd5c3e72fe
I20260709 12:19:31.114801 30786 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7"
dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
mode: GRACEFUL
new_leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:51866
I20260709 12:19:31.115413 30786 raft_consensus.cc:606] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 3 LEADER]: Received request to transfer leadership to TS b2657dfca3a04182a3c74dbd5c3e72fe
I20260709 12:19:31.171293 30956 raft_consensus.cc:993] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7: : Instructing follower b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:31.171845 30964 raft_consensus.cc:1081] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Signalling peer b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:31.173725 30712 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:42388
I20260709 12:19:31.174206 30712 raft_consensus.cc:493] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:31.174501 30712 raft_consensus.cc:3060] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.179804 30712 raft_consensus.cc:515] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.182204 30712 leader_election.cc:290] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 3 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:31.183002 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:31.183341 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a19c6e5d902745869661b7ee64f329e8" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:31.183924 30637 raft_consensus.cc:3055] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:31.184113 30786 raft_consensus.cc:3060] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.184305 30637 raft_consensus.cc:740] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:31.184552 30965 raft_consensus.cc:993] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb: : Instructing follower b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:31.185232 30958 raft_consensus.cc:1081] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 2 LEADER]: Signalling peer b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:31.185338 30637 consensus_queue.cc:260] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.186582 30637 raft_consensus.cc:3060] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.187083 30712 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eec3ea46dfb242f4b4ce66a97604aceb"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:42392
I20260709 12:19:31.187613 30712 raft_consensus.cc:493] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:31.187968 30712 raft_consensus.cc:3060] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.189954 30786 raft_consensus.cc:2468] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 3.
I20260709 12:19:31.191224 30679 leader_election.cc:304] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe; no voters: 
I20260709 12:19:31.192060 30963 raft_consensus.cc:2804] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:31.192536 30963 raft_consensus.cc:697] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 LEADER]: Becoming Leader. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:31.193501 30637 raft_consensus.cc:2468] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 3.
I20260709 12:19:31.193562 30963 consensus_queue.cc:237] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.194283 30712 raft_consensus.cc:515] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.196177 30712 leader_election.cc:290] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 3 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:31.197036 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec3ea46dfb242f4b4ce66a97604aceb" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:31.197677 30637 raft_consensus.cc:3060] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.198374 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec3ea46dfb242f4b4ce66a97604aceb" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:31.199009 30786 raft_consensus.cc:3055] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:31.199445 30786 raft_consensus.cc:740] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Running, Role: LEADER
I20260709 12:19:31.200181 30786 consensus_queue.cc:260] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.201167 30786 raft_consensus.cc:3060] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.204540 30637 raft_consensus.cc:2468] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 3.
I20260709 12:19:31.204007 30531 catalog_manager.cc:5697] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe reported cstate change: term changed from 2 to 3, leader changed from e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193) to b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194). New cstate: current_term: 3 leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:31.205801 30786 raft_consensus.cc:2468] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 3.
I20260709 12:19:31.206019 30678 leader_election.cc:304] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2657dfca3a04182a3c74dbd5c3e72fe, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:31.206947 30963 raft_consensus.cc:2804] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:31.207495 30963 raft_consensus.cc:697] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 LEADER]: Becoming Leader. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:31.208179 30963 consensus_queue.cc:237] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.224097 30531 catalog_manager.cc:5697] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe reported cstate change: term changed from 2 to 3, leader changed from 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195) to b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194). New cstate: current_term: 3 leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:31.411900 30956 raft_consensus.cc:993] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7: : Instructing follower b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:31.412405 30956 raft_consensus.cc:1081] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Signalling peer b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:31.413869 30712 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:42388
I20260709 12:19:31.414323 30712 raft_consensus.cc:493] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:31.414597 30712 raft_consensus.cc:3060] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.419494 30712 raft_consensus.cc:515] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.421461 30712 leader_election.cc:290] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 3 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:31.422382 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:31.422384 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:31.423218 30637 raft_consensus.cc:3055] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:31.423262 30786 raft_consensus.cc:3060] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.423663 30637 raft_consensus.cc:740] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:31.424440 30637 consensus_queue.cc:260] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.425385 30637 raft_consensus.cc:3060] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.428206 30786 raft_consensus.cc:2468] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 3.
I20260709 12:19:31.429262 30679 leader_election.cc:304] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe; no voters: 
I20260709 12:19:31.429970 30963 raft_consensus.cc:2804] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:31.430466 30637 raft_consensus.cc:2468] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 3.
I20260709 12:19:31.430595 30963 raft_consensus.cc:697] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 LEADER]: Becoming Leader. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:31.431610 30963 consensus_queue.cc:237] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.439316 30531 catalog_manager.cc:5697] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe reported cstate change: term changed from 2 to 3, leader changed from e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193) to b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194). New cstate: current_term: 3 leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:31.448251 30964 raft_consensus.cc:993] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7: : Instructing follower b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:31.449214 30956 raft_consensus.cc:1081] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Signalling peer b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:31.450799 30712 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4884fe1576a44b67862ba32ea082366a"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:42388
I20260709 12:19:31.451275 30712 raft_consensus.cc:493] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:31.451574 30712 raft_consensus.cc:3060] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.456089 30712 raft_consensus.cc:515] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.457618 30712 leader_election.cc:290] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 3 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:31.458685 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4884fe1576a44b67862ba32ea082366a" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:31.458724 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4884fe1576a44b67862ba32ea082366a" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:31.459246 30637 raft_consensus.cc:3055] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:31.459297 30786 raft_consensus.cc:3060] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.459540 30637 raft_consensus.cc:740] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:31.460225 30637 consensus_queue.cc:260] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.461054 30637 raft_consensus.cc:3060] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:31.463814 30786 raft_consensus.cc:2468] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 3.
I20260709 12:19:31.464816 30679 leader_election.cc:304] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, b2657dfca3a04182a3c74dbd5c3e72fe; no voters: 
I20260709 12:19:31.464990 30637 raft_consensus.cc:2468] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 3.
I20260709 12:19:31.465514 30963 raft_consensus.cc:2804] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:31.466018 30963 raft_consensus.cc:697] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 LEADER]: Becoming Leader. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:31.466826 30963 consensus_queue.cc:237] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.474773 30531 catalog_manager.cc:5697] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe reported cstate change: term changed from 2 to 3, leader changed from e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193) to b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194). New cstate: current_term: 3 leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:31.578761 30965 raft_consensus.cc:993] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb: : Instructing follower b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:31.579242 30958 raft_consensus.cc:1081] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 3 LEADER]: Signalling peer b2657dfca3a04182a3c74dbd5c3e72fe to start an election
I20260709 12:19:31.580755 30712 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
 from {username='slave'} at 127.0.0.1:42392
I20260709 12:19:31.581252 30712 raft_consensus.cc:493] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:31.581604 30712 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:31.586724 30712 raft_consensus.cc:515] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.589115 30712 leader_election.cc:290] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 4 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:31.590041 30786 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:31.590051 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:31.590927 30786 raft_consensus.cc:3055] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:19:31.591190 30637 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:31.591279 30786 raft_consensus.cc:740] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Running, Role: LEADER
I20260709 12:19:31.592084 30786 consensus_queue.cc:260] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.593236 30786 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:31.597718 30637 raft_consensus.cc:2468] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 4.
I20260709 12:19:31.598985 30678 leader_election.cc:304] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b2657dfca3a04182a3c74dbd5c3e72fe, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:31.599489 30786 raft_consensus.cc:2468] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2657dfca3a04182a3c74dbd5c3e72fe in term 4.
I20260709 12:19:31.599924 30963 raft_consensus.cc:2804] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:19:31.601405 30963 raft_consensus.cc:697] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 4 LEADER]: Becoming Leader. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:31.602486 30637 raft_consensus.cc:1275] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:31.602281 30963 consensus_queue.cc:237] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:31.603834 30975 consensus_queue.cc:1048] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:31.610956 30531 catalog_manager.cc:5697] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe reported cstate change: term changed from 3 to 4, leader changed from 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195) to b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194). New cstate: current_term: 4 leader_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:31.615559 30786 raft_consensus.cc:1275] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:31.616938 30933 consensus_queue.cc:1048] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:31.767444 30637 raft_consensus.cc:1275] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:31.768641 30933 consensus_queue.cc:1048] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:31.777475 30786 raft_consensus.cc:1275] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:31.780006 30975 consensus_queue.cc:1048] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:31.814347 30786 raft_consensus.cc:1275] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:31.815691 30975 consensus_queue.cc:1048] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:31.825462 30637 raft_consensus.cc:1275] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:31.827301 30985 consensus_queue.cc:1048] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:31.991817 30786 raft_consensus.cc:1275] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:31.991820 30637 raft_consensus.cc:1275] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:31.993136 30985 consensus_queue.cc:1048] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:31.993781 30933 consensus_queue.cc:1048] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:32.096824 30787 raft_consensus.cc:1275] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 4 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:19:32.098014 30933 consensus_queue.cc:1048] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:32.107537 30637 raft_consensus.cc:1275] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 4 FOLLOWER]: Refusing update from remote peer b2657dfca3a04182a3c74dbd5c3e72fe: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:19:32.109436 30933 consensus_queue.cc:1048] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:34.160676 30503 auto_leader_rebalancer-test.cc:328] The leader distribution is 
I20260709 12:19:34.161583 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:34.169093 30712 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "385b71fa8aea425f94dba57ebffc17fe"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
mode: GRACEFUL
new_leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
 from {username='slave'} at 127.0.0.1:42414
I20260709 12:19:34.169977 30712 raft_consensus.cc:606] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Received request to transfer leadership to TS e56eff8cb9af4916a3a0cb1f1077f3c7
I20260709 12:19:34.172950 30712 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "498b463cdd8b414c8a752d8c093c49ac"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
mode: GRACEFUL
new_leader_uuid: "07da37de9b1e4baca15164fe424a8fdb"
 from {username='slave'} at 127.0.0.1:42414
I20260709 12:19:34.173537 30712 raft_consensus.cc:606] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Received request to transfer leadership to TS 07da37de9b1e4baca15164fe424a8fdb
I20260709 12:19:34.175832 30712 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
mode: GRACEFUL
new_leader_uuid: "07da37de9b1e4baca15164fe424a8fdb"
 from {username='slave'} at 127.0.0.1:42414
I20260709 12:19:34.176317 30712 raft_consensus.cc:606] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 LEADER]: Received request to transfer leadership to TS 07da37de9b1e4baca15164fe424a8fdb
I20260709 12:19:34.178653 30712 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
mode: GRACEFUL
new_leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
 from {username='slave'} at 127.0.0.1:42414
I20260709 12:19:34.179147 30712 raft_consensus.cc:606] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 4 LEADER]: Received request to transfer leadership to TS e56eff8cb9af4916a3a0cb1f1077f3c7
I20260709 12:19:34.181478 30712 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "75f2a084d518442e8241188f2d30b856"
dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
mode: GRACEFUL
new_leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
 from {username='slave'} at 127.0.0.1:42414
I20260709 12:19:34.181926 30712 raft_consensus.cc:606] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Received request to transfer leadership to TS e56eff8cb9af4916a3a0cb1f1077f3c7
I20260709 12:19:34.183144 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:19:34.183621 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:34.196054 30993 raft_consensus.cc:993] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe: : Instructing follower 07da37de9b1e4baca15164fe424a8fdb to start an election
I20260709 12:19:34.196413 30933 raft_consensus.cc:993] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe: : Instructing follower e56eff8cb9af4916a3a0cb1f1077f3c7 to start an election
I20260709 12:19:34.196820 30992 raft_consensus.cc:1081] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Signalling peer 07da37de9b1e4baca15164fe424a8fdb to start an election
I20260709 12:19:34.196846 30993 raft_consensus.cc:1081] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 4 LEADER]: Signalling peer e56eff8cb9af4916a3a0cb1f1077f3c7 to start an election
I20260709 12:19:34.198836 30636 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7"
dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
 from {username='slave'} at 127.0.0.1:38382
I20260709 12:19:34.198829 30787 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "498b463cdd8b414c8a752d8c093c49ac"
dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
 from {username='slave'} at 127.0.0.1:51864
I20260709 12:19:34.199398 30636 raft_consensus.cc:493] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:34.199564 30787 raft_consensus.cc:493] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:34.199790 30636 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:34.200039 30787 raft_consensus.cc:3060] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.205927 30787 raft_consensus.cc:515] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.205927 30636 raft_consensus.cc:515] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.209825 30787 leader_election.cc:290] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 3 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949)
I20260709 12:19:34.210057 30712 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:34.211091 30712 raft_consensus.cc:3055] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:19:34.210171 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "498b463cdd8b414c8a752d8c093c49ac" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:34.211684 30637 raft_consensus.cc:3060] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.211520 30712 raft_consensus.cc:740] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:34.212001 30787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e82899b14fc4d10a58ca1f42c33e4f7" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:34.212663 30787 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:34.212517 30711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "498b463cdd8b414c8a752d8c093c49ac" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:34.213243 30711 raft_consensus.cc:3055] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:34.213025 30712 consensus_queue.cc:260] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.213567 30711 raft_consensus.cc:740] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:34.214260 30712 raft_consensus.cc:3060] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:19:34.214437 30711 consensus_queue.cc:260] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.215925 30711 raft_consensus.cc:3060] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.211441 30636 leader_election.cc:290] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 5 election: Requested vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:34.219755 30637 raft_consensus.cc:2468] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 3.
I20260709 12:19:34.219755 30787 raft_consensus.cc:2468] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 5.
I20260709 12:19:34.221393 30752 leader_election.cc:304] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:34.222379 30958 raft_consensus.cc:2804] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:34.222826 30712 raft_consensus.cc:2468] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 5.
I20260709 12:19:34.222922 30604 leader_election.cc:304] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:34.224483 30958 raft_consensus.cc:697] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 3 LEADER]: Becoming Leader. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Running, Role: LEADER
I20260709 12:19:34.224457 30711 raft_consensus.cc:2468] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 3.
I20260709 12:19:34.225649 30996 raft_consensus.cc:2804] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:19:34.225734 30958 consensus_queue.cc:237] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.227437 30996 raft_consensus.cc:697] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 5 LEADER]: Becoming Leader. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:34.228382 30996 consensus_queue.cc:237] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.236184 30534 catalog_manager.cc:5697] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb reported cstate change: term changed from 2 to 3, leader changed from b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194) to 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195). New cstate: current_term: 3 leader_uuid: "07da37de9b1e4baca15164fe424a8fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:34.237738 30535 catalog_manager.cc:5697] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 reported cstate change: term changed from 4 to 5, leader changed from b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194) to e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193). New cstate: current_term: 5 leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:34.351557 30992 raft_consensus.cc:993] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe: : Instructing follower e56eff8cb9af4916a3a0cb1f1077f3c7 to start an election
I20260709 12:19:34.352054 30993 raft_consensus.cc:1081] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Signalling peer e56eff8cb9af4916a3a0cb1f1077f3c7 to start an election
I20260709 12:19:34.353816 30637 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "385b71fa8aea425f94dba57ebffc17fe"
dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
 from {username='slave'} at 127.0.0.1:38382
I20260709 12:19:34.354378 30637 raft_consensus.cc:493] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:34.354725 30637 raft_consensus.cc:3060] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.359356 30637 raft_consensus.cc:515] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.360893 30637 leader_election.cc:290] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 3 election: Requested vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:34.361961 30787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "385b71fa8aea425f94dba57ebffc17fe" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:34.361961 30711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "385b71fa8aea425f94dba57ebffc17fe" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:34.362917 30787 raft_consensus.cc:3060] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.362951 30711 raft_consensus.cc:3055] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:34.363535 30711 raft_consensus.cc:740] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:34.364527 30711 consensus_queue.cc:260] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.365922 30711 raft_consensus.cc:3060] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.371168 30787 raft_consensus.cc:2468] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 3.
I20260709 12:19:34.372565 30604 leader_election.cc:304] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:34.373317 30996 raft_consensus.cc:2804] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:34.373497 30711 raft_consensus.cc:2468] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 3.
I20260709 12:19:34.373883 30996 raft_consensus.cc:697] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 LEADER]: Becoming Leader. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:34.374928 30996 consensus_queue.cc:237] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.382995 30535 catalog_manager.cc:5697] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 reported cstate change: term changed from 2 to 3, leader changed from b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194) to e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193). New cstate: current_term: 3 leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:34.584592 30992 raft_consensus.cc:993] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe: : Instructing follower e56eff8cb9af4916a3a0cb1f1077f3c7 to start an election
I20260709 12:19:34.585117 30993 raft_consensus.cc:1081] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Signalling peer e56eff8cb9af4916a3a0cb1f1077f3c7 to start an election
I20260709 12:19:34.586953 30637 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "75f2a084d518442e8241188f2d30b856"
dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
 from {username='slave'} at 127.0.0.1:38382
I20260709 12:19:34.587494 30637 raft_consensus.cc:493] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:34.587857 30637 raft_consensus.cc:3060] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.592969 30637 raft_consensus.cc:515] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.594786 30637 leader_election.cc:290] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 3 election: Requested vote from peers b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949), 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195:35831)
I20260709 12:19:34.595798 30711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75f2a084d518442e8241188f2d30b856" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:34.595995 30787 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75f2a084d518442e8241188f2d30b856" candidate_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
I20260709 12:19:34.596346 30711 raft_consensus.cc:3055] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:34.596590 30787 raft_consensus.cc:3060] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.596683 30711 raft_consensus.cc:740] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:34.597801 30711 consensus_queue.cc:260] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.598866 30711 raft_consensus.cc:3060] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:34.602566 30787 raft_consensus.cc:2468] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 3.
I20260709 12:19:34.604358 30604 leader_election.cc:304] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:34.604781 30711 raft_consensus.cc:2468] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e56eff8cb9af4916a3a0cb1f1077f3c7 in term 3.
I20260709 12:19:34.605700 30996 raft_consensus.cc:2804] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:34.606211 30992 raft_consensus.cc:993] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe: : Instructing follower 07da37de9b1e4baca15164fe424a8fdb to start an election
I20260709 12:19:34.606348 30996 raft_consensus.cc:697] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 LEADER]: Becoming Leader. State: Replica: e56eff8cb9af4916a3a0cb1f1077f3c7, State: Running, Role: LEADER
I20260709 12:19:34.606880 30992 raft_consensus.cc:1081] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 LEADER]: Signalling peer 07da37de9b1e4baca15164fe424a8fdb to start an election
I20260709 12:19:34.607571 30996 consensus_queue.cc:237] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.608803 30786 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6"
dest_uuid: "07da37de9b1e4baca15164fe424a8fdb"
 from {username='slave'} at 127.0.0.1:51864
I20260709 12:19:34.609429 30786 raft_consensus.cc:493] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:34.609798 30786 raft_consensus.cc:3060] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:34.616485 30786 raft_consensus.cc:515] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.616379 30535 catalog_manager.cc:5697] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 reported cstate change: term changed from 2 to 3, leader changed from b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194) to e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193). New cstate: current_term: 3 leader_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:34.618813 30786 leader_election.cc:290] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 4 election: Requested vote from peers e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449), b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194:41949)
I20260709 12:19:34.619477 30711 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe"
I20260709 12:19:34.619418 30637 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "535f09def3bd494687aa4a3edeab61d6" candidate_uuid: "07da37de9b1e4baca15164fe424a8fdb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7"
I20260709 12:19:34.620471 30637 raft_consensus.cc:3060] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:34.620280 30711 raft_consensus.cc:3055] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:19:34.621209 30711 raft_consensus.cc:740] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b2657dfca3a04182a3c74dbd5c3e72fe, State: Running, Role: LEADER
I20260709 12:19:34.622154 30711 consensus_queue.cc:260] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.623553 30711 raft_consensus.cc:3060] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:19:34.628564 30637 raft_consensus.cc:2468] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 4.
I20260709 12:19:34.629900 30752 leader_election.cc:304] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07da37de9b1e4baca15164fe424a8fdb, e56eff8cb9af4916a3a0cb1f1077f3c7; no voters: 
I20260709 12:19:34.630370 30786 raft_consensus.cc:1275] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 5 FOLLOWER]: Refusing update from remote peer e56eff8cb9af4916a3a0cb1f1077f3c7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:19:34.630740 30958 raft_consensus.cc:2804] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:19:34.631357 30958 raft_consensus.cc:697] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 4 LEADER]: Becoming Leader. State: Replica: 07da37de9b1e4baca15164fe424a8fdb, State: Running, Role: LEADER
I20260709 12:19:34.631381 30711 raft_consensus.cc:2468] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 07da37de9b1e4baca15164fe424a8fdb in term 4.
I20260709 12:19:34.632205 30996 consensus_queue.cc:1048] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:34.632488 30958 consensus_queue.cc:237] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } }
I20260709 12:19:34.642609 30535 catalog_manager.cc:5697] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb reported cstate change: term changed from 3 to 4, leader changed from b2657dfca3a04182a3c74dbd5c3e72fe (127.29.201.194) to 07da37de9b1e4baca15164fe424a8fdb (127.29.201.195). New cstate: current_term: 4 leader_uuid: "07da37de9b1e4baca15164fe424a8fdb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:34.646783 30711 raft_consensus.cc:1275] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 5 FOLLOWER]: Refusing update from remote peer e56eff8cb9af4916a3a0cb1f1077f3c7: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:19:34.650334 31011 consensus_queue.cc:1048] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:19:34.757217 30711 raft_consensus.cc:1275] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Refusing update from remote peer e56eff8cb9af4916a3a0cb1f1077f3c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:34.758608 31011 consensus_queue.cc:1048] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:34.769124 30786 raft_consensus.cc:1275] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Refusing update from remote peer e56eff8cb9af4916a3a0cb1f1077f3c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:34.770706 30997 consensus_queue.cc:1048] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:34.796113 30637 raft_consensus.cc:1275] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Refusing update from remote peer 07da37de9b1e4baca15164fe424a8fdb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:34.797493 30958 consensus_queue.cc:1048] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:34.807780 30711 raft_consensus.cc:1275] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Refusing update from remote peer 07da37de9b1e4baca15164fe424a8fdb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:34.809938 30995 consensus_queue.cc:1048] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:19:35.025779 30786 raft_consensus.cc:1275] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Refusing update from remote peer e56eff8cb9af4916a3a0cb1f1077f3c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:35.027197 30997 consensus_queue.cc:1048] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07da37de9b1e4baca15164fe424a8fdb" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:35.036772 30711 raft_consensus.cc:1275] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Refusing update from remote peer e56eff8cb9af4916a3a0cb1f1077f3c7: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:35.038728 30997 consensus_queue.cc:1048] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:35.118065 30637 raft_consensus.cc:1275] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 4 FOLLOWER]: Refusing update from remote peer 07da37de9b1e4baca15164fe424a8fdb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:19:35.119277 30995 consensus_queue.cc:1048] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e56eff8cb9af4916a3a0cb1f1077f3c7" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 43449 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:35.128414 30711 raft_consensus.cc:1275] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 4 FOLLOWER]: Refusing update from remote peer 07da37de9b1e4baca15164fe424a8fdb: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:19:35.130010 30958 consensus_queue.cc:1048] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2657dfca3a04182a3c74dbd5c3e72fe" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 41949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:19:35.184937 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:35.190557 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:35.191102 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:36.192370 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:36.198226 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:36.198704 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:37.199791 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:37.204926 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:37.205564 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:38.206923 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:38.212515 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:38.212988 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:39.214112 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:39.220077 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:39.220534 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:40.221652 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:40.226869 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:40.227320 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:41.228415 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:41.233640 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:41.234095 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:42.235275 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:42.240319 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:42.240796 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:43.241940 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:43.246891 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:43.247372 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:44.282310 30503 auto_leader_rebalancer-test.cc:340] The leader distribution is 
I20260709 12:19:44.283041 30503 tablet_server.cc:179] TabletServer@127.29.201.193:0 shutting down...
I20260709 12:19:44.312062 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:19:44.313016 30503 tablet_replica.cc:333] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7: stopping tablet replica
I20260709 12:19:44.313980 30503 raft_consensus.cc:2243] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.314803 30503 raft_consensus.cc:2272] T 498b463cdd8b414c8a752d8c093c49ac P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.318091 30503 tablet_replica.cc:333] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7: stopping tablet replica
I20260709 12:19:44.318914 30503 raft_consensus.cc:2243] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.319828 30503 raft_consensus.cc:2272] T 385b71fa8aea425f94dba57ebffc17fe P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.322005 30503 tablet_replica.cc:333] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7: stopping tablet replica
I20260709 12:19:44.322619 30503 raft_consensus.cc:2243] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.323318 30503 raft_consensus.cc:2272] T e7b159647196412382fcbeea841f3d93 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.325295 30503 tablet_replica.cc:333] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7: stopping tablet replica
I20260709 12:19:44.325933 30503 raft_consensus.cc:2243] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.326992 30503 raft_consensus.cc:2272] T 5e82899b14fc4d10a58ca1f42c33e4f7 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.328948 30503 tablet_replica.cc:333] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7: stopping tablet replica
I20260709 12:19:44.329517 30503 raft_consensus.cc:2243] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.330009 30503 raft_consensus.cc:2272] T 4884fe1576a44b67862ba32ea082366a P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.331943 30503 tablet_replica.cc:333] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7: stopping tablet replica
I20260709 12:19:44.332484 30503 raft_consensus.cc:2243] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.332970 30503 raft_consensus.cc:2272] T eec3ea46dfb242f4b4ce66a97604aceb P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.334818 30503 tablet_replica.cc:333] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7: stopping tablet replica
I20260709 12:19:44.335449 30503 raft_consensus.cc:2243] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.335975 30503 raft_consensus.cc:2272] T 535f09def3bd494687aa4a3edeab61d6 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.338045 30503 tablet_replica.cc:333] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7: stopping tablet replica
I20260709 12:19:44.338719 30503 raft_consensus.cc:2243] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.339699 30503 raft_consensus.cc:2272] T 75f2a084d518442e8241188f2d30b856 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:44.340543 30752 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:19:44.342168 30503 tablet_replica.cc:333] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7: stopping tablet replica
I20260709 12:19:44.342806 30503 raft_consensus.cc:2243] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.343298 30503 raft_consensus.cc:2272] T a19c6e5d902745869661b7ee64f329e8 P e56eff8cb9af4916a3a0cb1f1077f3c7 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:44.344154 30752 consensus_peers.cc:597] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb -> Peer e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449): Couldn't send request to peer e56eff8cb9af4916a3a0cb1f1077f3c7. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:44.440405 30503 tablet_server.cc:196] TabletServer@127.29.201.193:0 shutdown complete.
W20260709 12:19:44.441404 30752 consensus_peers.cc:597] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb -> Peer e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449): Couldn't send request to peer e56eff8cb9af4916a3a0cb1f1077f3c7. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:43449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:44.463112 30503 tablet_server.cc:179] TabletServer@127.29.201.194:0 shutting down...
I20260709 12:19:44.491477 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:19:44.492246 30503 tablet_replica.cc:333] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe: stopping tablet replica
I20260709 12:19:44.492949 30503 raft_consensus.cc:2243] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.493496 30503 raft_consensus.cc:2272] T e7b159647196412382fcbeea841f3d93 P b2657dfca3a04182a3c74dbd5c3e72fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.495586 30503 tablet_replica.cc:333] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe: stopping tablet replica
I20260709 12:19:44.496165 30503 raft_consensus.cc:2243] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.496639 30503 raft_consensus.cc:2272] T 385b71fa8aea425f94dba57ebffc17fe P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.498499 30503 tablet_replica.cc:333] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe: stopping tablet replica
I20260709 12:19:44.499082 30503 raft_consensus.cc:2243] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.499531 30503 raft_consensus.cc:2272] T 498b463cdd8b414c8a752d8c093c49ac P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.501444 30503 tablet_replica.cc:333] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe: stopping tablet replica
I20260709 12:19:44.502080 30503 raft_consensus.cc:2243] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.502604 30503 raft_consensus.cc:2272] T 5e82899b14fc4d10a58ca1f42c33e4f7 P b2657dfca3a04182a3c74dbd5c3e72fe [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.505056 30503 tablet_replica.cc:333] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe: stopping tablet replica
I20260709 12:19:44.505594 30503 raft_consensus.cc:2243] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.506480 30503 raft_consensus.cc:2272] T 4884fe1576a44b67862ba32ea082366a P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:19:44.507369 30678 consensus_peers.cc:597] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe -> Peer e56eff8cb9af4916a3a0cb1f1077f3c7 (127.29.201.193:43449): Couldn't send request to peer e56eff8cb9af4916a3a0cb1f1077f3c7. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:43449: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:19:44.508613 30503 tablet_replica.cc:333] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe: stopping tablet replica
I20260709 12:19:44.509120 30503 raft_consensus.cc:2243] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.509827 30503 raft_consensus.cc:2272] T eec3ea46dfb242f4b4ce66a97604aceb P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.511790 30503 tablet_replica.cc:333] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe: stopping tablet replica
I20260709 12:19:44.512220 30503 raft_consensus.cc:2243] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.512584 30503 raft_consensus.cc:2272] T 535f09def3bd494687aa4a3edeab61d6 P b2657dfca3a04182a3c74dbd5c3e72fe [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.514253 30503 tablet_replica.cc:333] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe: stopping tablet replica
I20260709 12:19:44.514773 30503 raft_consensus.cc:2243] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.515175 30503 raft_consensus.cc:2272] T 75f2a084d518442e8241188f2d30b856 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.516815 30503 tablet_replica.cc:333] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe: stopping tablet replica
I20260709 12:19:44.517324 30503 raft_consensus.cc:2243] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.518074 30503 raft_consensus.cc:2272] T a19c6e5d902745869661b7ee64f329e8 P b2657dfca3a04182a3c74dbd5c3e72fe [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.546754 30503 tablet_server.cc:196] TabletServer@127.29.201.194:0 shutdown complete.
I20260709 12:19:44.567499 30503 tablet_server.cc:179] TabletServer@127.29.201.195:0 shutting down...
I20260709 12:19:44.593451 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:19:44.594244 30503 tablet_replica.cc:333] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb: stopping tablet replica
I20260709 12:19:44.594992 30503 raft_consensus.cc:2243] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.595788 30503 raft_consensus.cc:2272] T e7b159647196412382fcbeea841f3d93 P 07da37de9b1e4baca15164fe424a8fdb [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.597689 30503 tablet_replica.cc:333] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb: stopping tablet replica
I20260709 12:19:44.598186 30503 raft_consensus.cc:2243] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.598773 30503 raft_consensus.cc:2272] T 385b71fa8aea425f94dba57ebffc17fe P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.600534 30503 tablet_replica.cc:333] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb: stopping tablet replica
I20260709 12:19:44.601055 30503 raft_consensus.cc:2243] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.601913 30503 raft_consensus.cc:2272] T 498b463cdd8b414c8a752d8c093c49ac P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.603648 30503 tablet_replica.cc:333] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb: stopping tablet replica
I20260709 12:19:44.604226 30503 raft_consensus.cc:2243] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.604732 30503 raft_consensus.cc:2272] T 5e82899b14fc4d10a58ca1f42c33e4f7 P 07da37de9b1e4baca15164fe424a8fdb [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.606592 30503 tablet_replica.cc:333] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb: stopping tablet replica
I20260709 12:19:44.607167 30503 raft_consensus.cc:2243] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.607601 30503 raft_consensus.cc:2272] T eec3ea46dfb242f4b4ce66a97604aceb P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.609282 30503 tablet_replica.cc:333] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb: stopping tablet replica
I20260709 12:19:44.609769 30503 raft_consensus.cc:2243] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.610281 30503 raft_consensus.cc:2272] T 4884fe1576a44b67862ba32ea082366a P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.612002 30503 tablet_replica.cc:333] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb: stopping tablet replica
I20260709 12:19:44.612495 30503 raft_consensus.cc:2243] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.613299 30503 raft_consensus.cc:2272] T 535f09def3bd494687aa4a3edeab61d6 P 07da37de9b1e4baca15164fe424a8fdb [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.615027 30503 tablet_replica.cc:333] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb: stopping tablet replica
I20260709 12:19:44.615501 30503 raft_consensus.cc:2243] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.615947 30503 raft_consensus.cc:2272] T 75f2a084d518442e8241188f2d30b856 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.617560 30503 tablet_replica.cc:333] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb: stopping tablet replica
I20260709 12:19:44.618028 30503 raft_consensus.cc:2243] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:19:44.618436 30503 raft_consensus.cc:2272] T a19c6e5d902745869661b7ee64f329e8 P 07da37de9b1e4baca15164fe424a8fdb [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.648200 30503 tablet_server.cc:196] TabletServer@127.29.201.195:0 shutdown complete.
I20260709 12:19:44.666931 30503 master.cc:562] Master@127.29.201.254:36891 shutting down...
I20260709 12:19:44.888442 30503 raft_consensus.cc:2243] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:19:44.889058 30503 raft_consensus.cc:2272] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:19:44.889402 30503 tablet_replica.cc:333] T 00000000000000000000000000000000 P fcb51a0848cb4dbdbc6cd9256992e573: stopping tablet replica
I20260709 12:19:44.908845 30503 master.cc:584] Master@127.29.201.254:36891 shutdown complete.
07da37de9b1e4baca15164fe424a8fdb  1
b2657dfca3a04182a3c74dbd5c3e72fe  4
e56eff8cb9af4916a3a0cb1f1077f3c7  4
07da37de9b1e4baca15164fe424a8fdb  3
b2657dfca3a04182a3c74dbd5c3e72fe  3
e56eff8cb9af4916a3a0cb1f1077f3c7  3
07da37de9b1e4baca15164fe424a8fdb  1
b2657dfca3a04182a3c74dbd5c3e72fe  8
e56eff8cb9af4916a3a0cb1f1077f3c7  0
07da37de9b1e4baca15164fe424a8fdb  3
b2657dfca3a04182a3c74dbd5c3e72fe  3
e56eff8cb9af4916a3a0cb1f1077f3c7  3
[       OK ] LeaderRebalancerTest.FunctionalTestForDivided (31465 ms)
[ RUN      ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260709 12:19:44.955999 30503 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.201.254:41269
I20260709 12:19:44.957203 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:44.963912 31037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:44.963924 31038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:44.964222 31040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:44.965806 30503 server_base.cc:1061] running on GCE node
I20260709 12:19:44.966903 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:44.967144 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:44.967319 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599584967296 us; error 0 us; skew 500 ppm
I20260709 12:19:44.967849 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:44.970875 30503 webserver.cc:533] Webserver started at http://127.29.201.254:45653/ using document root <none> and password file <none>
I20260709 12:19:44.971563 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:44.971818 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:44.972115 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:44.973364 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/master-0-root/instance:
uuid: "5a0169a5d1354fdab991b1801c78dd59"
format_stamp: "Formatted at 2026-07-09 12:19:44 on dist-test-slave-qgf5"
I20260709 12:19:44.978108 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20260709 12:19:44.981741 31045 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:44.982710 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.002s
I20260709 12:19:44.982980 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/master-0-root
uuid: "5a0169a5d1354fdab991b1801c78dd59"
format_stamp: "Formatted at 2026-07-09 12:19:44 on dist-test-slave-qgf5"
I20260709 12:19:44.983240 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:44.996541 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:44.997782 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:45.039731 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.254:41269
I20260709 12:19:45.039829 31096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.254:41269 every 8 connection(s)
I20260709 12:19:45.044234 31097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.055433 31097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59: Bootstrap starting.
I20260709 12:19:45.060937 31097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.065575 31097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59: No bootstrap required, opened a new log
I20260709 12:19:45.067905 31097 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0169a5d1354fdab991b1801c78dd59" member_type: VOTER }
I20260709 12:19:45.068411 31097 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.068679 31097 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a0169a5d1354fdab991b1801c78dd59, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.069350 31097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0169a5d1354fdab991b1801c78dd59" member_type: VOTER }
I20260709 12:19:45.069852 31097 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:19:45.070106 31097 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:19:45.070367 31097 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:45.074515 31097 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0169a5d1354fdab991b1801c78dd59" member_type: VOTER }
I20260709 12:19:45.075165 31097 leader_election.cc:304] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a0169a5d1354fdab991b1801c78dd59; no voters: 
I20260709 12:19:45.076398 31097 leader_election.cc:290] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:19:45.076740 31100 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:45.078209 31100 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [term 1 LEADER]: Becoming Leader. State: Replica: 5a0169a5d1354fdab991b1801c78dd59, State: Running, Role: LEADER
I20260709 12:19:45.078989 31100 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0169a5d1354fdab991b1801c78dd59" member_type: VOTER }
I20260709 12:19:45.079528 31097 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:19:45.084551 31102 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a0169a5d1354fdab991b1801c78dd59. Latest consensus state: current_term: 1 leader_uuid: "5a0169a5d1354fdab991b1801c78dd59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0169a5d1354fdab991b1801c78dd59" member_type: VOTER } }
I20260709 12:19:45.084816 31101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a0169a5d1354fdab991b1801c78dd59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a0169a5d1354fdab991b1801c78dd59" member_type: VOTER } }
I20260709 12:19:45.085309 31102 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [sys.catalog]: This master's current role is: LEADER
I20260709 12:19:45.085564 31101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [sys.catalog]: This master's current role is: LEADER
I20260709 12:19:45.089879 31108 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:19:45.095082 31108 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:19:45.097252 30503 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:19:45.105139 31108 catalog_manager.cc:1383] Generated new cluster ID: 015dbc2727664b5db06953549dca4a7d
I20260709 12:19:45.105571 31108 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:19:45.123138 31108 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:19:45.124522 31108 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:19:45.140761 31108 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59: Generated new TSK 0
I20260709 12:19:45.141398 31108 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:19:45.164400 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:45.171310 31118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:45.172430 31119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:45.174460 31121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:45.174772 30503 server_base.cc:1061] running on GCE node
I20260709 12:19:45.175791 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:45.176030 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:45.176213 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599585176193 us; error 0 us; skew 500 ppm
I20260709 12:19:45.176782 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:45.179453 30503 webserver.cc:533] Webserver started at http://127.29.201.193:38625/ using document root <none> and password file <none>
I20260709 12:19:45.180001 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:45.180226 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:45.180505 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:45.181790 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-0-root/instance:
uuid: "8f94dd3d7b0948e092e65f629d8af1de"
format_stamp: "Formatted at 2026-07-09 12:19:45 on dist-test-slave-qgf5"
I20260709 12:19:45.186723 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:19:45.190140 31126 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.191061 30503 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:19:45.191372 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-0-root
uuid: "8f94dd3d7b0948e092e65f629d8af1de"
format_stamp: "Formatted at 2026-07-09 12:19:45 on dist-test-slave-qgf5"
I20260709 12:19:45.191702 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:45.202157 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:45.203575 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:45.205077 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:45.207504 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:45.207742 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.208005 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:45.208259 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.249472 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.193:44183
I20260709 12:19:45.249579 31188 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.193:44183 every 8 connection(s)
I20260709 12:19:45.254683 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:45.261725 31193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:45.264143 31194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:45.266464 31196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:45.267024 30503 server_base.cc:1061] running on GCE node
I20260709 12:19:45.268821 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:45.269148 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:45.269362 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599585269339 us; error 0 us; skew 500 ppm
I20260709 12:19:45.269965 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:45.272090 31189 heartbeater.cc:344] Connected to a master server at 127.29.201.254:41269
I20260709 12:19:45.272462 31189 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:45.272995 30503 webserver.cc:533] Webserver started at http://127.29.201.194:34311/ using document root <none> and password file <none>
I20260709 12:19:45.273389 31189 heartbeater.cc:507] Master 127.29.201.254:41269 requested a full tablet report, sending...
I20260709 12:19:45.273542 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:45.273844 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:45.274129 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:45.275555 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-1-root/instance:
uuid: "e7f7f401a756482ea721ae7015101f72"
format_stamp: "Formatted at 2026-07-09 12:19:45 on dist-test-slave-qgf5"
I20260709 12:19:45.275842 31062 ts_manager.cc:194] Registered new tserver with Master: 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183)
I20260709 12:19:45.277782 31062 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59708
I20260709 12:19:45.280989 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:19:45.284917 31201 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.286150 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:19:45.286456 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-1-root
uuid: "e7f7f401a756482ea721ae7015101f72"
format_stamp: "Formatted at 2026-07-09 12:19:45 on dist-test-slave-qgf5"
I20260709 12:19:45.286888 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:45.318228 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:45.319777 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:45.321755 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:45.324497 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:45.324705 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.325034 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:45.325323 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.001s	sys 0.000s
I20260709 12:19:45.367249 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.194:39933
I20260709 12:19:45.367336 31263 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.194:39933 every 8 connection(s)
I20260709 12:19:45.372051 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:19:45.381464 31268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:45.384337 31267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:19:45.384840 31270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:19:45.385530 31264 heartbeater.cc:344] Connected to a master server at 127.29.201.254:41269
I20260709 12:19:45.385954 31264 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:45.386396 30503 server_base.cc:1061] running on GCE node
I20260709 12:19:45.386933 31264 heartbeater.cc:507] Master 127.29.201.254:41269 requested a full tablet report, sending...
I20260709 12:19:45.387387 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:19:45.387607 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:19:45.387823 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599585387801 us; error 0 us; skew 500 ppm
I20260709 12:19:45.388584 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:19:45.389168 31062 ts_manager.cc:194] Registered new tserver with Master: e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:19:45.390895 31062 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59718
I20260709 12:19:45.391481 30503 webserver.cc:533] Webserver started at http://127.29.201.195:35415/ using document root <none> and password file <none>
I20260709 12:19:45.392117 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:19:45.392381 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:19:45.392735 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:19:45.394081 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-2-root/instance:
uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
format_stamp: "Formatted at 2026-07-09 12:19:45 on dist-test-slave-qgf5"
I20260709 12:19:45.398804 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.001s
I20260709 12:19:45.402434 31275 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.403330 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:19:45.403674 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-2-root
uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
format_stamp: "Formatted at 2026-07-09 12:19:45 on dist-test-slave-qgf5"
I20260709 12:19:45.404019 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1783599553313367-30503-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:19:45.419960 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:19:45.421322 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:19:45.422994 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:19:45.425369 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:19:45.425704 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.425923 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:19:45.426075 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:19:45.467618 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.195:35629
I20260709 12:19:45.467819 31337 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.195:35629 every 8 connection(s)
I20260709 12:19:45.482103 31338 heartbeater.cc:344] Connected to a master server at 127.29.201.254:41269
I20260709 12:19:45.482514 31338 heartbeater.cc:461] Registering TS with master...
I20260709 12:19:45.483340 31338 heartbeater.cc:507] Master 127.29.201.254:41269 requested a full tablet report, sending...
I20260709 12:19:45.485569 31062 ts_manager.cc:194] Registered new tserver with Master: de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:45.485646 30503 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014701661s
I20260709 12:19:45.487227 31062 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:59722
I20260709 12:19:45.513217 31062 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59736:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:19:45.516263 31062 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:19:45.594960 31229 tablet_service.cc:1511] Processing CreateTablet for tablet e9cd04eb7d0b4e7fbac08a32889020e2 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:19:45.595341 31226 tablet_service.cc:1511] Processing CreateTablet for tablet 342a72e01a6c46fc9861bab2d1f4264f (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:19:45.596477 31229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9cd04eb7d0b4e7fbac08a32889020e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.597666 31227 tablet_service.cc:1511] Processing CreateTablet for tablet 8f12ddec531d42638c7ef2ec961ca399 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:19:45.599845 31228 tablet_service.cc:1511] Processing CreateTablet for tablet dda71ddf86394530aac956f3cc709af1 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:19:45.601186 31228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dda71ddf86394530aac956f3cc709af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.602949 31227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f12ddec531d42638c7ef2ec961ca399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.603343 31225 tablet_service.cc:1511] Processing CreateTablet for tablet 48cb82524513465785d32448b353ff4a (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:19:45.604652 31225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48cb82524513465785d32448b353ff4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.608265 31301 tablet_service.cc:1511] Processing CreateTablet for tablet 8f12ddec531d42638c7ef2ec961ca399 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:19:45.609845 31301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f12ddec531d42638c7ef2ec961ca399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.611898 31300 tablet_service.cc:1511] Processing CreateTablet for tablet 342a72e01a6c46fc9861bab2d1f4264f (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:19:45.613273 31300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 342a72e01a6c46fc9861bab2d1f4264f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.615491 31299 tablet_service.cc:1511] Processing CreateTablet for tablet 48cb82524513465785d32448b353ff4a (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:19:45.616928 31299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48cb82524513465785d32448b353ff4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.615618 31224 tablet_service.cc:1511] Processing CreateTablet for tablet d251201d5ba54aa8b4f8e536dbc3c39b (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:19:45.620371 31302 tablet_service.cc:1511] Processing CreateTablet for tablet dda71ddf86394530aac956f3cc709af1 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:19:45.621095 31224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d251201d5ba54aa8b4f8e536dbc3c39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.622108 31302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dda71ddf86394530aac956f3cc709af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.625176 31223 tablet_service.cc:1511] Processing CreateTablet for tablet 969ee28800d74736a2d0011634d1a014 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:19:45.626600 31223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 969ee28800d74736a2d0011634d1a014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.605515 31303 tablet_service.cc:1511] Processing CreateTablet for tablet e9cd04eb7d0b4e7fbac08a32889020e2 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:19:45.627900 31303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9cd04eb7d0b4e7fbac08a32889020e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.633404 31353 tablet_bootstrap.cc:492] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72: Bootstrap starting.
I20260709 12:19:45.634692 31296 tablet_service.cc:1511] Processing CreateTablet for tablet 37527f4161f344a4a4a010a4442419b8 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:19:45.636231 31296 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37527f4161f344a4a4a010a4442419b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.635089 31297 tablet_service.cc:1511] Processing CreateTablet for tablet 969ee28800d74736a2d0011634d1a014 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:19:45.633227 31298 tablet_service.cc:1511] Processing CreateTablet for tablet d251201d5ba54aa8b4f8e536dbc3c39b (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:19:45.638047 31298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d251201d5ba54aa8b4f8e536dbc3c39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.639848 31222 tablet_service.cc:1511] Processing CreateTablet for tablet 37527f4161f344a4a4a010a4442419b8 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:19:45.640488 31295 tablet_service.cc:1511] Processing CreateTablet for tablet 0f138178dea44e59804eaa33c221d4aa (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:19:45.641532 31297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 969ee28800d74736a2d0011634d1a014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.642046 31222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37527f4161f344a4a4a010a4442419b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.643616 31294 tablet_service.cc:1511] Processing CreateTablet for tablet cff8c0586c59461bb83a89ee55000d70 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:19:45.644708 31353 tablet_bootstrap.cc:654] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.644979 31294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff8c0586c59461bb83a89ee55000d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.647235 31226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 342a72e01a6c46fc9861bab2d1f4264f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.651000 31295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f138178dea44e59804eaa33c221d4aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.667613 31221 tablet_service.cc:1511] Processing CreateTablet for tablet cff8c0586c59461bb83a89ee55000d70 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:19:45.669063 31221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff8c0586c59461bb83a89ee55000d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.695309 31227 tablet_service.cc:1511] Processing CreateTablet for tablet 0f138178dea44e59804eaa33c221d4aa (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:19:45.696902 31227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f138178dea44e59804eaa33c221d4aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.768898 31353 tablet_bootstrap.cc:492] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72: No bootstrap required, opened a new log
I20260709 12:19:45.769444 31353 ts_tablet_manager.cc:1403] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72: Time spent bootstrapping tablet: real 0.136s	user 0.035s	sys 0.053s
I20260709 12:19:45.775557 31154 tablet_service.cc:1511] Processing CreateTablet for tablet e9cd04eb7d0b4e7fbac08a32889020e2 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:19:45.775921 31153 tablet_service.cc:1511] Processing CreateTablet for tablet dda71ddf86394530aac956f3cc709af1 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:19:45.783143 31148 tablet_service.cc:1511] Processing CreateTablet for tablet 969ee28800d74736a2d0011634d1a014 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:19:45.783865 31152 tablet_service.cc:1511] Processing CreateTablet for tablet 8f12ddec531d42638c7ef2ec961ca399 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:19:45.787967 31147 tablet_service.cc:1511] Processing CreateTablet for tablet 37527f4161f344a4a4a010a4442419b8 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:19:45.791137 31151 tablet_service.cc:1511] Processing CreateTablet for tablet 342a72e01a6c46fc9861bab2d1f4264f (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:19:45.794507 31150 tablet_service.cc:1511] Processing CreateTablet for tablet 48cb82524513465785d32448b353ff4a (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:19:45.780409 31149 tablet_service.cc:1511] Processing CreateTablet for tablet d251201d5ba54aa8b4f8e536dbc3c39b (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:19:45.798938 31146 tablet_service.cc:1511] Processing CreateTablet for tablet 0f138178dea44e59804eaa33c221d4aa (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:19:45.799286 31145 tablet_service.cc:1511] Processing CreateTablet for tablet cff8c0586c59461bb83a89ee55000d70 (DEFAULT_TABLE table=test-workload [id=1c1c68659e864ddd9c89bf95e1977f59]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:19:45.799994 31154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9cd04eb7d0b4e7fbac08a32889020e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.800660 31152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f12ddec531d42638c7ef2ec961ca399. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.801375 31145 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff8c0586c59461bb83a89ee55000d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.802117 31151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 342a72e01a6c46fc9861bab2d1f4264f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.803004 31147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37527f4161f344a4a4a010a4442419b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.803623 31150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48cb82524513465785d32448b353ff4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.811053 31149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d251201d5ba54aa8b4f8e536dbc3c39b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.811932 31146 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f138178dea44e59804eaa33c221d4aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.812594 31148 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 969ee28800d74736a2d0011634d1a014. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.814970 31153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dda71ddf86394530aac956f3cc709af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:19:45.820079 31353 raft_consensus.cc:359] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.820905 31353 raft_consensus.cc:385] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.821219 31353 raft_consensus.cc:740] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.822170 31353 consensus_queue.cc:260] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.824177 31355 tablet_bootstrap.cc:492] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1: Bootstrap starting.
I20260709 12:19:45.861814 31264 heartbeater.cc:499] Master 127.29.201.254:41269 was elected leader, sending a full tablet report...
I20260709 12:19:45.863304 31357 tablet_bootstrap.cc:492] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de: Bootstrap starting.
I20260709 12:19:45.873871 31357 tablet_bootstrap.cc:654] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.882076 31355 tablet_bootstrap.cc:654] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.889551 31353 ts_tablet_manager.cc:1434] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72: Time spent starting tablet: real 0.120s	user 0.013s	sys 0.015s
I20260709 12:19:45.891803 31353 tablet_bootstrap.cc:492] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72: Bootstrap starting.
I20260709 12:19:45.893548 31357 tablet_bootstrap.cc:492] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de: No bootstrap required, opened a new log
I20260709 12:19:45.894378 31357 ts_tablet_manager.cc:1403] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de: Time spent bootstrapping tablet: real 0.031s	user 0.012s	sys 0.014s
I20260709 12:19:45.898294 31357 raft_consensus.cc:359] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.898954 31353 tablet_bootstrap.cc:654] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.899271 31357 raft_consensus.cc:385] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.899741 31357 raft_consensus.cc:740] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.900519 31355 tablet_bootstrap.cc:492] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1: No bootstrap required, opened a new log
I20260709 12:19:45.900563 31357 consensus_queue.cc:260] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.901147 31355 ts_tablet_manager.cc:1403] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1: Time spent bootstrapping tablet: real 0.077s	user 0.014s	sys 0.005s
I20260709 12:19:45.905862 31355 raft_consensus.cc:359] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.906968 31355 raft_consensus.cc:385] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.907373 31355 raft_consensus.cc:740] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.908715 31355 consensus_queue.cc:260] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.914278 31353 tablet_bootstrap.cc:492] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72: No bootstrap required, opened a new log
I20260709 12:19:45.914872 31353 ts_tablet_manager.cc:1403] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.012s
I20260709 12:19:45.915576 31357 ts_tablet_manager.cc:1434] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de: Time spent starting tablet: real 0.021s	user 0.013s	sys 0.004s
I20260709 12:19:45.917183 31353 raft_consensus.cc:359] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.917833 31353 raft_consensus.cc:385] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.918133 31189 heartbeater.cc:499] Master 127.29.201.254:41269 was elected leader, sending a full tablet report...
I20260709 12:19:45.918848 31357 tablet_bootstrap.cc:492] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de: Bootstrap starting.
I20260709 12:19:45.918133 31353 raft_consensus.cc:740] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.919636 31338 heartbeater.cc:499] Master 127.29.201.254:41269 was elected leader, sending a full tablet report...
I20260709 12:19:45.920107 31355 ts_tablet_manager.cc:1434] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1: Time spent starting tablet: real 0.018s	user 0.006s	sys 0.010s
I20260709 12:19:45.920583 31353 consensus_queue.cc:260] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.921468 31355 tablet_bootstrap.cc:492] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1: Bootstrap starting.
I20260709 12:19:45.923295 31353 ts_tablet_manager.cc:1434] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.004s
I20260709 12:19:45.924381 31353 tablet_bootstrap.cc:492] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72: Bootstrap starting.
I20260709 12:19:45.927367 31355 tablet_bootstrap.cc:654] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.931072 31353 tablet_bootstrap.cc:654] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.934597 31357 tablet_bootstrap.cc:654] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.941812 31355 tablet_bootstrap.cc:492] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1: No bootstrap required, opened a new log
I20260709 12:19:45.942363 31355 ts_tablet_manager.cc:1403] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1: Time spent bootstrapping tablet: real 0.021s	user 0.007s	sys 0.012s
I20260709 12:19:45.945302 31357 tablet_bootstrap.cc:492] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de: No bootstrap required, opened a new log
I20260709 12:19:45.945886 31357 ts_tablet_manager.cc:1403] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de: Time spent bootstrapping tablet: real 0.027s	user 0.012s	sys 0.004s
I20260709 12:19:45.945483 31355 raft_consensus.cc:359] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.946368 31355 raft_consensus.cc:385] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.946782 31355 raft_consensus.cc:740] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.947773 31355 consensus_queue.cc:260] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.949719 31353 tablet_bootstrap.cc:492] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72: No bootstrap required, opened a new log
I20260709 12:19:45.949316 31357 raft_consensus.cc:359] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.950176 31357 raft_consensus.cc:385] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.950253 31353 ts_tablet_manager.cc:1403] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72: Time spent bootstrapping tablet: real 0.026s	user 0.010s	sys 0.008s
I20260709 12:19:45.950306 31355 ts_tablet_manager.cc:1434] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:19:45.950594 31357 raft_consensus.cc:740] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.951581 31355 tablet_bootstrap.cc:492] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1: Bootstrap starting.
I20260709 12:19:45.951624 31357 consensus_queue.cc:260] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.953722 31353 raft_consensus.cc:359] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.954222 31357 ts_tablet_manager.cc:1434] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:19:45.954499 31353 raft_consensus.cc:385] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.954901 31353 raft_consensus.cc:740] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.955543 31357 tablet_bootstrap.cc:492] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de: Bootstrap starting.
I20260709 12:19:45.955699 31353 consensus_queue.cc:260] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.958609 31355 tablet_bootstrap.cc:654] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.959844 31353 ts_tablet_manager.cc:1434] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:19:45.960819 31353 tablet_bootstrap.cc:492] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72: Bootstrap starting.
I20260709 12:19:45.963559 31357 tablet_bootstrap.cc:654] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.967178 31353 tablet_bootstrap.cc:654] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.971306 31355 tablet_bootstrap.cc:492] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1: No bootstrap required, opened a new log
I20260709 12:19:45.972033 31355 ts_tablet_manager.cc:1403] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1: Time spent bootstrapping tablet: real 0.021s	user 0.012s	sys 0.007s
I20260709 12:19:45.976161 31355 raft_consensus.cc:359] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.977190 31355 raft_consensus.cc:385] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.977591 31355 raft_consensus.cc:740] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.979956 31355 consensus_queue.cc:260] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.981523 31357 tablet_bootstrap.cc:492] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de: No bootstrap required, opened a new log
I20260709 12:19:45.982110 31357 ts_tablet_manager.cc:1403] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de: Time spent bootstrapping tablet: real 0.027s	user 0.018s	sys 0.007s
I20260709 12:19:45.982656 31355 ts_tablet_manager.cc:1434] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.001s
I20260709 12:19:45.983753 31355 tablet_bootstrap.cc:492] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1: Bootstrap starting.
I20260709 12:19:45.985574 31357 raft_consensus.cc:359] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.986435 31357 raft_consensus.cc:385] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.986807 31357 raft_consensus.cc:740] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.987618 31357 consensus_queue.cc:260] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.990223 31353 tablet_bootstrap.cc:492] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72: No bootstrap required, opened a new log
I20260709 12:19:45.990235 31357 ts_tablet_manager.cc:1434] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:19:45.990813 31353 ts_tablet_manager.cc:1403] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72: Time spent bootstrapping tablet: real 0.030s	user 0.019s	sys 0.007s
I20260709 12:19:45.991690 31357 tablet_bootstrap.cc:492] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de: Bootstrap starting.
I20260709 12:19:45.994011 31353 raft_consensus.cc:359] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.994783 31353 raft_consensus.cc:385] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:45.995100 31353 raft_consensus.cc:740] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Initialized, Role: FOLLOWER
I20260709 12:19:45.995870 31353 consensus_queue.cc:260] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:45.998051 31353 ts_tablet_manager.cc:1434] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:19:45.998111 31355 tablet_bootstrap.cc:654] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:45.999255 31353 tablet_bootstrap.cc:492] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72: Bootstrap starting.
I20260709 12:19:46.002228 31357 tablet_bootstrap.cc:654] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.007834 31353 tablet_bootstrap.cc:654] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.015951 31355 tablet_bootstrap.cc:492] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1: No bootstrap required, opened a new log
I20260709 12:19:46.016579 31355 ts_tablet_manager.cc:1403] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1: Time spent bootstrapping tablet: real 0.033s	user 0.013s	sys 0.005s
I20260709 12:19:46.018791 31353 tablet_bootstrap.cc:492] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72: No bootstrap required, opened a new log
I20260709 12:19:46.018791 31357 tablet_bootstrap.cc:492] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de: No bootstrap required, opened a new log
I20260709 12:19:46.018992 31355 raft_consensus.cc:359] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.019465 31357 ts_tablet_manager.cc:1403] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de: Time spent bootstrapping tablet: real 0.028s	user 0.010s	sys 0.006s
I20260709 12:19:46.019726 31355 raft_consensus.cc:385] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.019999 31353 ts_tablet_manager.cc:1403] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72: Time spent bootstrapping tablet: real 0.021s	user 0.014s	sys 0.004s
I20260709 12:19:46.020087 31355 raft_consensus.cc:740] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.020987 31355 consensus_queue.cc:260] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.021986 31357 raft_consensus.cc:359] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.022748 31357 raft_consensus.cc:385] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.023049 31357 raft_consensus.cc:740] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.023258 31355 ts_tablet_manager.cc:1434] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1: Time spent starting tablet: real 0.006s	user 0.001s	sys 0.007s
I20260709 12:19:46.023082 31353 raft_consensus.cc:359] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.023937 31353 raft_consensus.cc:385] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.024333 31355 tablet_bootstrap.cc:492] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1: Bootstrap starting.
I20260709 12:19:46.023866 31357 consensus_queue.cc:260] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.024344 31353 raft_consensus.cc:740] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.025434 31353 consensus_queue.cc:260] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.026548 31357 ts_tablet_manager.cc:1434] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de: Time spent starting tablet: real 0.007s	user 0.001s	sys 0.004s
I20260709 12:19:46.027603 31357 tablet_bootstrap.cc:492] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de: Bootstrap starting.
I20260709 12:19:46.030560 31353 ts_tablet_manager.cc:1434] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:19:46.031665 31353 tablet_bootstrap.cc:492] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72: Bootstrap starting.
I20260709 12:19:46.034296 31357 tablet_bootstrap.cc:654] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.038295 31353 tablet_bootstrap.cc:654] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.040907 31355 tablet_bootstrap.cc:654] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.048305 31357 tablet_bootstrap.cc:492] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de: No bootstrap required, opened a new log
I20260709 12:19:46.048305 31353 tablet_bootstrap.cc:492] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72: No bootstrap required, opened a new log
I20260709 12:19:46.048695 31355 tablet_bootstrap.cc:492] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1: No bootstrap required, opened a new log
I20260709 12:19:46.048971 31357 ts_tablet_manager.cc:1403] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.007s
I20260709 12:19:46.049079 31353 ts_tablet_manager.cc:1403] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72: Time spent bootstrapping tablet: real 0.018s	user 0.015s	sys 0.000s
I20260709 12:19:46.049254 31355 ts_tablet_manager.cc:1403] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1: Time spent bootstrapping tablet: real 0.025s	user 0.011s	sys 0.002s
I20260709 12:19:46.052119 31353 raft_consensus.cc:359] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.052278 31357 raft_consensus.cc:359] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.052522 31355 raft_consensus.cc:359] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.053084 31357 raft_consensus.cc:385] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.053274 31355 raft_consensus.cc:385] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.053084 31353 raft_consensus.cc:385] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.053534 31357 raft_consensus.cc:740] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.053752 31355 raft_consensus.cc:740] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.053830 31353 raft_consensus.cc:740] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.054581 31355 consensus_queue.cc:260] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.054425 31357 consensus_queue.cc:260] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.054739 31353 consensus_queue.cc:260] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.056803 31355 ts_tablet_manager.cc:1434] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:19:46.057886 31355 tablet_bootstrap.cc:492] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1: Bootstrap starting.
I20260709 12:19:46.059477 31353 ts_tablet_manager.cc:1434] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:19:46.060555 31353 tablet_bootstrap.cc:492] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72: Bootstrap starting.
I20260709 12:19:46.056829 31357 ts_tablet_manager.cc:1434] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:19:46.064301 31357 tablet_bootstrap.cc:492] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de: Bootstrap starting.
I20260709 12:19:46.071389 31355 tablet_bootstrap.cc:654] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.075435 31353 tablet_bootstrap.cc:654] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.076613 31357 tablet_bootstrap.cc:654] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.082858 31355 tablet_bootstrap.cc:492] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1: No bootstrap required, opened a new log
I20260709 12:19:46.083384 31355 ts_tablet_manager.cc:1403] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1: Time spent bootstrapping tablet: real 0.026s	user 0.009s	sys 0.008s
I20260709 12:19:46.086442 31355 raft_consensus.cc:359] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.087311 31355 raft_consensus.cc:385] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.087613 31355 raft_consensus.cc:740] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.088340 31355 consensus_queue.cc:260] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.088909 31357 tablet_bootstrap.cc:492] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de: No bootstrap required, opened a new log
I20260709 12:19:46.089617 31357 ts_tablet_manager.cc:1403] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de: Time spent bootstrapping tablet: real 0.026s	user 0.013s	sys 0.006s
I20260709 12:19:46.090994 31355 ts_tablet_manager.cc:1434] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:19:46.092990 31355 tablet_bootstrap.cc:492] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1: Bootstrap starting.
I20260709 12:19:46.092950 31357 raft_consensus.cc:359] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.093816 31357 raft_consensus.cc:385] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.094089 31357 raft_consensus.cc:740] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Initialized, Role: FOLLOWER
W20260709 12:19:46.094677 31110 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:19:46.094774 31357 consensus_queue.cc:260] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.097898 31357 ts_tablet_manager.cc:1434] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:19:46.098887 31357 tablet_bootstrap.cc:492] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de: Bootstrap starting.
I20260709 12:19:46.099714 31355 tablet_bootstrap.cc:654] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.104053 31353 tablet_bootstrap.cc:492] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72: No bootstrap required, opened a new log
I20260709 12:19:46.104599 31353 ts_tablet_manager.cc:1403] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72: Time spent bootstrapping tablet: real 0.044s	user 0.015s	sys 0.000s
I20260709 12:19:46.105415 31357 tablet_bootstrap.cc:654] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.107615 31353 raft_consensus.cc:359] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.108605 31353 raft_consensus.cc:385] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.109148 31353 raft_consensus.cc:740] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.112210 31355 tablet_bootstrap.cc:492] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1: No bootstrap required, opened a new log
I20260709 12:19:46.114773 31355 ts_tablet_manager.cc:1403] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.004s
I20260709 12:19:46.115312 31353 consensus_queue.cc:260] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.117535 31355 raft_consensus.cc:359] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.117964 31353 ts_tablet_manager.cc:1434] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72: Time spent starting tablet: real 0.013s	user 0.005s	sys 0.000s
I20260709 12:19:46.118170 31355 raft_consensus.cc:385] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.118535 31355 raft_consensus.cc:740] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.119349 31355 consensus_queue.cc:260] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.119982 31353 tablet_bootstrap.cc:492] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72: Bootstrap starting.
I20260709 12:19:46.120560 31357 tablet_bootstrap.cc:492] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de: No bootstrap required, opened a new log
I20260709 12:19:46.121016 31357 ts_tablet_manager.cc:1403] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de: Time spent bootstrapping tablet: real 0.022s	user 0.014s	sys 0.003s
I20260709 12:19:46.123970 31357 raft_consensus.cc:359] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.124737 31357 raft_consensus.cc:385] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.125033 31357 raft_consensus.cc:740] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.125794 31357 consensus_queue.cc:260] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.126981 31353 tablet_bootstrap.cc:654] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.128139 31357 ts_tablet_manager.cc:1434] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:19:46.129448 31355 ts_tablet_manager.cc:1434] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:19:46.131408 31357 tablet_bootstrap.cc:492] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de: Bootstrap starting.
I20260709 12:19:46.130834 31356 raft_consensus.cc:493] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.133603 31356 raft_consensus.cc:515] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.136092 31355 tablet_bootstrap.cc:492] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1: Bootstrap starting.
I20260709 12:19:46.138871 31356 leader_election.cc:290] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.140744 31357 tablet_bootstrap.cc:654] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.143172 31355 tablet_bootstrap.cc:654] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.156323 31355 tablet_bootstrap.cc:492] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1: No bootstrap required, opened a new log
I20260709 12:19:46.156997 31355 ts_tablet_manager.cc:1403] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.009s
I20260709 12:19:46.161768 31355 raft_consensus.cc:359] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.161774 31353 tablet_bootstrap.cc:492] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72: No bootstrap required, opened a new log
I20260709 12:19:46.162915 31355 raft_consensus.cc:385] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.163606 31353 ts_tablet_manager.cc:1403] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72: Time spent bootstrapping tablet: real 0.044s	user 0.012s	sys 0.008s
I20260709 12:19:46.163452 31355 raft_consensus.cc:740] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.164799 31355 consensus_queue.cc:260] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.166615 31353 raft_consensus.cc:359] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.167824 31353 raft_consensus.cc:385] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.168134 31353 raft_consensus.cc:740] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.168973 31353 consensus_queue.cc:260] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.169394 31362 raft_consensus.cc:493] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.170085 31362 raft_consensus.cc:515] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.171075 31353 ts_tablet_manager.cc:1434] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:19:46.172164 31353 tablet_bootstrap.cc:492] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72: Bootstrap starting.
I20260709 12:19:46.179194 31355 ts_tablet_manager.cc:1434] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1: Time spent starting tablet: real 0.022s	user 0.011s	sys 0.000s
I20260709 12:19:46.180249 31355 tablet_bootstrap.cc:492] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1: Bootstrap starting.
I20260709 12:19:46.187526 31353 tablet_bootstrap.cc:654] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.180907 31362 leader_election.cc:290] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:19:46.215047 31357 tablet_bootstrap.cc:492] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de: No bootstrap required, opened a new log
I20260709 12:19:46.215761 31357 ts_tablet_manager.cc:1403] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de: Time spent bootstrapping tablet: real 0.085s	user 0.028s	sys 0.015s
I20260709 12:19:46.218313 31362 raft_consensus.cc:493] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.218781 31357 raft_consensus.cc:359] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.218992 31362 raft_consensus.cc:515] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.219527 31357 raft_consensus.cc:385] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.219825 31357 raft_consensus.cc:740] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.220494 31357 consensus_queue.cc:260] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.221570 31362 leader_election.cc:290] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:19:46.222784 31357 ts_tablet_manager.cc:1434] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:19:46.223793 31357 tablet_bootstrap.cc:492] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de: Bootstrap starting.
I20260709 12:19:46.224043 31356 raft_consensus.cc:493] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.224529 31356 raft_consensus.cc:515] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.226493 31356 leader_election.cc:290] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.227105 31360 raft_consensus.cc:493] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.227573 31360 raft_consensus.cc:515] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.229180 31362 raft_consensus.cc:493] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.229743 31362 raft_consensus.cc:515] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.232136 31355 tablet_bootstrap.cc:654] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.230690 31357 tablet_bootstrap.cc:654] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.234285 31360 leader_election.cc:290] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.235005 31362 leader_election.cc:290] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:19:46.238547 31164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.239975 31164 raft_consensus.cc:2468] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 0.
I20260709 12:19:46.244359 31163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "969ee28800d74736a2d0011634d1a014" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.246275 31163 raft_consensus.cc:2468] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 0.
I20260709 12:19:46.246526 31205 leader_election.cc:304] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:46.247866 31356 raft_consensus.cc:2804] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.248498 31356 raft_consensus.cc:493] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.248963 31356 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.248654 31205 leader_election.cc:304] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:46.236027 31362 raft_consensus.cc:493] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.254560 31362 raft_consensus.cc:515] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.273527 31353 tablet_bootstrap.cc:492] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72: No bootstrap required, opened a new log
I20260709 12:19:46.274039 31353 ts_tablet_manager.cc:1403] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72: Time spent bootstrapping tablet: real 0.102s	user 0.012s	sys 0.000s
I20260709 12:19:46.274803 31360 raft_consensus.cc:493] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.275369 31360 raft_consensus.cc:515] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.276878 31353 raft_consensus.cc:359] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.277539 31360 leader_election.cc:290] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.277586 31353 raft_consensus.cc:385] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.278164 31353 raft_consensus.cc:740] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.278204 31360 raft_consensus.cc:493] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.280661 31353 consensus_queue.cc:260] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.286123 31362 leader_election.cc:290] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:19:46.297712 31396 raft_consensus.cc:493] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.298465 31395 raft_consensus.cc:2804] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.280445 31360 raft_consensus.cc:515] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.298478 31396 raft_consensus.cc:515] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.298985 31395 raft_consensus.cc:493] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.302673 31395 raft_consensus.cc:3060] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.312489 31163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.319319 31162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.320072 31162 raft_consensus.cc:2468] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 0.
I20260709 12:19:46.320753 31396 leader_election.cc:290] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.321123 31357 tablet_bootstrap.cc:492] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de: No bootstrap required, opened a new log
I20260709 12:19:46.321254 31396 raft_consensus.cc:493] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:46.321305 31279 leader_election.cc:343] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.321816 31396 raft_consensus.cc:515] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.322158 31164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.322784 31164 raft_consensus.cc:2468] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 0.
I20260709 12:19:46.322727 31279 leader_election.cc:304] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
I20260709 12:19:46.321593 31357 ts_tablet_manager.cc:1403] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de: Time spent bootstrapping tablet: real 0.098s	user 0.018s	sys 0.014s
I20260709 12:19:46.327677 31402 raft_consensus.cc:493] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.328254 31402 raft_consensus.cc:515] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.323942 31362 raft_consensus.cc:2804] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:19:46.330791 31205 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:44183, user_credentials={real_user=slave}} blocked reactor thread for 82528.7us
I20260709 12:19:46.333709 31279 leader_election.cc:304] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
I20260709 12:19:46.333878 31357 raft_consensus.cc:359] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.323832 31360 leader_election.cc:290] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.331079 31362 raft_consensus.cc:493] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.335078 31357 raft_consensus.cc:385] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.335839 31357 raft_consensus.cc:740] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.336706 31357 consensus_queue.cc:260] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.334939 31163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cff8c0586c59461bb83a89ee55000d70" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.338002 31163 raft_consensus.cc:2468] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 0.
I20260709 12:19:46.337373 31164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48cb82524513465785d32448b353ff4a" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.338655 31162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.339491 31162 raft_consensus.cc:2468] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 0.
I20260709 12:19:46.340588 31396 leader_election.cc:290] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.341133 31396 raft_consensus.cc:493] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.340891 31313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
I20260709 12:19:46.341593 31396 raft_consensus.cc:515] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.342361 31312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "969ee28800d74736a2d0011634d1a014" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
I20260709 12:19:46.343338 31312 raft_consensus.cc:2468] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 0.
I20260709 12:19:46.345358 31357 ts_tablet_manager.cc:1434] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de: Time spent starting tablet: real 0.021s	user 0.005s	sys 0.001s
I20260709 12:19:46.346360 31404 raft_consensus.cc:493] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:19:46.347100 31205 leader_election.cc:343] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.346964 31404 raft_consensus.cc:515] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.347833 31205 leader_election.cc:304] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:46.347997 31279 leader_election.cc:304] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
I20260709 12:19:46.348261 31310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dda71ddf86394530aac956f3cc709af1" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
I20260709 12:19:46.307768 31356 raft_consensus.cc:515] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.353175 31311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48cb82524513465785d32448b353ff4a" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
I20260709 12:19:46.353482 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9cd04eb7d0b4e7fbac08a32889020e2" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
I20260709 12:19:46.336647 31399 raft_consensus.cc:493] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.344480 31362 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.354674 31397 raft_consensus.cc:493] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.354166 31309 raft_consensus.cc:2468] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 0.
W20260709 12:19:46.354997 31205 leader_election.cc:343] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.343743 31396 leader_election.cc:290] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.354848 31395 raft_consensus.cc:515] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.361668 31399 raft_consensus.cc:515] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.357263 31360 raft_consensus.cc:493] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.368731 31399 leader_election.cc:290] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.355230 31397 raft_consensus.cc:515] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.375321 31397 leader_election.cc:290] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.362142 31205 leader_election.cc:304] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7f7f401a756482ea721ae7015101f72; no voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1
I20260709 12:19:46.378260 31399 raft_consensus.cc:493] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.379798 31162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dda71ddf86394530aac956f3cc709af1" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.380967 31162 raft_consensus.cc:2468] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 0.
I20260709 12:19:46.380861 31399 raft_consensus.cc:515] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.389731 31357 tablet_bootstrap.cc:492] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de: Bootstrap starting.
I20260709 12:19:46.361308 31402 leader_election.cc:290] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.367168 31396 raft_consensus.cc:2804] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.395542 31404 leader_election.cc:290] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:19:46.400646 31355 tablet_bootstrap.cc:492] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1: No bootstrap required, opened a new log
I20260709 12:19:46.401180 31355 ts_tablet_manager.cc:1403] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1: Time spent bootstrapping tablet: real 0.221s	user 0.014s	sys 0.007s
I20260709 12:19:46.401572 31353 ts_tablet_manager.cc:1434] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72: Time spent starting tablet: real 0.127s	user 0.024s	sys 0.048s
I20260709 12:19:46.402957 31353 tablet_bootstrap.cc:492] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72: Bootstrap starting.
I20260709 12:19:46.398689 31396 raft_consensus.cc:493] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.379065 31360 raft_consensus.cc:515] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.398476 31402 raft_consensus.cc:2749] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.400950 31404 raft_consensus.cc:2804] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.416110 31162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9cd04eb7d0b4e7fbac08a32889020e2" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.424185 31162 raft_consensus.cc:2468] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 0.
I20260709 12:19:46.413312 31396 raft_consensus.cc:3060] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.431308 31355 raft_consensus.cc:359] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.432422 31355 raft_consensus.cc:385] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.432864 31355 raft_consensus.cc:740] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.440136 31396 raft_consensus.cc:515] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.441363 31355 consensus_queue.cc:260] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.420047 31404 raft_consensus.cc:493] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.448506 31355 ts_tablet_manager.cc:1434] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1: Time spent starting tablet: real 0.047s	user 0.007s	sys 0.001s
I20260709 12:19:46.450094 31355 tablet_bootstrap.cc:492] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1: Bootstrap starting.
W20260709 12:19:46.411216 31279 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:44183, user_credentials={real_user=slave}} blocked reactor thread for 63303.8us
I20260709 12:19:46.454277 31405 raft_consensus.cc:2804] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.454711 31405 raft_consensus.cc:493] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.455042 31405 raft_consensus.cc:3060] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.449836 31404 raft_consensus.cc:3060] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.453248 31356 leader_election.cc:290] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.476045 31360 leader_election.cc:290] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.464979 31362 raft_consensus.cc:515] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.479328 31310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
W20260709 12:19:46.479759 31339 tablet.cc:2406] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:19:46.480356 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f138178dea44e59804eaa33c221d4aa" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
I20260709 12:19:46.479311 31357 tablet_bootstrap.cc:654] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.480976 31309 raft_consensus.cc:2468] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 0.
I20260709 12:19:46.476668 31396 leader_election.cc:290] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.484306 31403 raft_consensus.cc:493] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.485872 31395 leader_election.cc:290] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.486316 31310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cff8c0586c59461bb83a89ee55000d70" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
I20260709 12:19:46.474133 31162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f138178dea44e59804eaa33c221d4aa" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.480943 31399 leader_election.cc:290] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.493184 31311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
I20260709 12:19:46.495669 31313 raft_consensus.cc:2393] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e7f7f401a756482ea721ae7015101f72 in current term 1: Already voted for candidate de914d6d5a0046c99ab0fe321d40e8c1 in this term.
I20260709 12:19:46.495735 31162 raft_consensus.cc:2468] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 0.
I20260709 12:19:46.497639 31311 raft_consensus.cc:2468] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 0.
W20260709 12:19:46.498471 31205 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:35629, user_credentials={real_user=slave}} blocked reactor thread for 142287us
I20260709 12:19:46.498654 31362 leader_election.cc:290] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:19:46.501577 31164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cff8c0586c59461bb83a89ee55000d70" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.502868 31311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
I20260709 12:19:46.503105 31163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:46.502538 31164 raft_consensus.cc:2468] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 0.
I20260709 12:19:46.507578 31353 tablet_bootstrap.cc:654] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.504264 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:46.490577 31404 raft_consensus.cc:515] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.508762 31309 raft_consensus.cc:2393] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e7f7f401a756482ea721ae7015101f72 in current term 1: Already voted for candidate de914d6d5a0046c99ab0fe321d40e8c1 in this term.
I20260709 12:19:46.508515 31313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cff8c0586c59461bb83a89ee55000d70" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
I20260709 12:19:46.510406 31312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9cd04eb7d0b4e7fbac08a32889020e2" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
I20260709 12:19:46.504367 31163 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.510778 31279 leader_election.cc:304] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
I20260709 12:19:46.504402 31161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:46.513939 31161 raft_consensus.cc:3060] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.514065 31362 raft_consensus.cc:2804] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.504797 31162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:46.495128 31403 raft_consensus.cc:515] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.515338 31312 raft_consensus.cc:2468] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 0.
I20260709 12:19:46.516249 31362 raft_consensus.cc:493] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.506206 31160 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "969ee28800d74736a2d0011634d1a014" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:46.517694 31130 leader_election.cc:304] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
I20260709 12:19:46.524163 31160 raft_consensus.cc:3060] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.524513 31205 leader_election.cc:304] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:46.525331 31357 tablet_bootstrap.cc:492] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de: No bootstrap required, opened a new log
I20260709 12:19:46.525777 31357 ts_tablet_manager.cc:1403] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de: Time spent bootstrapping tablet: real 0.136s	user 0.016s	sys 0.011s
I20260709 12:19:46.522054 31362 raft_consensus.cc:3060] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.528158 31164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.531826 31164 raft_consensus.cc:2468] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 0.
I20260709 12:19:46.523411 31311 raft_consensus.cc:2468] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 0.
I20260709 12:19:46.536213 31312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:46.529119 31205 leader_election.cc:304] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:46.537781 31399 raft_consensus.cc:2804] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.538141 31399 raft_consensus.cc:493] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.538479 31399 raft_consensus.cc:3060] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.540657 31407 raft_consensus.cc:2804] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.541111 31407 raft_consensus.cc:493] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.541461 31407 raft_consensus.cc:3060] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.533380 31360 raft_consensus.cc:2804] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.542059 31360 raft_consensus.cc:493] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.543681 31360 raft_consensus.cc:3060] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.529316 31403 leader_election.cc:290] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:19:46.545397 31205 leader_election.cc:304] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:46.548959 31395 raft_consensus.cc:2804] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.553467 31205 leader_election.cc:304] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:46.554908 31357 raft_consensus.cc:359] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.555285 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "969ee28800d74736a2d0011634d1a014" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:46.555557 31396 raft_consensus.cc:2804] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.555816 31357 raft_consensus.cc:385] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.555967 31396 raft_consensus.cc:493] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.555992 31309 raft_consensus.cc:3060] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.556213 31357 raft_consensus.cc:740] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.556437 31396 raft_consensus.cc:3060] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.558064 31164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de" is_pre_election: true
I20260709 12:19:46.552500 31395 raft_consensus.cc:493] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.559828 31395 raft_consensus.cc:3060] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.561304 31160 raft_consensus.cc:2468] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 1.
I20260709 12:19:46.546030 31130 leader_election.cc:304] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
I20260709 12:19:46.554062 31312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
I20260709 12:19:46.563301 31312 raft_consensus.cc:2393] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e7f7f401a756482ea721ae7015101f72 in current term 1: Already voted for candidate de914d6d5a0046c99ab0fe321d40e8c1 in this term.
I20260709 12:19:46.563273 31130 leader_election.cc:304] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
W20260709 12:19:46.565054 31205 leader_election.cc:343] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:19:46.553958 31404 leader_election.cc:290] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:19:46.565678 31397 raft_consensus.cc:2804] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.570308 31357 consensus_queue.cc:260] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.572321 31406 raft_consensus.cc:2804] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.573318 31397 raft_consensus.cc:493] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.574158 31406 raft_consensus.cc:493] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.574271 31397 raft_consensus.cc:3060] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.574559 31406 raft_consensus.cc:3060] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.564214 31164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:46.582077 31357 ts_tablet_manager.cc:1434] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de: Time spent starting tablet: real 0.056s	user 0.004s	sys 0.007s
I20260709 12:19:46.579813 31164 raft_consensus.cc:3060] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.562327 31405 raft_consensus.cc:515] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.587075 31310 raft_consensus.cc:2393] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f94dd3d7b0948e092e65f629d8af1de in current term 1: Already voted for candidate de914d6d5a0046c99ab0fe321d40e8c1 in this term.
I20260709 12:19:46.585721 31309 raft_consensus.cc:2468] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 1.
I20260709 12:19:46.598574 31353 tablet_bootstrap.cc:492] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72: No bootstrap required, opened a new log
I20260709 12:19:46.599193 31353 ts_tablet_manager.cc:1403] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72: Time spent bootstrapping tablet: real 0.197s	user 0.013s	sys 0.018s
I20260709 12:19:46.601217 31163 raft_consensus.cc:2468] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 1.
I20260709 12:19:46.603996 31239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72" is_pre_election: true
I20260709 12:19:46.605077 31411 raft_consensus.cc:493] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:19:46.605216 31239 raft_consensus.cc:2468] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 0.
I20260709 12:19:46.596318 31399 raft_consensus.cc:515] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.606339 31355 tablet_bootstrap.cc:654] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:19:46.610219 31399 leader_election.cc:290] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.607010 31238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72" is_pre_election: true
I20260709 12:19:46.615103 31239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72" is_pre_election: true
I20260709 12:19:46.616349 31239 raft_consensus.cc:2393] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de914d6d5a0046c99ab0fe321d40e8c1 in current term 1: Already voted for candidate e7f7f401a756482ea721ae7015101f72 in this term.
I20260709 12:19:46.616238 31163 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cff8c0586c59461bb83a89ee55000d70" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:46.605592 31411 raft_consensus.cc:515] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.619866 31163 raft_consensus.cc:3060] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.620235 31405 leader_election.cc:290] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:19:46.624526 31362 raft_consensus.cc:515] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.628090 31362 leader_election.cc:290] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:19:46.629540 31239 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cff8c0586c59461bb83a89ee55000d70" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72" is_pre_election: true
I20260709 12:19:46.628019 31160 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f138178dea44e59804eaa33c221d4aa" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:46.632194 31234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:46.638957 31353 raft_consensus.cc:359] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.637382 31161 raft_consensus.cc:2468] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 1.
I20260709 12:19:46.633278 31163 raft_consensus.cc:2468] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 1.
I20260709 12:19:46.640182 31232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f138178dea44e59804eaa33c221d4aa" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:46.640225 31353 raft_consensus.cc:385] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.641607 31353 raft_consensus.cc:740] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.634074 31395 raft_consensus.cc:515] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.611402 31407 raft_consensus.cc:515] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.641902 31236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:46.643807 31279 leader_election.cc:304] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
I20260709 12:19:46.643811 31353 consensus_queue.cc:260] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.624526 31396 raft_consensus.cc:515] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.639464 31205 leader_election.cc:304] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:46.649752 31355 tablet_bootstrap.cc:492] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1: No bootstrap required, opened a new log
I20260709 12:19:46.650156 31355 ts_tablet_manager.cc:1403] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1: Time spent bootstrapping tablet: real 0.201s	user 0.009s	sys 0.025s
I20260709 12:19:46.645753 31362 raft_consensus.cc:2804] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.648692 31237 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f138178dea44e59804eaa33c221d4aa" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72" is_pre_election: true
I20260709 12:19:46.650841 31399 raft_consensus.cc:2804] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.651158 31238 raft_consensus.cc:2393] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de914d6d5a0046c99ab0fe321d40e8c1 in current term 1: Already voted for candidate e7f7f401a756482ea721ae7015101f72 in this term.
I20260709 12:19:46.651330 31411 leader_election.cc:290] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.652577 31353 ts_tablet_manager.cc:1434] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72: Time spent starting tablet: real 0.053s	user 0.008s	sys 0.000s
I20260709 12:19:46.653283 31205 leader_election.cc:304] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: de914d6d5a0046c99ab0fe321d40e8c1
I20260709 12:19:46.645599 31233 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cff8c0586c59461bb83a89ee55000d70" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:46.647019 31232 raft_consensus.cc:3060] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.660640 31235 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72" is_pre_election: true
I20260709 12:19:46.661283 31395 leader_election.cc:290] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.661628 31235 raft_consensus.cc:2393] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de914d6d5a0046c99ab0fe321d40e8c1 in current term 1: Already voted for candidate e7f7f401a756482ea721ae7015101f72 in this term.
I20260709 12:19:46.653307 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48cb82524513465785d32448b353ff4a" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" is_pre_election: true
I20260709 12:19:46.652415 31239 raft_consensus.cc:2393] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate de914d6d5a0046c99ab0fe321d40e8c1 in current term 1: Already voted for candidate e7f7f401a756482ea721ae7015101f72 in this term.
I20260709 12:19:46.661890 31238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72" is_pre_election: true
I20260709 12:19:46.655287 31234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72" is_pre_election: true
I20260709 12:19:46.660343 31399 raft_consensus.cc:697] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Becoming Leader. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:19:46.655640 31236 raft_consensus.cc:2393] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate de914d6d5a0046c99ab0fe321d40e8c1 in current term 1: Already voted for candidate e7f7f401a756482ea721ae7015101f72 in this term.
I20260709 12:19:46.666944 31395 raft_consensus.cc:2804] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.658906 31362 raft_consensus.cc:697] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 LEADER]: Becoming Leader. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Running, Role: LEADER
I20260709 12:19:46.666994 31230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cff8c0586c59461bb83a89ee55000d70" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72" is_pre_election: true
I20260709 12:19:46.669021 31230 raft_consensus.cc:2393] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f94dd3d7b0948e092e65f629d8af1de in current term 1: Already voted for candidate e7f7f401a756482ea721ae7015101f72 in this term.
I20260709 12:19:46.665078 31238 raft_consensus.cc:2468] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 0.
I20260709 12:19:46.669613 31395 raft_consensus.cc:697] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Becoming Leader. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:19:46.659179 31231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f138178dea44e59804eaa33c221d4aa" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72" is_pre_election: true
I20260709 12:19:46.674309 31397 raft_consensus.cc:515] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.675405 31395 consensus_queue.cc:237] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.676533 31362 consensus_queue.cc:237] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.665711 31309 raft_consensus.cc:2468] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 0.
I20260709 12:19:46.665477 31205 leader_election.cc:304] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: de914d6d5a0046c99ab0fe321d40e8c1
I20260709 12:19:46.678287 31129 leader_election.cc:304] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de; no voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72
I20260709 12:19:46.680786 31234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9cd04eb7d0b4e7fbac08a32889020e2" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72" is_pre_election: true
I20260709 12:19:46.681608 31234 raft_consensus.cc:2393] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f94dd3d7b0948e092e65f629d8af1de in current term 1: Already voted for candidate e7f7f401a756482ea721ae7015101f72 in this term.
I20260709 12:19:46.681684 31164 raft_consensus.cc:2468] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 1.
I20260709 12:19:46.681823 31130 leader_election.cc:304] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
I20260709 12:19:46.678983 31236 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:46.679610 31310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:46.682819 31310 raft_consensus.cc:3060] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.683293 31279 leader_election.cc:304] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
I20260709 12:19:46.682870 31399 consensus_queue.cc:237] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.679597 31396 leader_election.cc:290] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.679951 31231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48cb82524513465785d32448b353ff4a" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72" is_pre_election: true
I20260709 12:19:46.664740 31360 raft_consensus.cc:515] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.680663 31397 leader_election.cc:290] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 election: Requested vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.672405 31406 raft_consensus.cc:515] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.682912 31129 leader_election.cc:304] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de; no voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72
I20260709 12:19:46.689491 31356 raft_consensus.cc:2804] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.689316 31411 raft_consensus.cc:2749] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.690394 31236 raft_consensus.cc:3060] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.689574 31405 raft_consensus.cc:2804] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.695653 31278 leader_election.cc:304] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72; no voters: 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:19:46.692075 31397 raft_consensus.cc:2804] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:19:46.699476 31355 raft_consensus.cc:359] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.700238 31355 raft_consensus.cc:385] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:19:46.698740 31160 raft_consensus.cc:2393] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate de914d6d5a0046c99ab0fe321d40e8c1 in current term 1: Already voted for candidate 8f94dd3d7b0948e092e65f629d8af1de in this term.
I20260709 12:19:46.700673 31355 raft_consensus.cc:740] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Initialized, Role: FOLLOWER
I20260709 12:19:46.700237 31356 raft_consensus.cc:697] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Becoming Leader. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:19:46.691157 31231 raft_consensus.cc:2468] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 0.
I20260709 12:19:46.701613 31355 consensus_queue.cc:260] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.704952 31356 consensus_queue.cc:237] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.706311 31407 leader_election.cc:290] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.707334 31405 raft_consensus.cc:697] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 LEADER]: Becoming Leader. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Running, Role: LEADER
I20260709 12:19:46.708616 31355 ts_tablet_manager.cc:1434] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1: Time spent starting tablet: real 0.058s	user 0.004s	sys 0.002s
I20260709 12:19:46.708561 31405 consensus_queue.cc:237] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.709653 31164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9cd04eb7d0b4e7fbac08a32889020e2" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:46.709991 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9cd04eb7d0b4e7fbac08a32889020e2" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:46.710568 31164 raft_consensus.cc:2393] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e7f7f401a756482ea721ae7015101f72 in current term 1: Already voted for candidate 8f94dd3d7b0948e092e65f629d8af1de in this term.
I20260709 12:19:46.715068 31238 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f138178dea44e59804eaa33c221d4aa" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:46.715325 31234 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9cd04eb7d0b4e7fbac08a32889020e2" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:46.716009 31234 raft_consensus.cc:2393] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f94dd3d7b0948e092e65f629d8af1de in current term 1: Already voted for candidate e7f7f401a756482ea721ae7015101f72 in this term.
I20260709 12:19:46.700156 31397 raft_consensus.cc:493] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:19:46.716954 31397 raft_consensus.cc:3060] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.718575 31360 leader_election.cc:290] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 election: Requested vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.710556 31309 raft_consensus.cc:3060] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.725054 31414 raft_consensus.cc:2749] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.732470 31397 raft_consensus.cc:515] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.735546 31231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48cb82524513465785d32448b353ff4a" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:46.736159 31231 raft_consensus.cc:3060] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.736178 31311 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cff8c0586c59461bb83a89ee55000d70" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:46.739101 31160 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cff8c0586c59461bb83a89ee55000d70" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:46.740022 31160 raft_consensus.cc:2393] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e7f7f401a756482ea721ae7015101f72 in current term 1: Already voted for candidate de914d6d5a0046c99ab0fe321d40e8c1 in this term.
I20260709 12:19:46.713954 31232 raft_consensus.cc:2468] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 1.
I20260709 12:19:46.741520 31312 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:46.747776 31160 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dda71ddf86394530aac956f3cc709af1" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:46.748384 31160 raft_consensus.cc:3060] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.751144 31397 leader_election.cc:290] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 election: Requested vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.752290 31205 leader_election.cc:304] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7f7f401a756482ea721ae7015101f72; no voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1
I20260709 12:19:46.753535 31356 raft_consensus.cc:2749] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.754433 31310 raft_consensus.cc:2468] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 1.
I20260709 12:19:46.747776 31313 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dda71ddf86394530aac956f3cc709af1" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:46.755119 31403 raft_consensus.cc:2764] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:19:46.755266 31313 raft_consensus.cc:3060] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.754268 31164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:46.750487 31309 raft_consensus.cc:2468] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 1.
I20260709 12:19:46.760375 31062 catalog_manager.cc:5697] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:46.764407 31231 raft_consensus.cc:2468] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 1.
I20260709 12:19:46.767119 31278 leader_election.cc:304] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de914d6d5a0046c99ab0fe321d40e8c1; no voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72
I20260709 12:19:46.770673 31278 leader_election.cc:304] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: de914d6d5a0046c99ab0fe321d40e8c1; no voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72
I20260709 12:19:46.771831 31404 raft_consensus.cc:3060] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.772493 31129 leader_election.cc:304] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:46.772958 31236 raft_consensus.cc:2468] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 1.
I20260709 12:19:46.773380 31397 raft_consensus.cc:2804] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.773876 31397 raft_consensus.cc:697] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [term 1 LEADER]: Becoming Leader. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:19:46.773996 31310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9cd04eb7d0b4e7fbac08a32889020e2" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:46.774817 31310 raft_consensus.cc:2393] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f94dd3d7b0948e092e65f629d8af1de in current term 1: Already voted for candidate e7f7f401a756482ea721ae7015101f72 in this term.
I20260709 12:19:46.775004 31313 raft_consensus.cc:2468] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 1.
I20260709 12:19:46.774773 31397 consensus_queue.cc:237] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.776239 31405 raft_consensus.cc:2749] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.776362 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f138178dea44e59804eaa33c221d4aa" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:46.777318 31309 raft_consensus.cc:2393] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8f94dd3d7b0948e092e65f629d8af1de in current term 1: Already voted for candidate de914d6d5a0046c99ab0fe321d40e8c1 in this term.
I20260709 12:19:46.777882 31129 leader_election.cc:304] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:46.779528 31414 raft_consensus.cc:2804] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.779983 31414 raft_consensus.cc:697] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 LEADER]: Becoming Leader. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:19:46.780673 31160 raft_consensus.cc:2468] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 1.
I20260709 12:19:46.780812 31404 raft_consensus.cc:2749] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.771750 31164 raft_consensus.cc:2393] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e7f7f401a756482ea721ae7015101f72 in current term 1: Already voted for candidate de914d6d5a0046c99ab0fe321d40e8c1 in this term.
I20260709 12:19:46.778364 31205 leader_election.cc:304] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72; no voters: 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:19:46.780803 31414 consensus_queue.cc:237] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.786888 31130 leader_election.cc:304] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de; no voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72
I20260709 12:19:46.788221 31411 raft_consensus.cc:2749] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.788373 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "48cb82524513465785d32448b353ff4a" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:46.789005 31309 raft_consensus.cc:3060] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:19:46.790616 31395 raft_consensus.cc:2804] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.797752 31395 raft_consensus.cc:697] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Becoming Leader. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:19:46.799145 31395 consensus_queue.cc:237] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.801179 31205 leader_election.cc:304] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:46.802959 31278 leader_election.cc:304] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72; no voters: 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:19:46.803478 31399 raft_consensus.cc:2804] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.803537 31406 leader_election.cc:290] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 election: Requested vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:46.796991 31130 leader_election.cc:304] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de; no voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72
I20260709 12:19:46.804104 31399 raft_consensus.cc:697] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Becoming Leader. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:19:46.804585 31406 raft_consensus.cc:2749] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.802263 31205 leader_election.cc:304] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e7f7f401a756482ea721ae7015101f72; no voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1
I20260709 12:19:46.807296 31403 raft_consensus.cc:2804] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:19:46.804958 31399 consensus_queue.cc:237] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.808352 31403 raft_consensus.cc:697] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 LEADER]: Becoming Leader. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Running, Role: LEADER
I20260709 12:19:46.809613 31403 consensus_queue.cc:237] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:46.812247 31407 raft_consensus.cc:2749] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:19:46.814302 31059 catalog_manager.cc:5697] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193). New cstate: current_term: 1 leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.827950 31309 raft_consensus.cc:2468] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 1.
I20260709 12:19:46.836654 31060 catalog_manager.cc:5697] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f7f401a756482ea721ae7015101f72 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "e7f7f401a756482ea721ae7015101f72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.838922 31060 catalog_manager.cc:5697] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f7f401a756482ea721ae7015101f72 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "e7f7f401a756482ea721ae7015101f72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.840956 31060 catalog_manager.cc:5697] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f7f401a756482ea721ae7015101f72 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "e7f7f401a756482ea721ae7015101f72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.843056 31060 catalog_manager.cc:5697] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f7f401a756482ea721ae7015101f72 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "e7f7f401a756482ea721ae7015101f72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.849344 31062 catalog_manager.cc:5697] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193). New cstate: current_term: 1 leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.856442 31059 catalog_manager.cc:5697] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:46.858736 31059 catalog_manager.cc:5697] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:46.868366 31062 catalog_manager.cc:5697] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 reported cstate change: term changed from 0 to 1, leader changed from <none> to e7f7f401a756482ea721ae7015101f72 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "e7f7f401a756482ea721ae7015101f72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:46.901141 31232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e9cd04eb7d0b4e7fbac08a32889020e2"
dest_uuid: "e7f7f401a756482ea721ae7015101f72"
mode: GRACEFUL
new_leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:47210
I20260709 12:19:46.901821 31232 raft_consensus.cc:606] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Received request to transfer leadership to TS 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:19:46.907554 31232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dda71ddf86394530aac956f3cc709af1"
dest_uuid: "e7f7f401a756482ea721ae7015101f72"
mode: GRACEFUL
new_leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:47210
I20260709 12:19:46.908084 31232 raft_consensus.cc:606] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Received request to transfer leadership to TS 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:19:46.913196 31232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399"
dest_uuid: "e7f7f401a756482ea721ae7015101f72"
mode: GRACEFUL
new_leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:47210
I20260709 12:19:46.913761 31232 raft_consensus.cc:606] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Received request to transfer leadership to TS 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:19:46.918947 31232 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f"
dest_uuid: "e7f7f401a756482ea721ae7015101f72"
mode: GRACEFUL
new_leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:47210
I20260709 12:19:46.919528 31232 raft_consensus.cc:606] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Received request to transfer leadership to TS 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:19:46.935976 31309 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b"
dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
mode: GRACEFUL
new_leader_uuid: "e7f7f401a756482ea721ae7015101f72"
 from {username='slave'} at 127.0.0.1:57506
I20260709 12:19:46.936571 31309 raft_consensus.cc:606] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 LEADER]: Received request to transfer leadership to TS e7f7f401a756482ea721ae7015101f72
I20260709 12:19:46.952865 31164 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
mode: GRACEFUL
new_leader_uuid: "e7f7f401a756482ea721ae7015101f72"
 from {username='slave'} at 127.0.0.1:58582
I20260709 12:19:46.953423 31164 raft_consensus.cc:606] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 LEADER]: Received request to transfer leadership to TS e7f7f401a756482ea721ae7015101f72
I20260709 12:19:46.958576 31309 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0f138178dea44e59804eaa33c221d4aa"
dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
mode: GRACEFUL
new_leader_uuid: "e7f7f401a756482ea721ae7015101f72"
 from {username='slave'} at 127.0.0.1:57506
I20260709 12:19:46.959200 31309 raft_consensus.cc:606] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 LEADER]: Received request to transfer leadership to TS e7f7f401a756482ea721ae7015101f72
I20260709 12:19:47.091985 31399 consensus_queue.cc:1048] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:47.105625 31395 consensus_queue.cc:1048] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:47.131831 31399 consensus_queue.cc:1048] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.138872 31395 raft_consensus.cc:993] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72: : Instructing follower 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:19:47.139400 31407 raft_consensus.cc:1081] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Signalling peer 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:19:47.142994 31164 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:58562
I20260709 12:19:47.143626 31164 raft_consensus.cc:493] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:47.143989 31164 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.144019 31399 consensus_queue.cc:1048] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:47.151912 31164 raft_consensus.cc:515] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.154127 31164 leader_election.cc:290] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 2 election: Requested vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:47.156319 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:47.156673 31232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:47.156941 31309 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.157259 31232 raft_consensus.cc:3055] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:47.157567 31232 raft_consensus.cc:740] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:19:47.157615 31403 consensus_queue.cc:1048] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.158411 31232 consensus_queue.cc:260] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.159730 31232 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.164753 31309 raft_consensus.cc:2468] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 2.
I20260709 12:19:47.167245 31232 raft_consensus.cc:2468] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 2.
I20260709 12:19:47.168533 31129 leader_election.cc:304] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:47.169541 31414 raft_consensus.cc:2804] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:47.170037 31414 raft_consensus.cc:697] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Becoming Leader. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:19:47.170933 31414 consensus_queue.cc:237] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.172453 31362 raft_consensus.cc:993] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1: : Instructing follower e7f7f401a756482ea721ae7015101f72 to start an election
I20260709 12:19:47.172909 31362 raft_consensus.cc:1081] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 LEADER]: Signalling peer e7f7f401a756482ea721ae7015101f72 to start an election
I20260709 12:19:47.177681 31236 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b"
dest_uuid: "e7f7f401a756482ea721ae7015101f72"
 from {username='slave'} at 127.0.0.1:47192
I20260709 12:19:47.178339 31236 raft_consensus.cc:493] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:47.178768 31236 raft_consensus.cc:3060] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.181126 31362 consensus_queue.cc:1048] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:47.183336 31060 catalog_manager.cc:5697] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de reported cstate change: term changed from 1 to 2, leader changed from e7f7f401a756482ea721ae7015101f72 (127.29.201.194) to 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193). New cstate: current_term: 2 leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:47.187870 31236 raft_consensus.cc:515] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.190331 31236 leader_election.cc:290] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 2 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:47.192302 31160 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:47.197079 31362 consensus_queue.cc:1048] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.197700 31414 consensus_queue.cc:1048] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.198408 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:47.199048 31309 raft_consensus.cc:3055] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:47.199373 31309 raft_consensus.cc:740] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Running, Role: LEADER
I20260709 12:19:47.200476 31309 consensus_queue.cc:260] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.202112 31309 raft_consensus.cc:3060] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.215226 31309 raft_consensus.cc:2468] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 2.
I20260709 12:19:47.216722 31205 leader_election.cc:304] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72; no voters: 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:19:47.217865 31399 raft_consensus.cc:2804] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:47.218372 31399 raft_consensus.cc:697] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 2 LEADER]: Becoming Leader. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:19:47.220548 31399 consensus_queue.cc:237] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.221434 31360 consensus_queue.cc:1048] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.225911 31404 consensus_queue.cc:1048] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.229077 31062 catalog_manager.cc:5697] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 reported cstate change: term changed from 1 to 2, leader changed from de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195) to e7f7f401a756482ea721ae7015101f72 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "e7f7f401a756482ea721ae7015101f72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:47.234050 31407 consensus_queue.cc:1048] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:47.268119 31360 raft_consensus.cc:993] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de: : Instructing follower e7f7f401a756482ea721ae7015101f72 to start an election
I20260709 12:19:47.268572 31360 raft_consensus.cc:1081] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 LEADER]: Signalling peer e7f7f401a756482ea721ae7015101f72 to start an election
I20260709 12:19:47.269145 31404 consensus_queue.cc:1048] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.269171 31397 consensus_queue.cc:1048] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:47.270380 31231 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8"
dest_uuid: "e7f7f401a756482ea721ae7015101f72"
 from {username='slave'} at 127.0.0.1:47202
I20260709 12:19:47.270985 31231 raft_consensus.cc:493] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:47.271360 31231 raft_consensus.cc:3060] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.275203 31395 consensus_queue.cc:1048] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:47.278020 31403 consensus_queue.cc:1048] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.281520 31231 raft_consensus.cc:515] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.285511 31161 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:47.286360 31161 raft_consensus.cc:3055] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:47.286835 31161 raft_consensus.cc:740] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:19:47.287958 31161 consensus_queue.cc:260] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.290062 31161 raft_consensus.cc:3060] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.301189 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:47.301966 31309 raft_consensus.cc:3060] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.305469 31411 consensus_queue.cc:1048] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:47.312939 31161 raft_consensus.cc:2468] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 2.
I20260709 12:19:47.316092 31205 leader_election.cc:304] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:47.311179 31231 leader_election.cc:290] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 2 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:47.319671 31395 raft_consensus.cc:2804] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:47.320379 31395 raft_consensus.cc:697] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 2 LEADER]: Becoming Leader. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:19:47.323143 31309 raft_consensus.cc:2468] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 2.
I20260709 12:19:47.321610 31395 consensus_queue.cc:237] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.326021 31356 consensus_queue.cc:1048] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:47.335003 31404 raft_consensus.cc:993] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1: : Instructing follower e7f7f401a756482ea721ae7015101f72 to start an election
I20260709 12:19:47.337585 31404 raft_consensus.cc:1081] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 LEADER]: Signalling peer e7f7f401a756482ea721ae7015101f72 to start an election
I20260709 12:19:47.351377 31231 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0f138178dea44e59804eaa33c221d4aa"
dest_uuid: "e7f7f401a756482ea721ae7015101f72"
 from {username='slave'} at 127.0.0.1:47192
I20260709 12:19:47.352725 31231 raft_consensus.cc:493] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:47.353197 31231 raft_consensus.cc:3060] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.355255 31395 consensus_queue.cc:1048] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:19:47.362066 31231 raft_consensus.cc:515] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.364532 31231 leader_election.cc:290] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 2 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:47.369640 31395 consensus_queue.cc:1048] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.370083 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f138178dea44e59804eaa33c221d4aa" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:47.371130 31309 raft_consensus.cc:3055] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:47.371497 31309 raft_consensus.cc:740] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Running, Role: LEADER
I20260709 12:19:47.371621 31162 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0f138178dea44e59804eaa33c221d4aa" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:47.372865 31309 consensus_queue.cc:260] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.374238 31309 raft_consensus.cc:3060] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.381304 31395 consensus_queue.cc:1048] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.383477 31059 catalog_manager.cc:5697] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 reported cstate change: term changed from 1 to 2, leader changed from 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193) to e7f7f401a756482ea721ae7015101f72 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "e7f7f401a756482ea721ae7015101f72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:47.389101 31356 raft_consensus.cc:993] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72: : Instructing follower 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:19:47.389591 31356 raft_consensus.cc:1081] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Signalling peer 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:19:47.392304 31356 raft_consensus.cc:993] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72: : Instructing follower 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:19:47.392727 31356 raft_consensus.cc:1081] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Signalling peer 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:19:47.397272 31309 raft_consensus.cc:2468] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 2.
I20260709 12:19:47.399648 31160 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e9cd04eb7d0b4e7fbac08a32889020e2"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:58562
I20260709 12:19:47.400265 31160 raft_consensus.cc:493] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:47.404788 31160 raft_consensus.cc:3060] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.410431 31160 raft_consensus.cc:515] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.413405 31232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9cd04eb7d0b4e7fbac08a32889020e2" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:47.413673 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9cd04eb7d0b4e7fbac08a32889020e2" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:47.414062 31232 raft_consensus.cc:3055] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:47.414197 31309 raft_consensus.cc:3060] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.414422 31232 raft_consensus.cc:740] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:19:47.415319 31232 consensus_queue.cc:260] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.416244 31232 raft_consensus.cc:3060] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.419750 31164 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:58562
I20260709 12:19:47.420395 31164 raft_consensus.cc:493] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:47.420805 31164 raft_consensus.cc:3060] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.424630 31205 leader_election.cc:304] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72; no voters: 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:19:47.425884 31407 raft_consensus.cc:2804] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:47.423645 31232 raft_consensus.cc:2468] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 2.
I20260709 12:19:47.428536 31129 leader_election.cc:304] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:47.429423 31411 raft_consensus.cc:2804] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:47.429924 31411 raft_consensus.cc:697] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Becoming Leader. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:19:47.430126 31160 leader_election.cc:290] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 2 election: Requested vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:47.426421 31407 raft_consensus.cc:697] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [term 2 LEADER]: Becoming Leader. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:19:47.430783 31411 consensus_queue.cc:237] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.431376 31407 consensus_queue.cc:237] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.435171 31399 raft_consensus.cc:993] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72: : Instructing follower 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:19:47.435689 31399 raft_consensus.cc:1081] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Signalling peer 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:19:47.435853 31309 raft_consensus.cc:2468] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 2.
I20260709 12:19:47.443761 31164 raft_consensus.cc:515] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.444803 31160 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dda71ddf86394530aac956f3cc709af1"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:58562
I20260709 12:19:47.445407 31160 raft_consensus.cc:493] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:47.445791 31160 raft_consensus.cc:3060] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.448285 31060 catalog_manager.cc:5697] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de reported cstate change: term changed from 1 to 2, leader changed from e7f7f401a756482ea721ae7015101f72 (127.29.201.194) to 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193). New cstate: current_term: 2 leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:47.451198 31310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:47.449190 31232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:47.451802 31310 raft_consensus.cc:3060] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.452790 31232 raft_consensus.cc:3055] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:47.453253 31232 raft_consensus.cc:740] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:19:47.453109 31160 raft_consensus.cc:515] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.454377 31232 consensus_queue.cc:260] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.455760 31232 raft_consensus.cc:3060] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.456192 31231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dda71ddf86394530aac956f3cc709af1" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:47.456799 31231 raft_consensus.cc:3055] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:19:47.457108 31231 raft_consensus.cc:740] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:19:47.457931 31231 consensus_queue.cc:260] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.459059 31231 raft_consensus.cc:3060] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.459560 31310 raft_consensus.cc:2468] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 2.
I20260709 12:19:47.460959 31130 leader_election.cc:304] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
I20260709 12:19:47.461786 31411 raft_consensus.cc:2804] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:47.462253 31411 raft_consensus.cc:697] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Becoming Leader. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:19:47.463167 31411 consensus_queue.cc:237] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.464843 31232 raft_consensus.cc:2468] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 2.
I20260709 12:19:47.465497 31062 catalog_manager.cc:5697] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 reported cstate change: term changed from 1 to 2, leader changed from de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195) to e7f7f401a756482ea721ae7015101f72 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "e7f7f401a756482ea721ae7015101f72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:47.468731 31164 leader_election.cc:290] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 2 election: Requested vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:47.469926 31160 leader_election.cc:290] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 2 election: Requested vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:19:47.471107 31310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dda71ddf86394530aac956f3cc709af1" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:19:47.471715 31310 raft_consensus.cc:3060] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.471668 31231 raft_consensus.cc:2468] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 2.
I20260709 12:19:47.472877 31129 leader_election.cc:304] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:47.473727 31411 raft_consensus.cc:2804] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:19:47.474205 31411 raft_consensus.cc:697] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Becoming Leader. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:19:47.475064 31411 consensus_queue.cc:237] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:47.479096 31310 raft_consensus.cc:2468] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 2.
I20260709 12:19:47.484634 31059 catalog_manager.cc:5697] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de reported cstate change: term changed from 1 to 2, leader changed from e7f7f401a756482ea721ae7015101f72 (127.29.201.194) to 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193). New cstate: current_term: 2 leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:47.486241 31059 catalog_manager.cc:5697] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de reported cstate change: term changed from 1 to 2, leader changed from e7f7f401a756482ea721ae7015101f72 (127.29.201.194) to 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193). New cstate: current_term: 2 leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:19:47.605315 31160 raft_consensus.cc:3060] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.611212 31160 raft_consensus.cc:1275] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Refusing update from remote peer e7f7f401a756482ea721ae7015101f72: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.612706 31407 consensus_queue.cc:1048] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.620929 31310 raft_consensus.cc:1275] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Refusing update from remote peer e7f7f401a756482ea721ae7015101f72: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.623643 31356 consensus_queue.cc:1048] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.743918 31310 raft_consensus.cc:1275] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Refusing update from remote peer e7f7f401a756482ea721ae7015101f72: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.745203 31356 consensus_queue.cc:1048] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.753026 31160 raft_consensus.cc:1275] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Refusing update from remote peer e7f7f401a756482ea721ae7015101f72: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.755285 31407 consensus_queue.cc:1048] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.787107 31231 raft_consensus.cc:1275] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.788561 31411 consensus_queue.cc:1048] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.796518 31310 raft_consensus.cc:1275] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.798022 31397 consensus_queue.cc:1048] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.814457 31310 raft_consensus.cc:1275] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.815999 31397 consensus_queue.cc:1048] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.823886 31231 raft_consensus.cc:1275] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.826018 31397 consensus_queue.cc:1048] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.850950 31310 raft_consensus.cc:1275] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Refusing update from remote peer e7f7f401a756482ea721ae7015101f72: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.852346 31407 consensus_queue.cc:1048] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.859807 31160 raft_consensus.cc:3060] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:19:47.867416 31160 raft_consensus.cc:1275] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Refusing update from remote peer e7f7f401a756482ea721ae7015101f72: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.868824 31399 consensus_queue.cc:1048] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:19:47.904078 31310 raft_consensus.cc:1275] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.905380 31397 consensus_queue.cc:1048] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.913071 31231 raft_consensus.cc:1275] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.914753 31411 consensus_queue.cc:1048] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.930807 31231 raft_consensus.cc:1275] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.932027 31411 consensus_queue.cc:1048] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:47.940507 31310 raft_consensus.cc:1275] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:19:47.942189 31397 consensus_queue.cc:1048] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:19:49.997123 30503 auto_leader_rebalancer-test.cc:365] The leader distribution is 
I20260709 12:19:49.997855 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:50.016623 31160 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
mode: GRACEFUL
new_leader_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
 from {username='slave'} at 127.0.0.1:58596
I20260709 12:19:50.017144 31160 raft_consensus.cc:606] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Received request to transfer leadership to TS de914d6d5a0046c99ab0fe321d40e8c1
I20260709 12:19:50.019742 31160 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
mode: GRACEFUL
new_leader_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
 from {username='slave'} at 127.0.0.1:58596
I20260709 12:19:50.020318 31160 raft_consensus.cc:606] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Received request to transfer leadership to TS de914d6d5a0046c99ab0fe321d40e8c1
I20260709 12:19:50.021366 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:19:50.022030 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:50.170292 31397 raft_consensus.cc:993] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de: : Instructing follower de914d6d5a0046c99ab0fe321d40e8c1 to start an election
I20260709 12:19:50.170747 31458 raft_consensus.cc:1081] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Signalling peer de914d6d5a0046c99ab0fe321d40e8c1 to start an election
I20260709 12:19:50.172391 31310 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399"
dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
 from {username='slave'} at 127.0.0.1:57490
I20260709 12:19:50.172935 31310 raft_consensus.cc:493] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:50.173244 31310 raft_consensus.cc:3060] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:50.178227 31310 raft_consensus.cc:515] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:50.180050 31310 leader_election.cc:290] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 3 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:19:50.181119 31160 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:50.181119 31231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:50.181978 31231 raft_consensus.cc:3060] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:50.181978 31160 raft_consensus.cc:3055] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:50.182526 31160 raft_consensus.cc:740] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:19:50.183411 31160 consensus_queue.cc:260] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:50.184646 31160 raft_consensus.cc:3060] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:50.189592 31231 raft_consensus.cc:2468] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 3.
I20260709 12:19:50.191061 31278 leader_election.cc:304] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:50.191983 31403 raft_consensus.cc:2804] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:50.192101 31160 raft_consensus.cc:2468] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 3.
I20260709 12:19:50.193567 31403 raft_consensus.cc:697] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 LEADER]: Becoming Leader. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Running, Role: LEADER
I20260709 12:19:50.194478 31403 consensus_queue.cc:237] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:50.202131 31062 catalog_manager.cc:5697] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 reported cstate change: term changed from 2 to 3, leader changed from 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193) to de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195). New cstate: current_term: 3 leader_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:50.277325 31397 raft_consensus.cc:993] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de: : Instructing follower de914d6d5a0046c99ab0fe321d40e8c1 to start an election
I20260709 12:19:50.277812 31458 raft_consensus.cc:1081] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Signalling peer de914d6d5a0046c99ab0fe321d40e8c1 to start an election
I20260709 12:19:50.279327 31310 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f"
dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
 from {username='slave'} at 127.0.0.1:57490
I20260709 12:19:50.279850 31310 raft_consensus.cc:493] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:19:50.280200 31310 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:50.285302 31310 raft_consensus.cc:515] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:50.286947 31310 leader_election.cc:290] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 3 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:19:50.287930 31160 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:19:50.288115 31231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:19:50.288689 31160 raft_consensus.cc:3055] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:19:50.288761 31231 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:50.288995 31160 raft_consensus.cc:740] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:19:50.289786 31160 consensus_queue.cc:260] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:50.290817 31160 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:19:50.294520 31231 raft_consensus.cc:2468] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 3.
I20260709 12:19:50.295727 31278 leader_election.cc:304] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:19:50.296128 31160 raft_consensus.cc:2468] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 3.
I20260709 12:19:50.296518 31403 raft_consensus.cc:2804] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:19:50.297071 31403 raft_consensus.cc:697] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 LEADER]: Becoming Leader. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Running, Role: LEADER
I20260709 12:19:50.297926 31403 consensus_queue.cc:237] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:19:50.305819 31062 catalog_manager.cc:5697] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 reported cstate change: term changed from 2 to 3, leader changed from 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193) to de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195). New cstate: current_term: 3 leader_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: HEALTHY } } }
I20260709 12:19:50.598706 31160 raft_consensus.cc:1275] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Refusing update from remote peer de914d6d5a0046c99ab0fe321d40e8c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:50.599953 31459 consensus_queue.cc:1048] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:50.610394 31231 raft_consensus.cc:1275] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Refusing update from remote peer de914d6d5a0046c99ab0fe321d40e8c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:50.612097 31459 consensus_queue.cc:1048] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:50.735322 31231 raft_consensus.cc:1275] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Refusing update from remote peer de914d6d5a0046c99ab0fe321d40e8c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:50.736964 31468 consensus_queue.cc:1048] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:50.738466 31160 raft_consensus.cc:1275] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Refusing update from remote peer de914d6d5a0046c99ab0fe321d40e8c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:19:50.739989 31468 consensus_queue.cc:1048] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:19:51.023231 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:51.029278 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:51.029824 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:52.031015 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:52.037135 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:52.037657 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:53.038854 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:53.044801 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:53.045316 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:54.046679 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:54.053543 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:54.054113 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:55.055862 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:55.062302 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:55.062837 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:56.064015 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:56.072189 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:56.072602 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:57.073645 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:57.079737 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:57.080289 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:58.081254 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:58.086977 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:58.087448 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:19:59.088562 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:19:59.094523 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:19:59.095031 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:00.129935 30503 auto_leader_rebalancer-test.cc:384] The leader distribution is 
I20260709 12:20:00.141093 31310 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399"
dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
mode: GRACEFUL
new_leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:57506
I20260709 12:20:00.141832 31310 raft_consensus.cc:606] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 LEADER]: Received request to transfer leadership to TS 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:20:00.147980 31310 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f"
dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
mode: GRACEFUL
new_leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:57506
I20260709 12:20:00.148468 31310 raft_consensus.cc:606] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 LEADER]: Received request to transfer leadership to TS 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:20:00.156709 31231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b"
dest_uuid: "e7f7f401a756482ea721ae7015101f72"
mode: GRACEFUL
new_leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:47210
I20260709 12:20:00.157245 31231 raft_consensus.cc:606] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 2 LEADER]: Received request to transfer leadership to TS 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:20:00.162539 31231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "969ee28800d74736a2d0011634d1a014"
dest_uuid: "e7f7f401a756482ea721ae7015101f72"
mode: GRACEFUL
new_leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:47210
I20260709 12:20:00.163098 31231 raft_consensus.cc:606] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Received request to transfer leadership to TS 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:20:00.168500 31231 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8"
dest_uuid: "e7f7f401a756482ea721ae7015101f72"
mode: GRACEFUL
new_leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:47210
I20260709 12:20:00.169075 31231 raft_consensus.cc:606] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 2 LEADER]: Received request to transfer leadership to TS 8f94dd3d7b0948e092e65f629d8af1de
I20260709 12:20:00.170873 31468 raft_consensus.cc:993] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1: : Instructing follower 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:20:00.171391 31468 raft_consensus.cc:1081] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 LEADER]: Signalling peer 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:20:00.173128 31160 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:58578
I20260709 12:20:00.173763 31160 raft_consensus.cc:493] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:00.174206 31160 raft_consensus.cc:3060] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:00.180639 31160 raft_consensus.cc:515] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.182371 31160 leader_election.cc:290] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 4 election: Requested vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:20:00.183523 31310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:20:00.183764 31231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12ddec531d42638c7ef2ec961ca399" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:20:00.184356 31310 raft_consensus.cc:3055] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:00.184361 31231 raft_consensus.cc:3060] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:00.184860 31310 raft_consensus.cc:740] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Running, Role: LEADER
I20260709 12:20:00.185871 31310 consensus_queue.cc:260] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.187297 31310 raft_consensus.cc:3060] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:00.192124 31231 raft_consensus.cc:2468] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 4.
I20260709 12:20:00.193763 31129 leader_election.cc:304] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:20:00.194803 31458 raft_consensus.cc:2804] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:00.194824 31310 raft_consensus.cc:2468] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 4.
I20260709 12:20:00.195327 31458 raft_consensus.cc:697] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 4 LEADER]: Becoming Leader. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:20:00.196322 31458 consensus_queue.cc:237] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.208282 31059 catalog_manager.cc:5697] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de reported cstate change: term changed from 3 to 4, leader changed from de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195) to 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193). New cstate: current_term: 4 leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:00.211994 31487 raft_consensus.cc:993] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72: : Instructing follower 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:20:00.212664 31488 raft_consensus.cc:1081] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 2 LEADER]: Signalling peer 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:20:00.214579 31160 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:58562
I20260709 12:20:00.215200 31160 raft_consensus.cc:493] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:00.215524 31160 raft_consensus.cc:3060] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:00.222484 31160 raft_consensus.cc:515] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.224799 31160 leader_election.cc:290] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 3 election: Requested vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:20:00.225144 31231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:20:00.225292 31310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:20:00.225847 31231 raft_consensus.cc:3055] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:00.225970 31310 raft_consensus.cc:3060] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:00.226213 31231 raft_consensus.cc:740] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:20:00.227013 31231 consensus_queue.cc:260] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.227909 31231 raft_consensus.cc:3060] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:00.230598 31310 raft_consensus.cc:2468] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 3.
I20260709 12:20:00.232038 31130 leader_election.cc:304] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
I20260709 12:20:00.232462 31231 raft_consensus.cc:2468] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 3.
I20260709 12:20:00.232785 31458 raft_consensus.cc:2804] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:00.233217 31458 raft_consensus.cc:697] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 3 LEADER]: Becoming Leader. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:20:00.234066 31458 consensus_queue.cc:237] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.241019 31059 catalog_manager.cc:5697] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de reported cstate change: term changed from 2 to 3, leader changed from e7f7f401a756482ea721ae7015101f72 (127.29.201.194) to 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193). New cstate: current_term: 3 leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:00.262485 31468 raft_consensus.cc:993] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1: : Instructing follower 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:20:00.263147 31468 raft_consensus.cc:1081] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 LEADER]: Signalling peer 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:20:00.264926 31160 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:58578
I20260709 12:20:00.265532 31160 raft_consensus.cc:493] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:00.265887 31160 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:00.270591 31160 raft_consensus.cc:515] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.272229 31160 leader_election.cc:290] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 4 election: Requested vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:20:00.273193 31310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:20:00.273214 31231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:20:00.273819 31310 raft_consensus.cc:3055] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:00.273974 31231 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:00.274114 31310 raft_consensus.cc:740] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Running, Role: LEADER
I20260709 12:20:00.274899 31310 consensus_queue.cc:260] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.275770 31310 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:00.279138 31231 raft_consensus.cc:2468] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 4.
I20260709 12:20:00.280248 31129 leader_election.cc:304] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:20:00.280524 31310 raft_consensus.cc:2468] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 4.
I20260709 12:20:00.281173 31458 raft_consensus.cc:2804] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:00.281677 31458 raft_consensus.cc:697] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 4 LEADER]: Becoming Leader. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:20:00.282537 31458 consensus_queue.cc:237] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.290022 31059 catalog_manager.cc:5697] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de reported cstate change: term changed from 3 to 4, leader changed from de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195) to 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193). New cstate: current_term: 4 leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:00.461540 31487 raft_consensus.cc:993] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72: : Instructing follower 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:20:00.462019 31488 raft_consensus.cc:1081] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Signalling peer 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:20:00.463529 31160 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "969ee28800d74736a2d0011634d1a014"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:58562
I20260709 12:20:00.464049 31160 raft_consensus.cc:493] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:00.464356 31160 raft_consensus.cc:3060] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:00.469097 31160 raft_consensus.cc:515] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.470700 31160 leader_election.cc:290] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 2 election: Requested vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:20:00.471778 31231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "969ee28800d74736a2d0011634d1a014" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:20:00.471685 31310 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "969ee28800d74736a2d0011634d1a014" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:20:00.472410 31231 raft_consensus.cc:3055] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:00.472568 31310 raft_consensus.cc:3060] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:00.472709 31231 raft_consensus.cc:740] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:20:00.473590 31231 consensus_queue.cc:260] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.474529 31231 raft_consensus.cc:3060] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:00.477797 31310 raft_consensus.cc:2468] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 2.
I20260709 12:20:00.478996 31231 raft_consensus.cc:2468] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 2.
I20260709 12:20:00.478984 31130 leader_election.cc:304] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
I20260709 12:20:00.480125 31458 raft_consensus.cc:2804] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:00.480602 31458 raft_consensus.cc:697] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Becoming Leader. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:20:00.481482 31458 consensus_queue.cc:237] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.489259 31059 catalog_manager.cc:5697] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de reported cstate change: term changed from 1 to 2, leader changed from e7f7f401a756482ea721ae7015101f72 (127.29.201.194) to 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193). New cstate: current_term: 2 leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:00.574901 31487 raft_consensus.cc:993] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72: : Instructing follower 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:20:00.575380 31487 raft_consensus.cc:1081] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 2 LEADER]: Signalling peer 8f94dd3d7b0948e092e65f629d8af1de to start an election
I20260709 12:20:00.576844 31160 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
 from {username='slave'} at 127.0.0.1:58562
I20260709 12:20:00.577294 31160 raft_consensus.cc:493] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:00.577543 31160 raft_consensus.cc:3060] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:00.582391 31160 raft_consensus.cc:515] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.584102 31160 leader_election.cc:290] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 3 election: Requested vote from peers e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:20:00.585167 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:20:00.585162 31231 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37527f4161f344a4a4a010a4442419b8" candidate_uuid: "8f94dd3d7b0948e092e65f629d8af1de" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:20:00.585847 31309 raft_consensus.cc:3060] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:00.585917 31231 raft_consensus.cc:3055] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:00.586313 31231 raft_consensus.cc:740] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:20:00.587018 31231 consensus_queue.cc:260] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.588016 31231 raft_consensus.cc:3060] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:00.591337 31309 raft_consensus.cc:2468] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 3.
I20260709 12:20:00.592428 31130 leader_election.cc:304] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f94dd3d7b0948e092e65f629d8af1de, de914d6d5a0046c99ab0fe321d40e8c1; no voters: 
I20260709 12:20:00.593331 31458 raft_consensus.cc:2804] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:00.593636 31231 raft_consensus.cc:2468] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f94dd3d7b0948e092e65f629d8af1de in term 3.
I20260709 12:20:00.593849 31458 raft_consensus.cc:697] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 LEADER]: Becoming Leader. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:20:00.594839 31458 consensus_queue.cc:237] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:00.601733 31059 catalog_manager.cc:5697] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de reported cstate change: term changed from 2 to 3, leader changed from e7f7f401a756482ea721ae7015101f72 (127.29.201.194) to 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193). New cstate: current_term: 3 leader_uuid: "8f94dd3d7b0948e092e65f629d8af1de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:00.614949 31231 raft_consensus.cc:1275] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:00.616183 31458 consensus_queue.cc:1048] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:00.632833 31309 raft_consensus.cc:1275] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:00.634428 31489 consensus_queue.cc:1048] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:00.678050 31309 raft_consensus.cc:1275] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 4 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:00.679471 31489 consensus_queue.cc:1048] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:00.690950 31231 raft_consensus.cc:1275] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 4 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:00.693979 31504 consensus_queue.cc:1048] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:00.784405 31309 raft_consensus.cc:1275] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 4 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:00.785817 31504 consensus_queue.cc:1048] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:00.791710 31231 raft_consensus.cc:1275] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 4 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:00.793246 31489 consensus_queue.cc:1048] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:00.928956 31231 raft_consensus.cc:1275] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:00.930539 31489 consensus_queue.cc:1048] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:00.940066 31309 raft_consensus.cc:1275] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:00.941818 31504 consensus_queue.cc:1048] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:01.097108 31309 raft_consensus.cc:1275] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:01.098486 31489 consensus_queue.cc:1048] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:01.108916 31231 raft_consensus.cc:1275] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Refusing update from remote peer 8f94dd3d7b0948e092e65f629d8af1de: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:01.110397 31504 consensus_queue.cc:1048] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:03.212736 30503 auto_leader_rebalancer-test.cc:399] The leader distribution is 
I20260709 12:20:03.213433 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:03.222023 31160 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
mode: GRACEFUL
new_leader_uuid: "e7f7f401a756482ea721ae7015101f72"
 from {username='slave'} at 127.0.0.1:58596
I20260709 12:20:03.222666 31160 raft_consensus.cc:606] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 4 LEADER]: Received request to transfer leadership to TS e7f7f401a756482ea721ae7015101f72
I20260709 12:20:03.225320 31160 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "969ee28800d74736a2d0011634d1a014"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
mode: GRACEFUL
new_leader_uuid: "e7f7f401a756482ea721ae7015101f72"
 from {username='slave'} at 127.0.0.1:58596
I20260709 12:20:03.225777 31160 raft_consensus.cc:606] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Received request to transfer leadership to TS e7f7f401a756482ea721ae7015101f72
I20260709 12:20:03.228405 31160 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
mode: GRACEFUL
new_leader_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
 from {username='slave'} at 127.0.0.1:58596
I20260709 12:20:03.228928 31160 raft_consensus.cc:606] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 3 LEADER]: Received request to transfer leadership to TS de914d6d5a0046c99ab0fe321d40e8c1
I20260709 12:20:03.231462 31160 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "dda71ddf86394530aac956f3cc709af1"
dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
mode: GRACEFUL
new_leader_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
 from {username='slave'} at 127.0.0.1:58596
I20260709 12:20:03.231932 31160 raft_consensus.cc:606] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Received request to transfer leadership to TS de914d6d5a0046c99ab0fe321d40e8c1
I20260709 12:20:03.233018 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260709 12:20:03.233489 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:03.260571 31515 raft_consensus.cc:993] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de: : Instructing follower e7f7f401a756482ea721ae7015101f72 to start an election
I20260709 12:20:03.261112 31504 raft_consensus.cc:1081] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Signalling peer e7f7f401a756482ea721ae7015101f72 to start an election
I20260709 12:20:03.262883 31232 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "969ee28800d74736a2d0011634d1a014"
dest_uuid: "e7f7f401a756482ea721ae7015101f72"
 from {username='slave'} at 127.0.0.1:47202
I20260709 12:20:03.263444 31232 raft_consensus.cc:493] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:03.263765 31232 raft_consensus.cc:3060] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.269471 31232 raft_consensus.cc:515] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:03.271955 31232 leader_election.cc:290] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 3 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:20:03.272975 31160 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "969ee28800d74736a2d0011634d1a014" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:20:03.273151 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "969ee28800d74736a2d0011634d1a014" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:20:03.273674 31309 raft_consensus.cc:3060] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.273674 31160 raft_consensus.cc:3055] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:03.274235 31160 raft_consensus.cc:740] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:20:03.275022 31160 consensus_queue.cc:260] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:03.276263 31160 raft_consensus.cc:3060] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.281041 31309 raft_consensus.cc:2468] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 3.
I20260709 12:20:03.282305 31205 leader_election.cc:304] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:20:03.283123 31487 raft_consensus.cc:2804] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:03.283443 31160 raft_consensus.cc:2468] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 3.
I20260709 12:20:03.284350 31487 raft_consensus.cc:697] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 3 LEADER]: Becoming Leader. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:20:03.285418 31487 consensus_queue.cc:237] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:03.294158 31062 catalog_manager.cc:5697] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 reported cstate change: term changed from 2 to 3, leader changed from 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193) to e7f7f401a756482ea721ae7015101f72 (127.29.201.194). New cstate: current_term: 3 leader_uuid: "e7f7f401a756482ea721ae7015101f72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:03.322950 31504 raft_consensus.cc:993] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de: : Instructing follower e7f7f401a756482ea721ae7015101f72 to start an election
I20260709 12:20:03.323392 31515 raft_consensus.cc:1081] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 4 LEADER]: Signalling peer e7f7f401a756482ea721ae7015101f72 to start an election
I20260709 12:20:03.324829 31232 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f"
dest_uuid: "e7f7f401a756482ea721ae7015101f72"
 from {username='slave'} at 127.0.0.1:47202
I20260709 12:20:03.325363 31232 raft_consensus.cc:493] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:03.325630 31232 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:20:03.330461 31232 raft_consensus.cc:515] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:03.332165 31232 leader_election.cc:290] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 5 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195:35629)
I20260709 12:20:03.333025 31160 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:20:03.333658 31160 raft_consensus.cc:3055] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:20:03.333961 31160 raft_consensus.cc:740] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:20:03.334535 31309 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "342a72e01a6c46fc9861bab2d1f4264f" candidate_uuid: "e7f7f401a756482ea721ae7015101f72" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
I20260709 12:20:03.335084 31309 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:20:03.334721 31160 consensus_queue.cc:260] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:03.335889 31160 raft_consensus.cc:3060] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:20:03.341569 31309 raft_consensus.cc:2468] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 5.
I20260709 12:20:03.341982 31160 raft_consensus.cc:2468] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7f7f401a756482ea721ae7015101f72 in term 5.
I20260709 12:20:03.342948 31205 leader_election.cc:304] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:20:03.343611 31487 raft_consensus.cc:2804] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:20:03.344017 31487 raft_consensus.cc:697] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 5 LEADER]: Becoming Leader. State: Replica: e7f7f401a756482ea721ae7015101f72, State: Running, Role: LEADER
I20260709 12:20:03.344771 31487 consensus_queue.cc:237] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:03.351915 31062 catalog_manager.cc:5697] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 reported cstate change: term changed from 4 to 5, leader changed from 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193) to e7f7f401a756482ea721ae7015101f72 (127.29.201.194). New cstate: current_term: 5 leader_uuid: "e7f7f401a756482ea721ae7015101f72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:03.361156 31504 raft_consensus.cc:993] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de: : Instructing follower de914d6d5a0046c99ab0fe321d40e8c1 to start an election
I20260709 12:20:03.361663 31515 raft_consensus.cc:1081] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Signalling peer de914d6d5a0046c99ab0fe321d40e8c1 to start an election
I20260709 12:20:03.363209 31309 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "dda71ddf86394530aac956f3cc709af1"
dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
 from {username='slave'} at 127.0.0.1:57490
I20260709 12:20:03.363715 31309 raft_consensus.cc:493] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:03.364013 31309 raft_consensus.cc:3060] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.368534 31309 raft_consensus.cc:515] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:03.371027 31309 leader_election.cc:290] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 3 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:20:03.371887 31160 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dda71ddf86394530aac956f3cc709af1" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:20:03.372185 31232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dda71ddf86394530aac956f3cc709af1" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:20:03.372659 31160 raft_consensus.cc:3055] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:03.372769 31232 raft_consensus.cc:3060] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.373189 31160 raft_consensus.cc:740] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:20:03.374262 31160 consensus_queue.cc:260] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:03.375942 31160 raft_consensus.cc:3060] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:03.378737 31232 raft_consensus.cc:2468] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 3.
I20260709 12:20:03.380195 31278 leader_election.cc:304] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:20:03.381132 31468 raft_consensus.cc:2804] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:03.381707 31160 raft_consensus.cc:2468] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 3.
I20260709 12:20:03.382555 31468 raft_consensus.cc:697] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 LEADER]: Becoming Leader. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Running, Role: LEADER
I20260709 12:20:03.383590 31468 consensus_queue.cc:237] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:03.391806 31062 catalog_manager.cc:5697] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 reported cstate change: term changed from 2 to 3, leader changed from 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193) to de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195). New cstate: current_term: 3 leader_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:03.437721 31504 raft_consensus.cc:993] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de: : Instructing follower de914d6d5a0046c99ab0fe321d40e8c1 to start an election
I20260709 12:20:03.438320 31515 raft_consensus.cc:1081] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 3 LEADER]: Signalling peer de914d6d5a0046c99ab0fe321d40e8c1 to start an election
I20260709 12:20:03.440191 31309 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b"
dest_uuid: "de914d6d5a0046c99ab0fe321d40e8c1"
 from {username='slave'} at 127.0.0.1:57490
I20260709 12:20:03.440749 31309 raft_consensus.cc:493] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:03.441047 31309 raft_consensus.cc:3060] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:03.445489 31309 raft_consensus.cc:515] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:03.447114 31309 leader_election.cc:290] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 4 election: Requested vote from peers 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183), e7f7f401a756482ea721ae7015101f72 (127.29.201.194:39933)
I20260709 12:20:03.448056 31160 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "8f94dd3d7b0948e092e65f629d8af1de"
I20260709 12:20:03.448105 31232 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d251201d5ba54aa8b4f8e536dbc3c39b" candidate_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e7f7f401a756482ea721ae7015101f72"
I20260709 12:20:03.448729 31160 raft_consensus.cc:3055] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:20:03.448774 31232 raft_consensus.cc:3060] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:03.449007 31160 raft_consensus.cc:740] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 8f94dd3d7b0948e092e65f629d8af1de, State: Running, Role: LEADER
I20260709 12:20:03.449806 31160 consensus_queue.cc:260] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:03.451189 31160 raft_consensus.cc:3060] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:03.454700 31232 raft_consensus.cc:2468] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 4.
I20260709 12:20:03.455881 31160 raft_consensus.cc:2468] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate de914d6d5a0046c99ab0fe321d40e8c1 in term 4.
I20260709 12:20:03.455881 31278 leader_election.cc:304] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: de914d6d5a0046c99ab0fe321d40e8c1, e7f7f401a756482ea721ae7015101f72; no voters: 
I20260709 12:20:03.456786 31468 raft_consensus.cc:2804] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:20:03.457218 31468 raft_consensus.cc:697] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 4 LEADER]: Becoming Leader. State: Replica: de914d6d5a0046c99ab0fe321d40e8c1, State: Running, Role: LEADER
I20260709 12:20:03.458029 31468 consensus_queue.cc:237] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } }
I20260709 12:20:03.465368 31062 catalog_manager.cc:5697] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 reported cstate change: term changed from 3 to 4, leader changed from 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193) to de914d6d5a0046c99ab0fe321d40e8c1 (127.29.201.195). New cstate: current_term: 4 leader_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:03.723419 31160 raft_consensus.cc:1275] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Refusing update from remote peer e7f7f401a756482ea721ae7015101f72: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:03.725098 31487 consensus_queue.cc:1048] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:03.748700 31309 raft_consensus.cc:1275] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Refusing update from remote peer e7f7f401a756482ea721ae7015101f72: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:03.750296 31487 consensus_queue.cc:1048] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:03.839982 31164 raft_consensus.cc:1275] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 5 FOLLOWER]: Refusing update from remote peer e7f7f401a756482ea721ae7015101f72: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:20:03.841403 31487 consensus_queue.cc:1048] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:03.852743 31309 raft_consensus.cc:1275] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 5 FOLLOWER]: Refusing update from remote peer e7f7f401a756482ea721ae7015101f72: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:20:03.854588 31517 consensus_queue.cc:1048] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de914d6d5a0046c99ab0fe321d40e8c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:03.899796 31164 raft_consensus.cc:1275] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Refusing update from remote peer de914d6d5a0046c99ab0fe321d40e8c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:03.901243 31468 consensus_queue.cc:1048] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:03.909471 31232 raft_consensus.cc:1275] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Refusing update from remote peer de914d6d5a0046c99ab0fe321d40e8c1: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:03.911556 31468 consensus_queue.cc:1048] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:03.940953 31164 raft_consensus.cc:1275] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 4 FOLLOWER]: Refusing update from remote peer de914d6d5a0046c99ab0fe321d40e8c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:03.942368 31468 consensus_queue.cc:1048] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f94dd3d7b0948e092e65f629d8af1de" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 44183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:03.952296 31232 raft_consensus.cc:1275] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 4 FOLLOWER]: Refusing update from remote peer de914d6d5a0046c99ab0fe321d40e8c1: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:20:03.953874 31468 consensus_queue.cc:1048] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7f7f401a756482ea721ae7015101f72" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:04.234740 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:04.240748 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:04.241403 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:05.242530 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:05.247984 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:05.248498 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:06.249560 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:06.255435 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:06.255926 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:07.256868 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:07.262405 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:07.262907 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:08.263926 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:08.269438 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:08.269970 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:09.270931 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:09.277271 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:09.277746 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:10.278851 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:10.284559 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:10.285156 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:11.286181 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:11.292097 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:11.292582 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:12.293623 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:12.299022 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:20:12.299517 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:13.333519 30503 auto_leader_rebalancer-test.cc:410] The leader distribution is 
I20260709 12:20:13.334188 30503 tablet_server.cc:179] TabletServer@127.29.201.193:0 shutting down...
W20260709 12:20:13.353368 31205 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 2 similar messages]
W20260709 12:20:13.358537 31205 consensus_peers.cc:597] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 -> Peer 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183): Couldn't send request to peer 8f94dd3d7b0948e092e65f629d8af1de. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:13.368178 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:13.368945 30503 tablet_replica.cc:333] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de: stopping tablet replica
I20260709 12:20:13.369542 30503 raft_consensus.cc:2243] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.370326 30503 raft_consensus.cc:2272] T 37527f4161f344a4a4a010a4442419b8 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.372480 30503 tablet_replica.cc:333] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de: stopping tablet replica
I20260709 12:20:13.372977 30503 raft_consensus.cc:2243] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.373792 30503 raft_consensus.cc:2272] T 48cb82524513465785d32448b353ff4a P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.375829 30503 tablet_replica.cc:333] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de: stopping tablet replica
I20260709 12:20:13.376502 30503 raft_consensus.cc:2243] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.376974 30503 raft_consensus.cc:2272] T 342a72e01a6c46fc9861bab2d1f4264f P 8f94dd3d7b0948e092e65f629d8af1de [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.378978 30503 tablet_replica.cc:333] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de: stopping tablet replica
I20260709 12:20:13.379563 30503 raft_consensus.cc:2243] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.380376 30503 raft_consensus.cc:2272] T 8f12ddec531d42638c7ef2ec961ca399 P 8f94dd3d7b0948e092e65f629d8af1de [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.382280 30503 tablet_replica.cc:333] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de: stopping tablet replica
I20260709 12:20:13.382903 30503 raft_consensus.cc:2243] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.383319 30503 raft_consensus.cc:2272] T cff8c0586c59461bb83a89ee55000d70 P 8f94dd3d7b0948e092e65f629d8af1de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.385177 30503 tablet_replica.cc:333] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de: stopping tablet replica
I20260709 12:20:13.385757 30503 raft_consensus.cc:2243] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.386221 30503 raft_consensus.cc:2272] T d251201d5ba54aa8b4f8e536dbc3c39b P 8f94dd3d7b0948e092e65f629d8af1de [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.387946 30503 tablet_replica.cc:333] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de: stopping tablet replica
I20260709 12:20:13.388442 30503 raft_consensus.cc:2243] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.388955 30503 raft_consensus.cc:2272] T 0f138178dea44e59804eaa33c221d4aa P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.391059 30503 tablet_replica.cc:333] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de: stopping tablet replica
I20260709 12:20:13.391558 30503 raft_consensus.cc:2243] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.391975 30503 raft_consensus.cc:2272] T dda71ddf86394530aac956f3cc709af1 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.393708 30503 tablet_replica.cc:333] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de: stopping tablet replica
I20260709 12:20:13.394155 30503 raft_consensus.cc:2243] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.394981 30503 raft_consensus.cc:2272] T e9cd04eb7d0b4e7fbac08a32889020e2 P 8f94dd3d7b0948e092e65f629d8af1de [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.396796 30503 tablet_replica.cc:333] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de: stopping tablet replica
I20260709 12:20:13.397277 30503 raft_consensus.cc:2243] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.397703 30503 raft_consensus.cc:2272] T 969ee28800d74736a2d0011634d1a014 P 8f94dd3d7b0948e092e65f629d8af1de [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.429566 30503 tablet_server.cc:196] TabletServer@127.29.201.193:0 shutdown complete.
I20260709 12:20:13.451330 30503 tablet_server.cc:179] TabletServer@127.29.201.194:0 shutting down...
I20260709 12:20:13.483042 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:13.484169 30503 tablet_replica.cc:333] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72: stopping tablet replica
I20260709 12:20:13.485069 30503 raft_consensus.cc:2243] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.485904 30503 raft_consensus.cc:2272] T 37527f4161f344a4a4a010a4442419b8 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:20:13.487205 31279 consensus_peers.cc:597] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 -> Peer 8f94dd3d7b0948e092e65f629d8af1de (127.29.201.193:44183): Couldn't send request to peer 8f94dd3d7b0948e092e65f629d8af1de. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:44183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:13.489182 30503 tablet_replica.cc:333] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72: stopping tablet replica
I20260709 12:20:13.489899 30503 raft_consensus.cc:2243] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.490394 30503 raft_consensus.cc:2272] T e9cd04eb7d0b4e7fbac08a32889020e2 P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.492604 30503 tablet_replica.cc:333] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72: stopping tablet replica
I20260709 12:20:13.493151 30503 raft_consensus.cc:2243] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.493925 30503 raft_consensus.cc:2272] T 969ee28800d74736a2d0011634d1a014 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.495987 30503 tablet_replica.cc:333] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72: stopping tablet replica
I20260709 12:20:13.496549 30503 raft_consensus.cc:2243] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.496969 30503 raft_consensus.cc:2272] T dda71ddf86394530aac956f3cc709af1 P e7f7f401a756482ea721ae7015101f72 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.498852 30503 tablet_replica.cc:333] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72: stopping tablet replica
I20260709 12:20:13.499380 30503 raft_consensus.cc:2243] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.500219 30503 raft_consensus.cc:2272] T 0f138178dea44e59804eaa33c221d4aa P e7f7f401a756482ea721ae7015101f72 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.502303 30503 tablet_replica.cc:333] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72: stopping tablet replica
I20260709 12:20:13.502918 30503 raft_consensus.cc:2243] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.503377 30503 raft_consensus.cc:2272] T d251201d5ba54aa8b4f8e536dbc3c39b P e7f7f401a756482ea721ae7015101f72 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.505183 30503 tablet_replica.cc:333] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72: stopping tablet replica
I20260709 12:20:13.505703 30503 raft_consensus.cc:2243] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.506268 30503 raft_consensus.cc:2272] T 48cb82524513465785d32448b353ff4a P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.508170 30503 tablet_replica.cc:333] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72: stopping tablet replica
I20260709 12:20:13.508678 30503 raft_consensus.cc:2243] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.509467 30503 raft_consensus.cc:2272] T 342a72e01a6c46fc9861bab2d1f4264f P e7f7f401a756482ea721ae7015101f72 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.511449 30503 tablet_replica.cc:333] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72: stopping tablet replica
I20260709 12:20:13.511989 30503 raft_consensus.cc:2243] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.512461 30503 raft_consensus.cc:2272] T cff8c0586c59461bb83a89ee55000d70 P e7f7f401a756482ea721ae7015101f72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.514385 30503 tablet_replica.cc:333] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72: stopping tablet replica
I20260709 12:20:13.514930 30503 raft_consensus.cc:2243] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.515401 30503 raft_consensus.cc:2272] T 8f12ddec531d42638c7ef2ec961ca399 P e7f7f401a756482ea721ae7015101f72 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.544540 30503 tablet_server.cc:196] TabletServer@127.29.201.194:0 shutdown complete.
I20260709 12:20:13.565371 30503 tablet_server.cc:179] TabletServer@127.29.201.195:0 shutting down...
I20260709 12:20:13.594257 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:20:13.595127 30503 tablet_replica.cc:333] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1: stopping tablet replica
I20260709 12:20:13.595824 30503 raft_consensus.cc:2243] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.596432 30503 raft_consensus.cc:2272] T 8f12ddec531d42638c7ef2ec961ca399 P de914d6d5a0046c99ab0fe321d40e8c1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.598862 30503 tablet_replica.cc:333] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1: stopping tablet replica
I20260709 12:20:13.599514 30503 raft_consensus.cc:2243] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.600471 30503 raft_consensus.cc:2272] T cff8c0586c59461bb83a89ee55000d70 P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.602739 30503 tablet_replica.cc:333] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1: stopping tablet replica
I20260709 12:20:13.603358 30503 raft_consensus.cc:2243] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.603832 30503 raft_consensus.cc:2272] T 48cb82524513465785d32448b353ff4a P de914d6d5a0046c99ab0fe321d40e8c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.605989 30503 tablet_replica.cc:333] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1: stopping tablet replica
I20260709 12:20:13.606884 30503 raft_consensus.cc:2243] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.607402 30503 raft_consensus.cc:2272] T 342a72e01a6c46fc9861bab2d1f4264f P de914d6d5a0046c99ab0fe321d40e8c1 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.609346 30503 tablet_replica.cc:333] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1: stopping tablet replica
I20260709 12:20:13.610071 30503 raft_consensus.cc:2243] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.611024 30503 raft_consensus.cc:2272] T d251201d5ba54aa8b4f8e536dbc3c39b P de914d6d5a0046c99ab0fe321d40e8c1 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.613109 30503 tablet_replica.cc:333] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1: stopping tablet replica
I20260709 12:20:13.613742 30503 raft_consensus.cc:2243] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.614264 30503 raft_consensus.cc:2272] T 0f138178dea44e59804eaa33c221d4aa P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.616315 30503 tablet_replica.cc:333] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1: stopping tablet replica
I20260709 12:20:13.616917 30503 raft_consensus.cc:2243] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.617897 30503 raft_consensus.cc:2272] T dda71ddf86394530aac956f3cc709af1 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.620038 30503 tablet_replica.cc:333] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1: stopping tablet replica
I20260709 12:20:13.620610 30503 raft_consensus.cc:2243] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.621095 30503 raft_consensus.cc:2272] T e9cd04eb7d0b4e7fbac08a32889020e2 P de914d6d5a0046c99ab0fe321d40e8c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.623090 30503 tablet_replica.cc:333] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1: stopping tablet replica
I20260709 12:20:13.623634 30503 raft_consensus.cc:2243] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.624076 30503 raft_consensus.cc:2272] T 969ee28800d74736a2d0011634d1a014 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.625885 30503 tablet_replica.cc:333] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1: stopping tablet replica
I20260709 12:20:13.626437 30503 raft_consensus.cc:2243] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:13.626940 30503 raft_consensus.cc:2272] T 37527f4161f344a4a4a010a4442419b8 P de914d6d5a0046c99ab0fe321d40e8c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.656446 30503 tablet_server.cc:196] TabletServer@127.29.201.195:0 shutdown complete.
I20260709 12:20:13.676983 30503 master.cc:562] Master@127.29.201.254:41269 shutting down...
I20260709 12:20:13.703987 30503 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:20:13.704612 30503 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:13.704977 30503 tablet_replica.cc:333] T 00000000000000000000000000000000 P 5a0169a5d1354fdab991b1801c78dd59: stopping tablet replica
I20260709 12:20:13.728442 30503 master.cc:584] Master@127.29.201.254:41269 shutdown complete.
8f94dd3d7b0948e092e65f629d8af1de  5
de914d6d5a0046c99ab0fe321d40e8c1  1
e7f7f401a756482ea721ae7015101f72  4
8f94dd3d7b0948e092e65f629d8af1de  3
de914d6d5a0046c99ab0fe321d40e8c1  3
e7f7f401a756482ea721ae7015101f72  4
8f94dd3d7b0948e092e65f629d8af1de  8
de914d6d5a0046c99ab0fe321d40e8c1  1
e7f7f401a756482ea721ae7015101f72  1
8f94dd3d7b0948e092e65f629d8af1de  4
de914d6d5a0046c99ab0fe321d40e8c1  3
e7f7f401a756482ea721ae7015101f72  3
[       OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (28821 ms)
[ RUN      ] LeaderRebalancerTest.AddTserver
I20260709 12:20:13.778107 30503 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.201.254:37495
I20260709 12:20:13.779419 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:13.785926 31551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:13.786875 31552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:13.789633 30503 server_base.cc:1061] running on GCE node
W20260709 12:20:13.790023 31554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:13.791036 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:13.791289 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:13.791543 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599613791514 us; error 0 us; skew 500 ppm
I20260709 12:20:13.792133 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:13.795183 30503 webserver.cc:533] Webserver started at http://127.29.201.254:35507/ using document root <none> and password file <none>
I20260709 12:20:13.795768 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:13.795996 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:13.796303 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:13.797724 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/master-0-root/instance:
uuid: "ac4fa664fb214bda87cd96abb2116a4b"
format_stamp: "Formatted at 2026-07-09 12:20:13 on dist-test-slave-qgf5"
I20260709 12:20:13.802968 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.002s
I20260709 12:20:13.806526 31559 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:13.807397 30503 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:20:13.807710 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/master-0-root
uuid: "ac4fa664fb214bda87cd96abb2116a4b"
format_stamp: "Formatted at 2026-07-09 12:20:13 on dist-test-slave-qgf5"
I20260709 12:20:13.808024 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:13.834018 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:13.835457 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:13.876727 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.254:37495
I20260709 12:20:13.876833 31610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.254:37495 every 8 connection(s)
I20260709 12:20:13.881315 31611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:13.892987 31611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Bootstrap starting.
I20260709 12:20:13.897953 31611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:13.902403 31611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: No bootstrap required, opened a new log
I20260709 12:20:13.904716 31611 raft_consensus.cc:359] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4fa664fb214bda87cd96abb2116a4b" member_type: VOTER }
I20260709 12:20:13.905223 31611 raft_consensus.cc:385] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:13.905503 31611 raft_consensus.cc:740] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac4fa664fb214bda87cd96abb2116a4b, State: Initialized, Role: FOLLOWER
I20260709 12:20:13.906222 31611 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4fa664fb214bda87cd96abb2116a4b" member_type: VOTER }
I20260709 12:20:13.906764 31611 raft_consensus.cc:399] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:20:13.907032 31611 raft_consensus.cc:493] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:20:13.907297 31611 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:13.912142 31611 raft_consensus.cc:515] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4fa664fb214bda87cd96abb2116a4b" member_type: VOTER }
I20260709 12:20:13.912724 31611 leader_election.cc:304] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac4fa664fb214bda87cd96abb2116a4b; no voters: 
I20260709 12:20:13.913980 31611 leader_election.cc:290] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:20:13.914471 31614 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:13.916083 31614 raft_consensus.cc:697] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [term 1 LEADER]: Becoming Leader. State: Replica: ac4fa664fb214bda87cd96abb2116a4b, State: Running, Role: LEADER
I20260709 12:20:13.916975 31611 sys_catalog.cc:565] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:20:13.916860 31614 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4fa664fb214bda87cd96abb2116a4b" member_type: VOTER }
I20260709 12:20:13.924029 31616 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac4fa664fb214bda87cd96abb2116a4b. Latest consensus state: current_term: 1 leader_uuid: "ac4fa664fb214bda87cd96abb2116a4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4fa664fb214bda87cd96abb2116a4b" member_type: VOTER } }
I20260709 12:20:13.923942 31615 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac4fa664fb214bda87cd96abb2116a4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac4fa664fb214bda87cd96abb2116a4b" member_type: VOTER } }
I20260709 12:20:13.924751 31616 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:13.924860 31615 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [sys.catalog]: This master's current role is: LEADER
I20260709 12:20:13.927132 31623 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:20:13.933154 31623 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:20:13.934232 30503 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:20:13.943686 31623 catalog_manager.cc:1383] Generated new cluster ID: a07a12b8c67545918fda7a3c5aa3c7f5
I20260709 12:20:13.944023 31623 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:20:13.968580 31623 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:20:13.970080 31623 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:20:13.983952 31623 catalog_manager.cc:6070] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Generated new TSK 0
I20260709 12:20:13.984611 31623 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:20:14.001024 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:14.008112 31632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:14.008765 31633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:14.011307 31635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:14.011785 30503 server_base.cc:1061] running on GCE node
I20260709 12:20:14.012830 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:14.013022 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:14.013228 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599614013207 us; error 0 us; skew 500 ppm
I20260709 12:20:14.013793 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:14.016342 30503 webserver.cc:533] Webserver started at http://127.29.201.193:45053/ using document root <none> and password file <none>
I20260709 12:20:14.016832 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:14.017066 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:14.017366 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:14.018746 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/instance:
uuid: "b95aeeaf4c51442b8092cff1de674f2b"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-qgf5"
I20260709 12:20:14.023394 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.002s
I20260709 12:20:14.027037 31640 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.028051 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.001s
I20260709 12:20:14.028362 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root
uuid: "b95aeeaf4c51442b8092cff1de674f2b"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-qgf5"
I20260709 12:20:14.028743 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:14.044503 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:14.045878 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:14.047555 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:14.050001 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:14.050235 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.050510 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:14.050725 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.001s
I20260709 12:20:14.098109 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.193:41935
I20260709 12:20:14.098220 31702 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.193:41935 every 8 connection(s)
I20260709 12:20:14.103582 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:14.111430 31707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:14.115010 31708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:14.117709 30503 server_base.cc:1061] running on GCE node
W20260709 12:20:14.118757 31710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:14.119850 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:14.120105 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:14.120311 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599614120289 us; error 0 us; skew 500 ppm
I20260709 12:20:14.121127 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:14.123549 31703 heartbeater.cc:344] Connected to a master server at 127.29.201.254:37495
I20260709 12:20:14.123975 31703 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:14.124433 30503 webserver.cc:533] Webserver started at http://127.29.201.194:38759/ using document root <none> and password file <none>
I20260709 12:20:14.124805 31703 heartbeater.cc:507] Master 127.29.201.254:37495 requested a full tablet report, sending...
I20260709 12:20:14.124989 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:14.125376 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:14.125670 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:14.127199 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/instance:
uuid: "039d66a6c85e43c3936bf5017501b344"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-qgf5"
I20260709 12:20:14.127578 31576 ts_manager.cc:194] Registered new tserver with Master: b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:14.129534 31576 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42070
I20260709 12:20:14.132843 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:20:14.136727 31715 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.137792 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:20:14.138091 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root
uuid: "039d66a6c85e43c3936bf5017501b344"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-qgf5"
I20260709 12:20:14.138370 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:14.157416 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:14.158852 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:14.160476 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:14.163168 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:14.163430 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.163750 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:14.163933 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.209035 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.194:46639
I20260709 12:20:14.209161 31777 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.194:46639 every 8 connection(s)
I20260709 12:20:14.214994 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:14.222167 31781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:14.224231 31782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:14.228942 31778 heartbeater.cc:344] Connected to a master server at 127.29.201.254:37495
I20260709 12:20:14.229470 31778 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:14.230197 30503 server_base.cc:1061] running on GCE node
I20260709 12:20:14.230860 31778 heartbeater.cc:507] Master 127.29.201.254:37495 requested a full tablet report, sending...
W20260709 12:20:14.231009 31784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:14.232226 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:14.232501 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:14.232702 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599614232685 us; error 0 us; skew 500 ppm
I20260709 12:20:14.233476 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:14.233588 31576 ts_manager.cc:194] Registered new tserver with Master: 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:14.235770 31576 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42072
I20260709 12:20:14.236680 30503 webserver.cc:533] Webserver started at http://127.29.201.195:43751/ using document root <none> and password file <none>
I20260709 12:20:14.237314 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:14.237586 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:14.237964 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:14.239487 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root/instance:
uuid: "4d48c83c761f48079bc6ea176649dda0"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-qgf5"
I20260709 12:20:14.244843 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:20:14.248780 31789 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.249704 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:20:14.250021 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root
uuid: "4d48c83c761f48079bc6ea176649dda0"
format_stamp: "Formatted at 2026-07-09 12:20:14 on dist-test-slave-qgf5"
I20260709 12:20:14.250308 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:14.267784 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:14.269063 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:14.270720 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:14.273274 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:14.273516 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.273813 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:14.273993 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:14.319588 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.195:44645
I20260709 12:20:14.319633 31851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.195:44645 every 8 connection(s)
I20260709 12:20:14.335697 31852 heartbeater.cc:344] Connected to a master server at 127.29.201.254:37495
I20260709 12:20:14.336120 31852 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:14.337019 31852 heartbeater.cc:507] Master 127.29.201.254:37495 requested a full tablet report, sending...
I20260709 12:20:14.339421 31576 ts_manager.cc:194] Registered new tserver with Master: 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:14.340161 30503 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.017342883s
I20260709 12:20:14.341416 31576 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42080
I20260709 12:20:14.369962 31576 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42088:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:20:14.376979 31576 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:20:14.587742 31817 tablet_service.cc:1511] Processing CreateTablet for tablet f84a67c6dba04300a5240cc80c94718a (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:20:14.589567 31817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84a67c6dba04300a5240cc80c94718a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.589293 31814 tablet_service.cc:1511] Processing CreateTablet for tablet 1e27b968c91a4906aec537421f481269 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:20:14.590973 31814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e27b968c91a4906aec537421f481269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.592346 31815 tablet_service.cc:1511] Processing CreateTablet for tablet b043bbe90410471da70e6b00856b5c8d (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:20:14.593896 31815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b043bbe90410471da70e6b00856b5c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.590135 31816 tablet_service.cc:1511] Processing CreateTablet for tablet 7523cff6f9fb44b5ba8fc04726d85b65 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:20:14.595053 31816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7523cff6f9fb44b5ba8fc04726d85b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.596168 31813 tablet_service.cc:1511] Processing CreateTablet for tablet f933d4319cf444afb9f67bd265576a31 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:20:14.597735 31813 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f933d4319cf444afb9f67bd265576a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.602310 31812 tablet_service.cc:1511] Processing CreateTablet for tablet 945fd223ac334bce88917d1b9b4c72e1 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:20:14.603988 31812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 945fd223ac334bce88917d1b9b4c72e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.616950 31811 tablet_service.cc:1511] Processing CreateTablet for tablet 5d0d0402170d41b79dc30f93bee98766 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:20:14.618736 31811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0d0402170d41b79dc30f93bee98766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.623627 31816 tablet_service.cc:1511] Processing CreateTablet for tablet 70ada7cfaa154c8c9cc927936b46c30a (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:20:14.625162 31816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ada7cfaa154c8c9cc927936b46c30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.628015 31666 tablet_service.cc:1511] Processing CreateTablet for tablet b043bbe90410471da70e6b00856b5c8d (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:20:14.630940 31668 tablet_service.cc:1511] Processing CreateTablet for tablet f84a67c6dba04300a5240cc80c94718a (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:20:14.631227 31665 tablet_service.cc:1511] Processing CreateTablet for tablet 1e27b968c91a4906aec537421f481269 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:20:14.633898 31664 tablet_service.cc:1511] Processing CreateTablet for tablet f933d4319cf444afb9f67bd265576a31 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:20:14.636619 31663 tablet_service.cc:1511] Processing CreateTablet for tablet 945fd223ac334bce88917d1b9b4c72e1 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:20:14.639248 31662 tablet_service.cc:1511] Processing CreateTablet for tablet 5d0d0402170d41b79dc30f93bee98766 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:20:14.641829 31661 tablet_service.cc:1511] Processing CreateTablet for tablet 70ada7cfaa154c8c9cc927936b46c30a (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:20:14.643114 31815 tablet_service.cc:1511] Processing CreateTablet for tablet 5bb2dc5276714f75b6e3e61ca9275f84 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:20:14.644474 31815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bb2dc5276714f75b6e3e61ca9275f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.629657 31667 tablet_service.cc:1511] Processing CreateTablet for tablet 7523cff6f9fb44b5ba8fc04726d85b65 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:20:14.644840 31660 tablet_service.cc:1511] Processing CreateTablet for tablet 5bb2dc5276714f75b6e3e61ca9275f84 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:20:14.647140 31659 tablet_service.cc:1511] Processing CreateTablet for tablet 3a0925b3d6e9448684fd239b27453b69 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:20:14.629650 31666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b043bbe90410471da70e6b00856b5c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.649416 31663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 945fd223ac334bce88917d1b9b4c72e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.650161 31659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a0925b3d6e9448684fd239b27453b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.650871 31668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84a67c6dba04300a5240cc80c94718a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.655999 31816 tablet_service.cc:1511] Processing CreateTablet for tablet f2e7a775ae0d44108be0965c85245ecf (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:20:14.657106 31667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7523cff6f9fb44b5ba8fc04726d85b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.657399 31816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2e7a775ae0d44108be0965c85245ecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.662843 31810 tablet_service.cc:1511] Processing CreateTablet for tablet a5dfbfce13484fc0b1feba45daaf8247 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:20:14.664464 31810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5dfbfce13484fc0b1feba45daaf8247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.667141 31665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e27b968c91a4906aec537421f481269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.669063 31809 tablet_service.cc:1511] Processing CreateTablet for tablet 553d0b2dddba47779b0219c71ed9ce83 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:20:14.674574 31808 tablet_service.cc:1511] Processing CreateTablet for tablet 6499e78dd87c4e05be32f5681957a6b0 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:20:14.679593 31664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f933d4319cf444afb9f67bd265576a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.679703 31815 tablet_service.cc:1511] Processing CreateTablet for tablet 49cc564ccd0d411fa79831efa24f87bb (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:20:14.683108 31662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0d0402170d41b79dc30f93bee98766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.687142 31660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bb2dc5276714f75b6e3e61ca9275f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.687382 31743 tablet_service.cc:1511] Processing CreateTablet for tablet f84a67c6dba04300a5240cc80c94718a (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:20:14.689039 31743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84a67c6dba04300a5240cc80c94718a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.688678 31740 tablet_service.cc:1511] Processing CreateTablet for tablet 1e27b968c91a4906aec537421f481269 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:20:14.690049 31809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 553d0b2dddba47779b0219c71ed9ce83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.690287 31740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e27b968c91a4906aec537421f481269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.691704 31742 tablet_service.cc:1511] Processing CreateTablet for tablet 7523cff6f9fb44b5ba8fc04726d85b65 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:20:14.692076 31739 tablet_service.cc:1511] Processing CreateTablet for tablet f933d4319cf444afb9f67bd265576a31 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:20:14.618794 31866 tablet_bootstrap.cc:492] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:14.694458 31738 tablet_service.cc:1511] Processing CreateTablet for tablet 945fd223ac334bce88917d1b9b4c72e1 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:20:14.695145 31661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ada7cfaa154c8c9cc927936b46c30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.695500 31815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49cc564ccd0d411fa79831efa24f87bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.697331 31737 tablet_service.cc:1511] Processing CreateTablet for tablet 5d0d0402170d41b79dc30f93bee98766 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:20:14.700042 31736 tablet_service.cc:1511] Processing CreateTablet for tablet 70ada7cfaa154c8c9cc927936b46c30a (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:20:14.700889 31817 tablet_service.cc:1511] Processing CreateTablet for tablet b2c281519748458cb11517e3303d0a17 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:20:14.701989 31808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6499e78dd87c4e05be32f5681957a6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.702617 31735 tablet_service.cc:1511] Processing CreateTablet for tablet 5bb2dc5276714f75b6e3e61ca9275f84 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:20:14.705186 31734 tablet_service.cc:1511] Processing CreateTablet for tablet 3a0925b3d6e9448684fd239b27453b69 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:20:14.694458 31741 tablet_service.cc:1511] Processing CreateTablet for tablet b043bbe90410471da70e6b00856b5c8d (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:20:14.715183 31741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b043bbe90410471da70e6b00856b5c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.715956 31739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f933d4319cf444afb9f67bd265576a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.722909 31817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c281519748458cb11517e3303d0a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.723026 31734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a0925b3d6e9448684fd239b27453b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.727028 31737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0d0402170d41b79dc30f93bee98766. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.728628 31735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bb2dc5276714f75b6e3e61ca9275f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.730989 31736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ada7cfaa154c8c9cc927936b46c30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.735154 31738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 945fd223ac334bce88917d1b9b4c72e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.739696 31811 tablet_service.cc:1511] Processing CreateTablet for tablet 3a0925b3d6e9448684fd239b27453b69 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:20:14.741317 31811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a0925b3d6e9448684fd239b27453b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.743566 31742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7523cff6f9fb44b5ba8fc04726d85b65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.759124 31816 tablet_service.cc:1511] Processing CreateTablet for tablet 11075d6cbd1a40ccaf381a4eb70cbf24 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:20:14.760670 31816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11075d6cbd1a40ccaf381a4eb70cbf24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.747959 31866 tablet_bootstrap.cc:654] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.777571 31814 tablet_service.cc:1511] Processing CreateTablet for tablet 504cb3d29e9347fe98f010a3707a55cc (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:20:14.779160 31814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 504cb3d29e9347fe98f010a3707a55cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.812700 31810 tablet_service.cc:1511] Processing CreateTablet for tablet 181ddae071d549aab328e3c1d8553734 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:20:14.814910 31739 tablet_service.cc:1511] Processing CreateTablet for tablet f2e7a775ae0d44108be0965c85245ecf (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:20:14.840864 31739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2e7a775ae0d44108be0965c85245ecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.841866 31812 tablet_service.cc:1511] Processing CreateTablet for tablet 81c14d7d4fc14386b173a86e9623eb25 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:20:14.843655 31810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 181ddae071d549aab328e3c1d8553734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.847923 31665 tablet_service.cc:1511] Processing CreateTablet for tablet a5dfbfce13484fc0b1feba45daaf8247 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:20:14.849400 31665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5dfbfce13484fc0b1feba45daaf8247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.836741 31817 tablet_service.cc:1511] Processing CreateTablet for tablet 804df7046b7545d9890885896a333a3e (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:20:14.860014 31812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c14d7d4fc14386b173a86e9623eb25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.869143 31666 tablet_service.cc:1511] Processing CreateTablet for tablet 553d0b2dddba47779b0219c71ed9ce83 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:20:14.874950 31867 tablet_bootstrap.cc:492] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:14.873555 31741 tablet_service.cc:1511] Processing CreateTablet for tablet a5dfbfce13484fc0b1feba45daaf8247 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:20:14.876103 31741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5dfbfce13484fc0b1feba45daaf8247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.875085 31666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 553d0b2dddba47779b0219c71ed9ce83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.879031 31868 tablet_bootstrap.cc:492] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:14.889909 31868 tablet_bootstrap.cc:654] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.877501 31736 tablet_service.cc:1511] Processing CreateTablet for tablet 553d0b2dddba47779b0219c71ed9ce83 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:20:14.890295 31867 tablet_bootstrap.cc:654] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:14.865114 31660 tablet_service.cc:1511] Processing CreateTablet for tablet f2e7a775ae0d44108be0965c85245ecf (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:20:14.891279 31736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 553d0b2dddba47779b0219c71ed9ce83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.891950 31817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 804df7046b7545d9890885896a333a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.897501 31738 tablet_service.cc:1511] Processing CreateTablet for tablet 6499e78dd87c4e05be32f5681957a6b0 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:20:14.898654 31663 tablet_service.cc:1511] Processing CreateTablet for tablet 6499e78dd87c4e05be32f5681957a6b0 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:20:14.891911 31660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2e7a775ae0d44108be0965c85245ecf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.906456 31866 tablet_bootstrap.cc:492] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:14.907035 31866 ts_tablet_manager.cc:1403] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.289s	user 0.019s	sys 0.004s
I20260709 12:20:14.907701 31663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6499e78dd87c4e05be32f5681957a6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.912352 31664 tablet_service.cc:1511] Processing CreateTablet for tablet 49cc564ccd0d411fa79831efa24f87bb (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:20:14.918733 31866 raft_consensus.cc:359] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:14.919510 31808 tablet_service.cc:1511] Processing CreateTablet for tablet 992b9a77f3124e49a9bb783e08a9e856 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:20:14.919718 31734 tablet_service.cc:1511] Processing CreateTablet for tablet b2c281519748458cb11517e3303d0a17 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:20:14.926882 31815 tablet_service.cc:1511] Processing CreateTablet for tablet ab8ea7b957bd41e0a2e7967cbd101f36 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:20:14.929098 31668 tablet_service.cc:1511] Processing CreateTablet for tablet b2c281519748458cb11517e3303d0a17 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
W20260709 12:20:14.929435 31624 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:14.932170 31661 tablet_service.cc:1511] Processing CreateTablet for tablet 11075d6cbd1a40ccaf381a4eb70cbf24 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:20:14.937083 31867 tablet_bootstrap.cc:492] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:14.934366 31659 tablet_service.cc:1511] Processing CreateTablet for tablet 504cb3d29e9347fe98f010a3707a55cc (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:20:14.937616 31867 ts_tablet_manager.cc:1403] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.063s	user 0.002s	sys 0.012s
I20260709 12:20:14.935387 31815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab8ea7b957bd41e0a2e7967cbd101f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.940047 31808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 992b9a77f3124e49a9bb783e08a9e856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.943248 31742 tablet_service.cc:1511] Processing CreateTablet for tablet 49cc564ccd0d411fa79831efa24f87bb (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:20:14.924109 31738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6499e78dd87c4e05be32f5681957a6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.953819 31734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c281519748458cb11517e3303d0a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.957545 31664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49cc564ccd0d411fa79831efa24f87bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.958211 31659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 504cb3d29e9347fe98f010a3707a55cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.958930 31668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c281519748458cb11517e3303d0a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.961918 31737 tablet_service.cc:1511] Processing CreateTablet for tablet 11075d6cbd1a40ccaf381a4eb70cbf24 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:20:14.963275 31737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11075d6cbd1a40ccaf381a4eb70cbf24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.964792 31743 tablet_service.cc:1511] Processing CreateTablet for tablet 504cb3d29e9347fe98f010a3707a55cc (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:20:14.966166 31743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 504cb3d29e9347fe98f010a3707a55cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.966325 31661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11075d6cbd1a40ccaf381a4eb70cbf24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.961840 31666 tablet_service.cc:1511] Processing CreateTablet for tablet 804df7046b7545d9890885896a333a3e (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:20:14.970829 31814 tablet_service.cc:1511] Processing CreateTablet for tablet 04bdf51be7b94825a28028a5cfb96245 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:20:14.972250 31814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04bdf51be7b94825a28028a5cfb96245. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.972904 31667 tablet_service.cc:1511] Processing CreateTablet for tablet 181ddae071d549aab328e3c1d8553734 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:20:14.949400 31867 raft_consensus.cc:359] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:14.978952 31867 raft_consensus.cc:385] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.979353 31867 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.980136 31867 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:14.980763 31810 tablet_service.cc:1511] Processing CreateTablet for tablet ef31b84f462f424fa6162b810ea8c8a9 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:20:14.985205 31868 tablet_bootstrap.cc:492] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:14.985019 31735 tablet_service.cc:1511] Processing CreateTablet for tablet 181ddae071d549aab328e3c1d8553734 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:20:14.985668 31868 ts_tablet_manager.cc:1403] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.121s	user 0.011s	sys 0.004s
I20260709 12:20:14.985905 31810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef31b84f462f424fa6162b810ea8c8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.988458 31736 tablet_service.cc:1511] Processing CreateTablet for tablet 81c14d7d4fc14386b173a86e9623eb25 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:20:14.988409 31741 tablet_service.cc:1511] Processing CreateTablet for tablet 804df7046b7545d9890885896a333a3e (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:20:14.991293 31868 raft_consensus.cc:359] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:14.991444 31663 tablet_service.cc:1511] Processing CreateTablet for tablet 81c14d7d4fc14386b173a86e9623eb25 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:20:14.992156 31868 raft_consensus.cc:385] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:14.992507 31868 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:14.993558 31868 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:14.994850 31742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49cc564ccd0d411fa79831efa24f87bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:14.921916 31866 raft_consensus.cc:385] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.004567 31866 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.005633 31866 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.007192 31741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 804df7046b7545d9890885896a333a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.009337 31739 tablet_service.cc:1511] Processing CreateTablet for tablet 992b9a77f3124e49a9bb783e08a9e856 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:20:15.013972 31739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 992b9a77f3124e49a9bb783e08a9e856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.019040 31667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 181ddae071d549aab328e3c1d8553734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.024892 31666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 804df7046b7545d9890885896a333a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.026160 31815 tablet_service.cc:1511] Processing CreateTablet for tablet d30338be26ae44ff8e627f7b8a99036c (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:20:15.027115 31735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 181ddae071d549aab328e3c1d8553734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.027586 31815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30338be26ae44ff8e627f7b8a99036c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.034953 31736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c14d7d4fc14386b173a86e9623eb25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.034610 31660 tablet_service.cc:1511] Processing CreateTablet for tablet 992b9a77f3124e49a9bb783e08a9e856 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:20:15.043303 31867 ts_tablet_manager.cc:1434] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.105s	user 0.015s	sys 0.003s
I20260709 12:20:15.044137 31703 heartbeater.cc:499] Master 127.29.201.254:37495 was elected leader, sending a full tablet report...
I20260709 12:20:15.044373 31867 tablet_bootstrap.cc:492] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:15.047564 31663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c14d7d4fc14386b173a86e9623eb25. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.063514 31662 tablet_service.cc:1511] Processing CreateTablet for tablet ab8ea7b957bd41e0a2e7967cbd101f36 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:20:15.067909 31739 tablet_service.cc:1511] Processing CreateTablet for tablet 04bdf51be7b94825a28028a5cfb96245 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:20:15.068360 31817 tablet_service.cc:1511] Processing CreateTablet for tablet daa53c7009fe4881956dcfb5a20febde (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:20:15.069291 31739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04bdf51be7b94825a28028a5cfb96245. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.069669 31817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daa53c7009fe4881956dcfb5a20febde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.070989 31662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab8ea7b957bd41e0a2e7967cbd101f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.075500 31734 tablet_service.cc:1511] Processing CreateTablet for tablet ef31b84f462f424fa6162b810ea8c8a9 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:20:15.076938 31734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef31b84f462f424fa6162b810ea8c8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.083000 31660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 992b9a77f3124e49a9bb783e08a9e856. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.038775 31740 tablet_service.cc:1511] Processing CreateTablet for tablet ab8ea7b957bd41e0a2e7967cbd101f36 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:20:15.087927 31740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab8ea7b957bd41e0a2e7967cbd101f36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.091619 31778 heartbeater.cc:499] Master 127.29.201.254:37495 was elected leader, sending a full tablet report...
I20260709 12:20:15.092329 31868 ts_tablet_manager.cc:1434] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.106s	user 0.014s	sys 0.000s
I20260709 12:20:15.087872 31659 tablet_service.cc:1511] Processing CreateTablet for tablet 04bdf51be7b94825a28028a5cfb96245 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:20:15.096021 31659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04bdf51be7b94825a28028a5cfb96245. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.099282 31668 tablet_service.cc:1511] Processing CreateTablet for tablet ef31b84f462f424fa6162b810ea8c8a9 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:20:15.100678 31668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef31b84f462f424fa6162b810ea8c8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.112006 31852 heartbeater.cc:499] Master 127.29.201.254:37495 was elected leader, sending a full tablet report...
I20260709 12:20:15.112768 31868 tablet_bootstrap.cc:492] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:15.116230 31741 tablet_service.cc:1511] Processing CreateTablet for tablet d30338be26ae44ff8e627f7b8a99036c (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:20:15.121759 31874 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.122187 31665 tablet_service.cc:1511] Processing CreateTablet for tablet d30338be26ae44ff8e627f7b8a99036c (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:20:15.127595 31810 tablet_service.cc:1511] Processing CreateTablet for tablet fc47871ef692418388113c1d6c88b262 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:20:15.124266 31874 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.131726 31866 ts_tablet_manager.cc:1434] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.224s	user 0.010s	sys 0.003s
I20260709 12:20:15.132757 31873 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.134544 31873 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.156579 31874 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:15.154175 31665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30338be26ae44ff8e627f7b8a99036c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.166285 31867 tablet_bootstrap.cc:654] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.167222 31810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc47871ef692418388113c1d6c88b262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.168659 31868 tablet_bootstrap.cc:654] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.188398 31812 tablet_service.cc:1511] Processing CreateTablet for tablet 354c519f070b453dadcdef3f0d835c69 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:20:15.143978 31873 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:15.194032 31741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30338be26ae44ff8e627f7b8a99036c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.198364 31866 tablet_bootstrap.cc:492] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:15.211726 31815 tablet_service.cc:1511] Processing CreateTablet for tablet e59926725a06436e9899dbbbcae3f622 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:20:15.206609 31812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 354c519f070b453dadcdef3f0d835c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.218959 31815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59926725a06436e9899dbbbcae3f622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.219795 31743 tablet_service.cc:1511] Processing CreateTablet for tablet fc47871ef692418388113c1d6c88b262 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:20:15.221141 31743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc47871ef692418388113c1d6c88b262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.228971 31666 tablet_service.cc:1511] Processing CreateTablet for tablet fc47871ef692418388113c1d6c88b262 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:20:15.221822 31742 tablet_service.cc:1511] Processing CreateTablet for tablet b416bf5304744b14b72501e697b8c951 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:20:15.237416 31664 tablet_service.cc:1511] Processing CreateTablet for tablet daa53c7009fe4881956dcfb5a20febde (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:20:15.239013 31664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daa53c7009fe4881956dcfb5a20febde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.247722 31667 tablet_service.cc:1511] Processing CreateTablet for tablet b416bf5304744b14b72501e697b8c951 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:20:15.256613 31814 tablet_service.cc:1511] Processing CreateTablet for tablet b416bf5304744b14b72501e697b8c951 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:20:15.258148 31814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b416bf5304744b14b72501e697b8c951. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.256712 31738 tablet_service.cc:1511] Processing CreateTablet for tablet daa53c7009fe4881956dcfb5a20febde (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:20:15.263818 31736 tablet_service.cc:1511] Processing CreateTablet for tablet 354c519f070b453dadcdef3f0d835c69 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:20:15.283000 31666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc47871ef692418388113c1d6c88b262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.288702 31742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b416bf5304744b14b72501e697b8c951. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.293562 31866 tablet_bootstrap.cc:654] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.295475 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:15.296871 31868 tablet_bootstrap.cc:492] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
W20260709 12:20:15.297451 31717 leader_election.cc:343] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.297441 31868 ts_tablet_manager.cc:1403] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.185s	user 0.009s	sys 0.009s
I20260709 12:20:15.298318 31660 tablet_service.cc:1511] Processing CreateTablet for tablet 354c519f070b453dadcdef3f0d835c69 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:20:15.299755 31660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 354c519f070b453dadcdef3f0d835c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.301265 31868 raft_consensus.cc:359] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.302040 31868 raft_consensus.cc:385] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.302332 31868 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.303272 31868 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.319684 31740 tablet_service.cc:1511] Processing CreateTablet for tablet e59926725a06436e9899dbbbcae3f622 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:20:15.321667 31740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59926725a06436e9899dbbbcae3f622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.327718 31737 tablet_service.cc:1511] Processing CreateTablet for tablet d0a3795ad4d543f0998c3834aac7be45 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:20:15.331306 31667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b416bf5304744b14b72501e697b8c951. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.340369 31810 tablet_service.cc:1511] Processing CreateTablet for tablet d0a3795ad4d543f0998c3834aac7be45 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:20:15.342327 31810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a3795ad4d543f0998c3834aac7be45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.346581 31659 tablet_service.cc:1511] Processing CreateTablet for tablet e59926725a06436e9899dbbbcae3f622 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:20:15.348042 31659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59926725a06436e9899dbbbcae3f622. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.329675 31737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a3795ad4d543f0998c3834aac7be45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.353541 31736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 354c519f070b453dadcdef3f0d835c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.353678 31662 tablet_service.cc:1511] Processing CreateTablet for tablet d0a3795ad4d543f0998c3834aac7be45 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:20:15.355096 31662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0a3795ad4d543f0998c3834aac7be45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.357430 31738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daa53c7009fe4881956dcfb5a20febde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.369517 31867 tablet_bootstrap.cc:492] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:15.370373 31867 ts_tablet_manager.cc:1403] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.326s	user 0.017s	sys 0.008s
I20260709 12:20:15.385679 31872 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.386395 31872 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.397734 31872 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:15.416003 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:15.416020 31817 tablet_service.cc:1511] Processing CreateTablet for tablet 00854b2a70f24ef3a7b43d17477a0ccf (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:20:15.419227 31817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00854b2a70f24ef3a7b43d17477a0ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.418954 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:15.423554 31868 ts_tablet_manager.cc:1434] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.125s	user 0.005s	sys 0.003s
I20260709 12:20:15.434211 31867 raft_consensus.cc:359] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.439666 31867 raft_consensus.cc:385] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.469309 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:15.471336 31718 leader_election.cc:343] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.471890 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:15.473208 31873 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:15.474205 31868 tablet_bootstrap.cc:492] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:15.475653 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:15.478380 31643 leader_election.cc:343] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:15.464303 31791 leader_election.cc:343] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:15.480051 31791 leader_election.cc:343] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.480459 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:15.481263 31874 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:15.464841 31867 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.487485 31867 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:15.491605 31561 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 96229us
I20260709 12:20:15.493498 31868 tablet_bootstrap.cc:654] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.497920 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:15.498698 31753 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:15.498364 31660 tablet_service.cc:1511] Processing CreateTablet for tablet bc55086b81374990afb970c972568882 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:20:15.500355 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:15.502307 31814 tablet_service.cc:1511] Processing CreateTablet for tablet 9c849da11b414111b8461d6ae06c15b3 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:20:15.504294 31814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c849da11b414111b8461d6ae06c15b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.508139 31872 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:15.508600 31872 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:15.508981 31872 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.516539 31872 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.520289 31867 ts_tablet_manager.cc:1434] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.149s	user 0.009s	sys 0.000s
I20260709 12:20:15.520942 31866 tablet_bootstrap.cc:492] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:15.521482 31866 ts_tablet_manager.cc:1403] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.345s	user 0.017s	sys 0.015s
I20260709 12:20:15.524595 31866 raft_consensus.cc:359] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.525369 31866 raft_consensus.cc:385] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.525688 31866 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.526198 31666 tablet_service.cc:1511] Processing CreateTablet for tablet 00854b2a70f24ef3a7b43d17477a0ccf (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:20:15.526598 31866 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.526947 31736 tablet_service.cc:1511] Processing CreateTablet for tablet bc55086b81374990afb970c972568882 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:20:15.528242 31736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc55086b81374990afb970c972568882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.529388 31866 ts_tablet_manager.cc:1434] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20260709 12:20:15.530475 31866 tablet_bootstrap.cc:492] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:15.521477 31867 tablet_bootstrap.cc:492] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:15.531572 31815 tablet_service.cc:1511] Processing CreateTablet for tablet bc55086b81374990afb970c972568882 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:20:15.532401 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:15.532970 31815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc55086b81374990afb970c972568882. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:20:15.533855 31643 leader_election.cc:343] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.534584 31872 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:15.536269 31741 tablet_service.cc:1511] Processing CreateTablet for tablet 00854b2a70f24ef3a7b43d17477a0ccf (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:20:15.536952 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:15.537533 31753 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.537952 31660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc55086b81374990afb970c972568882. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.540524 31812 tablet_service.cc:1511] Processing CreateTablet for tablet c34755b3b9d040f69be353e35d7431e7 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:20:15.541157 31810 tablet_service.cc:1511] Processing CreateTablet for tablet 6821bb8779e64160887e3195aa5078a1 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:20:15.541862 31812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34755b3b9d040f69be353e35d7431e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.537608 31741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00854b2a70f24ef3a7b43d17477a0ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.545584 31753 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:15.548846 31866 tablet_bootstrap.cc:654] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.549667 31738 tablet_service.cc:1511] Processing CreateTablet for tablet 6821bb8779e64160887e3195aa5078a1 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:20:15.551709 31666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00854b2a70f24ef3a7b43d17477a0ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.552171 31738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6821bb8779e64160887e3195aa5078a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.559494 31867 tablet_bootstrap.cc:654] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.565585 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:15.542586 31810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6821bb8779e64160887e3195aa5078a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.567708 31662 tablet_service.cc:1511] Processing CreateTablet for tablet 9c849da11b414111b8461d6ae06c15b3 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:20:15.569079 31662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c849da11b414111b8461d6ae06c15b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.571731 31667 tablet_service.cc:1511] Processing CreateTablet for tablet 6821bb8779e64160887e3195aa5078a1 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:20:15.573069 31667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6821bb8779e64160887e3195aa5078a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.582556 31737 tablet_service.cc:1511] Processing CreateTablet for tablet 9c849da11b414111b8461d6ae06c15b3 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:20:15.584561 31737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c849da11b414111b8461d6ae06c15b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.588536 31742 tablet_service.cc:1511] Processing CreateTablet for tablet c34755b3b9d040f69be353e35d7431e7 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:20:15.589880 31742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34755b3b9d040f69be353e35d7431e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.592185 31664 tablet_service.cc:1511] Processing CreateTablet for tablet c34755b3b9d040f69be353e35d7431e7 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:20:15.593526 31664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34755b3b9d040f69be353e35d7431e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.597909 31808 tablet_service.cc:1511] Processing CreateTablet for tablet 45588632b8d64f9fb9c2e2ba18b20546 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:20:15.599246 31808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45588632b8d64f9fb9c2e2ba18b20546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.600677 31872 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:15.604339 31816 tablet_service.cc:1511] Processing CreateTablet for tablet bc0ece2345cd46edb250e06e8f5431d6 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:20:15.605749 31816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0ece2345cd46edb250e06e8f5431d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.614959 31668 tablet_service.cc:1511] Processing CreateTablet for tablet 45588632b8d64f9fb9c2e2ba18b20546 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:20:15.616227 31817 tablet_service.cc:1511] Processing CreateTablet for tablet 20650b4ef2514a05bde5dfd38a379f32 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:20:15.616822 31668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45588632b8d64f9fb9c2e2ba18b20546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.619499 31659 tablet_service.cc:1511] Processing CreateTablet for tablet 20650b4ef2514a05bde5dfd38a379f32 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:20:15.621330 31659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20650b4ef2514a05bde5dfd38a379f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.618559 31740 tablet_service.cc:1511] Processing CreateTablet for tablet 20650b4ef2514a05bde5dfd38a379f32 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:20:15.624495 31740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20650b4ef2514a05bde5dfd38a379f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.638334 31868 tablet_bootstrap.cc:492] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:15.638967 31868 ts_tablet_manager.cc:1403] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.191s	user 0.022s	sys 0.063s
I20260709 12:20:15.642333 31868 raft_consensus.cc:359] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.643208 31868 raft_consensus.cc:385] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.643528 31868 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.647128 31872 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:15.648113 31872 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.651249 31817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20650b4ef2514a05bde5dfd38a379f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.653733 31866 tablet_bootstrap.cc:492] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:15.654247 31866 ts_tablet_manager.cc:1403] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.124s	user 0.010s	sys 0.001s
I20260709 12:20:15.657053 31734 tablet_service.cc:1511] Processing CreateTablet for tablet 45588632b8d64f9fb9c2e2ba18b20546 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:20:15.658526 31734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45588632b8d64f9fb9c2e2ba18b20546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.658401 31739 tablet_service.cc:1511] Processing CreateTablet for tablet bc0ece2345cd46edb250e06e8f5431d6 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:20:15.659777 31739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0ece2345cd46edb250e06e8f5431d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.659610 31809 tablet_service.cc:1511] Processing CreateTablet for tablet 6e7ad549735d4871af75658a91bb98c4 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:20:15.661336 31866 raft_consensus.cc:359] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.662070 31866 raft_consensus.cc:385] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.661502 31868 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.664876 31867 tablet_bootstrap.cc:492] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:15.665481 31867 ts_tablet_manager.cc:1403] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.144s	user 0.010s	sys 0.007s
I20260709 12:20:15.670054 31868 ts_tablet_manager.cc:1434] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.031s	user 0.001s	sys 0.008s
I20260709 12:20:15.675658 31868 tablet_bootstrap.cc:492] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:15.677739 31867 raft_consensus.cc:359] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.678570 31867 raft_consensus.cc:385] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.678925 31867 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.679665 31867 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.682081 31867 ts_tablet_manager.cc:1434] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.000s
I20260709 12:20:15.662388 31866 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.704526 31866 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.691574 31809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e7ad549735d4871af75658a91bb98c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.696664 31811 tablet_service.cc:1511] Processing CreateTablet for tablet f6404cb9c2174326b06cbbb0d27d43f2 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:20:15.707835 31811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6404cb9c2174326b06cbbb0d27d43f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.697712 31665 tablet_service.cc:1511] Processing CreateTablet for tablet ffab0b886efd41099d99c20339a3c4ae (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:20:15.699851 31743 tablet_service.cc:1511] Processing CreateTablet for tablet 6e7ad549735d4871af75658a91bb98c4 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:20:15.700085 31735 tablet_service.cc:1511] Processing CreateTablet for tablet f6404cb9c2174326b06cbbb0d27d43f2 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:20:15.696664 31661 tablet_service.cc:1511] Processing CreateTablet for tablet f6404cb9c2174326b06cbbb0d27d43f2 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:20:15.711254 31743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e7ad549735d4871af75658a91bb98c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.712391 31891 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.713312 31665 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffab0b886efd41099d99c20339a3c4ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.712973 31891 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.716171 31891 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:15.717530 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:15.718988 31735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6404cb9c2174326b06cbbb0d27d43f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.719924 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:15.721886 31643 leader_election.cc:343] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:15.722738 31642 leader_election.cc:343] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.723403 31866 ts_tablet_manager.cc:1434] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.069s	user 0.006s	sys 0.003s
I20260709 12:20:15.723215 31642 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:15.724479 31891 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:15.726861 31661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6404cb9c2174326b06cbbb0d27d43f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.727075 31866 tablet_bootstrap.cc:492] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:15.731011 31660 tablet_service.cc:1511] Processing CreateTablet for tablet bc0ece2345cd46edb250e06e8f5431d6 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:20:15.732451 31660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0ece2345cd46edb250e06e8f5431d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.733623 31867 tablet_bootstrap.cc:492] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:15.707389 31738 tablet_service.cc:1511] Processing CreateTablet for tablet ffab0b886efd41099d99c20339a3c4ae (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:20:15.735587 31738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffab0b886efd41099d99c20339a3c4ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.700356 31664 tablet_service.cc:1511] Processing CreateTablet for tablet 6e7ad549735d4871af75658a91bb98c4 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:20:15.740146 31867 tablet_bootstrap.cc:654] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.740240 31664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e7ad549735d4871af75658a91bb98c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.742436 31812 tablet_service.cc:1511] Processing CreateTablet for tablet ffab0b886efd41099d99c20339a3c4ae (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:20:15.751487 31812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffab0b886efd41099d99c20339a3c4ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.766615 31659 tablet_service.cc:1511] Processing CreateTablet for tablet 55f4b2e07ada4e06a285803fcbfd6344 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:20:15.768262 31659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55f4b2e07ada4e06a285803fcbfd6344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.767903 31736 tablet_service.cc:1511] Processing CreateTablet for tablet 55f4b2e07ada4e06a285803fcbfd6344 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:20:15.769382 31736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55f4b2e07ada4e06a285803fcbfd6344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.743532 31868 tablet_bootstrap.cc:654] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.776189 31866 tablet_bootstrap.cc:654] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:15.776894 31662 tablet_service.cc:1511] Processing CreateTablet for tablet c22bf85ce45841fb80e303e2746080da (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:20:15.779778 31737 tablet_service.cc:1511] Processing CreateTablet for tablet c22bf85ce45841fb80e303e2746080da (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:20:15.791364 31662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22bf85ce45841fb80e303e2746080da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.803638 31737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22bf85ce45841fb80e303e2746080da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.808346 31808 tablet_service.cc:1511] Processing CreateTablet for tablet c22bf85ce45841fb80e303e2746080da (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:20:15.808871 31873 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.809535 31873 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.818748 31873 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:15.819242 31664 tablet_service.cc:1511] Processing CreateTablet for tablet 2c162dbe81714319bfeccd9d7cb7bfea (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:20:15.820905 31664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c162dbe81714319bfeccd9d7cb7bfea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.821393 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:15.814947 31808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22bf85ce45841fb80e303e2746080da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.823604 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:15.831848 31717 leader_election.cc:343] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.832566 31667 tablet_service.cc:1511] Processing CreateTablet for tablet e59f73cf80df4c20b6c17715783f2121 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:20:15.832696 31827 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:15.852550 31809 tablet_service.cc:1511] Processing CreateTablet for tablet 55f4b2e07ada4e06a285803fcbfd6344 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:20:15.858604 31809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55f4b2e07ada4e06a285803fcbfd6344. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.859297 31810 tablet_service.cc:1511] Processing CreateTablet for tablet e59f73cf80df4c20b6c17715783f2121 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:20:15.860883 31667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59f73cf80df4c20b6c17715783f2121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.862912 31814 tablet_service.cc:1511] Processing CreateTablet for tablet 2c162dbe81714319bfeccd9d7cb7bfea (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:20:15.849794 31740 tablet_service.cc:1511] Processing CreateTablet for tablet e59f73cf80df4c20b6c17715783f2121 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:20:15.867647 31666 tablet_service.cc:1511] Processing CreateTablet for tablet 8725620974f7465eb0371f48761a1625 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:20:15.868148 31740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59f73cf80df4c20b6c17715783f2121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.869024 31666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8725620974f7465eb0371f48761a1625. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.877471 31738 tablet_service.cc:1511] Processing CreateTablet for tablet 8725620974f7465eb0371f48761a1625 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:20:15.878929 31738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8725620974f7465eb0371f48761a1625. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.878988 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:15.880555 31873 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:15.880949 31873 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:15.881297 31873 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.885452 31867 tablet_bootstrap.cc:492] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:15.886091 31867 ts_tablet_manager.cc:1403] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.153s	user 0.018s	sys 0.029s
I20260709 12:20:15.886859 31574 catalog_manager.cc:5697] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:15.890725 31742 tablet_service.cc:1511] Processing CreateTablet for tablet 2c162dbe81714319bfeccd9d7cb7bfea (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:20:15.892164 31742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c162dbe81714319bfeccd9d7cb7bfea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.893180 31867 raft_consensus.cc:359] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.893945 31867 raft_consensus.cc:385] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.894255 31867 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.896231 31815 tablet_service.cc:1511] Processing CreateTablet for tablet 8725620974f7465eb0371f48761a1625 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:20:15.924732 31814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c162dbe81714319bfeccd9d7cb7bfea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.923367 31873 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.927546 31810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59f73cf80df4c20b6c17715783f2121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.928417 31872 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:15.928997 31872 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.933316 31872 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:15.934618 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:15.935317 31827 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:15.936003 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:15.936646 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:15.936781 31753 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:15.936832 31811 tablet_service.cc:1511] Processing CreateTablet for tablet 436783e8e1cc49edbad793a31645e2d6 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:20:15.937752 31872 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:15.938210 31872 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:15.938651 31872 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.940547 31868 tablet_bootstrap.cc:492] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:15.941174 31868 ts_tablet_manager.cc:1403] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.267s	user 0.027s	sys 0.023s
I20260709 12:20:15.938727 31738 tablet_service.cc:1511] Processing CreateTablet for tablet 436783e8e1cc49edbad793a31645e2d6 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:20:15.942914 31866 tablet_bootstrap.cc:492] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:15.943495 31866 ts_tablet_manager.cc:1403] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.217s	user 0.013s	sys 0.007s
W20260709 12:20:15.934542 31624 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:15.944494 31738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 436783e8e1cc49edbad793a31645e2d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.944558 31663 tablet_service.cc:1511] Processing CreateTablet for tablet 812ed9ab41214308b368b53417e6e9f3 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:20:15.947203 31872 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.947767 31663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 812ed9ab41214308b368b53417e6e9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.948187 31741 tablet_service.cc:1511] Processing CreateTablet for tablet 812ed9ab41214308b368b53417e6e9f3 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:20:15.949445 31741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 812ed9ab41214308b368b53417e6e9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.950728 31866 raft_consensus.cc:359] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.952644 31868 raft_consensus.cc:359] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.953502 31868 raft_consensus.cc:385] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.953855 31868 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.951608 31866 raft_consensus.cc:385] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:15.967300 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:15.969720 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:15.970913 31872 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:15.972106 31815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8725620974f7465eb0371f48761a1625. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:15.976473 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:15.974753 31753 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:15.982059 31717 leader_election.cc:343] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:15.984298 31827 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:15.987329 31816 tablet_service.cc:1511] Processing CreateTablet for tablet 812ed9ab41214308b368b53417e6e9f3 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:20:15.971954 31866 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:15.993650 31827 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:15.994097 31866 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.994993 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:15.996263 31872 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:15.996585 31866 ts_tablet_manager.cc:1434] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.053s	user 0.008s	sys 0.000s
I20260709 12:20:15.997049 31872 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:15.993135 31753 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:15.998131 31872 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.925678 31867 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:15.974089 31868 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.001340 31816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 812ed9ab41214308b368b53417e6e9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.007714 31866 tablet_bootstrap.cc:492] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:16.008666 31867 ts_tablet_manager.cc:1434] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.122s	user 0.008s	sys 0.000s
I20260709 12:20:16.009294 31811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 436783e8e1cc49edbad793a31645e2d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.009788 31867 tablet_bootstrap.cc:492] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:16.012228 31668 tablet_service.cc:1511] Processing CreateTablet for tablet 436783e8e1cc49edbad793a31645e2d6 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:20:16.014178 31668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 436783e8e1cc49edbad793a31645e2d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.014742 31873 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:16.016093 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:16.024777 31866 tablet_bootstrap.cc:654] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.025627 31867 tablet_bootstrap.cc:654] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.040150 31827 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.060266 31827 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:16.061546 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:16.062819 31873 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:16.063292 31873 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:16.064102 31873 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.065322 31812 tablet_service.cc:1511] Processing CreateTablet for tablet f79ad11703094c998ce1b3905e5c9ff6 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:20:16.079885 31868 ts_tablet_manager.cc:1434] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.138s	user 0.010s	sys 0.005s
I20260709 12:20:16.080603 31817 tablet_service.cc:1511] Processing CreateTablet for tablet b08faba9c13b4f0f85baf8b16c97d85b (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:20:16.082110 31817 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b08faba9c13b4f0f85baf8b16c97d85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.084702 31891 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.084954 31741 tablet_service.cc:1511] Processing CreateTablet for tablet f79ad11703094c998ce1b3905e5c9ff6 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:20:16.085230 31891 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.085287 31812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f79ad11703094c998ce1b3905e5c9ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.088644 31734 tablet_service.cc:1511] Processing CreateTablet for tablet b08faba9c13b4f0f85baf8b16c97d85b (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:20:16.095794 31868 tablet_bootstrap.cc:492] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:16.095947 31664 tablet_service.cc:1511] Processing CreateTablet for tablet f79ad11703094c998ce1b3905e5c9ff6 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:20:16.107326 31664 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f79ad11703094c998ce1b3905e5c9ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.116670 31741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f79ad11703094c998ce1b3905e5c9ff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.102437 31659 tablet_service.cc:1511] Processing CreateTablet for tablet b08faba9c13b4f0f85baf8b16c97d85b (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:20:16.117765 31734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b08faba9c13b4f0f85baf8b16c97d85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.118297 31659 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b08faba9c13b4f0f85baf8b16c97d85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.124003 31891 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:16.124642 31891 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:16.125716 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:16.127342 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:16.129834 31643 leader_election.cc:343] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.129395 31663 tablet_service.cc:1511] Processing CreateTablet for tablet 292851e482024b3a8dcd05c5dab04534 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
W20260709 12:20:16.147727 31642 leader_election.cc:343] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.148285 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:16.148546 31867 tablet_bootstrap.cc:492] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:16.149127 31867 ts_tablet_manager.cc:1403] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.140s	user 0.022s	sys 0.012s
I20260709 12:20:16.151929 31903 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:16.161576 31866 tablet_bootstrap.cc:492] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:16.162216 31866 ts_tablet_manager.cc:1403] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.155s	user 0.018s	sys 0.024s
I20260709 12:20:16.171305 31736 tablet_service.cc:1511] Processing CreateTablet for tablet 292851e482024b3a8dcd05c5dab04534 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:20:16.172721 31736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292851e482024b3a8dcd05c5dab04534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.176261 31873 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.177604 31868 tablet_bootstrap.cc:654] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.195070 31873 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.196480 31740 tablet_service.cc:1511] Processing CreateTablet for tablet 76c4ab1f56e44ff7b6adf7cfe850e01d (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:20:16.178150 31663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292851e482024b3a8dcd05c5dab04534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.180609 31812 tablet_service.cc:1511] Processing CreateTablet for tablet 76c4ab1f56e44ff7b6adf7cfe850e01d (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:20:16.183660 31815 tablet_service.cc:1511] Processing CreateTablet for tablet 292851e482024b3a8dcd05c5dab04534 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:20:16.188819 31735 tablet_service.cc:1511] Processing CreateTablet for tablet 35c247463bef4f77990becbf6089be7d (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:20:16.192358 31666 tablet_service.cc:1511] Processing CreateTablet for tablet 76c4ab1f56e44ff7b6adf7cfe850e01d (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:20:16.201234 31815 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 292851e482024b3a8dcd05c5dab04534. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.172335 31866 raft_consensus.cc:359] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.204205 31735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35c247463bef4f77990becbf6089be7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.206964 31866 raft_consensus.cc:385] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.207346 31810 tablet_service.cc:1511] Processing CreateTablet for tablet 35c247463bef4f77990becbf6089be7d (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:20:16.207867 31866 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.204236 31812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76c4ab1f56e44ff7b6adf7cfe850e01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.209811 31810 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35c247463bef4f77990becbf6089be7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.211572 31867 raft_consensus.cc:359] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:16.217036 31643 consensus_peers.cc:597] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:16.238171 31866 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.238847 31666 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76c4ab1f56e44ff7b6adf7cfe850e01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.240295 31740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76c4ab1f56e44ff7b6adf7cfe850e01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.240726 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:16.240971 31866 ts_tablet_manager.cc:1434] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.078s	user 0.007s	sys 0.003s
I20260709 12:20:16.242046 31866 tablet_bootstrap.cc:492] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
W20260709 12:20:16.242550 31717 leader_election.cc:343] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.231055 31867 raft_consensus.cc:385] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.247967 31867 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.250064 31738 tablet_service.cc:1511] Processing CreateTablet for tablet d11aa4e638cd4a1dad543baff3b3f965 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:20:16.251677 31738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d11aa4e638cd4a1dad543baff3b3f965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.268957 31873 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:16.270201 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:16.272603 31718 leader_election.cc:343] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.273097 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:16.274303 31873 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:16.282358 31667 tablet_service.cc:1511] Processing CreateTablet for tablet d11aa4e638cd4a1dad543baff3b3f965 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:20:16.291337 31668 tablet_service.cc:1511] Processing CreateTablet for tablet 35c247463bef4f77990becbf6089be7d (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:20:16.294028 31667 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d11aa4e638cd4a1dad543baff3b3f965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.295066 31866 tablet_bootstrap.cc:654] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.294158 31576 catalog_manager.cc:5697] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.305274 31809 tablet_service.cc:1511] Processing CreateTablet for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:20:16.306264 31735 tablet_service.cc:1511] Processing CreateTablet for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:20:16.306917 31809 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.312131 31663 tablet_service.cc:1511] Processing CreateTablet for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:20:16.324465 31868 tablet_bootstrap.cc:492] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:16.325212 31868 ts_tablet_manager.cc:1403] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.230s	user 0.012s	sys 0.012s
I20260709 12:20:16.308287 31867 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.328487 31660 tablet_service.cc:1511] Processing CreateTablet for tablet a22e63b9ca824c92ae9ba374d2f3242a (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:20:16.330772 31867 ts_tablet_manager.cc:1434] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.181s	user 0.006s	sys 0.004s
I20260709 12:20:16.330822 31868 raft_consensus.cc:359] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.336217 31868 raft_consensus.cc:385] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.336544 31868 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.340617 31814 tablet_service.cc:1511] Processing CreateTablet for tablet d11aa4e638cd4a1dad543baff3b3f965 (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:20:16.342763 31814 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d11aa4e638cd4a1dad543baff3b3f965. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.318679 31668 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35c247463bef4f77990becbf6089be7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.359161 31663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.358927 31868 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.360734 31905 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.361259 31905 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.365810 31905 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:16.367431 31812 tablet_service.cc:1511] Processing CreateTablet for tablet a22e63b9ca824c92ae9ba374d2f3242a (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:20:16.367755 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:16.368898 31812 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a22e63b9ca824c92ae9ba374d2f3242a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.369390 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:16.370082 31678 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:16.371465 31735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.371186 31741 tablet_service.cc:1511] Processing CreateTablet for tablet 2f2f85d943e24cfdb47fe2ad3185097f (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
W20260709 12:20:16.369370 31718 leader_election.cc:343] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.372527 31741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f2f85d943e24cfdb47fe2ad3185097f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.374907 31717 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:16.375881 31873 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:16.376268 31873 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.376601 31873 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.369371 31662 tablet_service.cc:1511] Processing CreateTablet for tablet 2f2f85d943e24cfdb47fe2ad3185097f (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:20:16.362491 31868 ts_tablet_manager.cc:1434] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.037s	user 0.008s	sys 0.004s
I20260709 12:20:16.378252 31662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f2f85d943e24cfdb47fe2ad3185097f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.389659 31660 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a22e63b9ca824c92ae9ba374d2f3242a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.390690 31811 tablet_service.cc:1511] Processing CreateTablet for tablet 2f2f85d943e24cfdb47fe2ad3185097f (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:20:16.392747 31811 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f2f85d943e24cfdb47fe2ad3185097f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.395689 31867 tablet_bootstrap.cc:492] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:16.407019 31868 tablet_bootstrap.cc:492] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:16.411896 31736 tablet_service.cc:1511] Processing CreateTablet for tablet a22e63b9ca824c92ae9ba374d2f3242a (DEFAULT_TABLE table=test-workload [id=d81b7fed68a84c348a74895b5b4310a8]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:20:16.413998 31736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a22e63b9ca824c92ae9ba374d2f3242a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:16.412480 31873 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.418016 31872 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.421216 31872 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.426467 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:16.427809 31872 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:16.429071 31642 leader_election.cc:343] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.430725 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:16.432665 31643 leader_election.cc:343] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.433204 31643 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:16.434427 31872 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:16.436568 31576 catalog_manager.cc:5697] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.444242 31867 tablet_bootstrap.cc:654] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.459542 31868 tablet_bootstrap.cc:654] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.468191 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:16.468912 31678 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.470206 31873 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:16.471920 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:16.488958 31874 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.489500 31874 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.491582 31874 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:16.492887 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:16.493125 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:16.494582 31867 tablet_bootstrap.cc:492] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:16.495136 31867 ts_tablet_manager.cc:1403] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.100s	user 0.013s	sys 0.009s
I20260709 12:20:16.496140 31866 tablet_bootstrap.cc:492] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:16.496585 31866 ts_tablet_manager.cc:1403] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.255s	user 0.037s	sys 0.032s
W20260709 12:20:16.498669 31718 leader_election.cc:343] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.503445 31867 raft_consensus.cc:359] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:16.504343 31867 raft_consensus.cc:385] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.504691 31867 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.505504 31867 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:16.508041 31867 ts_tablet_manager.cc:1434] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:20:16.509128 31867 tablet_bootstrap.cc:492] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
W20260709 12:20:16.494587 31791 leader_election.cc:343] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:16.511670 31791 leader_election.cc:343] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.512267 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:16.513932 31717 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:16.515057 31867 tablet_bootstrap.cc:654] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.518718 31874 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:16.519483 31866 raft_consensus.cc:359] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:16.520478 31678 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:16.520232 31866 raft_consensus.cc:385] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.521260 31866 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.522149 31717 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:16.523312 31873 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:16.523906 31873 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:16.524730 31873 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.530825 31866 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:16.533284 31866 ts_tablet_manager.cc:1434] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.036s	user 0.006s	sys 0.000s
I20260709 12:20:16.540416 31866 tablet_bootstrap.cc:492] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:16.548596 31868 tablet_bootstrap.cc:492] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:16.550727 31868 ts_tablet_manager.cc:1403] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.164s	user 0.014s	sys 0.008s
I20260709 12:20:16.555001 31866 tablet_bootstrap.cc:654] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.556054 31867 tablet_bootstrap.cc:492] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:16.556730 31867 ts_tablet_manager.cc:1403] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.048s	user 0.022s	sys 0.011s
I20260709 12:20:16.559940 31867 raft_consensus.cc:359] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.560670 31867 raft_consensus.cc:385] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.560825 31868 raft_consensus.cc:359] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.562973 31868 raft_consensus.cc:385] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.563344 31868 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.561029 31867 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.564941 31867 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.567417 31866 tablet_bootstrap.cc:492] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:16.567595 31867 ts_tablet_manager.cc:1434] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:20:16.567996 31866 ts_tablet_manager.cc:1403] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.029s	user 0.013s	sys 0.000s
I20260709 12:20:16.568181 31576 catalog_manager.cc:5697] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.569074 31867 tablet_bootstrap.cc:492] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:16.571287 31866 raft_consensus.cc:359] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.572139 31866 raft_consensus.cc:385] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.572552 31866 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.573446 31866 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.565057 31868 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.576169 31866 ts_tablet_manager.cc:1434] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:20:16.577103 31867 tablet_bootstrap.cc:654] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.577770 31868 ts_tablet_manager.cc:1434] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.026s	user 0.003s	sys 0.003s
I20260709 12:20:16.577827 31866 tablet_bootstrap.cc:492] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:16.578971 31868 tablet_bootstrap.cc:492] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:16.588178 31868 tablet_bootstrap.cc:654] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:16.589313 31853 tablet.cc:2406] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:20:16.600553 31903 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:16.606957 31866 tablet_bootstrap.cc:654] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.616004 31874 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.616362 31868 tablet_bootstrap.cc:492] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:16.617075 31868 ts_tablet_manager.cc:1403] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.038s	user 0.009s	sys 0.010s
I20260709 12:20:16.616869 31874 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.620738 31868 raft_consensus.cc:359] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.621668 31868 raft_consensus.cc:385] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.621730 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:16.622152 31868 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.622552 31678 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:16.622846 31874 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:16.622674 31873 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:16.623238 31868 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.624132 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:16.625103 31874 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:16.625551 31874 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.625777 31923 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.626049 31868 ts_tablet_manager.cc:1434] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:20:16.626005 31874 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.626267 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:16.627103 31752 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:16.626348 31923 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.631956 31903 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:16.632740 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:16.633713 31868 tablet_bootstrap.cc:492] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:16.635062 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:16.638095 31867 tablet_bootstrap.cc:492] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:16.638236 31752 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:16.638603 31867 ts_tablet_manager.cc:1403] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.070s	user 0.024s	sys 0.028s
W20260709 12:20:16.638775 31643 leader_election.cc:343] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.640452 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:16.641414 31867 raft_consensus.cc:359] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.642153 31867 raft_consensus.cc:385] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.642445 31867 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.642516 31872 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:16.643193 31872 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.643211 31867 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.643774 31872 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.645524 31867 ts_tablet_manager.cc:1434] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.001s
I20260709 12:20:16.645994 31868 tablet_bootstrap.cc:654] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.635802 31874 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.646657 31867 tablet_bootstrap.cc:492] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:16.649063 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:16.630995 31923 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:16.649766 31678 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.649695 31874 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:16.651042 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:16.651693 31752 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.652074 31925 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.652594 31925 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:16.654986 31925 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:16.656422 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:16.657155 31677 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:16.659073 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:16.659607 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:16.661099 31791 leader_election.cc:343] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.661917 31866 tablet_bootstrap.cc:492] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:16.662451 31866 ts_tablet_manager.cc:1403] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.085s	user 0.014s	sys 0.004s
I20260709 12:20:16.662424 31678 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:16.663813 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:16.665577 31866 raft_consensus.cc:359] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.666356 31866 raft_consensus.cc:385] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.666666 31866 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.666777 31752 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:16.667542 31866 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.668992 31874 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:16.669385 31874 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.669757 31874 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.670555 31926 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:16.671334 31926 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:16.672564 31926 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.669394 31867 tablet_bootstrap.cc:654] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.675877 31903 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.676625 31903 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.679804 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:16.682062 31866 ts_tablet_manager.cc:1434] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.019s	user 0.009s	sys 0.003s
I20260709 12:20:16.683432 31866 tablet_bootstrap.cc:492] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:16.684283 31874 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:16.687748 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:16.688248 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:16.689213 31791 leader_election.cc:343] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.691663 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:16.692822 31923 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:16.693619 31866 tablet_bootstrap.cc:654] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.695097 31867 tablet_bootstrap.cc:492] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:16.695622 31867 ts_tablet_manager.cc:1403] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.049s	user 0.008s	sys 0.006s
I20260709 12:20:16.680136 31903 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:16.699082 31867 raft_consensus.cc:359] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.702394 31868 tablet_bootstrap.cc:492] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:16.704200 31867 raft_consensus.cc:385] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.704661 31868 ts_tablet_manager.cc:1403] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.071s	user 0.000s	sys 0.014s
I20260709 12:20:16.705361 31866 tablet_bootstrap.cc:492] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:16.705147 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:16.705827 31866 ts_tablet_manager.cc:1403] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.008s
I20260709 12:20:16.705824 31678 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.706538 31874 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:16.708418 31868 raft_consensus.cc:359] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.713662 31868 raft_consensus.cc:385] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.713984 31678 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:16.704586 31867 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.715296 31867 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.717491 31867 ts_tablet_manager.cc:1434] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
I20260709 12:20:16.718395 31867 tablet_bootstrap.cc:492] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:16.719754 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:16.720674 31874 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:16.714005 31868 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.721148 31874 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:16.721812 31868 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.723991 31868 ts_tablet_manager.cc:1434] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.019s	user 0.005s	sys 0.000s
I20260709 12:20:16.708792 31866 raft_consensus.cc:359] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.709666 31872 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.728624 31872 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:16.722529 31874 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:16.730445 31867 tablet_bootstrap.cc:654] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.731032 31868 tablet_bootstrap.cc:492] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:16.731361 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:20:16.732873 31643 leader_election.cc:343] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.735591 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:16.736218 31752 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.737475 31868 tablet_bootstrap.cc:654] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.744927 31866 raft_consensus.cc:385] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.745565 31866 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.746762 31866 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.749830 31866 ts_tablet_manager.cc:1434] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.044s	user 0.005s	sys 0.000s
I20260709 12:20:16.751371 31866 tablet_bootstrap.cc:492] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:16.752529 31926 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.753060 31926 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.755544 31926 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:16.758353 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:16.758780 31866 tablet_bootstrap.cc:654] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.759766 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:16.761745 31791 leader_election.cc:343] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:16.762805 31791 leader_election.cc:343] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.763403 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:16.765012 31926 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:16.768314 31752 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:16.770154 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:16.771643 31872 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:16.772186 31872 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:16.772729 31868 tablet_bootstrap.cc:492] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:16.773510 31868 ts_tablet_manager.cc:1403] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.043s	user 0.024s	sys 0.016s
I20260709 12:20:16.773099 31872 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.772701 31574 catalog_manager.cc:5697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:16.774987 31574 catalog_manager.cc:5697] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.777240 31868 raft_consensus.cc:359] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:16.778182 31868 raft_consensus.cc:385] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.778538 31868 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.779412 31868 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:16.789316 31868 ts_tablet_manager.cc:1434] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 12:20:16.790693 31868 tablet_bootstrap.cc:492] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:16.791481 31922 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.792039 31922 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.796113 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:16.796886 31827 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:16.798686 31868 tablet_bootstrap.cc:654] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.798930 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:16.799675 31752 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:16.801658 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:16.802696 31891 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:16.803092 31891 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.803449 31891 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.804163 31867 tablet_bootstrap.cc:492] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:16.804668 31867 ts_tablet_manager.cc:1403] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.087s	user 0.032s	sys 0.000s
I20260709 12:20:16.807727 31867 raft_consensus.cc:359] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.808487 31867 raft_consensus.cc:385] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.808822 31867 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.809281 31922 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:16.809608 31867 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.814761 31891 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.827329 31867 ts_tablet_manager.cc:1434] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.022s	user 0.009s	sys 0.000s
I20260709 12:20:16.827621 31868 tablet_bootstrap.cc:492] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:16.826963 31574 catalog_manager.cc:5697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.827919 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:16.828179 31868 ts_tablet_manager.cc:1403] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.038s	user 0.029s	sys 0.007s
I20260709 12:20:16.828505 31867 tablet_bootstrap.cc:492] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:16.828603 31752 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.830276 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:16.830904 31827 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.831796 31868 raft_consensus.cc:359] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.832607 31868 raft_consensus.cc:385] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.832988 31868 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.833864 31868 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.835947 31867 tablet_bootstrap.cc:654] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.836618 31868 ts_tablet_manager.cc:1434] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:16.824466 31866 tablet_bootstrap.cc:492] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:16.840523 31866 ts_tablet_manager.cc:1403] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.090s	user 0.043s	sys 0.019s
I20260709 12:20:16.841527 31752 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:16.844368 31868 tablet_bootstrap.cc:492] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:16.847906 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:16.849263 31872 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:16.849952 31872 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:16.851497 31868 tablet_bootstrap.cc:654] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.851152 31872 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.841928 31891 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:16.857215 31827 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:16.862222 31866 raft_consensus.cc:359] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.863277 31866 raft_consensus.cc:385] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.863641 31866 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.864859 31866 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.868628 31868 tablet_bootstrap.cc:492] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:16.869246 31868 ts_tablet_manager.cc:1403] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.025s	user 0.018s	sys 0.005s
I20260709 12:20:16.872430 31868 raft_consensus.cc:359] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.873554 31868 raft_consensus.cc:385] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.873890 31868 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.875062 31868 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.876842 31866 ts_tablet_manager.cc:1434] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.036s	user 0.007s	sys 0.002s
I20260709 12:20:16.877228 31868 ts_tablet_manager.cc:1434] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:20:16.875488 31574 catalog_manager.cc:5697] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:16.880908 31868 tablet_bootstrap.cc:492] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:16.884236 31867 tablet_bootstrap.cc:492] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:16.884793 31867 ts_tablet_manager.cc:1403] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.057s	user 0.032s	sys 0.010s
I20260709 12:20:16.888098 31868 tablet_bootstrap.cc:654] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.889166 31866 tablet_bootstrap.cc:492] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:16.887683 31867 raft_consensus.cc:359] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.892175 31867 raft_consensus.cc:385] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.892553 31867 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.893316 31867 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.894673 31868 tablet_bootstrap.cc:492] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:16.895107 31926 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.895653 31867 ts_tablet_manager.cc:1434] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
I20260709 12:20:16.895644 31926 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.896735 31867 tablet_bootstrap.cc:492] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:16.897969 31866 tablet_bootstrap.cc:654] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.895196 31868 ts_tablet_manager.cc:1403] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.015s	user 0.013s	sys 0.000s
I20260709 12:20:16.899848 31874 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.900390 31874 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.903105 31868 raft_consensus.cc:359] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.903782 31868 raft_consensus.cc:385] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.904062 31868 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.904795 31868 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.907301 31867 tablet_bootstrap.cc:654] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.907437 31868 ts_tablet_manager.cc:1434] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:20:16.907652 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:16.908447 31868 tablet_bootstrap.cc:492] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:16.908648 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:16.910356 31791 leader_election.cc:343] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.911026 31874 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:16.915918 31866 tablet_bootstrap.cc:492] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
W20260709 12:20:16.912261 31791 leader_election.cc:343] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.918465 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:16.919198 31752 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:16.920284 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:16.921347 31874 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.921870 31874 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.924252 31874 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:16.924954 31867 tablet_bootstrap.cc:492] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:16.925396 31867 ts_tablet_manager.cc:1403] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.029s	user 0.011s	sys 0.009s
I20260709 12:20:16.926220 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:16.926471 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:16.927174 31677 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
W20260709 12:20:16.927469 31791 leader_election.cc:343] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.927963 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:16.928309 31867 raft_consensus.cc:359] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.928900 31928 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:16.929062 31867 raft_consensus.cc:385] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.928933 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:16.929402 31928 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.929499 31867 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.930076 31928 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.930821 31874 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:16.930604 31867 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.931517 31868 tablet_bootstrap.cc:654] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.930603 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:16.911733 31926 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:16.933705 31866 ts_tablet_manager.cc:1403] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.045s	user 0.011s	sys 0.007s
I20260709 12:20:16.937527 31866 raft_consensus.cc:359] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.939244 31928 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.940202 31874 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:16.940788 31874 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:16.941265 31874 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.941929 31928 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:20:16.939576 31791 leader_election.cc:343] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.942691 31866 raft_consensus.cc:385] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.942824 31867 ts_tablet_manager.cc:1434] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.017s	user 0.008s	sys 0.001s
I20260709 12:20:16.943585 31866 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.944526 31867 tablet_bootstrap.cc:492] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:16.944298 31866 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.946292 31866 ts_tablet_manager.cc:1434] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:20:16.951752 31867 tablet_bootstrap.cc:654] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:16.953980 31624 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:16.960819 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:16.962997 31866 tablet_bootstrap.cc:492] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:16.966034 31867 tablet_bootstrap.cc:492] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:16.966418 31867 ts_tablet_manager.cc:1403] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.022s	user 0.012s	sys 0.007s
W20260709 12:20:16.967814 31791 leader_election.cc:343] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:16.967729 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:16.968583 31751 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:16.969164 31867 raft_consensus.cc:359] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.969863 31867 raft_consensus.cc:385] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.970144 31867 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.970871 31867 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.973414 31868 tablet_bootstrap.cc:492] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:16.973876 31868 ts_tablet_manager.cc:1403] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.066s	user 0.025s	sys 0.019s
I20260709 12:20:16.976027 31867 ts_tablet_manager.cc:1434] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:20:16.980860 31866 tablet_bootstrap.cc:654] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.977008 31867 tablet_bootstrap.cc:492] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:16.982800 31751 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:16.984133 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:16.985020 31928 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:16.984898 31868 raft_consensus.cc:359] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.985586 31928 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:16.985831 31868 raft_consensus.cc:385] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:16.986255 31868 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:16.987167 31905 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:16.987078 31868 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.987663 31905 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:16.988497 31867 tablet_bootstrap.cc:654] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.989826 31868 ts_tablet_manager.cc:1434] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 12:20:16.989907 31905 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:16.992928 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:16.993624 31827 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:16.994141 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:16.994930 31868 tablet_bootstrap.cc:492] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:16.995036 31677 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:17.002149 31868 tablet_bootstrap.cc:654] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:16.995486 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:17.006907 31905 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.007407 31905 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.007911 31905 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.012168 31872 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.012707 31872 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.014994 31872 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:16.986653 31928 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.018940 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:17.018965 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:17.019969 31752 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:17.021045 31867 tablet_bootstrap.cc:492] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:17.021528 31867 ts_tablet_manager.cc:1403] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.045s	user 0.017s	sys 0.021s
W20260709 12:20:17.021471 31643 leader_election.cc:343] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.021979 31866 tablet_bootstrap.cc:492] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:17.022119 31677 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.022437 31866 ts_tablet_manager.cc:1403] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.074s	user 0.012s	sys 0.006s
I20260709 12:20:17.023078 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:17.025249 31866 raft_consensus.cc:359] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.026046 31866 raft_consensus.cc:385] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.025749 31867 raft_consensus.cc:359] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.026304 31866 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.026393 31867 raft_consensus.cc:385] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.026678 31867 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.027015 31866 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.027323 31867 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.028365 31891 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.028825 31891 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.029191 31891 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.030402 31867 ts_tablet_manager.cc:1434] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:20:17.031713 31867 tablet_bootstrap.cc:492] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:17.030402 31866 ts_tablet_manager.cc:1434] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.002s
I20260709 12:20:17.032888 31866 tablet_bootstrap.cc:492] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:17.038115 31867 tablet_bootstrap.cc:654] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.028909 31827 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:17.041136 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57099.7us
I20260709 12:20:17.038492 31866 tablet_bootstrap.cc:654] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.053136 31891 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.071065 31891 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:17.066259 31874 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.072460 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:17.074661 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:17.075278 31678 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.077495 31874 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:17.079851 31873 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:17.080152 31905 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:17.081790 31643 leader_election.cc:343] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.083561 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:17.084267 31676 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.084542 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:17.085134 31751 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.087798 31891 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.091496 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:17.087380 31873 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.092762 31873 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.094851 31752 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.088835 31891 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.102908 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:17.103614 31826 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:17.103776 31873 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:17.102916 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:17.109707 31891 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:17.112936 31642 leader_election.cc:343] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.113731 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:17.122105 31751 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:17.132431 31891 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.139104 31891 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.139356 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:17.140419 31825 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.139119 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:17.141589 31826 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 1: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:17.142216 31676 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:17.139562 31891 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.140727 31678 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:17.171319 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:17.174124 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:17.174467 31922 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:17.156447 31905 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:17.176410 31928 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:17.177794 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:17.179481 31676 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 1: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:17.174973 31922 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:17.179453 31928 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:17.175980 31825 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:17.183023 31928 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.183348 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:17.184396 31905 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:17.184266 31923 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:17.177146 31752 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:17.184899 31905 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:17.185797 31905 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.186718 31922 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.191121 31866 tablet_bootstrap.cc:492] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:17.191550 31866 ts_tablet_manager.cc:1403] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.159s	user 0.014s	sys 0.006s
I20260709 12:20:17.189829 31574 catalog_manager.cc:5697] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:17.204391 31868 tablet_bootstrap.cc:492] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:17.204922 31868 ts_tablet_manager.cc:1403] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.212s	user 0.038s	sys 0.013s
W20260709 12:20:17.206648 31643 consensus_peers.cc:597] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:17.208750 31867 tablet_bootstrap.cc:492] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:17.209225 31867 ts_tablet_manager.cc:1403] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.178s	user 0.036s	sys 0.073s
I20260709 12:20:17.210826 31866 raft_consensus.cc:359] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.211416 31866 raft_consensus.cc:385] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.211642 31866 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.212208 31866 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.219457 31868 raft_consensus.cc:359] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.220305 31868 raft_consensus.cc:385] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.220678 31868 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.221570 31868 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.215004 31956 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260709 12:20:17.223014 31891 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.226333 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:17.231074 31866 ts_tablet_manager.cc:1434] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.039s	user 0.007s	sys 0.001s
I20260709 12:20:17.231886 31866 tablet_bootstrap.cc:492] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:17.229972 31874 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:20:17.240545 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:17.241290 31824 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.227531 31717 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:17.245119 31642 leader_election.cc:343] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.246038 31905 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:17.250495 31867 raft_consensus.cc:359] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.251642 31867 raft_consensus.cc:385] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.252014 31867 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.252128 31576 catalog_manager.cc:5697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:17.253194 31867 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.255491 31753 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.270740 31873 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.266031 31891 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:17.273217 31873 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.274678 31868 ts_tablet_manager.cc:1434] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.069s	user 0.027s	sys 0.018s
I20260709 12:20:17.276405 31868 tablet_bootstrap.cc:492] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:17.277954 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:17.282403 31867 ts_tablet_manager.cc:1434] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.073s	user 0.009s	sys 0.001s
I20260709 12:20:17.277998 31873 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:17.285804 31717 leader_election.cc:343] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.284179 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:17.287492 31718 leader_election.cc:343] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.287936 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:17.288718 31956 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:17.288937 31928 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:17.289376 31905 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.289947 31905 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:17.291062 31867 tablet_bootstrap.cc:492] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:17.292876 31905 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:17.296641 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:17.298103 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:17.299557 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:17.299862 31824 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:17.302527 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:17.304771 31891 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:17.307546 31891 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:17.309373 31891 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.313079 31874 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:17.323169 31874 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:17.325441 31928 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.326840 31576 catalog_manager.cc:5697] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:17.331120 31867 tablet_bootstrap.cc:654] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.333326 31905 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.333861 31905 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.336150 31905 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:17.326027 31928 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.338325 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:17.339080 31826 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:17.340516 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:17.340873 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:17.341576 31905 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.340989 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:17.341614 31676 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:17.342288 31905 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.350561 31905 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.350836 31922 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:17.344798 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:17.357741 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:17.342401 31575 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:17.358875 31874 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:17.370779 31866 tablet_bootstrap.cc:654] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.341782 31928 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:17.406538 31868 tablet_bootstrap.cc:654] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.450750 31905 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.454718 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:17.455632 31676 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.512969 31905 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:17.512781 31956 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:17.520393 31891 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.521593 31922 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:20:17.524401 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:17.527420 31891 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.572621 31891 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:17.540874 31824 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.580263 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:17.581759 31866 tablet_bootstrap.cc:492] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:17.582285 31866 ts_tablet_manager.cc:1403] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.351s	user 0.035s	sys 0.023s
I20260709 12:20:17.582535 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:17.587464 31643 leader_election.cc:343] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.587816 31866 raft_consensus.cc:359] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.588617 31866 raft_consensus.cc:385] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:20:17.588778 31970 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:20:17.588898 31866 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.584620 31575 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:17.589692 31866 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.591235 31676 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:17.592002 31866 ts_tablet_manager.cc:1434] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:20:17.592015 31905 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:17.592417 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:17.593052 31866 tablet_bootstrap.cc:492] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:17.593403 31956 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:17.593986 31956 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:17.594967 31956 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.599680 31891 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.600533 31891 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.602305 31824 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:17.603256 31867 tablet_bootstrap.cc:492] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:17.603699 31867 ts_tablet_manager.cc:1403] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.313s	user 0.013s	sys 0.007s
W20260709 12:20:17.599825 31642 leader_election.cc:343] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.610313 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:17.655418 31922 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:17.660820 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 59996.9us
I20260709 12:20:17.669569 31867 raft_consensus.cc:359] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.674130 31867 raft_consensus.cc:385] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.674465 31867 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.675261 31867 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.677652 31867 ts_tablet_manager.cc:1434] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.074s	user 0.009s	sys 0.005s
I20260709 12:20:17.678611 31867 tablet_bootstrap.cc:492] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
W20260709 12:20:17.680222 31791 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:17.686424 31643 consensus_peers.cc:597] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:17.688562 31928 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:17.689782 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:17.690587 31956 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:20:17.691293 31642 leader_election.cc:343] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.664563 31891 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:17.699548 31868 tablet_bootstrap.cc:492] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:17.704391 31868 ts_tablet_manager.cc:1403] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.429s	user 0.033s	sys 0.044s
I20260709 12:20:17.709978 31866 tablet_bootstrap.cc:654] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.706580 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:17.712565 31928 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:20:17.719823 31824 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:17.734050 31926 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:17.740104 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:17.741842 31922 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.742295 31922 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.742707 31922 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.747324 31868 raft_consensus.cc:359] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.748432 31868 raft_consensus.cc:385] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.748819 31868 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.749972 31868 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.753012 31868 ts_tablet_manager.cc:1434] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.048s	user 0.008s	sys 0.000s
I20260709 12:20:17.754074 31868 tablet_bootstrap.cc:492] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:17.755757 31903 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:17.756278 31922 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.759586 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:17.760263 31750 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.760345 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:17.761060 31824 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.758390 31922 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:17.765200 31866 tablet_bootstrap.cc:492] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:17.765709 31866 ts_tablet_manager.cc:1403] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.173s	user 0.012s	sys 0.002s
I20260709 12:20:17.771600 31867 tablet_bootstrap.cc:654] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.778406 31750 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:17.780146 31642 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:17.781418 31903 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:17.779247 31576 catalog_manager.cc:5697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:17.783337 31866 raft_consensus.cc:359] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.783991 31866 raft_consensus.cc:385] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.784256 31866 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.784842 31903 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:17.786466 31903 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.786716 31866 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.792621 31866 ts_tablet_manager.cc:1434] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.027s	user 0.008s	sys 0.000s
I20260709 12:20:17.795951 31866 tablet_bootstrap.cc:492] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:17.801388 31827 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.816774 31868 tablet_bootstrap.cc:654] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.820492 31826 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.821403 31824 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:17.828236 31866 tablet_bootstrap.cc:654] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.850133 31903 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.850735 31903 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.850646 31574 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:17.861884 31903 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:17.863013 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:17.863178 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:17.865195 31642 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:17.866364 31903 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:17.869328 31642 consensus_peers.cc:597] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:17.872947 31868 tablet_bootstrap.cc:492] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:17.873528 31868 ts_tablet_manager.cc:1403] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.120s	user 0.012s	sys 0.010s
I20260709 12:20:17.876441 31868 raft_consensus.cc:359] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.877216 31868 raft_consensus.cc:385] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.877498 31868 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.878355 31868 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.881659 31868 ts_tablet_manager.cc:1434] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:20:17.882702 31868 tablet_bootstrap.cc:492] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:17.888983 31868 tablet_bootstrap.cc:654] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.890215 31866 tablet_bootstrap.cc:492] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:17.890909 31866 ts_tablet_manager.cc:1403] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.095s	user 0.007s	sys 0.019s
I20260709 12:20:17.894330 31866 raft_consensus.cc:359] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.895169 31866 raft_consensus.cc:385] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.895495 31866 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.897040 31867 tablet_bootstrap.cc:492] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:17.898103 31867 ts_tablet_manager.cc:1403] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.220s	user 0.017s	sys 0.006s
I20260709 12:20:17.900166 31866 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.902863 31866 ts_tablet_manager.cc:1434] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.001s
I20260709 12:20:17.903919 31866 tablet_bootstrap.cc:492] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:17.911979 31867 raft_consensus.cc:359] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.912724 31867 raft_consensus.cc:385] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.913017 31867 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.913815 31867 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.915235 31866 tablet_bootstrap.cc:654] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.912601 31868 tablet_bootstrap.cc:492] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:17.916230 31867 ts_tablet_manager.cc:1434] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.018s	user 0.000s	sys 0.006s
I20260709 12:20:17.916524 31868 ts_tablet_manager.cc:1403] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.034s	user 0.013s	sys 0.017s
I20260709 12:20:17.917197 31867 tablet_bootstrap.cc:492] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:17.920418 31868 raft_consensus.cc:359] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.921679 31868 raft_consensus.cc:385] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.922179 31868 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.923878 31867 tablet_bootstrap.cc:654] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.923545 31868 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.927371 31868 ts_tablet_manager.cc:1434] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.002s
I20260709 12:20:17.928987 31868 tablet_bootstrap.cc:492] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:17.930330 31922 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:17.939728 31868 tablet_bootstrap.cc:654] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.943197 31866 tablet_bootstrap.cc:492] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:17.943819 31866 ts_tablet_manager.cc:1403] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.040s	user 0.022s	sys 0.012s
I20260709 12:20:17.947553 31866 raft_consensus.cc:359] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.948673 31866 raft_consensus.cc:385] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.949079 31866 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.950351 31866 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.967514 31926 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.968056 31926 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.969334 31866 ts_tablet_manager.cc:1434] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.025s	user 0.010s	sys 0.000s
I20260709 12:20:17.970429 31866 tablet_bootstrap.cc:492] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:17.972833 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:17.973616 31677 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:17.974315 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:17.975241 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:17.975744 31926 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:17.963481 31867 tablet_bootstrap.cc:492] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
W20260709 12:20:17.976300 31791 leader_election.cc:343] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:17.976630 31867 ts_tablet_manager.cc:1403] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.060s	user 0.015s	sys 0.000s
I20260709 12:20:17.976315 31926 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:17.977553 31926 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:17.977926 31866 tablet_bootstrap.cc:654] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.977938 31926 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:17.979844 31867 raft_consensus.cc:359] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.980548 31867 raft_consensus.cc:385] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.980816 31867 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.981519 31867 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.984594 31868 tablet_bootstrap.cc:492] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:17.985152 31868 ts_tablet_manager.cc:1403] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.056s	user 0.021s	sys 0.017s
I20260709 12:20:17.987377 31867 ts_tablet_manager.cc:1434] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:20:17.988449 31867 tablet_bootstrap.cc:492] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:17.988232 31868 raft_consensus.cc:359] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.989025 31868 raft_consensus.cc:385] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:17.989180 31928 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:17.989382 31868 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:17.989713 31928 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.990306 31868 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:17.992363 31928 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:17.993317 31868 ts_tablet_manager.cc:1434] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:20:17.994354 31868 tablet_bootstrap.cc:492] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:17.995010 31867 tablet_bootstrap.cc:654] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:17.995136 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:17.996824 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:17.997591 31677 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:17.999027 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:20:18.000018 31791 leader_election.cc:343] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.000705 31868 tablet_bootstrap.cc:654] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.000968 31928 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.001338 31928 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.001735 31928 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.004329 31866 tablet_bootstrap.cc:492] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:18.004801 31866 ts_tablet_manager.cc:1403] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.035s	user 0.009s	sys 0.002s
I20260709 12:20:18.009598 31926 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.013267 31866 raft_consensus.cc:359] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.015580 31866 raft_consensus.cc:385] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:20:18.014261 31624 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:18.016132 31866 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.017498 31866 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.019333 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:18.019999 31867 tablet_bootstrap.cc:492] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:18.020098 31677 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.020468 31866 ts_tablet_manager.cc:1434] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.003s
I20260709 12:20:18.020557 31867 ts_tablet_manager.cc:1403] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.032s	user 0.019s	sys 0.011s
I20260709 12:20:18.021682 31866 tablet_bootstrap.cc:492] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:18.021991 31868 tablet_bootstrap.cc:492] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:18.022576 31868 ts_tablet_manager.cc:1403] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.028s	user 0.004s	sys 0.008s
I20260709 12:20:18.023881 31867 raft_consensus.cc:359] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.024693 31867 raft_consensus.cc:385] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.025059 31867 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.025810 31867 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.026044 31868 raft_consensus.cc:359] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.026973 31868 raft_consensus.cc:385] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.027374 31868 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.028211 31866 tablet_bootstrap.cc:654] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.027885 31928 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.028443 31868 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.034931 31867 ts_tablet_manager.cc:1434] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.014s	user 0.004s	sys 0.003s
I20260709 12:20:18.036213 31867 tablet_bootstrap.cc:492] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:18.036474 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:18.036967 31926 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.037276 31752 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.037792 31677 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:18.039296 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:18.041782 31926 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.041879 31868 ts_tablet_manager.cc:1434] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:20:18.041949 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:18.043285 31676 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.043562 31956 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.044425 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:18.044170 31956 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.045092 31751 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.043383 31868 tablet_bootstrap.cc:492] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:18.047432 31928 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.048626 31956 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.050259 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:18.055217 31752 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:18.055874 31926 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:18.061301 31751 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:18.058414 31926 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.067687 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:18.068308 31867 tablet_bootstrap.cc:654] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.068069 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:18.069206 31928 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.069304 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:18.069908 31874 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.073462 31956 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:18.073897 31874 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.060101 31676 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:18.076840 31868 tablet_bootstrap.cc:654] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.078943 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:18.081050 31866 tablet_bootstrap.cc:492] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:18.081595 31866 ts_tablet_manager.cc:1403] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.060s	user 0.019s	sys 0.005s
I20260709 12:20:18.081677 31874 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.082945 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:18.084641 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:18.085434 31826 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.085474 31866 raft_consensus.cc:359] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.086931 31866 raft_consensus.cc:385] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.087343 31866 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.088171 31866 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.093480 31866 ts_tablet_manager.cc:1434] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.011s	user 0.009s	sys 0.000s
I20260709 12:20:18.094592 31866 tablet_bootstrap.cc:492] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:18.097651 31576 catalog_manager.cc:5697] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:18.102512 31676 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.103256 31891 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.103729 31891 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.105871 31891 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.102100 31866 tablet_bootstrap.cc:654] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.102159 31867 tablet_bootstrap.cc:492] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:18.106734 31867 ts_tablet_manager.cc:1403] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.071s	user 0.006s	sys 0.011s
I20260709 12:20:18.107982 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:18.108721 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:18.109350 31827 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
W20260709 12:20:18.109686 31642 leader_election.cc:343] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.109818 31867 raft_consensus.cc:359] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.110669 31867 raft_consensus.cc:385] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.110793 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:18.111014 31867 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.100440 31928 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:18.112084 31891 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.111893 31867 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.112929 31891 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.113392 31891 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.114307 31867 ts_tablet_manager.cc:1434] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:20:18.117213 31867 tablet_bootstrap.cc:492] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:18.120865 31868 tablet_bootstrap.cc:492] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:18.121497 31868 ts_tablet_manager.cc:1403] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.078s	user 0.010s	sys 0.007s
I20260709 12:20:18.112748 31928 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.134444 31891 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.135375 31868 raft_consensus.cc:359] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.136207 31868 raft_consensus.cc:385] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.136595 31868 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.137921 31868 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.139235 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:20:18.141218 31642 leader_election.cc:343] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.144186 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:18.144845 31827 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.145048 31867 tablet_bootstrap.cc:654] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.147342 31868 ts_tablet_manager.cc:1434] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.026s	user 0.008s	sys 0.001s
I20260709 12:20:18.148479 31868 tablet_bootstrap.cc:492] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:18.149770 31956 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.162191 31956 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.162842 31956 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.166033 31956 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.166157 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:18.168258 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:18.168831 31718 leader_election.cc:343] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.169162 31676 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:18.171880 31717 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:18.173348 31956 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.173540 31868 tablet_bootstrap.cc:654] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.173805 31956 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.174320 31956 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.170392 31827 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:18.180641 31866 tablet_bootstrap.cc:492] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:18.181250 31866 ts_tablet_manager.cc:1403] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.087s	user 0.025s	sys 0.044s
I20260709 12:20:18.180987 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:18.182214 31922 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.182852 31922 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:18.183843 31922 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.184931 31866 raft_consensus.cc:359] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.185817 31866 raft_consensus.cc:385] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.186161 31866 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.187294 31866 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.191759 31868 tablet_bootstrap.cc:492] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:18.191879 31866 ts_tablet_manager.cc:1434] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:20:18.193056 31866 tablet_bootstrap.cc:492] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:18.196991 31956 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.199043 31868 ts_tablet_manager.cc:1403] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.051s	user 0.009s	sys 0.006s
I20260709 12:20:18.199828 31956 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.200371 31891 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.200726 31866 tablet_bootstrap.cc:654] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.200994 31867 tablet_bootstrap.cc:492] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:18.201486 31867 ts_tablet_manager.cc:1403] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.085s	user 0.012s	sys 0.010s
I20260709 12:20:18.203215 31868 raft_consensus.cc:359] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.203962 31868 raft_consensus.cc:385] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.204171 31867 raft_consensus.cc:359] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.204996 31867 raft_consensus.cc:385] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.205389 31867 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.206254 31867 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.207877 31874 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:18.208334 31867 ts_tablet_manager.cc:1434] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:20:18.209311 31867 tablet_bootstrap.cc:492] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:18.212946 31962 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:18.215591 31867 tablet_bootstrap.cc:654] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.216892 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:18.217536 31676 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.217308 31575 catalog_manager.cc:5697] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:18.231230 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:20:18.233029 31718 leader_election.cc:343] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.204254 31868 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.236866 31868 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.239713 31868 ts_tablet_manager.cc:1434] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.040s	user 0.007s	sys 0.000s
I20260709 12:20:18.239420 31872 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.245805 31868 tablet_bootstrap.cc:492] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:18.251603 31751 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.251971 31868 tablet_bootstrap.cc:654] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.259657 31575 catalog_manager.cc:5697] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:18.262550 31676 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:18.268230 31717 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:18.271667 31973 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.272215 31973 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:18.273078 31973 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.274940 31866 tablet_bootstrap.cc:492] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:18.275681 31866 ts_tablet_manager.cc:1403] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.083s	user 0.020s	sys 0.022s
I20260709 12:20:18.279623 31866 raft_consensus.cc:359] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.280670 31866 raft_consensus.cc:385] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.281068 31866 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.282094 31866 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.285396 31866 ts_tablet_manager.cc:1434] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:20:18.286968 31866 tablet_bootstrap.cc:492] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:18.298056 31866 tablet_bootstrap.cc:654] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.312450 31867 tablet_bootstrap.cc:492] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:18.313124 31867 ts_tablet_manager.cc:1403] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.104s	user 0.010s	sys 0.012s
I20260709 12:20:18.322494 31868 tablet_bootstrap.cc:492] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:18.323247 31868 ts_tablet_manager.cc:1403] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.078s	user 0.029s	sys 0.026s
I20260709 12:20:18.326159 31891 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.326315 31868 raft_consensus.cc:359] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.327095 31868 raft_consensus.cc:385] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.327412 31868 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.330220 31866 tablet_bootstrap.cc:492] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:18.330772 31866 ts_tablet_manager.cc:1403] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.044s	user 0.023s	sys 0.000s
I20260709 12:20:18.332435 31868 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.335027 31868 ts_tablet_manager.cc:1434] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.001s
I20260709 12:20:18.338042 31866 raft_consensus.cc:359] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.339044 31866 raft_consensus.cc:385] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.339423 31866 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.340216 31866 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.341743 31868 tablet_bootstrap.cc:492] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:18.342594 31866 ts_tablet_manager.cc:1434] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:20:18.348248 31866 tablet_bootstrap.cc:492] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:18.348950 31868 tablet_bootstrap.cc:654] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.351257 31867 raft_consensus.cc:359] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.352413 31867 raft_consensus.cc:385] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.352804 31867 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.353652 31867 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.355455 31866 tablet_bootstrap.cc:654] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.355496 31867 ts_tablet_manager.cc:1434] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.042s	user 0.006s	sys 0.001s
I20260709 12:20:18.361078 31867 tablet_bootstrap.cc:492] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:18.365584 31575 catalog_manager.cc:5697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:18.372524 31928 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.373163 31928 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.375844 31928 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.377485 31867 tablet_bootstrap.cc:654] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.378216 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:18.379212 31676 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:18.381973 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:18.382845 31751 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:18.385463 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:18.387828 31874 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.388273 31874 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.388638 31874 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.393692 31866 tablet_bootstrap.cc:492] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:18.394320 31866 ts_tablet_manager.cc:1403] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.046s	user 0.014s	sys 0.000s
I20260709 12:20:18.400651 31866 raft_consensus.cc:359] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.401544 31866 raft_consensus.cc:385] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.402052 31866 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.402896 31866 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.403555 31868 tablet_bootstrap.cc:492] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:18.404021 31868 ts_tablet_manager.cc:1403] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.063s	user 0.017s	sys 0.016s
I20260709 12:20:18.401301 31874 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.407542 31874 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.409085 31866 ts_tablet_manager.cc:1434] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:20:18.412086 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:18.412710 31676 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.413816 31866 tablet_bootstrap.cc:492] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:18.413537 31868 raft_consensus.cc:359] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.414183 31868 raft_consensus.cc:385] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.414451 31868 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.415078 31868 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.415575 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:18.416066 31926 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.416256 31751 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.416672 31926 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.417769 31868 ts_tablet_manager.cc:1434] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:20:18.418767 31868 tablet_bootstrap.cc:492] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:18.420259 31926 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.421118 31866 tablet_bootstrap.cc:654] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.424129 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:18.424885 31752 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:18.425045 31868 tablet_bootstrap.cc:654] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.427671 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:18.429328 31791 leader_election.cc:343] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.430107 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:18.431418 31926 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.431797 31926 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.432150 31926 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.436995 31973 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.437551 31973 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.441701 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:18.443240 31717 leader_election.cc:343] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.444715 31973 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.445879 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:18.452797 31826 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:18.453250 31867 tablet_bootstrap.cc:492] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:18.453869 31867 ts_tablet_manager.cc:1403] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.093s	user 0.005s	sys 0.008s
I20260709 12:20:18.454285 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:18.455911 31973 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.456425 31973 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.456866 31973 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.448932 31676 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:18.458444 31891 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.456998 31867 raft_consensus.cc:359] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.459074 31891 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.459789 31867 raft_consensus.cc:385] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.459827 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:18.460175 31867 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.460803 31874 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.461732 31874 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:18.461489 31867 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.447885 31926 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.462610 31874 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.464269 31926 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.464172 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:18.464944 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:18.465406 31891 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:18.466418 31643 leader_election.cc:343] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.467664 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:18.468192 31866 tablet_bootstrap.cc:492] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:18.468116 31973 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.468734 31866 ts_tablet_manager.cc:1403] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.055s	user 0.020s	sys 0.014s
W20260709 12:20:18.468989 31791 leader_election.cc:343] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:18.469075 31642 leader_election.cc:343] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.469643 31642 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:18.471230 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:18.471818 31676 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.471918 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:18.472599 31868 tablet_bootstrap.cc:492] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:18.472591 31753 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.472898 31891 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:18.473277 31868 ts_tablet_manager.cc:1403] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.055s	user 0.019s	sys 0.014s
I20260709 12:20:18.473770 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:18.474357 31826 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.476634 31868 raft_consensus.cc:359] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.477347 31868 raft_consensus.cc:385] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.477600 31868 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.478271 31868 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.472725 31973 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.480741 31753 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:18.481276 31826 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:18.482157 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:18.482587 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:18.483189 31926 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.483770 31926 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:18.484146 31973 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.484762 31926 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.486523 31866 raft_consensus.cc:359] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.487377 31866 raft_consensus.cc:385] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.487703 31866 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.484751 31973 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:18.488512 31866 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.489259 31676 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:18.472628 31867 ts_tablet_manager.cc:1434] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.003s
I20260709 12:20:18.489113 31973 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.490576 31867 tablet_bootstrap.cc:492] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:18.481323 31868 ts_tablet_manager.cc:1434] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:18.493351 31868 tablet_bootstrap.cc:492] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:18.496559 31866 ts_tablet_manager.cc:1434] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.003s
I20260709 12:20:18.497676 31866 tablet_bootstrap.cc:492] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:18.502792 31926 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.503505 31926 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.505107 31867 tablet_bootstrap.cc:654] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.505450 31868 tablet_bootstrap.cc:654] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.509545 31866 tablet_bootstrap.cc:654] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.506986 31926 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.507016 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:18.507594 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:18.511412 31791 leader_election.cc:343] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:18.512425 31791 leader_election.cc:343] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.512889 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:18.514065 31926 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:18.519450 31866 tablet_bootstrap.cc:492] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:18.520035 31866 ts_tablet_manager.cc:1403] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.023s	user 0.011s	sys 0.000s
I20260709 12:20:18.522559 31867 tablet_bootstrap.cc:492] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:18.523123 31867 ts_tablet_manager.cc:1403] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.033s	user 0.010s	sys 0.005s
I20260709 12:20:18.526433 31867 raft_consensus.cc:359] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.527221 31867 raft_consensus.cc:385] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.527540 31867 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.532351 31928 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:18.533614 31751 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:18.538302 31867 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.540911 31973 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.541388 31867 ts_tablet_manager.cc:1434] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.000s
I20260709 12:20:18.546833 31866 raft_consensus.cc:359] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.566722 31866 raft_consensus.cc:385] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.566841 31867 tablet_bootstrap.cc:492] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:18.567098 31866 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.567962 31866 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.568360 32004 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.570034 31891 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.570525 31891 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.572628 31891 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.574230 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:18.541478 31973 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:18.576407 31643 leader_election.cc:343] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.576538 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:18.577803 31866 ts_tablet_manager.cc:1434] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.057s	user 0.010s	sys 0.007s
I20260709 12:20:18.577836 31973 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.577673 31751 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:18.579108 31866 tablet_bootstrap.cc:492] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:18.578979 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:18.579651 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:18.579998 31827 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 1: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:20:18.581713 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:18.584028 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:18.565220 31576 catalog_manager.cc:5697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:18.585743 31891 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.586433 31891 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.586830 31973 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.586920 31891 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.574260 31867 tablet_bootstrap.cc:654] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.588063 31574 catalog_manager.cc:5697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:18.587304 31866 tablet_bootstrap.cc:654] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.616138 31576 catalog_manager.cc:5697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:18.555094 31928 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.064s
I20260709 12:20:18.634899 31868 tablet_bootstrap.cc:492] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:18.635517 31868 ts_tablet_manager.cc:1403] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.142s	user 0.033s	sys 0.026s
I20260709 12:20:18.639292 31868 raft_consensus.cc:359] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.640069 31868 raft_consensus.cc:385] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.640415 31868 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.639701 31962 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.641431 31868 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.642170 31962 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.643927 31868 ts_tablet_manager.cc:1434] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:18.645020 31868 tablet_bootstrap.cc:492] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:18.646158 31962 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.647035 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:18.650372 31717 leader_election.cc:343] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.652670 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:18.654207 31718 leader_election.cc:343] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.654935 31718 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:18.659910 32004 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:18.662078 31962 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:18.667330 31866 tablet_bootstrap.cc:492] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:18.667873 31866 ts_tablet_manager.cc:1403] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.089s	user 0.025s	sys 0.022s
I20260709 12:20:18.670943 31891 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.675858 31866 raft_consensus.cc:359] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.676644 31866 raft_consensus.cc:385] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.676923 31866 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.677780 31866 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.679582 31866 ts_tablet_manager.cc:1434] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.004s
I20260709 12:20:18.680538 31866 tablet_bootstrap.cc:492] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:18.682453 31973 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:18.685038 31867 tablet_bootstrap.cc:492] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:18.685508 31867 ts_tablet_manager.cc:1403] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.119s	user 0.027s	sys 0.021s
I20260709 12:20:18.686510 31923 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:18.687271 31874 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.687813 31874 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.690052 31866 tablet_bootstrap.cc:654] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.692456 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:18.693024 31868 tablet_bootstrap.cc:654] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.693234 31677 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:18.695344 31891 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.689999 31874 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.696691 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:18.697322 31827 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.697948 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:18.698552 31751 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.700815 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:18.701499 31752 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:18.698156 31867 raft_consensus.cc:359] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.702008 31867 raft_consensus.cc:385] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.702391 31867 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.703410 31867 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.705093 31874 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.705690 31874 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.708457 31874 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.711869 31867 ts_tablet_manager.cc:1434] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.026s	user 0.010s	sys 0.000s
I20260709 12:20:18.715204 31867 tablet_bootstrap.cc:492] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:18.718866 31827 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:18.720914 31751 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:18.712385 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:18.722994 31926 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.727510 31926 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.727666 31866 tablet_bootstrap.cc:492] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:18.727993 31926 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.728493 31866 ts_tablet_manager.cc:1403] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.048s	user 0.021s	sys 0.019s
I20260709 12:20:18.723224 31923 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.729733 31867 tablet_bootstrap.cc:654] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.729799 31923 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.731655 31866 raft_consensus.cc:359] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.713873 31868 tablet_bootstrap.cc:492] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:18.732484 31866 raft_consensus.cc:385] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.724340 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:18.731850 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:18.726238 31874 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.732940 31868 ts_tablet_manager.cc:1403] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.088s	user 0.012s	sys 0.001s
I20260709 12:20:18.732885 31866 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.733386 31874 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.733979 31866 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.735838 31868 raft_consensus.cc:359] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.736529 31866 ts_tablet_manager.cc:1434] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:18.732899 31923 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.735867 31874 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.737955 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:18.736537 31868 raft_consensus.cc:385] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.738230 31923 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.738552 31868 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.737526 31677 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:18.739619 31868 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.738782 31678 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 1: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:18.740785 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:18.737535 31866 tablet_bootstrap.cc:492] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:18.742166 31874 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.742561 31874 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.742965 31874 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.743666 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:18.745615 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:18.747709 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:18.748265 31866 tablet_bootstrap.cc:654] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:18.749238 31642 leader_election.cc:343] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:18.750344 31791 leader_election.cc:343] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.748353 31868 ts_tablet_manager.cc:1434] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.000s
I20260709 12:20:18.754489 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:18.754990 31868 tablet_bootstrap.cc:492] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:18.755532 31923 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:18.756482 31923 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.759908 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:18.760885 32004 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.762473 31866 tablet_bootstrap.cc:492] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:18.763039 31866 ts_tablet_manager.cc:1403] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.026s	user 0.014s	sys 0.004s
I20260709 12:20:18.762881 31874 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.764977 31874 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.766263 31866 raft_consensus.cc:359] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.767092 31866 raft_consensus.cc:385] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.767419 31866 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.768077 31868 tablet_bootstrap.cc:654] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.768225 31866 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.768543 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:18.769323 31678 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.771981 32004 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:18.773020 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:18.774554 31866 ts_tablet_manager.cc:1434] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.000s
W20260709 12:20:18.774958 31791 leader_election.cc:343] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.775560 31866 tablet_bootstrap.cc:492] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:18.775893 31827 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 1: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:18.776643 31867 tablet_bootstrap.cc:492] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:18.777094 31867 ts_tablet_manager.cc:1403] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.062s	user 0.010s	sys 0.002s
I20260709 12:20:18.779126 31643 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:18.780454 31678 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:18.781235 31891 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:18.782238 31867 raft_consensus.cc:359] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.782985 31867 raft_consensus.cc:385] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.783257 31867 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.783942 31867 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.785288 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:18.786124 31867 ts_tablet_manager.cc:1434] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:20:18.786245 31874 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:18.786900 31874 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:18.787132 31867 tablet_bootstrap.cc:492] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:18.788581 31866 tablet_bootstrap.cc:654] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.788332 31874 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.794267 31867 tablet_bootstrap.cc:654] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.801154 31868 tablet_bootstrap.cc:492] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:18.801641 31868 ts_tablet_manager.cc:1403] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.047s	user 0.014s	sys 0.000s
I20260709 12:20:18.803922 31868 raft_consensus.cc:359] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.804627 31868 raft_consensus.cc:385] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.804950 31868 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.805750 31868 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.810012 31868 ts_tablet_manager.cc:1434] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:18.811102 31868 tablet_bootstrap.cc:492] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:18.812764 31922 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.816807 31574 catalog_manager.cc:5697] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:18.824168 31956 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.826102 31956 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.843199 31866 tablet_bootstrap.cc:492] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:18.843724 31866 ts_tablet_manager.cc:1403] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.068s	user 0.017s	sys 0.000s
I20260709 12:20:18.827562 31926 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.838937 31874 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.847518 31973 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:18.848100 31973 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.850193 31973 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.851392 31956 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:18.852732 31576 catalog_manager.cc:5697] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:18.852121 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:18.857602 31827 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:18.858090 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:18.861289 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:18.862183 31962 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.862596 31962 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.862995 31962 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.863996 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:18.865499 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:18.866240 31676 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
W20260709 12:20:18.866679 31717 leader_election.cc:343] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:18.855474 31874 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.869489 31956 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:20:18.883505 31866 raft_consensus.cc:359] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.884117 31866 raft_consensus.cc:385] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.884368 31866 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.888168 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:18.890268 31676 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.891604 31973 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:18.893679 32004 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:18.896780 31866 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.900141 31867 tablet_bootstrap.cc:492] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:18.900684 31867 ts_tablet_manager.cc:1403] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.114s	user 0.026s	sys 0.007s
I20260709 12:20:18.906838 31866 ts_tablet_manager.cc:1434] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.063s	user 0.008s	sys 0.001s
I20260709 12:20:18.907853 31866 tablet_bootstrap.cc:492] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:18.913581 31928 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:18.902779 31822 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:18.922739 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:18.924167 31956 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:18.924669 31956 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:18.925078 31956 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:18.932026 31874 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.920359 31926 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:18.943353 31868 tablet_bootstrap.cc:654] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.937757 31867 raft_consensus.cc:359] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.943996 31867 raft_consensus.cc:385] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:18.944417 31867 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:18.904345 31962 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.949579 31867 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:18.954485 31676 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:18.960160 31867 ts_tablet_manager.cc:1434] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.059s	user 0.001s	sys 0.007s
I20260709 12:20:18.960991 31867 tablet_bootstrap.cc:492] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:18.970799 31973 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:18.980785 31866 tablet_bootstrap.cc:654] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:18.981058 31973 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.002894 31962 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.003860 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:19.004753 31751 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.023947 31867 tablet_bootstrap.cc:654] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.024519 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:19.025746 31752 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:19.026131 31956 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.026978 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:19.027537 31676 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:19.029462 32054 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:20:19.032063 31868 tablet_bootstrap.cc:492] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:19.032569 31868 ts_tablet_manager.cc:1403] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.222s	user 0.020s	sys 0.005s
I20260709 12:20:19.033854 31678 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.031888 31956 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.039610 31676 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:19.047464 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:19.048225 31827 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.050001 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:19.051577 32004 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.052048 32004 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.051779 31868 raft_consensus.cc:359] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.052448 32004 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.052556 31868 raft_consensus.cc:385] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.052893 31868 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.053817 31868 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.022207 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:19.085763 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:19.078292 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:19.087018 31923 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.087586 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:19.087548 31923 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.089949 31923 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.091704 31956 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:19.092098 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:19.092262 31956 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:19.085703 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:19.093165 31956 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.094046 31751 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:19.086606 31822 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.097347 31868 ts_tablet_manager.cc:1434] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.064s	user 0.007s	sys 0.002s
I20260709 12:20:19.098753 31868 tablet_bootstrap.cc:492] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:19.124722 31926 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:19.129349 32051 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:19.129971 32051 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:19.130960 32051 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:19.108893 31717 leader_election.cc:343] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.108783 31923 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.111511 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:19.149482 31874 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.169987 31827 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
W20260709 12:20:19.175217 31624 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 12:20:19.150031 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 48929.1us
I20260709 12:20:19.158324 31822 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:19.185304 32004 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.188036 32004 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:19.150161 31923 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.192750 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:19.194005 31922 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:19.195986 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:19.195988 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:19.199447 31751 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:19.199937 31922 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:19.200775 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:19.201745 31872 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.202155 31872 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.203663 31868 tablet_bootstrap.cc:654] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.202541 31872 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.230023 31867 tablet_bootstrap.cc:492] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:19.230852 31867 ts_tablet_manager.cc:1403] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.270s	user 0.015s	sys 0.006s
I20260709 12:20:19.205993 32065 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.232265 32065 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.177347 31874 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.238873 31826 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:19.239352 31874 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:19.194389 31923 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.224290 31866 tablet_bootstrap.cc:492] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:19.240353 31866 ts_tablet_manager.cc:1403] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.333s	user 0.020s	sys 0.028s
W20260709 12:20:19.225895 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 105972us
I20260709 12:20:19.243796 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:19.247480 31956 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:19.244073 31866 raft_consensus.cc:359] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.245484 31872 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.245926 31576 catalog_manager.cc:5697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:19.248126 31866 raft_consensus.cc:385] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.248560 31866 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.250073 31866 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.251151 31868 tablet_bootstrap.cc:492] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:19.251777 31868 ts_tablet_manager.cc:1403] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.153s	user 0.022s	sys 0.000s
I20260709 12:20:19.249418 31956 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:19.254727 31956 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.255910 31866 ts_tablet_manager.cc:1434] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.001s
I20260709 12:20:19.256907 31866 tablet_bootstrap.cc:492] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:19.259582 31867 raft_consensus.cc:359] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.260306 31867 raft_consensus.cc:385] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.260557 31867 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
W20260709 12:20:19.240792 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 145862us
I20260709 12:20:19.264109 32065 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.264930 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:19.266160 31678 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 1: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:20:19.266566 31867 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.268610 31867 ts_tablet_manager.cc:1434] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.037s	user 0.006s	sys 0.000s
I20260709 12:20:19.269596 31867 tablet_bootstrap.cc:492] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:19.269718 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:19.270334 31826 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:19.272315 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:19.275103 31872 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.277902 32065 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.278389 32065 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.277176 31574 catalog_manager.cc:5697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:19.293385 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:19.294040 31826 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.294977 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:19.288903 32065 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.310758 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:19.311388 31868 raft_consensus.cc:359] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.312057 31868 raft_consensus.cc:385] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.312265 31868 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.312399 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:19.312983 31751 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.312892 31868 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.314569 31868 ts_tablet_manager.cc:1434] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.063s	user 0.003s	sys 0.000s
I20260709 12:20:19.315572 31868 tablet_bootstrap.cc:492] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:19.318557 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:19.319485 31678 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 1: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:20:19.321705 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:19.323424 31923 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.325033 31751 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
W20260709 12:20:19.327356 32070 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 12:20:19.329540 32065 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.330812 31752 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 1: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:19.346921 31826 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:19.348663 32065 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.349061 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:19.349416 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:19.349769 31677 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 1: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:20:19.350239 31824 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 1: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:20:19.351650 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:19.354069 32065 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:19.361567 31867 tablet_bootstrap.cc:654] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.366202 31868 tablet_bootstrap.cc:654] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.367738 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:19.369900 32067 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:19.371201 32067 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:19.378401 31866 tablet_bootstrap.cc:654] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:19.377669 31791 leader_election.cc:343] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.379814 32067 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:19.380767 31791 leader_election.cc:343] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.381819 32004 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.383607 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:19.384363 32004 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:19.386847 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:19.388805 31922 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:19.392439 31891 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.393272 31891 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.398994 31891 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.399097 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:19.399935 31753 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:19.402004 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:19.404078 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:19.404649 31891 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.404915 31826 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:19.405207 31891 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.405733 31891 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.414793 31922 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:19.416172 31922 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.416200 31576 catalog_manager.cc:5697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:19.440403 31574 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:20:19.442548 31791 consensus_peers.cc:597] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:19.449718 31866 tablet_bootstrap.cc:492] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:19.450274 31866 ts_tablet_manager.cc:1403] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.194s	user 0.008s	sys 0.015s
I20260709 12:20:19.454105 32067 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.454923 31891 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.456655 31866 raft_consensus.cc:359] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.457345 31866 raft_consensus.cc:385] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.457428 31867 tablet_bootstrap.cc:492] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:19.457613 31866 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.457937 31867 ts_tablet_manager.cc:1403] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.189s	user 0.011s	sys 0.018s
I20260709 12:20:19.457876 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:19.458549 31752 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.458393 31866 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.460891 31866 ts_tablet_manager.cc:1434] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:20:19.462170 31866 tablet_bootstrap.cc:492] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:19.464241 31868 tablet_bootstrap.cc:492] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:19.464856 31868 ts_tablet_manager.cc:1403] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.149s	user 0.023s	sys 0.001s
I20260709 12:20:19.465524 31752 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:19.466820 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:19.467674 31922 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:19.468169 31866 tablet_bootstrap.cc:654] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.468155 31922 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:19.469363 31922 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.470206 31867 raft_consensus.cc:359] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.470981 31867 raft_consensus.cc:385] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.471606 31867 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.472389 31867 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.480190 31868 raft_consensus.cc:359] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.484297 31867 ts_tablet_manager.cc:1434] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.000s
I20260709 12:20:19.490573 31866 tablet_bootstrap.cc:492] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:19.491127 31866 ts_tablet_manager.cc:1403] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.029s	user 0.011s	sys 0.009s
I20260709 12:20:19.491385 31867 tablet_bootstrap.cc:492] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:19.487677 31573 catalog_manager.cc:5697] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:19.489650 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:19.493944 31868 raft_consensus.cc:385] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.490149 31891 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.493888 31826 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.499784 31867 tablet_bootstrap.cc:654] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.496992 31868 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.496031 31866 raft_consensus.cc:359] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.500603 31866 raft_consensus.cc:385] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.501107 31866 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.501911 31866 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.503010 31868 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.505300 31826 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:19.509887 31866 ts_tablet_manager.cc:1434] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.018s	user 0.004s	sys 0.002s
I20260709 12:20:19.510078 31868 ts_tablet_manager.cc:1434] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.045s	user 0.006s	sys 0.001s
I20260709 12:20:19.510938 31866 tablet_bootstrap.cc:492] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:19.511067 31868 tablet_bootstrap.cc:492] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:19.518249 31973 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.519017 31973 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.523124 32067 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.522500 31867 tablet_bootstrap.cc:492] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:19.523645 32067 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.524412 31867 ts_tablet_manager.cc:1403] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.033s	user 0.014s	sys 0.002s
I20260709 12:20:19.524351 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:19.525513 31973 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.525867 32067 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:19.528234 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:19.529973 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:19.531351 31791 leader_election.cc:343] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.531276 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:19.527310 31867 raft_consensus.cc:359] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.533445 31867 raft_consensus.cc:385] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.533742 31867 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
W20260709 12:20:19.533849 31791 leader_election.cc:343] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.534211 31956 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.534487 31867 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.537637 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:19.537724 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:19.538496 32067 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:19.538897 31973 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:19.542008 31866 tablet_bootstrap.cc:654] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.548374 31868 tablet_bootstrap.cc:654] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.549015 31576 catalog_manager.cc:5697] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:19.554070 31867 ts_tablet_manager.cc:1434] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.029s	user 0.006s	sys 0.000s
I20260709 12:20:19.558122 31866 tablet_bootstrap.cc:492] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:19.558596 31866 ts_tablet_manager.cc:1403] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.048s	user 0.009s	sys 0.007s
I20260709 12:20:19.561256 31866 raft_consensus.cc:359] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.561928 31866 raft_consensus.cc:385] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.562199 31866 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.562266 31868 tablet_bootstrap.cc:492] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:19.562836 31868 ts_tablet_manager.cc:1403] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.052s	user 0.013s	sys 0.000s
I20260709 12:20:19.562878 31866 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.565186 31866 ts_tablet_manager.cc:1434] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:19.566133 31866 tablet_bootstrap.cc:492] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:19.566300 31868 raft_consensus.cc:359] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.567085 31868 raft_consensus.cc:385] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.567394 31868 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.568151 31868 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.571903 31868 ts_tablet_manager.cc:1434] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:20:19.572988 31866 tablet_bootstrap.cc:654] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.573446 31868 tablet_bootstrap.cc:492] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:19.575431 31973 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.581449 31868 tablet_bootstrap.cc:654] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.589892 31867 tablet_bootstrap.cc:492] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:19.596529 31891 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.597198 31891 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.601388 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:19.602584 31753 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 1: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:19.607079 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:19.608237 31867 tablet_bootstrap.cc:654] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.610177 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:19.612434 32075 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.618749 31891 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.622014 31866 tablet_bootstrap.cc:492] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:19.595825 31868 tablet_bootstrap.cc:492] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:19.622556 31866 ts_tablet_manager.cc:1403] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.057s	user 0.023s	sys 0.019s
I20260709 12:20:19.625495 31866 raft_consensus.cc:359] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.626206 31866 raft_consensus.cc:385] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.626473 31866 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.627833 32067 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.628780 32075 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:19.622644 31868 ts_tablet_manager.cc:1403] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.049s	user 0.018s	sys 0.002s
I20260709 12:20:19.627265 31866 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.631853 31866 ts_tablet_manager.cc:1434] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.001s
I20260709 12:20:19.632789 31866 tablet_bootstrap.cc:492] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:19.635074 31868 raft_consensus.cc:359] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.636895 31868 raft_consensus.cc:385] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.637576 31868 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.639156 31868 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.642871 31868 ts_tablet_manager.cc:1434] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:20:19.643129 31866 tablet_bootstrap.cc:654] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.645256 31867 tablet_bootstrap.cc:492] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:19.645859 31867 ts_tablet_manager.cc:1403] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.056s	user 0.025s	sys 0.014s
I20260709 12:20:19.646711 31868 tablet_bootstrap.cc:492] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:19.648636 31867 raft_consensus.cc:359] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.649340 31867 raft_consensus.cc:385] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.649602 31867 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.650250 31867 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.651732 32067 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.652567 31867 ts_tablet_manager.cc:1434] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:20:19.654983 31867 tablet_bootstrap.cc:492] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:19.662132 31867 tablet_bootstrap.cc:654] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.670976 31973 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.671672 31973 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.675048 31867 tablet_bootstrap.cc:492] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:19.675189 31973 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.675642 31867 ts_tablet_manager.cc:1403] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.021s	user 0.013s	sys 0.006s
I20260709 12:20:19.676893 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:19.677561 31678 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:19.678930 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:19.678985 31867 raft_consensus.cc:359] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.679708 31867 raft_consensus.cc:385] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.679996 31867 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.680076 31973 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.680441 31973 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.681072 31973 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.680761 31867 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.683085 31867 ts_tablet_manager.cc:1434] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:19.685122 31867 tablet_bootstrap.cc:492] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:19.685295 31868 tablet_bootstrap.cc:654] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.687799 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:19.688496 31827 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:19.694060 31973 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.696472 31867 tablet_bootstrap.cc:654] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.703471 31866 tablet_bootstrap.cc:492] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:19.703989 31866 ts_tablet_manager.cc:1403] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.071s	user 0.040s	sys 0.011s
I20260709 12:20:19.707243 31866 raft_consensus.cc:359] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.711637 31866 raft_consensus.cc:385] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.711961 31866 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.713478 31678 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.713711 31866 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.716482 31866 ts_tablet_manager.cc:1434] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 12:20:19.717979 31866 tablet_bootstrap.cc:492] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:19.719376 32075 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.719949 32075 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.723354 32056 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:19.724875 31866 tablet_bootstrap.cc:654] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:19.726152 31868 tablet_bootstrap.cc:492] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:19.727036 31868 ts_tablet_manager.cc:1403] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.081s	user 0.015s	sys 0.005s
I20260709 12:20:19.732074 32075 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.733640 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:19.734608 31826 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:19.736757 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:19.738448 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:19.742242 31891 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:19.742755 31891 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:19.739779 31753 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:19.743124 31891 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.750622 31891 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.753959 31891 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.755936 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:19.765795 31826 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.775081 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:19.761665 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:19.787482 31678 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.779196 31956 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:19.761059 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:19.800735 31827 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.809345 31753 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.819236 31973 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:19.838582 31827 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:19.840014 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:19.841060 31973 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:19.815296 31868 raft_consensus.cc:359] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.840832 31928 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.841843 31973 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:19.842049 31928 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.843245 31973 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.846987 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:19.847213 31868 raft_consensus.cc:385] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.846498 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:19.866304 31926 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260709 12:20:19.876957 31868 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.877946 31868 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.852391 31867 tablet_bootstrap.cc:492] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:19.845358 31928 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:19.879313 31867 ts_tablet_manager.cc:1403] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.194s	user 0.067s	sys 0.037s
I20260709 12:20:19.857052 31922 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
W20260709 12:20:19.873510 31791 leader_election.cc:343] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:19.881371 31791 leader_election.cc:343] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.881819 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:19.882563 31928 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:19.897483 31891 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:19.899444 31867 raft_consensus.cc:359] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.900378 31867 raft_consensus.cc:385] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:19.900740 31867 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:19.902788 31867 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.905977 31928 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.906818 31928 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.907618 31867 ts_tablet_manager.cc:1434] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.001s
I20260709 12:20:19.909287 31928 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:19.912765 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:19.916965 31791 leader_election.cc:343] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.922974 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:19.932675 31926 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:19.933468 31926 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:19.936441 31926 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:19.938045 32004 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:20:19.938223 31791 leader_election.cc:343] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.938910 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:19.939730 32004 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:19.942768 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:19.944406 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:19.956017 31922 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:19.967036 31867 tablet_bootstrap.cc:492] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:19.973727 31573 catalog_manager.cc:5697] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:19.976603 31791 leader_election.cc:343] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:19.977062 31750 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:19.813491 31678 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:19.983479 31868 ts_tablet_manager.cc:1434] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.256s	user 0.007s	sys 0.000s
I20260709 12:20:19.984544 31868 tablet_bootstrap.cc:492] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
W20260709 12:20:20.005616 31791 leader_election.cc:343] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.006373 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:20.018857 32067 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:20.047423 32075 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:20.074904 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58155.1us
W20260709 12:20:20.137969 32076 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.091s	user 0.005s	sys 0.010s
I20260709 12:20:20.147536 31867 tablet_bootstrap.cc:654] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.152081 31891 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.152631 31891 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.157207 31891 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.162789 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:20.156925 32065 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.164220 32065 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.166903 31753 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
W20260709 12:20:20.167538 31643 leader_election.cc:343] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.170979 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:20.173451 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:20.182128 32105 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:20:20.199024 31642 leader_election.cc:343] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.199911 31642 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:20.204988 31891 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:20.206491 31866 tablet_bootstrap.cc:492] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:20.207265 31866 ts_tablet_manager.cc:1403] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.489s	user 0.136s	sys 0.167s
I20260709 12:20:20.212074 32105 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:20.218868 31826 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
W20260709 12:20:20.245752 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58732.2us
I20260709 12:20:20.247170 32065 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.245091 32105 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.250005 31866 raft_consensus.cc:359] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.250584 31866 raft_consensus.cc:385] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.250928 31866 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.252789 31866 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.256882 31866 ts_tablet_manager.cc:1434] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.049s	user 0.006s	sys 0.002s
I20260709 12:20:20.258040 31866 tablet_bootstrap.cc:492] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:20.267660 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:20.272426 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:20.313383 31718 leader_election.cc:343] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.318990 32109 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.327096 32109 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.330942 31867 tablet_bootstrap.cc:492] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:20.331202 31868 tablet_bootstrap.cc:654] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.331421 31867 ts_tablet_manager.cc:1403] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.365s	user 0.016s	sys 0.008s
I20260709 12:20:20.336109 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:20.336858 31677 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:20.339105 31574 catalog_manager.cc:5697] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:20.341738 31867 raft_consensus.cc:359] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.333937 32109 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:20.342463 31867 raft_consensus.cc:385] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.346164 31867 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.347000 31867 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.349097 31867 ts_tablet_manager.cc:1434] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
I20260709 12:20:20.350158 31867 tablet_bootstrap.cc:492] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:20.352452 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:20.331861 31717 leader_election.cc:343] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.353966 31866 tablet_bootstrap.cc:654] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.354271 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:20.357574 31791 leader_election.cc:343] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.358419 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:20:20.358996 31624 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:20.359702 32056 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:20.365031 32109 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.365717 32109 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.366227 32109 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.367221 31866 tablet_bootstrap.cc:492] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:20.367726 31866 ts_tablet_manager.cc:1403] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.110s	user 0.012s	sys 0.002s
I20260709 12:20:20.372222 31868 tablet_bootstrap.cc:492] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:20.372910 31868 ts_tablet_manager.cc:1403] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.389s	user 0.009s	sys 0.015s
I20260709 12:20:20.375790 31868 raft_consensus.cc:359] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.376579 31868 raft_consensus.cc:385] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.376847 31868 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.377519 31868 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.380482 31868 ts_tablet_manager.cc:1434] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.002s
I20260709 12:20:20.381192 31866 raft_consensus.cc:359] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.381960 31866 raft_consensus.cc:385] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.382251 31866 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.382987 31866 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.384405 31867 tablet_bootstrap.cc:654] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.384637 31868 tablet_bootstrap.cc:492] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:20.388533 32109 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.390709 32109 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:20.393028 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:20.393711 31677 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.401166 31677 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:20.403240 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:20.405014 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:20.405827 32109 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.406458 32109 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:20.407399 32109 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:20.411454 31791 leader_election.cc:343] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.412545 31866 ts_tablet_manager.cc:1434] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.045s	user 0.007s	sys 0.000s
I20260709 12:20:20.413658 31867 tablet_bootstrap.cc:492] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:20.414129 31867 ts_tablet_manager.cc:1403] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.064s	user 0.012s	sys 0.019s
I20260709 12:20:20.417238 31867 raft_consensus.cc:359] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.418007 31867 raft_consensus.cc:385] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.418279 31867 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.414180 31866 tablet_bootstrap.cc:492] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:20.419212 31867 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.421540 31867 ts_tablet_manager.cc:1434] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.003s
I20260709 12:20:20.424166 31867 tablet_bootstrap.cc:492] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:20.425886 31866 tablet_bootstrap.cc:654] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.430244 31868 tablet_bootstrap.cc:654] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.431164 31867 tablet_bootstrap.cc:654] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.452121 31973 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:20.483388 31868 tablet_bootstrap.cc:492] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:20.483994 31868 ts_tablet_manager.cc:1403] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.100s	user 0.021s	sys 0.014s
I20260709 12:20:20.487583 31868 raft_consensus.cc:359] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.488627 31868 raft_consensus.cc:385] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.489037 31868 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.492880 31866 tablet_bootstrap.cc:492] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:20.493423 31866 ts_tablet_manager.cc:1403] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.079s	user 0.016s	sys 0.018s
I20260709 12:20:20.493757 31956 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:20.497053 31866 raft_consensus.cc:359] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.498070 31866 raft_consensus.cc:385] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.498411 31866 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.489966 31868 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.502260 31868 ts_tablet_manager.cc:1434] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.000s
I20260709 12:20:20.503554 31868 tablet_bootstrap.cc:492] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:20.510602 31866 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.513845 31866 ts_tablet_manager.cc:1434] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.020s	user 0.006s	sys 0.003s
I20260709 12:20:20.515784 31868 tablet_bootstrap.cc:654] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.520587 31866 tablet_bootstrap.cc:492] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:20.530018 31866 tablet_bootstrap.cc:654] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.531445 31574 catalog_manager.cc:5697] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:20.537035 31868 tablet_bootstrap.cc:492] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:20.538159 31868 ts_tablet_manager.cc:1403] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.035s	user 0.013s	sys 0.003s
I20260709 12:20:20.541211 31868 raft_consensus.cc:359] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.541901 31868 raft_consensus.cc:385] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.542192 31868 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.542951 31868 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.543802 31867 tablet_bootstrap.cc:492] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:20.544410 31867 ts_tablet_manager.cc:1403] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.121s	user 0.041s	sys 0.056s
I20260709 12:20:20.545007 31866 tablet_bootstrap.cc:492] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:20.546530 31866 ts_tablet_manager.cc:1403] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.026s	user 0.015s	sys 0.001s
I20260709 12:20:20.549052 31866 raft_consensus.cc:359] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.549453 31868 ts_tablet_manager.cc:1434] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.003s
I20260709 12:20:20.549710 31866 raft_consensus.cc:385] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.550005 31866 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.550452 31868 tablet_bootstrap.cc:492] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:20.550797 31866 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.553149 31866 ts_tablet_manager.cc:1434] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:20:20.554034 31866 tablet_bootstrap.cc:492] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:20.560171 31866 tablet_bootstrap.cc:654] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.548563 31867 raft_consensus.cc:359] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.565151 31867 raft_consensus.cc:385] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.565588 31867 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.567227 31868 tablet_bootstrap.cc:654] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.566694 31867 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.569612 31867 ts_tablet_manager.cc:1434] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.000s
I20260709 12:20:20.570879 31867 tablet_bootstrap.cc:492] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:20.572271 31973 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.572806 31973 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.575745 31973 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.576081 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:20.577270 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:20.577718 31867 tablet_bootstrap.cc:654] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:20.577853 31717 leader_election.cc:343] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.577889 31827 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:20.579974 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:20.581291 31973 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.581650 31973 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.582052 31973 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.588917 31973 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.592027 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:20.593044 31891 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.593681 31866 tablet_bootstrap.cc:492] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:20.593578 31891 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.594048 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:20.594424 31866 ts_tablet_manager.cc:1403] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.041s	user 0.013s	sys 0.003s
I20260709 12:20:20.594583 31827 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.597430 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:20.598089 31826 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:20.599752 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:20.600623 31922 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.600978 31922 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.601336 31922 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.601723 31973 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.601256 31827 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:20.602788 31867 tablet_bootstrap.cc:492] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:20.593757 31868 tablet_bootstrap.cc:492] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:20.602953 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:20.603511 31867 ts_tablet_manager.cc:1403] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.033s	user 0.004s	sys 0.013s
I20260709 12:20:20.603511 31868 ts_tablet_manager.cc:1403] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.053s	user 0.011s	sys 0.004s
I20260709 12:20:20.604463 31973 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.605839 31973 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:20.606597 31866 raft_consensus.cc:359] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.607016 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:20.606895 31973 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.607519 31866 raft_consensus.cc:385] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.607981 31866 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
W20260709 12:20:20.608662 31642 leader_election.cc:343] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.608839 31866 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:20.609809 31717 leader_election.cc:343] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.610064 31891 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.611696 31866 ts_tablet_manager.cc:1434] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.003s
I20260709 12:20:20.612936 31866 tablet_bootstrap.cc:492] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:20.619055 31868 raft_consensus.cc:359] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.619840 31868 raft_consensus.cc:385] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.620124 31868 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.620666 31866 tablet_bootstrap.cc:654] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.620818 31868 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.623111 31868 ts_tablet_manager.cc:1434] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.001s
I20260709 12:20:20.624284 31868 tablet_bootstrap.cc:492] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:20.626343 31867 raft_consensus.cc:359] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.627314 31867 raft_consensus.cc:385] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.627635 31867 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.628548 31867 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.631668 31867 ts_tablet_manager.cc:1434] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.000s
I20260709 12:20:20.635002 31922 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.636513 31867 tablet_bootstrap.cc:492] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:20.638101 31973 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.638800 31973 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.639940 31573 catalog_manager.cc:5697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:20.641064 31973 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.638005 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:20.643328 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:20.644047 31868 tablet_bootstrap.cc:654] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.644771 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:20.645386 31826 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.646483 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:20.647177 31677 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:20.648710 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:20.643997 31827 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:20.650066 31867 tablet_bootstrap.cc:654] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.650660 31973 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.651067 31973 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:20:20.651402 31642 leader_election.cc:343] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.651728 31973 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.658834 31973 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.659580 31826 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:20.660974 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:20.661847 31891 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.661755 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:20.662339 31891 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:20.662410 31677 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.663203 31891 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.669097 31973 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.669740 31922 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.670193 31866 tablet_bootstrap.cc:492] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:20.671981 31866 ts_tablet_manager.cc:1403] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.059s	user 0.016s	sys 0.006s
I20260709 12:20:20.673135 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:20.674032 31827 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.670239 31677 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:20.679780 31866 raft_consensus.cc:359] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.680612 31866 raft_consensus.cc:385] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.680894 31866 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.681898 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:20.682894 31973 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.683421 31973 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:20.684269 31973 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.685263 31827 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:20.688519 31866 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.691115 31922 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:20.694748 31866 ts_tablet_manager.cc:1434] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.002s
I20260709 12:20:20.695988 31866 tablet_bootstrap.cc:492] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:20.699551 31573 catalog_manager.cc:5697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:20.700851 31867 tablet_bootstrap.cc:492] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:20.701340 31867 ts_tablet_manager.cc:1403] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.065s	user 0.021s	sys 0.008s
I20260709 12:20:20.704140 31868 tablet_bootstrap.cc:492] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:20.704041 31867 raft_consensus.cc:359] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.704655 31868 ts_tablet_manager.cc:1403] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.081s	user 0.009s	sys 0.004s
I20260709 12:20:20.704739 31867 raft_consensus.cc:385] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.705088 31867 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.705727 31867 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.707700 31868 raft_consensus.cc:359] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.708437 31868 raft_consensus.cc:385] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.708722 31868 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.709455 31867 ts_tablet_manager.cc:1434] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:20.709501 31868 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.711054 31867 tablet_bootstrap.cc:492] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:20.712268 31868 ts_tablet_manager.cc:1434] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.006s
I20260709 12:20:20.713248 31973 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.713773 31973 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.713258 31868 tablet_bootstrap.cc:492] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:20.718436 31867 tablet_bootstrap.cc:654] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.721414 31868 tablet_bootstrap.cc:654] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.723799 31973 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.725713 32075 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:20.727057 31866 tablet_bootstrap.cc:654] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.727631 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:20.728317 31827 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:20.726543 32065 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.741685 32065 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.745361 32109 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.746127 32109 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.733294 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:20.749647 32065 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:20.750649 31717 leader_election.cc:343] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.753111 31867 tablet_bootstrap.cc:492] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:20.751109 32109 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:20.760739 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:20.761431 31824 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:20.756772 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:20.760545 31867 ts_tablet_manager.cc:1403] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.050s	user 0.017s	sys 0.008s
I20260709 12:20:20.766373 31867 raft_consensus.cc:359] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.767184 31867 raft_consensus.cc:385] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.767565 31867 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.768594 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:20.769274 31750 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:20.769364 31868 tablet_bootstrap.cc:492] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:20.770134 31868 ts_tablet_manager.cc:1403] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.057s	user 0.008s	sys 0.016s
I20260709 12:20:20.769985 31867 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.771663 31574 catalog_manager.cc:5697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:20.772563 31867 ts_tablet_manager.cc:1434] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:20:20.773932 32075 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.773983 31868 raft_consensus.cc:359] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.774495 32075 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.775089 31868 raft_consensus.cc:385] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.775661 31868 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.776033 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:20.776530 32075 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.776835 31677 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:20.776574 31868 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.778420 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:20.779016 31826 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:20.780601 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:20.781383 32065 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.781709 32065 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.782043 32065 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.783160 31868 ts_tablet_manager.cc:1434] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.002s
I20260709 12:20:20.784608 31868 tablet_bootstrap.cc:492] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:20.788734 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:20.789256 32065 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.758111 31718 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:20.791472 31973 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.791383 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:20.791908 31973 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.792126 31752 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:20.793527 31868 tablet_bootstrap.cc:654] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.792264 31973 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.796497 32105 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.796903 32105 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.797204 32105 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.802476 31923 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:20.803148 31923 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.807683 31923 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.808382 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:20.815282 31751 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:20.808382 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:20.816377 31826 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:20.817883 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:20.810241 31868 tablet_bootstrap.cc:492] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:20.827149 31868 ts_tablet_manager.cc:1403] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.043s	user 0.011s	sys 0.011s
I20260709 12:20:20.827731 31922 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.828542 31922 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.830839 31922 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.811550 31791 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:20.818243 31973 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.839092 31973 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.831807 31868 raft_consensus.cc:359] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.840859 31868 raft_consensus.cc:385] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.841217 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:20.841337 31868 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
W20260709 12:20:20.842935 31717 leader_election.cc:343] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.843209 31868 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.845395 31867 tablet_bootstrap.cc:492] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:20.845786 31868 ts_tablet_manager.cc:1434] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.002s
I20260709 12:20:20.847259 31868 tablet_bootstrap.cc:492] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:20.851276 32004 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:20.851958 32004 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:20.852481 32004 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.854405 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:20.855562 31826 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:20.859192 31791 leader_election.cc:343] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:20.813879 32105 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.864442 32105 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.865671 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:20.865828 32004 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.869067 32065 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.866436 31822 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.866600 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:20.903483 32004 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:20.903930 31751 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.872751 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:20.899420 31826 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:20.876375 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:20.906615 31823 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 1: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:20.906679 31677 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.911387 31822 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:20.912912 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:20.913062 31718 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:20.913731 31891 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.913913 32065 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.914227 31891 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:20.914466 32065 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:20.915079 31891 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.915347 32065 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.919199 31868 tablet_bootstrap.cc:654] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.923671 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:20.925096 31677 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:20.926281 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:20.927178 31750 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 1: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:20.927481 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:20.928402 31973 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.929036 31973 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:20.929821 31973 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.931473 31791 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:20.927822 31922 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.936223 32143 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:20.936545 31867 tablet_bootstrap.cc:654] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:20.939220 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:20.939970 31750 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.943118 31751 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:20.945155 31922 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:20.946300 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:20.946956 31822 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:20.947327 32109 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:20.957211 31868 tablet_bootstrap.cc:492] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:20.957751 31868 ts_tablet_manager.cc:1403] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.111s	user 0.013s	sys 0.000s
I20260709 12:20:20.960671 31868 raft_consensus.cc:359] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.961374 31868 raft_consensus.cc:385] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.961671 31868 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.961465 31576 catalog_manager.cc:5697] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:20.963498 31866 tablet_bootstrap.cc:492] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:20.964020 31866 ts_tablet_manager.cc:1403] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.268s	user 0.055s	sys 0.082s
I20260709 12:20:20.965059 31573 catalog_manager.cc:5697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:20.966719 31866 raft_consensus.cc:359] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.967401 31866 raft_consensus.cc:385] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:20.967649 31866 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:20.968314 31866 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.968250 31573 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:20.973942 31822 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:20.974404 31868 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.976030 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:20.986109 31868 ts_tablet_manager.cc:1434] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.028s	user 0.002s	sys 0.004s
I20260709 12:20:20.987094 31868 tablet_bootstrap.cc:492] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:20.987975 31922 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:20.988508 31922 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:20.989338 31922 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:20.992060 31866 ts_tablet_manager.cc:1434] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.028s	user 0.002s	sys 0.005s
I20260709 12:20:20.993172 31866 tablet_bootstrap.cc:492] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:20.973942 31750 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:21.004654 31751 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.014287 32143 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:21.022022 31867 tablet_bootstrap.cc:492] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:21.022559 31867 ts_tablet_manager.cc:1403] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.179s	user 0.017s	sys 0.037s
I20260709 12:20:21.024150 31868 tablet_bootstrap.cc:654] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.027647 31866 tablet_bootstrap.cc:654] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.025236 31867 raft_consensus.cc:359] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.031486 31867 raft_consensus.cc:385] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.031805 31867 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.032464 31867 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.042518 32109 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.043128 32109 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.051867 31867 ts_tablet_manager.cc:1434] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.029s	user 0.005s	sys 0.002s
I20260709 12:20:21.051688 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:21.052975 31867 tablet_bootstrap.cc:492] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
W20260709 12:20:21.055168 31791 leader_election.cc:343] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.050318 31956 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.057284 31956 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.075484 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:21.080557 31791 leader_election.cc:343] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.081220 31791 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:21.082932 32143 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.045257 32109 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:21.087602 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:21.088399 31674 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:21.088814 31956 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.090248 31867 tablet_bootstrap.cc:654] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.092125 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:21.100589 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:20:21.101866 31718 leader_election.cc:343] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.101960 31973 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:21.102523 31973 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:21.102962 31973 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.106647 31573 catalog_manager.cc:5697] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:21.114763 31868 tablet_bootstrap.cc:492] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:21.117066 31868 ts_tablet_manager.cc:1403] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.130s	user 0.012s	sys 0.016s
I20260709 12:20:21.116101 31973 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.121062 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:21.120074 31866 tablet_bootstrap.cc:492] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:21.126176 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:21.126356 31866 ts_tablet_manager.cc:1403] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.133s	user 0.010s	sys 0.015s
I20260709 12:20:21.139763 31676 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:21.128273 31718 leader_election.cc:343] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.148073 31868 raft_consensus.cc:359] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.148835 31868 raft_consensus.cc:385] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.149086 31868 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.149989 31973 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.154199 31868 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.159190 31868 ts_tablet_manager.cc:1434] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.042s	user 0.007s	sys 0.000s
I20260709 12:20:21.160422 31868 tablet_bootstrap.cc:492] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:21.166491 31868 tablet_bootstrap.cc:654] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.150605 31866 raft_consensus.cc:359] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.172118 31866 raft_consensus.cc:385] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.172621 31866 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.173830 31866 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.178349 31868 tablet_bootstrap.cc:492] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:21.178894 31868 ts_tablet_manager.cc:1403] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.019s	user 0.011s	sys 0.006s
I20260709 12:20:21.181427 31868 raft_consensus.cc:359] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.182102 31868 raft_consensus.cc:385] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.182372 31868 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.184840 31867 tablet_bootstrap.cc:492] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:21.185293 31867 ts_tablet_manager.cc:1403] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.133s	user 0.012s	sys 0.011s
I20260709 12:20:21.187973 31867 raft_consensus.cc:359] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.189649 31867 raft_consensus.cc:385] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.189993 31867 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.190690 31867 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.192292 31867 ts_tablet_manager.cc:1434] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:20:21.192910 31973 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.193179 31867 tablet_bootstrap.cc:492] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:21.193382 31973 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.195838 31973 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.196310 31973 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.197599 31956 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.198400 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:21.199270 31823 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 1: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:20:21.202695 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:21.204941 31717 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:21.203776 31868 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.207267 32154 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.213292 31922 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:20:21.220556 31676 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:21.223421 31866 ts_tablet_manager.cc:1434] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.084s	user 0.005s	sys 0.002s
I20260709 12:20:21.224777 31866 tablet_bootstrap.cc:492] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:21.226089 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:21.226702 31867 tablet_bootstrap.cc:654] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.229619 31956 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:21.230223 31956 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:21.231124 31956 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:21.241804 31717 consensus_peers.cc:597] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:21.247716 31868 ts_tablet_manager.cc:1434] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.069s	user 0.001s	sys 0.005s
I20260709 12:20:21.253304 31868 tablet_bootstrap.cc:492] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:21.282053 31973 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.282759 31973 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.299073 31867 tablet_bootstrap.cc:492] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:21.299574 31867 ts_tablet_manager.cc:1403] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.107s	user 0.011s	sys 0.003s
I20260709 12:20:21.302523 31867 raft_consensus.cc:359] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.306943 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:21.307258 31867 raft_consensus.cc:385] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.308112 31867 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
W20260709 12:20:21.309005 31717 leader_election.cc:343] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.309106 31867 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.311610 31867 ts_tablet_manager.cc:1434] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 12:20:21.316046 32154 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.320120 31973 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.322819 31867 tablet_bootstrap.cc:492] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:21.329538 31868 tablet_bootstrap.cc:654] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.353968 31866 tablet_bootstrap.cc:654] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.354944 31973 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.358171 31676 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.363958 31867 tablet_bootstrap.cc:654] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.365137 31576 catalog_manager.cc:5697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:21.366365 31973 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.371183 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:21.373379 31822 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:21.370785 32154 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:21.376291 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:21.380115 31973 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:21.381253 31973 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:21.381698 31973 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.389333 31923 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.391167 32154 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.392922 31868 tablet_bootstrap.cc:492] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:21.393473 31868 ts_tablet_manager.cc:1403] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.140s	user 0.011s	sys 0.005s
I20260709 12:20:21.404326 31922 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.415539 31866 tablet_bootstrap.cc:492] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:21.416057 31866 ts_tablet_manager.cc:1403] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.192s	user 0.016s	sys 0.001s
I20260709 12:20:21.416707 31973 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.406216 31923 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:20:21.425967 31973 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.413206 31922 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.434362 31868 raft_consensus.cc:359] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.434531 31866 raft_consensus.cc:359] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.435214 31868 raft_consensus.cc:385] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.435283 31866 raft_consensus.cc:385] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.435621 31868 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.437754 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:21.438457 31823 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.435652 31866 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.443143 31866 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.444424 32105 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.445125 32105 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.448997 32105 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.450613 31922 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.451418 31867 tablet_bootstrap.cc:492] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:21.452033 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:21.455102 31867 ts_tablet_manager.cc:1403] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.138s	user 0.020s	sys 0.000s
I20260709 12:20:21.456961 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:21.447506 31868 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.459947 31866 ts_tablet_manager.cc:1434] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.044s	user 0.007s	sys 0.000s
W20260709 12:20:21.461488 31642 leader_election.cc:343] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.470268 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:21.472674 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:21.473482 31824 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
W20260709 12:20:21.474278 31643 leader_election.cc:343] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.474896 31643 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:21.476205 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:21.476455 31922 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.481002 31868 ts_tablet_manager.cc:1434] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.087s	user 0.014s	sys 0.001s
I20260709 12:20:21.483251 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:21.484359 31922 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:21.484800 31922 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:21.488619 31867 raft_consensus.cc:359] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.489519 31867 raft_consensus.cc:385] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.489305 32105 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:21.489872 31867 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.492007 31867 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.485263 31922 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:21.496240 31866 tablet_bootstrap.cc:492] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:21.505501 31867 ts_tablet_manager.cc:1434] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.050s	user 0.008s	sys 0.000s
I20260709 12:20:21.507242 31867 tablet_bootstrap.cc:492] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:21.508723 32154 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260709 12:20:21.540932 31868 tablet_bootstrap.cc:492] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:21.548626 31674 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.491798 31823 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:21.576121 32165 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.583212 32165 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.585549 32165 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.590332 31867 tablet_bootstrap.cc:654] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.593466 31922 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.633069 32143 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.633917 32143 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.640626 32143 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:21.665634 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:21.670011 31866 tablet_bootstrap.cc:654] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.673951 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:21.680766 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 126999us
I20260709 12:20:21.680871 32075 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:21.681381 31922 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.681432 31824 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:21.697746 31624 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:21.681573 31923 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:21.701776 31868 tablet_bootstrap.cc:654] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.705991 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:21.701622 32165 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.710968 32165 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.713271 32165 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.714268 32075 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.716629 32075 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.690322 31674 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:21.704567 31867 tablet_bootstrap.cc:492] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
W20260709 12:20:21.719900 31717 leader_election.cc:343] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.720414 31867 ts_tablet_manager.cc:1403] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.213s	user 0.016s	sys 0.000s
I20260709 12:20:21.721426 32075 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.722496 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:21.723953 31867 raft_consensus.cc:359] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.724686 31867 raft_consensus.cc:385] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.724978 31867 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.725661 31867 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.726209 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:21.728489 31867 ts_tablet_manager.cc:1434] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:21.732350 31867 tablet_bootstrap.cc:492] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:21.733705 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:21.683714 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:21.735265 31677 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 1: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:21.687549 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:20:21.733865 31643 leader_election.cc:343] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.738207 31867 tablet_bootstrap.cc:654] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.744383 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:21.745214 32143 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.750805 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:21.756244 31867 tablet_bootstrap.cc:492] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:21.756731 31642 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:21.758030 31867 ts_tablet_manager.cc:1403] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.026s	user 0.008s	sys 0.004s
I20260709 12:20:21.758239 31891 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:21.764740 31868 tablet_bootstrap.cc:492] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
W20260709 12:20:21.765084 31642 leader_election.cc:343] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.765645 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:21.766070 31868 ts_tablet_manager.cc:1403] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.225s	user 0.017s	sys 0.011s
I20260709 12:20:21.766564 31891 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.768925 31868 raft_consensus.cc:359] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.769709 31868 raft_consensus.cc:385] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.682293 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:21.770035 31868 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.761974 31867 raft_consensus.cc:359] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.770682 32154 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:21.771484 32154 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:21.771350 31868 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.770939 31867 raft_consensus.cc:385] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.773120 31868 ts_tablet_manager.cc:1434] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.003s
I20260709 12:20:21.773303 31867 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.772785 32154 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.774322 31868 tablet_bootstrap.cc:492] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:21.774322 31867 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.781174 31867 ts_tablet_manager.cc:1434] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.000s
I20260709 12:20:21.782426 31867 tablet_bootstrap.cc:492] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:21.783489 31868 tablet_bootstrap.cc:654] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:21.784839 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 102639us
I20260709 12:20:21.794239 31867 tablet_bootstrap.cc:654] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:21.797941 31717 leader_election.cc:343] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:21.800012 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 54411.9us
I20260709 12:20:21.805414 31866 tablet_bootstrap.cc:492] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:21.805958 31866 ts_tablet_manager.cc:1403] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.310s	user 0.019s	sys 0.007s
I20260709 12:20:21.807018 31868 tablet_bootstrap.cc:492] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:21.807567 31868 ts_tablet_manager.cc:1403] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.034s	user 0.013s	sys 0.006s
I20260709 12:20:21.807482 32004 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.808599 32004 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.812522 31867 tablet_bootstrap.cc:492] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:21.813135 31867 ts_tablet_manager.cc:1403] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.031s	user 0.011s	sys 0.004s
I20260709 12:20:21.813835 32004 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:21.815917 31867 raft_consensus.cc:359] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.816793 31867 raft_consensus.cc:385] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.817114 31867 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.817876 31867 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.821975 31867 ts_tablet_manager.cc:1434] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.008s	user 0.001s	sys 0.004s
I20260709 12:20:21.819758 31868 raft_consensus.cc:359] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.822469 31868 raft_consensus.cc:385] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.822850 31868 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.823700 31868 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.824507 31867 tablet_bootstrap.cc:492] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:21.825502 31868 ts_tablet_manager.cc:1434] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.017s	user 0.005s	sys 0.000s
I20260709 12:20:21.826529 31868 tablet_bootstrap.cc:492] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
W20260709 12:20:21.822682 31718 leader_election.cc:343] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.828413 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:21.829300 32165 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.831645 31867 tablet_bootstrap.cc:654] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.831100 32165 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:21.833321 31868 tablet_bootstrap.cc:654] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.835162 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:21.837296 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:21.838219 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:21.844012 31867 tablet_bootstrap.cc:492] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
W20260709 12:20:21.844318 31791 leader_election.cc:343] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.844749 31867 ts_tablet_manager.cc:1403] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.004s
I20260709 12:20:21.849107 31867 raft_consensus.cc:359] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.850188 31867 raft_consensus.cc:385] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.850589 31867 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.851542 31867 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.854419 32075 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.854848 31867 ts_tablet_manager.cc:1434] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.010s	user 0.012s	sys 0.000s
I20260709 12:20:21.855022 32075 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.856328 31867 tablet_bootstrap.cc:492] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
W20260709 12:20:21.848950 31791 leader_election.cc:343] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.857055 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:21.858510 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:21.860013 32004 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:20:21.861136 31642 leader_election.cc:343] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.863687 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:21.865387 31643 leader_election.cc:343] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.865911 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:21.867774 31866 raft_consensus.cc:359] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.868490 31866 raft_consensus.cc:385] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.868788 31866 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.869539 31866 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.870121 32075 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.870489 31922 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.870257 31867 tablet_bootstrap.cc:654] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.873241 31674 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.879884 31956 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:21.882426 32075 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:21.883145 32075 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.888435 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:21.889179 31751 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:21.891589 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:21.892390 31891 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:21.892728 31891 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:21.893040 31891 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.889814 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:21.896605 31822 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:21.905354 31891 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.907864 31867 tablet_bootstrap.cc:492] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:21.909117 31867 ts_tablet_manager.cc:1403] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.053s	user 0.012s	sys 0.010s
I20260709 12:20:21.908298 32075 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.915334 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:21.916033 31751 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:21.917032 31891 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:21.927769 31751 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:21.941213 31866 ts_tablet_manager.cc:1434] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.135s	user 0.003s	sys 0.006s
I20260709 12:20:21.943110 31866 tablet_bootstrap.cc:492] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
W20260709 12:20:21.953876 31718 leader_election.cc:343] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:21.960297 31718 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:21.961380 32165 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:21.952728 31576 catalog_manager.cc:5697] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:21.964794 31868 tablet_bootstrap.cc:492] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:21.965385 31868 ts_tablet_manager.cc:1403] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.139s	user 0.053s	sys 0.059s
I20260709 12:20:21.968186 31868 raft_consensus.cc:359] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.968946 31868 raft_consensus.cc:385] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.969214 31868 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.969579 31866 tablet_bootstrap.cc:654] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.969883 31868 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.973835 31868 ts_tablet_manager.cc:1434] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:20:21.974325 31867 raft_consensus.cc:359] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.974834 31868 tablet_bootstrap.cc:492] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:21.975071 31867 raft_consensus.cc:385] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:21.975399 31867 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:21.976104 31867 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.976656 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:21.977702 31923 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:21.978235 31867 ts_tablet_manager.cc:1434] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.069s	user 0.003s	sys 0.000s
I20260709 12:20:21.979327 31867 tablet_bootstrap.cc:492] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:21.981626 31868 tablet_bootstrap.cc:654] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:21.978152 31923 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:21.984882 31923 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:21.987512 31867 tablet_bootstrap.cc:654] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.005249 31674 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:22.007174 31868 tablet_bootstrap.cc:492] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:22.007052 32162 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:22.007714 31868 ts_tablet_manager.cc:1403] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.033s	user 0.011s	sys 0.011s
W20260709 12:20:22.007900 32187 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.051s	user 0.003s	sys 0.000s
I20260709 12:20:22.008553 31956 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:22.008935 31866 tablet_bootstrap.cc:492] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:22.008949 31956 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:22.009573 31866 ts_tablet_manager.cc:1403] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.067s	user 0.009s	sys 0.003s
I20260709 12:20:22.010073 31956 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.011775 31956 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:22.012243 31868 raft_consensus.cc:359] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.012928 31868 raft_consensus.cc:385] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.012461 31866 raft_consensus.cc:359] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.013229 31868 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.013336 31866 raft_consensus.cc:385] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.013710 31866 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.013901 31868 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.016274 31868 ts_tablet_manager.cc:1434] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:20:22.017215 31868 tablet_bootstrap.cc:492] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:22.028690 31866 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.031091 31868 tablet_bootstrap.cc:654] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.037142 31866 ts_tablet_manager.cc:1434] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.001s
I20260709 12:20:22.038355 31866 tablet_bootstrap.cc:492] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:22.042858 32004 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.043393 32004 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.045781 32004 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:22.048651 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:22.050818 31751 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:22.052711 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:22.053640 32143 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:22.054029 32143 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.054788 32143 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.055892 31866 tablet_bootstrap.cc:654] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.081792 32143 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:22.090664 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 185888us
I20260709 12:20:22.093966 32143 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:22.094699 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:22.095490 31751 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.099129 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:22.099922 31826 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.103626 31867 tablet_bootstrap.cc:492] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:22.104156 31867 ts_tablet_manager.cc:1403] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.125s	user 0.026s	sys 0.043s
I20260709 12:20:22.107064 31867 raft_consensus.cc:359] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.107805 31867 raft_consensus.cc:385] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.108098 31867 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.108732 31867 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.112375 31867 ts_tablet_manager.cc:1434] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:22.113319 31867 tablet_bootstrap.cc:492] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:22.125824 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:22.126610 31674 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:22.129156 31867 tablet_bootstrap.cc:654] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.133749 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:22.134585 31677 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.140057 31826 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:22.145474 31751 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:22.146894 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:22.148000 32143 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:22.148698 32143 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:22.149654 32143 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.151988 32190 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.152504 32190 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.154861 32190 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:22.149979 31575 catalog_manager.cc:5697] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:22.164301 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:22.164837 31868 tablet_bootstrap.cc:492] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:22.165354 31868 ts_tablet_manager.cc:1403] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.148s	user 0.023s	sys 0.040s
I20260709 12:20:22.165297 31751 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:22.166766 31642 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:22.168016 32190 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:22.168504 32190 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.168963 32190 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.174134 31866 tablet_bootstrap.cc:492] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:22.174690 31866 ts_tablet_manager.cc:1403] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.137s	user 0.031s	sys 0.014s
I20260709 12:20:22.176451 32190 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.178807 32190 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:22.168495 31868 raft_consensus.cc:359] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.179620 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:22.180017 31868 raft_consensus.cc:385] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.179955 31866 raft_consensus.cc:359] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.180480 31750 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.180860 31866 raft_consensus.cc:385] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.181273 31866 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.180464 31868 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.182240 31868 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.185012 31868 ts_tablet_manager.cc:1434] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.000s
I20260709 12:20:22.186097 31868 tablet_bootstrap.cc:492] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:22.192765 31868 tablet_bootstrap.cc:654] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.214956 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:22.218457 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:22.224920 31866 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.228695 31677 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:22.230393 31866 ts_tablet_manager.cc:1434] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.055s	user 0.009s	sys 0.000s
I20260709 12:20:22.230723 31956 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:22.231487 31866 tablet_bootstrap.cc:492] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:22.238197 31866 tablet_bootstrap.cc:654] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.244796 31750 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:22.245528 31867 tablet_bootstrap.cc:492] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:22.246239 31867 ts_tablet_manager.cc:1403] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.133s	user 0.041s	sys 0.041s
W20260709 12:20:22.248601 31643 leader_election.cc:343] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:22.253552 31643 leader_election.cc:343] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:22.251569 31956 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.261258 31642 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:22.263623 32192 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:22.264127 32192 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:22.264426 31868 tablet_bootstrap.cc:492] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:22.264896 31868 ts_tablet_manager.cc:1403] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.079s	user 0.025s	sys 0.020s
I20260709 12:20:22.264968 32192 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.267719 31868 raft_consensus.cc:359] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.268412 31868 raft_consensus.cc:385] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.257292 31956 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.268695 31868 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.270267 32143 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.270272 31868 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.274395 31573 catalog_manager.cc:5697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:22.271068 32143 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.277575 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:22.278236 31677 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:22.278503 32143 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:22.280808 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:22.281718 31674 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:22.283444 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:22.283903 32154 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.284543 32143 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:22.284497 32154 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.285101 32143 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.285601 32143 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.293939 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:22.294966 31675 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:22.295758 32065 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:22.304682 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:22.311000 32165 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:22.312163 32165 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.313122 32165 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.331247 31956 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:22.345247 31868 ts_tablet_manager.cc:1434] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.080s	user 0.007s	sys 0.001s
I20260709 12:20:22.346328 31868 tablet_bootstrap.cc:492] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:22.347836 31956 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.348289 31956 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.351176 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:22.355453 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:22.346918 31867 raft_consensus.cc:359] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.358659 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:22.358909 31824 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:22.358654 31867 raft_consensus.cc:385] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.359933 31867 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
W20260709 12:20:22.360364 31718 leader_election.cc:343] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:22.360602 31867 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:22.361816 31718 leader_election.cc:343] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:22.362596 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:22.363647 32065 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:22.364389 32065 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.364856 32065 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.368439 32143 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.369760 31868 tablet_bootstrap.cc:654] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.353140 31956 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:22.357156 32154 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:22.372340 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:22.373958 31677 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.374451 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:22.375041 31867 ts_tablet_manager.cc:1434] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.128s	user 0.024s	sys 0.046s
I20260709 12:20:22.375070 31752 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:22.376083 31867 tablet_bootstrap.cc:492] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:22.382827 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:22.383577 31676 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:22.385711 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:22.387043 31677 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:22.388276 31956 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:22.388648 31956 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.388950 31956 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.396497 32165 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.397771 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:22.412431 31956 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.417387 31956 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:22.419668 31866 tablet_bootstrap.cc:492] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:22.420156 31866 ts_tablet_manager.cc:1403] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.189s	user 0.008s	sys 0.010s
I20260709 12:20:22.420420 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:22.421231 31752 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.421066 32143 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:22.418488 32109 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:22.449544 31752 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:22.461514 31866 raft_consensus.cc:359] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.465086 31866 raft_consensus.cc:385] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.436489 32065 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.436539 32109 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:22.466724 31866 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.477083 31867 tablet_bootstrap.cc:654] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.476811 32109 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.479457 31866 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.484186 32165 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:22.543305 31866 ts_tablet_manager.cc:1434] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.123s	user 0.006s	sys 0.004s
I20260709 12:20:22.550907 31868 tablet_bootstrap.cc:492] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:22.552264 31868 ts_tablet_manager.cc:1403] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.206s	user 0.016s	sys 0.018s
I20260709 12:20:22.552659 31866 tablet_bootstrap.cc:492] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:22.503619 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:22.563978 31673 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.504323 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:22.484627 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:22.485814 32065 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:22.590097 31956 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.583245 31675 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.593907 31956 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.583329 31573 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:22.591982 31868 raft_consensus.cc:359] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.595727 31868 raft_consensus.cc:385] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.596109 31868 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.578711 32105 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.019s
I20260709 12:20:22.598466 31956 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:22.604364 31868 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.580535 31678 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.632081 31956 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.632762 31956 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.635835 31868 ts_tablet_manager.cc:1434] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.083s	user 0.007s	sys 0.002s
I20260709 12:20:22.637055 31868 tablet_bootstrap.cc:492] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:22.640064 31956 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:22.647995 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:22.648849 31671 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:22.651427 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:22.655611 31673 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:22.657106 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:22.657826 31822 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.663106 32211 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:20:22.683348 31573 catalog_manager.cc:5697] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:22.663645 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:22.704660 31867 tablet_bootstrap.cc:492] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:22.705225 31867 ts_tablet_manager.cc:1403] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.329s	user 0.006s	sys 0.031s
I20260709 12:20:22.703570 31678 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:22.681262 32109 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.023s
I20260709 12:20:22.709020 31956 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:22.710961 31956 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:22.711320 31956 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.725076 31867 raft_consensus.cc:359] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.725821 31867 raft_consensus.cc:385] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.726169 31867 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.726837 31867 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.726989 31956 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.728933 31956 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:22.730366 31867 ts_tablet_manager.cc:1434] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.025s	user 0.006s	sys 0.002s
I20260709 12:20:22.725896 31868 tablet_bootstrap.cc:654] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:22.710021 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 45196.9us
I20260709 12:20:22.732398 31867 tablet_bootstrap.cc:492] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:22.709920 31923 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260709 12:20:22.735746 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:22.736271 31891 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:20:22.739776 31717 leader_election.cc:343] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:22.744117 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:22.751681 31956 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:22.752048 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:22.752172 31956 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:22.752928 31827 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.753440 31956 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.754892 31717 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:22.766136 32065 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:22.767464 31827 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:22.766706 32065 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:22.773622 31822 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:22.796797 31675 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:22.773631 32065 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.800434 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:22.801012 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:22.801642 31678 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:22.837105 31866 tablet_bootstrap.cc:654] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.845038 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:22.847465 31868 tablet_bootstrap.cc:492] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:22.847955 31868 ts_tablet_manager.cc:1403] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.211s	user 0.009s	sys 0.010s
I20260709 12:20:22.841827 32075 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:22.851531 32075 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.854889 32075 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:22.855077 31867 tablet_bootstrap.cc:654] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:22.848649 32162 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:22.866146 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:22.867650 31643 leader_election.cc:343] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:22.864415 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:22.875147 32162 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:20:22.885087 31642 leader_election.cc:343] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:22.885555 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:22.887013 31891 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:22.885443 31868 raft_consensus.cc:359] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.891273 31868 raft_consensus.cc:385] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.891607 31868 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.892418 31868 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:22.893054 31643 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:22.900602 31866 tablet_bootstrap.cc:492] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:22.901360 31866 ts_tablet_manager.cc:1403] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.349s	user 0.013s	sys 0.017s
I20260709 12:20:22.914381 31868 ts_tablet_manager.cc:1434] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.066s	user 0.007s	sys 0.000s
I20260709 12:20:22.895699 32162 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.921034 31678 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:22.924989 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:22.925839 31868 tablet_bootstrap.cc:492] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
W20260709 12:20:22.932690 32223 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:20:22.932804 32065 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
W20260709 12:20:22.936082 31624 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260709 12:20:22.935731 32217 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.156s	user 0.004s	sys 0.000s
I20260709 12:20:22.942996 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:22.943559 32065 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:22.952349 32065 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:22.968210 31718 leader_election.cc:343] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:22.979712 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:22.980192 31866 raft_consensus.cc:359] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.980651 31823 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:22.980933 31866 raft_consensus.cc:385] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:22.982239 31866 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:22.982981 31866 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:22.990338 31867 tablet_bootstrap.cc:492] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:22.991876 31867 ts_tablet_manager.cc:1403] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.260s	user 0.014s	sys 0.003s
I20260709 12:20:23.003392 31866 ts_tablet_manager.cc:1434] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.102s	user 0.005s	sys 0.005s
I20260709 12:20:23.004815 31866 tablet_bootstrap.cc:492] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
W20260709 12:20:23.010043 31718 leader_election.cc:343] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:23.011538 31718 leader_election.cc:343] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.065428 31867 raft_consensus.cc:359] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.066210 31867 raft_consensus.cc:385] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.066521 31867 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.067174 31867 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.069519 31867 ts_tablet_manager.cc:1434] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.077s	user 0.005s	sys 0.000s
I20260709 12:20:23.070444 31867 tablet_bootstrap.cc:492] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:23.080497 31868 tablet_bootstrap.cc:654] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.122476 31866 tablet_bootstrap.cc:654] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.129074 31867 tablet_bootstrap.cc:654] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.131191 31573 catalog_manager.cc:5697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:23.133581 31573 catalog_manager.cc:5697] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:23.135317 31573 catalog_manager.cc:5697] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:23.140414 31866 tablet_bootstrap.cc:492] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:23.140980 31866 ts_tablet_manager.cc:1403] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.136s	user 0.013s	sys 0.000s
I20260709 12:20:23.140102 31573 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:23.147440 31868 tablet_bootstrap.cc:492] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:23.148061 31868 ts_tablet_manager.cc:1403] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.222s	user 0.012s	sys 0.021s
I20260709 12:20:23.150939 31868 raft_consensus.cc:359] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.153445 31868 raft_consensus.cc:385] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.153877 31868 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.153832 32165 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:23.154906 31868 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.156884 31868 ts_tablet_manager.cc:1434] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:20:23.157889 31868 tablet_bootstrap.cc:492] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:23.163990 31868 tablet_bootstrap.cc:654] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.167666 32210 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:23.168308 32210 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.170567 32210 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:23.173430 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:23.174587 31678 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:20:23.176456 31868 tablet_bootstrap.cc:492] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:23.176968 31868 ts_tablet_manager.cc:1403] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.019s	user 0.015s	sys 0.000s
I20260709 12:20:23.151356 31867 tablet_bootstrap.cc:492] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:23.178406 31867 ts_tablet_manager.cc:1403] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.108s	user 0.013s	sys 0.000s
I20260709 12:20:23.179780 31868 raft_consensus.cc:359] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.180513 31868 raft_consensus.cc:385] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.180792 31868 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.181107 31867 raft_consensus.cc:359] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.181823 31867 raft_consensus.cc:385] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.181588 31868 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.182082 31867 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.147789 31866 raft_consensus.cc:359] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.182775 31866 raft_consensus.cc:385] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.183061 31866 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.183580 31868 ts_tablet_manager.cc:1434] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 12:20:23.184561 31868 tablet_bootstrap.cc:492] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:23.190421 31868 tablet_bootstrap.cc:654] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.200377 31866 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.202940 31866 ts_tablet_manager.cc:1434] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.061s	user 0.006s	sys 0.002s
I20260709 12:20:23.203888 31866 tablet_bootstrap.cc:492] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:23.205669 31867 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.209996 31868 tablet_bootstrap.cc:492] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:23.210501 31868 ts_tablet_manager.cc:1403] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.026s	user 0.012s	sys 0.004s
I20260709 12:20:23.210744 31867 ts_tablet_manager.cc:1434] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.000s
I20260709 12:20:23.211673 31867 tablet_bootstrap.cc:492] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:23.213143 31868 raft_consensus.cc:359] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.213828 31868 raft_consensus.cc:385] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.214115 31868 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.214819 31868 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.229645 32210 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:20:23.244565 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 232157us
I20260709 12:20:23.247371 31866 tablet_bootstrap.cc:654] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.255918 31868 ts_tablet_manager.cc:1434] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.045s	user 0.010s	sys 0.001s
I20260709 12:20:23.256979 31868 tablet_bootstrap.cc:492] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:23.257571 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:23.258328 31750 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:23.266762 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:23.270395 32165 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.270887 32165 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.271222 32165 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.277652 31868 tablet_bootstrap.cc:654] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.289238 32165 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.289860 31822 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.300485 32165 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:23.301760 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:23.302809 31821 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.291113 31867 tablet_bootstrap.cc:654] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.308650 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:23.314898 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:23.320992 32235 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.324574 31868 tablet_bootstrap.cc:492] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:23.325019 31868 ts_tablet_manager.cc:1403] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.068s	user 0.013s	sys 0.003s
I20260709 12:20:23.314975 32143 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260709 12:20:23.311410 32109 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:23.328694 31866 tablet_bootstrap.cc:492] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:23.329154 31866 ts_tablet_manager.cc:1403] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.125s	user 0.011s	sys 0.018s
I20260709 12:20:23.327966 32109 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:23.331169 32109 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:23.334192 31717 leader_election.cc:343] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.321732 32235 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.334743 32162 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.348733 32162 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.373344 31821 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:23.376288 32192 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.376986 32192 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.388163 32109 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.395943 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:23.395088 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:23.404289 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:23.405258 32236 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:23.408718 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:23.409701 31675 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:20:23.411999 31642 leader_election.cc:343] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.401669 32192 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:23.417758 31753 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:23.419641 31866 raft_consensus.cc:359] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.420804 31866 raft_consensus.cc:385] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.421114 31866 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.421746 31866 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.393647 32109 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:20:23.435976 31718 consensus_peers.cc:597] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:23.438066 31868 raft_consensus.cc:359] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.439107 31868 raft_consensus.cc:385] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.439484 31868 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.431442 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:23.446508 31866 ts_tablet_manager.cc:1434] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.117s	user 0.008s	sys 0.000s
I20260709 12:20:23.447819 31866 tablet_bootstrap.cc:492] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:23.449815 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:23.446159 32190 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.452735 32190 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.454933 31868 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.455864 32162 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:23.457204 31868 ts_tablet_manager.cc:1434] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.132s	user 0.005s	sys 0.001s
W20260709 12:20:23.457093 31717 leader_election.cc:343] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.459388 31868 tablet_bootstrap.cc:492] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:23.415006 32236 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:23.468145 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:23.471232 31867 tablet_bootstrap.cc:492] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:23.471736 31867 ts_tablet_manager.cc:1403] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.260s	user 0.027s	sys 0.002s
I20260709 12:20:23.475224 32213 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:23.475368 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:23.535890 31956 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.536470 31956 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:23.478951 31718 leader_election.cc:343] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:23.542640 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63730.8us
I20260709 12:20:23.545995 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:23.552636 32238 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:23.477638 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:23.554481 32207 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:23.470163 32190 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:23.554746 32235 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:23.557075 31643 leader_election.cc:343] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.557515 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:23.558061 31922 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:23.559109 31922 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.559523 31922 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.561079 31956 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:23.564941 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:23.569054 31866 tablet_bootstrap.cc:654] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:23.569154 31717 leader_election.cc:343] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.568996 31922 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:23.571767 31643 leader_election.cc:343] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.572508 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:23.572539 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:23.573729 31752 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:20:23.573819 32004 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.582928 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:23.588865 32065 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.596180 32207 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:23.562803 31753 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:23.576454 31642 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:23.604910 31922 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:20:23.606774 31922 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.607350 31922 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:23.601683 31678 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:23.564220 32213 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:20:23.612632 32245 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.165s	user 0.000s	sys 0.003s
I20260709 12:20:23.574415 32004 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.604280 32065 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.614461 32143 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.622711 32065 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:23.625170 32143 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.622958 31922 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.627733 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:23.629894 32004 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:23.630234 32143 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:23.631141 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:23.631748 31752 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:23.632169 31717 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:23.632982 32207 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.633380 32207 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.633503 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:23.633780 32207 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.634212 31674 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:23.634325 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:20:23.632244 31791 leader_election.cc:343] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.632143 31922 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:23.485431 32237 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.122s
I20260709 12:20:23.641383 31642 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:20:23.643275 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 99558.6us
I20260709 12:20:23.643407 31752 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:20:23.643959 31922 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.644466 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:23.644930 31642 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:23.476131 32236 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.644513 31922 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.647600 31922 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.653333 32211 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.659652 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:23.666853 31922 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.660672 32211 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.659731 32208 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:23.658959 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:23.687873 31867 raft_consensus.cc:359] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.689203 31867 raft_consensus.cc:385] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.689617 31867 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.678232 32211 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.695554 31922 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:23.676283 31748 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
I20260709 12:20:23.700309 32237 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:23.704021 32237 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.705888 31866 tablet_bootstrap.cc:492] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:23.706418 31866 ts_tablet_manager.cc:1403] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.259s	user 0.014s	sys 0.012s
I20260709 12:20:23.707247 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:23.692946 31753 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 0.
W20260709 12:20:23.715472 31718 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:23.716948 31866 raft_consensus.cc:359] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.717852 31866 raft_consensus.cc:385] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:23.718178 31866 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:23.675571 32213 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.720484 31866 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.727793 31866 ts_tablet_manager.cc:1434] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:20:23.712862 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:23.712670 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:23.715852 32208 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:23.737821 31867 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.746096 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:23.749102 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:23.750463 31867 ts_tablet_manager.cc:1434] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.278s	user 0.010s	sys 0.002s
W20260709 12:20:23.754966 31643 leader_election.cc:343] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:23.757912 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:23.758518 31868 tablet_bootstrap.cc:654] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.758490 31677 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:23.730139 32235 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260709 12:20:23.739039 31822 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:23.766722 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:23.741595 32207 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.768632 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:23.775959 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:23.776546 32207 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:23.776687 31674 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 0.
I20260709 12:20:23.779479 31866 tablet_bootstrap.cc:492] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:23.783818 31751 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 1: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:23.784173 31867 tablet_bootstrap.cc:492] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:23.755537 32237 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:23.792105 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:23.792909 31678 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 1: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:20:23.792338 31717 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:23.769618 32165 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.816154 31642 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:23.834046 31922 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 12:20:23.835848 32252 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 12:20:23.832948 32165 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.857515 32165 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:23.861495 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 93032.1us
I20260709 12:20:23.858866 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:23.881076 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:23.789839 32208 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.911242 32251 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.911799 32251 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.912148 32251 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:20:23.915499 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47813.6us
I20260709 12:20:23.895143 32255 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:23.933488 32255 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:23.934083 32255 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:23.953351 31868 tablet_bootstrap.cc:492] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:23.954180 31868 ts_tablet_manager.cc:1403] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.495s	user 0.023s	sys 0.030s
I20260709 12:20:23.956393 32165 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:23.957576 31867 tablet_bootstrap.cc:654] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:23.958912 32165 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:24.012054 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 95064.3us
W20260709 12:20:24.039515 31718 leader_election.cc:343] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.040597 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:24.043313 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:24.043529 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:24.096266 31866 tablet_bootstrap.cc:654] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.096623 32004 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.101876 31673 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.108167 32271 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:23.961690 32211 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.114743 31827 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 1: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:20:24.128005 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 84754.3us
I20260709 12:20:24.126974 32162 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:24.132103 32162 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:24.132700 32162 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.133317 32211 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:20:24.137508 31718 leader_election.cc:343] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:20:24.139353 31718 leader_election.cc:343] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.141266 31718 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:24.143644 32235 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 12:20:24.147372 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 276896us
I20260709 12:20:24.122696 32004 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.160717 32004 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:24.195654 31868 raft_consensus.cc:359] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.196811 31868 raft_consensus.cc:385] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.197244 31868 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.186169 31576 catalog_manager.cc:5697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:24.197161 32251 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.206472 32251 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:24.208379 31867 tablet_bootstrap.cc:492] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:24.209218 31867 ts_tablet_manager.cc:1403] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.425s	user 0.013s	sys 0.012s
I20260709 12:20:24.228229 31826 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.234007 32256 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.242797 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:24.204229 31868 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.250012 31868 ts_tablet_manager.cc:1434] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.295s	user 0.012s	sys 0.000s
I20260709 12:20:24.253001 31673 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:24.261662 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:24.262343 31868 tablet_bootstrap.cc:492] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:24.262496 31678 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.267858 31866 tablet_bootstrap.cc:492] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:24.268759 31866 ts_tablet_manager.cc:1403] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.490s	user 0.012s	sys 0.021s
I20260709 12:20:24.286520 31678 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:24.274247 31866 raft_consensus.cc:359] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.289192 31866 raft_consensus.cc:385] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.289484 31866 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.234711 32256 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.320067 31674 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.285701 31867 raft_consensus.cc:359] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.328053 31867 raft_consensus.cc:385] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.328169 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:24.328612 31867 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.329525 31673 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 1: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:24.329690 31867 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.298715 32255 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.338264 31866 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.341787 32255 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:20:24.343107 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 139711us
I20260709 12:20:24.343955 31867 ts_tablet_manager.cc:1434] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.134s	user 0.012s	sys 0.002s
I20260709 12:20:24.347096 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:24.349557 32235 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:24.350292 32235 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:24.350435 32256 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:24.352828 32235 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:24.354911 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 43769.9us
I20260709 12:20:24.359050 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:24.360658 31717 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:24.345340 31867 tablet_bootstrap.cc:492] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
W20260709 12:20:24.362774 31718 leader_election.cc:343] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.367641 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:24.368834 31824 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 1: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:24.377102 31866 ts_tablet_manager.cc:1434] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.108s	user 0.005s	sys 0.004s
I20260709 12:20:24.373549 32065 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:24.383533 32065 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:24.384660 32065 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.387015 31678 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:24.389298 31866 tablet_bootstrap.cc:492] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:24.410679 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:24.414203 31868 tablet_bootstrap.cc:654] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.414072 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:24.369848 32162 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.428679 32162 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:24.433678 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:24.435496 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:24.437750 31717 leader_election.cc:343] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.444856 32004 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:24.445863 32004 consensus_queue.cc:1243] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:20:24.446805 32004 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:24.447207 32004 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:24.448143 32004 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.449278 32004 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:20:24.455396 31791 leader_election.cc:343] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.474552 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:24.476012 31820 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 1: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:24.519258 32235 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:24.528376 31866 tablet_bootstrap.cc:654] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.530699 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:24.531376 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:24.531998 31822 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 1: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:24.532254 31753 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 1: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:24.533808 31753 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:24.558431 32190 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.563496 32190 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.565507 31867 tablet_bootstrap.cc:654] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.572418 31822 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:20:24.573012 31822 pending_rounds.cc:85] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 1
I20260709 12:20:24.573381 31822 pending_rounds.cc:107] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:20:24.573736 31822 raft_consensus.cc:2889] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:20:24.575683 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:24.570950 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:24.584492 32065 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260709 12:20:24.586846 31643 leader_election.cc:343] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.587703 31868 tablet_bootstrap.cc:492] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:24.566257 32190 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:24.588514 32255 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.589046 32255 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.592985 32190 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:24.588557 31868 ts_tablet_manager.cc:1403] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.327s	user 0.020s	sys 0.002s
I20260709 12:20:24.595068 32255 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:24.595631 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:24.600764 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:24.605554 32213 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.608763 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:24.611691 32258 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.622095 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:24.623454 32255 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.635016 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:24.647442 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:24.654017 31868 raft_consensus.cc:359] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.657117 31868 raft_consensus.cc:385] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.657538 31868 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.664659 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:24.670467 31827 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.670791 31753 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:20:24.679937 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:24.683689 32275 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.686725 31868 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.692503 31747 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:24.704977 31868 ts_tablet_manager.cc:1434] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.109s	user 0.004s	sys 0.006s
I20260709 12:20:24.706429 31868 tablet_bootstrap.cc:492] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Bootstrap starting.
I20260709 12:20:24.707580 32235 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:24.711140 31867 tablet_bootstrap.cc:492] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:24.712563 31867 ts_tablet_manager.cc:1403] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.368s	user 0.032s	sys 0.018s
I20260709 12:20:24.721024 31867 raft_consensus.cc:359] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.721961 31867 raft_consensus.cc:385] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.722358 31867 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.723094 31866 tablet_bootstrap.cc:492] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:24.723598 31866 ts_tablet_manager.cc:1403] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.335s	user 0.036s	sys 0.058s
I20260709 12:20:24.723400 31867 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.726265 31867 ts_tablet_manager.cc:1434] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.004s
I20260709 12:20:24.726428 31866 raft_consensus.cc:359] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.728173 31867 tablet_bootstrap.cc:492] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:24.728233 31866 raft_consensus.cc:385] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.728852 31866 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.732877 32075 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:24.732901 31866 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.734505 31791 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:24.735915 31866 ts_tablet_manager.cc:1434] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 12:20:24.737142 31866 tablet_bootstrap.cc:492] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:24.739219 31642 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:24.742795 32190 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:24.743191 32190 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:24.743525 32190 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.752496 31573 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 1 to 3, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 3 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:24.756466 32004 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.757005 32004 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.764778 31868 tablet_bootstrap.cc:654] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.765600 31867 tablet_bootstrap.cc:654] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:24.776228 32004 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:24.781531 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:24.786527 31677 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 1: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:24.793503 32271 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:24.794095 32271 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.796365 32271 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:24.799621 31624 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:24.801954 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:24.803356 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:24.804615 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:24.806842 31717 leader_election.cc:343] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.807034 31866 tablet_bootstrap.cc:654] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
W20260709 12:20:24.812366 31718 leader_election.cc:343] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.813071 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:24.827728 32238 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.819353 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:24.835423 32211 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.836903 32190 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.850507 32210 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.854997 32190 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:24.856735 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:20:24.858419 31643 leader_election.cc:343] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:24.879089 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:24.879894 31750 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:24.895572 32235 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:24.902186 32211 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:24.911372 32271 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:24.920940 31750 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:24.931545 32238 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:24.946441 31642 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:24.959220 31866 tablet_bootstrap.cc:492] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:24.959759 31866 ts_tablet_manager.cc:1403] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.223s	user 0.009s	sys 0.008s
I20260709 12:20:24.964247 32208 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:24.964838 32208 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:24.965585 32208 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.975248 31867 tablet_bootstrap.cc:492] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:24.975400 31868 tablet_bootstrap.cc:492] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: No bootstrap required, opened a new log
I20260709 12:20:24.975857 31867 ts_tablet_manager.cc:1403] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.248s	user 0.024s	sys 0.062s
I20260709 12:20:24.975910 31868 ts_tablet_manager.cc:1403] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Time spent bootstrapping tablet: real 0.270s	user 0.036s	sys 0.025s
I20260709 12:20:24.968109 31866 raft_consensus.cc:359] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.976336 31866 raft_consensus.cc:385] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.976719 31866 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.977959 31866 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.980074 31868 raft_consensus.cc:359] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.980795 31868 raft_consensus.cc:385] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.981081 31868 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.981024 31867 raft_consensus.cc:359] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.981886 31867 raft_consensus.cc:385] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:24.982574 31868 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.984138 31866 ts_tablet_manager.cc:1434] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.024s	user 0.008s	sys 0.000s
I20260709 12:20:24.982194 31867 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:24.991115 31867 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:24.993798 31866 tablet_bootstrap.cc:492] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:25.001279 31868 ts_tablet_manager.cc:1434] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.000s
I20260709 12:20:25.009400 31867 ts_tablet_manager.cc:1434] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.001s
I20260709 12:20:25.011126 31867 tablet_bootstrap.cc:492] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:25.015489 32275 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.016121 32275 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.024616 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:25.026440 31643 leader_election.cc:343] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:25.028604 32275 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:25.030344 32264 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:20:25.031710 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:25.032454 31750 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:25.033715 31642 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:25.035203 32275 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:25.035686 32275 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:25.036013 31866 tablet_bootstrap.cc:654] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.036197 32275 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.043624 31867 tablet_bootstrap.cc:654] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.044689 31574 catalog_manager.cc:5697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:20:25.075229 31867 tablet_bootstrap.cc:492] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:25.075768 31867 ts_tablet_manager.cc:1403] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.065s	user 0.013s	sys 0.007s
I20260709 12:20:25.078961 31867 raft_consensus.cc:359] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.079823 31867 raft_consensus.cc:385] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.080112 31867 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.080401 32275 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.080854 31867 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.083092 31867 ts_tablet_manager.cc:1434] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:20:25.083220 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:25.083796 31750 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.085405 31867 tablet_bootstrap.cc:492] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Bootstrap starting.
I20260709 12:20:25.087314 31573 catalog_manager.cc:5697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 1, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:25.090605 31750 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:25.091851 31642 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:25.092634 31867 tablet_bootstrap.cc:654] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.092841 32075 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:25.099246 32075 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:25.100149 32075 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.094295 31866 tablet_bootstrap.cc:492] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:25.101645 31866 ts_tablet_manager.cc:1403] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.108s	user 0.026s	sys 0.002s
I20260709 12:20:25.105033 31866 raft_consensus.cc:359] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.105741 31866 raft_consensus.cc:385] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.106035 31866 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.106674 31866 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.108906 31866 ts_tablet_manager.cc:1434] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:20:25.109898 31866 tablet_bootstrap.cc:492] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:25.095888 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:20:25.112432 31643 leader_election.cc:343] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:20:25.094875 32275 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:25.117362 31866 tablet_bootstrap.cc:654] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.155601 31866 tablet_bootstrap.cc:492] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:25.156159 31866 ts_tablet_manager.cc:1403] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.046s	user 0.008s	sys 0.007s
I20260709 12:20:25.159121 31866 raft_consensus.cc:359] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.159718 31866 raft_consensus.cc:385] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.159979 31866 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.160645 31866 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.162415 31866 ts_tablet_manager.cc:1434] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:25.163683 31866 tablet_bootstrap.cc:492] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:25.169309 31866 tablet_bootstrap.cc:654] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.175120 31576 catalog_manager.cc:5697] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:25.184962 31867 tablet_bootstrap.cc:492] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: No bootstrap required, opened a new log
I20260709 12:20:25.185868 31867 ts_tablet_manager.cc:1403] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent bootstrapping tablet: real 0.101s	user 0.018s	sys 0.023s
I20260709 12:20:25.189643 31867 raft_consensus.cc:359] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.190493 31867 raft_consensus.cc:385] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.190878 31867 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.191633 31867 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.194947 31867 ts_tablet_manager.cc:1434] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.003s
I20260709 12:20:25.210489 31866 tablet_bootstrap.cc:492] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:25.211030 31866 ts_tablet_manager.cc:1403] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.048s	user 0.006s	sys 0.024s
I20260709 12:20:25.214124 32275 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.214084 31866 raft_consensus.cc:359] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.214958 31866 raft_consensus.cc:385] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.214753 32275 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.215281 31866 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.216079 31866 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.217043 32275 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:25.218751 31866 ts_tablet_manager.cc:1434] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:20:25.219853 31866 tablet_bootstrap.cc:492] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Bootstrap starting.
I20260709 12:20:25.221828 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:25.229038 31866 tablet_bootstrap.cc:654] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Neither blocks nor log segments found. Creating new log.
I20260709 12:20:25.222589 31749 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:25.247946 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:25.248605 31827 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:25.251261 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:25.253950 31922 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:25.254312 31922 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:25.254580 31922 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.260226 31573 catalog_manager.cc:5697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:25.263598 32256 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.264123 32256 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.266449 32256 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:25.269505 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:25.270207 31752 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:25.271008 31866 tablet_bootstrap.cc:492] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: No bootstrap required, opened a new log
I20260709 12:20:25.271327 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:25.271474 31866 ts_tablet_manager.cc:1403] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Time spent bootstrapping tablet: real 0.052s	user 0.012s	sys 0.015s
I20260709 12:20:25.271916 31827 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 0.
I20260709 12:20:25.272332 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:25.273437 32256 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:20:25.273761 32256 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:25.274088 32256 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.274859 31866 raft_consensus.cc:359] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.275565 31866 raft_consensus.cc:385] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:20:25.275840 31866 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Initialized, Role: FOLLOWER
I20260709 12:20:25.276468 31866 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.278025 31866 ts_tablet_manager.cc:1434] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.000s
I20260709 12:20:25.282675 32256 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.285907 31922 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.285986 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:25.288523 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:25.289834 31747 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.291014 31922 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:25.295358 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:25.296010 31749 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.297233 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:25.297838 31826 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.301371 31827 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.303592 31826 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:25.304342 31749 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:25.304832 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:25.305840 31747 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:25.306022 32190 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:25.306574 32190 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:25.307763 32256 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:25.307808 32190 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.311228 31827 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:25.308571 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:25.314441 32275 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:20:25.315029 32275 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:25.315894 32275 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.333890 31573 catalog_manager.cc:5697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:25.349851 31573 catalog_manager.cc:5697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 1, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:25.360414 32190 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:25.369871 31827 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.378728 32190 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:25.379773 32210 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.380391 32210 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.382445 32213 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.383014 32213 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.384963 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:25.385123 32213 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:25.388960 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:25.389900 31747 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 1: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:25.391602 31749 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 1: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:20:25.392208 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:25.382476 32210 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:25.393639 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:25.394443 31673 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 1: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:20:25.395922 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:25.397053 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:25.397997 32210 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.399710 32213 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.405751 32210 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:25.407348 32213 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:25.536432 32190 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:25.549520 32275 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:25.553325 32275 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:25.553855 32275 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:25.556612 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:25.557490 31749 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 1: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:25.560174 32275 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:25.561504 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:25.562261 31827 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 1: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:25.563333 31643 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:25.564600 32190 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:20:25.571880 32190 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:20:25.753903 31922 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:25.769845 32190 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:20:25.835017 31624 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:20:25.889601 32275 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:25.900871 32190 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:26.084647 31922 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:26.085142 31922 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:26.086875 31922 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:26.087745 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:26.088121 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:26.088411 31747 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:26.088830 31823 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:26.089517 31642 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:26.090301 31922 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:26.090668 31922 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:26.090970 31922 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:26.099623 31922 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:26.102514 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:26.103190 31749 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:26.103920 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:26.104516 31826 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:26.109377 31922 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:26.111359 31826 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:20:26.111761 31749 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:20:26.112447 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:26.113189 31922 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:26.113591 31922 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:26.114194 31922 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:26.124732 31573 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 0 to 2, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 2 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:26.200764 32271 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:26.201315 32271 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:26.204674 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:26.205281 31678 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:26.205861 32271 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:26.206677 31717 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:26.207495 32271 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:26.207580 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:26.207873 32271 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:26.208261 32271 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:26.208261 31826 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:26.215446 32271 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:26.217468 32271 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:26.218017 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:26.218262 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:26.218829 31826 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:26.218842 31678 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:26.223898 31678 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:20:26.224666 31826 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:20:26.224887 31717 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:26.226210 32271 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:26.226748 32271 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:26.227665 32271 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:26.243567 31573 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 0 to 2, leader changed from <none> to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:26.554739 31823 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:26.556711 31922 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:26.567368 31749 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:26.573252 31922 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:26.676474 31677 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:26.677662 32271 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:26.689823 31826 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:20:26.691671 32271 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:20:26.702488 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:20:26.741161 32335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:26.741286 32333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:20:26.741669 32332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:20:26.742990 30503 server_base.cc:1061] running on GCE node
I20260709 12:20:26.744514 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:20:26.744819 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:20:26.745075 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599626745053 us; error 0 us; skew 500 ppm
I20260709 12:20:26.745865 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:20:26.749022 30503 webserver.cc:533] Webserver started at http://127.29.201.196:38339/ using document root <none> and password file <none>
I20260709 12:20:26.749501 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:20:26.749675 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:20:26.750218 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:20:26.751688 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-3-root/instance:
uuid: "5206fd0087404d2da964a5d985a04ff4"
format_stamp: "Formatted at 2026-07-09 12:20:26 on dist-test-slave-qgf5"
I20260709 12:20:26.757074 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:20:26.760637 32340 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:26.761700 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:20:26.762085 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-3-root
uuid: "5206fd0087404d2da964a5d985a04ff4"
format_stamp: "Formatted at 2026-07-09 12:20:26 on dist-test-slave-qgf5"
I20260709 12:20:26.762436 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:20:26.801878 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:20:26.803702 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:20:26.805805 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:20:26.808574 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:20:26.808849 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:26.809146 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:20:26.809363 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:20:26.904934 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.196:44855
I20260709 12:20:26.912775 32402 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.196:44855 every 8 connection(s)
I20260709 12:20:26.915150 30503 auto_leader_rebalancer-test.cc:439] add a tserver: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:26.935828 32403 heartbeater.cc:344] Connected to a master server at 127.29.201.254:37495
I20260709 12:20:26.936336 32403 heartbeater.cc:461] Registering TS with master...
I20260709 12:20:26.937202 32403 heartbeater.cc:507] Master 127.29.201.254:37495 requested a full tablet report, sending...
I20260709 12:20:26.939451 31576 ts_manager.cc:194] Registered new tserver with Master: 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:26.941315 31576 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:33936
I20260709 12:20:27.945423 32403 heartbeater.cc:499] Master 127.29.201.254:37495 was elected leader, sending a full tablet report...
I20260709 12:20:28.030821 31747 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:28.037523 31678 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:28.039156 32408 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:28.039546 31827 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:28.042477 32408 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:28.049909 32327 raft_consensus.cc:2955] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:28.060248 31678 raft_consensus.cc:2955] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
W20260709 12:20:28.076463 31718 consensus_peers.cc:597] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 945fd223ac334bce88917d1b9b4c72e1. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:28.076016 31576 catalog_manager.cc:5697] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:28.091739 31677 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:28.097539 31827 raft_consensus.cc:2955] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:28.117048 31749 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:28.120088 31922 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:28.128365 31827 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:28.134002 32407 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:20:28.145066 31643 consensus_peers.cc:597] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2c281519748458cb11517e3303d0a17. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:28.152405 32423 raft_consensus.cc:2955] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:28.154443 31827 raft_consensus.cc:2955] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:28.154537 31749 raft_consensus.cc:2955] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:28.172922 31677 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:28.176932 31576 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:28.182260 31827 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:28.182981 31643 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 181ddae071d549aab328e3c1d8553734. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:28.183380 31749 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:28.183971 32407 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:28.184934 32423 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:28.198182 32423 raft_consensus.cc:2955] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:28.201525 31827 raft_consensus.cc:2955] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:28.203137 31749 raft_consensus.cc:2955] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:28.214401 31677 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:28.222096 31576 catalog_manager.cc:5697] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:28.235546 31643 consensus_peers.cc:597] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6499e78dd87c4e05be32f5681957a6b0. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:28.235986 31823 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:28.236743 31747 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:28.238782 31922 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:28.240473 31922 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:28.256109 32407 raft_consensus.cc:2955] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:28.258129 31749 raft_consensus.cc:2955] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:28.261829 31827 raft_consensus.cc:2955] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:28.281610 31573 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:28.598088 32434 ts_tablet_manager.cc:933] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:28.600260 32434 tablet_copy_client.cc:323] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.194:46639
I20260709 12:20:28.614923 31763 tablet_copy_service.cc:140] P 039d66a6c85e43c3936bf5017501b344: Received BeginTabletCopySession request for tablet 945fd223ac334bce88917d1b9b4c72e1 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:57176)
I20260709 12:20:28.615688 31763 tablet_copy_service.cc:161] P 039d66a6c85e43c3936bf5017501b344: Beginning new tablet copy session on tablet 945fd223ac334bce88917d1b9b4c72e1 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:57176: session id = 5206fd0087404d2da964a5d985a04ff4-945fd223ac334bce88917d1b9b4c72e1
I20260709 12:20:28.626029 31763 tablet_copy_source_session.cc:215] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:28.630594 32434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 945fd223ac334bce88917d1b9b4c72e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:28.645895 32434 tablet_copy_client.cc:806] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:28.646540 32434 tablet_copy_client.cc:670] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:28.658816 32434 tablet_copy_client.cc:538] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:28.659263 32438 ts_tablet_manager.cc:933] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:28.688182 32434 tablet_bootstrap.cc:492] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:28.691078 32438 tablet_copy_client.cc:323] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.193:41935
I20260709 12:20:28.742054 32443 ts_tablet_manager.cc:933] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:28.761845 32434 tablet_bootstrap.cc:492] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:28.770119 32434 tablet_bootstrap.cc:492] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:28.773552 32434 ts_tablet_manager.cc:1403] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.086s	user 0.039s	sys 0.030s
I20260709 12:20:28.786808 32434 raft_consensus.cc:359] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:28.787787 32434 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:28.788518 32434 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:28.792171 32443 tablet_copy_client.cc:323] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.193:41935
I20260709 12:20:28.817094 31688 tablet_copy_service.cc:140] P b95aeeaf4c51442b8092cff1de674f2b: Received BeginTabletCopySession request for tablet 181ddae071d549aab328e3c1d8553734 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34010)
I20260709 12:20:28.817930 31688 tablet_copy_service.cc:161] P b95aeeaf4c51442b8092cff1de674f2b: Beginning new tablet copy session on tablet 181ddae071d549aab328e3c1d8553734 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34010: session id = 5206fd0087404d2da964a5d985a04ff4-181ddae071d549aab328e3c1d8553734
I20260709 12:20:28.823129 31687 tablet_copy_service.cc:140] P b95aeeaf4c51442b8092cff1de674f2b: Received BeginTabletCopySession request for tablet b2c281519748458cb11517e3303d0a17 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34010)
I20260709 12:20:28.824059 31687 tablet_copy_service.cc:161] P b95aeeaf4c51442b8092cff1de674f2b: Beginning new tablet copy session on tablet b2c281519748458cb11517e3303d0a17 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34010: session id = 5206fd0087404d2da964a5d985a04ff4-b2c281519748458cb11517e3303d0a17
I20260709 12:20:28.844161 32434 ts_tablet_manager.cc:1434] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.069s	user 0.028s	sys 0.024s
I20260709 12:20:28.847291 32447 ts_tablet_manager.cc:933] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:28.849792 31763 tablet_copy_service.cc:342] P 039d66a6c85e43c3936bf5017501b344: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-945fd223ac334bce88917d1b9b4c72e1 received from {username='slave'} at 127.0.0.1:57176
I20260709 12:20:28.850354 31763 tablet_copy_service.cc:434] P 039d66a6c85e43c3936bf5017501b344: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-945fd223ac334bce88917d1b9b4c72e1 on tablet 945fd223ac334bce88917d1b9b4c72e1 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:28.851132 31687 tablet_copy_source_session.cc:215] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:28.852906 31688 tablet_copy_source_session.cc:215] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:28.857919 32438 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 181ddae071d549aab328e3c1d8553734. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:28.861390 32447 tablet_copy_client.cc:323] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.193:41935
I20260709 12:20:28.865972 32443 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c281519748458cb11517e3303d0a17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:28.869920 31687 tablet_copy_service.cc:140] P b95aeeaf4c51442b8092cff1de674f2b: Received BeginTabletCopySession request for tablet 6499e78dd87c4e05be32f5681957a6b0 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34010)
I20260709 12:20:28.870834 31687 tablet_copy_service.cc:161] P b95aeeaf4c51442b8092cff1de674f2b: Beginning new tablet copy session on tablet 6499e78dd87c4e05be32f5681957a6b0 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34010: session id = 5206fd0087404d2da964a5d985a04ff4-6499e78dd87c4e05be32f5681957a6b0
I20260709 12:20:28.878067 31687 tablet_copy_source_session.cc:215] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:28.883469 32447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6499e78dd87c4e05be32f5681957a6b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:28.892177 32438 tablet_copy_client.cc:806] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:28.893261 32438 tablet_copy_client.cc:670] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:28.902619 32438 tablet_copy_client.cc:538] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:28.912977 32447 tablet_copy_client.cc:806] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:28.913741 32447 tablet_copy_client.cc:670] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:28.920598 32447 tablet_copy_client.cc:538] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:28.915220 32443 tablet_copy_client.cc:806] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:28.923554 32443 tablet_copy_client.cc:670] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:28.917160 32438 tablet_bootstrap.cc:492] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:28.930667 32447 tablet_bootstrap.cc:492] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:28.931172 32443 tablet_copy_client.cc:538] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:28.948833 32443 tablet_bootstrap.cc:492] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:28.978719 32447 tablet_bootstrap.cc:492] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:28.979684 32447 tablet_bootstrap.cc:492] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:28.980366 32447 ts_tablet_manager.cc:1403] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.050s	user 0.028s	sys 0.001s
I20260709 12:20:28.983630 32447 raft_consensus.cc:359] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:28.984573 32447 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:28.986058 32447 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:28.990272 32438 tablet_bootstrap.cc:492] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:28.992472 32378 raft_consensus.cc:1217] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:28.992465 32447 ts_tablet_manager.cc:1434] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.012s	user 0.008s	sys 0.000s
I20260709 12:20:29.058331 32443 tablet_bootstrap.cc:492] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:29.068288 32443 tablet_bootstrap.cc:492] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:29.069199 32443 ts_tablet_manager.cc:1403] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.121s	user 0.022s	sys 0.010s
I20260709 12:20:29.069847 31687 tablet_copy_service.cc:342] P b95aeeaf4c51442b8092cff1de674f2b: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-6499e78dd87c4e05be32f5681957a6b0 received from {username='slave'} at 127.0.0.1:34010
I20260709 12:20:29.070390 31687 tablet_copy_service.cc:434] P b95aeeaf4c51442b8092cff1de674f2b: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-6499e78dd87c4e05be32f5681957a6b0 on tablet 6499e78dd87c4e05be32f5681957a6b0 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:29.072381 32443 raft_consensus.cc:359] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:29.073355 32443 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
W20260709 12:20:29.055786 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 62497.6us
I20260709 12:20:29.038426 32438 tablet_bootstrap.cc:492] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:29.081247 32438 ts_tablet_manager.cc:1403] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.164s	user 0.021s	sys 0.004s
I20260709 12:20:29.085153 32438 raft_consensus.cc:359] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:29.087123 32438 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:29.087803 32438 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:29.090359 32438 ts_tablet_manager.cc:1434] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.004s
I20260709 12:20:29.074085 32443 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:29.092945 31687 tablet_copy_service.cc:342] P b95aeeaf4c51442b8092cff1de674f2b: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-181ddae071d549aab328e3c1d8553734 received from {username='slave'} at 127.0.0.1:34010
I20260709 12:20:29.093433 31687 tablet_copy_service.cc:434] P b95aeeaf4c51442b8092cff1de674f2b: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-181ddae071d549aab328e3c1d8553734 on tablet 181ddae071d549aab328e3c1d8553734 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:29.095686 32443 ts_tablet_manager.cc:1434] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.000s
I20260709 12:20:29.098897 31687 tablet_copy_service.cc:342] P b95aeeaf4c51442b8092cff1de674f2b: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-b2c281519748458cb11517e3303d0a17 received from {username='slave'} at 127.0.0.1:34010
I20260709 12:20:29.099387 31687 tablet_copy_service.cc:434] P b95aeeaf4c51442b8092cff1de674f2b: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-b2c281519748458cb11517e3303d0a17 on tablet b2c281519748458cb11517e3303d0a17 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:29.114388 32378 raft_consensus.cc:1217] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:29.188459 32378 raft_consensus.cc:1217] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:29.260790 32378 raft_consensus.cc:1217] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:29.385679 32235 raft_consensus.cc:1064] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:29.389748 32235 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:29.407099 32378 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:29.409240 31678 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:29.411377 31827 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:29.410121 32235 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:29.413581 32327 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:29.423897 32444 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:29.442842 32444 raft_consensus.cc:2955] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.444257 31677 raft_consensus.cc:2955] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.448618 31823 raft_consensus.cc:2955] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.457137 32378 raft_consensus.cc:2955] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.474678 31576 catalog_manager.cc:5697] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.514513 31749 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:29.520669 32378 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:29.521944 31823 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:29.521996 32440 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:29.523501 32235 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:29.530176 32444 raft_consensus.cc:2955] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.536722 32453 raft_consensus.cc:1064] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:29.537266 32377 raft_consensus.cc:2955] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.543998 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 945fd223ac334bce88917d1b9b4c72e1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:29.539940 32453 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:29.555699 31575 catalog_manager.cc:5697] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:29.550998 31823 raft_consensus.cc:2955] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.574926 32378 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:29.576215 31823 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:29.577567 31922 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:29.581048 31749 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:29.581550 32407 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:29.584373 32407 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:29.589237 31668 tablet_service.cc:1558] Processing DeleteTablet for tablet 945fd223ac334bce88917d1b9b4c72e1 with delete_type TABLET_DATA_TOMBSTONED (TS b95aeeaf4c51442b8092cff1de674f2b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43180
I20260709 12:20:29.639981 32464 tablet_replica.cc:333] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:20:29.644505 32464 raft_consensus.cc:2243] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:29.645488 32464 raft_consensus.cc:2272] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:29.646224 32453 raft_consensus.cc:2955] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.649412 31752 raft_consensus.cc:2955] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.651484 32407 raft_consensus.cc:1064] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:29.654199 32407 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:29.659317 31823 raft_consensus.cc:2955] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.660982 32377 raft_consensus.cc:2955] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.684079 31575 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.710254 32464 ts_tablet_manager.cc:1916] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:29.718559 31678 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:29.729261 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57304.7us
I20260709 12:20:29.735924 31823 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:29.736456 32377 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:29.745107 32433 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:29.749858 31752 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:29.751673 32407 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:29.755995 32453 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:29.769771 32377 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:29.770049 31823 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:29.775298 32453 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:29.778975 32453 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:29.792352 32407 raft_consensus.cc:2955] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.801692 31823 raft_consensus.cc:2955] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:20:29.818135 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44992.9us
I20260709 12:20:29.798470 32464 ts_tablet_manager.cc:1929] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:29.831542 32464 log.cc:1199] T 945fd223ac334bce88917d1b9b4c72e1 P b95aeeaf4c51442b8092cff1de674f2b: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/wals/945fd223ac334bce88917d1b9b4c72e1
I20260709 12:20:29.830552 32377 raft_consensus.cc:2955] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.833381 31561 catalog_manager.cc:5028] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): tablet 945fd223ac334bce88917d1b9b4c72e1 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:29.847873 31573 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.850903 32453 raft_consensus.cc:2955] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.861318 31820 raft_consensus.cc:2955] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.868043 31747 raft_consensus.cc:2955] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.873512 31735 tablet_service.cc:1558] Processing DeleteTablet for tablet b2c281519748458cb11517e3303d0a17 with delete_type TABLET_DATA_TOMBSTONED (TS 039d66a6c85e43c3936bf5017501b344 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45150
I20260709 12:20:29.881987 32377 raft_consensus.cc:2955] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.897598 31922 raft_consensus.cc:1064] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:29.900182 31922 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:29.905220 31575 catalog_manager.cc:5697] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.916780 32377 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:29.919986 31827 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:29.919742 32453 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:29.923950 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b2c281519748458cb11517e3303d0a17 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:29.962227 32468 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:29.972003 31747 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:29.975143 32468 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:29.976012 32473 tablet_replica.cc:333] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:20:29.977129 32473 raft_consensus.cc:2243] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:29.977877 32473 raft_consensus.cc:2272] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:29.984891 32473 ts_tablet_manager.cc:1916] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:29.988900 31922 raft_consensus.cc:2955] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.992835 31827 raft_consensus.cc:2955] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:29.993160 31747 raft_consensus.cc:2955] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:30.002077 32378 raft_consensus.cc:2955] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:30.024781 31575 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:30.029907 32473 ts_tablet_manager.cc:1929] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:30.030514 32473 log.cc:1199] T b2c281519748458cb11517e3303d0a17 P 039d66a6c85e43c3936bf5017501b344: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/wals/b2c281519748458cb11517e3303d0a17
I20260709 12:20:30.039559 31561 catalog_manager.cc:5028] TS 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): tablet b2c281519748458cb11517e3303d0a17 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:30.079597 31674 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:30.086472 32378 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:30.088192 32475 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:30.088858 31747 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:30.093923 32475 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:30.094089 31674 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:30.100893 31753 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:30.096582 32475 raft_consensus.cc:2955] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:30.102514 32377 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:30.103147 31922 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:30.105233 31753 raft_consensus.cc:2955] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:30.106549 32433 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:30.110911 31922 raft_consensus.cc:2955] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:30.118046 31747 raft_consensus.cc:2955] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:30.121712 32377 raft_consensus.cc:2955] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:30.124943 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 6499e78dd87c4e05be32f5681957a6b0 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:30.117736 32378 raft_consensus.cc:2955] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:30.135901 31575 catalog_manager.cc:5697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:30.137290 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 181ddae071d549aab328e3c1d8553734 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:30.138854 31575 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:30.162887 31811 tablet_service.cc:1558] Processing DeleteTablet for tablet 181ddae071d549aab328e3c1d8553734 with delete_type TABLET_DATA_TOMBSTONED (TS 4d48c83c761f48079bc6ea176649dda0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41192
I20260709 12:20:30.165526 31812 tablet_service.cc:1558] Processing DeleteTablet for tablet 6499e78dd87c4e05be32f5681957a6b0 with delete_type TABLET_DATA_TOMBSTONED (TS 4d48c83c761f48079bc6ea176649dda0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41192
I20260709 12:20:30.189121 32480 tablet_replica.cc:333] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:20:30.193099 32480 raft_consensus.cc:2243] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:30.194031 32480 raft_consensus.cc:2272] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:30.201618 32480 ts_tablet_manager.cc:1916] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:30.220528 32480 ts_tablet_manager.cc:1929] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:30.221042 32480 log.cc:1199] T 181ddae071d549aab328e3c1d8553734 P 4d48c83c761f48079bc6ea176649dda0: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root/wals/181ddae071d549aab328e3c1d8553734
I20260709 12:20:30.222827 32480 tablet_replica.cc:333] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:20:30.222726 31562 catalog_manager.cc:5028] TS 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): tablet 181ddae071d549aab328e3c1d8553734 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:30.224036 32480 raft_consensus.cc:2243] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:30.224910 32480 raft_consensus.cc:2272] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:30.228500 32480 ts_tablet_manager.cc:1916] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:30.241127 32480 ts_tablet_manager.cc:1929] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:30.241580 32480 log.cc:1199] T 6499e78dd87c4e05be32f5681957a6b0 P 4d48c83c761f48079bc6ea176649dda0: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root/wals/6499e78dd87c4e05be32f5681957a6b0
I20260709 12:20:30.243046 31562 catalog_manager.cc:5028] TS 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): tablet 6499e78dd87c4e05be32f5681957a6b0 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:31.229378 31677 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:31.236550 31823 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:31.237151 31747 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:31.238041 32453 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:31.239707 32482 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:20:31.240875 31643 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b08faba9c13b4f0f85baf8b16c97d85b. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:31.248046 32453 raft_consensus.cc:2955] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:31.249938 31747 raft_consensus.cc:2955] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:31.249974 31823 raft_consensus.cc:2955] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:31.269989 31575 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 2 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:31.307060 31823 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:31.322441 31747 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:31.323359 31674 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:31.324366 32470 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:31.325407 32449 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:31.334568 32449 raft_consensus.cc:2955] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
W20260709 12:20:31.339715 31792 consensus_peers.cc:597] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e59f73cf80df4c20b6c17715783f2121. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:31.341168 31674 raft_consensus.cc:2955] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:31.345307 31747 raft_consensus.cc:2955] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:31.360041 31753 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:31.365478 31573 catalog_manager.cc:5697] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:31.380259 31677 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:31.384028 32235 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:31.400629 32235 raft_consensus.cc:2955] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:31.410693 31677 raft_consensus.cc:2955] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
W20260709 12:20:31.431821 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55679.3us
I20260709 12:20:31.428512 31574 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 2 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:31.438294 31753 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:31.459750 31823 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:20:31.503953 31718 consensus_peers.cc:597] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35c247463bef4f77990becbf6089be7d. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:20:31.504703 31718 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 436783e8e1cc49edbad793a31645e2d6. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:31.505540 32479 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260709 12:20:31.510061 31677 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:31.510682 31823 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:31.512025 32505 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:31.513044 32502 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:31.514365 31824 raft_consensus.cc:2955] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:31.532440 32479 raft_consensus.cc:2955] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:31.539489 31677 raft_consensus.cc:2955] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:31.558722 31823 raft_consensus.cc:2955] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:31.558314 31574 catalog_manager.cc:5697] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:31.816996 32510 ts_tablet_manager.cc:933] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:31.819165 32510 tablet_copy_client.cc:323] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.193:41935
I20260709 12:20:31.821092 31687 tablet_copy_service.cc:140] P b95aeeaf4c51442b8092cff1de674f2b: Received BeginTabletCopySession request for tablet b08faba9c13b4f0f85baf8b16c97d85b from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34010)
I20260709 12:20:31.821676 31687 tablet_copy_service.cc:161] P b95aeeaf4c51442b8092cff1de674f2b: Beginning new tablet copy session on tablet b08faba9c13b4f0f85baf8b16c97d85b from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34010: session id = 5206fd0087404d2da964a5d985a04ff4-b08faba9c13b4f0f85baf8b16c97d85b
I20260709 12:20:31.830192 31687 tablet_copy_source_session.cc:215] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:31.839097 32510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b08faba9c13b4f0f85baf8b16c97d85b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:31.861270 32510 tablet_copy_client.cc:806] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:31.861925 32510 tablet_copy_client.cc:670] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:31.867277 32510 tablet_copy_client.cc:538] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:31.877205 32510 tablet_bootstrap.cc:492] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:31.899935 32510 tablet_bootstrap.cc:492] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:31.900967 32510 tablet_bootstrap.cc:492] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:31.901678 32510 ts_tablet_manager.cc:1403] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.025s	user 0.013s	sys 0.010s
I20260709 12:20:31.905709 32510 raft_consensus.cc:359] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:31.906692 32510 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:31.907478 32510 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:31.919891 32513 ts_tablet_manager.cc:933] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:31.920224 32510 ts_tablet_manager.cc:1434] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.018s	user 0.010s	sys 0.010s
I20260709 12:20:31.922461 31687 tablet_copy_service.cc:342] P b95aeeaf4c51442b8092cff1de674f2b: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-b08faba9c13b4f0f85baf8b16c97d85b received from {username='slave'} at 127.0.0.1:34010
I20260709 12:20:31.922999 31687 tablet_copy_service.cc:434] P b95aeeaf4c51442b8092cff1de674f2b: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-b08faba9c13b4f0f85baf8b16c97d85b on tablet b08faba9c13b4f0f85baf8b16c97d85b with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:31.925493 32513 tablet_copy_client.cc:323] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.195:44645
I20260709 12:20:31.945771 31837 tablet_copy_service.cc:140] P 4d48c83c761f48079bc6ea176649dda0: Received BeginTabletCopySession request for tablet e59f73cf80df4c20b6c17715783f2121 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34642)
I20260709 12:20:31.946334 31837 tablet_copy_service.cc:161] P 4d48c83c761f48079bc6ea176649dda0: Beginning new tablet copy session on tablet e59f73cf80df4c20b6c17715783f2121 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34642: session id = 5206fd0087404d2da964a5d985a04ff4-e59f73cf80df4c20b6c17715783f2121
I20260709 12:20:31.956713 32510 ts_tablet_manager.cc:933] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:31.958107 31837 tablet_copy_source_session.cc:215] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:31.964994 32513 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59f73cf80df4c20b6c17715783f2121. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:31.971894 32510 tablet_copy_client.cc:323] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.194:46639
I20260709 12:20:31.975903 31763 tablet_copy_service.cc:140] P 039d66a6c85e43c3936bf5017501b344: Received BeginTabletCopySession request for tablet 436783e8e1cc49edbad793a31645e2d6 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:57176)
I20260709 12:20:31.976549 31763 tablet_copy_service.cc:161] P 039d66a6c85e43c3936bf5017501b344: Beginning new tablet copy session on tablet 436783e8e1cc49edbad793a31645e2d6 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:57176: session id = 5206fd0087404d2da964a5d985a04ff4-436783e8e1cc49edbad793a31645e2d6
I20260709 12:20:31.982702 32513 tablet_copy_client.cc:806] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:31.983394 32513 tablet_copy_client.cc:670] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:31.991189 32513 tablet_copy_client.cc:538] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:31.998411 31763 tablet_copy_source_session.cc:215] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:32.004763 32510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 436783e8e1cc49edbad793a31645e2d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:32.013011 32513 tablet_bootstrap.cc:492] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:32.019085 32520 ts_tablet_manager.cc:933] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:32.035953 32520 tablet_copy_client.cc:323] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.194:46639
I20260709 12:20:32.037788 31763 tablet_copy_service.cc:140] P 039d66a6c85e43c3936bf5017501b344: Received BeginTabletCopySession request for tablet 35c247463bef4f77990becbf6089be7d from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:57176)
I20260709 12:20:32.038447 31763 tablet_copy_service.cc:161] P 039d66a6c85e43c3936bf5017501b344: Beginning new tablet copy session on tablet 35c247463bef4f77990becbf6089be7d from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:57176: session id = 5206fd0087404d2da964a5d985a04ff4-35c247463bef4f77990becbf6089be7d
I20260709 12:20:32.063084 32513 tablet_bootstrap.cc:492] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:32.064249 32513 tablet_bootstrap.cc:492] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:32.065295 32513 ts_tablet_manager.cc:1403] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.053s	user 0.022s	sys 0.008s
I20260709 12:20:32.054154 32510 tablet_copy_client.cc:806] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:32.072376 32510 tablet_copy_client.cc:670] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:32.068995 32513 raft_consensus.cc:359] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:32.074285 32513 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:32.075114 32513 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:32.082274 32510 tablet_copy_client.cc:538] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:32.083174 31763 tablet_copy_source_session.cc:215] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:32.089148 32513 ts_tablet_manager.cc:1434] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.024s	user 0.009s	sys 0.000s
I20260709 12:20:32.090343 32520 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35c247463bef4f77990becbf6089be7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:32.091055 31837 tablet_copy_service.cc:342] P 4d48c83c761f48079bc6ea176649dda0: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-e59f73cf80df4c20b6c17715783f2121 received from {username='slave'} at 127.0.0.1:34642
I20260709 12:20:32.091583 31837 tablet_copy_service.cc:434] P 4d48c83c761f48079bc6ea176649dda0: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-e59f73cf80df4c20b6c17715783f2121 on tablet e59f73cf80df4c20b6c17715783f2121 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:32.110210 32520 tablet_copy_client.cc:806] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:32.110894 32520 tablet_copy_client.cc:670] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:32.120504 32520 tablet_copy_client.cc:538] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:32.134778 32510 tablet_bootstrap.cc:492] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:32.146207 32520 tablet_bootstrap.cc:492] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:32.182701 32520 tablet_bootstrap.cc:492] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:32.183729 32520 tablet_bootstrap.cc:492] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:32.184481 32520 ts_tablet_manager.cc:1403] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.039s	user 0.011s	sys 0.017s
I20260709 12:20:32.187601 32520 raft_consensus.cc:359] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:32.188541 32520 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:32.189849 32510 tablet_bootstrap.cc:492] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:32.189277 32520 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:32.190963 32510 tablet_bootstrap.cc:492] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:32.191715 32510 ts_tablet_manager.cc:1403] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.057s	user 0.032s	sys 0.020s
I20260709 12:20:32.192466 32520 ts_tablet_manager.cc:1434] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:20:32.194829 32510 raft_consensus.cc:359] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:32.200510 32510 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:32.201421 32510 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:32.204335 32510 ts_tablet_manager.cc:1434] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.005s
I20260709 12:20:32.206578 31763 tablet_copy_service.cc:342] P 039d66a6c85e43c3936bf5017501b344: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-436783e8e1cc49edbad793a31645e2d6 received from {username='slave'} at 127.0.0.1:57176
I20260709 12:20:32.207103 31763 tablet_copy_service.cc:434] P 039d66a6c85e43c3936bf5017501b344: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-436783e8e1cc49edbad793a31645e2d6 on tablet 436783e8e1cc49edbad793a31645e2d6 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:32.208079 31762 tablet_copy_service.cc:342] P 039d66a6c85e43c3936bf5017501b344: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-35c247463bef4f77990becbf6089be7d received from {username='slave'} at 127.0.0.1:57176
I20260709 12:20:32.208478 31762 tablet_copy_service.cc:434] P 039d66a6c85e43c3936bf5017501b344: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-35c247463bef4f77990becbf6089be7d on tablet 35c247463bef4f77990becbf6089be7d with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:32.309384 32378 raft_consensus.cc:1217] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 12:20:32.332669 32378 raft_consensus.cc:1217] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:32.387713 32378 raft_consensus.cc:1217] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:32.509231 32378 raft_consensus.cc:1217] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2]   Dedup: 2.2->[]
I20260709 12:20:32.706602 32475 raft_consensus.cc:1064] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:32.709157 32475 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:32.716753 31820 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:32.717829 32378 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEARNER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:32.720049 32527 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:32.720481 31753 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:32.722508 32482 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:32.723580 32527 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:32.738757 32527 raft_consensus.cc:2955] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:32.741544 31820 raft_consensus.cc:2955] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:32.742093 32378 raft_consensus.cc:2955] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:32.752173 31753 raft_consensus.cc:2955] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:32.760586 31574 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:32.796259 31674 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:32.802357 31823 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:32.802414 32378 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:32.804865 32527 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:32.806337 32453 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:32.818776 32527 raft_consensus.cc:2955] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:32.821892 32378 raft_consensus.cc:2955] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:32.821887 31823 raft_consensus.cc:2955] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:32.830826 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b08faba9c13b4f0f85baf8b16c97d85b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:32.847419 31576 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New cstate: current_term: 2 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:32.863094 31735 tablet_service.cc:1558] Processing DeleteTablet for tablet b08faba9c13b4f0f85baf8b16c97d85b with delete_type TABLET_DATA_TOMBSTONED (TS 039d66a6c85e43c3936bf5017501b344 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45150
I20260709 12:20:32.865105 32535 tablet_replica.cc:333] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:20:32.869225 32535 raft_consensus.cc:2243] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:32.872788 32525 raft_consensus.cc:1064] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:32.875314 32535 raft_consensus.cc:2272] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:32.879040 32535 ts_tablet_manager.cc:1916] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:32.892081 32470 raft_consensus.cc:1064] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:32.902897 32525 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:32.927213 32378 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:32.926607 32470 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:32.928527 31823 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:32.930342 32505 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:32.932948 32505 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:32.935361 32535 ts_tablet_manager.cc:1929] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260709 12:20:32.937093 32535 log.cc:1199] T b08faba9c13b4f0f85baf8b16c97d85b P 039d66a6c85e43c3936bf5017501b344: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/wals/b08faba9c13b4f0f85baf8b16c97d85b
I20260709 12:20:32.941751 31561 catalog_manager.cc:5028] TS 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): tablet b08faba9c13b4f0f85baf8b16c97d85b (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:32.948341 32377 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:32.950023 32470 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:32.955114 31674 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:32.956972 32479 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:32.970062 31753 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:32.970783 32505 raft_consensus.cc:2955] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:32.976317 31677 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:32.981945 32470 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:32.991484 32536 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:32.989024 32378 raft_consensus.cc:2955] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.014487 31576 catalog_manager.cc:5697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.022918 32470 raft_consensus.cc:2955] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.025171 31677 raft_consensus.cc:2955] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.045730 32377 raft_consensus.cc:2955] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.050098 31674 raft_consensus.cc:2955] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.074029 31823 raft_consensus.cc:2955] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.086159 31747 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:33.087590 31575 catalog_manager.cc:5697] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.088943 32479 raft_consensus.cc:1064] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:33.090811 31753 raft_consensus.cc:2955] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.096488 32479 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:33.103217 31677 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:33.105093 32505 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:33.109292 32377 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:33.112921 32502 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:33.126920 32377 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEARNER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:33.129163 32502 raft_consensus.cc:2955] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.127084 31674 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:33.130803 32547 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:33.135421 32525 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:33.141742 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 436783e8e1cc49edbad793a31645e2d6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:33.144526 32377 raft_consensus.cc:2955] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.145327 31820 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:33.148308 31823 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260709 12:20:33.149935 31678 raft_consensus.cc:2955] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.155138 32376 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:33.155743 31753 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:33.157313 32545 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:33.160569 32470 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:33.174561 31576 catalog_manager.cc:5697] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.183076 32537 raft_consensus.cc:2955] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.188637 32376 raft_consensus.cc:2955] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.191237 32547 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:33.212080 31749 raft_consensus.cc:2955] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.217432 31811 tablet_service.cc:1558] Processing DeleteTablet for tablet 436783e8e1cc49edbad793a31645e2d6 with delete_type TABLET_DATA_TOMBSTONED (TS 4d48c83c761f48079bc6ea176649dda0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41192
I20260709 12:20:33.220104 32553 tablet_replica.cc:333] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
W20260709 12:20:33.220705 32552 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:20:33.221189 32553 raft_consensus.cc:2243] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:33.221915 32553 raft_consensus.cc:2272] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:33.229843 32553 ts_tablet_manager.cc:1916] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:33.234844 31562 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet e59f73cf80df4c20b6c17715783f2121 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:33.236639 32502 raft_consensus.cc:2955] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.240304 31576 catalog_manager.cc:5697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:33.239907 32378 raft_consensus.cc:2955] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.256544 31668 tablet_service.cc:1558] Processing DeleteTablet for tablet e59f73cf80df4c20b6c17715783f2121 with delete_type TABLET_DATA_TOMBSTONED (TS b95aeeaf4c51442b8092cff1de674f2b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43180
I20260709 12:20:33.283902 32555 tablet_replica.cc:333] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:20:33.281544 31576 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.295678 32555 raft_consensus.cc:2243] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:33.296599 32555 raft_consensus.cc:2272] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:33.300216 32555 ts_tablet_manager.cc:1916] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:33.308085 31823 raft_consensus.cc:2955] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.310351 31674 raft_consensus.cc:2955] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.314195 32553 ts_tablet_manager.cc:1929] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:33.315388 32553 log.cc:1199] T 436783e8e1cc49edbad793a31645e2d6 P 4d48c83c761f48079bc6ea176649dda0: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root/wals/436783e8e1cc49edbad793a31645e2d6
I20260709 12:20:33.342145 31562 catalog_manager.cc:5028] TS 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): tablet 436783e8e1cc49edbad793a31645e2d6 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:33.361600 31749 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:33.367275 32378 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:33.370265 31824 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260709 12:20:33.370275 32502 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:33.372335 32502 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:33.376225 32555 ts_tablet_manager.cc:1929] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:33.378175 32555 log.cc:1199] T e59f73cf80df4c20b6c17715783f2121 P b95aeeaf4c51442b8092cff1de674f2b: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/wals/e59f73cf80df4c20b6c17715783f2121
I20260709 12:20:33.379829 32502 raft_consensus.cc:2955] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.383709 31561 catalog_manager.cc:5028] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): tablet e59f73cf80df4c20b6c17715783f2121 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:33.385452 31824 raft_consensus.cc:2955] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.386754 32378 raft_consensus.cc:2955] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.402580 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 35c247463bef4f77990becbf6089be7d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:33.403957 31576 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New cstate: current_term: 2 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:33.420339 31668 tablet_service.cc:1558] Processing DeleteTablet for tablet 35c247463bef4f77990becbf6089be7d with delete_type TABLET_DATA_TOMBSTONED (TS b95aeeaf4c51442b8092cff1de674f2b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43180
I20260709 12:20:33.421562 32555 tablet_replica.cc:333] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:20:33.422545 32555 raft_consensus.cc:2243] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:33.423277 32555 raft_consensus.cc:2272] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:33.426919 32555 ts_tablet_manager.cc:1916] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:33.442129 32555 ts_tablet_manager.cc:1929] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260709 12:20:33.442481 32555 log.cc:1199] T 35c247463bef4f77990becbf6089be7d P b95aeeaf4c51442b8092cff1de674f2b: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/wals/35c247463bef4f77990becbf6089be7d
I20260709 12:20:33.444128 31561 catalog_manager.cc:5028] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): tablet 35c247463bef4f77990becbf6089be7d (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:34.512984 31677 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:34.520838 31820 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:34.521641 31752 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:20:34.522334 31643 consensus_peers.cc:597] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45588632b8d64f9fb9c2e2ba18b20546. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:34.523314 32561 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:34.524555 32527 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:34.535380 32527 raft_consensus.cc:2955] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:34.536965 31752 raft_consensus.cc:2955] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:34.537881 31820 raft_consensus.cc:2955] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:34.552704 31573 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 3, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 3 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:34.561488 31823 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:34.594332 31752 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:34.595945 31674 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:34.598169 32545 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:34.599342 32537 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:20:34.608654 31792 consensus_peers.cc:597] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d30338be26ae44ff8e627f7b8a99036c. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:34.620231 32537 raft_consensus.cc:2955] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:34.622006 31677 raft_consensus.cc:2955] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:34.631309 31752 raft_consensus.cc:2955] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:34.642107 31823 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:34.646255 31576 catalog_manager.cc:5697] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:34.660948 31749 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:34.666639 31677 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:34.667016 31792 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5dfbfce13484fc0b1feba45daaf8247. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:34.667551 32545 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:34.676258 32545 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:34.679147 32571 raft_consensus.cc:2955] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:34.683584 31752 raft_consensus.cc:2955] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:34.697897 31749 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:34.700392 31677 raft_consensus.cc:2955] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:34.702085 31576 catalog_manager.cc:5697] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:34.712267 31674 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:34.713181 31718 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70ada7cfaa154c8c9cc927936b46c30a. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:34.715440 31823 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:34.718024 32502 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:20:34.721900 32547 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:34.736748 32579 raft_consensus.cc:2955] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:34.742347 31677 raft_consensus.cc:2955] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:34.759279 31573 catalog_manager.cc:5697] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:34.763983 31820 raft_consensus.cc:2955] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:35.035319 32582 ts_tablet_manager.cc:933] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:35.040304 32582 tablet_copy_client.cc:323] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.193:41935
I20260709 12:20:35.042969 31687 tablet_copy_service.cc:140] P b95aeeaf4c51442b8092cff1de674f2b: Received BeginTabletCopySession request for tablet 45588632b8d64f9fb9c2e2ba18b20546 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34010)
I20260709 12:20:35.043627 31687 tablet_copy_service.cc:161] P b95aeeaf4c51442b8092cff1de674f2b: Beginning new tablet copy session on tablet 45588632b8d64f9fb9c2e2ba18b20546 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34010: session id = 5206fd0087404d2da964a5d985a04ff4-45588632b8d64f9fb9c2e2ba18b20546
I20260709 12:20:35.053857 31687 tablet_copy_source_session.cc:215] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:35.058301 32582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45588632b8d64f9fb9c2e2ba18b20546. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:35.080549 32582 tablet_copy_client.cc:806] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:35.081240 32582 tablet_copy_client.cc:670] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:35.085526 32582 tablet_copy_client.cc:538] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:35.095415 32582 tablet_bootstrap.cc:492] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:35.132834 32582 tablet_bootstrap.cc:492] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:35.134003 32582 tablet_bootstrap.cc:492] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:35.134860 32582 ts_tablet_manager.cc:1403] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.040s	user 0.012s	sys 0.015s
I20260709 12:20:35.138257 32582 raft_consensus.cc:359] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:35.139292 32582 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:35.139766 32586 ts_tablet_manager.cc:933] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:35.140123 32582 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:35.147590 32586 tablet_copy_client.cc:323] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.194:46639
I20260709 12:20:35.148166 32582 ts_tablet_manager.cc:1434] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.001s
I20260709 12:20:35.150480 31762 tablet_copy_service.cc:140] P 039d66a6c85e43c3936bf5017501b344: Received BeginTabletCopySession request for tablet 70ada7cfaa154c8c9cc927936b46c30a from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:57176)
I20260709 12:20:35.151206 31762 tablet_copy_service.cc:161] P 039d66a6c85e43c3936bf5017501b344: Beginning new tablet copy session on tablet 70ada7cfaa154c8c9cc927936b46c30a from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:57176: session id = 5206fd0087404d2da964a5d985a04ff4-70ada7cfaa154c8c9cc927936b46c30a
I20260709 12:20:35.152333 31687 tablet_copy_service.cc:342] P b95aeeaf4c51442b8092cff1de674f2b: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-45588632b8d64f9fb9c2e2ba18b20546 received from {username='slave'} at 127.0.0.1:34010
I20260709 12:20:35.152851 31687 tablet_copy_service.cc:434] P b95aeeaf4c51442b8092cff1de674f2b: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-45588632b8d64f9fb9c2e2ba18b20546 on tablet 45588632b8d64f9fb9c2e2ba18b20546 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:35.166054 31762 tablet_copy_source_session.cc:215] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:35.171469 32586 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70ada7cfaa154c8c9cc927936b46c30a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:35.202030 32586 tablet_copy_client.cc:806] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:35.202757 32586 tablet_copy_client.cc:670] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:35.210373 32586 tablet_copy_client.cc:538] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:35.215549 32582 ts_tablet_manager.cc:933] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:35.218379 32582 tablet_copy_client.cc:323] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.195:44645
I20260709 12:20:35.220500 31837 tablet_copy_service.cc:140] P 4d48c83c761f48079bc6ea176649dda0: Received BeginTabletCopySession request for tablet d30338be26ae44ff8e627f7b8a99036c from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34642)
I20260709 12:20:35.221064 31837 tablet_copy_service.cc:161] P 4d48c83c761f48079bc6ea176649dda0: Beginning new tablet copy session on tablet d30338be26ae44ff8e627f7b8a99036c from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34642: session id = 5206fd0087404d2da964a5d985a04ff4-d30338be26ae44ff8e627f7b8a99036c
I20260709 12:20:35.229199 31837 tablet_copy_source_session.cc:215] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:35.231259 32586 tablet_bootstrap.cc:492] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:35.232215 32582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d30338be26ae44ff8e627f7b8a99036c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:35.249433 32582 tablet_copy_client.cc:806] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:35.250352 32582 tablet_copy_client.cc:670] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:35.257702 32586 tablet_bootstrap.cc:492] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:35.257831 32582 tablet_copy_client.cc:538] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:35.258836 32586 tablet_bootstrap.cc:492] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:35.259644 32586 ts_tablet_manager.cc:1403] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.029s	user 0.021s	sys 0.005s
I20260709 12:20:35.264128 32591 ts_tablet_manager.cc:933] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:35.267099 32586 raft_consensus.cc:359] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:35.268440 32586 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:35.269447 32586 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:35.272120 32582 tablet_bootstrap.cc:492] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:35.272881 32591 tablet_copy_client.cc:323] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.195:44645
I20260709 12:20:35.275496 31837 tablet_copy_service.cc:140] P 4d48c83c761f48079bc6ea176649dda0: Received BeginTabletCopySession request for tablet a5dfbfce13484fc0b1feba45daaf8247 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34642)
I20260709 12:20:35.276078 31837 tablet_copy_service.cc:161] P 4d48c83c761f48079bc6ea176649dda0: Beginning new tablet copy session on tablet a5dfbfce13484fc0b1feba45daaf8247 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34642: session id = 5206fd0087404d2da964a5d985a04ff4-a5dfbfce13484fc0b1feba45daaf8247
I20260709 12:20:35.277552 32586 ts_tablet_manager.cc:1434] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.018s	user 0.007s	sys 0.000s
I20260709 12:20:35.279358 31762 tablet_copy_service.cc:342] P 039d66a6c85e43c3936bf5017501b344: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-70ada7cfaa154c8c9cc927936b46c30a received from {username='slave'} at 127.0.0.1:57176
I20260709 12:20:35.279850 31762 tablet_copy_service.cc:434] P 039d66a6c85e43c3936bf5017501b344: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-70ada7cfaa154c8c9cc927936b46c30a on tablet 70ada7cfaa154c8c9cc927936b46c30a with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:35.295396 31837 tablet_copy_source_session.cc:215] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:35.299378 32591 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5dfbfce13484fc0b1feba45daaf8247. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:35.308727 32582 tablet_bootstrap.cc:492] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:35.310045 32582 tablet_bootstrap.cc:492] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:35.310945 32582 ts_tablet_manager.cc:1403] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.039s	user 0.019s	sys 0.006s
I20260709 12:20:35.318022 32582 raft_consensus.cc:359] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:35.320361 32582 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:35.321170 32582 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:35.324041 32582 ts_tablet_manager.cc:1434] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.001s
I20260709 12:20:35.325194 32591 tablet_copy_client.cc:806] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:35.325906 32591 tablet_copy_client.cc:670] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:35.326047 31837 tablet_copy_service.cc:342] P 4d48c83c761f48079bc6ea176649dda0: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-d30338be26ae44ff8e627f7b8a99036c received from {username='slave'} at 127.0.0.1:34642
I20260709 12:20:35.326556 31837 tablet_copy_service.cc:434] P 4d48c83c761f48079bc6ea176649dda0: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-d30338be26ae44ff8e627f7b8a99036c on tablet d30338be26ae44ff8e627f7b8a99036c with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:35.331226 32591 tablet_copy_client.cc:538] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:35.340837 32591 tablet_bootstrap.cc:492] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:35.366341 32591 tablet_bootstrap.cc:492] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:35.367417 32591 tablet_bootstrap.cc:492] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:35.368127 32591 ts_tablet_manager.cc:1403] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.028s	user 0.020s	sys 0.005s
I20260709 12:20:35.371657 32591 raft_consensus.cc:359] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:35.372678 32591 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:35.373679 32591 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:35.376187 32591 ts_tablet_manager.cc:1434] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:20:35.378832 31837 tablet_copy_service.cc:342] P 4d48c83c761f48079bc6ea176649dda0: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-a5dfbfce13484fc0b1feba45daaf8247 received from {username='slave'} at 127.0.0.1:34642
I20260709 12:20:35.379336 31837 tablet_copy_service.cc:434] P 4d48c83c761f48079bc6ea176649dda0: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-a5dfbfce13484fc0b1feba45daaf8247 on tablet a5dfbfce13484fc0b1feba45daaf8247 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:35.476220 32378 raft_consensus.cc:1217] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEARNER]: Deduplicated request from leader. Original: 3.2->[3.3-3.3]   Dedup: 3.3->[]
I20260709 12:20:35.623438 32378 raft_consensus.cc:1217] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:35.667001 32378 raft_consensus.cc:1217] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:35.844936 32378 raft_consensus.cc:1217] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:36.013939 32527 raft_consensus.cc:1064] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:36.016429 32527 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 1, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:36.026000 32378 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEARNER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:20:36.028384 31823 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:20:36.028904 31749 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:20:36.027844 32527 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:36.031558 32584 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:36.032433 32559 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:36.056427 32559 raft_consensus.cc:2955] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.068373 31749 raft_consensus.cc:2955] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.077800 31823 raft_consensus.cc:2955] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.095876 31573 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 3 to 4, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 3 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.113953 32578 raft_consensus.cc:1064] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:36.117056 32578 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:36.126353 32378 raft_consensus.cc:2955] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Committing config change with OpId 3.4: config changed from index 3 to 4, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.150472 31674 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:36.150875 31823 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:36.151937 32578 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:36.151934 32378 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:36.153203 32547 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:36.157892 32502 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:36.183239 31677 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:36.190011 32377 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:20:36.191802 32584 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:36.192328 31677 raft_consensus.cc:2955] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.201906 31820 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:20:36.185542 32578 raft_consensus.cc:2955] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.203452 32584 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:36.208725 32602 raft_consensus.cc:2955] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.218652 31820 raft_consensus.cc:2955] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.220304 32595 raft_consensus.cc:1064] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:36.223315 32595 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:36.224282 32378 raft_consensus.cc:2955] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Committing config change with OpId 3.5: config changed from index 4 to 5, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.235245 32377 raft_consensus.cc:2955] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.246527 32376 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:36.249522 32595 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:36.251188 31674 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:36.255527 31748 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:36.258723 32545 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:36.273854 31823 raft_consensus.cc:2955] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.278093 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 45588632b8d64f9fb9c2e2ba18b20546 with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:36.280215 31575 catalog_manager.cc:5697] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:36.284770 31574 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index 4 to 5, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New cstate: current_term: 3 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:36.299221 31735 tablet_service.cc:1558] Processing DeleteTablet for tablet 45588632b8d64f9fb9c2e2ba18b20546 with delete_type TABLET_DATA_TOMBSTONED (TS 039d66a6c85e43c3936bf5017501b344 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:45150
I20260709 12:20:36.309890 32571 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:36.318080 32571 raft_consensus.cc:1064] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:36.320920 32571 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:36.322614 31748 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:36.379786 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 56604.3us
I20260709 12:20:36.404939 32615 tablet_replica.cc:333] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:20:36.405296 32378 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:36.410509 32615 raft_consensus.cc:2243] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:36.411607 32615 raft_consensus.cc:2272] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:36.420461 32502 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:36.420940 32378 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:36.429865 32615 ts_tablet_manager.cc:1916] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:36.431437 32595 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:36.434127 31674 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:36.445470 32616 raft_consensus.cc:2955] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.447937 32502 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:36.455987 31674 raft_consensus.cc:2955] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.458002 32377 raft_consensus.cc:2955] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.445747 32620 raft_consensus.cc:2955] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.473325 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 70ada7cfaa154c8c9cc927936b46c30a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:36.488068 31672 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:36.493027 31748 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:36.494889 31672 raft_consensus.cc:2955] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.503578 31575 catalog_manager.cc:5697] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:36.513679 31747 raft_consensus.cc:2955] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.529152 32615 ts_tablet_manager.cc:1929] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.4
I20260709 12:20:36.529738 32615 log.cc:1199] T 45588632b8d64f9fb9c2e2ba18b20546 P 039d66a6c85e43c3936bf5017501b344: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/wals/45588632b8d64f9fb9c2e2ba18b20546
I20260709 12:20:36.519466 32595 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260709 12:20:36.537889 32621 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:36.549979 31561 catalog_manager.cc:5028] TS 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): tablet 45588632b8d64f9fb9c2e2ba18b20546 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:36.583165 31811 tablet_service.cc:1558] Processing DeleteTablet for tablet 70ada7cfaa154c8c9cc927936b46c30a with delete_type TABLET_DATA_TOMBSTONED (TS 4d48c83c761f48079bc6ea176649dda0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41192
I20260709 12:20:36.581329 32378 raft_consensus.cc:2955] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.609989 31576 catalog_manager.cc:5697] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:36.629743 32626 tablet_replica.cc:333] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:20:36.636116 32545 raft_consensus.cc:2955] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.640640 31673 raft_consensus.cc:2955] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.638674 32626 raft_consensus.cc:2243] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:36.647686 32626 raft_consensus.cc:2272] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:36.658550 31748 raft_consensus.cc:2955] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.663868 32626 ts_tablet_manager.cc:1916] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:36.669220 32378 raft_consensus.cc:2955] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.681632 31823 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:36.689657 32377 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:36.690987 31749 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:36.691896 32621 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:36.693370 32622 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:36.691879 31575 catalog_manager.cc:5697] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:36.715014 32545 raft_consensus.cc:2955] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.716882 32378 raft_consensus.cc:2955] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.721987 31748 raft_consensus.cc:2955] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.732388 31823 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:36.739578 32378 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:36.740795 31672 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:36.741021 31575 catalog_manager.cc:5697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.742379 32622 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:36.750604 32620 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:36.757016 31668 tablet_service.cc:1558] Processing DeleteTablet for tablet d30338be26ae44ff8e627f7b8a99036c with delete_type TABLET_DATA_TOMBSTONED (TS b95aeeaf4c51442b8092cff1de674f2b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43180
I20260709 12:20:36.745836 32626 ts_tablet_manager.cc:1929] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:36.760820 32626 log.cc:1199] T 70ada7cfaa154c8c9cc927936b46c30a P 4d48c83c761f48079bc6ea176649dda0: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root/wals/70ada7cfaa154c8c9cc927936b46c30a
I20260709 12:20:36.763815 32627 tablet_replica.cc:333] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:20:36.765228 32627 raft_consensus.cc:2243] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:36.766126 32627 raft_consensus.cc:2272] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:36.767308 32620 raft_consensus.cc:2955] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.769397 31562 catalog_manager.cc:5028] TS 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): tablet 70ada7cfaa154c8c9cc927936b46c30a (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:36.770241 31562 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet d30338be26ae44ff8e627f7b8a99036c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:36.771771 32627 ts_tablet_manager.cc:1916] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:36.773401 32378 raft_consensus.cc:2955] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.776872 31672 raft_consensus.cc:2955] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:36.779383 31562 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet a5dfbfce13484fc0b1feba45daaf8247 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:36.799304 31576 catalog_manager.cc:5697] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:36.808056 32627 ts_tablet_manager.cc:1929] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:36.808645 32627 log.cc:1199] T d30338be26ae44ff8e627f7b8a99036c P b95aeeaf4c51442b8092cff1de674f2b: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/wals/d30338be26ae44ff8e627f7b8a99036c
I20260709 12:20:36.812139 31561 catalog_manager.cc:5028] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): tablet d30338be26ae44ff8e627f7b8a99036c (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:36.820907 31735 tablet_service.cc:1558] Processing DeleteTablet for tablet a5dfbfce13484fc0b1feba45daaf8247 with delete_type TABLET_DATA_TOMBSTONED (TS 039d66a6c85e43c3936bf5017501b344 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45150
I20260709 12:20:36.821848 32615 tablet_replica.cc:333] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:20:36.822978 32615 raft_consensus.cc:2243] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:36.823839 32615 raft_consensus.cc:2272] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:36.827471 32615 ts_tablet_manager.cc:1916] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:36.854866 32615 ts_tablet_manager.cc:1929] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:36.855305 32615 log.cc:1199] T a5dfbfce13484fc0b1feba45daaf8247 P 039d66a6c85e43c3936bf5017501b344: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/wals/a5dfbfce13484fc0b1feba45daaf8247
I20260709 12:20:36.856747 31561 catalog_manager.cc:5028] TS 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): tablet a5dfbfce13484fc0b1feba45daaf8247 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:37.917196 31827 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
W20260709 12:20:37.925311 31792 consensus_peers.cc:597] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c34755b3b9d040f69be353e35d7431e7. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:37.927078 31748 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:37.927245 31672 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:37.928793 32545 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:37.929714 32620 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:37.940924 32545 raft_consensus.cc:2955] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:37.942480 31748 raft_consensus.cc:2955] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:37.949625 31673 raft_consensus.cc:2955] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:37.968935 31575 catalog_manager.cc:5697] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:37.985940 31673 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:38.004993 31823 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:38.007556 31643 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00854b2a70f24ef3a7b43d17477a0ccf. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:38.008222 31748 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:38.008931 32600 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:38.010214 32628 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 12:20:38.029748 32628 raft_consensus.cc:2955] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:38.034003 31823 raft_consensus.cc:2955] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:38.035454 31748 raft_consensus.cc:2955] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:38.051921 31748 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:38.052707 31574 catalog_manager.cc:5697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:38.062670 31823 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:38.064673 31673 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:38.065450 32629 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:38.066836 32608 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:20:38.069350 31718 consensus_peers.cc:597] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e7ad549735d4871af75658a91bb98c4. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:38.087288 32629 raft_consensus.cc:2955] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:38.091629 31823 raft_consensus.cc:2955] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:38.091771 31673 raft_consensus.cc:2955] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:38.105304 31827 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:38.112161 31574 catalog_manager.cc:5697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:38.123780 31748 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:38.124567 31792 consensus_peers.cc:597] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 04bdf51be7b94825a28028a5cfb96245. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:38.125370 32634 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:38.126128 31673 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:38.127872 32634 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:38.163991 32633 raft_consensus.cc:2955] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:38.168339 31749 raft_consensus.cc:2955] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:38.168833 31673 raft_consensus.cc:2955] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:38.216995 31576 catalog_manager.cc:5697] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:38.383558 32650 ts_tablet_manager.cc:933] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:38.385489 32650 tablet_copy_client.cc:323] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.195:44645
I20260709 12:20:38.387550 31837 tablet_copy_service.cc:140] P 4d48c83c761f48079bc6ea176649dda0: Received BeginTabletCopySession request for tablet c34755b3b9d040f69be353e35d7431e7 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34642)
I20260709 12:20:38.388144 31837 tablet_copy_service.cc:161] P 4d48c83c761f48079bc6ea176649dda0: Beginning new tablet copy session on tablet c34755b3b9d040f69be353e35d7431e7 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34642: session id = 5206fd0087404d2da964a5d985a04ff4-c34755b3b9d040f69be353e35d7431e7
I20260709 12:20:38.396787 31837 tablet_copy_source_session.cc:215] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:38.403944 32650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c34755b3b9d040f69be353e35d7431e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:38.420620 32652 ts_tablet_manager.cc:933] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:38.423195 32652 tablet_copy_client.cc:323] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.193:41935
I20260709 12:20:38.424788 31687 tablet_copy_service.cc:140] P b95aeeaf4c51442b8092cff1de674f2b: Received BeginTabletCopySession request for tablet 00854b2a70f24ef3a7b43d17477a0ccf from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34010)
I20260709 12:20:38.425408 31687 tablet_copy_service.cc:161] P b95aeeaf4c51442b8092cff1de674f2b: Beginning new tablet copy session on tablet 00854b2a70f24ef3a7b43d17477a0ccf from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34010: session id = 5206fd0087404d2da964a5d985a04ff4-00854b2a70f24ef3a7b43d17477a0ccf
I20260709 12:20:38.426280 32650 tablet_copy_client.cc:806] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:38.426975 32650 tablet_copy_client.cc:670] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:38.431699 32650 tablet_copy_client.cc:538] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:38.435900 31687 tablet_copy_source_session.cc:215] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:38.439621 32652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00854b2a70f24ef3a7b43d17477a0ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:38.442159 32650 tablet_bootstrap.cc:492] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:38.460421 32652 tablet_copy_client.cc:806] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:38.461131 32652 tablet_copy_client.cc:670] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:38.465723 32652 tablet_copy_client.cc:538] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:38.472600 32650 tablet_bootstrap.cc:492] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:38.473590 32650 tablet_bootstrap.cc:492] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:38.474313 32650 ts_tablet_manager.cc:1403] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.032s	user 0.019s	sys 0.004s
I20260709 12:20:38.475735 32652 tablet_bootstrap.cc:492] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:38.477496 32650 raft_consensus.cc:359] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:38.478418 32650 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:38.479241 32650 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:38.498232 32650 ts_tablet_manager.cc:1434] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.024s	user 0.005s	sys 0.003s
I20260709 12:20:38.502140 31837 tablet_copy_service.cc:342] P 4d48c83c761f48079bc6ea176649dda0: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-c34755b3b9d040f69be353e35d7431e7 received from {username='slave'} at 127.0.0.1:34642
I20260709 12:20:38.502686 31837 tablet_copy_service.cc:434] P 4d48c83c761f48079bc6ea176649dda0: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-c34755b3b9d040f69be353e35d7431e7 on tablet c34755b3b9d040f69be353e35d7431e7 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:38.516407 32652 tablet_bootstrap.cc:492] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:38.518123 32652 tablet_bootstrap.cc:492] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:38.519279 32652 ts_tablet_manager.cc:1403] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.044s	user 0.025s	sys 0.000s
I20260709 12:20:38.522748 32652 raft_consensus.cc:359] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:38.523675 32652 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:38.524422 32652 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:38.532819 32652 ts_tablet_manager.cc:1434] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.013s	user 0.001s	sys 0.004s
I20260709 12:20:38.536231 31687 tablet_copy_service.cc:342] P b95aeeaf4c51442b8092cff1de674f2b: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-00854b2a70f24ef3a7b43d17477a0ccf received from {username='slave'} at 127.0.0.1:34010
I20260709 12:20:38.536710 31687 tablet_copy_service.cc:434] P b95aeeaf4c51442b8092cff1de674f2b: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-00854b2a70f24ef3a7b43d17477a0ccf on tablet 00854b2a70f24ef3a7b43d17477a0ccf with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:38.600662 32652 ts_tablet_manager.cc:933] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:38.602411 32652 tablet_copy_client.cc:323] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.195:44645
I20260709 12:20:38.604003 31837 tablet_copy_service.cc:140] P 4d48c83c761f48079bc6ea176649dda0: Received BeginTabletCopySession request for tablet 04bdf51be7b94825a28028a5cfb96245 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34642)
I20260709 12:20:38.604661 31837 tablet_copy_service.cc:161] P 4d48c83c761f48079bc6ea176649dda0: Beginning new tablet copy session on tablet 04bdf51be7b94825a28028a5cfb96245 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34642: session id = 5206fd0087404d2da964a5d985a04ff4-04bdf51be7b94825a28028a5cfb96245
I20260709 12:20:38.612919 31837 tablet_copy_source_session.cc:215] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:38.616299 32652 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04bdf51be7b94825a28028a5cfb96245. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:38.629248 32652 tablet_copy_client.cc:806] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:38.630090 32652 tablet_copy_client.cc:670] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:38.635416 32652 tablet_copy_client.cc:538] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:38.641567 32650 ts_tablet_manager.cc:933] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:38.643625 32652 tablet_bootstrap.cc:492] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:38.643636 32650 tablet_copy_client.cc:323] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.194:46639
I20260709 12:20:38.646178 31762 tablet_copy_service.cc:140] P 039d66a6c85e43c3936bf5017501b344: Received BeginTabletCopySession request for tablet 6e7ad549735d4871af75658a91bb98c4 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:57176)
I20260709 12:20:38.646852 31762 tablet_copy_service.cc:161] P 039d66a6c85e43c3936bf5017501b344: Beginning new tablet copy session on tablet 6e7ad549735d4871af75658a91bb98c4 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:57176: session id = 5206fd0087404d2da964a5d985a04ff4-6e7ad549735d4871af75658a91bb98c4
I20260709 12:20:38.660585 31762 tablet_copy_source_session.cc:215] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:38.664506 32650 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e7ad549735d4871af75658a91bb98c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:38.667943 32652 tablet_bootstrap.cc:492] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:38.669039 32652 tablet_bootstrap.cc:492] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:38.670151 32652 ts_tablet_manager.cc:1403] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.027s	user 0.011s	sys 0.012s
I20260709 12:20:38.674827 32652 raft_consensus.cc:359] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:38.675815 32652 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:38.676548 32652 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:38.679468 32652 ts_tablet_manager.cc:1434] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.004s
I20260709 12:20:38.681180 31837 tablet_copy_service.cc:342] P 4d48c83c761f48079bc6ea176649dda0: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-04bdf51be7b94825a28028a5cfb96245 received from {username='slave'} at 127.0.0.1:34642
I20260709 12:20:38.681607 31837 tablet_copy_service.cc:434] P 4d48c83c761f48079bc6ea176649dda0: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-04bdf51be7b94825a28028a5cfb96245 on tablet 04bdf51be7b94825a28028a5cfb96245 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:38.692526 32650 tablet_copy_client.cc:806] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:38.693267 32650 tablet_copy_client.cc:670] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:38.698894 32650 tablet_copy_client.cc:538] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:38.712675 32650 tablet_bootstrap.cc:492] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:38.742624 32650 tablet_bootstrap.cc:492] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:38.743634 32650 tablet_bootstrap.cc:492] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:38.744336 32650 ts_tablet_manager.cc:1403] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.032s	user 0.020s	sys 0.008s
I20260709 12:20:38.746974 32650 raft_consensus.cc:359] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:38.747793 32650 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:38.748431 32650 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:38.750525 32650 ts_tablet_manager.cc:1434] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:20:38.752732 31762 tablet_copy_service.cc:342] P 039d66a6c85e43c3936bf5017501b344: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-6e7ad549735d4871af75658a91bb98c4 received from {username='slave'} at 127.0.0.1:57176
I20260709 12:20:38.753139 31762 tablet_copy_service.cc:434] P 039d66a6c85e43c3936bf5017501b344: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-6e7ad549735d4871af75658a91bb98c4 on tablet 6e7ad549735d4871af75658a91bb98c4 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:38.815572 32377 raft_consensus.cc:1217] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:38.962740 32377 raft_consensus.cc:1217] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:39.055944 32377 raft_consensus.cc:1217] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:39.168977 32377 raft_consensus.cc:1217] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:39.260922 32545 raft_consensus.cc:1064] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:39.262794 32545 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:39.275314 32377 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.277366 32637 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:39.280871 31749 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.282404 32633 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:39.289860 31672 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.299491 32545 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:39.304493 32662 raft_consensus.cc:2955] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.307232 31749 raft_consensus.cc:2955] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.323979 32377 raft_consensus.cc:2955] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.332489 31575 catalog_manager.cc:5697] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.343724 31672 raft_consensus.cc:2955] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.463485 32584 raft_consensus.cc:1064] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:39.477989 32584 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:39.578408 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 73499us
I20260709 12:20:39.600106 32662 raft_consensus.cc:1064] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:39.608141 31748 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.610724 32662 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:39.612584 32674 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260709 12:20:39.631129 32377 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.632817 32545 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:39.635088 31672 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.641825 32545 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:39.662372 31752 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.659296 31823 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:39.667886 32545 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:39.674516 32377 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:39.677171 32545 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:39.684553 32671 raft_consensus.cc:2955] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.685257 32633 raft_consensus.cc:2955] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.691694 32377 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.691160 31752 raft_consensus.cc:2955] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.699679 31748 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:39.702848 32375 raft_consensus.cc:2955] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.704413 32637 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:39.702828 31672 raft_consensus.cc:2955] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.714707 32673 raft_consensus.cc:1064] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:39.716336 32377 raft_consensus.cc:2955] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.722821 31827 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.722746 32673 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:39.719571 31752 raft_consensus.cc:2955] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.735642 32680 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:39.757550 31573 catalog_manager.cc:5697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:39.761736 32584 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:39.766599 31562 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c34755b3b9d040f69be353e35d7431e7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:39.778702 31575 catalog_manager.cc:5697] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.802165 31822 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:39.803596 31673 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.805274 32616 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:39.806607 31668 tablet_service.cc:1558] Processing DeleteTablet for tablet c34755b3b9d040f69be353e35d7431e7 with delete_type TABLET_DATA_TOMBSTONED (TS b95aeeaf4c51442b8092cff1de674f2b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43180
I20260709 12:20:39.814044 32377 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.818434 31749 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:39.822072 32671 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:39.823217 32378 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:39.831647 31822 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:39.833827 32545 raft_consensus.cc:2955] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.839185 32616 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:39.840515 32675 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:39.842131 31749 raft_consensus.cc:2955] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.845687 32671 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:39.854161 32377 raft_consensus.cc:2955] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.859468 32690 tablet_replica.cc:333] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:20:39.860342 31562 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 04bdf51be7b94825a28028a5cfb96245 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:39.864483 31573 catalog_manager.cc:5697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:39.880398 32690 raft_consensus.cc:2243] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:39.896786 31668 tablet_service.cc:1558] Processing DeleteTablet for tablet 04bdf51be7b94825a28028a5cfb96245 with delete_type TABLET_DATA_TOMBSTONED (TS b95aeeaf4c51442b8092cff1de674f2b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43180
I20260709 12:20:39.896458 32672 raft_consensus.cc:2955] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.899015 31673 raft_consensus.cc:2955] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.897397 32682 raft_consensus.cc:2955] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.903131 32690 raft_consensus.cc:2272] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:39.908074 31823 raft_consensus.cc:2955] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.931210 32377 raft_consensus.cc:2955] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.936468 32378 raft_consensus.cc:2955] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.946852 32690 ts_tablet_manager.cc:1916] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:39.953464 31575 catalog_manager.cc:5697] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.962555 31822 raft_consensus.cc:2955] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.976504 31573 catalog_manager.cc:5697] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.990214 31749 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:39.979223 31752 raft_consensus.cc:2955] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:39.999898 32377 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:40.000810 31673 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:40.002444 32672 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:40.006258 31677 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:40.013252 32675 raft_consensus.cc:2955] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:40.016561 31672 raft_consensus.cc:2955] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:40.025190 32378 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:40.027729 32682 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:40.032008 31822 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:40.034799 32672 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:40.037235 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 6e7ad549735d4871af75658a91bb98c4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:40.042882 32690 ts_tablet_manager.cc:1929] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:40.043493 32690 log.cc:1199] T c34755b3b9d040f69be353e35d7431e7 P b95aeeaf4c51442b8092cff1de674f2b: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/wals/c34755b3b9d040f69be353e35d7431e7
I20260709 12:20:40.043895 32674 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:40.045354 32690 tablet_replica.cc:333] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:20:40.045646 32677 raft_consensus.cc:2955] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:40.046710 31561 catalog_manager.cc:5028] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): tablet c34755b3b9d040f69be353e35d7431e7 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:40.051633 32378 raft_consensus.cc:2955] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:40.051705 31823 raft_consensus.cc:2955] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:40.071565 32377 raft_consensus.cc:2955] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:40.084612 31576 catalog_manager.cc:5697] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:40.104540 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 00854b2a70f24ef3a7b43d17477a0ccf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:40.106963 32690 raft_consensus.cc:2243] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:40.107919 32690 raft_consensus.cc:2272] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:40.113330 32690 ts_tablet_manager.cc:1916] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:40.114413 31811 tablet_service.cc:1558] Processing DeleteTablet for tablet 6e7ad549735d4871af75658a91bb98c4 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:41192
I20260709 12:20:40.118211 31575 catalog_manager.cc:5697] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:40.141461 31735 tablet_service.cc:1558] Processing DeleteTablet for tablet 00854b2a70f24ef3a7b43d17477a0ccf with delete_type TABLET_DATA_TOMBSTONED (TS 039d66a6c85e43c3936bf5017501b344 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45150
I20260709 12:20:40.156358 32696 tablet_replica.cc:333] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:20:40.161067 32696 raft_consensus.cc:2243] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:40.162029 32696 raft_consensus.cc:2272] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:40.164106 32697 tablet_replica.cc:333] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:20:40.165515 32697 raft_consensus.cc:2243] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:40.166430 32697 raft_consensus.cc:2272] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:40.166785 32696 ts_tablet_manager.cc:1916] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:40.170083 32697 ts_tablet_manager.cc:1916] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:40.164160 32690 ts_tablet_manager.cc:1929] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:40.173573 32690 log.cc:1199] T 04bdf51be7b94825a28028a5cfb96245 P b95aeeaf4c51442b8092cff1de674f2b: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/wals/04bdf51be7b94825a28028a5cfb96245
I20260709 12:20:40.175040 31561 catalog_manager.cc:5028] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): tablet 04bdf51be7b94825a28028a5cfb96245 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:40.183905 32696 ts_tablet_manager.cc:1929] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:40.184514 32696 log.cc:1199] T 6e7ad549735d4871af75658a91bb98c4 P 4d48c83c761f48079bc6ea176649dda0: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root/wals/6e7ad549735d4871af75658a91bb98c4
I20260709 12:20:40.186215 31562 catalog_manager.cc:5028] TS 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): tablet 6e7ad549735d4871af75658a91bb98c4 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:40.187284 32697 ts_tablet_manager.cc:1929] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:40.187687 32697 log.cc:1199] T 00854b2a70f24ef3a7b43d17477a0ccf P 039d66a6c85e43c3936bf5017501b344: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/wals/00854b2a70f24ef3a7b43d17477a0ccf
I20260709 12:20:40.189056 31561 catalog_manager.cc:5028] TS 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): tablet 00854b2a70f24ef3a7b43d17477a0ccf (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:41.222489 31672 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:41.229949 31823 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:41.230161 31643 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f2f85d943e24cfdb47fe2ad3185097f. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.230762 31752 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:41.231828 32669 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:41.232928 32700 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:41.248467 32584 raft_consensus.cc:2955] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:41.259590 31823 raft_consensus.cc:2955] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:41.260900 31748 raft_consensus.cc:2955] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:41.268779 31752 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:41.272866 31574 catalog_manager.cc:5697] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:41.298233 31718 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c22bf85ce45841fb80e303e2746080da. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.299711 31827 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:41.301503 31672 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:41.301638 32675 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:41.302898 32699 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:41.314101 32699 raft_consensus.cc:2955] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:41.316077 31672 raft_consensus.cc:2955] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:41.329541 31752 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:41.334034 31575 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:41.366044 31672 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:41.367671 31823 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:41.366160 31827 raft_consensus.cc:2955] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
W20260709 12:20:41.368563 31718 consensus_peers.cc:597] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 804df7046b7545d9890885896a333a3e. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.369458 32616 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:41.377118 32675 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:41.388070 32699 raft_consensus.cc:2955] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:41.396131 31677 raft_consensus.cc:2955] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:41.396659 31823 raft_consensus.cc:2955] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:41.407567 31827 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:41.409163 31576 catalog_manager.cc:5697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:41.420766 31792 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 11075d6cbd1a40ccaf381a4eb70cbf24. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:41.420930 31752 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:41.421517 31672 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:41.422458 32545 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:41.424291 32698 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:41.438285 32545 raft_consensus.cc:2955] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:41.443096 31677 raft_consensus.cc:2955] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:41.443840 31752 raft_consensus.cc:2955] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:41.507095 31576 catalog_manager.cc:5697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260709 12:20:41.549476 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55502.3us
W20260709 12:20:41.559829 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62282.5us
W20260709 12:20:41.568596 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 76702.3us
I20260709 12:20:41.800200 32724 ts_tablet_manager.cc:933] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:41.825702 32724 tablet_copy_client.cc:323] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.194:46639
I20260709 12:20:41.826437 32723 ts_tablet_manager.cc:933] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:41.827572 31762 tablet_copy_service.cc:140] P 039d66a6c85e43c3936bf5017501b344: Received BeginTabletCopySession request for tablet 804df7046b7545d9890885896a333a3e from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:57176)
I20260709 12:20:41.828171 31762 tablet_copy_service.cc:161] P 039d66a6c85e43c3936bf5017501b344: Beginning new tablet copy session on tablet 804df7046b7545d9890885896a333a3e from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:57176: session id = 5206fd0087404d2da964a5d985a04ff4-804df7046b7545d9890885896a333a3e
I20260709 12:20:41.835804 31762 tablet_copy_source_session.cc:215] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:41.839386 32724 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 804df7046b7545d9890885896a333a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:41.858018 32723 tablet_copy_client.cc:323] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.193:41935
I20260709 12:20:41.860590 31687 tablet_copy_service.cc:140] P b95aeeaf4c51442b8092cff1de674f2b: Received BeginTabletCopySession request for tablet 2f2f85d943e24cfdb47fe2ad3185097f from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34010)
I20260709 12:20:41.861306 31687 tablet_copy_service.cc:161] P b95aeeaf4c51442b8092cff1de674f2b: Beginning new tablet copy session on tablet 2f2f85d943e24cfdb47fe2ad3185097f from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34010: session id = 5206fd0087404d2da964a5d985a04ff4-2f2f85d943e24cfdb47fe2ad3185097f
I20260709 12:20:41.869267 32724 tablet_copy_client.cc:806] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:41.871591 32724 tablet_copy_client.cc:670] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:41.876153 31687 tablet_copy_source_session.cc:215] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:41.880690 32724 tablet_copy_client.cc:538] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:41.881080 32723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f2f85d943e24cfdb47fe2ad3185097f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:41.892746 32724 tablet_bootstrap.cc:492] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:41.891738 32727 ts_tablet_manager.cc:933] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:41.902431 32728 ts_tablet_manager.cc:933] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:41.918376 32727 tablet_copy_client.cc:323] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.194:46639
I20260709 12:20:41.922837 31762 tablet_copy_service.cc:140] P 039d66a6c85e43c3936bf5017501b344: Received BeginTabletCopySession request for tablet c22bf85ce45841fb80e303e2746080da from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:57176)
I20260709 12:20:41.923607 31762 tablet_copy_service.cc:161] P 039d66a6c85e43c3936bf5017501b344: Beginning new tablet copy session on tablet c22bf85ce45841fb80e303e2746080da from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:57176: session id = 5206fd0087404d2da964a5d985a04ff4-c22bf85ce45841fb80e303e2746080da
I20260709 12:20:41.920698 32728 tablet_copy_client.cc:323] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.195:44645
I20260709 12:20:41.929867 31837 tablet_copy_service.cc:140] P 4d48c83c761f48079bc6ea176649dda0: Received BeginTabletCopySession request for tablet 11075d6cbd1a40ccaf381a4eb70cbf24 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34642)
I20260709 12:20:41.930490 31837 tablet_copy_service.cc:161] P 4d48c83c761f48079bc6ea176649dda0: Beginning new tablet copy session on tablet 11075d6cbd1a40ccaf381a4eb70cbf24 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34642: session id = 5206fd0087404d2da964a5d985a04ff4-11075d6cbd1a40ccaf381a4eb70cbf24
I20260709 12:20:41.935851 31762 tablet_copy_source_session.cc:215] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:41.939796 32727 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22bf85ce45841fb80e303e2746080da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:41.946393 31837 tablet_copy_source_session.cc:215] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:41.930526 32723 tablet_copy_client.cc:806] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:41.948818 32723 tablet_copy_client.cc:670] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:41.950537 32728 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11075d6cbd1a40ccaf381a4eb70cbf24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:41.956128 32723 tablet_copy_client.cc:538] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:41.966604 32723 tablet_bootstrap.cc:492] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:41.968310 32724 tablet_bootstrap.cc:492] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:41.969575 32724 tablet_bootstrap.cc:492] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:41.970515 32724 ts_tablet_manager.cc:1403] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.078s	user 0.031s	sys 0.013s
I20260709 12:20:41.971648 32727 tablet_copy_client.cc:806] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:41.972745 32727 tablet_copy_client.cc:670] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:41.974239 32724 raft_consensus.cc:359] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:41.975306 32724 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:41.976958 32727 tablet_copy_client.cc:538] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:41.976107 32724 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:41.981428 32728 tablet_copy_client.cc:806] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:41.982040 32728 tablet_copy_client.cc:670] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:41.994438 32728 tablet_copy_client.cc:538] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:41.995944 32727 tablet_bootstrap.cc:492] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:42.000540 32724 ts_tablet_manager.cc:1434] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.030s	user 0.014s	sys 0.007s
I20260709 12:20:42.003898 31762 tablet_copy_service.cc:342] P 039d66a6c85e43c3936bf5017501b344: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-804df7046b7545d9890885896a333a3e received from {username='slave'} at 127.0.0.1:57176
I20260709 12:20:42.004106 32723 tablet_bootstrap.cc:492] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.004390 31762 tablet_copy_service.cc:434] P 039d66a6c85e43c3936bf5017501b344: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-804df7046b7545d9890885896a333a3e on tablet 804df7046b7545d9890885896a333a3e with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:42.011301 32723 tablet_bootstrap.cc:492] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:42.012413 32723 ts_tablet_manager.cc:1403] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.046s	user 0.017s	sys 0.009s
I20260709 12:20:42.014281 32728 tablet_bootstrap.cc:492] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:42.016057 32723 raft_consensus.cc:359] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:42.017171 32723 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:42.018069 32723 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:42.020993 32723 ts_tablet_manager.cc:1434] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.001s
I20260709 12:20:42.026844 31687 tablet_copy_service.cc:342] P b95aeeaf4c51442b8092cff1de674f2b: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-2f2f85d943e24cfdb47fe2ad3185097f received from {username='slave'} at 127.0.0.1:34010
I20260709 12:20:42.027344 31687 tablet_copy_service.cc:434] P b95aeeaf4c51442b8092cff1de674f2b: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-2f2f85d943e24cfdb47fe2ad3185097f on tablet 2f2f85d943e24cfdb47fe2ad3185097f with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:42.051524 32727 tablet_bootstrap.cc:492] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.052525 32728 tablet_bootstrap.cc:492] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:42.053750 32728 tablet_bootstrap.cc:492] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:42.054574 32728 ts_tablet_manager.cc:1403] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.041s	user 0.022s	sys 0.009s
I20260709 12:20:42.052626 32727 tablet_bootstrap.cc:492] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:42.055528 32727 ts_tablet_manager.cc:1403] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.060s	user 0.014s	sys 0.012s
I20260709 12:20:42.058648 32728 raft_consensus.cc:359] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:42.059680 32728 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:42.061026 32728 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:42.068429 32728 ts_tablet_manager.cc:1434] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.013s	user 0.009s	sys 0.000s
I20260709 12:20:42.070083 32727 raft_consensus.cc:359] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:42.070891 32727 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:42.071753 31837 tablet_copy_service.cc:342] P 4d48c83c761f48079bc6ea176649dda0: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-11075d6cbd1a40ccaf381a4eb70cbf24 received from {username='slave'} at 127.0.0.1:34642
I20260709 12:20:42.072295 31837 tablet_copy_service.cc:434] P 4d48c83c761f48079bc6ea176649dda0: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-11075d6cbd1a40ccaf381a4eb70cbf24 on tablet 11075d6cbd1a40ccaf381a4eb70cbf24 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:42.073597 32727 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:42.079082 32727 ts_tablet_manager.cc:1434] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.000s
I20260709 12:20:42.096180 31762 tablet_copy_service.cc:342] P 039d66a6c85e43c3936bf5017501b344: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-c22bf85ce45841fb80e303e2746080da received from {username='slave'} at 127.0.0.1:57176
I20260709 12:20:42.096647 31762 tablet_copy_service.cc:434] P 039d66a6c85e43c3936bf5017501b344: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-c22bf85ce45841fb80e303e2746080da on tablet c22bf85ce45841fb80e303e2746080da with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:42.272723 32378 raft_consensus.cc:1217] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:42.334987 32378 raft_consensus.cc:1217] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:42.361106 32378 raft_consensus.cc:1217] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:42.465749 32378 raft_consensus.cc:1217] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:42.679061 32698 raft_consensus.cc:1064] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:42.681686 32698 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:42.689730 32377 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.691367 32545 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:42.697193 31748 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.700347 31672 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.700536 32545 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:42.701830 32671 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:42.715212 32671 raft_consensus.cc:2955] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:42.724666 31748 raft_consensus.cc:2955] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:42.744704 31576 catalog_manager.cc:5697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:42.739275 31672 raft_consensus.cc:2955] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:42.758699 32378 raft_consensus.cc:2955] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:42.801342 31827 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:42.807849 32377 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:42.808039 31672 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:42.809289 32671 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:42.810305 32740 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:42.823441 32375 raft_consensus.cc:2955] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:42.827545 32616 raft_consensus.cc:1064] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:42.830029 32616 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:42.834326 31672 raft_consensus.cc:2955] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:42.819460 32740 raft_consensus.cc:2955] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:42.859652 31574 catalog_manager.cc:5697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:42.868394 31672 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.868624 32375 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.869218 31827 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.880327 31562 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 11075d6cbd1a40ccaf381a4eb70cbf24 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:42.883611 31735 tablet_service.cc:1558] Processing DeleteTablet for tablet 11075d6cbd1a40ccaf381a4eb70cbf24 with delete_type TABLET_DATA_TOMBSTONED (TS 039d66a6c85e43c3936bf5017501b344 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45150
I20260709 12:20:42.883272 32720 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260709 12:20:42.893559 32746 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:42.896802 32675 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:20:42.898080 32722 raft_consensus.cc:1064] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:42.902243 32722 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:42.931895 32748 tablet_replica.cc:333] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:20:42.943840 32375 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.944912 31823 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.945969 31752 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:42.958369 32675 raft_consensus.cc:2955] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:42.953260 32745 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260709 12:20:42.972369 32700 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:42.973380 32584 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:20:42.997107 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49837us
W20260709 12:20:43.004186 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 46687.8us
I20260709 12:20:43.005770 32616 raft_consensus.cc:1064] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:43.009121 31827 raft_consensus.cc:2955] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.001811 31573 catalog_manager.cc:5697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:43.025148 32748 raft_consensus.cc:2243] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.026094 32748 raft_consensus.cc:2272] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.031909 32377 raft_consensus.cc:2955] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.043993 32748 ts_tablet_manager.cc:1916] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:43.046880 32616 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:43.100021 31827 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:43.101099 32378 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:43.105588 32746 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:43.105938 31750 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:43.107513 32746 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:43.107879 32745 raft_consensus.cc:2955] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.110077 32375 raft_consensus.cc:2955] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.122519 32378 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:43.127507 31822 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:43.137924 31574 catalog_manager.cc:5697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.154856 32746 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:43.156509 32746 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:43.165395 32720 raft_consensus.cc:2955] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.168290 32375 raft_consensus.cc:2955] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.177096 31827 raft_consensus.cc:2955] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.184434 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 804df7046b7545d9890885896a333a3e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:43.195400 31673 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:43.210812 31752 raft_consensus.cc:2955] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.214541 31822 raft_consensus.cc:2955] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.194584 31574 catalog_manager.cc:5697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:43.228576 32746 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:43.248030 32748 ts_tablet_manager.cc:1929] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:43.248695 32757 raft_consensus.cc:2955] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.253302 31823 raft_consensus.cc:2955] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.259716 31668 tablet_service.cc:1558] Processing DeleteTablet for tablet 804df7046b7545d9890885896a333a3e with delete_type TABLET_DATA_TOMBSTONED (TS b95aeeaf4c51442b8092cff1de674f2b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43180
I20260709 12:20:43.272322 32748 log.cc:1199] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 039d66a6c85e43c3936bf5017501b344: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/wals/11075d6cbd1a40ccaf381a4eb70cbf24
I20260709 12:20:43.277261 31561 catalog_manager.cc:5028] TS 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): tablet 11075d6cbd1a40ccaf381a4eb70cbf24 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:43.276640 32375 raft_consensus.cc:2955] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.287794 32764 tablet_replica.cc:333] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:20:43.289290 32764 raft_consensus.cc:2243] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.290081 32764 pending_rounds.cc:70] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b: Trying to abort 1 pending ops.
I20260709 12:20:43.293135 31672 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:43.302345 32377 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:43.305382 32760 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:43.305799 31820 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:43.290473 32764 pending_rounds.cc:77] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7305624137026007040 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "804df7046b7545d9890885896a333a3e" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } } }
I20260709 12:20:43.316335 32760 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:43.315565 32764 raft_consensus.cc:2917] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20260709 12:20:43.319563 32764 raft_consensus.cc:2889] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260709 12:20:43.320009 32764 raft_consensus.cc:2272] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.320150 31672 raft_consensus.cc:2955] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.326599 32722 raft_consensus.cc:2955] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.330451 32764 ts_tablet_manager.cc:1916] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:43.331522 32378 raft_consensus.cc:2955] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.333523 31576 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.338924 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2f2f85d943e24cfdb47fe2ad3185097f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:43.349881 31827 raft_consensus.cc:2955] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.367660 31574 catalog_manager.cc:5697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.370033 32764 ts_tablet_manager.cc:1929] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:43.370534 32764 log.cc:1199] T 804df7046b7545d9890885896a333a3e P b95aeeaf4c51442b8092cff1de674f2b: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/wals/804df7046b7545d9890885896a333a3e
I20260709 12:20:43.382336 31735 tablet_service.cc:1558] Processing DeleteTablet for tablet 2f2f85d943e24cfdb47fe2ad3185097f with delete_type TABLET_DATA_TOMBSTONED (TS 039d66a6c85e43c3936bf5017501b344 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45150
I20260709 12:20:43.383033 31561 catalog_manager.cc:5028] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): tablet 804df7046b7545d9890885896a333a3e (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:43.385507 32748 tablet_replica.cc:333] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:20:43.386670 32748 raft_consensus.cc:2243] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.387326 32748 raft_consensus.cc:2272] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.390275 32748 ts_tablet_manager.cc:1916] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:43.404996 31752 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:43.409870 32378 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:43.410830 31672 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:43.412328 32616 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:43.413098 32675 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:43.417801 32748 ts_tablet_manager.cc:1929] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:43.419091 32748 log.cc:1199] T 2f2f85d943e24cfdb47fe2ad3185097f P 039d66a6c85e43c3936bf5017501b344: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/wals/2f2f85d943e24cfdb47fe2ad3185097f
I20260709 12:20:43.422217 31561 catalog_manager.cc:5028] TS 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): tablet 2f2f85d943e24cfdb47fe2ad3185097f (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:43.422565 32757 raft_consensus.cc:2955] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.426657 31672 raft_consensus.cc:2955] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.435003 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet c22bf85ce45841fb80e303e2746080da with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:43.438786 32378 raft_consensus.cc:2955] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.448433 31574 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:43.467675 31811 tablet_service.cc:1558] Processing DeleteTablet for tablet c22bf85ce45841fb80e303e2746080da with delete_type TABLET_DATA_TOMBSTONED (TS 4d48c83c761f48079bc6ea176649dda0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41192
I20260709 12:20:43.471689 32766 tablet_replica.cc:333] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:20:43.474004 32766 raft_consensus.cc:2243] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:43.474797 32766 raft_consensus.cc:2272] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:43.477573 32766 ts_tablet_manager.cc:1916] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:43.491374 32766 ts_tablet_manager.cc:1929] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:43.491736 32766 log.cc:1199] T c22bf85ce45841fb80e303e2746080da P 4d48c83c761f48079bc6ea176649dda0: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root/wals/c22bf85ce45841fb80e303e2746080da
I20260709 12:20:43.493105 31562 catalog_manager.cc:5028] TS 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): tablet c22bf85ce45841fb80e303e2746080da (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:44.569092 31673 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:44.577315 31752 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:44.577713 31822 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:44.577522 31643 consensus_peers.cc:597] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 354c519f070b453dadcdef3f0d835c69. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:44.579008 32760 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:44.580050   300 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:44.589129 32760 raft_consensus.cc:2955] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.592130 31752 raft_consensus.cc:2955] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.592219 31822 raft_consensus.cc:2955] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.612002 31574 catalog_manager.cc:5697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.621384 31820 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
W20260709 12:20:44.632277 31792 consensus_peers.cc:597] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e27b968c91a4906aec537421f481269. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:44.633142 31750 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:44.634986 32758 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:44.637787 31673 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:44.644784 32758 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:44.680274 32698 raft_consensus.cc:2955] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.684085 31750 raft_consensus.cc:2955] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.692710 31673 raft_consensus.cc:2955] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.709259 31672 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:44.717577 31750 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:44.719494 31820 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:44.722110 32745 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:20:44.725703 31643 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 812ed9ab41214308b368b53417e6e9f3. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:44.725723 32745 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:44.727784 31574 catalog_manager.cc:5697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.742093 32760 raft_consensus.cc:2955] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.753458 31750 raft_consensus.cc:2955] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.754778 31820 raft_consensus.cc:2955] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.760564 31672 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:44.771637 31574 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:44.801303 31820 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:44.804095 31643 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6821bb8779e64160887e3195aa5078a1. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:44.805277 32760 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:20:44.815333 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47272.9us
I20260709 12:20:44.818210 31752 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:44.820261 32745 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:44.826339 32745 raft_consensus.cc:2955] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.828929 31820 raft_consensus.cc:2955] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.843690 31752 raft_consensus.cc:2955] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:44.863637 31574 catalog_manager.cc:5697] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:45.146755   322 ts_tablet_manager.cc:933] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:45.148854   322 tablet_copy_client.cc:323] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.195:44645
I20260709 12:20:45.151198 31837 tablet_copy_service.cc:140] P 4d48c83c761f48079bc6ea176649dda0: Received BeginTabletCopySession request for tablet 1e27b968c91a4906aec537421f481269 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34642)
I20260709 12:20:45.151893 31837 tablet_copy_service.cc:161] P 4d48c83c761f48079bc6ea176649dda0: Beginning new tablet copy session on tablet 1e27b968c91a4906aec537421f481269 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34642: session id = 5206fd0087404d2da964a5d985a04ff4-1e27b968c91a4906aec537421f481269
I20260709 12:20:45.160998 31837 tablet_copy_source_session.cc:215] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:45.166270   322 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e27b968c91a4906aec537421f481269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.183365   322 tablet_copy_client.cc:806] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:45.184180   322 tablet_copy_client.cc:670] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:45.188988   322 tablet_copy_client.cc:538] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:45.194990   324 ts_tablet_manager.cc:933] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:45.199208   324 tablet_copy_client.cc:323] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.193:41935
I20260709 12:20:45.201576   322 tablet_bootstrap.cc:492] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:45.201719 31687 tablet_copy_service.cc:140] P b95aeeaf4c51442b8092cff1de674f2b: Received BeginTabletCopySession request for tablet 354c519f070b453dadcdef3f0d835c69 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34010)
I20260709 12:20:45.202339 31687 tablet_copy_service.cc:161] P b95aeeaf4c51442b8092cff1de674f2b: Beginning new tablet copy session on tablet 354c519f070b453dadcdef3f0d835c69 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34010: session id = 5206fd0087404d2da964a5d985a04ff4-354c519f070b453dadcdef3f0d835c69
I20260709 12:20:45.228245 31687 tablet_copy_source_session.cc:215] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:45.231211   326 ts_tablet_manager.cc:933] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:45.247534   326 tablet_copy_client.cc:323] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.193:41935
I20260709 12:20:45.260179 31687 tablet_copy_service.cc:140] P b95aeeaf4c51442b8092cff1de674f2b: Received BeginTabletCopySession request for tablet 6821bb8779e64160887e3195aa5078a1 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34010)
I20260709 12:20:45.260783 31687 tablet_copy_service.cc:161] P b95aeeaf4c51442b8092cff1de674f2b: Beginning new tablet copy session on tablet 6821bb8779e64160887e3195aa5078a1 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34010: session id = 5206fd0087404d2da964a5d985a04ff4-6821bb8779e64160887e3195aa5078a1
I20260709 12:20:45.271091 31687 tablet_copy_source_session.cc:215] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:45.281644   326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6821bb8779e64160887e3195aa5078a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.287249   324 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 354c519f070b453dadcdef3f0d835c69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.311677   327 ts_tablet_manager.cc:933] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:45.312815   326 tablet_copy_client.cc:806] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:45.313898   326 tablet_copy_client.cc:670] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
W20260709 12:20:45.314872 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56586.6us
I20260709 12:20:45.318667   326 tablet_copy_client.cc:538] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:45.379396   326 tablet_bootstrap.cc:492] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:45.383662   327 tablet_copy_client.cc:323] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.193:41935
I20260709 12:20:45.396215 31687 tablet_copy_service.cc:140] P b95aeeaf4c51442b8092cff1de674f2b: Received BeginTabletCopySession request for tablet 812ed9ab41214308b368b53417e6e9f3 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34010)
I20260709 12:20:45.396806 31687 tablet_copy_service.cc:161] P b95aeeaf4c51442b8092cff1de674f2b: Beginning new tablet copy session on tablet 812ed9ab41214308b368b53417e6e9f3 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34010: session id = 5206fd0087404d2da964a5d985a04ff4-812ed9ab41214308b368b53417e6e9f3
I20260709 12:20:45.407087 31687 tablet_copy_source_session.cc:215] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:45.423070   327 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 812ed9ab41214308b368b53417e6e9f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:45.408492   322 tablet_bootstrap.cc:492] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:45.427137   322 tablet_bootstrap.cc:492] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:45.431063   322 ts_tablet_manager.cc:1403] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.230s	user 0.068s	sys 0.084s
I20260709 12:20:45.434018   324 tablet_copy_client.cc:806] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:45.434774   324 tablet_copy_client.cc:670] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:45.434904   322 raft_consensus.cc:359] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:45.436343   322 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:45.442956   324 tablet_copy_client.cc:538] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:45.444185   322 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:45.454236   324 tablet_bootstrap.cc:492] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:45.442495   326 tablet_bootstrap.cc:492] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:45.459867   326 tablet_bootstrap.cc:492] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:45.460801   326 ts_tablet_manager.cc:1403] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.082s	user 0.025s	sys 0.000s
I20260709 12:20:45.464375   326 raft_consensus.cc:359] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:45.465279   327 tablet_copy_client.cc:806] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:45.464874   322 ts_tablet_manager.cc:1434] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.033s	user 0.010s	sys 0.011s
I20260709 12:20:45.465992   327 tablet_copy_client.cc:670] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:45.465368   326 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:45.466988   326 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:45.469095 31837 tablet_copy_service.cc:342] P 4d48c83c761f48079bc6ea176649dda0: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-1e27b968c91a4906aec537421f481269 received from {username='slave'} at 127.0.0.1:34642
I20260709 12:20:45.469630 31837 tablet_copy_service.cc:434] P 4d48c83c761f48079bc6ea176649dda0: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-1e27b968c91a4906aec537421f481269 on tablet 1e27b968c91a4906aec537421f481269 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:45.479619   326 ts_tablet_manager.cc:1434] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.018s	user 0.009s	sys 0.004s
I20260709 12:20:45.485414   327 tablet_copy_client.cc:538] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:45.498885 31687 tablet_copy_service.cc:342] P b95aeeaf4c51442b8092cff1de674f2b: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-6821bb8779e64160887e3195aa5078a1 received from {username='slave'} at 127.0.0.1:34010
I20260709 12:20:45.499464 31687 tablet_copy_service.cc:434] P b95aeeaf4c51442b8092cff1de674f2b: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-6821bb8779e64160887e3195aa5078a1 on tablet 6821bb8779e64160887e3195aa5078a1 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:45.515018   327 tablet_bootstrap.cc:492] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:45.530419   324 tablet_bootstrap.cc:492] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:45.531668   324 tablet_bootstrap.cc:492] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:45.532465   324 ts_tablet_manager.cc:1403] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.078s	user 0.021s	sys 0.025s
I20260709 12:20:45.536348   324 raft_consensus.cc:359] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:45.541532   324 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:45.542320   324 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:45.545194   324 ts_tablet_manager.cc:1434] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.012s	user 0.002s	sys 0.004s
I20260709 12:20:45.549131 31687 tablet_copy_service.cc:342] P b95aeeaf4c51442b8092cff1de674f2b: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-354c519f070b453dadcdef3f0d835c69 received from {username='slave'} at 127.0.0.1:34010
I20260709 12:20:45.549582 31687 tablet_copy_service.cc:434] P b95aeeaf4c51442b8092cff1de674f2b: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-354c519f070b453dadcdef3f0d835c69 on tablet 354c519f070b453dadcdef3f0d835c69 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:45.557615   327 tablet_bootstrap.cc:492] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:45.558502   327 tablet_bootstrap.cc:492] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:45.559196   327 ts_tablet_manager.cc:1403] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.051s	user 0.024s	sys 0.005s
I20260709 12:20:45.561371   327 raft_consensus.cc:359] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:45.561967   327 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:45.562500   327 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:45.564711   327 ts_tablet_manager.cc:1434] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:20:45.566972 31687 tablet_copy_service.cc:342] P b95aeeaf4c51442b8092cff1de674f2b: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-812ed9ab41214308b368b53417e6e9f3 received from {username='slave'} at 127.0.0.1:34010
I20260709 12:20:45.567389 31687 tablet_copy_service.cc:434] P b95aeeaf4c51442b8092cff1de674f2b: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-812ed9ab41214308b368b53417e6e9f3 on tablet 812ed9ab41214308b368b53417e6e9f3 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:45.684870 32378 raft_consensus.cc:1217] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:45.728611 32378 raft_consensus.cc:1217] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:45.750415 32378 raft_consensus.cc:1217] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:45.810102 32378 raft_consensus.cc:1217] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:46.133929 32745 raft_consensus.cc:1064] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:46.136754 32745 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:46.160308 31750 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.162971 32378 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.164041 31822 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.162443 32745 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:46.166500   334 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.168994   308 raft_consensus.cc:1064] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:46.169085   333 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:46.172220   308 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:46.196246 31750 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.196288 31672 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.198603 32758 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.200538 32758 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.198660 32375 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.219013   333 raft_consensus.cc:1064] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:46.220625   331 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:46.221764   333 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:46.235252 31672 raft_consensus.cc:2955] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.241050 32745 raft_consensus.cc:2955] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.243989 32378 raft_consensus.cc:2955] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.268131 32375 raft_consensus.cc:2955] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.272783 31820 raft_consensus.cc:2955] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.280962   341 raft_consensus.cc:1064] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:46.291245   341 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:46.294394 31573 catalog_manager.cc:5697] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.303238   344 raft_consensus.cc:2955] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:20:46.331427 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 70703.5us
I20260709 12:20:46.333263 31752 raft_consensus.cc:2955] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.334367 31750 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.334779 31820 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.342015 32378 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.342890   317 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.351212   334 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:46.363517   334 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:46.366600 31752 raft_consensus.cc:2955] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.385210   333 raft_consensus.cc:2955] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.386729 31820 raft_consensus.cc:2955] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.388967 32378 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.391134 31822 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.390974   317 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.392823   317 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:46.400454 31575 catalog_manager.cc:5697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.408682 31750 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:46.412986 31750 raft_consensus.cc:2955] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.425474   317 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:46.430014 32375 raft_consensus.cc:2955] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.469393   317 raft_consensus.cc:2955] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.471124 31820 raft_consensus.cc:2955] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.492604 32375 raft_consensus.cc:2955] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.492776 31752 raft_consensus.cc:2955] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.529946 31576 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:46.540251 31822 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:46.550148 31752 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.551669 32375 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.551923 32758 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:46.553728   308 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:46.563149   344 raft_consensus.cc:2955] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.563927 31750 raft_consensus.cc:2955] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.566645 32375 raft_consensus.cc:2955] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.565652 31574 catalog_manager.cc:5697] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.577592 31562 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 1e27b968c91a4906aec537421f481269 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:46.592867 31673 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:46.607347 32378 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.608960   350 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:20:46.615700 31750 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.616582   350 raft_consensus.cc:2955] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.617497   334 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:46.619458 31572 catalog_manager.cc:5697] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.625543 31750 raft_consensus.cc:2955] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.626142 32378 raft_consensus.cc:2955] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.639015 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 354c519f070b453dadcdef3f0d835c69 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:46.645460 31668 tablet_service.cc:1558] Processing DeleteTablet for tablet 1e27b968c91a4906aec537421f481269 with delete_type TABLET_DATA_TOMBSTONED (TS b95aeeaf4c51442b8092cff1de674f2b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43180
I20260709 12:20:46.648123   364 tablet_replica.cc:333] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:20:46.653388   364 raft_consensus.cc:2243] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:46.654287   364 raft_consensus.cc:2272] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:46.658759 31673 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:46.659169 31672 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:46.660583 31576 catalog_manager.cc:5697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.665758 32375 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.665938 31750 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.666390   364 ts_tablet_manager.cc:1916] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:46.667590   333 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:46.667950 32378 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.674153   350 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:46.674401 31827 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:46.674463   334 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.006s
I20260709 12:20:46.682521   317 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:20:46.688230   333 raft_consensus.cc:2955] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.689502   350 raft_consensus.cc:2955] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.694092 31750 raft_consensus.cc:2955] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.700297 32375 raft_consensus.cc:2955] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.701440 32377 raft_consensus.cc:2955] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.703315   364 ts_tablet_manager.cc:1929] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:46.703862   364 log.cc:1199] T 1e27b968c91a4906aec537421f481269 P b95aeeaf4c51442b8092cff1de674f2b: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/wals/1e27b968c91a4906aec537421f481269
I20260709 12:20:46.709245 31827 raft_consensus.cc:2955] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.705874 31561 catalog_manager.cc:5028] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): tablet 1e27b968c91a4906aec537421f481269 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:46.716473 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 6821bb8779e64160887e3195aa5078a1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:46.727109 31811 tablet_service.cc:1558] Processing DeleteTablet for tablet 354c519f070b453dadcdef3f0d835c69 with delete_type TABLET_DATA_TOMBSTONED (TS 4d48c83c761f48079bc6ea176649dda0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41192
I20260709 12:20:46.730333 31573 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:46.735067 31572 catalog_manager.cc:5697] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index 3 to 4, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:46.743654 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 812ed9ab41214308b368b53417e6e9f3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:46.747999   365 tablet_replica.cc:333] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:20:46.749382   365 raft_consensus.cc:2243] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:46.750169   365 raft_consensus.cc:2272] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:46.756095 31811 tablet_service.cc:1558] Processing DeleteTablet for tablet 6821bb8779e64160887e3195aa5078a1 with delete_type TABLET_DATA_TOMBSTONED (TS 4d48c83c761f48079bc6ea176649dda0 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41192
I20260709 12:20:46.757028   365 ts_tablet_manager.cc:1916] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:46.757459 31735 tablet_service.cc:1558] Processing DeleteTablet for tablet 812ed9ab41214308b368b53417e6e9f3 with delete_type TABLET_DATA_TOMBSTONED (TS 039d66a6c85e43c3936bf5017501b344 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45150
I20260709 12:20:46.781859   365 ts_tablet_manager.cc:1929] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:46.782178   366 tablet_replica.cc:333] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:20:46.782366   365 log.cc:1199] T 354c519f070b453dadcdef3f0d835c69 P 4d48c83c761f48079bc6ea176649dda0: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root/wals/354c519f070b453dadcdef3f0d835c69
I20260709 12:20:46.783322   366 raft_consensus.cc:2243] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:46.784034   366 raft_consensus.cc:2272] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:46.784284   365 tablet_replica.cc:333] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:20:46.784310 31562 catalog_manager.cc:5028] TS 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): tablet 354c519f070b453dadcdef3f0d835c69 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:46.785836   365 raft_consensus.cc:2243] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:20:46.786557   365 raft_consensus.cc:2272] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:20:46.787272   366 ts_tablet_manager.cc:1916] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:46.789860   365 ts_tablet_manager.cc:1916] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:20:46.808080   365 ts_tablet_manager.cc:1929] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:46.808245   366 ts_tablet_manager.cc:1929] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:20:46.808626   365 log.cc:1199] T 6821bb8779e64160887e3195aa5078a1 P 4d48c83c761f48079bc6ea176649dda0: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root/wals/6821bb8779e64160887e3195aa5078a1
I20260709 12:20:46.808714   366 log.cc:1199] T 812ed9ab41214308b368b53417e6e9f3 P 039d66a6c85e43c3936bf5017501b344: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/wals/812ed9ab41214308b368b53417e6e9f3
I20260709 12:20:46.810531 31562 catalog_manager.cc:5028] TS 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): tablet 6821bb8779e64160887e3195aa5078a1 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:46.811187 31561 catalog_manager.cc:5028] TS 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): tablet 812ed9ab41214308b368b53417e6e9f3 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:20:46.916720 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:46.973080 31750 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:20:46.973804 31750 raft_consensus.cc:606] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Received request to transfer leadership to TS 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:46.976650 31750 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:20:46.977218 31750 raft_consensus.cc:606] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:46.980324 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:20:46.980898 31752 raft_consensus.cc:606] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Received request to transfer leadership to TS 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:46.983367 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:20:46.983750 31752 raft_consensus.cc:606] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Received request to transfer leadership to TS 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:46.986187 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:20:46.986783 31752 raft_consensus.cc:606] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:46.988996   329 raft_consensus.cc:993] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:46.989511   329 raft_consensus.cc:1081] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Signalling peer 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:46.991179 32375 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1"
dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:33954
I20260709 12:20:46.991789 32375 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:46.992126 32375 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.005004 32375 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:47.008144 32375 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:47.012056 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:47.012635 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:47.012814 31752 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:47.013207 31752 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:47.013222 31820 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.014019 31752 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:47.015569 31752 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.020794 31673 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:20:47.021049 31820 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:47.021479 31673 raft_consensus.cc:606] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:47.023284 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:20:47.024794 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:47.025964 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:47.027220 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:20:47.028250   336 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:47.029495 31752 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:47.029826   336 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:20:47.030970   336 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:47.041126 31571 catalog_manager.cc:5697] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 1 to 2, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.092964   301 raft_consensus.cc:993] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:47.093506   301 raft_consensus.cc:1081] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Signalling peer 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:47.095108 32377 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4"
dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:33954
I20260709 12:20:47.095605 32377 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:47.095947 32377 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.101033 32377 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:47.103154 32377 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:47.103838 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:47.104086 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:47.104476 31673 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.104676 31750 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:47.105016 31750 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:47.105772 31750 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:47.106745   301 raft_consensus.cc:993] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:20:47.106864 31750 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.107447   302 raft_consensus.cc:1081] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Signalling peer 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:20:47.109041 31820 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:41204
I20260709 12:20:47.109536 31820 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:47.109901 31820 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.110796 31673 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:47.112478 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:47.113324   336 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:47.113765   336 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:20:47.114710   336 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:47.117264 31820 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:47.119078 31750 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:47.120267 31820 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:47.121197 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:47.121490 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:47.121788 31673 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.122061 31750 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:47.122380 31750 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:47.123247 31750 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:47.124310 31750 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.124297 31571 catalog_manager.cc:5697] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 1 to 2, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.171787 31750 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:47.182570 31673 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:47.189420 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:47.191093   344 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:47.191628   344 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:47.192584   344 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:47.214152 31571 catalog_manager.cc:5697] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 2, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 2 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.250602   334 raft_consensus.cc:993] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: : Instructing follower 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:20:47.251134   334 raft_consensus.cc:1081] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Signalling peer 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:20:47.252651 31820 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:41210
I20260709 12:20:47.253206 31820 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:47.253548 31820 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.267674 31820 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:47.269169 31820 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:47.270298 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:47.270454 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:47.270896 31673 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:47.271121 31750 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.271260 31673 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:47.272037 31673 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:47.273049 31673 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.276780 31750 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:47.277814 31673 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:47.278060 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:47.278937   344 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:47.279441   344 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:47.280174   344 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:47.290692 31571 catalog_manager.cc:5697] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 2, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 2 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.347858   301 raft_consensus.cc:993] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:47.348372   302 raft_consensus.cc:1081] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Signalling peer 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:47.350122 32377 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6"
dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:33954
I20260709 12:20:47.350796 32377 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:47.351125 32377 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.356038 32377 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:47.358340 32377 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:47.359714 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:47.359817 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:47.360631 31750 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:47.360657 31677 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.361182 31750 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:47.362174 31750 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:47.363739 31750 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.368338 31677 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:47.369474 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:47.370287   336 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:47.370223 31750 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:47.370929   336 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:20:47.371858   336 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:47.381001 31571 catalog_manager.cc:5697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 1 to 2, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.420739 31822 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:47.422158   336 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:47.425045   302 raft_consensus.cc:993] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:20:47.425609   301 raft_consensus.cc:1081] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Signalling peer 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:20:47.427070 31820 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:41204
I20260709 12:20:47.427623 31820 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:47.428153 31820 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.434180 31820 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:47.438061 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:47.438824 31677 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.446230 31677 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:47.447252 31750 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:47.447381 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:47.448192   344 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:47.448681   344 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:47.449540   344 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:47.451301 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:47.451925 31749 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:47.452208 31749 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:47.453284 31749 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:47.454542 31749 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:47.454929   336 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:47.455967 31820 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:47.463475 31749 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:47.473927 31576 catalog_manager.cc:5697] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 2, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 2 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:20:47.514662 31677 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:47.516140   336 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:47.531347 31749 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:47.532691   336 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:47.665992 31749 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:47.667560   331 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:47.681581 31673 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:47.683784   308 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:47.823666 31822 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:47.829730 31750 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260709 12:20:47.829823 31792 consensus_peers.cc:597] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc47871ef692418388113c1d6c88b262. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:47.830009 31673 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:47.831207   331 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:47.832170   344 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:47.840770   344 raft_consensus.cc:2955] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:47.842770 31673 raft_consensus.cc:2955] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:47.843954 31749 raft_consensus.cc:2955] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:47.852752 31750 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:47.855178   308 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:47.868714 31571 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:47.883411 31673 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:47.885089   308 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:47.906708 31750 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:47.931926 31749 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:47.934741   331 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:47.960458 31673 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260709 12:20:47.961067 31718 consensus_peers.cc:597] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c162dbe81714319bfeccd9d7cb7bfea. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:47.962970   336 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:47.992524 31673 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:47.995916   331 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.006064 31749 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:48.014184   336 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 12:20:48.034489 31822 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.043484   301 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.076761 31672 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.080061   301 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:48.090155 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:20:48.107301 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 78799.9us
W20260709 12:20:48.247489 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 110344us
W20260709 12:20:48.364706 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 61400.7us
W20260709 12:20:48.430514 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78657us
I20260709 12:20:48.439967   418 ts_tablet_manager.cc:933] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:48.476713   417 raft_consensus.cc:2955] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:48.491321   418 tablet_copy_client.cc:323] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.195:44645
I20260709 12:20:48.494949 31837 tablet_copy_service.cc:140] P 4d48c83c761f48079bc6ea176649dda0: Received BeginTabletCopySession request for tablet fc47871ef692418388113c1d6c88b262 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34642)
I20260709 12:20:48.495625 31837 tablet_copy_service.cc:161] P 4d48c83c761f48079bc6ea176649dda0: Beginning new tablet copy session on tablet fc47871ef692418388113c1d6c88b262 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34642: session id = 5206fd0087404d2da964a5d985a04ff4-fc47871ef692418388113c1d6c88b262
I20260709 12:20:48.496024 31820 raft_consensus.cc:2955] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:48.508121 31672 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "181ddae071d549aab328e3c1d8553734"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:20:48.509073 31672 raft_consensus.cc:606] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Received request to transfer leadership to TS 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:48.513473 31673 raft_consensus.cc:2955] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:48.526448 31837 tablet_copy_source_session.cc:215] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:48.518877 31674 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:48.531031 31571 catalog_manager.cc:5697] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:48.532830   418 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc47871ef692418388113c1d6c88b262. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:48.573011   418 tablet_copy_client.cc:806] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:48.573725   418 tablet_copy_client.cc:670] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:48.584617   418 tablet_copy_client.cc:538] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:48.587651   429 ts_tablet_manager.cc:933] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:48.597410   418 tablet_bootstrap.cc:492] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:48.599802   429 tablet_copy_client.cc:323] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.194:46639
I20260709 12:20:48.602936 31827 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.603880 31748 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.606118 31762 tablet_copy_service.cc:140] P 039d66a6c85e43c3936bf5017501b344: Received BeginTabletCopySession request for tablet 2c162dbe81714319bfeccd9d7cb7bfea from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:57176)
I20260709 12:20:48.607718   420 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:48.608595 31762 tablet_copy_service.cc:161] P 039d66a6c85e43c3936bf5017501b344: Beginning new tablet copy session on tablet 2c162dbe81714319bfeccd9d7cb7bfea from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:57176: session id = 5206fd0087404d2da964a5d985a04ff4-2c162dbe81714319bfeccd9d7cb7bfea
I20260709 12:20:48.611109   350 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:20:48.617746 31643 consensus_peers.cc:597] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f6404cb9c2174326b06cbbb0d27d43f2. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:48.619388 31762 tablet_copy_source_session.cc:215] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:48.622089 31820 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
mode: GRACEFUL
new_leader_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:41934
I20260709 12:20:48.623025 31820 raft_consensus.cc:606] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Received request to transfer leadership to TS 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:48.625842   429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c162dbe81714319bfeccd9d7cb7bfea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:48.630548 31748 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:20:48.631263 31748 raft_consensus.cc:606] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Received request to transfer leadership to TS 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:48.641522 31748 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:20:48.642161 31748 raft_consensus.cc:606] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Received request to transfer leadership to TS 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:48.645778 31824 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
mode: GRACEFUL
new_leader_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:41934
I20260709 12:20:48.646416 31824 raft_consensus.cc:606] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Received request to transfer leadership to TS 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:48.651520 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:20:48.653893 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:48.667640 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:48.679325   422 raft_consensus.cc:2955] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:48.683355   418 tablet_bootstrap.cc:492] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:48.684726   418 tablet_bootstrap.cc:492] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:48.685755   418 ts_tablet_manager.cc:1403] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.089s	user 0.022s	sys 0.006s
I20260709 12:20:48.687520 31750 raft_consensus.cc:2955] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:48.689805   418 raft_consensus.cc:359] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:48.687520 31827 raft_consensus.cc:2955] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:48.690723   418 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:48.703620   418 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:48.706465   418 ts_tablet_manager.cc:1434] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.020s	user 0.000s	sys 0.006s
I20260709 12:20:48.709280 31674 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:48.714437 31837 tablet_copy_service.cc:342] P 4d48c83c761f48079bc6ea176649dda0: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-fc47871ef692418388113c1d6c88b262 received from {username='slave'} at 127.0.0.1:34642
I20260709 12:20:48.714967 31837 tablet_copy_service.cc:434] P 4d48c83c761f48079bc6ea176649dda0: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-fc47871ef692418388113c1d6c88b262 on tablet fc47871ef692418388113c1d6c88b262 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:48.716971 31573 catalog_manager.cc:5697] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:48.719293 31827 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.719297 31748 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260709 12:20:48.721397   420 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:20:48.726498 31643 consensus_peers.cc:597] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b25009db0c5a47a9ab9f6f8b2e5c1b17. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:20:48.731601   412 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:48.746176   429 tablet_copy_client.cc:806] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:48.747422   429 tablet_copy_client.cc:670] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:48.751364   429 tablet_copy_client.cc:538] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:48.765218   422 raft_consensus.cc:2955] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:48.769807   429 tablet_bootstrap.cc:492] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:48.771540 31820 raft_consensus.cc:2955] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:48.770386   416 raft_consensus.cc:993] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:48.773180   416 raft_consensus.cc:1081] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Signalling peer 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:48.779865 32377 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a"
dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:33954
I20260709 12:20:48.780437 32377 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:48.780810 32377 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.780582 31748 raft_consensus.cc:2955] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:48.788579 32377 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:48.796434 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:48.797101 31673 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.804468 31571 catalog_manager.cc:5697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index -1 to 2, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:20:48.811146 31673 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:48.812515 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:48.814699   379 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:48.815008 32377 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:48.815219   379 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:20:48.816116   379 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:48.816565 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:48.817174 31748 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:48.817441 31748 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:48.820595   412 raft_consensus.cc:993] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: : Instructing follower 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:48.821125   412 raft_consensus.cc:1081] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Signalling peer 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:48.824379 32375 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "181ddae071d549aab328e3c1d8553734"
dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:33962
I20260709 12:20:48.824923 32375 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:48.825409 32375 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.830492 31748 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:48.838325   400 raft_consensus.cc:993] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: : Instructing follower 039d66a6c85e43c3936bf5017501b344 to start an election
I20260709 12:20:48.839056   399 raft_consensus.cc:1081] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Signalling peer 039d66a6c85e43c3936bf5017501b344 to start an election
I20260709 12:20:48.840690 31748 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.841302 31750 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:45166
I20260709 12:20:48.844208 31750 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:48.844583 31750 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.849346 32375 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:48.858203   429 tablet_bootstrap.cc:492] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:48.861151 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:48.862418 32375 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:48.862687 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:48.863319 31747 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.863613 32377 raft_consensus.cc:1217] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:48.864199   429 tablet_bootstrap.cc:492] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:48.866748   429 ts_tablet_manager.cc:1403] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.097s	user 0.026s	sys 0.018s
I20260709 12:20:48.860028 31750 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:48.872120 31750 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:48.874219 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:48.875092 31674 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.882215 31748 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:48.870024   429 raft_consensus.cc:359] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:48.884150   429 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:48.886240 31571 catalog_manager.cc:5697] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 1 to 2, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:48.884857   429 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:48.890173   429 ts_tablet_manager.cc:1434] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.000s
I20260709 12:20:48.862676 31673 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:48.892496 31673 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:48.890197 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:48.893288 31827 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:48.893633 31827 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:48.893358 31673 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:48.894758 31673 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.896389 31762 tablet_copy_service.cc:342] P 039d66a6c85e43c3936bf5017501b344: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-2c162dbe81714319bfeccd9d7cb7bfea received from {username='slave'} at 127.0.0.1:57176
I20260709 12:20:48.896847 31762 tablet_copy_service.cc:434] P 039d66a6c85e43c3936bf5017501b344: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-2c162dbe81714319bfeccd9d7cb7bfea on tablet 2c162dbe81714319bfeccd9d7cb7bfea with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:48.894361 31827 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:48.906955 31827 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:48.911057 31673 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:48.912519 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:48.913897   379 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:48.916774 31674 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:20:48.917236   379 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:20:48.918123 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:48.918280   379 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:48.918995   417 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:48.919660   417 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:48.875110 31747 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:48.920545   417 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:48.929710 31827 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:20:48.954072 31571 catalog_manager.cc:5697] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 1 to 2, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:48.959017 31576 catalog_manager.cc:5697] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 1 to 2, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:48.970504 32377 raft_consensus.cc:1217] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:48.993208   417 raft_consensus.cc:993] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:48.993696   416 raft_consensus.cc:1081] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Signalling peer 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:48.995232 32377 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da"
dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:33954
I20260709 12:20:48.995790 32377 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:48.996138 32377 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.002776 32377 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:49.005542 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:49.006142 31674 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.006546 32377 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:49.008584 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:49.009191 31747 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:49.009505 31747 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:49.010267 31747 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:49.011586 31747 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.016073 31674 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:49.017032 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:49.017846   379 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:49.018330   379 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:20:49.019249 31747 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:49.019251   379 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:49.023432   429 ts_tablet_manager.cc:933] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:49.031984   429 tablet_copy_client.cc:323] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.193:41935
I20260709 12:20:49.033612 31687 tablet_copy_service.cc:140] P b95aeeaf4c51442b8092cff1de674f2b: Received BeginTabletCopySession request for tablet f6404cb9c2174326b06cbbb0d27d43f2 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34010)
I20260709 12:20:49.034190 31687 tablet_copy_service.cc:161] P b95aeeaf4c51442b8092cff1de674f2b: Beginning new tablet copy session on tablet f6404cb9c2174326b06cbbb0d27d43f2 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34010: session id = 5206fd0087404d2da964a5d985a04ff4-f6404cb9c2174326b06cbbb0d27d43f2
I20260709 12:20:49.034873 31571 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 1 to 2, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:49.042295 31687 tablet_copy_source_session.cc:215] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:49.047684   429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6404cb9c2174326b06cbbb0d27d43f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:49.076987   429 tablet_copy_client.cc:806] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:49.077720   429 tablet_copy_client.cc:670] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:49.082098   429 tablet_copy_client.cc:538] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:49.089071   331 raft_consensus.cc:993] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: : Instructing follower 039d66a6c85e43c3936bf5017501b344 to start an election
I20260709 12:20:49.089677   331 raft_consensus.cc:1081] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Signalling peer 039d66a6c85e43c3936bf5017501b344 to start an election
I20260709 12:20:49.091755 31747 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:45166
I20260709 12:20:49.092447 31747 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:49.092754 31747 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.099447   429 tablet_bootstrap.cc:492] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:49.100085 31747 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:49.103173 31747 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:49.103173 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:49.105189 31674 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.107749 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:49.108283 31827 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:49.108546 31827 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:49.109205 31827 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:49.113708 31674 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:20:49.113783 31827 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:49.116338 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:49.117233   416 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:49.117728   416 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:49.123932 31827 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:20:49.118606   416 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:49.133507   429 tablet_bootstrap.cc:492] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:49.134383   429 tablet_bootstrap.cc:492] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:49.135147   429 ts_tablet_manager.cc:1403] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.036s	user 0.023s	sys 0.008s
I20260709 12:20:49.138222   429 raft_consensus.cc:359] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:49.139099   429 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:49.139840   429 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:49.143464   429 ts_tablet_manager.cc:1434] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:20:49.149712 31687 tablet_copy_service.cc:342] P b95aeeaf4c51442b8092cff1de674f2b: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-f6404cb9c2174326b06cbbb0d27d43f2 received from {username='slave'} at 127.0.0.1:34010
I20260709 12:20:49.148000 31576 catalog_manager.cc:5697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 1 to 2, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:20:49.150218 31687 tablet_copy_service.cc:434] P b95aeeaf4c51442b8092cff1de674f2b: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-f6404cb9c2174326b06cbbb0d27d43f2 on tablet f6404cb9c2174326b06cbbb0d27d43f2 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:49.161617   429 ts_tablet_manager.cc:933] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:49.163439   429 tablet_copy_client.cc:323] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.193:41935
I20260709 12:20:49.165148 31687 tablet_copy_service.cc:140] P b95aeeaf4c51442b8092cff1de674f2b: Received BeginTabletCopySession request for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34010)
I20260709 12:20:49.165629 31687 tablet_copy_service.cc:161] P b95aeeaf4c51442b8092cff1de674f2b: Beginning new tablet copy session on tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34010: session id = 5206fd0087404d2da964a5d985a04ff4-b25009db0c5a47a9ab9f6f8b2e5c1b17
I20260709 12:20:49.172077 31687 tablet_copy_source_session.cc:215] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:20:49.175314   429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:20:49.193007   429 tablet_copy_client.cc:806] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:20:49.193586   429 tablet_copy_client.cc:670] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:20:49.197305   429 tablet_copy_client.cc:538] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:20:49.204306   429 tablet_bootstrap.cc:492] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:20:49.210745 31750 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:49.212272   379 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:49.278283   429 tablet_bootstrap.cc:492] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:20:49.279385   429 tablet_bootstrap.cc:492] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:20:49.280108   429 ts_tablet_manager.cc:1403] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.076s	user 0.019s	sys 0.004s
I20260709 12:20:49.287113 31673 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:49.286696   429 raft_consensus.cc:359] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:49.287829   429 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:20:49.290279   429 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:49.293367   429 ts_tablet_manager.cc:1434] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:20:49.299113 31687 tablet_copy_service.cc:342] P b95aeeaf4c51442b8092cff1de674f2b: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-b25009db0c5a47a9ab9f6f8b2e5c1b17 received from {username='slave'} at 127.0.0.1:34010
I20260709 12:20:49.299669 31687 tablet_copy_service.cc:434] P b95aeeaf4c51442b8092cff1de674f2b: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-b25009db0c5a47a9ab9f6f8b2e5c1b17 on tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:49.299847   452 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 12:20:49.338737   400 raft_consensus.cc:1064] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:49.345760 31748 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:49.348574   452 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:49.349542   400 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:49.364254 31674 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:49.373761 31750 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:49.374125   331 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:49.386886 31673 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:49.391402   336 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:49.398559 32377 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:49.400117   399 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:49.413101   457 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:49.417191 31827 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:49.420006   416 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:49.440469 32375 raft_consensus.cc:1217] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:49.449944   416 raft_consensus.cc:1064] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:49.451639 31750 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:49.452649   416 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:49.454923   379 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:49.468264 32376 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:49.473448   396 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:49.509063 31674 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:49.519320   336 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:49.520277 31824 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:49.532423   400 raft_consensus.cc:2955] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:20:49.572999 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48307.6us
W20260709 12:20:49.796586 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 101127us
I20260709 12:20:49.823529 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:20:49.891527   462 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
W20260709 12:20:49.893071 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 193245us
I20260709 12:20:49.944096 31747 raft_consensus.cc:2955] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:49.896373   467 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
W20260709 12:20:50.053020   472 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
W20260709 12:20:50.076025 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 106406us
I20260709 12:20:50.171957 31674 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:20:50.172992 31674 raft_consensus.cc:606] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Received request to transfer leadership to TS 5206fd0087404d2da964a5d985a04ff4
W20260709 12:20:50.201732 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 68173.3us
I20260709 12:20:50.245934 31573 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:50.288066 32377 raft_consensus.cc:2955] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:20:50.292321   400 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.704s	user 0.014s	sys 0.001s
W20260709 12:20:50.293268 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 112638us
I20260709 12:20:50.303844 32375 raft_consensus.cc:1217] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2]   Dedup: 1.2->[]
I20260709 12:20:50.343809 31748 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:20:50.344604 31748 raft_consensus.cc:606] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
W20260709 12:20:50.346714 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 139145us
I20260709 12:20:50.356379 31748 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:20:50.357995 31748 raft_consensus.cc:606] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
W20260709 12:20:50.352720 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54726.3us
W20260709 12:20:50.387831 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 191508us
I20260709 12:20:50.394785 31674 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:20:50.395541 31674 raft_consensus.cc:606] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Received request to transfer leadership to TS 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:50.402968 31676 raft_consensus.cc:2955] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:50.408341 31827 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:50.416759 31673 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:20:50.417440 31673 raft_consensus.cc:606] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Received request to transfer leadership to TS 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:50.464490   467 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:50.473610 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:20:50.475523 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:50.476387 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:20:50.516705   509 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:20:50.527524   479 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:50.528379   479 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:50.533583   479 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:50.548012   467 raft_consensus.cc:993] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:20:50.548832   467 raft_consensus.cc:1081] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Signalling peer 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:20:50.557737   506 raft_consensus.cc:1064] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:50.561271   506 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:50.565977   495 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:50.573323 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:50.633167   495 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:50.594060   507 raft_consensus.cc:993] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: : Instructing follower 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:50.607823 31826 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:41204
I20260709 12:20:50.635179   507 raft_consensus.cc:1081] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Signalling peer 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:50.635200 31826 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:50.635860 31826 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:50.636111   495 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:50.637444 31820 raft_consensus.cc:2955] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:50.643556 31826 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:50.645807 31826 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:50.655268 32372 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3"
dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:33962
I20260709 12:20:50.656008 32372 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:50.656450 32372 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:50.662701 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:50.687552 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:50.688405 31676 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:50.697791 31676 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:50.699582 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:50.700414 31748 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:50.700788 31748 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:50.701663 31748 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:50.631050   484 raft_consensus.cc:2955] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:50.705354 31748 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:50.720595 32372 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:50.732812 32372 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:50.736620 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:50.737380 31820 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:50.726178 31748 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:50.742223 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:50.743144   490 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:50.743741   490 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:50.744740   490 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:50.748476 31820 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:50.758885 31676 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:50.759317   505 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:50.759605   406 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:50.759876   505 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:50.760355   406 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:50.763861 31671 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:50.762502   505 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:50.783732 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 97473.1us
I20260709 12:20:50.791857 31673 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:50.800545 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:20:50.801627   479 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:50.802160   479 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:20:50.803040   479 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:50.808292   473 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:50.809033   473 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:50.819794   514 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:50.820439   514 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:50.839691   514 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:50.841545 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:50.844671   473 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:50.795925   406 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:50.845178 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:50.846210 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:50.846923 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:50.848927 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:50.849674 31674 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:50.850098 31674 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:50.851106 31674 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:50.852674 31674 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:50.853166 31827 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:50.854832 32373 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:50.861691   506 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260709 12:20:50.880600 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:50.883495 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:50.883644   379 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:20:50.885298 31674 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:50.888235   498 raft_consensus.cc:993] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:20:50.888777   498 raft_consensus.cc:1081] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Signalling peer 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:20:50.889079   406 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:20:50.919862   406 raft_consensus.cc:993] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: : Instructing follower 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:50.920349   406 raft_consensus.cc:1081] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Signalling peer 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:50.922221 32377 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69"
dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:33962
I20260709 12:20:50.923005 32377 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:50.923467 32377 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:50.934198 32377 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:50.935204 31677 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:50.939893 32377 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:50.945029 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:50.945881 31677 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:50.946429 31677 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:50.947319 31677 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:50.948503 31677 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:50.950007   467 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:50.950700   467 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:50.952859   467 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:50.953593 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:51.016155   473 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:51.017524   473 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:51.022305   473 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:51.053848   473 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:51.054793   473 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:51.057778   473 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:51.069007 31677 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:51.129371   481 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:20:51.385113   525 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.310s	user 0.001s	sys 0.000s
W20260709 12:20:51.444007 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:51.638975   481 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:51.639730   476 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:20:51.477332 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:51.640306   476 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:51.641235   481 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:51.642179 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:51.518297   488 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:51.643121   488 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:51.643482 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:51.645251   488 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:51.645982 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:51.520416   527 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:51.647125   527 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:51.648252   476 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:51.647490 31573 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 1 to 2, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:51.649288   527 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:51.524240 31572 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42070 (request call id 219) took 1059 ms. Trace:
I20260709 12:20:51.535351 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:51.543969   528 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:51.651795 31747 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:51.651865   528 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:51.545653   529 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:51.655052 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:51.654919   529 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:51.657245   529 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:51.546610   498 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:51.547104   468 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:20:51.548856 31827 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.053s	sys 0.078s
W20260709 12:20:51.660633 31827 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.766s	user 0.056s	sys 0.079s
W20260709 12:20:51.561473 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 590156us
I20260709 12:20:51.564666   531 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:51.564139 32373 raft_consensus.cc:2955] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:20:51.598143 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.081s	user 0.022s	sys 0.035s
W20260709 12:20:51.664492 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.150s	user 0.022s	sys 0.035s
W20260709 12:20:51.664891 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 1.18208e+06us
I20260709 12:20:51.666361 31747 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:51.620503 31571 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 232) took 1266 ms. Trace:
W20260709 12:20:51.628230 32375 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.733s	user 0.013s	sys 0.004s
W20260709 12:20:51.673566 32375 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.014s	sys 0.004s
I20260709 12:20:51.646539 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:51.659982 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:51.674486 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:51.674255 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:51.675704 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:51.654425   528 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:51.681339 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:51.683759 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:51.689281 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:51.689411 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:51.701928 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:51.678339 31571 rpcz_store.cc:276] 0709 12:20:50.353744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:50.353894 (+   150us) service_pool.cc:224] Handling call
0709 12:20:51.620432 (+1266538us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":615896,"reported_tablets":1}
I20260709 12:20:51.661648 31572 rpcz_store.cc:276] 0709 12:20:50.464455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:50.468638 (+  4183us) service_pool.cc:224] Handling call
0709 12:20:51.524095 (+1055457us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2146,"reported_tablets":1}
I20260709 12:20:51.727769 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:51.726974 31678 raft_consensus.cc:2955] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:51.731341 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:51.732703   412 raft_consensus.cc:993] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: : Instructing follower 039d66a6c85e43c3936bf5017501b344 to start an election
I20260709 12:20:51.732754   507 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:51.733278   412 raft_consensus.cc:1081] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Signalling peer 039d66a6c85e43c3936bf5017501b344 to start an election
I20260709 12:20:51.733621 31747 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:51.735345 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:51.736411 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:51.743232 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:51.748662   539 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:51.749239   539 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:51.751991 31827 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:51.766247 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:51.769236 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:51.770159 31827 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:41204
I20260709 12:20:51.742062   412 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:51.771384 31827 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:51.775992 31827 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:51.747258   514 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:51.781661 31753 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:45178
I20260709 12:20:51.784786   539 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:51.779587 31820 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:51.786974 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:51.785873 31748 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:51.799717 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:51.770517   505 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:51.773527   479 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:51.813165 31677 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:51.867770 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:51.876050 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 82284us
W20260709 12:20:51.884882 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:51.797191   468 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:51.778148   507 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.108s
I20260709 12:20:51.853624   479 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:20:51.914136 31827 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:51.868664   478 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
W20260709 12:20:51.924790   537 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 12:20:51.952718   498 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:51.953442   498 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:51.960875   498 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:51.964811 31753 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:51.936437 31827 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:52.219420 31753 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:51.965684 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:20:52.217849 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 342146us
I20260709 12:20:52.225163 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:51.942724   478 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:52.230603 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:52.231312 31749 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:52.231580 31749 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:52.232146 31749 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:52.232784 31749 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:20:52.246032 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:52.153663   532 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:20:52.115245 31774 debug-util.cc:398] Leaking SignalData structure 0x7b0800165120 after lost signal to thread 30506
I20260709 12:20:52.203640   543 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:52.236764   478 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:52.252164 31749 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:20:52.290017 31753 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:52.242195   490 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:52.296509 31753 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:52.248335   465 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:52.300673 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:52.301137 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:52.322561 31678 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:20:52.323817 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74450.3us
I20260709 12:20:52.303596   336 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 12:20:52.313108   490 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:52.327806 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:52.325845 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:52.329792 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:52.330351 31826 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.126881   479 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:52.304319 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44880.8us
I20260709 12:20:52.295122 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 240) took 1237 ms. Trace:
I20260709 12:20:52.335117   452 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:52.337317 31678 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:52.340111 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:52.339277 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:52.340986 31820 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:52.342094 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:52.343784 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:52.262753   532 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:52.371012 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:52.326030 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:52.341944 31774 debug-util.cc:398] Leaking SignalData structure 0x7b0800165800 after lost signal to thread 31610
W20260709 12:20:52.379483 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:52.300511   543 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:52.332996   465 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:52.387092   548 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260709 12:20:52.373629   532 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:52.388800   481 raft_consensus.cc:1064] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:20:52.392868 31753 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:52.389400   399 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:52.399566   536 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:52.334858 32373 rpcz_store.cc:276] 0709 12:20:51.058049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.563267 (+505218us) service_pool.cc:224] Handling call
0709 12:20:51.563708 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:51.563978 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:20:51.771055 (+207077us) log.cc:844] Serialized 14 byte log entry
0709 12:20:52.294798 (+523743us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:52.294865 (+    67us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:20:52.294904 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:52.294962 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:52.295065 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":38172,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":7353,"thread_start_us":459898,"threads_started":1}
I20260709 12:20:52.404201   533 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.013s
W20260709 12:20:52.424903 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:52.426242 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 89678.1us
I20260709 12:20:52.432574 31826 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
W20260709 12:20:52.439002   546 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.547s	user 0.014s	sys 0.015s
I20260709 12:20:52.412268   505 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
W20260709 12:20:52.442320   550 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:20:52.454936 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:52.431401 31774 debug-util.cc:398] Leaking SignalData structure 0x7b0800333c40 after lost signal to thread 31702
I20260709 12:20:52.457482   465 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:20:52.457795 31774 debug-util.cc:398] Leaking SignalData structure 0x7b08002f73e0 after lost signal to thread 31777
I20260709 12:20:52.459285   543 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:52.459475 31774 debug-util.cc:398] Leaking SignalData structure 0x7b0800354440 after lost signal to thread 31851
I20260709 12:20:52.440743   490 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:52.449689   544 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:52.449851   489 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:20:52.450373 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:52.459187 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:52.444913   412 raft_consensus.cc:2955] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:20:52.460988 31774 debug-util.cc:398] Leaking SignalData structure 0x7b0800354d20 after lost signal to thread 32402
I20260709 12:20:52.461678   544 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:52.470296   489 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:52.470919   514 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:20:52.492944 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 100612us
W20260709 12:20:52.493932 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 155360us
W20260709 12:20:52.487212 31774 debug-util.cc:398] Leaking SignalData structure 0x7b0800354d40 after lost signal to thread 32403
I20260709 12:20:52.498049 31749 raft_consensus.cc:2955] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:52.501571 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:52.507675   553 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:20:52.509966 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 87366.2us
I20260709 12:20:52.513178 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:52.487143   543 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:52.517779 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:52.521715 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:52.494908   514 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:52.503219   478 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:20:52.524436 31774 debug-util.cc:398] Leaking SignalData structure 0x7b0800350120 after lost signal to thread 475
I20260709 12:20:52.508368   553 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:52.538506   532 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:52.423666   536 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:52.535871   543 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:52.598682 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61023.2us
I20260709 12:20:52.587266   569 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.588945 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:52.589514 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 64137.3us
I20260709 12:20:52.592224 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:52.594424 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:52.542729   553 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:52.604044 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 176817us
I20260709 12:20:52.605576 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:52.606850 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:20:52.611471 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 313637us
I20260709 12:20:52.609411   554 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:52.614447 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:52.614924   544 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:52.617554 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:52.618508   535 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:52.619554 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:52.618916 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:52.549063 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:52.636106 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:52.545326   489 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:52.637059   406 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:52.638302   544 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:52.638900   544 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:52.635653 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:52.613168   554 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:52.583652   514 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:52.663625 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 46127us
I20260709 12:20:52.612071   465 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:20:52.672595   555 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.204s	user 0.001s	sys 0.000s
I20260709 12:20:52.673087   543 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:52.672857   465 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:52.668200 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49880us
I20260709 12:20:52.672060   551 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:52.611321   569 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:52.744817 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:52.668589   566 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:52.764932   544 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:52.793390 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:52.813117 31818 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:52.821111 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 977) took 1046 ms. Trace:
I20260709 12:20:52.821591 31750 rpcz_store.cc:276] 0709 12:20:51.774850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.780581 (+  5731us) service_pool.cc:224] Handling call
0709 12:20:52.223249 (+442668us) spinlock_profiling.cc:243] Waited 2.93 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:20:52.223404 (+   155us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:52.372300 (+148896us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:52.372332 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:52.372373 (+    41us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:20:52.372416 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:52.820850 (+448434us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400404cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:20:52.820911 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:52.821043 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":348796288}
I20260709 12:20:52.824374   570 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:52.817816   514 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:52.775236   551 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:52.845633 31573 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 145) took 1927 ms. Trace:
I20260709 12:20:52.834203   570 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:52.845757   465 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:52.856781   554 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:52.856312 31572 catalog_manager.cc:5697] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:52.859093   481 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:52.883150   569 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:52.912456 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 373297us
I20260709 12:20:52.913740 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:52.899900   574 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:20:52.846194 31573 rpcz_store.cc:276] 0709 12:20:50.918310 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:50.919310 (+  1000us) service_pool.cc:224] Handling call
0709 12:20:52.845544 (+1926234us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:51.709568 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:51.719591 (+ 10023us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:51.719627 (+    36us) write_op.cc:435] Acquired schema lock
0709 12:20:51.719651 (+    24us) tablet.cc:662] Decoding operations
0709 12:20:51.720036 (+   385us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:51.720093 (+    57us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:51.720136 (+    43us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:20:51.720343 (+   207us) tablet.cc:701] Row locks acquired
0709 12:20:51.720365 (+    22us) write_op.cc:260] PREPARE: finished
0709 12:20:51.720515 (+   150us) write_op.cc:270] Start()
0709 12:20:51.720639 (+   124us) write_op.cc:276] Timestamp: P: 1783599651720465 usec, L: 0
0709 12:20:51.720667 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:20:51.782090 (+ 61423us) log.cc:844] Serialized 539 byte log entry
0709 12:20:52.313485 (+531395us) op_driver.cc:464] REPLICATION: finished
0709 12:20:52.587891 (+274406us) spinlock_profiling.cc:243] Waited 4 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8f9ccb0ac 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe90080bef0
0709 12:20:52.590666 (+  2775us) write_op.cc:301] APPLY: starting
0709 12:20:52.590764 (+    98us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:52.590872 (+   108us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:52.590897 (+    25us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:52.591174 (+   277us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:52.591341 (+   167us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:20:52.591390 (+    49us) write_op.cc:312] APPLY: finished
0709 12:20:52.591593 (+   203us) log.cc:844] Serialized 23 byte log entry
0709 12:20:52.595181 (+  3588us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:52.595280 (+    99us) write_op.cc:454] Released schema lock
0709 12:20:52.595322 (+    42us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":600123,"reported_tablets":2,"child_traces":[["op",{"apply.queue_time_us":265065,"apply.run_cpu_time_us":1653,"apply.run_wall_time_us":7725,"num_ops":1,"prepare.queue_time_us":58206,"prepare.run_cpu_time_us":3544,"prepare.run_wall_time_us":97564,"replication_time_us":529717,"spinlock_wait_cycles":9204480,"thread_start_us":316636,"threads_started":2}]]}
I20260709 12:20:52.925072 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:52.957686 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:52.962481 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:52.984421   568 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:52.927800   536 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:53.019277   557 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:52.954851 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:53.032956 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1410) took 1116 ms. Trace:
I20260709 12:20:53.033535 31825 rpcz_store.cc:276] 0709 12:20:51.911157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.923080 (+ 11923us) service_pool.cc:224] Handling call
0709 12:20:52.919439 (+996359us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:20:52.928309 (+  8870us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:52.978505 (+ 50196us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:52.984519 (+  6014us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:52.998018 (+ 13499us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:20:53.009222 (+ 11204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:53.013555 (+  4333us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:53.025068 (+ 11513us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37919104}
I20260709 12:20:53.040369   452 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:53.033056   558 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:20:53.065968 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 140876us
I20260709 12:20:53.067363   577 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.068150   577 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:53.072217   551 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:53.074322 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:53.075806 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1309) took 1017 ms. Trace:
W20260709 12:20:53.082249 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 358207us
W20260709 12:20:53.106305 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 438244us
I20260709 12:20:53.107564 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:53.115746 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:53.076463 31818 rpcz_store.cc:276] 0709 12:20:52.059006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:52.059208 (+   202us) service_pool.cc:224] Handling call
0709 12:20:52.410312 (+351104us) spinlock_profiling.cc:243] Waited 4.23 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:20:52.410550 (+   238us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:52.813641 (+403091us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:53.075402 (+261761us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b54004466d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:20:53.075717 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":606041088}
I20260709 12:20:53.117362 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:53.170125 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 86810.9us
I20260709 12:20:53.171073 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:53.171396   570 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:53.177815 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:53.178699 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:53.182898 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:53.170205 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62763.3us
I20260709 12:20:53.190362 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:53.183300   336 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:53.192507 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:53.182420   478 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:53.210315 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:53.189092   545 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:53.224363 31677 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:53.224834 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:53.208087 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:53.235342 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:53.251984 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:53.255646   548 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.256309   548 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.257697   570 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:53.261636   576 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:53.263015   452 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:53.262531   576 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:53.264854   452 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:53.245685 31818 raft_consensus.cc:2955] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:20:53.282899 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:53.284111 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 282877us
I20260709 12:20:53.285811 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:53.304682 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:53.299010 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:53.316453 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:53.327042 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.011s	sys 0.075s
W20260709 12:20:53.327688 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.011s	sys 0.075s
W20260709 12:20:53.328105 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 657403us
W20260709 12:20:53.330539 31678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.004s	sys 0.010s
I20260709 12:20:53.332197 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:53.332453 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:53.331049 31678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.504s	user 0.005s	sys 0.010s
I20260709 12:20:53.279631   577 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:53.335892   587 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:53.298920 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:53.327435   528 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:53.337399   467 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:53.327273   558 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:53.356570   467 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:53.350518 31671 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 1.
I20260709 12:20:53.374800   467 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:53.350934   452 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:53.345165   531 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:53.379261   465 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.399966   489 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.400709   489 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:53.382880   465 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.406754   581 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.342684 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:20:53.416335 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 73755us
I20260709 12:20:53.406052   489 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:53.415535   578 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:53.408980   505 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.436254   498 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:53.417938   558 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:53.422570 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:53.441815 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 484244us
I20260709 12:20:53.443852 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:53.444912   399 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:53.447126   465 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:53.458062   578 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.452745   588 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:53.453869   543 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.466893   543 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:53.469201   543 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:53.459564 32375 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20260709 12:20:53.433732 31641 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.038s	sys 0.063s
I20260709 12:20:53.471107   399 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:53.489075   558 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:53.407569   581 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.490479   558 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.447808   505 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:53.479087   531 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:53.441679   548 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:53.447906   498 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:53.509706 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:53.511361 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:53.470947   588 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.506121   484 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:20:53.539999 31641 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.791s	user 0.039s	sys 0.063s
I20260709 12:20:53.544411 31749 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:53.540952   484 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.501921   505 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:53.548797   484 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:53.562672 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 89582.7us
I20260709 12:20:53.565380 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:53.516113   399 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:53.572165 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:53.575716 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:53.576584 31750 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 1.
I20260709 12:20:53.581770   578 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:53.572192   468 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.044s	sys 0.048s
I20260709 12:20:53.608048 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:53.609020 32378 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:53.570699   484 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:20:53.611549 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 389030us
I20260709 12:20:53.615077 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:53.604532   468 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.045s	sys 0.049s
W20260709 12:20:53.626483 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 405977us
I20260709 12:20:53.576149   533 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260709 12:20:53.636991 31791 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:53.655256 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:53.679116 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:53.679472 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:53.679745 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:53.681715 32377 raft_consensus.cc:2955] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:53.634465   495 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.686322 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:53.577385   576 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:53.686522 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:53.687129 31673 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:53.688966 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:53.689308 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:53.636525   491 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:53.695721 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:53.669708   406 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:53.680801   484 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:53.680967   465 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:53.694121   588 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:53.693714 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:53.697710   584 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.694746 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:53.721489   498 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:53.725543 31751 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 1.
I20260709 12:20:53.706197 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:53.694521   495 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.715350   406 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.749814   556 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:53.704224   491 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:53.735806   468 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:53.755687 31674 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:53.744103   554 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:53.723047   465 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:53.809098   581 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:53.812769 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:53.809099   541 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.809307   560 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.817546 31674 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:53.815158   563 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:53.815850   599 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:53.750571   556 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.904318   491 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:53.812800   576 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:53.916604   597 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:53.892387   541 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.926908   598 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.777588   468 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.914672   406 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:53.914294 31671 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
W20260709 12:20:53.976444 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 280232us
I20260709 12:20:53.918146   399 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.941459   479 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:54.009493   561 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:53.915325   560 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.919467   489 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:54.010200   561 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:54.009833   479 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:53.927673   598 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:53.955533 31669 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.696s	user 0.003s	sys 0.025s
I20260709 12:20:53.929759   581 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:53.918900   576 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:53.950832 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:53.951752   495 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:53.948096   564 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:53.917287   597 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:53.755574   584 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:53.919152   599 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:54.061190   599 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:54.066751 31644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.033s	sys 0.086s
I20260709 12:20:53.948381   602 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:20:54.029423   590 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.779s	user 0.065s	sys 0.124s
I20260709 12:20:53.936115   563 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:54.081034   590 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.065s	sys 0.125s
W20260709 12:20:54.068559 31644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.033s	sys 0.086s
I20260709 12:20:54.083602 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:54.084437 31671 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:54.085505 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:54.086870   479 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:54.025399 31669 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:20:54.014756   399 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:54.099798   465 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:54.099677   584 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:54.067394   564 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:54.104104   563 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:54.013926   489 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:54.029320   556 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:54.087548 32343 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:54.111567   489 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:54.031016   581 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:54.136152   597 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:54.133247   598 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:54.130364 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 77376.2us
I20260709 12:20:54.119493   556 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:54.119140   578 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:54.128101   595 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:54.143702   479 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:54.147910   541 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:54.147817   479 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:54.119762   484 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:54.136206   576 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:54.102797   602 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:54.148727   556 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:54.154261   578 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:54.163058   595 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:54.488039 31790 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.013s	sys 0.024s
W20260709 12:20:54.546253 31790 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.644s	user 0.014s	sys 0.026s
I20260709 12:20:54.464564 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:54.557698   336 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:20:54.559312 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 94908.3us
W20260709 12:20:54.453783 31818 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.075s	user 0.010s	sys 0.005s
I20260709 12:20:54.463182   582 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:54.562875   582 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:54.324023 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:54.521411 31672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.704s	user 0.016s	sys 0.025s
W20260709 12:20:54.567315 31672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.017s	sys 0.025s
I20260709 12:20:54.447501   399 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:20:54.362134 31669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1327) took 1916 ms (client timeout 1781 ms). Trace:
I20260709 12:20:54.416566 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 245) took 1212 ms. Trace:
I20260709 12:20:54.321482   597 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:54.351359 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 896) took 1758 ms. Trace:
W20260709 12:20:54.541775 31717 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.781s (SENT) [suppressed 1 similar messages]
I20260709 12:20:54.542562 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:54.543314 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:54.527024 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:54.571067 31678 rpcz_store.cc:276] 0709 12:20:52.593372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:52.593527 (+   155us) service_pool.cc:224] Handling call
0709 12:20:52.593860 (+   333us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:52.795849 (+201989us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:52.795883 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:52.804169 (+  8286us) log.cc:844] Serialized 26 byte log entry
0709 12:20:53.339925 (+535756us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:20:53.339968 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:54.329509 (+989541us) spinlock_profiling.cc:243] Waited 183 ms on lock 0x7b54002e6b54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:20:54.329573 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:54.329639 (+    66us) raft_consensus.cc:1685] finished
0709 12:20:54.329660 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:54.351201 (+ 21541us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":24754,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":136,"spinlock_wait_cycles":422040448,"thread_start_us":512663,"threads_started":1,"wal-append.queue_time_us":645599,"wal-append.run_cpu_time_us":9494,"wal-append.run_wall_time_us":1006996}
W20260709 12:20:54.788887 31793 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.891s	user 0.020s	sys 0.040s
W20260709 12:20:55.308511 31793 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.411s	user 0.030s	sys 0.048s
I20260709 12:20:55.058029   578 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:55.283392 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.709s	user 0.008s	sys 0.031s
W20260709 12:20:55.309767 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.735s	user 0.010s	sys 0.032s
I20260709 12:20:54.651520 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:54.979868   587 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.967s	user 0.035s	sys 0.046s
W20260709 12:20:54.788403 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.876s	user 0.028s	sys 0.009s
W20260709 12:20:55.311419   587 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.299s	user 0.035s	sys 0.046s
W20260709 12:20:55.311875 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.400s	user 0.033s	sys 0.022s
I20260709 12:20:54.722595   593 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.297892 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:54.709843 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:54.668318 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:54.636628 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:54.850917 32344 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.781s	user 0.003s	sys 0.025s
W20260709 12:20:54.177057   467 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.801s	user 0.012s	sys 0.002s
W20260709 12:20:55.086108 32377 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.298s	user 0.005s	sys 0.073s
W20260709 12:20:54.628793 32341 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.006s	sys 0.042s
I20260709 12:20:54.569352 32374 rpcz_store.cc:276] 0709 12:20:53.203863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.203979 (+   116us) service_pool.cc:224] Handling call
0709 12:20:54.415394 (+1211415us) spinlock_profiling.cc:243] Waited 9.38 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:20:54.415697 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:54.415967 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:54.416012 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:54.416085 (+    73us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:20:54.416132 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:54.416222 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:54.416431 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21582080}
I20260709 12:20:54.570406   564 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:20:54.483719   412 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.066s	user 0.019s	sys 0.034s
I20260709 12:20:55.026465   556 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:55.309005 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 748130us
W20260709 12:20:55.289989 31818 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.001s	sys 0.010s
W20260709 12:20:55.260850 31644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.159s	user 0.013s	sys 0.013s
I20260709 12:20:54.792196 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 907) took 1113 ms. Trace:
I20260709 12:20:55.343441 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1007) took 1033 ms. Trace:
I20260709 12:20:55.344061 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1005) took 1037 ms. Trace:
W20260709 12:20:55.344976 31818 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.001s	sys 0.010s
I20260709 12:20:55.344707 31750 rpcz_store.cc:276] 0709 12:20:54.306280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:54.307126 (+   846us) service_pool.cc:224] Handling call
0709 12:20:55.343962 (+1036836us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:55.343943 31747 rpcz_store.cc:276] 0709 12:20:54.309871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:54.310012 (+   141us) service_pool.cc:224] Handling call
0709 12:20:55.343211 (+1033199us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:55.346560 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1422) took 2165 ms. Trace:
W20260709 12:20:55.251789 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.240s	user 0.007s	sys 0.016s
W20260709 12:20:55.347463 31644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.246s	user 0.013s	sys 0.024s
I20260709 12:20:55.342970 31677 rpcz_store.cc:276] 0709 12:20:53.678602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.678828 (+   226us) service_pool.cc:224] Handling call
0709 12:20:53.679083 (+   255us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:53.780534 (+101451us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:53.780584 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:53.780650 (+    66us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:20:53.780701 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:54.791850 (+1011149us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54001d5e54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:20:54.791902 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:54.792107 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":232731520}
W20260709 12:20:55.354686 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.344s	user 0.017s	sys 0.023s
I20260709 12:20:55.365186 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:55.366608 31823 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
W20260709 12:20:55.368381 32341 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.443s	user 0.022s	sys 0.068s
W20260709 12:20:54.205096   509 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.009s	sys 0.019s
W20260709 12:20:55.371138   509 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.768s	user 0.031s	sys 0.045s
W20260709 12:20:54.568332 31669 rpcz_store.cc:269] 0709 12:20:52.446103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:52.446330 (+   227us) service_pool.cc:224] Handling call
0709 12:20:54.361835 (+1915505us) spinlock_profiling.cc:243] Waited 734 ms on lock 0x7b540017c5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:20:54.362009 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1688087936}
I20260709 12:20:55.373508   484 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:20:55.374766 32344 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.270s	user 0.007s	sys 0.047s
I20260709 12:20:55.376480 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:54.720536 31641 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.706s	user 0.010s	sys 0.012s
W20260709 12:20:54.640281   558 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.911s	user 0.007s	sys 0.028s
I20260709 12:20:55.380466   479 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:55.374567   484 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:55.404196   484 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:54.173044   468 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:54.419836 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.413940 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1004) took 1044 ms. Trace:
I20260709 12:20:54.736943   560 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:55.378787 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:54.247129   602 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:55.347374   572 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:55.443351   593 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.446127   593 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:20:55.447206 31641 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.381s	user 0.013s	sys 0.024s
I20260709 12:20:55.448334 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1003) took 1143 ms. Trace:
I20260709 12:20:55.448786 31748 rpcz_store.cc:276] 0709 12:20:54.304503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:54.304702 (+   199us) service_pool.cc:224] Handling call
0709 12:20:54.531922 (+227220us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:54.662414 (+130492us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:54.662462 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:54.662518 (+    56us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:20:54.662555 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:55.447670 (+785115us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54002f62d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:20:55.447870 (+   200us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:55.448126 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":271031040}
W20260709 12:20:55.450742   412 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.940s	user 0.041s	sys 0.049s
I20260709 12:20:55.455057   556 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:55.456123   556 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.461153   561 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.462107   561 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:55.462878   561 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.463725   582 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.465353   561 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:55.464884   597 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.468117   597 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.469059   583 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.468967   467 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.470037   583 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.470981   581 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.471873   581 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:55.472785   583 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:55.472668   581 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.473443   583 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.474908 31826 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:55.424862 31749 rpcz_store.cc:276] 0709 12:20:54.304886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:54.305564 (+   678us) service_pool.cc:224] Handling call
0709 12:20:55.314827 (+1009263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:55.342250 (+ 27423us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:55.342301 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:55.342368 (+    67us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:20:55.342402 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:55.342495 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:55.348940 (+  6445us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:20:55.480291   479 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.481282   479 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.484426   479 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.378731   528 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:20:55.408969 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 1.39781e+06us
W20260709 12:20:55.438489 32377 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.589s	user 0.007s	sys 0.077s
I20260709 12:20:55.491080   582 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:20:55.491654   558 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.684s	user 0.020s	sys 0.047s
I20260709 12:20:55.492008   582 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.446429   548 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:55.347057 31818 rpcz_store.cc:276] 0709 12:20:53.181004 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.182341 (+  1337us) service_pool.cc:224] Handling call
0709 12:20:53.231082 (+ 48741us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:53.242748 (+ 11666us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:20:54.561489 (+1318741us) log.cc:844] Serialized 14 byte log entry
0709 12:20:55.345971 (+784482us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:55.346076 (+   105us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:20:55.346163 (+    87us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:55.346281 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:55.346475 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":783712,"threads_started":1}
I20260709 12:20:55.495493 32370 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:55.497967   579 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:20:55.509723 32370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 371) took 1861 ms (client timeout 1842 ms). Trace:
I20260709 12:20:55.495968   595 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.485198   479 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:20:55.474294   583 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.531831 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 370) took 1890 ms. Trace:
W20260709 12:20:55.550392 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49192.1us
I20260709 12:20:55.551096   587 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.552275   587 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.554775   528 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.558872   582 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.559739   582 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.560096   467 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.560618   582 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.510079   579 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.538918 32377 rpcz_store.cc:276] 0709 12:20:53.636978 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.637107 (+   129us) service_pool.cc:224] Handling call
0709 12:20:53.681255 (+ 44148us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:53.681547 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:20:53.762060 (+ 80513us) log.cc:844] Serialized 14 byte log entry
0709 12:20:55.519918 (+1757858us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:55.520031 (+   113us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:20:55.520126 (+    95us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:55.527497 (+  7371us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:55.527756 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8204,"thread_start_us":1730363,"threads_started":1}
I20260709 12:20:55.503970   593 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:55.529284   597 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:55.495602   548 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:55.510396 32370 rpcz_store.cc:269] 0709 12:20:53.648054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.673787 (+ 25733us) service_pool.cc:224] Handling call
0709 12:20:54.419647 (+745860us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:20:55.509549 (+1089902us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24012160}
I20260709 12:20:55.576221   624 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.558429   528 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.588505   618 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:55.593060   578 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:55.593420   618 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:55.574596   543 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:20:55.604260 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 1.05947e+06us
I20260709 12:20:55.590452   572 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:55.541832 31677 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:20:55.612578   491 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:55.615370   631 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:55.615862   583 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.616241   568 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.616858   484 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:55.629674   484 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.616875   597 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.618906   548 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.619695   468 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.620541   581 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:55.587275   602 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.601504   618 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.613309   620 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:20:55.620572 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 306018us
I20260709 12:20:55.627014 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:55.621544   507 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:55.634114   556 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.609956 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:55.631363 31677 raft_consensus.cc:606] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Received request to transfer leadership to TS 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:55.648128   620 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.648291   468 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.647902   564 raft_consensus.cc:2955] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:55.651501   620 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.654731   624 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:55.656584 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 92776us
I20260709 12:20:55.657621   624 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.657994   604 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:55.658546   604 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:55.658991   604 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.634552   484 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.662526   578 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.634469   491 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.664302   631 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.666706   579 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.634546   597 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.629693   568 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.609651   561 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.623904   593 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.635800   583 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.636741   467 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.720959   602 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.723430   507 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.729158 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:55.752357 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 84455us
I20260709 12:20:55.727394   582 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.755739 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.756803   491 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:55.757673 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 174828us
I20260709 12:20:55.758023   507 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:55.750799 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 405982us
I20260709 12:20:55.760332   467 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.758669   623 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:55.780274 32374 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:55.761104 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.762356   619 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.781821 32378 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:55.782038   616 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.782006   619 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.782891   616 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.763069   602 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:55.764746   632 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.785297   616 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:55.765470 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.676185   572 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.785804   632 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.786363 32373 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:55.788236   568 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.788472   632 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.790959 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:55.791772 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:55.792596 31818 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:55.794375 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:55.795248 31818 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 1.
I20260709 12:20:55.796396 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:55.767941 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.770006 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.774236 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.801234 31670 raft_consensus.cc:1813] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Handling vote request from an unknown peer 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:55.802019 31670 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:20:55.758647   484 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:55.805526   631 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.774956 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:55.776302 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.766054   578 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.810966 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:55.811776   587 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.790983   593 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.813148 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:55.811004 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:55.772918 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45018us
I20260709 12:20:55.771265 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.780931   623 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.759123 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.822845 32370 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:55.771484   468 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.824394 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.784719   467 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.784756   619 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:55.794698   625 raft_consensus.cc:993] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: : Instructing follower 039d66a6c85e43c3936bf5017501b344 to start an election
W20260709 12:20:55.825250 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66959.7us
I20260709 12:20:55.805655 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:55.775504   584 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:55.836423 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.838083 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:55.838582   584 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.840130   587 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:55.840636   587 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.841055   587 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.841399   584 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.732059   556 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:55.782827 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.807837   572 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.817121   602 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:55.817731 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:55.818558   623 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.848491 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.820600 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.879361 32373 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:55.822075 32378 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 LEARNER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:55.766822 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.881893 31677 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:55.829818   625 raft_consensus.cc:1081] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Signalling peer 039d66a6c85e43c3936bf5017501b344 to start an election
I20260709 12:20:55.830407 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.856096 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:20:55.819269 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:55.886686 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:20:55.887060   639 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:55.887670   639 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.888159   639 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.887868 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:55.887887 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:20:55.889119 31748 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:55.889653   468 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:55.890158   468 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.890594   468 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.890801   593 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.891239   467 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:55.850183 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.867158 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:55.849918 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.848667 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:20:55.877604 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60046.5us
I20260709 12:20:55.899427 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:55.847723 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.901050 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:55.901160 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:55.878422   572 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:20:55.902302   572 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:55.902853   572 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:55.891815   593 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.904300   637 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:55.891745   467 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.905328   467 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.894035   484 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.906534   593 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.907226   602 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:55.907047   484 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.907603 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.908397 32370 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:55.907055   619 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:55.909536   619 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:55.909730   484 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.894688   558 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.909940 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:20:55.909955   619 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:55.910720   558 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.809134   561 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.919569   528 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.929207 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:55.887012   556 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.931489 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:55.932729 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.937168 31674 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:55.939123 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:55.903031   636 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:55.924896   484 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:55.925612   568 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.951608   568 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.951831   528 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.953807   572 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:55.949214 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.843360   583 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:55.957041 31678 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:20:55.959213 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:55.954198   528 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.917567   631 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:55.941212 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.925935   578 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.936764   625 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:55.972858 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.963050 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.954334   633 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:55.954206   568 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.978534 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:55.978127   558 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:55.979475   633 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.985438 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:55.974195   556 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:55.991009   561 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:55.990813   616 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:55.991178 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.992919   633 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:55.993701 31748 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:20:55.958195   484 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:55.979471 32374 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:55.950271 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:20:56.000532 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.977785   631 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.001430 32376 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:56.016057 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.008865   578 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:55.995800   556 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:56.014356 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:56.001245 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51267.6us
I20260709 12:20:56.004967 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:56.017868 31827 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:56.005323 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.019953 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:56.020458 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.021003   597 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:56.020121   631 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:56.021859   558 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:55.996357   484 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.004848   582 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:20:56.020071 31676 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:20:56.023003   582 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:56.023869   582 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:56.023768 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:20:56.024008 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:20:56.020880 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.013970 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.025198 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:56.004745   468 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.032931 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.033787 31824 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:56.003785 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.009029 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.035239 31821 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 1.
I20260709 12:20:56.037779   636 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:56.038460   636 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:56.040418 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.041078 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.041189 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.039352   636 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.042876   568 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:56.046887   568 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:56.047469   568 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.043563   468 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:56.038398   578 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:56.049697 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:55.977224   625 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:56.053206 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.056563 31676 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:20:56.048115 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:55.995424   564 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:41192 (request call id 74) took 4212 ms. Trace:
I20260709 12:20:56.065778 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.067721 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:56.069262 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.070118 31670 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:20:56.083976 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:56.086346 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.087182 31670 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:56.087416 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:56.088241 32378 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:56.079710 31674 raft_consensus.cc:2376] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2. Re-sending same reply.
I20260709 12:20:56.088395 31573 catalog_manager.cc:5697] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 1 to 2, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:56.059404   578 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:56.091558 31572 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42072 (request call id 237) took 4154 ms. Trace:
I20260709 12:20:56.091374   578 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.072427 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.048763   572 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:20:56.078796 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:56.083842 31675 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:20:56.083976 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.099888 31748 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:56.100175 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.084467 31749 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:20:56.084772 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:56.085075 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.096195 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:56.102059 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.088315 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.035468 31819 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:56.104825 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:20:56.107511   613 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:20:56.080018   467 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.109193 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 81) took 2779 ms. Trace:
I20260709 12:20:56.109588 31672 rpcz_store.cc:276] 0709 12:20:53.329608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.329765 (+   157us) service_pool.cc:224] Handling call
0709 12:20:53.330161 (+   396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:53.330498 (+   337us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:53.330527 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:53.815932 (+485405us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:20:53.816535 (+   603us) log.cc:844] Serialized 26 byte log entry
0709 12:20:55.477793 (+1661258us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:20:55.477946 (+   153us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:55.478108 (+   162us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:56.108720 (+630612us) raft_consensus.cc:1685] finished
0709 12:20:56.108746 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:56.108990 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1947,"raft.queue_time_us":4511,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":158,"spinlock_wait_cycles":31954688,"thread_start_us":750986,"threads_started":1,"wal-append.queue_time_us":2010660}
I20260709 12:20:56.111219   467 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:56.099117   578 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:56.112161 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.056773   625 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.118834 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:56.108593   622 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.176s	user 0.000s	sys 0.004s
I20260709 12:20:56.127701 31826 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.125217 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:56.131349 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:56.131671 32372 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.051483   639 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.057400   604 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:56.136726 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:56.132858 31750 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.147243 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.148963 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:56.156653   639 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:56.157660 31749 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:56.097337   484 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.164494   484 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:56.165535 31642 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:56.139889 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.168352   618 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.170868 31824 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:56.170694 32372 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:20:56.175884   583 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:56.178269   587 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.179590 32375 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:56.187667 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:56.193077   623 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:56.193434 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:56.194093 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.194933 31824 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:56.195173 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.195744 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.196990 31677 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:56.199995 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:56.201254   623 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:56.201771   623 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:56.202185   623 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.199931 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.192412 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:20:56.238731 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 73232.6us
W20260709 12:20:56.235975 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:56.243641 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 51313.5us
I20260709 12:20:56.247591 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.247808 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:56.248649 32371 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:20:56.249115 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:56.250168 32369 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:56.242376 31678 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:20:56.268091 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.269902 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.268553 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.273548 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.278528   619 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:56.281164 31669 raft_consensus.cc:2955] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:56.283025 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.283352 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:56.284626 32342 connection.cc:256] RPC call timeout handler was delayed by -1.3453287310003361s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1583s for a response
I20260709 12:20:56.297554 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.300285 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.309718 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.310675 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.311020 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.311575 31752 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:56.312456 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.260735   643 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:56.313128 31751 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
W20260709 12:20:56.304052 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 166122us
I20260709 12:20:56.325928 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.214813   556 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.079s
I20260709 12:20:56.327709 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:56.328856 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.311053 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.332271 32369 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:20:56.334785 31753 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:56.334818   604 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:56.340906   604 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:56.341149 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:56.338369 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.314388 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:56.343067   584 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:20:56.343477   584 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:56.343847   584 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:56.345892 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.346552 31749 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:56.352839 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:56.344115   583 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:56.362496 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.341383   604 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:56.362957 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:56.364923   587 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:56.365325 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:56.362478 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60507.8us
I20260709 12:20:56.369313 31750 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:20:56.365393 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.323666   618 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:56.373335 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.374732 31678 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:56.365648   639 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260709 12:20:56.387661 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.389799 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.370450   604 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.396217 31750 raft_consensus.cc:606] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:56.398690 31750 raft_consensus.cc:614] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:56.399423 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.400205 31818 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:56.356045   643 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:56.079416   564 rpcz_store.cc:276] 0709 12:20:51.760548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.777792 (+ 17244us) service_pool.cc:224] Handling call
0709 12:20:51.797915 (+ 20123us) log.cc:844] Serialized 508 byte log entry
0709 12:20:52.436770 (+638855us) spinlock_profiling.cc:243] Waited 2.35 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:20:52.893793 (+457023us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b5400277a20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4adbe 00007fe8fcd55930 00007fe8fcd558d8 00007fe8fcd55858 00007fe8fcd55813 00007fe8fcd54a9c 00007fe902c402f6 00007fe902c3aaa8 00007fe8fcd917cb 00007fe8fcd942c0
0709 12:20:55.961269 (+3067476us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44023,"raft.queue_time_us":44427,"raft.run_cpu_time_us":21432,"raft.run_wall_time_us":468127,"spinlock_wait_cycles":10375808,"thread_start_us":109672,"threads_started":2,"wal-append.queue_time_us":54209,"wal-append.run_cpu_time_us":35009,"wal-append.run_wall_time_us":2045397}
I20260709 12:20:56.401455   643 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.381083 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.409749 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:56.388759   619 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:56.414968 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.403443 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.422446 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.422868 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.423121   467 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:56.423517   467 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:56.423619 31821 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:56.423769 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:56.416771 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.423957   467 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.430146 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.430837 31824 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:56.432453 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:56.435097 31824 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:56.435814 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:56.440311 31826 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:20:56.441236 31747 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:56.438772 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:56.414534   652 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:56.448901 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.451622 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.454909 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:56.421332   626 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 12:20:56.442462   484 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:56.457768 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:56.442376   468 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:56.466315 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.468904 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:56.470777 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:56.470764 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:56.466923   630 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:20:56.473779   630 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:20:56.092001 31572 rpcz_store.cc:276] 0709 12:20:51.937373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:51.942249 (+  4876us) service_pool.cc:224] Handling call
0709 12:20:56.091266 (+4149017us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:53.487308 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:53.489682 (+  2374us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:53.489729 (+    47us) write_op.cc:435] Acquired schema lock
0709 12:20:53.489757 (+    28us) tablet.cc:662] Decoding operations
0709 12:20:53.490256 (+   499us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:53.490333 (+    77us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:53.490387 (+    54us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:20:53.490646 (+   259us) tablet.cc:701] Row locks acquired
0709 12:20:53.490676 (+    30us) write_op.cc:260] PREPARE: finished
0709 12:20:53.490876 (+   200us) write_op.cc:270] Start()
0709 12:20:53.491029 (+   153us) write_op.cc:276] Timestamp: P: 1783599653490814 usec, L: 0
0709 12:20:53.491061 (+    32us) op_driver.cc:348] REPLICATION: starting
0709 12:20:53.575722 (+ 84661us) log.cc:844] Serialized 603 byte log entry
0709 12:20:54.471147 (+895425us) spinlock_profiling.cc:243] Waited 2.68 ms on lock 0x7b54002828d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd0e28 00007fe90080b581 00007fe90080ad58 00007fe90080f9f1 00007fe90080f9a9 00007fe90080f939 00007fe90080f901 00007fe90080eb2d 0000000000388634 0000000000388479
0709 12:20:54.906132 (+434985us) op_driver.cc:464] REPLICATION: finished
0709 12:20:55.923500 (+1017368us) write_op.cc:301] APPLY: starting
0709 12:20:55.923667 (+   167us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:55.923879 (+   212us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:55.923924 (+    45us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:55.924265 (+   341us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:55.924526 (+   261us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:20:55.924581 (+    55us) write_op.cc:312] APPLY: finished
0709 12:20:55.927135 (+  2554us) log.cc:844] Serialized 23 byte log entry
0709 12:20:55.927450 (+   315us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:55.927590 (+   140us) write_op.cc:454] Released schema lock
0709 12:20:55.927668 (+    78us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:55.928206 (+   538us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:55.928248 (+    42us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":249496,"reported_tablets":3,"child_traces":[["op",{"apply.queue_time_us":1009118,"apply.run_cpu_time_us":2953,"apply.run_wall_time_us":4748,"num_ops":1,"prepare.queue_time_us":400256,"prepare.run_cpu_time_us":9059,"prepare.run_wall_time_us":1012445,"replication_time_us":1350505,"spinlock_wait_cycles":6155648,"thread_start_us":492410,"threads_started":2}]]}
I20260709 12:20:56.474149   630 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:56.474516 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.468561 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:20:56.469123 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.463508   652 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:56.483891 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.483098 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:56.486948 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.475350 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:56.491796 31824 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:56.496879   578 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:20:56.497452   578 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:56.497843   578 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:56.475348   484 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:20:56.503522 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:56.514380 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.543386 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:56.488673   652 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:56.561410   646 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
I20260709 12:20:56.513018 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.498998   572 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:56.467121   582 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.486713   633 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:20:56.587661 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 73402.8us
W20260709 12:20:56.584901 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 85411.4us
I20260709 12:20:56.495730   556 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:56.589972 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.550487   620 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:56.524699   484 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:56.595513   572 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:56.599026 32378 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:20:56.600939 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.604791 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:56.615247 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 102426us
I20260709 12:20:56.624406 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:56.603691   572 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.600921   484 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:56.600327   556 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:56.600767   620 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:56.594784   619 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.027s
I20260709 12:20:56.634330 31826 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:56.642509 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.643533 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.643896 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.645092 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.643591 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:56.642333 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:56.646765   616 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:56.655300 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:56.633157   620 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.634430   633 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:20:56.667462 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 234323us
W20260709 12:20:56.670995 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 298064us
I20260709 12:20:56.678507 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.679216 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:56.684312 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.686082   635 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.670595   602 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:20:56.685308 31745 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
W20260709 12:20:56.694312 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 70140.9us
W20260709 12:20:56.722465 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:56.745496 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.696527 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:56.675778   633 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:56.659694   623 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.744359 31826 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
W20260709 12:20:56.752477 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56024.8us
I20260709 12:20:56.695832   602 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:20:56.746903   649 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.196s	user 0.000s	sys 0.002s
I20260709 12:20:56.702893   582 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:56.773752   578 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:56.778108 31747 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:56.752401   468 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:56.755280   633 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:56.742708   467 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:56.799002 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.775877   602 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:56.803402 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:20:56.802313 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 116722us
I20260709 12:20:56.805212   652 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.808431 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:56.809296   582 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.694402   556 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:56.777745 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:56.809567   652 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:56.815922 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:56.836743 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:20:57.114722 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:56.877866 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:20:57.125735 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 314358us
I20260709 12:20:56.837709 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:56.806209 31670 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:56.859292 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.173507 31753 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:57.177368 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:20:57.180357 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 342766us
I20260709 12:20:56.934959 31562 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet fc47871ef692418388113c1d6c88b262 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:20:57.079429   467 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:56.811923   468 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:20:56.959532 31825 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.191094 32371 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.191172 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:57.183580   658 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.373s	user 0.018s	sys 0.018s
I20260709 12:20:57.183578   650 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:20:56.933629   584 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.589s	user 0.009s	sys 0.002s
W20260709 12:20:57.081827   625 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.949s	user 0.011s	sys 0.000s
W20260709 12:20:56.919884   604 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.503s	user 0.003s	sys 0.006s
W20260709 12:20:56.967435 31669 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.587s	user 0.011s	sys 0.000s
W20260709 12:20:57.070137   636 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.024s	user 0.014s	sys 0.000s
W20260709 12:20:56.932746   665 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:20:57.251681 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:57.253325 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:20:57.210958 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.214645 31750 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:20:57.254189   673 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:56.918108   616 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:57.254765   616 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:56.942720   556 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:56.998077 32374 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:57.129033   635 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.194152 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:56.935066 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:56.934096   623 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:56.854351 31745 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
W20260709 12:20:57.240576 31562 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 305719us
W20260709 12:20:57.221518 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44246.6us
I20260709 12:20:56.936796 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:56.810470   631 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.409s
I20260709 12:20:57.245611 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:57.246771 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:20:57.215953   630 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.741s	user 0.017s	sys 0.000s
I20260709 12:20:57.240589 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:57.139137   633 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.115546   620 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.213686   468 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:56.997493   484 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:56.889282   568 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.841s	user 0.006s	sys 0.008s
I20260709 12:20:57.141392   602 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.255636   616 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.220446 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:57.214520   664 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:57.217572   661 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:57.069247   572 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.267169 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.243220   491 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260709 12:20:57.256453   673 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:57.262660   582 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.260730   650 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.331094   681 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:20:57.243479   625 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.242179   584 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.268636 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:57.249528   604 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.258553   636 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.344362 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1373) took 1027 ms. Trace:
W20260709 12:20:57.344870 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 99339.8us
I20260709 12:20:57.351323 31749 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
W20260709 12:20:57.351363 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 84298.1us
W20260709 12:20:57.353091 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.333055   578 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.338083   468 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:57.343918 32369 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.353048   616 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.344161   661 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:57.399957 31827 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.335081   630 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:57.360239 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 176803us
I20260709 12:20:57.345863   673 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.364584 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:57.421319 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57189us
I20260709 12:20:57.354395   635 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.340713   568 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.426358 31676 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.363855 31673 rpcz_store.cc:276] 0709 12:20:56.336998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.352330 (+ 15332us) service_pool.cc:224] Handling call
0709 12:20:56.392304 (+ 39974us) spinlock_profiling.cc:243] Waited 5.94 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:20:56.450581 (+ 58277us) spinlock_profiling.cc:243] Waited 11.9 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0e9c 00007fe8f9cb0a5b 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:20:56.453921 (+  3340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:20:56.454147 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:20:56.454184 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:20:56.455026 (+   842us) log.cc:844] Serialized 26 byte log entry
0709 12:20:56.593874 (+138848us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:20:56.595312 (+  1438us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:56.596676 (+  1364us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:20:57.341058 (+744382us) raft_consensus.cc:1685] finished
0709 12:20:57.341081 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:57.341230 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":40980608,"thread_start_us":138425,"threads_started":1,"wal-append.queue_time_us":295400}
W20260709 12:20:57.436066 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 71560.7us
I20260709 12:20:57.437126 32342 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.364418   631 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:57.398881   664 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:57.457942 32376 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:20:57.457752   668 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:57.451294 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:57.458926   630 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.460930 31748 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.462118   568 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:57.463652 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 945) took 1016 ms. Trace:
I20260709 12:20:57.464025   587 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:20:57.464156 31670 rpcz_store.cc:276] 0709 12:20:56.447519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.458873 (+ 11354us) service_pool.cc:224] Handling call
0709 12:20:56.459207 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:57.182683 (+723476us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:57.463349 (+280666us) spinlock_profiling.cc:243] Waited 326 ms on lock 0x7b54001d5e54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:20:57.463590 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":749687680}
I20260709 12:20:57.454388   631 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.468366   631 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.469674   484 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.427848   661 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.478008 31718 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.480716 31671 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 2: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:20:57.488837 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.495165 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.495926 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.496563   604 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.496973 31752 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.497087   639 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.482404 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 928) took 1202 ms. Trace:
I20260709 12:20:57.500000 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.495355   602 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:57.504222 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:20:57.502367 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 65368.4us
I20260709 12:20:57.502871 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.505306 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.506248 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.507629 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.507522 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.507690 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:57.498740   663 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:20:57.495191 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.474181   664 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.518921   688 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.521459 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:57.500597   678 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:57.509864 31751 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:45178
I20260709 12:20:57.544305 31751 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:57.544751 31751 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:57.530305 31746 raft_consensus.cc:2368] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4.
W20260709 12:20:57.531512 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53621.6us
I20260709 12:20:57.530459   672 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:57.533207   657 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:57.481822   668 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:57.540840   683 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.131s	user 0.001s	sys 0.000s
I20260709 12:20:57.521752   667 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:57.509598   654 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:57.530589 31744 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260709 12:20:57.547004   620 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.554895   657 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.549139 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:57.587309 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 150705us
I20260709 12:20:57.556744   688 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:57.559124 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:57.562974 32369 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:20:57.597714 32370 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.563584 31751 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.564617   636 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.565398   625 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.567080 32373 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.567229   664 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:20:57.567919   675 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.229s	user 0.000s	sys 0.001s
I20260709 12:20:57.615082 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:57.616452   684 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:20:57.559902 32376 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 263) took 1433 ms (client timeout 1748 ms). Trace:
I20260709 12:20:57.582615   633 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.500802 31669 rpcz_store.cc:276] 0709 12:20:56.268922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.280364 (+ 11442us) service_pool.cc:224] Handling call
0709 12:20:56.280718 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:56.281024 (+   306us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:20:57.252338 (+971314us) log.cc:844] Serialized 14 byte log entry
0709 12:20:57.411215 (+158877us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:20:57.427885 (+ 16670us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:20:57.430035 (+  2150us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:57.443304 (+ 13269us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:20:57.471297 (+ 27993us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7225,"raft.queue_time_us":49967,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":19137,"thread_start_us":90910,"threads_started":1,"wal-append.queue_time_us":203147}
I20260709 12:20:57.589497   572 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:57.590341   688 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.569860   667 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:57.592581 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:57.597359   682 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:20:57.602524 32369 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.602403   670 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:57.604664 31751 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.605679 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1372) took 1322 ms. Trace:
I20260709 12:20:57.579126   584 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.569810   678 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.607782   654 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.611896   676 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:57.613113 31753 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 3. Ops: []
I20260709 12:20:57.609229   587 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:20:57.583904 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 258669us
I20260709 12:20:57.624323 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.567747   672 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.617020   684 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:57.622948 32376 rpcz_store.cc:269] 0709 12:20:56.126601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.127598 (+   997us) service_pool.cc:224] Handling call
0709 12:20:57.559663 (+1432065us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b54003f3254. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:20:57.559839 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2905514880}
I20260709 12:20:57.636639 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.636862 31821 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:57.657164   643 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:57.638873 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.657626   643 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:57.657686   654 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:20:57.642361 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49911.2us
I20260709 12:20:57.658826 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.658670   643 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.639067 31676 rpcz_store.cc:276] 0709 12:20:56.282938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.283928 (+   990us) service_pool.cc:224] Handling call
0709 12:20:56.303466 (+ 19538us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:57.429773 (+1126307us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:57.605450 (+175677us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54002e7f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:20:57.605614 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2543390080}
I20260709 12:20:57.638492 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.659729 32378 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.660036 31818 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:57.660143   643 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.660518 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 269) took 1467 ms. Trace:
I20260709 12:20:57.647632 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:57.635020   667 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:57.643368 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.650862 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.665275 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.642465   664 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:57.660902 32374 rpcz_store.cc:276] 0709 12:20:56.192835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.199253 (+  6418us) service_pool.cc:224] Handling call
0709 12:20:56.997723 (+798470us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:57.259711 (+261988us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:57.660318 (+400607us) spinlock_profiling.cc:243] Waited 786 ms on lock 0x7b54006473d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:20:57.660454 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1807293696}
I20260709 12:20:57.675606 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:57.677199   684 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.671882   664 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:57.678284   664 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.678522   484 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:57.679093   484 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:57.677412 31744 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.679597   484 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.686131   688 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.686985 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.687857 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.688602 31753 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:57.666810   620 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.672907   653 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:20:57.697520 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46864.4us
I20260709 12:20:57.704030   694 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:57.704859   678 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:57.704761   694 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:57.698442   693 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.707444   694 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:57.711795 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1031) took 1079 ms. Trace:
I20260709 12:20:57.712302 31749 rpcz_store.cc:276] 0709 12:20:56.632367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.632542 (+   175us) service_pool.cc:224] Handling call
0709 12:20:56.649172 (+ 16630us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:57.398824 (+749652us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:57.711523 (+312699us) spinlock_profiling.cc:243] Waited 697 ms on lock 0x7b5400064354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:20:57.711725 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1602040192}
I20260709 12:20:57.713874 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:57.716118 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:57.717061 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 91921.5us
I20260709 12:20:57.719398 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.720471 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:57.714932   653 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:57.721470   633 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:57.721931   633 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:57.722368   633 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.722492   653 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.723033 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.726959 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.727632 31823 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.727849   620 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:57.728837   659 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:57.729368   659 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.730134 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:57.730873   468 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.731793   659 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.732508 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1014) took 1528 ms. Trace:
I20260709 12:20:57.732870 31748 rpcz_store.cc:276] 0709 12:20:56.204087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.204225 (+   138us) service_pool.cc:224] Handling call
0709 12:20:56.212285 (+  8060us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:57.461344 (+1249059us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:57.732271 (+270927us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54002f5dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:20:57.732452 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2871195520}
I20260709 12:20:57.733861   468 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.735198 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.737000 31818 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:20:57.739102   672 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:57.743829 31822 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:57.740034   650 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.749318   693 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:57.750590   693 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:57.750939   693 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:57.753286 31823 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:20:57.743423 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.756814 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:57.762614 31792 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.762717 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:57.754987   620 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:57.771894 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.755697   654 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:57.773754 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.777159 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:57.779320 31823 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:57.762598   467 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:20:57.769769   556 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.514s	user 0.006s	sys 0.006s
I20260709 12:20:57.779978 31674 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:20:57.804105 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:57.795125 31717 leader_election.cc:336] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.727s (SENT)
I20260709 12:20:57.807189 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.808890 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:57.775518   620 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:57.778944   678 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:20:57.812094 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.762245   668 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:57.780661   602 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.810402 31674 raft_consensus.cc:606] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Received request to transfer leadership to TS 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:57.751794   693 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.819573 31672 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:57.777586   689 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.831696 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:57.818215   620 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.808779   467 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:57.815026 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:57.845954 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:57.809326   639 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260709 12:20:57.847095 32374 raft_consensus.cc:2955] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:57.848408   689 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:57.848796   689 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:57.814971   695 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:57.828924   676 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:20:57.818899   468 raft_consensus.cc:2764] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260709 12:20:57.810426 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:57.831686   578 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.805219   654 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.815315   679 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:20:57.840088 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 318389us
I20260709 12:20:57.818501   678 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:57.842252   625 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:57.842901 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:57.843628 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:57.843832 32370 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
W20260709 12:20:57.845314 31671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1377) took 1506 ms (client timeout 1727 ms). Trace:
I20260709 12:20:57.864300   602 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.821830 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:20:57.869837 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46407.5us
I20260709 12:20:57.822960   616 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.850991   467 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.821729   484 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.891748 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.876735   695 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:20:57.888810 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67105.9us
I20260709 12:20:57.889293 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.889670 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.876695 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:57.892776 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:57.819260   673 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:57.878595 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:57.894259 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:20:57.891333 31671 rpcz_store.cc:269] 0709 12:20:56.338708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.356729 (+ 18021us) service_pool.cc:224] Handling call
0709 12:20:57.845073 (+1488344us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b540017c0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:20:57.845217 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2312653056}
I20260709 12:20:57.901374   673 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:57.889938   678 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:57.895238   695 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.895018 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.894647 32372 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:20:57.894809   578 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.913771 32369 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.895442   639 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:20:57.849640   689 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:57.897506 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:57.915283 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:57.898255 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:57.878482   623 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.915953   670 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:57.891897   468 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.916469 31748 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:57.916513   670 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.899168 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:57.906927 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.907027 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.907277 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.908478 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.908943 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.909586 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:57.878319   668 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:20:57.890897   679 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:57.909989 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.918396 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.920326 32378 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:57.926347 31820 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:57.919292   672 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:20:57.926781 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:57.903728   572 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:57.924925   679 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:57.906981   677 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.878937   661 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.922076 31818 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:57.915288   689 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:57.904121   653 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:57.937511 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.956643 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.921113 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:57.963483 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.965077   618 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.964983   633 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.966396   664 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:57.968008 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.969084 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:57.930611   572 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:57.970470 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.977835 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.980911 31823 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 2: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:57.920635   616 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 12:20:57.993687 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:57.995318 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:57.999202 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:57.958132   564 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.958264   623 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:58.001865   687 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.998659   672 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:58.006208 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.977773   561 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:20:58.037065 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51572.5us
I20260709 12:20:58.012820 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:57.977994   618 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.041230 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:58.044044 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 73710.8us
I20260709 12:20:58.045078 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.998785   584 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:20:58.053682 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 96571.7us
I20260709 12:20:58.050402 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.055579 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:57.998488   556 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:20:58.058719 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 138170us
I20260709 12:20:58.005509   619 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:57.964641   653 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.063506   639 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:58.038415 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:57.997143   677 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260709 12:20:58.053681   616 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.039224   564 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:20:58.059109   618 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.051358   578 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.057272   657 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:58.087396   661 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:58.088047   484 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:58.045820   623 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.073498   604 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.074366 31818 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:58.081228   686 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.075960   687 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:58.076555   561 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:58.082787 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:58.088078   691 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260709 12:20:58.060629   584 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:58.110311   687 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:58.072427   556 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.077142   667 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.121415 31826 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.099617   654 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:58.125080 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:58.109681 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:58.127000   650 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.127496   650 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:58.127647   653 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:58.127758 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:58.115417   568 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260709 12:20:58.128367   650 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.128804   672 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.129652   650 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.129163 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:58.135452   568 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.107590 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:58.110889   561 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:58.138260 31820 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:20:58.139801   568 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:20:58.162662 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55191.9us
I20260709 12:20:58.174329 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:20:58.175436 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 119954us
I20260709 12:20:58.175789   639 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:58.176256 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.177059   713 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:58.132529   687 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.192968   713 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:58.193506   713 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.139676   691 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:58.135442   633 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:58.177145   639 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.177464 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.138664   587 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:58.116262   620 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.139124   673 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:58.143954   561 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.192019 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 383) took 1938 ms. Trace:
I20260709 12:20:58.208437   708 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.211031 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.201381   639 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:58.212327 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.192428   659 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.178781   631 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.196604   556 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:58.201118 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 69810.7us
I20260709 12:20:58.215049   587 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:20:58.208123 32373 rpcz_store.cc:276] 0709 12:20:56.253287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.260215 (+  6928us) service_pool.cc:224] Handling call
0709 12:20:57.566877 (+1306662us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:57.611419 (+ 44542us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:20:58.139271 (+527852us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54003f3250. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:20:58.178203 (+ 38932us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2980833152}
I20260709 12:20:58.225222   616 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:58.205412   677 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260709 12:20:58.228199   584 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:58.230794   667 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:58.197284   691 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:20:58.198611   664 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:58.228982   467 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:58.277973 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 265226us
I20260709 12:20:58.237790   708 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:58.212566   639 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.290848 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:58.295082 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:58.244692   616 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:20:58.297261   643 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.634s	user 0.010s	sys 0.005s
I20260709 12:20:58.305101 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:20:58.304752 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 222900us
I20260709 12:20:58.293819   708 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:58.316974 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:20:58.352233 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 152447us
I20260709 12:20:58.317912 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:58.318424   693 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.302301   673 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:20:58.304134   667 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.321108 32370 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:58.324949   572 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:58.331127 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:58.330487   653 raft_consensus.cc:2955] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:20:58.335366   704 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:58.337543   706 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:58.374775 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:58.338569 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:58.375862 31677 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 1.
I20260709 12:20:58.340258 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.340873 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:58.341998   468 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:20:58.342514 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.343066   491 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.298105   668 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.241219 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:58.312315 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:58.303468   616 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.359031 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:20:58.382217 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:58.319010   467 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:58.385427   707 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:20:58.385526   712 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:58.386052   667 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.386194   707 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:58.386333   712 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:58.377943 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.297114   670 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:20:58.381553 31674 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:20:58.358718   708 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:58.383687 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52647.4us
I20260709 12:20:58.360384   639 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.334384   657 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260709 12:20:58.371475   623 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:20:58.402419 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 190154us
I20260709 12:20:58.373770   706 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:58.406937 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:58.408218 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.410012 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:58.382180   468 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:58.413693 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.414471 31673 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:20:58.382339   491 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:58.395121   582 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.417567 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:58.418967 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:58.425209   667 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.426025   667 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:58.387238   707 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.432883 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:58.399016 31671 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:58.439296 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:58.439594 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:58.438060 31677 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:20:58.442806 31677 raft_consensus.cc:606] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Received request to transfer leadership to TS 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:58.443266 31677 raft_consensus.cc:614] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:20:58.445152 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:58.445859 31751 raft_consensus.cc:2368] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4.
I20260709 12:20:58.448801 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:58.409341 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 168244us
I20260709 12:20:58.451135 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:58.371796   587 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:58.454036 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:58.417567   672 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.405211   717 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260709 12:20:58.464473 31752 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:20:58.450776   564 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 12:20:58.458262 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:58.458938 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:58.445605   620 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:58.460072 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:58.462507 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:58.462872 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:58.453301 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:58.439019   694 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.458021   693 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:20:58.444644   584 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.434588   625 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.472863 31750 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:58.473678 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:20:58.469875   679 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.508s	user 0.009s	sys 0.002s
I20260709 12:20:58.378553   704 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.465329 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:58.480929 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:58.480928 31642 leader_election.cc:400] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4.
I20260709 12:20:58.382908   691 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:58.483255 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:20:58.504559 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62649.9us
I20260709 12:20:58.506944 31672 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.509276 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.511276 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:58.512269 31670 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:20:58.512799 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.513530 31670 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.490051 31747 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:58.514609 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:58.515239 31747 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:58.515748 31676 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.516057 31747 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:20:58.516568 31747 raft_consensus.cc:606] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:58.516803 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.517824 31747 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.495574 31642 leader_election.cc:403] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:20:58.496466 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:20:58.519256 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57290.2us
I20260709 12:20:58.519817 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:58.520053 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:58.500515   618 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.524683   584 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:20:58.525686 31824 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.480778 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:58.465792   468 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:58.527228 31669 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:58.527748   468 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.475080 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:58.477106   695 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.555s	user 0.010s	sys 0.000s
I20260709 12:20:58.521260 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:58.489573 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:58.460520   673 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:58.536721 31747 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.543344 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:58.545004 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:58.407618   706 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:58.528048   723 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20260709 12:20:58.529162   625 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.486547   679 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.552709 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:58.556913 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:58.562953 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.541615   694 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.567382   712 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:58.568218 31746 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:20:58.549044   668 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.563234   667 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.583298 31746 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:58.584082 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:58.565642 31745 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:58.602718 31750 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: [1.3-1.3]
I20260709 12:20:58.603737   712 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:58.604559 31747 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:58.604334   712 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.601259   708 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:58.551446   715 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.559679   670 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:58.609856   656 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:58.553014   664 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 12:20:58.538262   695 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.627812 31822 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.587247   713 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.629017 31820 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.556098   718 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.222s
I20260709 12:20:58.639941 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:20:58.645839   650 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.516s	user 0.014s	sys 0.000s
I20260709 12:20:58.692569 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.693475 31677 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 2: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:20:58.628518   582 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:20:58.697371 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 91381.4us
I20260709 12:20:58.633772   564 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:20:58.726673 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 88711.3us
I20260709 12:20:58.731349 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:58.704506   564 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:20:58.725484   678 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.823s	user 0.009s	sys 0.004s
W20260709 12:20:58.732167   720 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:20:58.729792 31825 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.750900 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.733610   582 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:58.755087 31678 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:58.755367   582 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:58.755748   713 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:58.757670   582 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:58.740425 31818 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:20:58.751917   716 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:20:58.775579 31818 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:58.786124   623 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.785974 31818 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:58.786793   704 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:58.776239 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:58.751009   733 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:58.787638   733 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:58.789569 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 88809.3us
I20260709 12:20:58.789901   733 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:58.790647   695 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:58.791512   679 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:58.793062   708 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:58.777525   618 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.795202   564 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:58.795740   618 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:58.796851   618 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.786573   623 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:58.798295   604 raft_consensus.cc:993] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: : Instructing follower 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:58.798934   604 raft_consensus.cc:1081] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Signalling peer 5206fd0087404d2da964a5d985a04ff4 to start an election
I20260709 12:20:58.798753   623 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:58.799963   623 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.784693   712 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:58.797479   734 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:20:58.804288 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 52870.5us
I20260709 12:20:58.790205 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:58.742139   678 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:58.822607 31678 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:20:58.823743 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:58.819814 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:58.825891 32373 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
W20260709 12:20:58.814277   561 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.591s	user 0.007s	sys 0.003s
I20260709 12:20:58.832121 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.834622 31792 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:20:58.841543   734 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.842025   734 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:58.842734 31821 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:58.839876   678 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:58.852842 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:58.854287   734 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:20:58.855531   687 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.662s	user 0.010s	sys 0.002s
I20260709 12:20:58.856204   734 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.856709   736 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:58.857213   736 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:58.857620   736 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.855556 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:20:58.859473   584 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4.
I20260709 12:20:58.857656 32373 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
W20260709 12:20:58.862383 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.862931 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.855983   687 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.844664   561 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.868499 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:20:58.869387 31824 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.870326   687 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:58.874153 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:20:58.875624 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.877317 31826 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 2: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:58.880434 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.883989 31573 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 146) took 5846 ms. Trace:
I20260709 12:20:58.886723   704 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:58.888048   704 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.888482   704 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:58.889564   704 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:58.895372 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:58.879341   616 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.907408 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:58.907869 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:58.908282   616 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:58.909700 31827 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.904704 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.913309 31824 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:58.843538   568 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.920022 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.920490 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.884439 31573 rpcz_store.cc:276] 0709 12:20:53.030267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:53.030440 (+   173us) service_pool.cc:224] Handling call
0709 12:20:58.876506 (+5846066us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:56.363372 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:56.457125 (+ 93753us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:56.457174 (+    49us) write_op.cc:435] Acquired schema lock
0709 12:20:56.457214 (+    40us) tablet.cc:662] Decoding operations
0709 12:20:56.457645 (+   431us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:56.457732 (+    87us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:56.457765 (+    33us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:20:56.458055 (+   290us) tablet.cc:701] Row locks acquired
0709 12:20:56.458102 (+    47us) write_op.cc:260] PREPARE: finished
0709 12:20:56.458302 (+   200us) write_op.cc:270] Start()
0709 12:20:56.458448 (+   146us) write_op.cc:276] Timestamp: P: 1783599656458241 usec, L: 0
0709 12:20:56.458471 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:20:56.459210 (+   739us) log.cc:844] Serialized 539 byte log entry
0709 12:20:56.712162 (+252952us) op_driver.cc:464] REPLICATION: finished
0709 12:20:57.175101 (+462939us) write_op.cc:301] APPLY: starting
0709 12:20:57.175189 (+    88us) tablet.cc:1367] starting BulkCheckPresence
0709 12:20:57.175288 (+    99us) tablet.cc:1370] finished BulkCheckPresence
0709 12:20:57.175408 (+   120us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:20:57.616466 (+441058us) spinlock_profiling.cc:243] Waited 3.62 ms on lock 0x7b5c0021fd0c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902fc7331 00007fe902fc7221 00007fe902fc1f6c 00007fe90072230b 00007fe900721fd3 00007fe900820e34 00007fe90080c97c 00007fe900810d61 00007fe900810d19 00007fe900810ca9 00007fe900810c71
0709 12:20:57.616745 (+   279us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:20:57.616915 (+   170us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:20:57.616961 (+    46us) write_op.cc:312] APPLY: finished
0709 12:20:57.617169 (+   208us) log.cc:844] Serialized 23 byte log entry
0709 12:20:57.698608 (+ 81439us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:57.702369 (+  3761us) write_op.cc:454] Released schema lock
0709 12:20:57.716582 (+ 14213us) write_op.cc:341] FINISH: Updating metrics
0709 12:20:57.767874 (+ 51292us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:20:57.770292 (+  2418us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":74484,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":424039,"apply.run_cpu_time_us":23728,"apply.run_wall_time_us":580065,"num_ops":1,"prepare.queue_time_us":188396,"prepare.run_cpu_time_us":3992,"prepare.run_wall_time_us":158488,"replication_time_us":208473,"spinlock_wait_cycles":8320256,"thread_start_us":293580,"threads_started":3,"wal-append.queue_time_us":150773,"wal-append.run_cpu_time_us":1965,"wal-append.run_wall_time_us":1062570}]]}
I20260709 12:20:58.869405   561 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:58.926438 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.927078 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.927438   616 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.927807 32370 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.928333 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.929219   568 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.927013 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:58.930608   468 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:58.933564 32374 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b"
dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:33962
I20260709 12:20:58.940335 31753 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:58.945475 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:58.946583 31678 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: []
I20260709 12:20:58.947327 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:58.947885 31751 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:58.949313 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.950289 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.935931   584 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.015s
I20260709 12:20:58.936510   678 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.014s
I20260709 12:20:58.935700   706 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:58.952174   678 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:58.952535   678 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:58.953354 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:58.938135 32374 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:20:58.953418   678 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.954228 32374 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:58.954605 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.954843   678 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:58.950002 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.948689 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.934216   639 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.958876   468 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:20:58.959410 31744 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:20:58.959573 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:58.950048   704 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.960507   468 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:58.961648   639 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.962003   639 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:20:58.962994   564 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.962889   639 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:58.964725   639 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.965444   618 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:58.917508   713 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 12:20:58.968232   713 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:58.968557   713 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:58.973875 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.980623 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.981810 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:58.982039 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:58.986023 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.973140 32377 raft_consensus.cc:2955] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:20:58.990712 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 136971us
I20260709 12:20:58.992627 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.994844 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:58.998652 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:58.999118 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.995151 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.969880   713 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.002130 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:59.006870 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:59.010505 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:59.002755   670 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:20:58.999392   664 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.024439   745 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.022647 31753 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:58.990583   679 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:20:59.026656 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.997144   616 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:59.003808   677 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.030400 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.996843   730 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260709 12:20:59.012219   602 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.053064 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:58.997303   668 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 12:20:59.001672   687 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260709 12:20:59.022326   587 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.029234   679 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:20:59.025766   732 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:20:59.053570   713 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:59.011650   723 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 12:20:59.027251   616 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:59.054384   693 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.081383   616 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:59.081615 32374 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.082163 32378 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:59.025746 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:20:59.093564 31752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1418) took 1204 ms (client timeout 1512 ms). Trace:
W20260709 12:20:59.094224 31752 rpcz_store.cc:269] 0709 12:20:57.889241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:57.889423 (+   182us) service_pool.cc:224] Handling call
0709 12:20:59.093267 (+1203844us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b5400212fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:20:59.093429 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1313727360}
I20260709 12:20:59.009687   728 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.054s
W20260709 12:20:59.021970   733 consensus_queue.cc:1175] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "039d66a6c85e43c3936bf5017501b344" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260709 12:20:59.058475   707 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:59.097566   616 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.097946   707 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.100107   707 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:59.069444   727 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.073457 31746 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:59.104864   686 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:20:59.107034 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:59.108529 31749 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:59.084040   705 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:20:59.117707 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 104939us
I20260709 12:20:59.118752 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:59.073436   695 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.046s
I20260709 12:20:59.120806 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:59.122248 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:59.123538 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:59.117933   639 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:20:59.124358   673 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.122356   666 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:59.141676 31751 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:59.144071 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:59.124545   708 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:59.151120 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:59.152019 31752 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:20:59.154742 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:59.147014 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:20:59.119812   736 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.131305   686 consensus_queue.cc:1243] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:20:59.142032   726 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:59.151535   723 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.155114   623 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.116751   690 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.014s
I20260709 12:20:59.184002 31824 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:59.180526 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:59.168391 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:59.149123   746 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.150856   708 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:59.182006 31676 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:59.150925   639 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.175575   726 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:59.179687   584 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.189227 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:59.175817 31675 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:20:59.172753   661 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.018s
I20260709 12:20:59.177030   668 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.014s
I20260709 12:20:59.197582 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:59.190760   694 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:20:59.244325 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 116368us
I20260709 12:20:59.192154   666 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:59.246119   666 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:59.195991   726 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.248888   668 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:59.249235   639 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:59.249490   668 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:59.250347   668 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.251487   708 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.252554 31751 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:20:59.253064   708 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:20:59.253513 32342 leader_election.cc:336] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.570s (SENT)
I20260709 12:20:59.253937 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:59.248999 32374 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:59.259306 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:20:59.259956   673 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.260663   604 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:59.260546   673 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.260862 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:20:59.262727   718 consensus_queue.cc:1175] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5206fd0087404d2da964a5d985a04ff4" responder_term: 1 status { last_received { term: 1 index: 3 } last_committed_idx: 3 last_received_current_leader { term: 1 index: 3 } }
I20260709 12:20:59.262918   639 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:20:59.261114   604 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:59.271246   712 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:59.271864   604 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.272099   712 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.273903 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1388) took 1152 ms. Trace:
I20260709 12:20:59.274354 31826 rpcz_store.cc:276] 0709 12:20:58.121364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:58.124820 (+  3456us) service_pool.cc:224] Handling call
0709 12:20:59.273626 (+1148806us) spinlock_profiling.cc:243] Waited 751 ms on lock 0x7b54001671d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:20:59.273802 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1728068992}
I20260709 12:20:59.196417   556 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.275586 31823 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:59.263552   639 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:20:59.265300 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 67801.3us
I20260709 12:20:59.251701   668 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:20:59.269558 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.284756   739 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:59.285375   739 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.285823   572 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:59.247102   726 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:59.277163   704 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:59.278985   639 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:59.297389 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:20:59.312615 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:59.302887 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:59.319092 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.288235   750 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.294473 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:20:59.300707 31571 catalog_manager.cc:5697] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 1 to 2, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 2 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:20:59.333132 31571 catalog_manager.cc:5697] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:20:59.294373   556 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.037s
W20260709 12:20:59.355758 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:59.360136   744 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.125s	user 0.000s	sys 0.001s
I20260709 12:20:59.333012   747 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.056s
I20260709 12:20:59.333271 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 96) took 1032 ms. Trace:
I20260709 12:20:59.337340 31675 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:59.384323 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:59.426702 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:59.371343   670 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:59.412245   736 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:20:59.460870 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 88577.7us
I20260709 12:20:59.423291   564 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.438374   620 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.439035   739 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:59.423705 31675 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:59.438174   664 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.475353   754 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.471649 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:20:59.495398 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 198148us
I20260709 12:20:59.487083 31675 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.462191   491 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.499923 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:59.413273 31671 rpcz_store.cc:276] 0709 12:20:58.300678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:58.302239 (+  1561us) service_pool.cc:224] Handling call
0709 12:20:59.333020 (+1030781us) spinlock_profiling.cc:243] Waited 4.2 ms on lock 0x7b54002e8454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:20:59.333164 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9668864}
I20260709 12:20:59.482529   712 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:59.503978 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:59.516144   556 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:20:59.484790   673 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:59.523310   556 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:59.532510   556 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:59.533022   604 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:59.493220   759 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.052s
I20260709 12:20:59.501833 31675 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:59.485565   670 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.525558 31642 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:59.540369 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:20:59.541472   670 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:59.541591 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:59.542500   754 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:59.526422   672 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:20:59.502259   753 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:59.482805   564 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.547942 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:20:59.550843 31753 raft_consensus.cc:1813] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Handling vote request from an unknown peer b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:59.551679 31753 raft_consensus.cc:2410] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:20:59.561817 31752 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:59.570665 31748 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:20:59.573746   625 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:59.574012   672 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:59.574486   672 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:59.575403   639 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.577481 31747 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
W20260709 12:20:59.578567 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 242603us
I20260709 12:20:59.582541 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:59.585789 31822 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 2: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:59.587213 31826 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:59.569784   568 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.577754   639 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:59.578115   572 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:20:59.594730 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 132590us
I20260709 12:20:59.587852   759 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.599653   704 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.643815 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:20:59.646538   759 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.649159   759 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:20:59.587779   712 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 12:20:59.575443   672 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:59.699795 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.638661 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:59.665762   723 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.706275 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:20:59.706435 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:59.707129 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 111632us
W20260709 12:20:59.710870 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:59.711345 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.596300   670 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.712898 31753 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:59.701753 32369 raft_consensus.cc:1813] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Handling vote request from an unknown peer b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:20:59.675451   736 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:59.728051 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:59.663702   713 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:59.726397 31643 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:20:59.745093   713 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.725145   670 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.746130   730 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:20:59.747623   713 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:59.747746   670 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:59.749245 32369 raft_consensus.cc:2410] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:20:59.728611   770 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:59.732060 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:59.732347 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:59.712466   747 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:20:59.733670   764 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:20:59.735044 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:20:59.735954 31820 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:59.593379   568 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.665258   729 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:20:59.754295 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.746557   730 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:59.847647 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:59.854022   564 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:59.855146   766 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:59.709254   708 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:20:59.599642   572 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:20:59.856376 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:20:59.701869   723 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.599170   758 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.130s
I20260709 12:20:59.862002 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:20:59.862622 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 111204us
I20260709 12:20:59.863978 31821 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 2: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:59.882385 31827 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:59.863915 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:20:59.754873   736 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.889772 32369 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:20:59.861191   670 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:59.890950   670 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.891350   758 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:59.891921   758 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:20:59.709743   666 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.720288   556 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:20:59.868016 31826 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 2: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:59.871040 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:59.877763 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:20:59.867246 31825 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 2: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:20:59.896245   736 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:59.874620   730 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.896667   691 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:20:59.866953   772 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.880357 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:59.871913   602 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:20:59.900293   681 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:20:59.900615   602 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:59.900930   681 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:20:59.868227   661 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260709 12:20:59.901597   602 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.884102 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:59.887307   568 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:59.872416   639 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:59.889081 31827 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:20:59.892974 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:20:59.917084 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:20:59.917786 31827 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
W20260709 12:20:59.897754 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:20:59.866712   668 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.580s	user 0.013s	sys 0.006s
I20260709 12:20:59.871995   729 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.968004 31675 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:20:59.968187   572 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:59.899240   556 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:20:59.969873   670 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:20:59.879197   771 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:20:59.906175 32377 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:20:59.881132   766 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.977233   767 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:59.902194   672 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:59.919251 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:20:59.907047   772 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.937577   625 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.969668   758 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.863461   770 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:20:59.893810   708 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.970690   739 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:59.978552   602 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:20:59.966130   639 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:20:59.971760   736 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:00.014564 31827 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:20:59.972609   747 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:00.021692 31820 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:20:59.977288 32369 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:00.012411 31821 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 2: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:00.001763   556 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.992480   754 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:00.015189   764 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:20:59.968045   729 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:00.019034 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.028430   686 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.025027 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1057) took 1044 ms. Trace:
I20260709 12:21:00.026343   736 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.034691   758 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.028286   747 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.035035   760 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.011744   625 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.035641   572 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.036796   753 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:20:59.901909   681 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.047706   713 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:20:59.987690   771 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.064091   736 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.051503   764 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.028970   739 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.053860 31819 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 1.
I20260709 12:21:00.036520   556 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.054687   624 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
W20260709 12:21:00.104562 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 208103us
I20260709 12:21:00.134778 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1490) took 1742 ms. Trace:
I20260709 12:21:00.064919   686 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.135298 31823 rpcz_store.cc:276] 0709 12:20:58.389291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:58.389412 (+   121us) service_pool.cc:224] Handling call
0709 12:20:59.275379 (+885967us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:59.276055 (+   676us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:00.131856 (+855801us) spinlock_profiling.cc:243] Waited 886 ms on lock 0x7b54001671d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:00.131999 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2036798336}
I20260709 12:21:00.140807 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:00.143666 31823 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:00.145578   695 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.149132 31753 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:00.076774 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:00.152647   695 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.154560   764 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.084349   572 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.155220 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.156104 32374 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.157819 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:00.158059   686 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.159930 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.160358   729 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.123409   568 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:00.054910 31746 rpcz_store.cc:276] 0709 12:20:58.980920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:58.994015 (+ 13095us) service_pool.cc:224] Handling call
0709 12:20:59.021911 (+ 27896us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:59.105569 (+ 83658us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:00.000624 (+895055us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x7b5400064354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:00.019458 (+ 18834us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":92136192}
I20260709 12:21:00.134006   766 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.085290   713 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.173723   713 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.174815   729 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:00.175825   750 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:00.175933 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:00.170979 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 104510us
I20260709 12:21:00.176779 32369 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:21:00.167423   764 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260709 12:21:00.177145   729 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:21:00.177925   764 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:00.178437   764 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:00.179628   747 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:00.179466   764 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.180290   568 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.177188 32343 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:00.181285   670 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:00.181876   670 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.155097   695 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:00.184549   752 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:00.190996   770 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:00.191183   752 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.185117   771 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.185335   624 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:00.191689   752 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.160554   686 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:00.192230   686 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.185564 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.182992 31792 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:00.185755 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.185173 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:00.186237   772 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.187358   625 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.187695   670 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.188073   639 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.189302 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.190121   708 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.193928 31678 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.195618 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.196533 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.182523   568 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.200695   730 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.200703   739 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.201922   556 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.208309 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.234009 31822 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:00.234159 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.180811   764 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.157737   572 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.217636 31674 raft_consensus.cc:1813] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Handling vote request from an unknown peer 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:00.103803   760 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.205099   686 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.183466   750 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:00.204452   766 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:00.211005   770 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:00.234843 31753 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:00.235175 31678 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
W20260709 12:21:00.235551 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:00.245492 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:00.246001 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.215441   727 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.238348 31674 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:21:00.239403 31744 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.236846 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.219188   767 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.237610   568 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.235353 32374 raft_consensus.cc:2410] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 3 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:21:00.248050 31752 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:00.232293 32369 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.248709 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.264129 31827 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:00.242573   572 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.247802 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.266062   568 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.247125   770 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.267976 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:00.280380 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.268503   728 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:00.268637 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.269562   723 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.262205   705 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:00.270241   653 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
W20260709 12:21:00.268038 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.839s	user 0.000s	sys 0.031s
W20260709 12:21:00.287735 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.000s	sys 0.031s
W20260709 12:21:00.288113 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 859159us
I20260709 12:21:00.289227 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:00.263801   708 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.272620   760 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.274412   763 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.290596 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:00.282466   728 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:00.305333 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.305789   728 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.281824   767 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:00.295637   708 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.296716   616 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:00.255451   766 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.300149 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.310120 31752 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:00.301474 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.254925   727 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.305363 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.283159   770 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.308231   767 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.313618   690 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:00.290151   653 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:00.316661   767 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.306733   763 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.320626 31818 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.324244   659 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:00.324538   766 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.324771   659 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:00.325748 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:00.327250 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:00.327797 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.328002   778 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.328114   708 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:00.328604 31825 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:00.328817   778 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.330174 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:00.330610 31827 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:00.304173   639 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.337211   755 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:00.340421 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.279882   572 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.344733 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.307122 31746 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:00.349471 32378 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:21:00.317413   760 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.368875 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.369864 31752 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:00.370160 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.355424   783 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.370865 31753 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:00.371220 31748 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:00.329699 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.347728   730 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.356168 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.374125 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.360208 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.361779 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:00.361297   736 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.363991   604 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.316449   690 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:00.429174 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.353715 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:00.335081   728 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.430186 31752 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.339334   755 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.339326   653 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.337857   778 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.343365   670 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.387187 32372 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:00.387380 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.394757 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.400066 31753 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:00.449863 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.400259 31750 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:00.427162 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:00.451376 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.445570   670 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.452131 31745 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:00.418452   602 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:00.420236 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.427085 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.313874   750 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.431437 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:00.433209 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 150672us
I20260709 12:21:00.445536 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:00.444208   771 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:00.450914 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.402086 31751 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:21:00.451937 32370 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:00.428010 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.461866 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.462194 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.461872 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.417589   572 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:00.463572 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.464037   670 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.464231 31669 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:00.465335 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.466014 31669 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.453274 31749 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:00.325773   659 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:00.454735 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 99054.4us
I20260709 12:21:00.434259 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.440161   678 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:00.457080 31718 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:00.446725 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 93142.2us
I20260709 12:21:00.458940 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:00.436005 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45366.2us
I20260709 12:21:00.462513 31753 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.400758   730 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:00.487931 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.459503 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:00.452000 31750 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:00.458362 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.477669 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:00.478170 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.449054 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.453778   758 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:00.482558 31675 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:00.480705 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.480916 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:00.482095   653 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.392283   783 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.479882   771 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:00.477375 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.508535 31748 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:00.494400 31671 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:00.490120 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:00.491822 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:00.514947 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:00.492992 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:00.448971   708 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:00.499430 31751 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.500118 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.494038   572 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:00.436928   604 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.457676   602 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.449666   672 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260709 12:21:00.483561   678 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:00.559332 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.560105 31820 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:00.560432   678 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.561192 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.561625   678 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:00.561923 31820 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:00.562703 31669 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:00.562896 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.505584 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:00.563808 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.565022 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.482764   736 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.519618 31752 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.476146   691 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:00.466557   770 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.522419 31753 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.468109   686 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.501942   784 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:00.524771   661 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:00.493175   766 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.523708 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.565798   715 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.498183   758 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.587523   758 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.587342   715 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.498883   568 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.590931 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:00.591099   770 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.519100   624 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:00.591826   624 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.504083   767 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.504179   746 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.567984   785 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:00.558740 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:00.595599 31747 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.497488 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.510891 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.567636 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:00.596434 31676 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.594300 31753 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:00.597258 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.597359 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:00.598069   736 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.598229 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:00.599013   736 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:00.512203 31821 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:00.564904   771 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:00.559043 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.512606   783 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.599987   718 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:00.589411   715 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:00.601035   718 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:00.601409   718 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:00.497376   755 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.586913 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.635478 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1413) took 1645 ms. Trace:
I20260709 12:21:00.586968 31751 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:21:00.593838 31677 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.637635 31823 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:00.592226 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.512104   666 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.598383 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.635661 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:00.599087   761 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.897s	user 0.009s	sys 0.015s
I20260709 12:21:00.529517 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:00.563803 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 69500.3us
I20260709 12:21:00.592195   752 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.528029   572 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.638787 31820 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:00.605669 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.640695 31826 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:00.529785   714 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:00.636077 31673 rpcz_store.cc:276] 0709 12:20:58.989846 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:58.989988 (+   142us) service_pool.cc:224] Handling call
0709 12:20:59.334778 (+344790us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:00.176136 (+841358us) spinlock_profiling.cc:243] Waited 7.24 ms on lock 0x7b1800077260. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcdcdbfb 00007fe8fcdd864b
0709 12:21:00.176275 (+   139us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:00.176303 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:00.176861 (+   558us) log.cc:844] Serialized 26 byte log entry
0709 12:21:00.259560 (+ 82699us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:00.259619 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:00.259741 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:00.458014 (+198273us) raft_consensus.cc:1685] finished
0709 12:21:00.458035 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:00.635253 (+177218us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b54002e8450. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:00.635417 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19063,"raft.queue_time_us":91678,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":13441,"spinlock_wait_cycles":778571520,"thread_start_us":178222,"threads_started":2,"wal-append.queue_time_us":155186}
I20260709 12:21:00.603039   727 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:00.634662 31669 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 2: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:00.603376 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:00.566216   733 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260709 12:21:00.592239   624 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:00.644656   761 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.941s	user 0.009s	sys 0.015s
I20260709 12:21:00.648936 31673 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:00.624737   787 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:00.585932   661 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:00.655471 31747 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:00.592906 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:00.658722 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 55422us
I20260709 12:21:00.638731   726 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:00.651541 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.529487   708 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.612360   784 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:00.647222 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 117821us
I20260709 12:21:00.669188 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:00.651458 31825 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:00.670442 31676 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:00.652320   770 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.644351 31671 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:00.675113 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.648964   745 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.660761 31670 raft_consensus.cc:2368] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:00.649351   572 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.650890   714 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.677305   602 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:00.677798 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.678447 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.662673   616 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:00.585708   691 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.681802   747 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:00.672530   787 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:00.682852 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:00.687980 31673 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.711817 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.651990   778 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:00.684657   770 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.693830 31820 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:00.678966 31744 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:00.667910 31818 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:00.667681 32370 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:21:00.739378   602 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.744841   639 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.748539 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1498) took 2225 ms. Trace:
I20260709 12:21:00.684808   763 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.748477   602 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.749222   604 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.749895   686 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.691856   726 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:00.670982   661 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.690531 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:21:00.751859   789 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
W20260709 12:21:00.752082 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61625us
I20260709 12:21:00.752722   715 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:00.752799 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:00.753271   715 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.753692   715 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.754022 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:00.644474   746 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.754465   770 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:00.755051   770 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:00.748581   639 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.755591   770 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.756172   746 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.756829   736 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:21:00.751895   619 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:21:00.757310   639 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.757527   723 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:00.757906   723 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.758252   723 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.751868 31671 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:00.758320   602 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.759799   784 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.760308 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.760974 31826 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 1.
I20260709 12:21:00.667912   705 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:00.763151   705 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.765388   705 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.749004 31818 rpcz_store.cc:276] 0709 12:20:58.522700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:58.701988 (+179288us) service_pool.cc:224] Handling call
0709 12:20:58.733403 (+ 31415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:59.104568 (+371165us) spinlock_profiling.cc:243] Waited 21.8 ms on lock 0x7b18001d9880. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fce10646 00007fe8fcd4a58e 00007fe8fcdcbf9e 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4
0709 12:21:00.320146 (+1215578us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b58000b00d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd310aa 00007fe8fcdb3447 00007fe8fcdcbfe7 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4
0709 12:21:00.748242 (+428096us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:00.748478 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1805283072}
I20260709 12:21:00.767267 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:00.698153   727 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.767984   727 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:00.769670   782 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:21:00.770231   727 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.602330   718 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.771972 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 286) took 1126 ms. Trace:
I20260709 12:21:00.772447   704 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.773018   691 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.773128   704 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.744830   568 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:00.774068   568 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:00.774499   568 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:00.743587 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.757345   736 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:00.776754   704 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.777045   736 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.778034 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.779888   708 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:00.780337 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.781172 31825 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 2: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:00.781482   708 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.782003   708 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.772341 32377 rpcz_store.cc:276] 0709 12:20:59.645173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:59.645293 (+   120us) service_pool.cc:224] Handling call
0709 12:20:59.645530 (+   237us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:20:59.972279 (+326749us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:00.771749 (+799470us) spinlock_profiling.cc:243] Waited 260 ms on lock 0x7b54005c30d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:00.771916 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":598951168}
I20260709 12:21:00.782265 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:00.699882   745 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.784096 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.787274 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.788039 31750 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.790602   745 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.791428 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:00.792824 31671 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:00.793134 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.793790 31750 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:00.700558   785 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.094s
I20260709 12:21:00.706657   714 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.700359   774 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.685387   753 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.685763   670 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.750680 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.751421   726 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:00.798148 31750 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:21:00.802181 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 299585us
I20260709 12:21:00.715065 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:00.706465   733 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.703927   616 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:00.810513 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.708230   747 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:00.815322 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:00.810878 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 108094us
I20260709 12:21:00.843431 31751 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
W20260709 12:21:00.825146 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 110124us
I20260709 12:21:00.747498   787 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:00.890502 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:00.758280   770 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:00.819124 31791 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 2.057s (SENT) [suppressed 13 similar messages]
I20260709 12:21:00.842075 31752 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:00.814416   678 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:00.845455 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.846478 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:00.867660 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:00.845826 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.845139 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.890093 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:00.891209 32377 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.892283 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.934911 31751 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:00.900079 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.843010 31669 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:00.936944 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.839876 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.938488 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:00.939121 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:00.939744 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.940476 31671 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.903038 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.948953   633 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:00.949702 31745 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:00.949774   770 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.950047 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:00.903864 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:00.950418   770 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.904901 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.909180 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.952338 32378 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:21:00.910496 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.953660 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:00.910390 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:00.910877 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.912170 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:00.912669 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.913206 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.917479 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:00.825582   753 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.013702 31675 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.822383   730 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:00.943111 32375 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:01.015015   795 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.015537   753 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.016531 31673 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.946067 31827 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:01.018812 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:00.946516 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 100127us
I20260709 12:21:00.818521   774 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.963066   625 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:00.940614   690 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.948804 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:00.950907 31749 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.948359 31750 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:00.908075 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:01.033624 31826 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:21:01.035266 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:00.844666   733 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.010525 32374 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:00.918442   568 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:00.923094   624 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.038229 31671 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:01.042869 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:01.043716   624 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:00.963871 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:00.963899 31820 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:01.045434 31820 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:00.970803 31747 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:00.972743 32377 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
W20260709 12:21:00.973670 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.035120   715 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.049453   715 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:00.999030 31819 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:01.050736   796 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:01.051236   796 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:01.051644   796 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:01.053671   661 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.032676   625 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.055918   625 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.038693   752 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.059198 31749 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:01.037773   572 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.061039 31822 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:01.061708   568 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.062613   623 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:01.063791 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:01.064541 32377 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:01.064469 31670 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:01.063333   572 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:01.066438   728 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.627s	user 0.007s	sys 0.004s
I20260709 12:21:00.839941 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:00.924544   791 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
W20260709 12:21:00.965677 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 274577us
I20260709 12:21:00.845866   757 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.815125   758 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.014586 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:01.019865 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:00.843843   771 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.149s
W20260709 12:21:01.082242 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62471.5us
I20260709 12:21:01.083024 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:01.083918 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:01.020469 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:01.084656 31751 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:00.840807   790 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:00.828682   726 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:00.832760   670 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:00.918363   727 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.091s
I20260709 12:21:01.024448 31745 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:01.027364   770 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:00.954494 32376 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:21:00.975661   678 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:00.925405   747 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.000406   708 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:01.000182   767 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.093891   785 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:01.107801   785 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.107908   659 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.108268   785 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.998270   712 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:01.030910 31669 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:00.972234 32370 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:00.845146   616 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:01.035039 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:01.118108 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 83158.3us
I20260709 12:21:00.840224   778 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:01.094398   783 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:01.070139 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 251032us
W20260709 12:21:01.124809 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:01.125643 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.127041 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:01.119761   778 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.071908 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:01.079777 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:01.079998 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:01.037840   653 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.529s	user 0.009s	sys 0.002s
I20260709 12:21:01.093408 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:01.085637   694 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 12:21:01.055882   661 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:01.109228 31674 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:01.066601   774 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:01.097451 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:01.113416 31753 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:01.092002   633 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:01.120632 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:01.015621   795 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.130278 31747 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:01.130278 31750 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:01.131996 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:01.152911 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:21:01.134208 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62378.9us
I20260709 12:21:01.112463   712 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:01.149590   633 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.151736 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:01.161244 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:01.123780   767 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:01.136194   747 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:01.133333   778 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.142190   706 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:01.149421   784 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.095852   714 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.133049   770 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.150840   736 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:01.162762   705 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:01.150578   799 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:01.181737 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:01.163772   774 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:01.167852   712 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.173702   633 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.173343   653 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:01.173063   766 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.177273   767 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.095450   757 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.094223   790 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.120401   787 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.218256   625 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.177065   715 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:01.167711   619 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:01.174080   678 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.189790   670 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:01.198702   758 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:01.200402 31675 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:01.165994   752 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
W20260709 12:21:01.208618 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 79912.8us
I20260709 12:21:01.183214   661 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:01.231670   802 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 12:21:01.232098   661 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.202034 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:01.176697   723 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.213748 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:01.213819   639 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:01.191805   733 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:01.210043   653 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.236552   653 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.236724 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:01.236987 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:01.237890 31671 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:21:01.224017 32372 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:01.186282   795 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.207716   736 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:01.188946   799 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:01.211017   774 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:01.201936   796 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.230686 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:01.187577   770 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.192024   705 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.188133   784 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.236260   639 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.215147   767 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:01.240286   714 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:01.241521 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:01.241466 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.226091   619 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.237970   730 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:01.240964   728 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:01.225696   787 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.249771   661 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.250231 31827 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:01.250550   723 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:01.251139   625 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.241276   715 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 12:21:01.287894 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:01.276542   774 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.288874   784 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.214326   766 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.289474 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:01.290150 31824 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:01.261880   790 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.291874   766 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.293095 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:01.293773 31678 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:01.281052   730 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.299201   730 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:01.257712 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:01.301481 32369 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:01.301151   785 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.260330 31791 leader_election.cc:400] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:01.263592 32377 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 3: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:01.303227   757 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.264452 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:01.304550   766 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:01.305096   766 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.305503   766 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.305847   770 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:01.307207 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:01.304561 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:01.310148 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:01.311142   714 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:01.311203 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:01.311735   714 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:01.312026 31821 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:01.312288   714 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:01.313056 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:01.314016 31821 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 2: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:01.314849 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:01.315133   491 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:01.315495 31821 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:01.248386   799 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:01.308521   718 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:01.318380 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:01.319032 31824 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:01.320374 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:01.321039 31750 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:01.271333 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:01.318390 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 59325.8us
I20260709 12:21:01.244668 32376 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:01.324267 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:01.329845 31747 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.332996 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:01.267705   705 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:01.271549 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:01.276440 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:01.339480 31749 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:01.257735   736 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:01.341812   736 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:01.315642   670 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:01.343418 31671 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:01.343621   670 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:01.342780   736 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.342698   639 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.338053   633 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.326007 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:01.327383 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:01.269474   619 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:01.323791 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:01.335484 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.272262   728 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:01.340786 31823 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:21:01.361027 31750 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:01.365089 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:01.335202   757 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260709 12:21:01.354758 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:01.366496 31752 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:01.254602 32370 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:01.367482 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:01.381340   767 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.384544   767 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.384718 31675 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:01.387833 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1408) took 1052 ms. Trace:
I20260709 12:21:01.388233 31822 rpcz_store.cc:276] 0709 12:21:00.335813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.602277 (+266464us) service_pool.cc:224] Handling call
0709 12:21:01.387627 (+785350us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b5400446bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:01.387753 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":902001536}
W20260709 12:21:01.392724 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 311195us
I20260709 12:21:01.395673   796 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.396147 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:01.397459 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:01.397584 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:01.398401 32377 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:21:01.399045 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:01.399276 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:01.400110 31752 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:01.402037 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:01.403091 31824 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 2: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:01.404452 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:01.405112 31826 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:01.408224 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:01.409008 31678 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.413218 31751 raft_consensus.cc:2368] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3.
W20260709 12:21:01.414335 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.426174 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:01.428643 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:01.434747 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:01.435731 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 106749us
I20260709 12:21:01.435791 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:01.436107   661 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:01.436637   661 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.436614 31748 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:01.437314   661 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.437683   678 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.437942 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:01.438436 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:01.438396   678 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.439219 31748 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:01.439051 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:01.436419 31823 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:01.402961   796 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:01.435880 31678 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:01.440558 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:01.441568 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:01.473865 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:01.474846 31822 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:01.475575 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:01.369700   730 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:01.474489 31748 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.481287   785 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.369755   770 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:01.482384 31824 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:01.440363   810 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:01.484917 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:01.417378   723 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:01.407825   733 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.431708   625 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.489843 31747 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:01.409521   715 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:01.487495   810 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.500823   715 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.501190   810 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.441732   704 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:01.495007 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:01.508111 31749 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.499362 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:01.499711 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:01.486028   784 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.431339   564 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.501248   715 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:01.492233 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:01.519232 31823 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:01.519719   705 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:01.520287 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:01.521065 31823 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:01.477949 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:01.431087   616 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.477237 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:01.517155   564 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.476331   619 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.522085 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:01.505892   704 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.485220   730 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:01.527976 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:01.511706 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:01.449047   653 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.495539   785 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:01.502831 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:01.473903   726 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:01.520385   705 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:01.534548 31672 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:01.584548 31823 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 2: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:01.540697 31749 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:01.473483   796 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.506263   723 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:01.542239 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:01.649328 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:01.650190 31676 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
W20260709 12:21:01.650360 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 108197us
W20260709 12:21:01.547223 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.515483   771 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.651479 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:01.667424 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:01.668480 31749 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:01.567649 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:01.506942 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:01.507418   625 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.529721 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:01.502942   733 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.548497   604 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.568784   704 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:01.653739 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:01.750176 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 96523.1us
I20260709 12:21:01.529377   784 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.657196   619 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:01.657212 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:01.577324   556 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:01.659555   653 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:01.660645   659 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.552s	user 0.006s	sys 0.005s
I20260709 12:21:01.665555   814 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:01.663175   705 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.664237   564 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:01.575011   745 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:01.750941   785 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:01.579043 31819 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:01.572923   730 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:01.759851 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:01.649133   796 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.779361   556 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.665879   672 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:01.648511 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:01.673954   623 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:01.578562   778 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.763262   604 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:01.767547   712 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.555s	user 0.010s	sys 0.000s
W20260709 12:21:01.671922 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:01.760649   758 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:01.794706 31745 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:01.797787 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:01.770133   736 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:01.801973 31822 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:01.746718 31825 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:01.747117   771 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:01.801924 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 133054us
I20260709 12:21:01.792076   633 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.889807 31748 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:01.879169 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 984) took 1374 ms. Trace:
I20260709 12:21:01.880581 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1091) took 1063 ms. Trace:
I20260709 12:21:01.795604   745 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.951033   745 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:01.893781   808 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.362s	user 0.001s	sys 0.000s
I20260709 12:21:01.894161 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:01.894794   766 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.589s	user 0.004s	sys 0.009s
I20260709 12:21:01.895555   726 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:21:01.814599   714 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.502s	user 0.016s	sys 0.000s
I20260709 12:21:01.797070   616 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:01.938616 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:01.972226 31791 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:01.795063   619 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.096s
W20260709 12:21:01.975771   805 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.777s	user 0.017s	sys 0.038s
I20260709 12:21:01.886744   723 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:01.981700   718 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.665s	user 0.009s	sys 0.000s
I20260709 12:21:01.984401   604 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.984751 32370 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:01.985354   564 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:01.968199 31671 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:01.989177   715 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.886937   672 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.991937   784 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:01.993145   672 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:01.801206   785 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.994805 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1413) took 1631 ms. Trace:
I20260709 12:21:01.995357   733 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:01.900446   758 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.883920   623 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.942809 31674 rpcz_store.cc:276] 0709 12:21:00.505155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.510969 (+  5814us) service_pool.cc:224] Handling call
0709 12:21:01.878774 (+1367805us) spinlock_profiling.cc:243] Waited 537 ms on lock 0x7b54001d5e54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:01.879043 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1235917696}
I20260709 12:21:01.792507   799 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.886454   639 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:01.963686 31676 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:01.892390   712 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:01.965301 31822 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:01.885610   778 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:01.920239   736 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.003087 31677 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:02.006150 31643 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:02.010924 31824 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.520s	user 0.007s	sys 0.001s
I20260709 12:21:02.011343 31824 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:01.948071 31753 rpcz_store.cc:276] 0709 12:21:00.816964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.817070 (+   106us) service_pool.cc:224] Handling call
0709 12:21:00.963721 (+146651us) spinlock_profiling.cc:243] Waited 5.78 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:01.880322 (+916601us) spinlock_profiling.cc:243] Waited 68.8 ms on lock 0x7b54001ee1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:01.880492 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":171528448}
I20260709 12:21:02.013623 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:01.995234 31818 rpcz_store.cc:276] 0709 12:21:00.340293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.953567 (+613274us) service_pool.cc:224] Handling call
0709 12:21:00.953852 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:00.954057 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:21:01.184501 (+230444us) log.cc:844] Serialized 14 byte log entry
0709 12:21:01.891381 (+706880us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:01.901562 (+ 10181us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:01.920994 (+ 19432us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:01.955487 (+ 34493us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:01.971367 (+ 15880us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":245416,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":35505,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":75,"thread_start_us":859278,"threads_started":3,"wal-append.queue_time_us":394019}
I20260709 12:21:02.015100   770 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:01.954375   810 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:01.938238   706 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:02.017727 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 114) took 1137 ms. Trace:
I20260709 12:21:02.023770 31677 rpcz_store.cc:276] 0709 12:21:00.880618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.901264 (+ 20646us) service_pool.cc:224] Handling call
0709 12:21:02.017556 (+1116292us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54001d6354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:02.017670 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2418071936}
I20260709 12:21:01.998904   666 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:01.951069   572 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:02.006872   766 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:02.019304   799 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:02.019526   758 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:01.985881   564 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.019949   817 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:02.025820   770 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:01.998668   661 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.561s	user 0.009s	sys 0.000s
W20260709 12:21:02.032542   816 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:21:02.007906   714 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:02.015471   736 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.002874   785 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:02.032042 31752 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.627s	user 0.009s	sys 0.000s
I20260709 12:21:02.035539   616 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:02.003679   625 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:02.033025 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:02.022235 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:02.054344 31822 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:02.023687   653 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:02.035771 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:02.029886   706 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:02.020004   745 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.033737   564 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.052625 31752 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:02.033828 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:02.056938 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:21:02.125524 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 68681.9us
I20260709 12:21:02.127030 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:21:02.026408 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 369271us
I20260709 12:21:02.050824   718 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:02.130780 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:02.057791   604 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:02.168941 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:02.169793 31677 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:02.170970 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:02.171694 31677 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:02.159557 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:02.177656 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:02.044747   661 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:02.198722 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:02.062299   619 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:02.199648 31674 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:02.179818 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:02.202394 31642 leader_election.cc:400] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3.
I20260709 12:21:02.202865 31642 leader_election.cc:403] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:02.203920 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:02.205552 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:02.205968 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:02.206302 31752 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:02.208402 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:02.208870 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:02.209200 31793 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.511s	user 0.007s	sys 0.044s
I20260709 12:21:02.209573 31753 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:21:02.209824 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:02.211087 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:02.211350 31748 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:02.220289 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:02.158782   760 raft_consensus.cc:2764] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:21:02.223276 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:02.172736   625 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.213662 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:02.208710   784 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.008s
I20260709 12:21:02.197196   604 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:02.179278   730 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.200700   639 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:02.212200 31749 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:21:02.225242 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:02.228456 31748 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:02.202889   619 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.253521 31747 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:02.177469   491 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:02.207649   747 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:02.211817   814 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:02.177088   726 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:02.235827 31753 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:02.180884   616 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:02.243822 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:02.212061   818 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:02.212776   715 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:02.262339 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:02.233530   623 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:02.247251   758 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.259712   619 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:02.271673 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61914.7us
I20260709 12:21:02.273310 31752 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:02.178426   718 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:02.256017   723 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:02.276067 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:02.264830   726 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.256771 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:02.261622   712 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:02.241864   625 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.282315   726 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.286223 31746 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:02.287930   745 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:02.288322   823 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:02.288882   823 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.289273   823 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.273272   584 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:02.277228   814 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:02.284745   616 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:02.291391   814 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.291103   604 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:02.291812   814 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.292234   616 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.284946   818 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:02.295259   714 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:02.295763   818 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.297134   730 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:02.279623   723 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.299654   723 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:02.264691   491 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.300899   771 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:02.301074   584 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:02.302079   584 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.284106   623 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:02.305342   623 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.276607   766 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:02.306383 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:02.281803   810 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:02.305446 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:02.277324 31669 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:21:02.322999 31671 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:02.325519 31672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 959) took 3816 ms (client timeout 2057 ms). Trace:
W20260709 12:21:02.326179 31672 rpcz_store.cc:269] 0709 12:20:58.508852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:58.509031 (+   179us) service_pool.cc:224] Handling call
0709 12:21:02.325245 (+3816214us) spinlock_profiling.cc:243] Waited 2.98 s on lock 0x7b54003921d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:02.325368 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6855877376}
I20260709 12:21:02.336623 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1534) took 1319 ms. Trace:
I20260709 12:21:02.278174 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:02.345273   747 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:02.345803   747 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.346289   747 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:02.337160 31827 rpcz_store.cc:276] 0709 12:21:01.017639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.018495 (+   856us) service_pool.cc:224] Handling call
0709 12:21:02.336347 (+1317852us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b54003c96d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:02.336538 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":530399488}
W20260709 12:21:02.347805 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 69733us
I20260709 12:21:02.349390 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:02.350301 31678 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:02.350580 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:02.371305   572 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260709 12:21:02.377962 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:02.383137 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:02.384895 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:02.385957 31672 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:02.390147   678 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:02.390475   830 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:02.396284 31744 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:02.400029 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:02.400607   783 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.401052 32369 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:02.401345   783 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.404686   690 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:02.409436   706 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:02.411438   572 consensus_queue.cc:1243] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:21:02.411921 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 290) took 1894 ms. Trace:
I20260709 12:21:02.401081 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:02.414057 31744 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:02.412596 32372 rpcz_store.cc:276] 0709 12:21:00.517027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.521051 (+  4024us) service_pool.cc:224] Handling call
0709 12:21:02.411648 (+1890597us) spinlock_profiling.cc:243] Waited 622 ms on lock 0x7b54006c0554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:02.411830 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1429580544}
I20260709 12:21:02.414794 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:02.415544 31749 raft_consensus.cc:2368] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4.
I20260709 12:21:02.416129 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:02.416949 31669 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:02.419315 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1424) took 1149 ms. Trace:
I20260709 12:21:02.419899 31819 rpcz_store.cc:276] 0709 12:21:01.270236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.272020 (+  1784us) service_pool.cc:224] Handling call
0709 12:21:01.274321 (+  2301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:01.801632 (+527311us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:02.418966 (+617334us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b54001c8ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:02.419233 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":689418112}
I20260709 12:21:02.432080 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:02.433133 31821 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:02.433622 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1539) took 1033 ms. Trace:
I20260709 12:21:02.433969 31821 rpcz_store.cc:276] 0709 12:21:01.400436 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.403463 (+  3027us) service_pool.cc:224] Handling call
0709 12:21:02.431870 (+1028407us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:02.433551 (+  1681us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":51016960}
W20260709 12:21:02.451318 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 98355.9us
I20260709 12:21:02.454062 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:02.456130 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:02.456804 31749 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:02.457041 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52837.7us
I20260709 12:21:02.461449 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:02.461599 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:02.462249 31672 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:02.462448 31677 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:02.463352 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:02.464321 31753 raft_consensus.cc:2368] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:02.464350 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:02.465299 31677 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:02.468591 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:02.469571 31748 raft_consensus.cc:2368] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:02.478996 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:02.479943 31672 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:02.488008 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:02.488583 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:02.492637 31678 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:02.500311 31677 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:02.502414 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:02.509150 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:02.511005 31674 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.511406 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:02.512109 31749 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:02.513986 31677 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:02.475909   799 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.533553 31672 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:02.499687   572 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:02.537683 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 261720us
W20260709 12:21:02.535795 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 47287.9us
I20260709 12:21:02.538897 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:02.540016 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:02.508121   715 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.508211   727 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:02.510902   730 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:02.558946 31672 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:02.540378 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:02.567067 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:02.572058 31642 leader_election.cc:400] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4.
I20260709 12:21:02.572435 31642 leader_election.cc:403] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:21:02.526372   783 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:02.535055   828 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:02.523831   718 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:02.562532   584 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:02.540009 31792 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:02.501742   830 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.059s
I20260709 12:21:02.580739 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:02.572012 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 107598us
I20260709 12:21:02.558995   572 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.536226   799 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:02.536517 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:02.601869 32342 leader_election.cc:400] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:02.602237 32342 leader_election.cc:403] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:21:02.571200   712 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:02.575698 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:02.608347 31749 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.565438   730 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.564962 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:02.590437 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 101042us
I20260709 12:21:02.625247 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:02.573899   705 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:02.583853 31746 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:02.582194   556 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:02.620100 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 80155.6us
I20260709 12:21:02.633898 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:02.597311   572 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.580890   820 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4.
I20260709 12:21:02.618165 31749 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:02.579677   656 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260709 12:21:02.630487   705 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:02.618754 31675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1437) took 1543 ms (client timeout 2002 ms). Trace:
I20260709 12:21:02.601053   624 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.028s
I20260709 12:21:02.596486   491 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:02.590502 31669 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:02.630026 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:02.586911   771 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:02.597630   796 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.706s	user 0.014s	sys 0.000s
I20260709 12:21:02.565217   715 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.597050   584 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.641507   799 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:02.641731 32372 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 2: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:02.587610   678 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260709 12:21:02.644186 31674 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:02.644354 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:02.663789   838 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:02.659495 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:02.637722 31744 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:02.625501   730 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:02.573472   727 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:02.607894   712 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:02.624580   661 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:02.624737   828 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:02.637809   781 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:02.676565 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 74832.1us
I20260709 12:21:02.660530   705 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:02.655609 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50328.3us
I20260709 12:21:02.692308 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:02.668926   830 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.702008 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:02.683708 32370 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:02.672173   774 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:02.679736   584 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.676209   624 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:02.701383 32342 leader_election.cc:400] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:02.697070 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:02.720813 32342 leader_election.cc:403] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:21:02.721123 31674 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:02.711796   818 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:02.690701   706 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:02.700021   623 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:02.660869 31675 rpcz_store.cc:269] 0709 12:21:01.075221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.236380 (+161159us) service_pool.cc:224] Handling call
0709 12:21:02.591656 (+1355276us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b5400037354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:02.605587 (+ 13931us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":338204544}
I20260709 12:21:02.697379   712 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.713761 31717 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:02.677117   796 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.676155   771 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:02.696169   704 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.929s	user 0.007s	sys 0.012s
I20260709 12:21:02.715075   835 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:02.723357   670 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:02.747824 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:02.748819 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 118865us
I20260709 12:21:02.698457   726 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.750075   760 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:02.724294   783 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:02.775255   783 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:02.776643   783 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.751981 31674 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:02.722843   624 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:02.778793   624 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:02.780122   624 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:02.779433 31572 catalog_manager.cc:5697] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 2 to 3, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 3 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:21:02.755261 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 95889.6us
I20260709 12:21:02.725458   751 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.725785   706 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:02.761957   766 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:02.753595   704 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.753887   670 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:02.765163   827 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:02.770733   819 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:02.759362   623 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.785349 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:02.701015   828 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.721997   774 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.812266 31670 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:02.756551   818 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:02.840255   818 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:02.837989   733 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:02.841418   733 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.843089   708 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:02.843681   708 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.762538   714 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:02.845788   714 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:02.759438   760 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.846968   760 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.848733 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:02.777611 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1007) took 1353 ms. Trace:
I20260709 12:21:02.849627 31671 rpcz_store.cc:276] 0709 12:21:01.424104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.989544 (+565440us) service_pool.cc:224] Handling call
0709 12:21:01.990037 (+   493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:02.323585 (+333548us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:02.777301 (+453716us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b54003917d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:02.777524 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":764967808}
I20260709 12:21:02.850540   604 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:02.850992   604 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:02.851857   604 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:02.826404   666 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:02.853852   666 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:02.827337   849 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:02.854971   849 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:02.855922   849 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.756678   824 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:02.859261   824 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:02.853166 31671 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:02.829504 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:02.879413 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 50013.5us
I20260709 12:21:02.726858   830 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:02.908464 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:02.908922   785 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:02.909333 32370 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:02.909448   785 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:02.803978   706 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:02.910467   785 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.910360 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:02.912349 31819 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.815032   827 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:02.841269   818 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:02.827590   704 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:02.829365   815 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.066s	user 0.003s	sys 0.020s
I20260709 12:21:02.880676 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:02.883150 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:02.883411 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:02.955145 32375 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
W20260709 12:21:02.908689 31670 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1002) took 1506 ms (client timeout 1925 ms). Trace:
I20260709 12:21:02.911820   785 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:02.976240 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:02.814769   670 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.977155 32374 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.832401   819 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:02.971724 31670 rpcz_store.cc:269] 0709 12:21:01.401841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.403260 (+  1419us) service_pool.cc:224] Handling call
0709 12:21:02.908440 (+1505180us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54002553d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:02.908610 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2879671040}
I20260709 12:21:02.947845   835 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:02.757637   845 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:02.915282 31674 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:02.952021 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:03.006439 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:03.008328   733 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:02.818684   836 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.140s
I20260709 12:21:02.947659   718 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:03.016492 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 221429us
W20260709 12:21:03.009049 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 128764us
I20260709 12:21:03.018348 31571 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42070 (request call id 220) took 6598 ms. Trace:
I20260709 12:21:03.019338 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:03.019977 31827 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.980931 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:03.011307   715 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:03.012573 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 60733.2us
I20260709 12:21:03.027774 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:02.913839   694 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:02.834095   623 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.002959   835 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:21:02.973526 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:03.037731 31824 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:03.008733   828 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:03.101009 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:03.017625   818 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:03.023211 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:03.019994   846 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:03.185639 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:03.189131   781 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:03.195061 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:03.196219 31745 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.845s	user 0.027s	sys 0.021s
W20260709 12:21:03.205355   564 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.034s	user 0.002s	sys 0.016s
I20260709 12:21:03.083143 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:03.044626 32371 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:02.978977   819 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.057081 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:03.208618 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 481323us
I20260709 12:21:03.244236 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1476) took 1836 ms. Trace:
W20260709 12:21:03.206542   842 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.178s	user 0.000s	sys 0.000s
W20260709 12:21:03.289121 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:03.288533 31820 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.012s	sys 0.028s
W20260709 12:21:03.289342 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 94381.6us
W20260709 12:21:03.247087 31677 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.003s	sys 0.020s
I20260709 12:21:03.024935   604 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:03.299666 31824 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:03.244212   736 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.186s	user 0.006s	sys 0.026s
I20260709 12:21:03.038478   718 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.183599   625 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:03.298341   770 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.251s	user 0.016s	sys 0.001s
I20260709 12:21:03.314496 31750 rpcz_store.cc:276] 0709 12:21:01.421243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.422172 (+   929us) service_pool.cc:224] Handling call
0709 12:21:01.422649 (+   477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:01.423141 (+   492us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:01.423169 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:01.423620 (+   451us) log.cc:844] Serialized 26 byte log entry
0709 12:21:01.573020 (+149400us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:01.573065 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:01.573139 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:02.828748 (+1255609us) spinlock_profiling.cc:243] Waited 9.68 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:03.182743 (+353995us) raft_consensus.cc:1685] finished
0709 12:21:03.182775 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:03.244139 (+ 61364us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":97,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":19171,"spinlock_wait_cycles":22260096,"thread_start_us":72756,"threads_started":1,"wal-append.queue_time_us":160737}
I20260709 12:21:03.301580   795 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:21:03.321888 31820 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.015s	sys 0.033s
I20260709 12:21:03.249754   751 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:03.246729   846 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:03.018851 31571 rpcz_store.cc:276] 0709 12:20:56.424902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.444489 (+ 19587us) service_pool.cc:224] Handling call
0709 12:21:03.018281 (+6573792us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:20:59.682405 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:20:59.699674 (+ 17269us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:20:59.699715 (+    41us) write_op.cc:435] Acquired schema lock
0709 12:20:59.699735 (+    20us) tablet.cc:662] Decoding operations
0709 12:20:59.700358 (+   623us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:20:59.700443 (+    85us) write_op.cc:641] Partition lock acquired for write op
0709 12:20:59.700486 (+    43us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:20:59.700771 (+   285us) tablet.cc:701] Row locks acquired
0709 12:20:59.700794 (+    23us) write_op.cc:260] PREPARE: finished
0709 12:20:59.700947 (+   153us) write_op.cc:270] Start()
0709 12:20:59.701078 (+   131us) write_op.cc:276] Timestamp: P: 1783599659700906 usec, L: 0
0709 12:20:59.701108 (+    30us) op_driver.cc:348] REPLICATION: starting
0709 12:20:59.701920 (+   812us) log.cc:844] Serialized 994 byte log entry
0709 12:21:00.508230 (+806310us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8f9ccb0ac 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b
0709 12:21:01.164963 (+656733us) op_driver.cc:464] REPLICATION: finished
0709 12:21:01.239460 (+ 74497us) write_op.cc:301] APPLY: starting
0709 12:21:01.245178 (+  5718us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:01.257131 (+ 11953us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:01.259068 (+  1937us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:01.293066 (+ 33998us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:01.293297 (+   231us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 12:21:01.293376 (+    79us) write_op.cc:312] APPLY: finished
0709 12:21:01.293621 (+   245us) log.cc:844] Serialized 29 byte log entry
0709 12:21:01.293802 (+   181us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:01.293903 (+   101us) write_op.cc:454] Released schema lock
0709 12:21:01.293946 (+    43us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:01.294320 (+   374us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:01.294347 (+    27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":53462,"reported_tablets":11,"spinlock_wait_cycles":18816,"child_traces":[["op",{"apply.queue_time_us":60403,"apply.run_cpu_time_us":2394,"apply.run_wall_time_us":64719,"num_ops":2,"prepare.queue_time_us":212840,"prepare.run_cpu_time_us":28095,"prepare.run_wall_time_us":1001145,"raft.queue_time_us":360255,"raft.run_cpu_time_us":11977,"raft.run_wall_time_us":93221,"replication_time_us":1433664,"spinlock_wait_cycles":23921024,"thread_start_us":1344728,"threads_started":4,"wal-append.queue_time_us":953100,"wal-append.run_cpu_time_us":138102,"wal-append.run_wall_time_us":1633654}]]}
W20260709 12:21:03.336019   572 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.698s	user 0.009s	sys 0.022s
I20260709 12:21:03.336031   781 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:03.336230 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:03.055387   694 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.253321   726 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:03.246995   616 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.005s	sys 0.017s
I20260709 12:21:03.207194   830 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.342543 31827 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:03.102576   845 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.267s
I20260709 12:21:03.106194 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.288651 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:03.300029   838 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.008s	sys 0.015s
W20260709 12:21:03.324302 31677 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.003s	sys 0.020s
I20260709 12:21:03.252007 31745 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:03.301268   672 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:03.346731 31572 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.006s	sys 0.027s
I20260709 12:21:03.209712   727 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.358381 31824 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:03.387631 32369 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:03.390851 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:03.365484   781 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.392174 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:03.392817 32369 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:21:03.393422   781 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:03.395241   827 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.396130   694 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.397073 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:03.397334   747 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.050s	user 0.007s	sys 0.011s
I20260709 12:21:03.375974   841 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:03.398002 31818 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:03.397907 31819 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:03.398273   841 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:03.398883   841 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:03.398977 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:03.399864 32372 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:03.403024   855 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:03.403760   855 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.317647   836 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:03.405041   836 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:03.343832   817 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.299s	user 0.014s	sys 0.003s
I20260709 12:21:03.371277 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:03.340754   771 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.342617   829 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:03.303195   564 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.348801   708 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:03.361295   714 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.399044 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:03.373311   616 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.006s	sys 0.017s
I20260709 12:21:03.389258 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:03.434763 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:03.362468   846 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:03.440692   844 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:21:03.437152 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:03.496389 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.498260 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.499192 31753 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:03.439128 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:03.386770 31572 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.007s	sys 0.027s
W20260709 12:21:03.387595   838 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.009s	sys 0.016s
I20260709 12:21:03.349753   736 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.380033 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.379570   726 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:03.359648   770 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:03.438642   666 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.552s	user 0.003s	sys 0.011s
I20260709 12:21:03.369107   572 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.520674 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:03.510641 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 71558.6us
I20260709 12:21:03.440065   817 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:03.441711 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1102) took 1225 ms. Trace:
I20260709 12:21:03.514124 31824 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:03.523160 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 205653us
I20260709 12:21:03.514861 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:03.523896 31749 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:03.515743 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.517040 31823 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:03.518121 31752 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:03.397902   747 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:03.510281   758 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.226s	user 0.019s	sys 0.016s
I20260709 12:21:03.440289   727 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20260709 12:21:03.527439 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:03.514776 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1544) took 2086 ms. Trace:
I20260709 12:21:03.546535 31827 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:03.535593 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 166835us
I20260709 12:21:03.409562   672 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.524024 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:03.517745   704 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:03.541535 31826 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:03.542049 31749 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:03.372900   835 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.548002 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:03.516548   838 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:21:03.528988   690 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.124s	user 0.021s	sys 0.009s
W20260709 12:21:03.520596   823 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.231s	user 0.016s	sys 0.029s
I20260709 12:21:03.555451 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:03.542045   814 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.250s	user 0.014s	sys 0.013s
W20260709 12:21:03.556021 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 118996us
W20260709 12:21:03.541046   723 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.241s	user 0.032s	sys 0.019s
I20260709 12:21:03.545305 31820 rpcz_store.cc:276] 0709 12:21:01.428348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:01.454821 (+ 26473us) service_pool.cc:224] Handling call
0709 12:21:02.442268 (+987447us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:02.442559 (+   291us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:02.443061 (+   502us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:02.443103 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:02.443748 (+   645us) log.cc:844] Serialized 26 byte log entry
0709 12:21:02.672025 (+228277us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:03.359553 (+687528us) log.cc:844] Serialized 14 byte log entry
0709 12:21:03.514373 (+154820us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:03.514457 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:03.514500 (+    43us) raft_consensus.cc:1685] finished
0709 12:21:03.514569 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:03.514718 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27671,"prepare.queue_time_us":687934,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":47,"raft.queue_time_us":103274,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":35987,"spinlock_wait_cycles":48545152,"thread_start_us":1056205,"threads_started":3,"wal-append.queue_time_us":278725}
I20260709 12:21:03.601358 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.602545 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:03.559950 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.602876 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.603406 31823 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:03.517321   678 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:03.565601 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.570488 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:03.550455   760 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.703s	user 0.011s	sys 0.019s
I20260709 12:21:03.536913   616 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:03.542261   758 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3.
I20260709 12:21:03.539532   828 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:03.491081   829 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.583848   865 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:03.548735   771 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:03.549021 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.598065 31678 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
W20260709 12:21:03.566653 32374 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.589s	user 0.011s	sys 0.004s
I20260709 12:21:03.525384   726 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.603451 31752 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:03.551811   849 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:03.567565 31753 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:03.560117   851 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20260709 12:21:03.525197   666 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.567034   704 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:03.611634   728 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:03.613114 31561 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f6404cb9c2174326b06cbbb0d27d43f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b95aeeaf4c51442b8092cff1de674f2b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:21:03.615202   852 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.168s	user 0.000s	sys 0.000s
W20260709 12:21:03.617667   705 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.903s	user 0.012s	sys 0.009s
I20260709 12:21:03.562719   857 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:03.579237   817 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:03.618325   838 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.618772 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.584558 31746 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:03.522928 31745 rpcz_store.cc:276] 0709 12:21:02.210334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.211975 (+  1641us) service_pool.cc:224] Handling call
0709 12:21:02.350430 (+138455us) spinlock_profiling.cc:243] Waited 3.38 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0e9c 00007fe8f9cb0a5b 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:03.410183 (+1059753us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7775488}
I20260709 12:21:03.622442 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.608222 32375 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:03.626991 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:03.686249 31750 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:03.631192 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.593068   623 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.634840 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.686975 31823 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:03.631563 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:03.687956 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:03.688287   623 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:03.688750   623 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:03.650683   861 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:03.687980 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.622865   836 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:03.692183 31752 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260709 12:21:03.652974 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55987us
I20260709 12:21:03.655153 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:03.654353 31747 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:03.656744 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:03.693588 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:03.635582   865 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:03.641533 32374 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:03.636972   653 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:03.700958   712 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.950s	user 0.012s	sys 0.015s
I20260709 12:21:03.701565   653 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.701769   712 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:03.703399 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:03.701278   865 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:03.704272   712 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.705049 31672 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:03.601197   823 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.606775   814 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.610882   723 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.623368   760 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.625545   616 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.665725 32378 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:03.628153   828 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.671914 31753 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:03.680496   726 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.681404   572 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.720013   866 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:03.656590   849 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.659404   704 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:03.662842   730 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.964s	user 0.015s	sys 0.015s
W20260709 12:21:03.688009 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 56580.4us
I20260709 12:21:03.727353 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:03.642009 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 94030.4us
W20260709 12:21:03.635317   670 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.658s	user 0.008s	sys 0.017s
I20260709 12:21:03.689141   623 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:03.733547   867 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:03.734155 32374 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:03.734575 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:03.735242 32376 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:03.545930 31821 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:03.688030 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.694355 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:03.684252   817 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:03.696696   584 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.969s	user 0.010s	sys 0.005s
I20260709 12:21:03.694936 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:03.748713 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53852.1us
I20260709 12:21:03.723877   736 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:03.700172 31676 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:03.700433   625 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:03.685982   706 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.774s	user 0.017s	sys 0.018s
I20260709 12:21:03.700964   862 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:03.700461   846 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.673533   728 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:03.713455 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.689736 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.715288 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.683938   829 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.721526 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.761250 31825 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:21:03.735306 32372 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
W20260709 12:21:03.721994   619 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.427s	user 0.015s	sys 0.006s
W20260709 12:21:03.665582 31561 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 52744.8us
W20260709 12:21:03.729594 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:03.765465 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:03.705035 32373 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:03.673938   705 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:03.740773 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46474.3us
I20260709 12:21:03.741483 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.742095   864 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:03.736294   730 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.696362   666 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.672607   783 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260709 12:21:03.750061 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:03.750676   771 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:03.699970   758 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:03.751353   712 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:03.689574   861 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.722739   866 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.708734 31750 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:03.757714   706 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.757807 31751 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:03.760025 31747 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:03.754779   625 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:03.742936   726 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:03.752616   584 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.773180   828 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:03.773291 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.778925 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:03.766718   619 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.780400   760 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.738391   670 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:03.812350   869 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:21:03.813072   706 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.813299   670 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.757472   862 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:03.690994   857 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.785467 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:03.814493 31753 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:03.786664   736 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260709 12:21:03.781536   705 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.728401   841 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.736104   604 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:03.789748   666 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:03.835506   604 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:03.836869   584 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.837358   824 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.787776   829 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:03.835541   666 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.838591   849 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:03.839243   616 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:03.839619   823 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:03.795092 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 160098us
I20260709 12:21:03.790414 31675 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:03.734150   867 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.843178   723 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.798525 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:03.815677 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:03.780443   846 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.781080   864 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.816282   758 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:03.817389   694 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:03.814239   862 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:03.796562   625 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:03.809243 31818 raft_consensus.cc:2368] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4.
I20260709 12:21:03.826519 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:03.827453   785 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.851s	user 0.020s	sys 0.000s
I20260709 12:21:03.865978 31749 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:03.809625   726 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:03.838367   829 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:03.851508 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:03.880033 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:03.880650 31751 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:03.852105 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53637.1us
I20260709 12:21:03.852604 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:03.859416 31823 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:03.860944 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:03.884848 31746 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:03.866377 32369 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:03.886937 31825 raft_consensus.cc:2368] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:03.888054 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:03.888485 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.889883 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:03.883854 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:03.888326   619 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:03.873720 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:03.899506   873 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:21:03.899260   704 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.900467 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:03.901234   864 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.861578 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:03.903944 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:03.904801 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:03.855758   823 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:03.858412   723 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:03.908018 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:03.906394 31751 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:03.902292   704 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.908949 32377 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:03.895542   616 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 12:21:03.917846 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:03.889897 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:03.915274   670 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:03.924520   568 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:03.905790   666 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.836980   604 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.907029 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:03.920521   760 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:03.916505   706 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:03.890259   623 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.944141   623 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.927561 31825 raft_consensus.cc:2368] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:03.945504 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 127) took 1305 ms. Trace:
I20260709 12:21:03.945680 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.947093 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.947722 31825 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:21:03.948084   817 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:03.948431   847 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:03.948675 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:03.949332 31825 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:03.949031   847 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.918344   619 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:03.950225   861 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:03.950932   619 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.951344 31671 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:03.952268 31820 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:03.935314   730 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.953116   619 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:03.928359   859 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:03.952971 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:03.918735   783 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:03.953910   859 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:03.954087   783 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:03.954952   859 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.956698 32376 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:03.903687   625 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.921438   867 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.920889   771 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:03.929697   568 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:03.935545   670 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:03.980002   670 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.936591 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:03.920229   823 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:03.925323   751 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:03.939132 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:03.939708 31669 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:03.939853 32343 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:03.941485   726 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.000380   823 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.902853   829 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:03.922353   723 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:03.946218 31675 rpcz_store.cc:276] 0709 12:21:02.640046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.765435 (+125389us) service_pool.cc:224] Handling call
0709 12:21:02.765713 (+   278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:03.843283 (+1077570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:03.945139 (+101856us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002b0f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:03.945414 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2336656640}
I20260709 12:21:03.960373   760 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:03.960803   625 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.961230 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:03.961632   706 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.926234 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:03.967144 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:04.007583 31824 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:03.967396 31673 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:03.973102   774 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.009795   877 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:04.010314   877 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:04.010706   877 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:04.011092 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 438) took 1096 ms. Trace:
I20260709 12:21:03.989779 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.987624   751 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.991654   796 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.981381   568 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.025554   856 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:04.006300 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 65960.7us
I20260709 12:21:04.026055   856 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:04.026507   856 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:03.992473   747 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.992918 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:04.013134 32371 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.670s	user 0.012s	sys 0.000s
I20260709 12:21:03.990069 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:03.980944 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:03.990522   653 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.982223   866 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:03.980356   865 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.037334 31819 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:04.037937 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1553) took 1012 ms. Trace:
I20260709 12:21:03.963503   857 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:04.003623 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 67154.8us
I20260709 12:21:03.978060   770 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:03.996248   718 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:03.981079   771 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:04.015825 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 89652.6us
I20260709 12:21:04.023190 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:04.011511   810 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:03.954980   783 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:03.994969 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:04.027891 31818 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:04.013849 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:04.012616 32370 rpcz_store.cc:276] 0709 12:21:02.914441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.952416 (+ 37975us) service_pool.cc:224] Handling call
0709 12:21:04.010231 (+1057815us) spinlock_profiling.cc:243] Waited 5.85 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:04.010460 (+   229us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:04.010685 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:04.010711 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:04.010776 (+    65us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:04.010821 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.010886 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:04.011031 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13459712}
I20260709 12:21:04.034289   723 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.035338 31671 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:04.035763   760 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.026738 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:04.042376 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.044706 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:04.021193   774 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:04.070960 31747 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:04.055104 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60245.6us
I20260709 12:21:04.071705 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:04.038339 31819 rpcz_store.cc:276] 0709 12:21:03.025653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.980591 (+954938us) service_pool.cc:224] Handling call
0709 12:21:04.037860 (+ 57269us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:04.071777 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:04.072551 31747 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:04.072832 31718 leader_election.cc:400] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4.
I20260709 12:21:04.075158 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:04.050282 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:04.075215 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.077533 31747 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:04.077339 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.078270 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:04.078444 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:04.079252 31747 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:04.061600   857 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:04.057452 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:04.056792 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1442) took 1025 ms. Trace:
I20260709 12:21:04.033402 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:04.033646 32371 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:04.084841 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:04.084892 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:04.060225   718 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:04.084583 31824 rpcz_store.cc:276] 0709 12:21:03.031807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.013508 (+981701us) service_pool.cc:224] Handling call
0709 12:21:04.056729 (+ 43221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:04.066550   672 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.080057 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.088518 32375 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.065382   860 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:04.071086   819 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.055953   810 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.093752 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.085810 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:04.062080   747 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:21:04.082841   653 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:04.082993   796 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:04.081449 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:04.125167 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:04.104451 32377 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 295) took 1221 ms (client timeout 1626 ms). Trace:
I20260709 12:21:04.100860 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:04.109486 32375 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:04.049456   862 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:04.127526 32377 rpcz_store.cc:269] 0709 12:21:02.882653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.882823 (+   170us) service_pool.cc:224] Handling call
0709 12:21:04.104254 (+1221431us) spinlock_profiling.cc:243] Waited 926 ms on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:04.104391 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2128865024}
I20260709 12:21:04.109020 31746 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:04.111622 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:04.089556 31675 raft_consensus.cc:1813] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Handling vote request from an unknown peer 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:04.087137   624 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:04.176071 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.177330 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:04.111497 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:04.180945 32372 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:04.181077   862 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:04.092391 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:04.100209 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 73593.9us
I20260709 12:21:04.091086   718 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:04.124768   876 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:04.122327 31669 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:04.125484   879 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.090s	user 0.002s	sys 0.000s
I20260709 12:21:04.125815 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1555) took 1099 ms. Trace:
I20260709 12:21:04.097683 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 299) took 1141 ms. Trace:
I20260709 12:21:04.097939 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:04.126039   796 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:04.073868   774 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.128451 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:04.091863   870 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.186826 31675 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:21:04.187463 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:04.189056 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:04.203284   880 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
I20260709 12:21:04.089936   857 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:04.189426 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:04.203279   718 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.210371 31823 rpcz_store.cc:276] 0709 12:21:03.026282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.687703 (+661421us) service_pool.cc:224] Handling call
0709 12:21:04.125650 (+437947us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b5400379454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:04.125749 (+    99us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":250908416}
I20260709 12:21:04.190919 31825 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
W20260709 12:21:04.186832   884 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
W20260709 12:21:04.214049 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 89664.2us
I20260709 12:21:04.215616   670 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:04.189394 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:04.123412   865 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.104717 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:04.125524   771 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.223965   857 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.224946   774 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.213089   796 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:04.209517 32371 rpcz_store.cc:276] 0709 12:21:02.952817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.971579 (+ 18762us) service_pool.cc:224] Handling call
0709 12:21:04.094225 (+1122646us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:04.227625 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:04.212896 32375 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:04.194205   604 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:04.228960 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:04.206907   751 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:04.231460 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:04.124106   653 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:04.233065 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:04.228013 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:04.233676 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:04.226203   870 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.238098 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:04.218137   568 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.230283 31818 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.281150 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 121) took 1701 ms. Trace:
I20260709 12:21:04.281666 31672 rpcz_store.cc:276] 0709 12:21:02.579840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.585478 (+  5638us) service_pool.cc:224] Handling call
0709 12:21:02.585865 (+   387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:03.705480 (+1119615us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.280816 (+575336us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54002e7a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:04.281071 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2573399424}
I20260709 12:21:04.291244 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:04.292039 31673 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.273718   653 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:04.277827   771 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:04.242240   670 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.272503 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:04.273339 31750 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.276383 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:04.271441   678 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.279170   604 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:04.234372 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1565) took 1039 ms. Trace:
I20260709 12:21:04.103987   819 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.305470 31677 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:04.267941 32343 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:04.300302 31669 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:04.306139   714 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:04.318547 31750 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:04.323609 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:04.324949 31673 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:04.322853 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1439) took 1292 ms. Trace:
I20260709 12:21:04.329031 31827 rpcz_store.cc:276] 0709 12:21:03.030542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.370943 (+340401us) service_pool.cc:224] Handling call
0709 12:21:04.322535 (+951592us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54000fb754. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:04.322772 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":261035392}
I20260709 12:21:04.330219 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:04.331825 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:04.317865 31746 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:04.318904   564 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:04.322315   751 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:04.307317 31825 rpcz_store.cc:276] 0709 12:21:03.191863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.948402 (+756539us) service_pool.cc:224] Handling call
0709 12:21:04.230818 (+282416us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:04.308149   678 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.337045 31669 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:04.327803   860 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:04.327430   714 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:04.343729   881 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.138s	user 0.000s	sys 0.002s
W20260709 12:21:04.335330 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 83716.8us
I20260709 12:21:04.345327   568 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:04.345865 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:04.332705 31827 raft_consensus.cc:2368] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3.
I20260709 12:21:04.350829 31718 leader_election.cc:400] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3.
I20260709 12:21:04.350430 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:04.353500 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:04.360241 31827 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:21:04.337941 31671 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:04.358387 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1550) took 1333 ms. Trace:
I20260709 12:21:04.355953   751 raft_consensus.cc:2764] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20260709 12:21:04.365984 31822 rpcz_store.cc:276] 0709 12:21:03.024722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.026763 (+  2041us) service_pool.cc:224] Handling call
0709 12:21:03.393049 (+366286us) spinlock_profiling.cc:243] Waited 5.36 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:03.393245 (+   196us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:03.406166 (+ 12921us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:03.406201 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:03.406251 (+    50us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:03.406286 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.358153 (+951867us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b54002675d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:04.358206 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:04.358331 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41524864}
I20260709 12:21:04.344200   714 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:04.367560 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:04.372884 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:04.373562 31822 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:04.373684   819 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:04.374084 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1567) took 1181 ms. Trace:
I20260709 12:21:04.348322   860 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:04.374437 31822 rpcz_store.cc:276] 0709 12:21:03.192489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.372509 (+1180020us) service_pool.cc:224] Handling call
0709 12:21:04.374032 (+  1523us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:04.388331 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:04.364784 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:04.338205   604 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.371587   670 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:04.384447   845 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:04.408978 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:04.433236 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1444) took 1400 ms. Trace:
I20260709 12:21:04.433794 31822 rpcz_store.cc:276] 0709 12:21:03.032437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.388491 (+1356054us) service_pool.cc:224] Handling call
0709 12:21:04.433138 (+ 44647us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:04.379074 32343 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:04.407634   827 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.465765   855 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:04.407501   877 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.469161   856 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.469779   877 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.470366 31751 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:04.418883   870 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.473196   653 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.474929   847 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:04.405107   774 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.457022   888 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:04.479003   888 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.480649   653 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.481031 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:04.481405   888 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:04.483279   814 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:04.485962 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:04.486347 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1558) took 1459 ms. Trace:
I20260709 12:21:04.508498 31820 rpcz_store.cc:276] 0709 12:21:03.027239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.618454 (+591215us) service_pool.cc:224] Handling call
0709 12:21:04.485992 (+867538us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b540043dad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:04.486258 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":616138112}
I20260709 12:21:04.509780 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:04.505350 31678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1021) took 1674 ms (client timeout 2037 ms). Trace:
I20260709 12:21:04.482151   774 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:04.407843   564 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:04.449079   850 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:04.517637 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 119896us
I20260709 12:21:04.485932 31751 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
W20260709 12:21:04.510480 31678 rpcz_store.cc:269] 0709 12:21:02.830895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.848452 (+ 17557us) service_pool.cc:224] Handling call
0709 12:21:04.505156 (+1656704us) spinlock_profiling.cc:243] Waited 748 ms on lock 0x7b540036f1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:04.505275 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1721338752}
W20260709 12:21:04.505573 31676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1452) took 2053 ms (client timeout 2230 ms). Trace:
I20260709 12:21:04.518862 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:04.518955 31676 rpcz_store.cc:269] 0709 12:21:02.452272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.452414 (+   142us) service_pool.cc:224] Handling call
0709 12:21:04.505319 (+2052905us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54001815d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:04.505487 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2590493952}
I20260709 12:21:04.406481 31825 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:21:04.509925 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:04.514992 31823 raft_consensus.cc:2410] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:04.456633 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:04.520597 31820 raft_consensus.cc:1813] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Handling vote request from an unknown peer 039d66a6c85e43c3936bf5017501b344
W20260709 12:21:04.520695 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 64187.5us
I20260709 12:21:04.522384 31820 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:21:04.524542 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:04.525192 31820 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:04.525512 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:04.533916 31752 raft_consensus.cc:2368] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:04.535193 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.535990 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:04.510145   799 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:04.546481 31752 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:04.555538 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:04.514875   564 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:04.454437   845 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.530814   877 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:04.569509 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:04.560137 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:04.575028 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 306549us
I20260709 12:21:04.558462 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:04.516060   855 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.041s
W20260709 12:21:04.522106 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 151936us
I20260709 12:21:04.582623 31820 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:04.587262 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:04.527578 31818 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:04.555088 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:04.590272 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:04.594848 32372 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.595060 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:04.585518   564 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:04.596997 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1556) took 1570 ms. Trace:
I20260709 12:21:04.597405 31818 rpcz_store.cc:276] 0709 12:21:03.026588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.596119 (+1569531us) service_pool.cc:224] Handling call
0709 12:21:04.596470 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:04.596651 (+   181us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:04.596682 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:04.596725 (+    43us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:04.596763 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.596823 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:04.596941 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:04.521837   850 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.587772   877 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:04.574759   796 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.599958   877 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:04.600525   850 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:04.600836 31672 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
W20260709 12:21:04.570745 31821 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1436) took 1544 ms (client timeout 1523 ms). Trace:
I20260709 12:21:04.593868 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:04.601637 31821 rpcz_store.cc:269] 0709 12:21:03.016125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.016667 (+   542us) service_pool.cc:224] Handling call
0709 12:21:04.549225 (+1532558us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b54001676d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:04.560727 (+ 11502us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":369980800}
I20260709 12:21:04.602491 31820 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.569684 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:04.569052 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:04.593792 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:04.594962 32369 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:04.569970 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:04.549737   727 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.603741 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.593901 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:04.556457 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:04.555100   706 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:04.582602 31718 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:04.587481 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:04.603003   771 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:04.607774   653 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:04.676294 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:04.677246 31672 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:04.676335   653 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.679560 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:04.680704 31676 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:04.607987   859 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:04.683087 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:04.684800 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:04.662081 32370 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:04.637353 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54857.7us
W20260709 12:21:04.656349 31826 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1432) took 1777 ms (client timeout 2230 ms). Trace:
I20260709 12:21:04.531448   860 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:04.657405 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 67407.8us
I20260709 12:21:04.657688 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 439) took 1707 ms. Trace:
I20260709 12:21:04.636943   751 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:04.631892 31825 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:21:04.665742 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:04.669112 32372 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:04.697871   827 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.698341 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:04.698396   827 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.700655   841 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.701604 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:04.702389 31824 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:04.703109 31821 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:21:04.703954 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1443) took 1671 ms. Trace:
I20260709 12:21:04.704353 31821 rpcz_store.cc:276] 0709 12:21:03.032134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.627017 (+1594883us) service_pool.cc:224] Handling call
0709 12:21:04.681867 (+ 54850us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:04.703612 (+ 21745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.703888 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:04.705952 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:04.647832   888 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:04.706843   706 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:04.628150   829 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:04.708513 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:04.709407 31823 raft_consensus.cc:2368] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4.
I20260709 12:21:04.673235 31746 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:04.711334 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:04.711663   856 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:04.712389   856 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:04.712769   856 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:04.713478 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1563) took 1522 ms. Trace:
I20260709 12:21:04.713680   856 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.645481 31820 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:04.685989   625 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:04.715099   625 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.715555 31820 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:04.689864 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:04.716214 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1566) took 1524 ms. Trace:
I20260709 12:21:04.691438 32378 rpcz_store.cc:276] 0709 12:21:02.950858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.951601 (+   743us) service_pool.cc:224] Handling call
0709 12:21:02.979670 (+ 28069us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:03.717553 (+737883us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.657320 (+939767us) spinlock_profiling.cc:243] Waited 686 ms on lock 0x7b540071d4d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:04.657597 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1577330432}
I20260709 12:21:04.716590 31820 rpcz_store.cc:276] 0709 12:21:03.192171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.715041 (+1522870us) service_pool.cc:224] Handling call
0709 12:21:04.715383 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:04.715960 (+   577us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.716144 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:04.717870 31820 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:04.592519   799 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
W20260709 12:21:04.687728 31826 rpcz_store.cc:269] 0709 12:21:02.879313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.880381 (+  1068us) service_pool.cc:224] Handling call
0709 12:21:04.656030 (+1775649us) spinlock_profiling.cc:243] Waited 587 ms on lock 0x7b54000fad54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:04.656263 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1349709952}
I20260709 12:21:04.692548 32342 leader_election.cc:400] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:04.627266   727 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.693531 31717 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:04.636660   568 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:04.647621 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:04.687769   859 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.735008 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.662668   771 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:04.735718 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:04.735939 31750 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:04.736661   868 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:04.737289   868 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:04.737555 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:04.738324 31671 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
W20260709 12:21:04.679224 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55502.9us
I20260709 12:21:04.727118 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:04.744580 31718 leader_election.cc:400] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4.
I20260709 12:21:04.745049 31718 leader_election.cc:403] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:21:04.694356 31818 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:21:04.734194 32342 leader_election.cc:403] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:21:04.740916 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:04.719501 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1570) took 1169 ms. Trace:
I20260709 12:21:04.684293   823 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:04.745679   885 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:04.748175   870 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4.
I20260709 12:21:04.749089 32370 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:04.687394 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:04.735980   771 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:04.738188 31825 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:04.751621 31820 rpcz_store.cc:276] 0709 12:21:03.548574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.717399 (+1168825us) service_pool.cc:224] Handling call
0709 12:21:04.717695 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:04.718220 (+   525us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.718380 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:04.746974 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:04.754733   751 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:04.714118 31819 rpcz_store.cc:276] 0709 12:21:03.191107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.712462 (+1521355us) service_pool.cc:224] Handling call
0709 12:21:04.712869 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:04.713078 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:04.713107 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:04.713175 (+    68us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:04.713210 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.713281 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:04.713406 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:04.755898 32378 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:04.747227   856 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:04.739320   855 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
W20260709 12:21:04.757021 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 64612.8us
I20260709 12:21:04.750362 31818 pending_rounds.cc:85] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 1
I20260709 12:21:04.758601   747 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:04.720166 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:04.756469 31823 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:21:04.783401 31823 pending_rounds.cc:85] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 1
I20260709 12:21:04.749502   851 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:04.739394   568 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:04.738261   868 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.739404   810 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.777712   819 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:04.749788   760 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260709 12:21:04.764153   845 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:04.811193 31821 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:04.792490 31671 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:21:04.815356 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:04.773765 32343 leader_election.cc:400] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:04.792688 31673 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:04.778012 31818 pending_rounds.cc:107] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:21:04.783787 31823 pending_rounds.cc:107] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:21:04.788815   851 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:04.793186   855 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.795805   727 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:04.790386   751 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260709 12:21:04.839712   855 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:04.840580   751 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:04.841029   751 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:04.842012   855 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:21:04.758525   823 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.843448 31671 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:04.844914   823 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:04.789450   774 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:04.796797 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:21:04.846658 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49951.4us
I20260709 12:21:04.797549   877 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.749858   846 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 12:21:04.800387 31670 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:04.800603   672 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:04.803803   653 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.865868 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:04.866596 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:04.866998 31561 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f6404cb9c2174326b06cbbb0d27d43f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b95aeeaf4c51442b8092cff1de674f2b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 2 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:04.762943   887 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.026s
I20260709 12:21:04.867599   846 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:04.867734   827 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:04.868067   846 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:04.868564   887 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:04.868992   887 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:04.869029   846 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.830394 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:04.832276 31823 raft_consensus.cc:2889] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:04.778362   747 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:04.836834 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:04.870002   887 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.854120   845 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:04.781126   799 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 12:21:04.832702 31818 raft_consensus.cc:2889] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:04.879647 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:04.872006 31677 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:04.872931 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:04.837864   564 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.817091 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:04.917286 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:04.819088   796 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.863225 32375 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:04.877300 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:04.871191 31673 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:04.873273 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.876506 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.876936 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.927453 31751 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:04.903967 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:04.908826 31820 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260709 12:21:04.928550 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.929421 31751 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:04.913216 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:04.916057 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:04.873517   857 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:04.918159 32378 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:04.923444 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:04.911190 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:04.938134 31678 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:21:04.934366 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:04.932096   888 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:04.933777 32372 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:21:04.853648 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 94524us
I20260709 12:21:04.934448 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:04.917928 31669 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:04.956851 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 43742.9us
I20260709 12:21:04.957800 31792 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:04.873232   887 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:04.930521 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:04.959280 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:04.864758   877 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.962384 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.870524   846 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:04.952569 31750 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:04.940492 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:04.950553 31674 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:04.921859   653 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:04.872262   747 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:04.926492   718 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.691s	user 0.012s	sys 0.001s
I20260709 12:21:04.875247   845 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:04.944419   694 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:04.876101   870 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.089s
I20260709 12:21:04.963018 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:04.981834 31673 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
W20260709 12:21:04.873286   885 consensus_queue.cc:1175] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260709 12:21:04.842344   751 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:04.968981 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:04.969352 31753 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:04.958603   814 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.962384   670 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:04.960249 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.976162   796 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:04.938050   727 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:04.934430 32370 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:05.010758 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:05.012303 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:04.965679 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:05.013434 31821 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:04.984395   718 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:04.996949 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:04.967592   827 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:05.000976 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:05.024120 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:04.995900 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:04.998037   796 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:05.007800   727 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:05.012586   564 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:04.995661 31751 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:05.014882 32378 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:04.978715   888 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 12:21:05.014835   855 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:05.022094 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:05.016244   718 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:04.974154 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1557) took 1947 ms. Trace:
I20260709 12:21:04.939440   770 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260709 12:21:04.977705   568 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:21:05.038178 31673 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:04.984995   694 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:05.055583 31752 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:05.038281   653 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:05.038997   796 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:05.042150   829 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:05.038549   827 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:05.103662 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 238498us
I20260709 12:21:05.105051 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:04.992579 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:05.106953 31753 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:05.001832   814 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:05.115511 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:05.122135 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:05.080186   694 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:05.123847   819 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:05.107713 31825 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:05.102176   827 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:05.112711   774 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:05.126260   564 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:05.148983 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:05.109107   653 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:05.160948 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:05.162194 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:05.165344 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1561) took 1986 ms. Trace:
I20260709 12:21:05.169179 31823 rpcz_store.cc:276] 0709 12:21:03.178537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.710001 (+1531464us) service_pool.cc:224] Handling call
0709 12:21:04.710343 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:04.871679 (+161336us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:05.164946 (+293267us) spinlock_profiling.cc:243] Waited 42.8 ms on lock 0x7b5400166cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:05.165258 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98499456}
I20260709 12:21:05.129009   823 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:05.057642 31818 rpcz_store.cc:276] 0709 12:21:03.026917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.598415 (+1571498us) service_pool.cc:224] Handling call
0709 12:21:04.608429 (+ 10014us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:04.937229 (+328800us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.968489 (+ 31260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:05.137960   829 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:05.133572   672 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260709 12:21:05.164534   860 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:21:05.231431 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 190632us
W20260709 12:21:05.231079 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 273370us
W20260709 12:21:05.261811 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 265978us
W20260709 12:21:05.156908 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 183466us
I20260709 12:21:05.137008   877 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:21:05.247644 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 86803.8us
I20260709 12:21:05.221321   770 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:05.225057   855 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:21:05.261335 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:05.263746 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:05.265276 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:05.227895 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1453) took 1168 ms. Trace:
I20260709 12:21:05.137715   845 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:05.272941 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:05.168322   774 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:05.280460 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:05.243407   823 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:05.295828   907 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
W20260709 12:21:05.295828   921 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20260709 12:21:05.295830   910 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:21:05.327014 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:05.286146 31825 rpcz_store.cc:276] 0709 12:21:04.059200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.737696 (+678496us) service_pool.cc:224] Handling call
0709 12:21:04.738055 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:05.156916 (+418861us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:05.224890 (+ 67974us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:05.322875   904 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.185s	user 0.000s	sys 0.001s
I20260709 12:21:05.300074 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:05.348277 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1451) took 1289 ms. Trace:
I20260709 12:21:05.349305 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:05.319278 31670 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:05.351114   920 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:05.339900 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:05.351653   929 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:05.352246   929 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:05.355933   774 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:05.357033 32370 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:05.357189   774 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:05.357867   774 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:05.358783   856 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:21:05.340056   771 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.570s	user 0.012s	sys 0.001s
I20260709 12:21:05.361502 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:05.351699   920 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:05.363308 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:05.296583   924 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:05.341243 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:05.364982 31678 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:05.348738 31824 rpcz_store.cc:276] 0709 12:21:04.058612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.731462 (+672850us) service_pool.cc:224] Handling call
0709 12:21:04.732024 (+   562us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:04.733799 (+  1775us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:04.733834 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:04.747061 (+ 13227us) log.cc:844] Serialized 26 byte log entry
0709 12:21:05.005766 (+258705us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:05.005831 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:05.005954 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:05.348041 (+342087us) raft_consensus.cc:1685] finished
0709 12:21:05.348063 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:05.348214 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1279,"raft.queue_time_us":4468,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":124,"thread_start_us":228390,"threads_started":1,"wal-append.queue_time_us":346704}
I20260709 12:21:05.332470   727 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:05.346664   914 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:21:05.319681   653 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:05.317170   868 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 12:21:05.317292 32343 leader_election.cc:400] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:05.317687   847 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:05.361304   771 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:05.306602   823 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:05.386003 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 123751us
I20260709 12:21:05.389678 31672 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:05.440281 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:05.440963 31672 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:05.385453 31670 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260709 12:21:05.397204 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 124321us
I20260709 12:21:05.358289   774 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:05.369833 32373 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
W20260709 12:21:05.435155 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:05.304404   770 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.066s
W20260709 12:21:05.449540   926 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:21:05.449776   796 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:05.451124 31825 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:05.444625 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:05.326704   819 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:05.390686   888 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.092s
I20260709 12:21:05.364281   829 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:05.480798   916 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.463s
W20260709 12:21:05.456748   919 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:21:05.439637   747 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:05.648046   925 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.070s	user 0.000s	sys 0.001s
I20260709 12:21:05.496992   929 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:05.519465   928 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:21:05.591325   771 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:05.620080   827 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:05.781774   823 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:05.518492 31669 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.562s	user 0.011s	sys 0.000s
I20260709 12:21:05.522271 31678 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:21:05.620361   887 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.652s	user 0.010s	sys 0.001s
I20260709 12:21:05.981556 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1511) took 2129 ms. Trace:
I20260709 12:21:05.981873   930 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:05.983006   930 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:05.983443 31669 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:05.984438 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1050) took 1212 ms. Trace:
I20260709 12:21:05.984860 31669 rpcz_store.cc:276] 0709 12:21:04.771796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.772436 (+   640us) service_pool.cc:224] Handling call
0709 12:21:05.984294 (+1211858us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:05.985956   930 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:05.455999 31671 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:05.459472 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:05.511801   770 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:05.992307 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 547592us
I20260709 12:21:05.993124   770 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:05.994817   604 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:05.995740 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 708384us
I20260709 12:21:05.992597 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 315) took 1457 ms. Trace:
I20260709 12:21:05.997643 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:05.994027 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:05.998948   774 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:05.999502 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 539977us
I20260709 12:21:05.519765 31675 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
W20260709 12:21:06.002864 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 563333us
I20260709 12:21:06.003188   937 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:06.004482   937 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:05.982331 31748 rpcz_store.cc:276] 0709 12:21:03.851428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:03.853277 (+  1849us) service_pool.cc:224] Handling call
0709 12:21:03.853786 (+   509us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:03.865555 (+ 11769us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:03.865591 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:03.866119 (+   528us) log.cc:844] Serialized 26 byte log entry
0709 12:21:04.558433 (+692314us) spinlock_profiling.cc:243] Waited 9.41 ms on lock 0x7b18000df360. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:21:04.828545 (+270112us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:04.828609 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:04.828713 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:05.559840 (+731127us) raft_consensus.cc:1685] finished
0709 12:21:05.559869 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:05.981199 (+421330us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11249,"raft.queue_time_us":11608,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":11877,"spinlock_wait_cycles":21644544,"thread_start_us":269863,"threads_started":1,"wal-append.queue_time_us":1174880}
I20260709 12:21:06.004719 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.005194   937 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:05.965917   796 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:05.457267 31672 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:05.590798   694 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.007442   937 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.009606 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:06.010885   694 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.002328   888 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:06.020788 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.707s	user 0.004s	sys 0.004s
I20260709 12:21:06.027436 31822 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:06.027525   936 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:06.025218 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.005s	sys 0.004s
I20260709 12:21:06.032640 31674 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:06.020212   888 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:06.007514 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:06.012617 31672 raft_consensus.cc:2376] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3. Re-sending same reply.
I20260709 12:21:06.035575   841 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:06.038897   727 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:06.038223 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.043292   625 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:06.045881 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.042012   888 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:06.052451 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55185.1us
I20260709 12:21:06.064642   855 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:06.067572   862 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:06.068133   862 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:21:06.044871 32376 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.658s	user 0.023s	sys 0.018s
I20260709 12:21:06.053426   770 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.053791   946 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:06.074666   862 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.075172   946 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.058501 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:06.076967   862 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.087078 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 452) took 1264 ms. Trace:
I20260709 12:21:06.088258 32369 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
W20260709 12:21:06.070108 32376 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.024s	sys 0.019s
I20260709 12:21:06.087893 32374 rpcz_store.cc:276] 0709 12:21:04.822563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.827861 (+  5298us) service_pool.cc:224] Handling call
0709 12:21:06.047962 (+1220101us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:06.085958 (+ 37996us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:06.086423 (+   465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:06.086462 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:06.086555 (+    93us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:06.086602 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:06.086706 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:06.086876 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29417216}
I20260709 12:21:06.095319 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:06.095877 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 57990.3us
I20260709 12:21:06.097185 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.099625 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:05.998279 32371 rpcz_store.cc:276] 0709 12:21:04.535198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.537892 (+  2694us) service_pool.cc:224] Handling call
0709 12:21:04.623224 (+ 85332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:04.679144 (+ 55920us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:04.684577 (+  5433us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:04.752746 (+ 68169us) log.cc:844] Serialized 26 byte log entry
0709 12:21:05.223645 (+470899us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:05.226809 (+  3164us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:05.233812 (+  7003us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:05.650180 (+416368us) raft_consensus.cc:1685] finished
0709 12:21:05.650230 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:05.992310 (+342080us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12059,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":11767,"thread_start_us":385508,"threads_started":1,"wal-append.queue_time_us":879634}
I20260709 12:21:06.072448 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:06.104681 31818 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:06.055058 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:06.090734 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:06.058297   913 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.824s	user 0.017s	sys 0.009s
I20260709 12:21:06.100443   727 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:06.101107   885 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:06.102258 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 97859.7us
I20260709 12:21:06.071462   859 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.102159 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:06.105494 31672 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:06.104878 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:06.106211 32375 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:06.107939   946 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:05.511945   920 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.111506   770 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:06.111862   747 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.113595 31669 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:06.114449 31672 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:06.113670 31673 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:06.118007 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.111510   705 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.119062 31818 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:06.119468   885 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.120136 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.122494   727 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:06.119532   705 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.128352 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.129559 31818 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:06.130155 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.134778 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:06.135651   705 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.126276   827 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.137480 32375 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:06.138216   841 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.139051   823 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.140077 31745 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:06.122093   885 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.155380 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:06.156924 31818 raft_consensus.cc:2368] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:06.151638   920 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.157948 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.172518   568 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.169193   862 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:06.173437   568 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.174659 31825 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:06.175011   678 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:06.164856   705 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.176157   568 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.176276   705 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.164736   841 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:06.180265 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 80899.7us
I20260709 12:21:06.181059   841 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.181562 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.184439 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:06.184749 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:06.185384 32370 raft_consensus.cc:1813] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Handling vote request from an unknown peer b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.186035 32370 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:06.185923 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:06.187036 31818 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:21:06.188084 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 53555.4us
I20260709 12:21:06.189105 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:06.189405   823 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:06.190038   751 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.190572   705 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.190212   823 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.194444   862 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:06.195994   862 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:06.196393 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1446) took 2139 ms. Trace:
I20260709 12:21:06.196552   862 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:06.196069 31561 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f6404cb9c2174326b06cbbb0d27d43f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b95aeeaf4c51442b8092cff1de674f2b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:06.196890 31826 rpcz_store.cc:276] 0709 12:21:04.057232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.719088 (+661856us) service_pool.cc:224] Handling call
0709 12:21:06.193220 (+1474132us) spinlock_profiling.cc:243] Waited 3.69 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:06.195424 (+  2204us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:06.195803 (+   379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:06.195837 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:06.195887 (+    50us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:06.195922 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:06.195980 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:06.196122 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8496896}
I20260709 12:21:06.197604   827 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:06.193084   678 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:06.202085   814 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:06.202517 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.205432 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:06.202847 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.207757 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:06.199604   851 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.210392   851 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.204582   823 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.215163 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.221057 31823 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:06.221261 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.222258 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:06.223824 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:06.215230 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.224723   939 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.224866 31821 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:06.225400   939 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.222982 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.216717   855 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:06.236182 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:06.239261 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1049) took 1485 ms. Trace:
I20260709 12:21:06.239699 31675 rpcz_store.cc:276] 0709 12:21:04.754285 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.754773 (+   488us) service_pool.cc:224] Handling call
0709 12:21:04.755185 (+   412us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:06.003246 (+1248061us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:06.238779 (+235533us) spinlock_profiling.cc:243] Waited 727 ms on lock 0x7b5400046ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:06.239062 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1671782272}
I20260709 12:21:06.219326   920 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.230682   841 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:06.243745   841 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:06.244151 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:06.244891 32342 connection.cc:256] RPC call timeout handler was delayed by -1.075461321000148s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
I20260709 12:21:06.236934   938 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:21:06.245837 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64519.2us
I20260709 12:21:06.247560 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:06.247656 31819 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:06.230250   568 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:06.248384 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1469) took 1365 ms. Trace:
I20260709 12:21:06.248528   723 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:06.249019   723 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:06.248831 31819 rpcz_store.cc:276] 0709 12:21:04.882658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.909693 (+ 27035us) service_pool.cc:224] Handling call
0709 12:21:06.246644 (+1336951us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:06.247043 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:06.248087 (+  1044us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:06.248327 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28635520}
I20260709 12:21:06.249601   723 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:06.249888 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.245719   939 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.244189   841 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:06.251412   841 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:06.253064   851 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.253590   851 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.254127   908 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:06.254598   908 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:06.236491 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:06.260807 31823 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:06.261621   908 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.262240   799 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:06.263821 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:06.242575   920 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.278436   908 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:06.278836   946 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:06.279608   946 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:06.281018 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:06.280915   946 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.281807 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:06.282606 31643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.909s (SENT) [suppressed 8 similar messages]
I20260709 12:21:06.283124 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:06.284696 31670 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:06.285245 31678 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:06.267598   823 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:06.269999 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.270991 31826 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:21:06.265456 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1044) took 1561 ms. Trace:
I20260709 12:21:06.274340 31669 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:06.274039   888 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:06.274871 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 118424us
I20260709 12:21:06.276206 31819 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:06.291069 31827 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:06.293807 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:06.294246 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:06.295086 32376 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:06.295795 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1529) took 1364 ms. Trace:
I20260709 12:21:06.295872   823 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.296384 31745 rpcz_store.cc:276] 0709 12:21:04.931752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.934122 (+  2370us) service_pool.cc:224] Handling call
0709 12:21:06.295524 (+1361402us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002963d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:06.295682 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2342578944}
I20260709 12:21:06.276780 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.278870   799 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:06.303712   799 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:06.304256   799 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.271871 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.287772 31826 pending_rounds.cc:85] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 1
I20260709 12:21:06.291195   920 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:06.291335 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.301924   885 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:06.310738   568 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:21:06.311179 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 63757.4us
I20260709 12:21:06.311434   885 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.312029   568 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:06.312546   568 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:06.313336   855 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:06.314136   885 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.302727   916 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.314759   861 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:06.315467 31821 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:06.315186   916 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.316720 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.317811   916 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.307811 31826 pending_rounds.cc:107] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:21:06.319519 31826 raft_consensus.cc:2889] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:06.321489 31826 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:06.303031 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:06.313704   568 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.315326   861 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:06.318257   855 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:06.310298   920 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.295207 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.288146 31676 rpcz_store.cc:276] 0709 12:21:04.703515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.703925 (+   410us) service_pool.cc:224] Handling call
0709 12:21:06.201104 (+1497179us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:06.203853 (+  2749us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:06.204198 (+   345us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:06.204227 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:06.204911 (+   684us) log.cc:844] Serialized 26 byte log entry
0709 12:21:06.229862 (+ 24951us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:06.229910 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:06.229991 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:06.264943 (+ 34952us) raft_consensus.cc:1685] finished
0709 12:21:06.264966 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:06.265239 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":36688256,"thread_start_us":24442,"threads_started":1,"wal-append.queue_time_us":57606}
I20260709 12:21:06.330555 31671 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:06.323372   827 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.343195 31826 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:21:06.356297 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 133) took 1090 ms. Trace:
I20260709 12:21:06.357064 31823 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:06.332289   851 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.339107 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:06.318326 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:06.368827 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63043.8us
I20260709 12:21:06.356822 31674 rpcz_store.cc:276] 0709 12:21:05.266755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:05.266925 (+   170us) service_pool.cc:224] Handling call
0709 12:21:05.282491 (+ 15566us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:06.035483 (+752992us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:06.341159 (+305676us) spinlock_profiling.cc:243] Waited 742 ms on lock 0x7b54000479d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:06.356171 (+ 15012us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1706472320}
I20260709 12:21:06.321427   723 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.343257   568 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:06.384718 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 87068.9us
I20260709 12:21:06.375659   827 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.381449   888 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:06.405138 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 85907.6us
I20260709 12:21:06.404168 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:06.384850 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1519) took 1516 ms. Trace:
W20260709 12:21:06.383355 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1461) took 1804 ms (client timeout 2012 ms). Trace:
I20260709 12:21:06.341609   862 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.391597 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:06.414695   823 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.416296 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:06.416489   862 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.383857 31752 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:06.417850 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:06.418332 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:06.389820 32373 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:06.419224 31752 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:06.419703 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.420518 31821 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:06.388384 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:06.422382 32377 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 453) took 1599 ms (client timeout 2046 ms). Trace:
W20260709 12:21:06.422866 32377 rpcz_store.cc:269] 0709 12:21:04.822979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.823699 (+   720us) service_pool.cc:224] Handling call
0709 12:21:06.373932 (+1550233us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:06.422275 (+ 48343us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27738496}
I20260709 12:21:06.392977   920 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.385604 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:06.394894   799 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.425599 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:06.382202 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:06.386585 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.390949 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:06.386072 31748 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:06.390704   705 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260709 12:21:06.392904 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:06.388018 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:06.396449 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.395182 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:06.394507 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:06.420519 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:06.386510 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:06.436887 31823 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:21:06.426362 31746 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.434578 31749 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
W20260709 12:21:06.410460 31822 rpcz_store.cc:269] 0709 12:21:04.578725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.594478 (+ 15753us) service_pool.cc:224] Handling call
0709 12:21:04.872780 (+278302us) spinlock_profiling.cc:243] Waited 1.48 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:06.383063 (+1510283us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400446bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:06.383230 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2492766336}
I20260709 12:21:06.410104 31750 rpcz_store.cc:276] 0709 12:21:04.868473 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.877030 (+  8557us) service_pool.cc:224] Handling call
0709 12:21:06.384355 (+1507325us) spinlock_profiling.cc:243] Waited 25.8 ms on lock 0x7b5400295ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:06.384618 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":59430784}
I20260709 12:21:06.441813 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.443337 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:06.437844 31670 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:06.426395   799 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:06.447109 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 107049us
I20260709 12:21:06.438174   862 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:06.458148 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.459741 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:06.459606   841 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.461567 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:06.462071 31751 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:21:06.446873 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:06.453930 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:06.462235   841 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.451395   888 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:06.453950   955 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:06.451406 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 69491.9us
I20260709 12:21:06.467284 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 134) took 1200 ms. Trace:
I20260709 12:21:06.470176 31748 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:06.473712   920 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.474411   920 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.474778   718 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:06.475306   718 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:06.475728   718 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.476876   920 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.477532   920 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:06.479988 31750 raft_consensus.cc:2410] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:06.480088 31572 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 233) took 9873 ms. Trace:
W20260709 12:21:06.483457   918 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.864s	user 0.009s	sys 0.005s
I20260709 12:21:06.471052 31677 rpcz_store.cc:276] 0709 12:21:05.267295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:05.267426 (+   131us) service_pool.cc:224] Handling call
0709 12:21:06.359722 (+1092296us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:06.467198 (+107476us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6199040}
I20260709 12:21:06.486443   859 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:06.487783 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.395862   861 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.489974   861 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.490911   705 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:06.486159 31825 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:21:06.491144 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1141) took 1819 ms. Trace:
I20260709 12:21:06.458374   862 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.491609 31747 rpcz_store.cc:276] 0709 12:21:04.672009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:04.673583 (+  1574us) service_pool.cc:224] Handling call
0709 12:21:04.673883 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:04.674115 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:04.674139 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:04.876535 (+202396us) spinlock_profiling.cc:243] Waited 10.3 ms on lock 0x7b2c00001158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:21:04.876948 (+   413us) log.cc:844] Serialized 26 byte log entry
0709 12:21:05.154796 (+277848us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:05.154831 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:05.154903 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:06.440860 (+1285957us) spinlock_profiling.cc:243] Waited 2.05 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:06.490728 (+ 49868us) raft_consensus.cc:1685] finished
0709 12:21:06.490752 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:06.490953 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28520064,"thread_start_us":277441,"threads_started":1,"wal-append.queue_time_us":741605}
I20260709 12:21:06.493251   862 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.490303 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:06.448733   823 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:06.472990 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:06.494442 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:06.449375   827 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.500653   564 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:06.506973 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 70249.2us
I20260709 12:21:06.496250   705 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:06.507064 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45630.1us
I20260709 12:21:06.497447   955 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.508939 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:06.510043   705 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:06.509927   888 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.510736   705 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:06.511291   705 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:06.511622 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:06.511745   564 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:06.512496   888 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.496732   823 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:06.513620   823 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.514274   814 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:06.515612   814 consensus_queue.cc:1243] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:21:06.516285   814 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:06.516698   814 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:06.517784   814 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.495585   946 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:06.513950   564 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:06.519207   564 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:06.519706   564 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:06.520133   766 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:06.520848 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:06.522287   939 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:06.522473   885 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:06.523082   885 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:06.523308   814 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.523797 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:06.524138   885 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.514485   678 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
I20260709 12:21:06.528221 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:06.529268 31576 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 3 to 4, term changed from 1 to 2, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to <none>, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New cstate: current_term: 2 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:06.531229   916 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:06.531665   916 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:06.531553   771 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:06.532310   916 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.532696   771 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:06.531011   939 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:06.510182   955 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:06.489001 31821 raft_consensus.cc:2368] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3.
W20260709 12:21:06.535818 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 66336.9us
I20260709 12:21:06.515798 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:06.522293 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 68529.7us
I20260709 12:21:06.514437 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.533318   771 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:06.531064   819 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260709 12:21:06.547516 31576 catalog_manager.cc:5697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 1 to 2, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:06.549852   956 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:06.550127   868 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:06.550426   956 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:06.550746   868 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:06.550977   956 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:06.551872   868 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:06.553002 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.553562 31746 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:06.554448 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:06.538820 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.556466 31822 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:06.549597 31576 catalog_manager.cc:5697] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 1 to 2, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:21:06.565868 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 329497us
I20260709 12:21:06.555347 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:06.572635 31576 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 2 to 3, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 3 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:06.608505 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.572054   678 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:06.608496 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:06.569204   568 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:06.632501 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 102162us
W20260709 12:21:06.631762 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 57388us
W20260709 12:21:06.635592 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.608397   932 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:21:06.643469 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 127804us
I20260709 12:21:06.645710 32343 leader_election.cc:400] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3.
I20260709 12:21:06.650388   859 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:06.568667   718 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:06.652201 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.652963   718 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.654932   751 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:06.656123   799 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:06.656541   799 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:06.636214 31576 catalog_manager.cc:5697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:06.657698 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:06.657500   799 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.658424 31673 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.659041 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:06.658576 31576 catalog_manager.cc:5697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 1 to 2, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 2 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:06.654100   961 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:06.660897 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:06.661518   961 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:06.660776 31576 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 2 to 3, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 3 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:06.661875   961 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:06.662364   819 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:06.659063 31824 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260709 12:21:06.663241 31824 pending_rounds.cc:85] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 4
I20260709 12:21:06.663453 31824 pending_rounds.cc:107] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260709 12:21:06.663664 31824 raft_consensus.cc:2889] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:06.664114 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.662887   819 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:06.664988 31823 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:06.665797   819 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.666531 31826 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260709 12:21:06.639078   723 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:06.666939 31826 pending_rounds.cc:85] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 1
I20260709 12:21:06.667261 31826 pending_rounds.cc:107] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:21:06.667511 31826 raft_consensus.cc:2889] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:06.667717 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 96609.4us
I20260709 12:21:06.668695 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:06.669257   799 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.671850   568 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.670763   961 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.667826   819 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:06.480599 31572 rpcz_store.cc:276] 0709 12:20:56.609654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.626186 (+ 16532us) service_pool.cc:224] Handling call
0709 12:21:06.479848 (+9853662us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:03.528509 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:03.650566 (+122057us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:03.650598 (+    32us) write_op.cc:435] Acquired schema lock
0709 12:21:03.654253 (+  3655us) tablet.cc:662] Decoding operations
0709 12:21:03.705698 (+ 51445us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:03.713642 (+  7944us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:03.718225 (+  4583us) tablet.cc:685] Acquiring locks for 1 operations
0709 12:21:03.738206 (+ 19981us) tablet.cc:701] Row locks acquired
0709 12:21:03.744129 (+  5923us) write_op.cc:260] PREPARE: finished
0709 12:21:03.756624 (+ 12495us) write_op.cc:270] Start()
0709 12:21:03.763683 (+  7059us) write_op.cc:276] Timestamp: P: 1783599663755150 usec, L: 0
0709 12:21:03.766066 (+  2383us) op_driver.cc:348] REPLICATION: starting
0709 12:21:03.841734 (+ 75668us) log.cc:844] Serialized 544 byte log entry
0709 12:21:04.468073 (+626339us) op_driver.cc:464] REPLICATION: finished
0709 12:21:04.973921 (+505848us) write_op.cc:301] APPLY: starting
0709 12:21:04.980634 (+  6713us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:04.987752 (+  7118us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:05.007506 (+ 19754us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:05.007797 (+   291us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:05.007977 (+   180us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0709 12:21:05.008027 (+    50us) write_op.cc:312] APPLY: finished
0709 12:21:05.008253 (+   226us) log.cc:844] Serialized 23 byte log entry
0709 12:21:05.008441 (+   188us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:05.008553 (+   112us) write_op.cc:454] Released schema lock
0709 12:21:05.008595 (+    42us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:05.008946 (+   351us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:05.009013 (+    67us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1398696,"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":483512,"apply.run_cpu_time_us":2358,"apply.run_wall_time_us":44506,"num_ops":1,"prepare.queue_time_us":659536,"prepare.run_cpu_time_us":34405,"prepare.run_wall_time_us":419629,"raft.queue_time_us":239688,"raft.run_cpu_time_us":26999,"raft.run_wall_time_us":301669,"replication_time_us":701828,"thread_start_us":1222548,"threads_started":4,"wal-append.queue_time_us":143800,"wal-append.run_cpu_time_us":24778,"wal-append.run_wall_time_us":2111267}]]}
I20260709 12:21:06.679289   564 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.637132   868 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:06.683194 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 168879us
I20260709 12:21:06.636893   939 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:06.683320   961 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:06.720137 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:06.723186 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:06.728037   678 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:21:06.728271   958 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:21:06.728812   678 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:06.731961 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:06.732897   932 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:06.733213 31643 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:06.734886   814 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:06.737011   814 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:06.737358   814 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:06.739152 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.740510 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.741554 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:06.741958 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.742678 31818 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:06.734571   932 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:06.749032 32375 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:06.751942 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 143125us
W20260709 12:21:06.791126 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57979.4us
I20260709 12:21:06.792560 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:06.674263   568 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.729916   678 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.706238   766 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:06.796057 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 127504us
I20260709 12:21:06.718319   965 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.791435   932 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:06.805474 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.718667   823 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:06.818512 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 185984us
I20260709 12:21:06.820281 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.796547   939 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:06.808841   859 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:06.821805   946 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260709 12:21:06.830663 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:06.831404 31748 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:06.831245   939 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.797911   908 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.812592   568 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:06.830665 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.832108   916 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.831650 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:06.832412   965 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.816828   932 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:06.738479   814 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.796351   694 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 12:21:06.834146   859 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:06.827749 32376 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:21:06.817783 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:06.822220 32373 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:21:06.828435 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:06.865250 31643 leader_election.cc:400] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3.
I20260709 12:21:06.865645 31643 leader_election.cc:403] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:06.866438 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:06.869611 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:06.849720 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.848093   568 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:06.843271   604 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260709 12:21:06.873612   706 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.877333   916 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:06.822187   771 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.854485   723 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:06.863968   859 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:06.879019 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:06.880795   564 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:06.853005   955 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:21:06.881642 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51179.4us
I20260709 12:21:06.915302 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:06.886871   841 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:06.858507 32371 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:06.886221 32375 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:06.885028 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:06.894680 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:06.861440   930 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:06.842551   938 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 12:21:06.870849 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:06.908491 31750 raft_consensus.cc:2410] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:21:06.908684 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.909529 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:06.887516   568 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:06.883536 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:06.950304 31824 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:06.903388   814 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:06.888018   718 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260709 12:21:06.903153 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:06.885385   705 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:06.986225 31673 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:06.957158 31747 raft_consensus.cc:2410] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:21:06.905869   706 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.959676 31822 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:06.964614 31827 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
W20260709 12:21:07.003532 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 145423us
I20260709 12:21:06.967931 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:07.025738 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:06.970046   963 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.090s	user 0.000s	sys 0.001s
I20260709 12:21:06.965441   974 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.048349   974 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.965642   973 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:06.966646   962 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:21:07.052153 31677 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:06.925055   956 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.053659 32378 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:21:07.035049   931 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3.
I20260709 12:21:07.055135   965 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.974932   955 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:07.055846 32377 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:07.056028   974 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.056342   955 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.008514   766 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.059044   766 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.045259   819 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.016s
I20260709 12:21:07.061167   604 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.061676   819 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:07.062091   819 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:07.061937   604 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.063205   819 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.064909   819 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:07.065248   604 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:07.066121   604 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:06.983016 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:07.045323 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:07.045369   972 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:07.041385   885 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 12:21:06.964370   930 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:06.952461   723 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:06.949610 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:06.959525 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:06.979300   823 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:06.990876 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:07.046689   771 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.067773 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:07.045490 31819 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:07.070514   706 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.044116 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:07.071970 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 104173us
W20260709 12:21:07.091467 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:06.969817   966 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:07.045189   705 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:07.058480   955 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.047220   979 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:07.085327 31821 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:07.108098 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:07.120721 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:07.076606   972 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:07.151994 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47011.4us
I20260709 12:21:07.158434 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:07.087637 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:07.168233 31748 raft_consensus.cc:2368] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:07.173440 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.177529 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.181986 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.181970 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:07.094280   965 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:07.174778 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:07.169453 31678 raft_consensus.cc:2376] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3. Re-sending same reply.
I20260709 12:21:07.092773   974 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.062s
I20260709 12:21:07.189824 31750 raft_consensus.cc:2410] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:07.104568 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:07.194463 31676 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:07.154886   706 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.041s
I20260709 12:21:07.088105   604 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.176714 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:07.158584   972 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:07.196646 32376 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:21:07.215266 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 287579us
I20260709 12:21:07.210932   771 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.226837 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:07.227705 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:07.154130   855 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:21:07.168030   861 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
W20260709 12:21:07.231359   868 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.530s	user 0.013s	sys 0.000s
W20260709 12:21:07.229333 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:07.231225 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49352.2us
I20260709 12:21:07.231796 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:07.214571   705 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.238448 32370 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:07.224072 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:07.248348   799 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.579s	user 0.010s	sys 0.000s
W20260709 12:21:07.232822 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 148045us
I20260709 12:21:07.249578 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:07.251153 31642 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:07.184957   966 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.263288   956 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.264623   966 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.265326   939 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:21:07.228430 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 363293us
I20260709 12:21:07.225077   862 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:07.233282 31670 raft_consensus.cc:2410] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:07.225106 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:07.247082 31753 raft_consensus.cc:2410] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:07.247417 31675 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:07.248769 31745 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:07.235134 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:07.235076 31746 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:21:07.271019 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:07.287874 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.288959   930 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:07.290846   975 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:21:07.287963   862 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.254823   751 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.291604   862 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.292030   751 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.293967   862 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:07.292030   950 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:07.294791   950 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:07.237202   766 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:07.295181   950 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:07.295555 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:07.290900 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 65886.4us
I20260709 12:21:07.296953 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:07.297690 31674 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:07.238775   987 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:07.317138 31674 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:21:07.237488   979 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.063s
I20260709 12:21:07.329782 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:07.330112 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 59173.1us
I20260709 12:21:07.331290 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:07.239038   771 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.287873 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:07.241129   705 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.337399 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:07.338429 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:07.339530 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 86265.3us
I20260709 12:21:07.352047 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:07.336720   694 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:07.353561 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:07.357833   723 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:07.358068   706 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:07.358507   723 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:07.342459   938 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:07.367051 31673 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:07.372105 32374 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:07.375432 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:07.376420 32375 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:07.343021   987 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:07.380388 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:07.349848   908 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:07.359556   723 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:07.385618 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:07.387426 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:07.390108 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:07.359580   706 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:07.418465 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 117958us
I20260709 12:21:07.378341   799 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.366739   974 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:07.378320   855 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:07.388110 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:07.399444   983 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:21:07.357911   993 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:07.384968 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:07.423121 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:07.456544 31826 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:07.433923 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:07.437362 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:07.457813 31823 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:07.441604 31749 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:07.442548 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:07.443493   771 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.447026 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:07.447602   799 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.429518   855 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:21:07.450925   859 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.529s	user 0.011s	sys 0.000s
I20260709 12:21:07.388419   694 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:07.396742   823 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:07.452317 31675 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:07.451871   965 raft_consensus.cc:2955] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:07.465170   950 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:07.470248   993 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:07.454831 32370 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:07.455366   980 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:07.455683 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:07.477421 31678 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:21:07.471833 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:07.447099   961 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.724s	user 0.006s	sys 0.006s
I20260709 12:21:07.479132   985 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.091s
I20260709 12:21:07.489952   604 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:07.478909   993 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.451212 31821 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:07.475102   694 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:07.477625   950 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:21:07.436196   939 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:07.418562   966 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:07.442602   938 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.057s
I20260709 12:21:07.498867 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:07.497460 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 107372us
I20260709 12:21:07.499130 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.502054 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:07.503080 32373 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260709 12:21:07.503633 31746 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:07.479893   859 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.498920 31825 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:07.505962   920 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:07.506909   987 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.517416 31750 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:07.491955 32374 raft_consensus.cc:2955] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:07.492895 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:07.519060 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.506392   799 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:07.523581 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:07.427292   989 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.081s
I20260709 12:21:07.518860   966 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:07.547868 31820 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:07.549701 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:07.510300   694 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:07.520051   961 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:07.567538   705 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:07.567412   961 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.570806   799 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:07.570951   966 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:07.571389   799 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.573652   799 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.546883 32373 pending_rounds.cc:85] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 4
I20260709 12:21:07.549614 32371 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:07.560617 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:07.563766 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:07.564659 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:07.570322   961 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.487391   855 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.550715   920 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:07.579396 31675 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:21:07.587486 31675 raft_consensus.cc:606] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:07.587947 31675 raft_consensus.cc:614] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:07.592483 32377 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:07.593539 32371 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:21:07.590137   920 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.596024 31744 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:07.596237   920 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:07.597203   859 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:07.599639   568 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.593s	user 0.011s	sys 0.000s
I20260709 12:21:07.600090   568 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.600786 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:07.601522 31819 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:07.601788   966 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.602608 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:07.604545   991 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:07.604948   991 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:07.577137 32373 pending_rounds.cc:107] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 12:21:07.605605 32373 raft_consensus.cc:2889] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:07.583128   705 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.606355   705 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.606179   991 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.608757   705 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.608829   991 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:07.610345 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:07.610777   751 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:07.611970 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:07.612691 31825 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
W20260709 12:21:07.613436   819 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.548s	user 0.014s	sys 0.001s
I20260709 12:21:07.613704 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:07.615657 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:07.616405 31821 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:07.585036  1010 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:07.585944   994 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:07.605079 31819 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:07.619452   994 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:07.619504  1010 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:07.620416   994 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.604837   972 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.593581 31826 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:21:07.575323   932 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.704s	user 0.005s	sys 0.006s
I20260709 12:21:07.628978 31820 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:07.602319   568 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.620766  1010 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.622467 31825 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 2: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:07.657963   568 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.658702   568 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:21:07.661192   568 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.666988 31745 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 4. Ops: []
I20260709 12:21:07.667862  1005 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:07.669580   930 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:07.670053   930 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:07.670677   972 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:07.671018   930 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.672963   930 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:07.618135 31821 raft_consensus.cc:2955] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:07.642369   994 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:07.691485 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:07.692327 32371 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:07.643158 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:07.650898   855 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:07.696887 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:07.698330 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:07.639943   932 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.711688 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.712193 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.712623 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.714146 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.714836 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:07.718735 31752 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 2: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:07.721630 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:07.722460 31753 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:07.722596 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:07.718293   823 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:07.724411   823 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:07.724805   823 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:07.700117   972 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.732065 32377 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:07.732671 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:07.737082 31671 raft_consensus.cc:2368] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
W20260709 12:21:07.738448 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:07.739228 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:07.741751 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:07.708798 32370 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:07.735028 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:21:07.748072 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:07.698887   705 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 12:21:07.749413 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:07.750347 31791 leader_election.cc:400] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:07.750669 31791 leader_election.cc:403] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260709 12:21:07.751600 31820 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
mode: GRACEFUL
new_leader_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:41934
I20260709 12:21:07.694584   931 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 12:21:07.729641   751 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:07.730073 31744 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:07.745275   961 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:07.750746   814 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:07.746735 31561 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f6404cb9c2174326b06cbbb0d27d43f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b95aeeaf4c51442b8092cff1de674f2b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:07.757611   930 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260709 12:21:07.753887 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:07.746937   771 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:07.733112   694 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.774098 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:07.758265   862 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:07.734058   972 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.762082   568 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:07.770613   723 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 12:21:07.778775   771 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.782441   998 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:07.769507   814 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:07.785874   950 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.792095 31751 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:07.791831   950 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.792725   694 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.793259 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:07.793489   771 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.794786   989 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.789354   814 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:07.795436   989 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.798555   568 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:07.799717   568 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:07.773284  1002 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:07.775885   973 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:07.794509 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:07.813197 31820 raft_consensus.cc:606] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Received request to transfer leadership to TS 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:07.813889 31820 raft_consensus.cc:614] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:21:07.814402 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:07.811764   973 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.816740 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:07.831112 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:07.832085 31749 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:07.834583 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:07.843350  1013 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.272s	user 0.003s	sys 0.000s
I20260709 12:21:07.843935 31672 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:07.843034 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.820391 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.839169   973 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:07.849509 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:07.865393 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 119915us
I20260709 12:21:07.869822 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:07.867933 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 135319us
I20260709 12:21:07.838991   946 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:07.900379 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.905958 31749 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:07.907089 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.910797 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.912357 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:07.912654  1012 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:21:07.913713 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.900050   946 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:07.918954 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:07.919709 31827 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:07.914431 31753 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:07.908522   965 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:45150 (request call id 74) took 11768 ms. Trace:
I20260709 12:21:07.924319 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:07.927223 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:07.876777   931 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:07.931949 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:07.933135 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:07.923924   965 rpcz_store.cc:276] 0709 12:20:56.109251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:20:56.109394 (+   143us) service_pool.cc:224] Handling call
0709 12:20:56.184492 (+ 75098us) log.cc:844] Serialized 508 byte log entry
0709 12:20:56.308704 (+124212us) spinlock_profiling.cc:243] Waited 2.39 ms on lock 0x7b5400350820. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd55b26 00007fe8fcd4f49a 00007fe8fcd4e714 00007fe8fcd2eeaa 00007fe8fcd35d53 00007fe8fcd35cd9 00007fe8fcd35c69 00007fe8fcd35c31 00007fe8fcd34dfd 0000000000388634
0709 12:21:07.863227 (+11554523us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12982,"raft.run_cpu_time_us":3881,"raft.run_wall_time_us":64974,"spinlock_wait_cycles":5528192,"thread_start_us":57066,"threads_started":1,"wal-append.queue_time_us":67990,"wal-append.run_cpu_time_us":2248,"wal-append.run_wall_time_us":1069050}
I20260709 12:21:07.934243   908 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:07.863034 31819 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:07.938234 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:07.940620 31827 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:07.941952 31827 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:07.943574 31827 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:07.931481 31744 raft_consensus.cc:2410] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:07.947255  1029 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:21:07.947532 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:07.949858 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:07.953457 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:07.967162 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:07.971306 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:07.971570 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:07.972010  1015 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:07.972149 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 2c162dbe81714319bfeccd9d7cb7bfea with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:21:07.972491  1015 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:07.972954  1015 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:07.973127 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:07.974488 31744 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:07.974615 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:07.940465   799 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:21:07.981794 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:07.978824   604 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:07.981281 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:07.980405 31827 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260709 12:21:07.971164 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:07.939889   694 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:07.988075 31676 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:07.987006 31678 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:07.993788 31827 pending_rounds.cc:85] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 4
I20260709 12:21:07.996420 31827 pending_rounds.cc:107] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 12:21:07.996851 31827 raft_consensus.cc:2889] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:07.990074 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:07.947261   931 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:08.003191 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.006210 31672 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:07.980154   908 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:07.986773 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 212714us
I20260709 12:21:08.051455 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:07.991209 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.057868 31749 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:07.991153   961 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:08.060765 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:07.992376 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:07.998777 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:07.992602   920 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:08.073839   564 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:08.094373   973 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:08.093755 31749 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:07.989391   814 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:08.053200 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:08.055539 31669 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:08.063373   946 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:08.063787   961 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:08.062386   994 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:08.120366 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:08.116673   908 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:08.153419 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:07.990999   997 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:08.062425   823 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:08.159747   961 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:08.104916 31746 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:08.160463 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:08.162515 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 73808us
I20260709 12:21:08.135418 31825 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:08.174255 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 207163us
I20260709 12:21:08.166317 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.166952 31672 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:08.169806 31751 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:08.104610   920 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.161079  1001 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:08.120183   973 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.152159   994 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.184092   997 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:08.164470   908 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.232111 31676 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:08.204200  1035 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:08.216640  1001 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:08.236156 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:08.236845 31675 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:08.236826  1001 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:08.239492  1025 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260709 12:21:08.217777   961 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:08.243242 31674 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:08.246284 31675 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:08.259367  1033 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:08.260035  1033 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:08.263010   997 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:08.265421   932 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:08.268476   823 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:08.228809 31670 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:08.292760   973 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:08.293200   855 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.582s	user 0.009s	sys 0.003s
I20260709 12:21:08.293439 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.293341  1035 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.296295   994 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:08.297586 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:08.169747   931 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:08.303169 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:08.303905 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:08.391763 32378 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:08.354084 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:08.352730 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.298321 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.395283 31677 raft_consensus.cc:1813] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Handling vote request from an unknown peer 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:08.396180 31677 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 3: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:08.396612 31823 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:08.262503  1033 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:08.245960   564 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:08.352962 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:08.410526 31672 raft_consensus.cc:2368] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4.
I20260709 12:21:08.411283 31676 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:08.413141 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.273818 31819 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:21:08.248543   956 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:08.247267  1015 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.307734   814 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:08.306595   920 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:08.460916 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:08.352716 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.393846 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:08.469542 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.466104 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.409863 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:08.481508 31751 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:08.483224 31675 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:08.480790 31672 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:08.459928   987 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:08.471591   994 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:08.493893 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:08.492676 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 168467us
I20260709 12:21:08.491539 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:08.425091   564 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:08.487236   993 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.917s	user 0.018s	sys 0.028s
I20260709 12:21:08.420862 31669 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:08.487015   908 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:08.511632 31751 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:08.470070   956 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:08.604287 32378 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
W20260709 12:21:08.590966 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:08.562160 31672 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:08.606707 31823 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:08.618690 31823 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:08.596446  1048 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.137s	user 0.000s	sys 0.000s
I20260709 12:21:08.512310 31746 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:08.559335  1034 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20260709 12:21:08.523167   993 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.953s	user 0.018s	sys 0.029s
W20260709 12:21:08.594980  1054 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:21:08.622314 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:08.594585   932 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:08.524644 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:08.566644   564 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:08.594954 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:08.596343 31745 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:08.569198  1001 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:08.631284 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:08.632794 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:08.630975 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:08.595302  1033 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:08.588213   994 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.639755 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:08.645395 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.646039 31745 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:08.646175 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:08.664989 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:08.654697  1052 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:21:08.656193 31676 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:08.514068   987 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:08.659691 31820 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260709 12:21:08.705029 31820 pending_rounds.cc:85] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 4
I20260709 12:21:08.705399 31820 pending_rounds.cc:107] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 12:21:08.707691 31820 raft_consensus.cc:2889] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:08.705147 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 65522.9us
I20260709 12:21:08.700659 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.661914 31748 raft_consensus.cc:2955] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:08.718811 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:08.721081 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:08.648109 31821 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:08.711730 31819 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:08.647814   993 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:08.700527   920 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:08.723330 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:08.740342 31746 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:08.704106   939 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:08.702360   997 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:08.741315   966 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:08.731712   979 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:08.741447   939 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.639676   932 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:08.744313   932 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:08.745777 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:08.746125 31825 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.574s	user 0.009s	sys 0.005s
I20260709 12:21:08.746577 31825 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:08.746687 31744 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:08.723627   973 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:08.751085   956 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:08.752195  1035 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:08.753245   972 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:08.733142  1015 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:08.753545 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:08.722585 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:08.731269   987 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:08.742228   979 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:08.757763   979 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:08.703296  1030 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:21:08.648564  1001 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:08.761061   939 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:08.761739  1001 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:08.696652  1033 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:08.763077 31671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1061) took 1844 ms (client timeout 2050 ms). Trace:
W20260709 12:21:08.763540 31671 rpcz_store.cc:269] 0709 12:21:06.918384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:06.928577 (+ 10193us) service_pool.cc:224] Handling call
0709 12:21:08.762842 (+1834265us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b540022ae54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:08.763004 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1485089024}
I20260709 12:21:08.764493  1033 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:08.740397   994 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:08.769912 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:08.771317 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 48834.6us
I20260709 12:21:08.772362 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:08.779644 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:08.780308 31820 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:08.774139 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:08.783224 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:08.783982 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:08.785032 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:08.785743 31824 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:08.785610 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:08.787369 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:08.789001 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 140581us
W20260709 12:21:08.775158 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 76548.5us
I20260709 12:21:08.796582 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:08.798784 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:08.816537 31748 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:08.795342   932 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:08.778196   994 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:08.806916 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:08.806957 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:08.810468 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:08.812278 31674 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:21:08.802150 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:08.778802   974 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 12:21:08.809293  1035 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:08.800813  1033 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:08.825091 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:08.828877 32378 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:08.803925 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:08.807435   920 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:08.820478 31671 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:08.835112  1035 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.836758  1030 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:08.837333 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:08.837817   862 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:08.837659  1001 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:21:08.839944 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 43485us
I20260709 12:21:08.841604 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.841939  1015 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:08.799331   973 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260709 12:21:08.842315 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.815694   993 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:08.814668 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:08.815004   678 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:08.824728   994 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:08.880955 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.881309   994 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:08.884521   966 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:08.885210   966 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:08.846540  1030 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:08.886358  1030 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:08.886288   966 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.843683 31819 raft_consensus.cc:1813] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Handling vote request from an unknown peer 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:08.890450 31819 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:08.833667 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.877878   993 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:08.891711 31669 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:08.892280 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:08.893042   993 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:08.875897   908 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:08.896616 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:08.897437  1035 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:08.897683 32376 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:08.895810   979 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.845994   972 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260709 12:21:08.899852   972 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:08.900241   972 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:08.856424 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:08.901503 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45188.8us
I20260709 12:21:08.901239   972 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:08.864161 31674 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:08.880203   678 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.902902 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:08.904299 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:08.866892  1033 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:08.852262   974 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:08.824674   932 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.870784   564 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.009s
I20260709 12:21:08.877238 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:08.905431 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:08.921967 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:08.946414 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1617) took 1491 ms. Trace:
I20260709 12:21:08.947046 31822 rpcz_store.cc:276] 0709 12:21:07.455396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:07.465046 (+  9650us) service_pool.cc:224] Handling call
0709 12:21:08.907602 (+1442556us) spinlock_profiling.cc:243] Waited 5.73 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0e9c 00007fe8f9cb0a5b 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:08.907874 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:08.934662 (+ 26788us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:08.934691 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:08.946028 (+ 11337us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:08.946087 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:08.946173 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:08.946333 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14796416}
W20260709 12:21:08.949137 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 43798.1us
I20260709 12:21:08.957883 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1632) took 1196 ms. Trace:
I20260709 12:21:08.902554   972 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:08.970081 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:08.922698  1015 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:08.971647 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:08.973073   939 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:08.973600   932 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:08.974064  1015 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:08.904850   678 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:08.897360   908 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:08.974663 31671 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:08.970284 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.977520   859 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:08.900233   979 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:08.936332 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:08.985742 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49530.1us
I20260709 12:21:08.933161   974 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:08.956859  1005 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:08.959923   946 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:08.958451 31827 rpcz_store.cc:276] 0709 12:21:07.760998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:08.180085 (+419087us) service_pool.cc:224] Handling call
0709 12:21:08.199853 (+ 19768us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:08.263169 (+ 63316us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:08.263199 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:08.263641 (+   442us) log.cc:844] Serialized 26 byte log entry
0709 12:21:08.519315 (+255674us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:08.519380 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:08.904484 (+385104us) spinlock_profiling.cc:243] Waited 62.9 ms on lock 0x7b54001e9bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:08.904552 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:08.904617 (+    65us) raft_consensus.cc:1685] finished
0709 12:21:08.904637 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:08.957787 (+ 53150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4332,"raft.run_cpu_time_us":75,"raft.run_wall_time_us":75,"spinlock_wait_cycles":144749440,"thread_start_us":255180,"threads_started":1,"wal-append.queue_time_us":373207}
I20260709 12:21:08.966679   930 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:08.994810 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:09.002292 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.991863   974 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.014214 31746 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:08.994576 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:08.990415  1070 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:08.997114   930 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:08.997375   946 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:09.000799  1005 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:09.034760 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:08.986608 32343 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:09.047582 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 61053.5us
I20260709 12:21:09.048914 32343 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:21:09.056635 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 249744us
W20260709 12:21:09.076865 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 222150us
I20260709 12:21:09.029614   930 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:09.020150   908 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:09.081019 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:09.092963 31674 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:09.039830   946 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.075403 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:09.091970 31717 leader_election.cc:400] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4.
I20260709 12:21:09.089226 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:09.076452   987 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:09.093027 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 142271us
I20260709 12:21:09.095419 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:09.082676  1070 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:09.098241 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 266609us
I20260709 12:21:09.083248  1072 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:09.099323 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:09.101065 31791 leader_election.cc:400] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:09.101823 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:09.106158 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:09.096856   751 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:09.114548 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:09.116786 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:09.099253  1070 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:09.109400   920 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:09.118690 31671 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:09.107438  1015 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:09.123085  1035 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:09.114225  1005 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.098347   799 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 12:21:09.123214  1070 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.098085   979 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 12:21:09.102077   823 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 12:21:09.098510   987 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:09.134390   920 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:09.134974  1015 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:09.108358  1072 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.151762 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:09.137498   973 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:09.161499   987 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:09.160399 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54291.5us
W20260709 12:21:09.168684 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 66965.2us
I20260709 12:21:09.208983 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:09.210414 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58744.8us
I20260709 12:21:09.141062   678 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.032s
I20260709 12:21:09.211323 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:09.213163 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:09.178591 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:09.167938 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 64048.2us
I20260709 12:21:09.215335 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:09.133873  1075 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.075s
I20260709 12:21:09.224356 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:09.219305 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:09.224392 31748 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:09.227128 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:09.237673 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:09.228030 31753 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:09.208333 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:09.150606  1035 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.093s
I20260709 12:21:09.229710  1071 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:09.212167  1033 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:09.163949  1015 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:09.164173   920 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.231912   931 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:09.220733   973 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:09.291030   973 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:09.291615   950 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:09.291805  1073 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:21:09.291880  1071 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:09.293205 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1575) took 1020 ms. Trace:
I20260709 12:21:09.293068  1071 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.293589 31752 rpcz_store.cc:276] 0709 12:21:08.273007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:08.560821 (+287814us) service_pool.cc:224] Handling call
0709 12:21:08.561191 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:09.001202 (+440011us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:09.001245 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:09.013630 (+ 12385us) log.cc:844] Serialized 26 byte log entry
0709 12:21:09.161933 (+148303us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:09.161971 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:09.292781 (+130810us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400295ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:09.292828 (+    47us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:09.292873 (+    45us) raft_consensus.cc:1685] finished
0709 12:21:09.292894 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:09.293137 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":736,"raft.queue_time_us":16282,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"spinlock_wait_cycles":343094656,"thread_start_us":425123,"threads_started":2,"wal-append.queue_time_us":262114}
I20260709 12:21:09.274211 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:09.275174   678 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:09.277458  1001 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:09.278077  1069 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
I20260709 12:21:09.213166  1081 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:09.248443   861 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:09.253250   855 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:21:09.242442 31561 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f6404cb9c2174326b06cbbb0d27d43f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b95aeeaf4c51442b8092cff1de674f2b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:21:09.297883  1066 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.203s	user 0.000s	sys 0.001s
I20260709 12:21:09.258430 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:09.319626 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1573) took 1047 ms. Trace:
W20260709 12:21:09.334117 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 96351.7us
I20260709 12:21:09.289218 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:09.328037 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:09.321638 31749 rpcz_store.cc:276] 0709 12:21:08.272333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:08.273433 (+  1100us) service_pool.cc:224] Handling call
0709 12:21:08.356173 (+ 82740us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:08.356689 (+   516us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:08.356734 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:08.378542 (+ 21808us) log.cc:844] Serialized 26 byte log entry
0709 12:21:08.747509 (+368967us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b180048ece0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:21:09.056271 (+308762us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:09.056332 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:09.056435 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:09.297855 (+241420us) raft_consensus.cc:1685] finished
0709 12:21:09.297879 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:09.319535 (+ 21656us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6841,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":6071,"spinlock_wait_cycles":32996736,"thread_start_us":308525,"threads_started":1,"wal-append.queue_time_us":761642}
W20260709 12:21:09.330422 31561 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 88160.1us
I20260709 12:21:09.299657  1033 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.309767   994 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.314177  1001 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:09.457537 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 219849us
I20260709 12:21:09.460089 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1643) took 1080 ms. Trace:
I20260709 12:21:09.312490   678 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.402027  1079 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.129s
I20260709 12:21:09.473496 31823 rpcz_store.cc:276] 0709 12:21:08.379423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:08.622903 (+243480us) service_pool.cc:224] Handling call
0709 12:21:08.635553 (+ 12650us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:08.649258 (+ 13705us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:08.649313 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:08.650851 (+  1538us) log.cc:844] Serialized 26 byte log entry
0709 12:21:09.000893 (+350042us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:09.000937 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:09.001013 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:09.459850 (+458837us) raft_consensus.cc:1685] finished
0709 12:21:09.459869 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:09.460011 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2263,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":56008,"thread_start_us":349360,"threads_started":1,"wal-append.queue_time_us":430764}
I20260709 12:21:09.446691   931 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:09.481019 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:09.415226  1081 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.056s
W20260709 12:21:09.481400 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 145552us
I20260709 12:21:09.458940  1090 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:09.459172  1091 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:09.490615 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:09.471832  1086 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:09.485581   966 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.513155 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:09.485343  1035 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:09.523142 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:09.502733  1091 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:09.496708  1105 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.533226 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:09.544488 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 242221us
I20260709 12:21:09.550212   887 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:09.550248 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:09.552250 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:09.554186 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:09.554900 31748 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
W20260709 12:21:09.544145  1030 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.657s	user 0.009s	sys 0.004s
I20260709 12:21:09.530167  1072 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:09.561545 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 70897.6us
I20260709 12:21:09.525851   799 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.030s
I20260709 12:21:09.551039   861 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.008s
I20260709 12:21:09.537019  1091 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.564312 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:09.525141   966 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.558526  1079 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:09.558681  1075 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:09.527612  1035 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.502065  1090 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.562924   885 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:09.522163  1086 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.562086   931 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.595261   885 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.595608  1095 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260709 12:21:09.596333  1104 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:09.596940  1083 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.274s	user 0.000s	sys 0.001s
I20260709 12:21:09.597774   885 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.595791  1096 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260709 12:21:09.596848  1095 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:09.598580  1115 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.596907  1099 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:09.598706  1093 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:09.609745  1082 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.256s	user 0.001s	sys 0.000s
I20260709 12:21:09.610018  1093 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.603515  1098 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:09.611030  1086 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.543708  1105 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.570479   887 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.024s
I20260709 12:21:09.612663  1093 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:09.596860  1104 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.611250  1098 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.613735  1105 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.614248  1100 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:09.614694  1100 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:09.615119  1100 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.615310  1098 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.614884  1115 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.615899  1098 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:09.617455  1115 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:09.609424  1001 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:09.619369  1104 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.602298  1033 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:09.603201  1089 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20260709 12:21:09.584817  1079 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:09.620662  1079 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:09.621479   930 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.525s	user 0.007s	sys 0.004s
I20260709 12:21:09.621898   930 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.603971 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1513) took 1490 ms. Trace:
I20260709 12:21:09.624213  1035 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:09.592103   855 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:09.609436  1099 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:09.622829 31678 rpcz_store.cc:276] 0709 12:21:08.113182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:08.113359 (+   177us) service_pool.cc:224] Handling call
0709 12:21:08.142149 (+ 28790us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:08.158703 (+ 16554us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:08.158749 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:08.165489 (+  6740us) log.cc:844] Serialized 26 byte log entry
0709 12:21:08.403882 (+238393us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:08.467075 (+ 63193us) log.cc:844] Serialized 14 byte log entry
0709 12:21:08.467460 (+   385us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:08.467564 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:09.534179 (+1066615us) spinlock_profiling.cc:243] Waited 5.48 ms on lock 0x7b1800246b20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:21:09.601866 (+ 67687us) raft_consensus.cc:1685] finished
0709 12:21:09.603753 (+  1887us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:09.603909 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":62886,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":57,"raft.queue_time_us":9377,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":14847,"spinlock_wait_cycles":12606848,"thread_start_us":237893,"threads_started":1,"wal-append.queue_time_us":325290}
W20260709 12:21:09.625525   751 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.511s	user 0.010s	sys 0.001s
I20260709 12:21:09.625983   751 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:09.625689  1099 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.628379   966 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:09.626231 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:09.607200   994 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.601574  1090 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.567984  1030 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.638908   751 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.652529 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:09.653295 31678 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:09.654948   751 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.655622   989 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:09.656761   946 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.559s	user 0.004s	sys 0.006s
I20260709 12:21:09.657171   751 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:09.658200   930 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.609020   939 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.629516  1095 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:09.657186   946 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.671010  1015 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.668444   973 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.676112   814 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:09.676472 31669 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:09.676759   814 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.677670   973 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.679113   814 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:09.679844 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:09.680116   814 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:09.680482 31753 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:09.680372  1079 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.680621   814 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.681769   920 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.682376 31671 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:21:09.682997   814 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:09.683522  1095 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.684757  1095 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:09.695672   920 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.674988   946 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.689148  1015 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.716526 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:09.713053 32376 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.814s	user 0.017s	sys 0.003s
W20260709 12:21:09.718182   974 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.678s	user 0.011s	sys 0.004s
I20260709 12:21:09.737651  1079 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:09.733513   920 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:09.750212   920 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:09.754489   920 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:09.759336  1030 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.760825 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:09.761969 31673 raft_consensus.cc:2368] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:09.765913   972 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:09.766971  1030 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.769368  1030 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:09.773545   987 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.511s	user 0.011s	sys 0.000s
I20260709 12:21:09.774407   987 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.779524  1076 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:09.780197  1076 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:09.781316  1076 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.783393 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:09.807279 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:09.808748 31673 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.738652 32376 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:09.766553   972 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:09.799679  1076 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.782356   920 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:09.818096  1070 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.669s	user 0.013s	sys 0.000s
I20260709 12:21:09.826794   972 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.831841   862 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:09.832459   920 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.832538   862 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.827379  1070 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.835705 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:09.836301  1079 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:09.836356 31820 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.837078  1079 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.838683 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:09.839232   814 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:09.839803   814 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.840474 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:09.841418 31675 raft_consensus.cc:2368] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6.
I20260709 12:21:09.841854   814 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:09.843178 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:09.843922 32370 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:09.845520 31677 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:21:09.846160   972 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.846330 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:09.847008 31677 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:09.847106 31823 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:21:09.849578 31717 leader_election.cc:400] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6.
I20260709 12:21:09.850004 31717 leader_election.cc:403] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:21:09.842454 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:09.851486 31675 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:09.853044 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:09.858430 31820 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:09.869269  1078 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:09.869443 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:09.869668  1114 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:09.870249  1095 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:09.870612 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:09.870795 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:09.871338 31827 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:09.868110 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:09.872608 31823 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:09.873415  1095 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:09.873869  1095 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:09.874249  1095 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:09.856091 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:09.845722   862 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:09.877064   862 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:09.878439 31792 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:09.879499   862 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:09.880810 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:09.871338 31825 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:09.881570   862 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:09.883248  1079 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:09.842450   814 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.884203   814 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.891285   564 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:09.891914   564 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:09.893095   564 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.799376  1100 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.897369  1030 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:09.900586  1100 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.901053   862 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:09.931818   862 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:09.901749   564 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:09.936193 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:09.937001 31823 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:09.937412 31750 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:09.938508   956 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:09.939107   956 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.939520 31826 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:09.941041 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:09.941718 31820 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:09.909092  1114 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:09.915391  1074 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.617s	user 0.004s	sys 0.000s
I20260709 12:21:09.923482 31671 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:09.921433 31573 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 1 to 3, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:09.930805 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:09.930984   814 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:09.943305 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:09.970983 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:09.966418  1114 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.971499   841 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:09.957427  1030 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:09.973479  1030 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:09.979183 31735 tablet_service.cc:1558] Processing DeleteTablet for tablet fc47871ef692418388113c1d6c88b262 with delete_type TABLET_DATA_TOMBSTONED (TS 039d66a6c85e43c3936bf5017501b344 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:45150
I20260709 12:21:09.747454   974 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.002290 31820 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:09.980908 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:09.864923 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 338) took 1052 ms. Trace:
I20260709 12:21:09.992460  1079 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:09.992909 31573 catalog_manager.cc:5697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 1 to 2, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:09.993899  1114 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:09.995874  1030 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:10.006348   841 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:10.008422   997 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:10.008970   939 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:10.030411 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 462819us
I20260709 12:21:10.011204  1070 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:09.992506   987 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:10.031710 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 467505us
I20260709 12:21:10.046396 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:10.048623 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:10.009650 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:21:10.032356 31674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.580s	user 0.009s	sys 0.010s
W20260709 12:21:10.053124 31674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.009s	sys 0.010s
W20260709 12:21:10.055867 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46323.5us
I20260709 12:21:10.060145 31675 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:10.037636 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:10.096030 31673 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
W20260709 12:21:10.124256 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 75727.8us
I20260709 12:21:10.127019  1078 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6.
I20260709 12:21:10.129056   956 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:10.129546   950 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:10.131000   568 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:10.132508   568 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:10.132953   568 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:10.027365   932 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:10.058810 31576 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 147) took 7201 ms. Trace:
I20260709 12:21:10.023557   841 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:10.133927   568 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.158978   568 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:10.145916   678 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:10.130250   950 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.168804   950 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:10.169539 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:10.148262   814 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:10.148981 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:10.171096 31826 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:10.170919   814 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.173481   814 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:10.150049 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:10.174539 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:10.175143 31827 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:10.152074  1095 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.178009  1095 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:10.179173   862 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:10.164144   974 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:10.183082   979 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:10.183669   979 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:10.184643   979 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.087688   961 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:21:10.156980  1091 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.567s	user 0.006s	sys 0.006s
I20260709 12:21:10.188014  1091 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.189158 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:10.189548 31749 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:10.189889 31672 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.157428  1081 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:10.193346  1081 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:10.193964  1081 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:10.199824 31672 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:10.200992  1091 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:10.201509 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 76032.2us
I20260709 12:21:10.204043 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:10.205843   974 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:10.206480   974 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:10.207496   974 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.210371   932 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:10.203176 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:10.146782   855 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:10.187085 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.213596   932 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:10.154422 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:10.057967  1114 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:10.087483 31745 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:10.163832   841 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:10.043466 31746 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.008s	sys 0.056s
W20260709 12:21:10.217128 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.909s	user 0.004s	sys 0.016s
W20260709 12:21:10.232007 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.923s	user 0.005s	sys 0.017s
W20260709 12:21:10.220841 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 183329us
I20260709 12:21:10.223064 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:10.225488 31820 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:10.233384 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:10.224351   961 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:10.235396   961 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:10.232512 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 924095us
I20260709 12:21:10.247170 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:10.247542 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.055156  1079 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.247924 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.252004 31748 raft_consensus.cc:2410] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 12:21:10.231691 31746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.690s	user 0.010s	sys 0.057s
I20260709 12:21:10.254057 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.230835 31826 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:10.256556 31751 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:10.020469 32376 rpcz_store.cc:276] 0709 12:21:08.767103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:08.778710 (+ 11607us) service_pool.cc:224] Handling call
0709 12:21:08.896459 (+117749us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:09.819806 (+923347us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7144704}
I20260709 12:21:10.252920 31747 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:10.227854  1114 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:10.258860 31749 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:10.260489 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.222851  1095 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:10.238018   678 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.038s
I20260709 12:21:10.231863 31671 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:10.268280 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:10.275020 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:10.274286 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.277460 31672 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
W20260709 12:21:10.284235 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.741s	user 0.031s	sys 0.032s
W20260709 12:21:10.304023 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.032s	sys 0.033s
W20260709 12:21:10.304484 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 761094us
I20260709 12:21:10.305335 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:10.305998 32375 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:10.226809   855 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.310290 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:10.312160  1101 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:10.312885   678 consensus_queue.cc:1243] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 2.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:21:10.313637  1079 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:10.281585  1114 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.278843 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:10.320779 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:10.253120   799 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:10.324579 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.737s	user 0.041s	sys 0.055s
I20260709 12:21:10.322615 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:10.324735 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:10.336367 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.748s	user 0.041s	sys 0.055s
I20260709 12:21:10.328732 31748 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:10.336830 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 748927us
I20260709 12:21:10.326531 31820 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
W20260709 12:21:10.338124 31676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.002s	sys 0.027s
W20260709 12:21:10.338508 31676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.002s	sys 0.027s
I20260709 12:21:10.329113 31749 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:21:10.340095 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:10.258067   932 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:10.340867 32375 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:10.341439 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:10.257784 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.271471 32370 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:10.271950 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:10.276372  1095 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:10.327138 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:10.327886  1099 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.005s	sys 0.029s
I20260709 12:21:10.352257 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:10.352430 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:10.325702   931 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.022s	sys 0.014s
I20260709 12:21:10.247912  1081 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.374084 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:10.375737 31678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.705s	user 0.018s	sys 0.022s
I20260709 12:21:10.368882 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:10.376308 31678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.706s	user 0.018s	sys 0.022s
W20260709 12:21:10.347519 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 100438us
I20260709 12:21:10.135625 31576 rpcz_store.cc:276] 0709 12:21:02.853736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:02.854028 (+   292us) service_pool.cc:224] Handling call
0709 12:21:10.055087 (+7201059us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:06.880741 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:06.881524 (+   783us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:06.881588 (+    64us) write_op.cc:435] Acquired schema lock
0709 12:21:06.881609 (+    21us) tablet.cc:662] Decoding operations
0709 12:21:06.971678 (+ 90069us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:06.971803 (+   125us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:06.971845 (+    42us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:21:06.972871 (+  1026us) tablet.cc:701] Row locks acquired
0709 12:21:06.972929 (+    58us) write_op.cc:260] PREPARE: finished
0709 12:21:06.973175 (+   246us) write_op.cc:270] Start()
0709 12:21:06.973372 (+   197us) write_op.cc:276] Timestamp: P: 1783599666973090 usec, L: 0
0709 12:21:06.973418 (+    46us) op_driver.cc:348] REPLICATION: starting
0709 12:21:07.092360 (+118942us) log.cc:844] Serialized 2886 byte log entry
0709 12:21:07.507925 (+415565us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b54002828d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd0e28 00007fe90080b581 00007fe90080ad58 00007fe90080f9f1 00007fe90080f9a9 00007fe90080f939 00007fe90080f901 00007fe90080eb2d 0000000000388634 0000000000388479
0709 12:21:07.575513 (+ 67588us) op_driver.cc:464] REPLICATION: finished
0709 12:21:07.744318 (+168805us) spinlock_profiling.cc:243] Waited 9.77 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8f9ccb0ac 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe90080bef0
0709 12:21:07.871686 (+127368us) write_op.cc:301] APPLY: starting
0709 12:21:07.871795 (+   109us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:07.872058 (+   263us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:07.872102 (+    44us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:08.203999 (+331897us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:08.238838 (+ 34839us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0709 12:21:08.238918 (+    80us) write_op.cc:312] APPLY: finished
0709 12:21:08.239323 (+   405us) log.cc:844] Serialized 59 byte log entry
0709 12:21:08.239570 (+   247us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:08.239709 (+   139us) write_op.cc:454] Released schema lock
0709 12:21:08.239778 (+    69us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:08.765297 (+525519us) spinlock_profiling.cc:243] Waited 9.96 ms on lock 0x7b180016a380. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:21:08.765450 (+   153us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:08.765486 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":10,"child_traces":[["op",{"apply.queue_time_us":288487,"apply.run_cpu_time_us":5771,"apply.run_wall_time_us":368503,"num_ops":7,"prepare.queue_time_us":184869,"prepare.run_cpu_time_us":52428,"prepare.run_wall_time_us":627477,"raft.queue_time_us":154747,"raft.run_cpu_time_us":31883,"raft.run_wall_time_us":357542,"replication_time_us":570778,"spinlock_wait_cycles":130932608,"thread_start_us":614396,"threads_started":4,"wal-append.queue_time_us":176098,"wal-append.run_cpu_time_us":24467,"wal-append.run_wall_time_us":1996274}]]}
I20260709 12:21:10.321851  1101 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:10.350284   980 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.799s	user 0.033s	sys 0.065s
I20260709 12:21:10.371229 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:10.368111 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.350323  1030 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:10.371083   931 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.023s	sys 0.014s
I20260709 12:21:10.390208 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:10.348661  1075 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:10.335983   799 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:10.367676  1095 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.377635 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:10.366894  1099 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.006s	sys 0.030s
I20260709 12:21:10.373723 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.397517   956 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:10.400003 31748 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:10.366016   946 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260709 12:21:10.359577   932 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:10.407320 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1092) took 1134 ms. Trace:
I20260709 12:21:10.410231 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:10.400302   980 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.858s	user 0.033s	sys 0.066s
I20260709 12:21:10.416826 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:10.413914 31791 leader_election.cc:400] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:10.471567 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.404425 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:10.418339 32371 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:21:10.407397 31670 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.014s	sys 0.033s
I20260709 12:21:10.414517 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.496304 31678 rpcz_store.cc:276] 0709 12:21:09.272765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.654065 (+381300us) service_pool.cc:224] Handling call
0709 12:21:09.654406 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:09.654601 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:21:09.670358 (+ 15757us) log.cc:844] Serialized 14 byte log entry
0709 12:21:10.406843 (+736485us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:10.406921 (+    78us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:10.406981 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.407074 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:10.407234 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10297,"prepare.queue_time_us":13281,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":30256,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":5447,"thread_start_us":706033,"threads_started":1}
I20260709 12:21:10.500742 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.414690   799 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:10.470341 31825 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.007s	sys 0.015s
I20260709 12:21:10.408746  1070 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:10.525447 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:10.486532   956 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:10.402268  1131 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:10.408763   939 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:10.410595   564 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:10.528782 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 118729us
I20260709 12:21:10.557171  1131 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:10.546984 31825 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.008s	sys 0.015s
I20260709 12:21:10.558210 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:21:10.505522 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 91838.7us
I20260709 12:21:10.469534  1091 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:10.559803  1131 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:10.560674 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.561657 31749 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:10.561957   939 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:10.557175 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:10.563056   939 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:10.565071 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:10.533727 31674 raft_consensus.cc:2368] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:10.534065 31672 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:10.534454 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.568575 31673 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:10.570778  1079 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:10.571375  1079 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.573706  1079 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:10.578204 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.579998 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.584336   814 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:10.585660   814 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:10.586838   564 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:10.587306   564 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:10.588270   564 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.590026   564 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:10.544693 31670 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.777s	user 0.014s	sys 0.033s
I20260709 12:21:10.544078 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.544481 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.600103 31748 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:10.537262 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:10.602262 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 65138.2us
I20260709 12:21:10.414149  1092 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.117s
I20260709 12:21:10.496701 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.553383 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:10.604075 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:10.529474 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 112758us
W20260709 12:21:10.605605 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:10.606062 31718 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:10.607064   814 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:10.607197 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:10.607705   814 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:10.608486 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:10.566197  1015 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:10.591398 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:10.591933 31751 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:10.405678  1030 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:10.527550  1099 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:10.591495  1126 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:21:10.537855   972 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:10.601306 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.618688 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.619504 31748 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:10.400667  1101 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:10.615913  1091 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:10.600419 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1589) took 1380 ms. Trace:
I20260709 12:21:10.621883   814 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:10.622112  1091 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:10.622740  1076 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.611690 31747 raft_consensus.cc:2368] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4.
I20260709 12:21:10.624151 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.625257 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.531360  1121 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:10.550591   956 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.626564 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:10.626586 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:10.650163 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.622537  1091 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:10.705343  1132 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.175s	user 0.000s	sys 0.000s
I20260709 12:21:10.626013 31747 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:10.696408 31748 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:10.696398 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 66738.8us
W20260709 12:21:10.481720 31735 thread.cc:641] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.016s	sys 0.012s
W20260709 12:21:10.487691  1005 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.640s	user 0.009s	sys 0.015s
I20260709 12:21:10.628273 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:10.673956 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:10.625993  1075 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:10.600225   946 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.095s
I20260709 12:21:10.717748  1030 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:10.701141  1015 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:10.673782   855 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:10.724140  1081 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:10.724216 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:10.725021 31820 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:10.728014 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.729351 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:10.730141 31678 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:10.730808 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:10.731076 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:10.731166 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:10.731804 32375 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:10.731943 32371 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:21:10.734061 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:10.735947 32375 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:10.736339 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:10.737354 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.738500 31669 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:21:10.739079 31669 raft_consensus.cc:606] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:10.739907 31669 raft_consensus.cc:614] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:10.740429  1075 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:10.740907  1075 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:10.741284  1075 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.748092 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1537) took 1532 ms. Trace:
I20260709 12:21:10.748529 31676 rpcz_store.cc:276] 0709 12:21:09.215233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.237173 (+ 21940us) service_pool.cc:224] Handling call
0709 12:21:09.266210 (+ 29037us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:09.455377 (+189167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:09.455996 (+   619us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:09.672928 (+216932us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:21:09.673320 (+   392us) log.cc:844] Serialized 26 byte log entry
0709 12:21:09.876368 (+203048us) spinlock_profiling.cc:243] Waited 7.35 ms on lock 0x7b4800080058. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:21:10.339047 (+462679us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:10.339085 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.339150 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:10.747870 (+408720us) raft_consensus.cc:1685] finished
0709 12:21:10.747891 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:10.748030 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":143174,"raft.queue_time_us":284081,"raft.run_cpu_time_us":3897,"raft.run_wall_time_us":314797,"spinlock_wait_cycles":41545856,"thread_start_us":842005,"threads_started":2,"wal-append.queue_time_us":854663}
I20260709 12:21:10.751200 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:10.754161  1099 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:10.754496 31676 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:10.754767  1099 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:10.755028 31561 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f6404cb9c2174326b06cbbb0d27d43f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b95aeeaf4c51442b8092cff1de674f2b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:10.755563  1099 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:10.712699 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1590) took 1230 ms. Trace:
I20260709 12:21:10.759622   972 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:10.760155   972 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.759503 31746 rpcz_store.cc:276] 0709 12:21:09.482114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.482264 (+   150us) service_pool.cc:224] Handling call
0709 12:21:09.508027 (+ 25763us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:09.540968 (+ 32941us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:09.541007 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:09.541563 (+   556us) log.cc:844] Serialized 26 byte log entry
0709 12:21:10.287096 (+745533us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:10.305286 (+ 18190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.322229 (+ 16943us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:10.674747 (+352518us) raft_consensus.cc:1685] finished
0709 12:21:10.694896 (+ 20149us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:10.706645 (+ 11749us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18049,"raft.queue_time_us":8024,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"thread_start_us":715907,"threads_started":1,"wal-append.queue_time_us":867135}
W20260709 12:21:10.764125   568 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.605s	user 0.011s	sys 0.003s
I20260709 12:21:10.766511 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.767409 31674 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:10.768095   997 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:10.767719  1099 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:10.768623   997 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:10.772295 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.773321  1075 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:10.775591   568 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:10.776473 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:10.770244   972 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.778383   931 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:10.778848   931 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:10.779094   972 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:10.779728   931 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.781301   931 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:10.782408  1070 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:10.674212  1086 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:10.788690 31824 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:10.676551 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:10.715452 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 87270.1us
I20260709 12:21:10.776332 31749 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:10.708827 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 304504us
I20260709 12:21:10.794108 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:10.797792 31824 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:10.797681 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:10.789944 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:10.783128  1079 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:10.590554   799 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:10.715008 31735 thread.cc:608] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.017s	sys 0.012s
I20260709 12:21:10.806701 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:10.790589  1005 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.943s	user 0.009s	sys 0.016s
I20260709 12:21:10.810801 31671 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:10.782997  1070 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:10.792888   987 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:10.825141 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:10.829734 31822 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:10.800765   855 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:10.742230  1091 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.839099 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:10.807862  1079 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:10.835597 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:10.825264   841 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.574s	user 0.011s	sys 0.000s
I20260709 12:21:10.806337  1131 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.036s
I20260709 12:21:10.849831 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.860284 31753 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:21:10.825559  1030 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:10.714702  1121 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:10.823158 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:10.829243   987 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:10.863046 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:10.621894 31745 rpcz_store.cc:276] 0709 12:21:09.217506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.220683 (+  3177us) service_pool.cc:224] Handling call
0709 12:21:09.307436 (+ 86753us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:10.233299 (+925863us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.547322 (+314023us) spinlock_profiling.cc:243] Waited 716 ms on lock 0x7b54001a4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:10.597542 (+ 50220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1646513536}
I20260709 12:21:10.858588 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:10.806161  1114 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:10.862485 31643 connection.cc:256] RPC call timeout handler was delayed by -1.4319452110003112s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1906s for a response
I20260709 12:21:10.879709 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:10.881572 31643 connection.cc:256] RPC call timeout handler was delayed by -1.3617452110001977s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19840000000000002s for a response
I20260709 12:21:10.882234 31751 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:10.899760 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:10.900581  1121 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:10.882839 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:10.848707   916 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:10.884714 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.884975  1015 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.863587  1105 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:10.859004  1078 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.028s
I20260709 12:21:10.886911 31826 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:10.886790 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:10.887643 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 159) took 2093 ms. Trace:
I20260709 12:21:10.864603   841 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:10.889328  1075 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:10.891278   932 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:10.863885  1076 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:10.882879 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.871685  1030 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:10.895274 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:10.924466 31822 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:10.921793 31753 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:10.896252 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.896413   997 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:10.848824  1079 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:10.876873   987 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:10.858247  1099 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:10.903198 31750 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.025s	sys 0.048s
W20260709 12:21:10.936280 31750 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.026s	sys 0.048s
I20260709 12:21:10.905969 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:10.908134 31677 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.888s	user 0.115s	sys 0.190s
W20260709 12:21:10.938436 31677 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.919s	user 0.115s	sys 0.190s
W20260709 12:21:10.903923 31752 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.037s	sys 0.036s
I20260709 12:21:10.912480 31820 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:21:10.940649 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:10.913327 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:10.915071 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:10.901197  1081 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:10.944888 31747 raft_consensus.cc:2410] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:10.944583  1081 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:10.945353  1129 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.251s	user 0.001s	sys 0.001s
I20260709 12:21:10.903749   568 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:10.946336 31747 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:10.947163   568 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:10.947932   568 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:10.902691  1070 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:10.949240 31752 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.037s	sys 0.037s
W20260709 12:21:10.949715  1128 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:21:10.949873  1070 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:10.950431  1070 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:10.906052   862 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:10.951882   862 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:10.942206 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:10.952841   862 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.953931   862 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:10.954272 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:10.907459  1119 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:10.956923  1081 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:10.957423 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1587) took 1741 ms. Trace:
I20260709 12:21:10.957522  1119 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:10.958014 31744 rpcz_store.cc:276] 0709 12:21:09.215976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.238686 (+ 22710us) service_pool.cc:224] Handling call
0709 12:21:09.910526 (+671840us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:09.910754 (+   228us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:10.091434 (+180680us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:10.101891 (+ 10457us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:10.133966 (+ 32075us) log.cc:844] Serialized 26 byte log entry
0709 12:21:10.503175 (+369209us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:10.529072 (+ 25897us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.532167 (+  3095us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:10.957082 (+424915us) raft_consensus.cc:1685] finished
0709 12:21:10.957111 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:10.957360 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":211899,"raft.queue_time_us":144904,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":29595,"spinlock_wait_cycles":12067072,"thread_start_us":349589,"threads_started":1,"wal-append.queue_time_us":821236}
I20260709 12:21:10.960834  1119 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:10.961737 31744 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:10.928537 31748 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
W20260709 12:21:10.930734   974 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.012s	sys 0.038s
I20260709 12:21:10.900593 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:10.909250  1131 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:10.978158 31751 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:10.955112 31820 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:21:10.956041  1070 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:10.977571 31675 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.018s	sys 0.028s
I20260709 12:21:10.904287   916 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:10.917800  1105 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:10.983356 31753 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:10.979694 31675 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.018s	sys 0.028s
I20260709 12:21:10.928575  1030 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:10.976086 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:10.907641  1101 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:10.977707   974 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.012s	sys 0.038s
I20260709 12:21:10.996556 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:10.997715 31827 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.020s	sys 0.062s
I20260709 12:21:10.979349  1005 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:10.979970 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:11.001148 31827 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.020s	sys 0.062s
I20260709 12:21:10.985818  1100 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:11.035957 31820 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:10.999207  1098 raft_consensus.cc:2764] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:21:10.986361  1131 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:11.041088   974 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:11.055157 31822 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
W20260709 12:21:11.015548   979 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.827s	user 0.013s	sys 0.043s
I20260709 12:21:11.068224 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:11.059212  1078 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:11.125353  1078 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:11.127192 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:11.127553   916 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:11.127969 32372 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:11.129670  1078 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:11.131294 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:11.132154 31822 raft_consensus.cc:2376] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3. Re-sending same reply.
I20260709 12:21:11.133391 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:11.134115 31822 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.140774 32372 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
W20260709 12:21:11.144094  1135 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:21:11.145738 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 114) took 1210 ms. Trace:
I20260709 12:21:11.146360 31825 rpcz_store.cc:276] 0709 12:21:09.935394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.940123 (+  4729us) service_pool.cc:224] Handling call
0709 12:21:09.940503 (+   380us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:09.940900 (+   397us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:09.940930 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:09.941448 (+   518us) log.cc:844] Serialized 26 byte log entry
0709 12:21:10.559077 (+617629us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:10.559130 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.559217 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:11.145428 (+586211us) raft_consensus.cc:1685] finished
0709 12:21:11.145458 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:11.145655 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18099,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":130,"thread_start_us":617202,"threads_started":1,"wal-append.queue_time_us":1049973}
I20260709 12:21:11.148749 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:11.149371 31825 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:11.154512 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:11.158777 31824 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:11.163823 31672 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:11.122740 31819 raft_consensus.cc:2410] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
W20260709 12:21:11.152119   979 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.933s	user 0.014s	sys 0.043s
I20260709 12:21:11.122141 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:11.241473 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:11.123224  1076 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.137094 31818 raft_consensus.cc:2955] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:21:11.251343 31641 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.013s	sys 0.011s
W20260709 12:21:11.254309 32375 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.015s	sys 0.010s
I20260709 12:21:11.162395  1091 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:11.166944  1114 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:11.221376   862 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:11.257777 31641 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.014s	sys 0.011s
I20260709 12:21:11.240384  1081 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:11.230870 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:11.255879  1136 tablet_replica.cc:333] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:21:11.063930  1105 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:11.258267 32375 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.015s	sys 0.010s
I20260709 12:21:11.152626   974 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:11.295742 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:11.316766 31674 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:11.295181 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:11.202353   956 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:11.195999 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:11.257143 31671 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:11.259120   841 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:11.325338 31824 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:11.296339 31744 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:21:11.319149 32376 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:10.916425 31670 rpcz_store.cc:276] 0709 12:21:08.794106 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:08.798550 (+  4444us) service_pool.cc:224] Handling call
0709 12:21:09.754021 (+955471us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:09.754215 (+   194us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:09.754618 (+   403us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:09.754672 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:09.755239 (+   567us) log.cc:844] Serialized 26 byte log entry
0709 12:21:10.614650 (+859411us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:10.618604 (+  3954us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.638821 (+ 20217us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:10.887274 (+248453us) raft_consensus.cc:1685] finished
0709 12:21:10.887307 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:10.887577 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4044,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":119,"spinlock_wait_cycles":29087104,"thread_start_us":843478,"threads_started":1,"wal-append.queue_time_us":1075590}
I20260709 12:21:11.307219  1081 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:11.301338   862 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:11.284160  1076 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:11.340588 31678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.015s	sys 0.019s
I20260709 12:21:11.341487 31822 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
W20260709 12:21:11.471403 31678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.015s	sys 0.019s
W20260709 12:21:11.327932   950 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.003s	sys 0.020s
W20260709 12:21:11.350854 31716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.021s	sys 0.026s
W20260709 12:21:11.465763 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.004s	sys 0.022s
W20260709 12:21:11.379006 31792 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.033s	sys 0.078s
W20260709 12:21:11.482980 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.004s	sys 0.022s
W20260709 12:21:11.481812 31716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.021s	sys 0.026s
W20260709 12:21:11.344161 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.000s	sys 0.045s
W20260709 12:21:11.483387 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 679142us
W20260709 12:21:11.351467  1102 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.967s	user 0.014s	sys 0.042s
W20260709 12:21:11.377032   961 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.847s	user 0.014s	sys 0.026s
W20260709 12:21:11.410110  1139 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:21:11.340327   799 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:11.447482  1114 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.480005  1119 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:11.431699  1136 raft_consensus.cc:2243] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Raft consensus shutting down.
W20260709 12:21:11.468451 31748 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.026s	sys 0.040s
W20260709 12:21:11.431326  1144 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
W20260709 12:21:11.548516  1102 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.175s	user 0.014s	sys 0.042s
W20260709 12:21:11.506031 31821 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.018s	sys 0.039s
I20260709 12:21:11.546550  1100 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:11.354279 32374 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
W20260709 12:21:11.402474  1146 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
W20260709 12:21:11.417279  1134 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
I20260709 12:21:11.552775 31753 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:21:11.341058 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:11.483203 31792 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.033s	sys 0.078s
I20260709 12:21:11.567268   956 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:11.384454 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 440591us
W20260709 12:21:11.512856 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.714s	user 0.000s	sys 0.055s
I20260709 12:21:11.486593 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:11.598923 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 809001us
I20260709 12:21:11.599126 31747 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
W20260709 12:21:11.409627  1141 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:21:11.341503  1101 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:11.608276  1081 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:11.514367 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:11.610718  1136 raft_consensus.cc:2272] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:11.611084 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96789.1us
I20260709 12:21:11.612411 31791 leader_election.cc:400] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4.
I20260709 12:21:11.612751 31791 leader_election.cc:403] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:11.614123 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:11.355540 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:11.523551 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:11.615904 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 92418.7us
W20260709 12:21:11.447108 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 444220us
W20260709 12:21:11.561872 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:11.545482  1079 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.594s	user 0.008s	sys 0.003s
I20260709 12:21:11.621995  1079 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:11.623129 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 122) took 1404 ms. Trace:
I20260709 12:21:11.624615  1079 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:11.623605 31827 rpcz_store.cc:276] 0709 12:21:10.218613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.218845 (+   232us) service_pool.cc:224] Handling call
0709 12:21:10.230258 (+ 11413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:10.232470 (+  2212us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:10.232504 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:10.233032 (+   528us) log.cc:844] Serialized 26 byte log entry
0709 12:21:11.013996 (+780964us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:11.014046 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:11.014126 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:11.622872 (+608746us) raft_consensus.cc:1685] finished
0709 12:21:11.622901 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:11.623065 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1633,"raft.queue_time_us":1766,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"thread_start_us":745225,"threads_started":1,"wal-append.queue_time_us":1049812}
I20260709 12:21:11.625360  1079 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:11.625805 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:11.545915   862 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:11.626703   862 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4.
I20260709 12:21:11.626740 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:11.628280 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:11.628459 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:11.628907 31822 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.629040 31826 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:11.629611   987 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.675s	user 0.011s	sys 0.001s
W20260709 12:21:11.435169  1145 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
W20260709 12:21:11.586501  1142 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.231s	user 0.001s	sys 0.001s
W20260709 12:21:11.410481  1143 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:21:11.632133 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:11.569664  1114 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:11.632849 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1176) took 1391 ms. Trace:
I20260709 12:21:11.632954 31670 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:11.634532  1100 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:11.633409 31750 rpcz_store.cc:276] 0709 12:21:10.241222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.241393 (+   171us) service_pool.cc:224] Handling call
0709 12:21:10.255115 (+ 13722us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:10.263196 (+  8081us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:10.263231 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:10.271741 (+  8510us) log.cc:844] Serialized 26 byte log entry
0709 12:21:10.988484 (+716743us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:11.452876 (+464392us) log.cc:844] Serialized 14 byte log entry
0709 12:21:11.453220 (+   344us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:11.453354 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:11.632614 (+179260us) raft_consensus.cc:1685] finished
0709 12:21:11.632635 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:11.632791 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6294,"prepare.queue_time_us":54438,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":7974,"raft.run_cpu_time_us":427,"raft.run_wall_time_us":42815,"thread_start_us":1115801,"threads_started":2,"wal-append.queue_time_us":970941}
I20260709 12:21:11.539470  1078 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.635027  1114 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:11.635654  1078 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:11.636324  1170 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:21:11.637279  1095 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.156s	user 0.013s	sys 0.000s
I20260709 12:21:11.637789  1095 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:11.638185  1078 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:11.477708 31819 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20260709 12:21:11.592626  1086 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:11.639067  1086 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.639768 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:11.640723 31819 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:11.641660 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:11.567238 31748 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.026s	sys 0.040s
I20260709 12:21:11.465557   993 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:11.552300   961 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.024s	user 0.014s	sys 0.026s
W20260709 12:21:11.515517   939 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.949s	user 0.017s	sys 0.076s
W20260709 12:21:11.540017  1130 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:21:11.645516 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:11.579159 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:11.647431 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:11.502614   950 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.003s	sys 0.020s
I20260709 12:21:11.654762 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:11.429741  1098 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:11.577471 31821 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.018s	sys 0.040s
W20260709 12:21:11.588937 31823 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.765s	user 0.016s	sys 0.082s
I20260709 12:21:11.486593 31752 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:11.605731 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:11.666742 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 61186.2us
I20260709 12:21:11.668579 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:11.607095 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1549) took 1734 ms. Trace:
W20260709 12:21:11.665534 31823 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.841s	user 0.017s	sys 0.082s
I20260709 12:21:11.625993  1079 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:11.400434  1131 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:11.343851 31824 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: [3.5-4.6]
I20260709 12:21:11.650813 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:11.577342 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:11.571883   859 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.038s	sys 0.064s
I20260709 12:21:11.657912  1101 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:11.658907 31822 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:11.642354 31819 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:11.673887 31824 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:21:11.674582 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:11.653610 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.853s	user 0.008s	sys 0.024s
W20260709 12:21:11.785581 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.985s	user 0.009s	sys 0.024s
I20260709 12:21:11.675611 31826 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
W20260709 12:21:11.739471  1092 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.012s	user 0.008s	sys 0.028s
I20260709 12:21:11.671042   956 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:11.653467   939 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.087s	user 0.017s	sys 0.077s
I20260709 12:21:11.687618 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:11.710325 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 59624.4us
I20260709 12:21:11.544534  1076 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:11.663225  1081 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.742690 32371 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:11.669298 31677 rpcz_store.cc:276] 0709 12:21:09.873049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.873194 (+   145us) service_pool.cc:224] Handling call
0709 12:21:09.873493 (+   299us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:10.018646 (+145153us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 12:21:10.018694 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:10.019326 (+   632us) log.cc:844] Serialized 26 byte log entry
0709 12:21:10.939043 (+919717us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:10.939104 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.939208 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:11.606812 (+667604us) raft_consensus.cc:1685] finished
0709 12:21:11.606849 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:11.607030 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":144651,"raft.queue_time_us":147926,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":27685,"thread_start_us":919314,"threads_started":1,"wal-append.queue_time_us":1210639}
I20260709 12:21:11.630182   987 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:11.635031  1100 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:11.657958  1159 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:11.786056 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 985916us
W20260709 12:21:11.793474  1092 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.083s	user 0.008s	sys 0.029s
I20260709 12:21:11.800377 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1559) took 1584 ms. Trace:
I20260709 12:21:11.743880   993 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:11.699265  1098 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:11.805207 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 117691us
W20260709 12:21:11.764072   859 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.873s	user 0.039s	sys 0.065s
I20260709 12:21:11.810760 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:11.699733  1078 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:11.658070  1164 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:21:11.886461 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 87118.9us
I20260709 12:21:11.890794 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:11.901458 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:11.791028  1101 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:11.747802  1131 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.893023 31675 rpcz_store.cc:276] 0709 12:21:10.215619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.218956 (+  3337us) service_pool.cc:224] Handling call
0709 12:21:10.219415 (+   459us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:10.219708 (+   293us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:10.219751 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:10.308097 (+ 88346us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:21:10.319771 (+ 11674us) log.cc:844] Serialized 26 byte log entry
0709 12:21:10.984595 (+664824us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:10.984636 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:10.984714 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:11.792011 (+807297us) raft_consensus.cc:1685] finished
0709 12:21:11.792033 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:11.800293 (+  8260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":59980672,"thread_start_us":623525,"threads_started":1,"wal-append.queue_time_us":927068}
I20260709 12:21:11.748083  1091 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:21:11.911254 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 293037us
I20260709 12:21:11.813280  1159 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:11.911576 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 293946us
W20260709 12:21:11.891577  1149 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:21:11.813088  1100 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:11.920717 31676 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:21:11.813407  1157 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:11.896754  1136 ts_tablet_manager.cc:1916] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:11.919132  1131 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:11.948647   841 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:11.949334   961 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:21:11.949853 31818 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.582s	user 0.005s	sys 0.005s
I20260709 12:21:11.918090 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:11.802251   956 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:11.973254 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 55246.2us
I20260709 12:21:11.951550  1079 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:11.803421 32376 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
W20260709 12:21:11.956037 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 298909us
I20260709 12:21:11.903121   993 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:11.959017 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1607) took 1381 ms. Trace:
I20260709 12:21:11.906256  1114 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:11.907246  1164 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:11.909452  1113 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:11.970988 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:12.053164 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 82388.3us
I20260709 12:21:11.985129 31748 rpcz_store.cc:276] 0709 12:21:10.577239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.963929 (+386690us) service_pool.cc:224] Handling call
0709 12:21:10.964663 (+   734us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:10.964925 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:21:11.642974 (+678049us) log.cc:844] Serialized 14 byte log entry
0709 12:21:11.956402 (+313428us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:11.956455 (+    53us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:11.956501 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:11.956592 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:11.958936 (+  2344us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2316,"prepare.queue_time_us":44312,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"thread_start_us":988217,"threads_started":2}
I20260709 12:21:11.987138 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:11.988878 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:11.989650 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:11.813772  1081 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:12.055789 31675 raft_consensus.cc:2368] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:12.057085 31673 raft_consensus.cc:2368] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4.
I20260709 12:21:12.075380 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:11.973026 31825 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.823s	user 0.015s	sys 0.001s
W20260709 12:21:12.102810 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 127888us
I20260709 12:21:12.104467 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:12.106361 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:11.956137  1159 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:12.126067 32377 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:21:12.053421  1030 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.055s	user 0.015s	sys 0.000s
W20260709 12:21:12.141489 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 64365.3us
I20260709 12:21:11.998337   993 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:12.144011 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 330683us
I20260709 12:21:12.052104  1164 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:12.149009 32378 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:12.052066   841 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:11.960136   950 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.167s
I20260709 12:21:12.070315  1178 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:12.135667 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 488) took 1281 ms. Trace:
I20260709 12:21:12.109227 31825 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:12.052624   961 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:12.102988  1113 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:12.102259  1157 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:12.135667 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:12.137878  1179 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:21:12.157915 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 255826us
I20260709 12:21:12.147521 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1526) took 1354 ms. Trace:
I20260709 12:21:12.157662  1095 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:12.069947  1114 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:12.199602 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1532) took 1410 ms. Trace:
I20260709 12:21:12.137188  1105 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260709 12:21:12.146746  1030 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:12.211431 31820 rpcz_store.cc:276] 0709 12:21:10.789384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:11.331246 (+541862us) service_pool.cc:224] Handling call
0709 12:21:11.331643 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:11.331919 (+   276us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:21:11.800295 (+468376us) log.cc:844] Serialized 14 byte log entry
0709 12:21:12.199083 (+398788us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:12.199170 (+    87us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:12.199237 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:12.199333 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:12.199518 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8751,"prepare.queue_time_us":568379,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":68,"thread_start_us":718154,"threads_started":2}
I20260709 12:21:12.230333 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:12.162160   939 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 12:21:12.187938  1098 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:12.195480 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:12.243623 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:12.193043 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1545) took 1314 ms. Trace:
I20260709 12:21:12.197006 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1195) took 1593 ms. Trace:
I20260709 12:21:12.129052  1101 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:12.223829  1078 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:12.241011 31669 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.008s	sys 0.011s
I20260709 12:21:12.209751 31670 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
W20260709 12:21:12.229849  1137 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.005s	sys 0.011s
I20260709 12:21:12.175474 32374 rpcz_store.cc:276] 0709 12:21:10.854478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.858185 (+  3707us) service_pool.cc:224] Handling call
0709 12:21:12.087501 (+1229316us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b54006dd654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:12.125180 (+ 37679us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1023878016}
W20260709 12:21:12.283985  1152 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.473s	user 0.000s	sys 0.001s
I20260709 12:21:12.237401  1183 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:12.293548 31669 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.008s	sys 0.011s
I20260709 12:21:12.306360 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:12.199334  1157 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:12.308140 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:12.281299   987 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:12.291196  1148 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.444s	user 0.001s	sys 0.001s
I20260709 12:21:12.312819 31827 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260709 12:21:12.302001   939 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:12.313333 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 170) took 2051 ms. Trace:
I20260709 12:21:12.338253 31678 rpcz_store.cc:276] 0709 12:21:10.262293 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.742826 (+480533us) service_pool.cc:224] Handling call
0709 12:21:10.743283 (+   457us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:10.743958 (+   675us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:10.743990 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:10.744579 (+   589us) log.cc:844] Serialized 26 byte log entry
0709 12:21:11.471929 (+727350us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:11.471993 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:11.472096 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:12.313070 (+840974us) raft_consensus.cc:1685] finished
0709 12:21:12.313095 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:12.313255 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":106,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":112,"thread_start_us":726782,"threads_started":1,"wal-append.queue_time_us":927900}
I20260709 12:21:12.272399 31825 rpcz_store.cc:276] 0709 12:21:10.872571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:11.148445 (+275874us) service_pool.cc:224] Handling call
0709 12:21:12.186653 (+1038208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:12.313694  1150 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:12.298215 32369 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:12.293938  1137 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.005s	sys 0.011s
I20260709 12:21:12.315608 31826 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:12.307798 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:12.305845 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:12.297983 31671 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.949s	user 0.006s	sys 0.007s
I20260709 12:21:12.319159  1183 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:12.319912  1177 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.107s	user 0.000s	sys 0.001s
I20260709 12:21:12.320063   974 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:12.320971  1131 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:12.321396  1180 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:21:12.321264 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:12.232044   841 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:12.322710   961 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:12.317286  1101 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:12.381006 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:12.327931  1114 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:12.329805  1030 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:12.331715  1174 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.333s	user 0.001s	sys 0.000s
I20260709 12:21:12.277233 31744 rpcz_store.cc:276] 0709 12:21:10.604074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.967101 (+363027us) service_pool.cc:224] Handling call
0709 12:21:10.995515 (+ 28414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:11.341654 (+346139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:12.167526 (+825872us) spinlock_profiling.cc:243] Waited 255 ms on lock 0x7b54002f67d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:12.192792 (+ 25266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":587504896}
I20260709 12:21:12.344658 31824 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:12.314035  1078 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:12.317830  1188 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:12.323086  1190 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:12.386795  1188 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:12.366786 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:12.354545 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 211484us
I20260709 12:21:12.600450  1220 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:12.865512  1220 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:12.867574  1102 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:12.868076  1220 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:12.443804 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:13.152098 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 708352us
W20260709 12:21:12.468734 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 277050us
I20260709 12:21:12.414944 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:12.408625 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:12.541664 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:13.162405 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:13.163287 31744 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:13.164219 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:13.165010 31744 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:13.165839 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:12.510080 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:13.170133 31747 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:13.171212 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:12.410902 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:13.262110 31748 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:12.513532 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:12.510619 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:12.509845 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:12.492534 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:12.451256 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:13.291963   862 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:13.292313 31819 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:12.193506 31821 rpcz_store.cc:276] 0709 12:21:10.787724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.792261 (+  4537us) service_pool.cc:224] Handling call
0709 12:21:10.849445 (+ 57184us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:10.877987 (+ 28542us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:11.700435 (+822448us) log.cc:844] Serialized 14 byte log entry
0709 12:21:11.972401 (+271966us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:11.997180 (+ 24779us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:12.053090 (+ 55910us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:12.103167 (+ 50077us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:12.142248 (+ 39081us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":912587,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":184899,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":11411,"thread_start_us":939029,"threads_started":2,"wal-append.queue_time_us":485189}
I20260709 12:21:12.495535 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1525) took 1708 ms. Trace:
I20260709 12:21:12.365296 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1523) took 1578 ms. Trace:
I20260709 12:21:13.337314 31823 rpcz_store.cc:276] 0709 12:21:10.786760 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.789600 (+  2840us) service_pool.cc:224] Handling call
0709 12:21:10.805399 (+ 15799us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:10.805627 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:11.671158 (+865531us) log.cc:844] Serialized 14 byte log entry
0709 12:21:12.364753 (+693595us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:12.364850 (+    97us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:12.364925 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:12.365034 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:12.365228 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":983313,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":229390,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":136,"thread_start_us":1539857,"threads_started":3,"wal-append.queue_time_us":570443}
I20260709 12:21:13.339504 31823 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:12.588106 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:13.177625  1225 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:13.195405  1226 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:13.220467  1227 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:13.533726  1225 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:13.533881  1226 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.534456  1225 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:13.534397  1227 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.536505  1226 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.536756  1227 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.221581  1228 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:13.537516  1228 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:12.793977 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:13.238147 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:13.544862 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 306801us
I20260709 12:21:13.545989 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:12.866508 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:13.548019  1119 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.548558  1119 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:12.794554 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:12.349006 31671 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:13.552959 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:13.553355 31671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1568) took 2504 ms (client timeout 1648 ms). Trace:
I20260709 12:21:13.533696 31822 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:13.553800 31671 rpcz_store.cc:269] 0709 12:21:11.049069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:11.058960 (+  9891us) service_pool.cc:224] Handling call
0709 12:21:13.553192 (+2494232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:13.554922 31671 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:21:13.555544 31671 raft_consensus.cc:606] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Received request to transfer leadership to TS 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:13.555948 31671 raft_consensus.cc:614] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:12.804944 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:13.557642 31672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1120) took 1561 ms (client timeout 1838 ms). Trace:
W20260709 12:21:13.557996 31672 rpcz_store.cc:269] 0709 12:21:11.996172 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.089769 (+ 93597us) service_pool.cc:224] Handling call
0709 12:21:12.804756 (+714987us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:13.557581 (+752825us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28521856}
I20260709 12:21:13.189054 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:12.857785 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:13.277501 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1113) took 1289 ms. Trace:
I20260709 12:21:13.152451 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:13.298451 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:13.597049 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:13.597393 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 299014us
I20260709 12:21:13.219305 31750 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:13.599838 31750 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:13.600719 31750 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.602041 31750 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:13.175398 31746 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:13.603437 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:13.603852 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1202) took 1091 ms. Trace:
I20260709 12:21:13.604305 31746 rpcz_store.cc:276] 0709 12:21:12.512762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.513685 (+   923us) service_pool.cc:224] Handling call
0709 12:21:13.603778 (+1090093us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:12.512177 31749 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:21:13.606575  1226 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:13.606705 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1201) took 1095 ms. Trace:
I20260709 12:21:13.607185 31749 rpcz_store.cc:276] 0709 12:21:12.511342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.511524 (+   182us) service_pool.cc:224] Handling call
0709 12:21:12.511925 (+   401us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:13.606324 (+1094399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.606623 (+   299us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:13.607854  1232 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:13.608481  1232 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:13.597723 31678 raft_consensus.cc:2376] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3. Re-sending same reply.
I20260709 12:21:13.609397  1232 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:12.546073 31792 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.517s (SENT) [suppressed 5 similar messages]
W20260709 12:21:13.291652 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 799122us
I20260709 12:21:13.617828 31791 leader_election.cc:400] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:13.618156 31791 leader_election.cc:403] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:13.336650 31818 rpcz_store.cc:276] 0709 12:21:10.787407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.792301 (+  4894us) service_pool.cc:224] Handling call
0709 12:21:11.136391 (+344090us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:11.136612 (+   221us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:11.136924 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:12.052387 (+915463us) log.cc:844] Serialized 14 byte log entry
0709 12:21:12.450196 (+397809us) log.cc:844] Serialized 14 byte log entry
0709 12:21:12.494007 (+ 43811us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:12.495093 (+  1086us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:12.495170 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:12.495277 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:12.495447 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23323,"prepare.queue_time_us":67868,"prepare.run_cpu_time_us":88,"prepare.run_wall_time_us":4893,"raft.queue_time_us":28821,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":343,"spinlock_wait_cycles":6704512,"thread_start_us":244912,"threads_started":1,"wal-append.queue_time_us":434834}
I20260709 12:21:13.621800 31818 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:13.623678   862 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:12.365859 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:13.627014 31825 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:12.444389 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:13.286314 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:13.303761 31819 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:13.636581 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1661) took 1268 ms. Trace:
I20260709 12:21:13.637009 31819 rpcz_store.cc:276] 0709 12:21:12.368009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.371212 (+  3203us) service_pool.cc:224] Handling call
0709 12:21:13.636503 (+1265291us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:13.638684 31819 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:21:13.639436 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1669) took 1155 ms. Trace:
I20260709 12:21:13.639853  1226 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:13.639784 31819 rpcz_store.cc:276] 0709 12:21:12.483980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.637919 (+1153939us) service_pool.cc:224] Handling call
0709 12:21:13.638372 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:13.639107 (+   735us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.639373 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:13.640950  1226 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:12.628083 31824 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:12.629477 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:12.482075 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:12.505568 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.010s	sys 0.019s
I20260709 12:21:12.790391 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 350) took 2052 ms. Trace:
I20260709 12:21:13.651676 32371 rpcz_store.cc:276] 0709 12:21:10.738373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.739002 (+   629us) service_pool.cc:224] Handling call
0709 12:21:10.739473 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:11.812847 (+1073374us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:12.790048 (+977201us) spinlock_profiling.cc:243] Waited 937 ms on lock 0x7b54006473d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:12.790307 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2154446208}
I20260709 12:21:13.652478 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:13.190413 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:13.223727 32376 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:13.655118  1171 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:12.347786   678 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.658824   678 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.662035   678 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:12.513576   799 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.654551 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:13.666266   799 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.667238 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:13.668756   799 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:12.484452   859 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.008s	sys 0.005s
W20260709 12:21:13.671788   859 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.724s	user 0.010s	sys 0.006s
I20260709 12:21:13.671164 31825 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:12.513557   961 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:13.673686   961 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.254369   972 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.675562   972 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:13.676241 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 1.13011e+06us
W20260709 12:21:13.677495 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:13.678210   972 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:12.335711   987 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:13.680807   987 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:12.476006  1086 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.836s	user 0.012s	sys 0.002s
W20260709 12:21:12.490000  1091 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.000s	sys 0.014s
W20260709 12:21:13.685592  1091 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.713s	user 0.001s	sys 0.015s
I20260709 12:21:13.685935  1091 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:13.686774   961 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:13.686926  1091 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.683250 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:12.314172  1113 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:12.494138  1114 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.695940  1114 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:13.697269 31699 debug-util.cc:398] Leaking SignalData structure 0x7b0800177b40 after lost signal to thread 30503
W20260709 12:21:13.697775 31699 debug-util.cc:398] Leaking SignalData structure 0x7b08001e81c0 after lost signal to thread 30506
I20260709 12:21:13.697954  1248 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:12.500561   993 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.701088  1114 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:12.383293  1030 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:13.711931  1030 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:12.596742  1136 ts_tablet_manager.cc:1929] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260709 12:21:13.718168  1136 log.cc:1199] T fc47871ef692418388113c1d6c88b262 P 039d66a6c85e43c3936bf5017501b344: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/wals/fc47871ef692418388113c1d6c88b262
I20260709 12:21:12.344228  1150 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:12.374269  1170 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.721330  1170 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.723817  1170 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:12.366436  1181 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:13.725237  1181 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:12.390116  1188 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:12.540836  1192 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.164s	user 0.001s	sys 0.000s
I20260709 12:21:13.732059  1249 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.194253  1197 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:13.734513  1197 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.737083  1197 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:12.833350  1198 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:13.747020  1198 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.298243  1209 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:12.561964  1211 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:13.756062  1211 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.758323  1211 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.132230  1220 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.235608   932 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:13.759795  1220 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.760408   932 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:12.337997   939 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:13.764312   939 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.765565   939 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:12.446017   974 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:13.768127   974 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:13.768411  1251 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.768920  1251 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.769068   974 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.770238   974 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:12.568971   980 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:13.771469   980 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:13.772332   980 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.773584   980 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:13.776154  1251 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:13.039214  1005 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:13.211345  1076 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:13.779297  1005 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.779773  1076 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.230715  1078 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.132754  1095 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.783383  1095 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.039876  1102 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:13.787503  1102 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:12.594856  1176 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.416s	user 0.004s	sys 0.000s
I20260709 12:21:12.481793  1189 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:13.016600  1196 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:13.795454  1196 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.797610  1196 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:13.264714 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 355) took 2133 ms. Trace:
I20260709 12:21:13.800005 32369 rpcz_store.cc:276] 0709 12:21:11.131167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:11.134825 (+  3658us) service_pool.cc:224] Handling call
0709 12:21:11.135314 (+   489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:12.347087 (+1211773us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.264303 (+917216us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:13.264594 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2639209344}
I20260709 12:21:13.312891 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:13.180963 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:13.248754   946 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:13.299608   950 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.808274   950 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:21:13.809864 30508 debug-util.cc:398] Leaking SignalData structure 0x7b08003a7020 after lost signal to thread 31792
I20260709 12:21:13.133425  1075 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:12.548262  1079 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:12.367393  1081 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:12.592017  1100 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.623s	user 0.012s	sys 0.000s
W20260709 12:21:13.332018  1217 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.771s	user 0.003s	sys 0.002s
I20260709 12:21:12.374930  1159 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:12.583920  1164 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:12.833484  1173 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:12.496053  1182 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 12:21:12.579929  1183 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:12.495705  1193 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 12:21:13.192970  1194 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.274389  1204 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:13.149317  1212 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.586s	user 0.001s	sys 0.001s
I20260709 12:21:12.468066  1179 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:13.340745 31821 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:13.560978  1228 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:13.562821  1119 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:13.564165 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:13.566964 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:13.571739   931 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:13.578394  1225 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.579491  1236 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:13.584611 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:13.585105 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:13.577376 31822 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:13.593123  1235 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:13.593710 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 727228us
I20260709 12:21:13.594908 31677 rpcz_store.cc:276] 0709 12:21:11.987886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:11.989260 (+  1374us) service_pool.cc:224] Handling call
0709 12:21:13.276572 (+1287312us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:13.276838 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:13.277066 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:13.277103 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:13.277157 (+    54us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:13.277203 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.277266 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:13.277433 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":33682816}
W20260709 12:21:13.191946 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:13.610970  1178 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:13.611414 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:13.625034 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 201) took 1212 ms. Trace:
I20260709 12:21:13.632431 31826 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 4 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:13.632683 31820 raft_consensus.cc:1813] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Handling vote request from an unknown peer b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:13.634450  1200 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 5 current term: 5 is higher.
I20260709 12:21:13.641770 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1670) took 1154 ms. Trace:
I20260709 12:21:13.643606 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1668) took 1166 ms. Trace:
I20260709 12:21:13.649924 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1664) took 1203 ms. Trace:
I20260709 12:21:12.599834 32372 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:13.653594 31827 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1662) took 1281 ms (client timeout 1705 ms). Trace:
W20260709 12:21:13.654476 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 1.17254e+06us
I20260709 12:21:13.645503 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:13.663893   678 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:13.666776   862 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:13.668941 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:13.669713  1246 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:12.442989   956 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:13.672740 31825 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1659) took 1363 ms (client timeout 1781 ms). Trace:
I20260709 12:21:13.680122 31753 raft_consensus.cc:2368] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:13.665134 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:13.683780   987 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.684566  1086 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.698516  1248 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:13.699100 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:13.698869 31699 debug-util.cc:398] Leaking SignalData structure 0x7b08001da9c0 after lost signal to thread 31610
I20260709 12:21:13.704532 31751 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:13.703442   993 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.716267  1030 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.716629  1114 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:13.721035  1150 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.728578  1136 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45150 (request call id 75) took 3750 ms. Trace:
I20260709 12:21:13.732695  1249 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:12.517939  1157 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:13.741245 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1578) took 1366 ms. Trace:
I20260709 12:21:13.745364  1181 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:13.752521  1209 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.754182  1198 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:12.521648  1213 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:13.762697  1220 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:13.781644  1005 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.781669  1076 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:13.762356   932 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:13.806468   946 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.860100  1204 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:13.651249 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.742s	user 0.014s	sys 0.019s
W20260709 12:21:13.875535 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 1.97406e+06us
I20260709 12:21:13.876433  1173 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.877094   946 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.794449  1189 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.861958  1248 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:13.881439  1081 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:13.882897  1179 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.887058 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:13.887979  1209 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:13.888386  1189 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.889539  1236 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:13.890079  1236 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:13.890118  1079 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.891125  1236 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.892445  1236 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:13.892550  1075 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.894796 31744 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:13.895300  1213 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.897737  1213 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.862779   678 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:13.898856   678 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:13.899201 31677 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:13.860508 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:13.865778  1136 rpcz_store.cc:276] 0709 12:21:09.978478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:09.978802 (+   324us) service_pool.cc:224] Handling call
0709 12:21:11.065235 (+1086433us) ts_tablet_manager.cc:1151] Deleting tablet fc47871ef692418388113c1d6c88b262
0709 12:21:11.151755 (+ 86520us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:21:12.569004 (+1417249us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:12.596535 (+ 27531us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:13.728484 (+1131949us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41324,"tablet-delete.queue_time_us":1057125,"thread_start_us":826112,"threads_started":1}
I20260709 12:21:13.900746 31678 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:13.901749 31561 catalog_manager.cc:5028] TS 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): tablet fc47871ef692418388113c1d6c88b262 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:21:13.902066 31678 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:13.863628 31750 rpcz_store.cc:276] 0709 12:21:12.412912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.413263 (+   351us) service_pool.cc:224] Handling call
0709 12:21:12.413768 (+   505us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:13.219157 (+805389us) spinlock_profiling.cc:243] Waited 53.6 ms on lock 0x7b54004d0ba0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:21:13.624644 (+405487us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:13.624680 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:13.624729 (+    49us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:13.624762 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.624843 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:13.624973 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6578,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"spinlock_wait_cycles":123312384,"thread_start_us":3697,"threads_started":1}
W20260709 12:21:13.864684 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:13.902983 31753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1200) took 1393 ms (client timeout 1822 ms). Trace:
I20260709 12:21:13.863953 31823 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:13.903659 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1206) took 1388 ms. Trace:
I20260709 12:21:13.904145   993 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:13.904587 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1667) took 1432 ms. Trace:
I20260709 12:21:13.904690 31750 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:13.905050 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:13.904958 31823 rpcz_store.cc:276] 0709 12:21:12.472727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.347217 (+874490us) service_pool.cc:224] Handling call
0709 12:21:13.347510 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:13.903841 (+556331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.904379 (+   538us) spinlock_profiling.cc:243] Waited 516 ms on lock 0x7b540030e8d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:13.904534 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1187300224}
I20260709 12:21:13.905681 31749 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:13.906175 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:13.862341 31827 rpcz_store.cc:269] 0709 12:21:12.371921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.372388 (+   467us) service_pool.cc:224] Handling call
0709 12:21:12.629312 (+256924us) spinlock_profiling.cc:243] Waited 6.68 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:13.653536 (+1024224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15361920}
I20260709 12:21:13.907213 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:13.904695   993 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:13.908483 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:13.908282   993 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.909193 31827 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:13.909972 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:13.897754 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1203) took 1384 ms. Trace:
I20260709 12:21:13.911049 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:13.911772 31748 raft_consensus.cc:2368] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3.
W20260709 12:21:13.867060 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1675) took 1335 ms (client timeout 1595 ms). Trace:
I20260709 12:21:13.866587 31673 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
W20260709 12:21:13.912256 31822 rpcz_store.cc:269] 0709 12:21:12.532043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.587776 (+ 55733us) service_pool.cc:224] Handling call
0709 12:21:13.866993 (+1279217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:13.904170 31747 rpcz_store.cc:276] 0709 12:21:12.514795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.172074 (+657279us) service_pool.cc:224] Handling call
0709 12:21:13.172396 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:13.902863 (+730467us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:13.902893 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:13.902933 (+    40us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:13.902960 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.903457 (+   497us) spinlock_profiling.cc:243] Waited 730 ms on lock 0x7b5400064354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:13.903499 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:13.903607 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1679381760}
I20260709 12:21:13.868757 31820 raft_consensus.cc:2410] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 4 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:21:13.913548 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1673) took 1404 ms. Trace:
W20260709 12:21:13.865350 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:13.862602  1225 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:13.914242 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:13.914187 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 268765us
I20260709 12:21:13.914564 31823 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:13.862151  1100 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.872174  1178 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:13.872973 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1567) took 2824 ms. Trace:
I20260709 12:21:13.867986 31824 rpcz_store.cc:276] 0709 12:21:12.446563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.448494 (+  1931us) service_pool.cc:224] Handling call
0709 12:21:12.627577 (+179083us) spinlock_profiling.cc:243] Waited 6.78 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:12.627845 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:13.649620 (+1021775us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.649861 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15592192}
I20260709 12:21:13.934260 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:13.873775 31674 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:21:13.874039  1249 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:13.867789 31818 rpcz_store.cc:276] 0709 12:21:12.477005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.621016 (+1144011us) service_pool.cc:224] Handling call
0709 12:21:13.621502 (+   486us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:13.622377 (+   875us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.643541 (+ 21164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:13.866791 31819 rpcz_store.cc:276] 0709 12:21:12.487383 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.640801 (+1153418us) service_pool.cc:224] Handling call
0709 12:21:13.641181 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:13.641388 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:13.641426 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:13.641478 (+    52us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:13.641517 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.641583 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:13.641710 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:13.810530   950 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:21:13.844434  1102 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.786021  1095 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:13.874608  1194 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.878746  1173 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.889662  1150 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:13.894837  1075 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.901829 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1665) took 1448 ms. Trace:
W20260709 12:21:13.903422 31753 rpcz_store.cc:269] 0709 12:21:12.510001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.510390 (+   389us) service_pool.cc:224] Handling call
0709 12:21:13.902819 (+1392429us) spinlock_profiling.cc:243] Waited 395 ms on lock 0x7b54004047d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:13.902933 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":909505408}
I20260709 12:21:13.942979   939 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:13.782457  1078 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:13.912853 31748 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:13.864565 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 195675us
I20260709 12:21:13.975502 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:13.975847 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:13.977399 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:13.979861 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:13.863209 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 198413us
I20260709 12:21:13.871799 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1660) took 1534 ms. Trace:
I20260709 12:21:13.981014 31826 rpcz_store.cc:276] 0709 12:21:12.349837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.367432 (+ 17595us) service_pool.cc:224] Handling call
0709 12:21:13.871732 (+1504300us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:13.913954 31820 rpcz_store.cc:276] 0709 12:21:12.512085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.286048 (+773963us) service_pool.cc:224] Handling call
0709 12:21:13.913491 (+627443us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:13.982338 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:13.982394 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:13.983108 31673 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:13.869166 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:13.864705 31675 rpcz_store.cc:276] 0709 12:21:12.374582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.258995 (+884413us) service_pool.cc:224] Handling call
0709 12:21:13.259350 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:13.737671 (+478321us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:13.737706 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:13.737750 (+    44us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:13.737776 (+    26us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.740855 (+  3079us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b540017d9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:13.740937 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:13.741147 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1099471232}
I20260709 12:21:13.984579 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:13.985368 31675 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:13.985479 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:13.870683 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:13.916647 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:13.987012 31823 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:13.987452 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 116852us
W20260709 12:21:13.871968 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 287448us
I20260709 12:21:13.990245 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:13.990983  1246 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:13.991408  1246 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:13.991762  1246 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:13.991891  1078 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:13.992484  1220 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:13.992897   946 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:13.993072 31670 raft_consensus.cc:2368] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 3. Current term is 4.
I20260709 12:21:13.912396 31746 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:13.994279 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 175) took 1200 ms. Trace:
I20260709 12:21:13.959374  1005 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260709 12:21:13.994746 31670 rpcz_store.cc:276] 0709 12:21:12.793695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.552666 (+758971us) service_pool.cc:224] Handling call
0709 12:21:13.994046 (+441380us) spinlock_profiling.cc:243] Waited 399 ms on lock 0x7b54002e8454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:13.994217 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":916741888}
I20260709 12:21:13.953504  1248 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:13.995997 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:13.899235 31699 debug-util.cc:398] Leaking SignalData structure 0x7b08003a1e40 after lost signal to thread 31702
I20260709 12:21:13.922447 32378 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:13.888796  1119 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 12:21:13.923996  1086 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:13.924234  1164 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:13.863070  1159 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:13.913241 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:14.047509 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.047853 31822 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.049801 31673 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:14.052174 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.053042 31673 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:13.871245   980 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:13.903623 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 491) took 1797 ms. Trace:
I20260709 12:21:14.058831 32372 rpcz_store.cc:276] 0709 12:21:12.106250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.126445 (+ 20195us) service_pool.cc:224] Handling call
0709 12:21:12.192467 (+ 66022us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:13.871587 (+1679120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.898571 (+ 26984us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b5400595454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:13.903556 (+  4985us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":936248064}
I20260709 12:21:13.892479  1079 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:13.933050  1178 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.035902  1075 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.031s
I20260709 12:21:14.066911 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.068451 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.049706 31670 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.035676  1114 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:14.070205 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:13.936571 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:14.071887 31699 debug-util.cc:398] Leaking SignalData structure 0x7b08003a6a40 after lost signal to thread 31777
I20260709 12:21:14.072258 31824 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
W20260709 12:21:14.072448 31699 debug-util.cc:398] Leaking SignalData structure 0x7b080010c580 after lost signal to thread 31778
I20260709 12:21:14.073407 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:13.936820 31749 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
W20260709 12:21:14.074108 31699 debug-util.cc:398] Leaking SignalData structure 0x7b080032e640 after lost signal to thread 31851
I20260709 12:21:14.074913 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.076022 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.077055 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:13.931941 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:21:14.079252 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 147426us
I20260709 12:21:14.079909 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:14.080672 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:13.944272 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1114) took 1956 ms. Trace:
I20260709 12:21:14.082839   987 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:14.083264   987 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.083624   987 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:14.086858  1030 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:14.087436  1030 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:13.911053 31751 rpcz_store.cc:276] 0709 12:21:12.513133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.513261 (+   128us) service_pool.cc:224] Handling call
0709 12:21:13.897556 (+1384295us) spinlock_profiling.cc:243] Waited 423 ms on lock 0x7b54002f62d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:13.897694 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":971912576}
I20260709 12:21:13.942516 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:13.935601 31572 catalog_manager.cc:5697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 2 to 4, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 4 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:13.956804 31745 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:13.958580 31642 leader_election.cc:400] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3.
I20260709 12:21:14.089689 31642 leader_election.cc:403] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:21:13.865128 31825 rpcz_store.cc:269] 0709 12:21:12.309726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.346278 (+ 36552us) service_pool.cc:224] Handling call
0709 12:21:13.672667 (+1326389us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:14.089956 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 147554us
I20260709 12:21:13.985600 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:13.986176  1249 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:13.910522 31827 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:13.989352  1189 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:13.921543 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:13.996675 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:14.114246 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:13.999030 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.001708 31744 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:14.003544  1194 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:14.003952 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.003679   931 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:14.004241 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:13.974404 31821 rpcz_store.cc:276] 0709 12:21:12.453737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.334842 (+881105us) service_pool.cc:224] Handling call
0709 12:21:13.335232 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:13.901550 (+566318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.901769 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:14.004873  1102 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:14.005481   862 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:14.006122 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:13.946826  1213 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 12:21:13.921778  1204 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:14.009308  1179 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:14.098418   961 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:14.011195   939 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:14.153113  1150 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:14.013963 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:13.942116   956 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.043s
I20260709 12:21:14.163869  1150 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:14.164259 31676 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:14.014071 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:13.986390 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:13.933684 31669 rpcz_store.cc:276] 0709 12:21:11.048613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:11.058774 (+ 10161us) service_pool.cc:224] Handling call
0709 12:21:11.642710 (+583936us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:11.642988 (+   278us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:11.643464 (+   476us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:11.643509 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:11.644398 (+   889us) log.cc:844] Serialized 52 byte log entry
0709 12:21:12.319230 (+674832us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:12.319305 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:12.319398 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:13.872633 (+1553235us) raft_consensus.cc:1685] finished
0709 12:21:13.872665 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:13.872893 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":28040,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":69138,"spinlock_wait_cycles":36327168,"thread_start_us":670323,"threads_started":1,"wal-append.queue_time_us":700993}
I20260709 12:21:14.165545 31676 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:14.018965 31643 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:14.019398 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.133411  1179 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:14.163921 31673 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:14.020733   932 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:14.022836  1220 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:14.168143   932 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:14.167563  1179 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:14.023094  1078 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:14.024938 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:14.027420 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47651.1us
I20260709 12:21:14.177958 31791 leader_election.cc:400] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4.
I20260709 12:21:14.028699 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:14.035300 31823 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:14.178380 31791 leader_election.cc:403] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:21:14.178341 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.047066  1246 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:13.850034 31573 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42072 (request call id 238) took 13368 ms (client timeout 15000 ms). Trace:
I20260709 12:21:14.179425 31678 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:14.179996 31791 leader_election.cc:400] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:14.180459 31791 leader_election.cc:403] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
W20260709 12:21:14.090157 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 136935us
I20260709 12:21:14.082052 31674 rpcz_store.cc:276] 0709 12:21:11.988187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:11.989338 (+  1151us) service_pool.cc:224] Handling call
0709 12:21:13.873524 (+1884186us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:13.935649 (+ 62125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:13.944082 (+  8433us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b540022a950. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:13.944218 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4062465280}
I20260709 12:21:14.092200 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:14.074182 31824 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:14.078452 31749 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260709 12:21:14.185832 31749 pending_rounds.cc:85] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 1
I20260709 12:21:14.186265 31749 pending_rounds.cc:107] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 12:21:14.186580 31749 raft_consensus.cc:2889] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:14.093011 31822 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:14.169077  1179 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.189265 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:14.189913 31822 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.189946 31753 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:14.190977 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.124653 31746 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:14.191529  1246 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:14.191807 31751 raft_consensus.cc:2410] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:14.153913   961 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.095966 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:14.192385 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.192565   961 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:14.112725 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:14.193149 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 97259.3us
I20260709 12:21:14.169018  1078 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:14.194423 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:14.195397  1078 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:14.196482 31561 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f6404cb9c2174326b06cbbb0d27d43f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b95aeeaf4c51442b8092cff1de674f2b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:14.113972 32374 raft_consensus.cc:2368] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4.
I20260709 12:21:14.114420 31819 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:14.197754 32376 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:14.198710   799 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:14.198813 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.199251   799 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:14.199463 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.200464 31827 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:14.200785 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.197800  1249 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:14.201444 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.201989 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.090137 31572 catalog_manager.cc:5697] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3, now has a pending config: VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194), VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195), VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:21:14.093506 31699 debug-util.cc:398] Leaking SignalData structure 0x7b08003d0260 after lost signal to thread 31852
W20260709 12:21:14.203722 31699 debug-util.cc:398] Leaking SignalData structure 0x7b08003341e0 after lost signal to thread 32402
I20260709 12:21:14.203390 31572 catalog_manager.cc:5697] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 2, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 2 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:14.204869 31572 catalog_manager.cc:5697] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:13.997327  1248 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:14.120635 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.207715   987 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:14.011101 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.113567 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:14.163160   939 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:14.002357 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.206105 31572 catalog_manager.cc:5697] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 3 to 5, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 5 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:14.167757  1220 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:14.166374 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 147547us
I20260709 12:21:14.027508 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:14.165174  1150 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:14.173693 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 184605us
I20260709 12:21:14.230443 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:14.044682 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.196667  1079 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4.
I20260709 12:21:14.197199  1189 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:14.199661   799 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.202215 31827 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:14.202029 31824 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
W20260709 12:21:14.208498 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 116385us
I20260709 12:21:14.115104 31825 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:14.209491 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 180850us
I20260709 12:21:14.105485  1075 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:14.119940 31752 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:14.193042 31823 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:14.210038 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:14.215148 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.217373 31572 catalog_manager.cc:5697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 3 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:14.292307 31820 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:14.293181 31572 catalog_manager.cc:5697] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 3 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:14.032287   678 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:14.088344  1030 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:14.250531 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:14.234550 31822 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:14.163160 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.228971 32375 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
W20260709 12:21:14.294902 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.301860 31820 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:21:14.302836 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1672) took 1811 ms. Trace:
I20260709 12:21:14.303517 31827 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:14.303391 31820 rpcz_store.cc:276] 0709 12:21:12.491908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.300952 (+1809044us) service_pool.cc:224] Handling call
0709 12:21:14.301391 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:14.302483 (+  1092us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.302766 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:14.305353 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1671) took 1816 ms. Trace:
I20260709 12:21:14.305768 31823 rpcz_store.cc:276] 0709 12:21:12.489015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.304572 (+1815557us) service_pool.cc:224] Handling call
0709 12:21:14.304866 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:14.305038 (+   172us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:14.305062 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:14.305102 (+    40us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:14.305133 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.305187 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:14.305299 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:14.258744 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:14.296731 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 86764.6us
I20260709 12:21:14.296916 31572 catalog_manager.cc:5697] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:21:14.315347 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 64888.4us
I20260709 12:21:14.319820 31572 catalog_manager.cc:5697] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:14.321400 31572 catalog_manager.cc:5697] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 2, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:14.211428   956 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:14.325598 31718 leader_election.cc:400] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4.
I20260709 12:21:14.227512  1204 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:14.290057  1248 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.292707 32372 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:14.338383 31822 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:14.340353 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.347621 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:14.291276  1213 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:14.412218 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.418124 31824 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:14.356563 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.419731 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:14.420396 31751 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
W20260709 12:21:14.179554 31573 rpcz_store.cc:269] 0709 12:21:00.482075 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:00.487252 (+  5177us) service_pool.cc:224] Handling call
0709 12:21:00.774134 (+286882us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54002828d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3f090 00007fe8fcdddf3a 00007fe902b93e9c 00007fe902c9204e 00007fe9011532a4 00007fe901153232 00007fe901153161 00007fe9011530dc 00007fe901152392 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:13.849933 (+13075799us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:11.092507 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:11.345493 (+252986us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:11.348700 (+  3207us) write_op.cc:435] Acquired schema lock
0709 12:21:11.351867 (+  3167us) tablet.cc:662] Decoding operations
0709 12:21:11.396236 (+ 44369us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:11.403742 (+  7506us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:11.405836 (+  2094us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:21:11.475961 (+ 70125us) tablet.cc:701] Row locks acquired
0709 12:21:11.478824 (+  2863us) write_op.cc:260] PREPARE: finished
0709 12:21:11.525306 (+ 46482us) write_op.cc:270] Start()
0709 12:21:11.547416 (+ 22110us) write_op.cc:276] Timestamp: P: 1783599671525239 usec, L: 0
0709 12:21:11.547457 (+    41us) op_driver.cc:348] REPLICATION: starting
0709 12:21:11.584536 (+ 37079us) log.cc:844] Serialized 935 byte log entry
0709 12:21:13.746740 (+2162204us) op_driver.cc:464] REPLICATION: finished
0709 12:21:13.765627 (+ 18887us) write_op.cc:301] APPLY: starting
0709 12:21:13.765728 (+   101us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:13.765848 (+   120us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:13.771669 (+  5821us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:13.772079 (+   410us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:13.772243 (+   164us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0709 12:21:13.772286 (+    43us) write_op.cc:312] APPLY: finished
0709 12:21:13.772513 (+   227us) log.cc:844] Serialized 29 byte log entry
0709 12:21:13.773505 (+   992us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:13.773594 (+    89us) write_op.cc:454] Released schema lock
0709 12:21:13.773635 (+    41us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:13.774662 (+  1027us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:13.774692 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":150808,"reported_tablets":8,"spinlock_wait_cycles":265990016,"child_traces":[["op",{"apply.queue_time_us":18631,"apply.run_cpu_time_us":1960,"apply.run_wall_time_us":8980,"num_ops":2,"prepare.queue_time_us":998472,"prepare.run_cpu_time_us":24402,"prepare.run_wall_time_us":1303716,"raft.queue_time_us":13962,"raft.run_cpu_time_us":1824,"raft.run_wall_time_us":3971,"replication_time_us":2199079,"spinlock_wait_cycles":2983168,"thread_start_us":1183381,"threads_started":4,"wal-append.queue_time_us":744222}]]}
I20260709 12:21:14.415612 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.356020 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.417080 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.421157 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:14.423559 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 76043.5us
I20260709 12:21:14.441617 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.447222 31824 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:14.447377 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.448663 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.407238 31718 leader_election.cc:403] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:14.326566 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:14.450857 31749 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:14.454680 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 224310us
I20260709 12:21:14.448920 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:14.463985 31825 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:14.328598 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1676) took 1786 ms. Trace:
I20260709 12:21:14.520820 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1674) took 1999 ms. Trace:
I20260709 12:21:14.521014 31827 rpcz_store.cc:276] 0709 12:21:12.542716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.303055 (+1760339us) service_pool.cc:224] Handling call
0709 12:21:14.303332 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:14.319540 (+ 16208us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.328526 (+  8986us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:14.521200 31825 rpcz_store.cc:276] 0709 12:21:12.527327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.301004 (+1773677us) service_pool.cc:224] Handling call
0709 12:21:14.463207 (+162203us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:14.463509 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:14.520550 (+ 57041us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.520763 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2847616}
I20260709 12:21:14.431834 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:14.390810 31745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 202) took 1975 ms (client timeout 2102 ms). Trace:
I20260709 12:21:14.356096  1005 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.122s
W20260709 12:21:14.403160  1236 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.510s	user 0.005s	sys 0.006s
I20260709 12:21:14.531191  1173 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.539418  1173 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:14.540540  1264 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.132s	user 0.000s	sys 0.000s
I20260709 12:21:14.542963 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:14.502206 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 175706us
I20260709 12:21:14.543670 31825 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:14.544128 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:14.544512 31825 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fc47871ef692418388113c1d6c88b262"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
mode: GRACEFUL
new_leader_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:41934
I20260709 12:21:14.363458  1075 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.545468  1075 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:14.546921   993 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:14.547497   993 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:14.547808  1075 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:14.549160 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.331772 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:14.378981 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.515669 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:14.549247 31748 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:14.350555  1246 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.351336   932 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:21:14.374733 31607 debug-util.cc:398] Leaking SignalData structure 0x7b080019e380 after lost signal to thread 30503
I20260709 12:21:14.326762   859 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.230s
I20260709 12:21:14.377867  1100 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:14.571094 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.356675  1194 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.197s
I20260709 12:21:14.537575 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:14.393211 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.426833 31572 catalog_manager.cc:5697] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 3 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:14.417912  1213 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:14.438922  1235 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:14.339488 31818 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:14.558035 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.559156 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.623909 31677 raft_consensus.cc:2410] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260709 12:21:14.622380 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 308817us
I20260709 12:21:14.625141 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.315979   931 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.201s
I20260709 12:21:14.560535 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.522929 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:14.642127 31824 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:14.401697 31670 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:14.367503 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:14.567718 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:14.532083 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 206633us
I20260709 12:21:14.649062 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:21:14.555193  1261 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.190s	user 0.000s	sys 0.001s
I20260709 12:21:14.575155 32375 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260709 12:21:14.576812 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:14.531235  1113 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:14.554239  1204 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 4
W20260709 12:21:14.564775 31607 debug-util.cc:398] Leaking SignalData structure 0x7b080019e180 after lost signal to thread 30506
I20260709 12:21:14.554801  1266 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:14.605274 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.565229   980 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:14.642879  1178 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.582s	user 0.010s	sys 0.000s
I20260709 12:21:14.514609  1248 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:14.608589 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:14.691051 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.660135 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:21:14.691172 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 171134us
I20260709 12:21:14.564535  1170 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.565692  1173 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:14.622879 31672 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:14.626034 31572 catalog_manager.cc:5697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:14.696074 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:14.645187 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.605630  1213 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.647322 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:14.701653 31824 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:14.531154 31745 rpcz_store.cc:269] 0709 12:21:12.414097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:12.414443 (+   346us) service_pool.cc:224] Handling call
0709 12:21:14.381066 (+1966623us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b54004042d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:14.388571 (+  7505us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1444974464}
I20260709 12:21:14.565414  1246 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:14.604775   987 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:14.548367   993 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:14.610773 31747 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.015s	sys 0.006s
W20260709 12:21:14.708840 31747 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.015s	sys 0.009s
I20260709 12:21:14.655068 31675 raft_consensus.cc:2410] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:14.653005 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:14.687067 31678 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:14.688390 31749 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:14.650446  1119 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3.
I20260709 12:21:14.624401 31821 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:14.572955   932 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:14.645995 31669 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:14.642254   678 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:14.648185  1235 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:14.684115  1200 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:14.683802 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:14.715833 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:14.717738 31572 catalog_manager.cc:5697] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:14.759965 31572 catalog_manager.cc:5697] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:14.685278 31819 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
W20260709 12:21:14.724831 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:14.764430 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.737999 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.701372  1078 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:14.772467  1078 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:14.759855 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:14.772998  1078 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:14.773887  1263 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.122s	user 0.001s	sys 0.000s
I20260709 12:21:14.690505  1113 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:14.689546  1189 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:14.738773 31824 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
W20260709 12:21:14.735416 32369 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.001s	sys 0.034s
W20260709 12:21:14.735090 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08000361e0 after lost signal to thread 31610
I20260709 12:21:14.778565 31821 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [3.4-3.4]
I20260709 12:21:14.782702 31745 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:21:14.782543   932 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:14.784098 31745 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
W20260709 12:21:14.785497   961 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.592s	user 0.011s	sys 0.000s
I20260709 12:21:14.785670 31669 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:14.777591  1113 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:14.759253  1235 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:14.744184 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.794545 31671 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:14.689990  1178 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:14.743950 31749 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:14.705616  1170 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:14.761736 31572 catalog_manager.cc:5697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:14.769001 31672 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:14.809049 31672 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:14.778792 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.779714 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:14.779845 32369 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.002s	sys 0.035s
I20260709 12:21:14.758967  1200 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:21:14.744376 32370 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.027s	sys 0.035s
I20260709 12:21:14.766394  1249 raft_consensus.cc:2764] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:21:14.775208  1194 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:14.811074 31572 catalog_manager.cc:5697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:14.814818 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 289747us
W20260709 12:21:14.809544 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 286181us
I20260709 12:21:14.817338 31825 raft_consensus.cc:606] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Received request to transfer leadership to TS 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:14.819239 31825 raft_consensus.cc:614] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:21:14.820307 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:14.825578 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:14.827793 31673 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:14.818882 31572 catalog_manager.cc:5697] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 2 to 4, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 4 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:14.786223   961 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:14.832167 31572 catalog_manager.cc:5697] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 3 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:14.835858 31820 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:14.832526 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:14.816608 32370 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.027s	sys 0.035s
I20260709 12:21:14.842780 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:14.846829 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:14.846649 31572 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 3 to 4, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 4 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:14.849403 31572 catalog_manager.cc:5697] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:14.814086 31818 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:14.854029 31678 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:21:14.854668 31678 pending_rounds.cc:85] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 2
I20260709 12:21:14.854964 31678 pending_rounds.cc:107] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:21:14.855271 31678 raft_consensus.cc:2889] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:14.856048 31820 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:14.817204  1200 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:14.857913 31820 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:14.861146 31820 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:14.830719  1005 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:14.861778 31572 catalog_manager.cc:5697] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:14.866822 31572 catalog_manager.cc:5697] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 3 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:14.840656 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:14.844034  1159 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:14.837824  1100 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.032s
W20260709 12:21:14.864900 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800248e60 after lost signal to thread 31702
I20260709 12:21:14.869382 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:14.837319   980 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.033s
I20260709 12:21:14.873746 31572 catalog_manager.cc:5697] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 3, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:21:14.880545 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 164808us
I20260709 12:21:14.848555  1248 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:14.867358 31819 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:14.866183  1079 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:14.872500   678 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:14.881815 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 189857us
I20260709 12:21:14.953235 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:14.968703 32342 leader_election.cc:400] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 3. Current term is 4.
I20260709 12:21:14.969250 32342 leader_election.cc:403] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260709 12:21:14.981572 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:14.984436 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:14.991290 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78497.1us
I20260709 12:21:14.993315 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:15.021576 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:14.884663 31669 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:14.871479  1005 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:15.028483 31677 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
W20260709 12:21:15.015550 31607 debug-util.cc:398] Leaking SignalData structure 0x7b080035df60 after lost signal to thread 31749
I20260709 12:21:14.960635   678 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:15.011243  1248 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:15.015789  1267 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:14.879642  1277 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.148s
W20260709 12:21:15.009869 31744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1207) took 2448 ms (client timeout 1838 ms). Trace:
I20260709 12:21:15.009299  1273 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:15.063134 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 236716us
I20260709 12:21:15.031587  1246 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:15.029263 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1558) took 1140 ms. Trace:
I20260709 12:21:15.043195 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:14.910902  1189 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:15.042490 31818 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:15.045121  1204 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4.
W20260709 12:21:15.065419 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08002f6700 after lost signal to thread 31777
I20260709 12:21:15.039878   678 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:15.076068  1276 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260709 12:21:15.075400  1271 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.196s	user 0.000s	sys 0.001s
W20260709 12:21:15.102346 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 273118us
I20260709 12:21:15.139438 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:15.141034 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 119548us
I20260709 12:21:15.142136 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:15.143692 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:15.146572  1292 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:15.076936  1170 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:15.085393 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800382340 after lost signal to thread 31778
W20260709 12:21:15.149298  1279 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:21:15.211867 31747 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:15.177802 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1556) took 1289 ms. Trace:
I20260709 12:21:15.216588 31822 rpcz_store.cc:276] 0709 12:21:13.888108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.743597 (+855489us) service_pool.cc:224] Handling call
0709 12:21:14.757296 (+ 13699us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:14.757612 (+   316us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:14.757650 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:14.776436 (+ 18786us) log.cc:844] Serialized 26 byte log entry
0709 12:21:14.944283 (+167847us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:14.944351 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.944459 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:15.177515 (+233056us) raft_consensus.cc:1685] finished
0709 12:21:15.177536 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:15.177717 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":128034,"threads_started":1,"wal-append.queue_time_us":289267}
I20260709 12:21:15.148491  1227 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:15.088022  1178 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:15.167272  1292 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:15.225688 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:15.047770 31744 rpcz_store.cc:269] 0709 12:21:12.515056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.165576 (+650520us) service_pool.cc:224] Handling call
0709 12:21:14.910953 (+1745377us) spinlock_profiling.cc:243] Waited 728 ms on lock 0x7b54002134d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:14.963304 (+ 52351us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1674992896}
W20260709 12:21:15.234246 31792 leader_election.cc:336] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.517s (SENT)
I20260709 12:21:15.243103 31792 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:15.175630  1272 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.298s	user 0.000s	sys 0.001s
W20260709 12:21:15.175011  1278 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
W20260709 12:21:15.176508  1179 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.988s	user 0.010s	sys 0.003s
W20260709 12:21:15.175681 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800372b20 after lost signal to thread 31792
W20260709 12:21:15.244833 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 180581us
W20260709 12:21:15.318933 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003dc6e0 after lost signal to thread 31824
I20260709 12:21:15.170883   961 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:15.319344 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003dc6a0 after lost signal to thread 31825
I20260709 12:21:15.162362  1284 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:15.173862  1277 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:15.318806 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:15.237986 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 368681us
I20260709 12:21:15.077219 31819 rpcz_store.cc:276] 0709 12:21:13.888602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.790925 (+902323us) service_pool.cc:224] Handling call
0709 12:21:14.825400 (+ 34475us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:14.959631 (+134231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:15.025014 (+ 65383us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:15.224354  1114 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:15.227898  1227 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:15.322252 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 182718us
W20260709 12:21:15.325191 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 181532us
I20260709 12:21:15.263981  1280 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:15.361735 31791 leader_election.cc:400] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3.
I20260709 12:21:15.246650  1246 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:15.365013 31749 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:15.364915 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:15.362962 31719 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.008s	sys 0.017s
W20260709 12:21:15.321393 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003dc7c0 after lost signal to thread 31826
I20260709 12:21:15.347834  1178 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:15.359997 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:15.349261  1114 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:15.399503 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1550) took 1513 ms. Trace:
I20260709 12:21:15.396771 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:15.351759 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:15.373310 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:15.400135 31719 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.008s	sys 0.017s
W20260709 12:21:15.388232 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:15.398545 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:15.401209 31823 rpcz_store.cc:276] 0709 12:21:13.886512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.308519 (+422007us) service_pool.cc:224] Handling call
0709 12:21:14.308785 (+   266us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:14.310117 (+  1332us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:14.310177 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:14.405405 (+ 95228us) log.cc:844] Serialized 26 byte log entry
0709 12:21:14.697941 (+292536us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:14.697988 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:14.698051 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:15.378421 (+680370us) raft_consensus.cc:1685] finished
0709 12:21:15.378453 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:15.399424 (+ 20971us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5212,"raft.run_cpu_time_us":916,"raft.run_wall_time_us":28585,"thread_start_us":196082,"threads_started":1,"wal-append.queue_time_us":417312}
I20260709 12:21:15.363477  1179 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:15.360468   980 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:21:15.395968 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003dc760 after lost signal to thread 31827
I20260709 12:21:15.364409  1290 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:15.364315 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:15.364334  1280 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:15.414814 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55197.9us
I20260709 12:21:15.420099 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:15.420231 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:15.395905  1178 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:21:15.422338  1213 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.704s	user 0.014s	sys 0.016s
I20260709 12:21:15.402138   946 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:15.423306   946 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:15.416095   980 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:15.422942  1178 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:15.424358  1178 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:15.401942  1005 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:15.418828  1290 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:15.425540   980 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:15.421726 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:15.418835  1298 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:15.420971  1164 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:21:15.364473  1289 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:21:15.427699 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:15.405055 32374 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:15.428037 31826 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 12:21:15.413468  1246 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:15.418123  1280 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:15.426911  1078 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.653s	user 0.015s	sys 0.001s
W20260709 12:21:15.421749  1286 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
W20260709 12:21:15.422761 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003d54a0 after lost signal to thread 31851
I20260709 12:21:15.481365 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1677) took 2938 ms. Trace:
I20260709 12:21:15.513495 31826 rpcz_store.cc:276] 0709 12:21:12.543133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.301823 (+1758690us) service_pool.cc:224] Handling call
0709 12:21:15.427500 (+1125677us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:15.427699 (+   199us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:15.481024 (+ 53325us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:15.481292 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27172736}
I20260709 12:21:15.485623 31677 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:15.515821 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:15.516413 31677 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:15.412329  1304 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:15.490850 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 70816.5us
I20260709 12:21:15.489059  1300 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:15.499761 32342 connection.cc:256] RPC call timeout handler was delayed by -1.8229959669997697s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1664s for a response
I20260709 12:21:15.417184 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 363) took 1499 ms. Trace:
I20260709 12:21:15.502686 31678 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:21:15.503266 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:15.407153  1227 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:15.508508 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:15.520658 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:15.528614 31674 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:15.526903 32342 connection.cc:256] RPC call timeout handler was delayed by -1.5790959669998301s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1935s for a response
W20260709 12:21:15.529613 32342 connection.cc:256] RPC call timeout handler was delayed by -1.4287959670000419s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2102s for a response
W20260709 12:21:15.530032 32342 connection.cc:256] RPC call timeout handler was delayed by -1.3315959669998847s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.221s for a response
I20260709 12:21:15.490063  1246 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:15.442729  1005 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.047s
W20260709 12:21:15.511385 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 83760us
W20260709 12:21:15.511987 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 93382.2us
I20260709 12:21:15.444545  1297 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:15.422809  1213 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:15.508931  1307 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:15.416678  1284 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.101s
W20260709 12:21:15.586484 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54221.1us
I20260709 12:21:15.424216   946 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:15.589339 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:15.644047 31675 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:15.519963  1303 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:15.435302  1256 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.082s
I20260709 12:21:15.634698 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:15.437399  1292 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:15.635859  1164 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:15.645255 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:15.597122 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 373974us
I20260709 12:21:15.514578  1078 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:15.513070 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003d53e0 after lost signal to thread 31852
I20260709 12:21:15.585785  1297 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:15.603811   859 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:15.585104  1293 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:15.585608  1246 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:15.546572 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:15.679359 31677 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:15.657389  1204 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:15.663753 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:15.646458   946 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:15.665818 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:15.670446 31791 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:15.660610 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:15.695300 31676 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:15.648826 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:15.677206 31675 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
W20260709 12:21:15.704831 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 335571us
I20260709 12:21:15.634402   931 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:15.661460 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:15.695832 31747 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:15.705754 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:15.715792 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:15.716094 31676 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:15.670265  1297 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:15.527163  1300 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:15.722945 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:15.723526  1310 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:21:15.723955 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:15.725528 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:15.725960 31676 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:15.708367 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:15.669636  1301 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:15.680339  1293 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:15.680895  1256 consensus_queue.cc:1243] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:21:15.716889   931 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:15.677984   859 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:15.723628 32374 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:15.727196 31676 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:15.723491  1300 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:15.728552   799 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.414s	user 0.015s	sys 0.000s
I20260709 12:21:15.747588 31676 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:15.716087 32342 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:15.716805  1299 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:15.674854 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003174e0 after lost signal to thread 32342
I20260709 12:21:15.732725  1293 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:15.748847 31676 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:15.724483  1290 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:15.712054   961 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
W20260709 12:21:15.757323 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 96816.9us
W20260709 12:21:15.762516 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46781.1us
I20260709 12:21:15.743307  1256 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:15.765359 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:15.752827  1295 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.259s	user 0.000s	sys 0.001s
I20260709 12:21:15.769757 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:15.770792 32376 raft_consensus.cc:2368] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5.
I20260709 12:21:15.724623  1269 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260709 12:21:15.531253  1304 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:15.652557  1303 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:15.750313 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800339460 after lost signal to thread 32343
W20260709 12:21:15.785403 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 115018us
I20260709 12:21:15.770874  1189 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:15.804963 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:15.743773  1284 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.029s
I20260709 12:21:15.768157  1290 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:15.806907  1321 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:15.763307  1005 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:15.749790  1299 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:15.807367   931 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:15.772156   972 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:15.767198  1256 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:15.822280 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:15.838584 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:15.840221 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:15.756085   799 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:15.824322   972 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:15.828428  1311 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:15.831166  1267 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.781s	user 0.003s	sys 0.008s
I20260709 12:21:15.818395  1290 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:15.814618  1189 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:15.764323  1164 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:15.821722  1213 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:15.834869  1315 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:15.852262  1113 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.064s	user 0.008s	sys 0.005s
I20260709 12:21:15.819131  1005 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:15.819371  1299 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:15.825235 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800366f00 after lost signal to thread 32372
I20260709 12:21:15.814818  1298 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:15.857097  1314 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:15.857481   859 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:15.853307  1266 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.150s	user 0.015s	sys 0.015s
I20260709 12:21:15.861881 32375 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:15.856037  1290 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:15.850268   972 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:15.852500 31674 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
W20260709 12:21:15.860178  1306 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.220s	user 0.000s	sys 0.001s
W20260709 12:21:15.863667 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:15.884562 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 348860us
I20260709 12:21:15.884812 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:15.860987  1189 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:15.856350  1267 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 3. Current term is 4.
W20260709 12:21:15.892019  1100 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.981s	user 0.015s	sys 0.021s
I20260709 12:21:15.871971  1078 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:15.872268 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:15.823905  1321 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:15.871001  1005 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:15.841886  1313 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.123s
I20260709 12:21:15.900394 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:15.882215  1312 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:15.906306  1100 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.020s	user 0.015s	sys 0.021s
I20260709 12:21:15.889583  1266 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:15.893067 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003b1140 after lost signal to thread 32402
I20260709 12:21:15.868100   931 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:15.902631  1315 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:15.906163 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:15.890321  1314 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:15.854592  1311 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:15.926786  1314 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:15.927667  1311 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:15.917961 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:15.935236  1311 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:15.935878  1311 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:15.936337  1311 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:15.915897  1284 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:15.875188  1113 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:15.953184 32378 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:15.892860   859 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.030s
W20260709 12:21:15.922698 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800360920 after lost signal to thread 678
I20260709 12:21:15.966223 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1691) took 1550 ms. Trace:
I20260709 12:21:15.967702 31822 rpcz_store.cc:276] 0709 12:21:14.415998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.218374 (+802376us) service_pool.cc:224] Handling call
0709 12:21:15.316751 (+ 98377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:15.317742 (+   991us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:15.317772 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:15.318347 (+   575us) log.cc:844] Serialized 26 byte log entry
0709 12:21:15.320477 (+  2130us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:15.320548 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:15.320629 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:15.965927 (+645298us) raft_consensus.cc:1685] finished
0709 12:21:15.965949 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:15.966145 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":1671,"threads_started":1,"wal-append.queue_time_us":185457}
I20260709 12:21:15.955816  1313 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:15.953121  1312 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:15.963441 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800303de0 after lost signal to thread 799
W20260709 12:21:15.977617 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 168124us
I20260709 12:21:15.887987  1256 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:15.532589 32370 rpcz_store.cc:276] 0709 12:21:13.918084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.924637 (+  6553us) service_pool.cc:224] Handling call
0709 12:21:13.924952 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:13.925263 (+   311us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:14.123749 (+198486us) log.cc:844] Serialized 14 byte log entry
0709 12:21:15.210664 (+1086915us) spinlock_profiling.cc:243] Waited 6.83 ms on lock 0x7b18002ef6c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a
0709 12:21:15.395287 (+184623us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:15.397491 (+  2204us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:15.402052 (+  4561us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:15.409684 (+  7632us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:15.416104 (+  6420us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5924,"prepare.queue_time_us":129086,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":90,"raft.queue_time_us":137501,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":4964,"spinlock_wait_cycles":15714688,"thread_start_us":1048908,"threads_started":3,"wal-append.queue_time_us":720238}
I20260709 12:21:15.982726 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:16.024106 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:16.025208 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:15.969980  1284 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:16.028323  1337 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:16.036865  1337 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:16.037286  1337 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:16.033639  1319 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:16.038117  1319 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:16.002311 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800360c00 after lost signal to thread 859
I20260709 12:21:16.044049 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:15.958593  1298 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.045156 31675 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:16.046047 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:15.976778  1313 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:15.981695  1312 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:16.071779 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:21:16.087833 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:16.047389 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003b3b20 after lost signal to thread 939
W20260709 12:21:16.097455 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 257341us
I20260709 12:21:16.050143  1256 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:16.072621  1312 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:16.081595 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:16.085280   931 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:16.096390 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003b3aa0 after lost signal to thread 956
I20260709 12:21:16.106175 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:16.092358   799 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.120465 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:16.121964 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:16.100958  1100 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:16.127990 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:16.131120 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:16.110954 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08002f8ac0 after lost signal to thread 961
I20260709 12:21:16.043697  1284 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.141582 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:16.101063  1304 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:16.105715   931 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:16.145221  1235 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.332s	user 0.008s	sys 0.014s
I20260709 12:21:16.149861  1235 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:16.151952  1178 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.727s	user 0.011s	sys 0.000s
I20260709 12:21:16.152232  1235 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:16.153851  1310 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.420s	user 0.000s	sys 0.003s
I20260709 12:21:16.156920 31677 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
W20260709 12:21:16.159346   678 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.996s	user 0.010s	sys 0.001s
I20260709 12:21:16.159440  1321 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:16.159888   678 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.126847  1303 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.140162  1164 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260709 12:21:16.145695  1200 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.275s	user 0.013s	sys 0.013s
I20260709 12:21:16.145370  1315 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:16.129858  1100 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:16.162727 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:16.148684   931 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:16.164980 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:16.178776  1340 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:16.179483  1332 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:21:16.179849   678 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:16.161401  1323 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:16.185010  1189 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.187955  1319 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:16.187842  1311 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.188609  1189 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.174948  1321 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260709 12:21:16.170554  1100 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:16.193039  1277 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.843s	user 0.010s	sys 0.000s
I20260709 12:21:16.173416  1314 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:16.206802  1323 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:16.208319  1323 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.209645  1287 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:16.210155  1287 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:16.194324 32376 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:21:16.177412  1113 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.196259  1277 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:16.214672  1150 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:16.221130  1273 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.144s	user 0.010s	sys 0.003s
I20260709 12:21:16.221801  1273 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:16.181274  1164 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 12:21:16.216243   678 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:16.180233  1284 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.196625  1300 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.172955  1200 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.147557  1313 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.232942  1313 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:16.228896  1200 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:16.219097  1314 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:16.235306  1314 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.193090  1246 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.242507  1114 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.244086  1114 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:16.244894  1331 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:16.245703  1331 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.232806  1299 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:16.253876  1280 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.755s	user 0.013s	sys 0.000s
I20260709 12:21:16.261106  1299 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.262107  1331 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.262537  1287 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.266330  1287 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:16.269686  1311 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.247474  1290 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:16.267163  1297 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.544s	user 0.006s	sys 0.005s
W20260709 12:21:16.274581  1324 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.461s	user 0.002s	sys 0.006s
I20260709 12:21:16.275591  1297 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:16.277225  1344 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.087s	user 0.002s	sys 0.000s
I20260709 12:21:16.279165  1297 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.287042  1337 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.289072  1337 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:16.235833  1320 consensus_queue.cc:1175] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 2 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:21:16.140652 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003aa340 after lost signal to thread 974
W20260709 12:21:16.321496 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003b3e00 after lost signal to thread 980
W20260709 12:21:16.321985 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003a8ec0 after lost signal to thread 987
W20260709 12:21:16.322428 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800352ba0 after lost signal to thread 993
W20260709 12:21:16.322916 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08002ee7e0 after lost signal to thread 1030
W20260709 12:21:16.324013 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003b84a0 after lost signal to thread 1079
W20260709 12:21:16.324468 31607 debug-util.cc:398] Leaking SignalData structure 0x7b080027d480 after lost signal to thread 1081
W20260709 12:21:16.324829 31607 debug-util.cc:398] Leaking SignalData structure 0x7b080033a3a0 after lost signal to thread 1086
W20260709 12:21:16.325210 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08001638a0 after lost signal to thread 1091
W20260709 12:21:16.325716 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003aa5e0 after lost signal to thread 1100
W20260709 12:21:16.163048 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60143.2us
I20260709 12:21:16.326602 31669 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:16.328043 31746 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:16.332013 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1639) took 1126 ms. Trace:
I20260709 12:21:16.333420 31678 raft_consensus.cc:2410] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:16.340004 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 364) took 1190 ms. Trace:
I20260709 12:21:16.224610   678 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:16.342219 32377 raft_consensus.cc:2410] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:16.361925  1200 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:16.362696   678 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:16.357470 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800334ce0 after lost signal to thread 1113
W20260709 12:21:16.365854  1204 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.659s	user 0.012s	sys 0.000s
I20260709 12:21:16.366976  1204 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:16.361353 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:16.349421 31744 rpcz_store.cc:276] 0709 12:21:15.205645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.583968 (+378323us) service_pool.cc:224] Handling call
0709 12:21:15.649543 (+ 65575us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:15.899502 (+249959us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:15.904606 (+  5104us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:15.961458 (+ 56852us) log.cc:844] Serialized 26 byte log entry
0709 12:21:16.123152 (+161694us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:16.130010 (+  6858us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.144327 (+ 14317us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:16.331200 (+186873us) raft_consensus.cc:1685] finished
0709 12:21:16.331244 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.331658 (+   414us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1581,"raft.queue_time_us":72599,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":8726,"thread_start_us":340618,"threads_started":2,"wal-append.queue_time_us":182003}
I20260709 12:21:16.374679 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:16.363313  1200 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.376590 31745 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
W20260709 12:21:16.365976 31607 debug-util.cc:398] Leaking SignalData structure 0x7b080031c0e0 after lost signal to thread 1114
I20260709 12:21:16.365929   678 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:16.374284   972 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.391317   972 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:16.395135 32374 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.640s	user 0.011s	sys 0.012s
I20260709 12:21:16.395977 32374 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:16.396375 31750 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:16.355814 32370 rpcz_store.cc:276] 0709 12:21:15.149934 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.129054 (+979120us) service_pool.cc:224] Handling call
0709 12:21:16.339642 (+210588us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:16.403031 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1212) took 2074 ms. Trace:
I20260709 12:21:16.403429 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:16.404091  1246 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:16.403659 31750 rpcz_store.cc:276] 0709 12:21:14.328157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.741368 (+413211us) service_pool.cc:224] Handling call
0709 12:21:16.395089 (+1653721us) spinlock_profiling.cc:243] Waited 5.43 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:16.395776 (+   687us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.400897 (+  5121us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.402907 (+  2010us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12498688}
I20260709 12:21:16.405593 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:16.407409 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:16.407531  1200 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.407652  1290 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.283757  1300 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:16.409665 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:16.402388 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800220c80 after lost signal to thread 1136
W20260709 12:21:16.421173 31748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1211) took 2093 ms (client timeout 1504 ms). Trace:
I20260709 12:21:16.421535 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:16.421886 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1642) took 1215 ms. Trace:
W20260709 12:21:16.421805 31748 rpcz_store.cc:269] 0709 12:21:14.327776 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.327955 (+   179us) service_pool.cc:224] Handling call
0709 12:21:14.415439 (+ 87484us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:16.420735 (+2005296us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b54000f2dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:16.420937 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":266859136}
I20260709 12:21:16.422348 31750 rpcz_store.cc:276] 0709 12:21:15.206478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.419439 (+1212961us) service_pool.cc:224] Handling call
0709 12:21:16.421062 (+  1623us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.421476 (+   414us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.421511 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:16.421578 (+    67us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:16.421639 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.421700 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.421818 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:16.423727 31744 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:16.424762 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1644) took 1217 ms. Trace:
I20260709 12:21:16.413460 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 367) took 1164 ms. Trace:
I20260709 12:21:16.425571 32374 rpcz_store.cc:276] 0709 12:21:15.248599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.722554 (+473955us) service_pool.cc:224] Handling call
0709 12:21:16.412873 (+690319us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:16.427978 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1637) took 1208 ms. Trace:
I20260709 12:21:16.428581 31747 rpcz_store.cc:276] 0709 12:21:15.205167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.225350 (+ 20183us) service_pool.cc:224] Handling call
0709 12:21:16.413516 (+1188166us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b54004042d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:16.413872 (+   356us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1074269568}
I20260709 12:21:16.403057  1280 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.428220 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1635) took 1223 ms. Trace:
I20260709 12:21:16.428527 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 363) took 1292 ms. Trace:
I20260709 12:21:16.425309 31748 rpcz_store.cc:276] 0709 12:21:15.207113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.423391 (+1216278us) service_pool.cc:224] Handling call
0709 12:21:16.423862 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.424303 (+   441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.424340 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:16.424420 (+    80us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:16.424467 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.424540 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.424670 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:16.434224 31752 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:16.434751 31749 rpcz_store.cc:276] 0709 12:21:15.204618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.204773 (+   155us) service_pool.cc:224] Handling call
0709 12:21:15.205166 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:15.370089 (+164923us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.427631 (+1057542us) spinlock_profiling.cc:243] Waited 160 ms on lock 0x7b54002134d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:16.427994 (+   363us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":367056128}
I20260709 12:21:16.424782 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1643) took 1218 ms. Trace:
I20260709 12:21:16.443447 31750 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:16.445077  1100 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.445952 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 141) took 2247 ms. Trace:
I20260709 12:21:16.436205 32374 rpcz_store.cc:276] 0709 12:21:15.136658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.426766 (+1290108us) service_pool.cc:224] Handling call
0709 12:21:16.427550 (+   784us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.427992 (+   442us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:16.428024 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:16.428109 (+    85us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:16.428195 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.428299 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.428427 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:16.453303 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:16.472122 32373 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:16.472678 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 356) took 1489 ms. Trace:
I20260709 12:21:16.473242 32373 rpcz_store.cc:276] 0709 12:21:14.983543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.008217 (+ 24674us) service_pool.cc:224] Handling call
0709 12:21:16.452818 (+1444601us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:16.472611 (+ 19793us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4215168}
I20260709 12:21:16.453768 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:16.475360 32370 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:21:16.447031 31744 rpcz_store.cc:276] 0709 12:21:15.206812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.422812 (+1216000us) service_pool.cc:224] Handling call
0709 12:21:16.423305 (+   493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.424473 (+  1168us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.424720 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:16.455132 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 354) took 1474 ms. Trace:
I20260709 12:21:16.456895 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 501) took 2041 ms. Trace:
W20260709 12:21:16.455858  1289 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.906s	user 0.008s	sys 0.006s
I20260709 12:21:16.459117 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 361) took 1428 ms. Trace:
W20260709 12:21:16.461673 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 51384.8us
I20260709 12:21:16.463258 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:16.467171  1349 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.144s	user 0.002s	sys 0.000s
W20260709 12:21:16.439792 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08002f8b40 after lost signal to thread 1150
I20260709 12:21:16.484529 32378 rpcz_store.cc:276] 0709 12:21:14.415378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.415526 (+   148us) service_pool.cc:224] Handling call
0709 12:21:16.456451 (+2040925us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b54004312d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:16.456672 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3289416448}
I20260709 12:21:16.439138  1280 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:16.490756  1289 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.981s	user 0.008s	sys 0.007s
I20260709 12:21:16.487653 32376 rpcz_store.cc:276] 0709 12:21:15.031020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.972849 (+941829us) service_pool.cc:224] Handling call
0709 12:21:16.458558 (+485709us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54005947d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:16.458842 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":312330112}
I20260709 12:21:16.480932 32375 rpcz_store.cc:276] 0709 12:21:14.980987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.981182 (+   195us) service_pool.cc:224] Handling call
0709 12:21:16.454661 (+1473479us) spinlock_profiling.cc:243] Waited 878 ms on lock 0x7b5400426dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:16.454917 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2019884800}
I20260709 12:21:16.480446 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 366) took 1303 ms. Trace:
W20260709 12:21:16.493945 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08002831e0 after lost signal to thread 1157
I20260709 12:21:16.494592 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:16.497529 31670 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
W20260709 12:21:16.504611 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800383460 after lost signal to thread 1159
I20260709 12:21:16.471194 31820 rpcz_store.cc:276] 0709 12:21:14.198444 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.871664 (+673220us) service_pool.cc:224] Handling call
0709 12:21:14.872567 (+   903us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:21:14.953634 (+ 81067us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:14.953688 (+    54us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:21:15.018530 (+ 64842us) log.cc:844] Serialized 78 byte log entry
0709 12:21:15.157503 (+138973us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:15.157982 (+   479us) log.cc:844] Serialized 14 byte log entry
0709 12:21:16.444763 (+1286781us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b540036ed20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd50733 00007fe8fcdd5dbf 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c
0709 12:21:16.444879 (+   116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.445104 (+   225us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:16.445186 (+    82us) raft_consensus.cc:1685] finished
0709 12:21:16.445340 (+   154us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.445632 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":223,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":41102,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":56737,"spinlock_wait_cycles":30583168,"thread_start_us":81033,"threads_started":1,"wal-append.queue_time_us":113241,"wal-append.run_cpu_time_us":3966,"wal-append.run_wall_time_us":1074381}
W20260709 12:21:16.515158 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003d5f60 after lost signal to thread 1164
I20260709 12:21:16.520183 31672 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.520727 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:16.521142 32369 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:16.520767 31752 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:16.523695 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:16.532382 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:16.540463 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:16.537127 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:16.541702 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800383520 after lost signal to thread 1170
I20260709 12:21:16.500923 32370 rpcz_store.cc:276] 0709 12:21:15.173246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.473903 (+1300657us) service_pool.cc:224] Handling call
0709 12:21:16.474899 (+   996us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:16.476198 (+  1299us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.476893 (+   695us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:16.549984  1299 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:16.550318  1289 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:16.550567 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:16.550752  1299 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.552094 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:16.553387 31827 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:21:16.553620  1299 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:16.547437 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:16.554054 31827 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1685) took 2192 ms (client timeout 2240 ms). Trace:
I20260709 12:21:16.554570 31752 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:16.532449 32371 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:16.509779  1005 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.551044  1289 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.555063 32375 raft_consensus.cc:2410] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:16.557339 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:16.548972 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:16.546885 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:16.554572  1227 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:16.561182 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:16.555577 31672 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:16.554486 31827 rpcz_store.cc:269] 0709 12:21:14.361395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.522095 (+160700us) service_pool.cc:224] Handling call
0709 12:21:16.520076 (+1997981us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:16.553914 (+ 33838us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2601216}
I20260709 12:21:16.569693  1289 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:16.570060 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 178) took 1086 ms. Trace:
I20260709 12:21:16.570516 31669 rpcz_store.cc:276] 0709 12:21:15.483295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.489952 (+  6657us) service_pool.cc:224] Handling call
0709 12:21:15.531993 (+ 42041us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.348515 (+816522us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.569745 (+221230us) spinlock_profiling.cc:243] Waited 624 ms on lock 0x7b54001d5954. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:16.569978 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1435880192}
I20260709 12:21:16.574509 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:41934 (request call id 3) took 2166 ms. Trace:
I20260709 12:21:16.575199 31825 rpcz_store.cc:276] 0709 12:21:14.408660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.544362 (+135702us) service_pool.cc:224] Handling call
0709 12:21:16.574043 (+2029681us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b540043dad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdcf853 00007fe901d9d60a 00007fe8fb3237d4 00007fe8fb323762 00007fe8fb323691 00007fe8fb32360c 00007fe8fb3228c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:16.574234 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":626080000}
W20260709 12:21:16.557592 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800387140 after lost signal to thread 1176
I20260709 12:21:16.561509 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 352) took 1580 ms. Trace:
W20260709 12:21:16.579888 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003539c0 after lost signal to thread 1179
W20260709 12:21:16.580331 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08003020e0 after lost signal to thread 1181
W20260709 12:21:16.580780 31607 debug-util.cc:398] Leaking SignalData structure 0x7b080035c0a0 after lost signal to thread 1182
I20260709 12:21:16.579977 32369 rpcz_store.cc:276] 0709 12:21:14.975910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.009436 (+ 33526us) service_pool.cc:224] Handling call
0709 12:21:16.449977 (+1440541us) spinlock_profiling.cc:243] Waited 2.95 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:16.450330 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.542352 (+ 92022us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.556124 (+ 13772us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6783360}
W20260709 12:21:16.581084 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800367420 after lost signal to thread 1183
I20260709 12:21:16.581847 31675 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
I20260709 12:21:16.583057 31753 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:16.583770 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1213) took 2255 ms. Trace:
W20260709 12:21:16.579406 32377 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 357) took 1595 ms (client timeout 1705 ms). Trace:
W20260709 12:21:16.584619 32377 rpcz_store.cc:269] 0709 12:21:14.983932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.984081 (+   149us) service_pool.cc:224] Handling call
0709 12:21:16.579104 (+1595023us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b540053a554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:16.579272 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3117933952}
I20260709 12:21:16.585325 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:16.591135  1320 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:21:16.559057 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 349472us
I20260709 12:21:16.579115 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:16.579380 32374 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:16.584182 31753 rpcz_store.cc:276] 0709 12:21:14.328452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.746350 (+417898us) service_pool.cc:224] Handling call
0709 12:21:16.582275 (+1835925us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:16.582617 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.583517 (+   900us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.583709 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4295552}
I20260709 12:21:16.589460 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:16.606451 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:16.607015  1269 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:16.608711  1301 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:16.609164  1301 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:16.610244 32369 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:16.610468  1301 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.612373  1301 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:16.613183 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:16.615129 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:16.615561 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:16.616204 32373 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:16.616284  1289 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:16.617801  1273 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:16.617947 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:16.618592  1273 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:16.619179 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:16.620319  1079 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:16.620852  1079 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:16.608701 32375 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:16.625015 31792 connection.cc:256] RPC call timeout handler was delayed by -1.1895053260000168s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
W20260709 12:21:16.625916 31792 leader_election.cc:336] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.714s (SENT)
W20260709 12:21:16.614749 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:16.626737 32378 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260709 12:21:16.627635 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1214) took 2298 ms. Trace:
I20260709 12:21:16.628019 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:16.628108 31746 rpcz_store.cc:276] 0709 12:21:14.328766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.773750 (+444984us) service_pool.cc:224] Handling call
0709 12:21:14.774106 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.349408 (+1575302us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.627328 (+277920us) spinlock_profiling.cc:243] Waited 1.55 s on lock 0x7b54002f71d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:16.627573 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3571907328}
I20260709 12:21:16.632963 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:16.633240 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:16.633773 32377 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:16.634123 31670 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:16.635856 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:16.637753 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:16.614295 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:16.639134 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:21:16.639842  1249 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:16.640537  1312 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.527s	user 0.009s	sys 0.009s
I20260709 12:21:16.640837  1249 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.641198  1312 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.630218 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:16.643365 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:16.644086 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:16.644922 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:16.645020 31673 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:16.638527 31678 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:16.619874  1273 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.614979 31669 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:16.596128 31607 debug-util.cc:398] Leaking SignalData structure 0x7b080024f500 after lost signal to thread 1188
I20260709 12:21:16.603466 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:16.634020 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:16.653788 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:16.646391 32374 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
W20260709 12:21:16.661902 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 48797.2us
W20260709 12:21:16.650735 31607 debug-util.cc:398] Leaking SignalData structure 0x7b080035c460 after lost signal to thread 1189
I20260709 12:21:16.635875 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:16.665570 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 76476.1us
I20260709 12:21:16.667886  1327 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:16.668712  1327 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:16.622349  1079 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:16.671473  1293 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.796s	user 0.014s	sys 0.012s
I20260709 12:21:16.671942  1079 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:16.672149  1293 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:16.674994  1358 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.081s	user 0.002s	sys 0.000s
I20260709 12:21:16.679188 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:16.682701 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:16.682932 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 57155.9us
W20260709 12:21:16.683719 31792 connection.cc:256] RPC call timeout handler was delayed by -1.1769053259999964s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
W20260709 12:21:16.684294 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:16.684695 32370 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:16.684876 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:16.686034  1289 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:16.668092  1005 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:16.687222 31792 leader_election.cc:336] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.726s (SENT)
I20260709 12:21:16.686766 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:16.686995  1256 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:16.652042 31744 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:16.691182 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:16.694736 31825 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:16.679500 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 503) took 2059 ms. Trace:
I20260709 12:21:16.695544 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:16.699338 31673 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:16.700840  1296 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:16.701424 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:16.703575 32377 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
W20260709 12:21:16.696830 31607 debug-util.cc:398] Leaking SignalData structure 0x7b080039ee80 after lost signal to thread 1192
I20260709 12:21:16.705500 31672 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260709 12:21:16.705497 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:16.705104  1178 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:16.694949  1100 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.706825 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:16.707108 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1641) took 1501 ms. Trace:
I20260709 12:21:16.705330   678 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:16.707604  1256 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.708559 31825 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:16.686830  1289 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.669981  1327 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:16.709040 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800045e20 after lost signal to thread 1193
I20260709 12:21:16.718156 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1608) took 1951 ms. Trace:
I20260709 12:21:16.709795  1178 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:16.718844 31672 rpcz_store.cc:276] 0709 12:21:14.766546 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.768388 (+  1842us) service_pool.cc:224] Handling call
0709 12:21:14.768819 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.508886 (+1740067us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b4c00337600. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb0a1a20 00007fe8fb0b2a29 00007fe8fb078644 00007fe8fb09ff2d 00007fe8fb09fb93 00007fe8fcdd018b 00007fe8fcdd0cb5 00007fe8fcdcbf30 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416
0709 12:21:16.709227 (+200341us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.718069 (+  8842us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":52290432}
I20260709 12:21:16.711650 32369 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:16.748335   931 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.571s	user 0.009s	sys 0.009s
I20260709 12:21:16.732136 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:16.749012   931 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.749953 31822 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:16.733983 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:16.750597 31674 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:16.733870 31672 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:16.735366 31670 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:16.736102 31753 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:16.712217 31745 rpcz_store.cc:276] 0709 12:21:15.206187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.807418 (+601231us) service_pool.cc:224] Handling call
0709 12:21:15.832607 (+ 25189us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.378161 (+545554us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.706819 (+328658us) spinlock_profiling.cc:243] Waited 543 ms on lock 0x7b5400295ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:16.707049 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1248347008}
I20260709 12:21:16.754385 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:16.738970 31607 debug-util.cc:398] Leaking SignalData structure 0x7b080039bc60 after lost signal to thread 1211
I20260709 12:21:16.754805  1289 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:16.755324 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08002c80e0 after lost signal to thread 1213
W20260709 12:21:16.739492 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 46424.8us
I20260709 12:21:16.760247 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:16.760131  1299 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:16.760857 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:16.761969  1299 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:16.762929  1030 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEADER]: Stepping down as leader of term 3
W20260709 12:21:16.740716 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 56393.5us
I20260709 12:21:16.763351  1030 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:16.764689  1030 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.766062 31825 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:16.766274 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:16.742851 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:16.767135 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:16.743487 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:21:16.743435 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51566.8us
I20260709 12:21:16.766531  1159 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:16.741950  1249 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:16.771104 31824 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:16.783963 31673 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
W20260709 12:21:16.785517 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:16.791936 31669 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:16.800009 31751 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
W20260709 12:21:16.793061  1086 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 2 current term: 2 is higher.
I20260709 12:21:16.803709  1102 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20260709 12:21:16.804356 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:16.804896  1102 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:16.805318  1102 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:16.805630 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:16.806862 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:16.807019 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:16.762615  1299 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:21:16.826093 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:16.729625  1005 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:16.832626 31827 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:16.825587 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:16.823894 31825 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
W20260709 12:21:16.844444 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44076.3us
I20260709 12:21:16.841877 31825 pending_rounds.cc:85] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 1
I20260709 12:21:16.696642 32371 rpcz_store.cc:276] 0709 12:21:14.619933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.622383 (+  2450us) service_pool.cc:224] Handling call
0709 12:21:14.691877 (+ 69494us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.532065 (+1840188us) spinlock_profiling.cc:243] Waited 46.7 ms on lock 0x7b54001c6c20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:21:16.555483 (+ 23418us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.679052 (+123569us) spinlock_profiling.cc:243] Waited 471 ms on lock 0x7b540031d654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:16.679429 (+   377us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1191396608}
I20260709 12:21:16.790326  1289 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:16.813419  1030 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:16.807041  1150 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:16.817054 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:16.828842 32373 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:16.825846  1114 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:16.797679   678 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.859017 31825 pending_rounds.cc:107] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:21:16.875118 31825 raft_consensus.cc:2889] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:16.788398  1159 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.882877 31678 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:16.854550  1249 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:16.900493 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:16.861227  1293 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:16.866828  1150 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:16.902297 31820 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
W20260709 12:21:16.902441 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 88535us
I20260709 12:21:16.806499  1102 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.890482 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:16.919574 31824 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:16.920681 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 204) took 1108 ms. Trace:
I20260709 12:21:16.921224 31751 rpcz_store.cc:276] 0709 12:21:15.812008 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:15.884526 (+ 72518us) service_pool.cc:224] Handling call
0709 12:21:16.920436 (+1035910us) spinlock_profiling.cc:243] Waited 832 ms on lock 0x7b54001ee6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:16.920604 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1912555776}
I20260709 12:21:16.923147 31827 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:16.926767  1075 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:16.927361  1075 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.929747  1075 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.930570 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:16.931695  1159 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:16.932065 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:16.934100 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:16.935458  1178 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.937458  1178 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:16.931998 31746 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:16.939008 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:16.940825  1289 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:16.910267  1150 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:16.911962 31823 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:16.889463  1249 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:16.960631 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:16.975523 31823 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:16.975764 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 68254.7us
I20260709 12:21:16.961397 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1614) took 2149 ms. Trace:
I20260709 12:21:16.959084 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:16.979054 31751 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:16.970824 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 95979.4us
I20260709 12:21:16.964007 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:16.955503  1102 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:16.989537 31822 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:16.990482  1269 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:16.991160 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:16.992012 31822 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:16.964316  1330 raft_consensus.cc:2764] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:21:16.943567   931 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.963790 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:16.998041  1367 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:16.998481  1367 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:16.900072 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:16.999624  1367 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:16.958268 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:16.901582  1005 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.004276  1005 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:16.964190 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:16.994993  1277 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 12:21:16.961123  1249 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:17.014744 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:16.942567 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:17.006251 31744 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:17.019439 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:17.047230 31750 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:17.048530 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:17.006578 31818 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:17.066136 31826 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:16.956760   678 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:16.977435 31671 rpcz_store.cc:276] 0709 12:21:14.811402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:14.860323 (+ 48921us) service_pool.cc:224] Handling call
0709 12:21:14.877019 (+ 16696us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:15.176543 (+299524us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:15.176582 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:15.177270 (+   688us) log.cc:844] Serialized 26 byte log entry
0709 12:21:16.586523 (+1409253us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b180020a400. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:21:16.744108 (+157585us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:16.744579 (+   471us) log.cc:844] Serialized 14 byte log entry
0709 12:21:16.744916 (+   337us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:16.745013 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:16.960888 (+215875us) raft_consensus.cc:1685] finished
0709 12:21:16.960998 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:16.961177 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":116990,"prepare.queue_time_us":177,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":296987,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":8295,"spinlock_wait_cycles":53913856,"thread_start_us":418421,"threads_started":2,"wal-append.queue_time_us":1673419}
I20260709 12:21:16.943538  1289 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:17.011646  1256 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:17.076081 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:17.007292  1277 consensus_queue.cc:1243] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
W20260709 12:21:17.077862 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.018877  1159 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:16.967130  1299 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:17.009110 32369 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:17.012871  1353 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:17.093108 31827 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:17.088567 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:17.087877 31824 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:17.092542 31678 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:17.006688  1293 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.082s
I20260709 12:21:17.078272 31821 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260709 12:21:17.104072 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:17.106153 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:17.106330 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:17.108991 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:17.109714 31826 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:17.112984  1312 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:17.115271 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:17.115937 31826 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:17.120164 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:17.121083 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:17.123697 31824 raft_consensus.cc:2410] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:21:17.104355 31821 pending_rounds.cc:85] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 4
I20260709 12:21:17.139477 31752 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:17.141793 31673 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:17.141968  1351 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:17.142163 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:17.143221 31749 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:17.143760 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:17.144830  1293 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:17.135310 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:17.155227 31745 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:17.196307 31823 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:17.158903  1353 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:21:17.163548 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 215238us
I20260709 12:21:17.174801  1289 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.175309   678 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.175925  1114 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:17.177248  1079 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:17.209626 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:17.087643   972 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260709 12:21:17.116719 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:17.118750 31819 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:17.097505  1075 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260709 12:21:17.203688 31823 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:17.203842 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:17.205857 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:17.196251 31821 pending_rounds.cc:107] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260709 12:21:17.214054 31824 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:17.214936 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:17.187297 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:17.215837  1312 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:21:17.247591 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:17.249571 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:17.251434   931 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.252149   931 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:17.259124 31818 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:17.227128   678 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.265295 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1618) took 1369 ms. Trace:
I20260709 12:21:17.225273  1313 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:17.197590  1351 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:17.232343  1293 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:21:17.216995 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 73334.9us
W20260709 12:21:17.242404 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 124440us
I20260709 12:21:17.238080 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:17.283385 31671 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:17.272048 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:17.275352 31673 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:17.286000 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:17.225274 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:17.286784 31826 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:17.287575 31822 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:17.267696 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:17.292819 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:17.268574 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.282481 31827 raft_consensus.cc:2376] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5. Re-sending same reply.
I20260709 12:21:17.295857 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:17.272064  1351 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:17.215399 31821 raft_consensus.cc:2889] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:17.276063  1293 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:17.265870 31674 rpcz_store.cc:276] 0709 12:21:15.896570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.094571 (+198001us) service_pool.cc:224] Handling call
0709 12:21:16.094907 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.751197 (+656290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.264689 (+513492us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b540017c0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:17.265138 (+   449us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1507066240}
I20260709 12:21:17.271989  1114 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:17.306165 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:17.313568 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:17.265839   678 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:17.291910  1079 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 12:21:17.207410  1289 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.322866  1330 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:17.323342 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:17.302842  1293 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:17.327072 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:17.327903 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.328689 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:17.330899  1102 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:17.331593  1313 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:17.332763  1330 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:17.333139  1330 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:17.333483  1290 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:17.334046  1290 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:17.333338  1249 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:17.327754 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:17.307880   931 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:17.342918  1249 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:17.334049  1330 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.342243 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:17.352274 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:17.354020  1277 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:17.360134 31826 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:17.295718  1256 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.033s
W20260709 12:21:17.368923 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.039s	sys 0.109s
I20260709 12:21:17.369102 31824 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:17.369624 31822 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
W20260709 12:21:17.372087 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.517s	user 0.039s	sys 0.109s
I20260709 12:21:17.324481  1353 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.374390 31671 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:21:17.374454 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 574788us
I20260709 12:21:17.372251 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:17.376149 31823 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:17.358789  1078 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.010s
W20260709 12:21:17.379834 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.359258 31819 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:17.390851 31822 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:17.397835  1330 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:17.398097  1327 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:17.399325  1303 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:17.399700  1303 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:17.382114  1079 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:17.408080  1079 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:17.408751 31675 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
W20260709 12:21:17.406688 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.407401  1290 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:17.386852  1249 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:17.423036 31824 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:21:17.391356  1313 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:17.400786  1303 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:17.439350 31818 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:17.459868 32342 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:17.419986  1078 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 12:21:17.463856  1303 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:17.464751  1078 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:21:17.464970 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.465369  1078 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:17.424322  1114 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:17.467002  1114 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.469509  1114 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:17.471598 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:21:17.476235 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.461607  1150 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:17.484504  1150 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:17.389503  1353 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.504637 31826 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:17.434340  1330 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:17.518460 31673 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:17.492885  1313 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:17.520061  1273 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:21:17.524200 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:17.518568 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 102365us
I20260709 12:21:17.520807 31824 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:21:17.528457  1371 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:17.529333  1330 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.531859  1114 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.532327   987 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:17.533862  1159 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:17.535215  1256 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:17.526407  1030 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.657s	user 0.006s	sys 0.008s
I20260709 12:21:17.530320  1353 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:17.561220  1371 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:17.561632 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:17.494616  1078 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.561925  1369 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:17.562613  1273 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:17.562674  1289 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:17.563297  1313 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:17.562583  1290 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:17.596253 31674 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:17.596817  1249 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:17.575052 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:17.578315  1371 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:17.510418  1351 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:17.607709 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.556792  1159 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:17.555230   987 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:17.600255 31671 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 2.
I20260709 12:21:17.630453 31676 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 12:21:17.631151 31676 pending_rounds.cc:85] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 5
I20260709 12:21:17.631451 31676 pending_rounds.cc:107] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:21:17.631744 31676 raft_consensus.cc:2889] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:17.632192 31676 pending_rounds.cc:107] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:21:17.632494 31676 raft_consensus.cc:2889] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:17.589681  1273 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:17.637874 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 76317.7us
I20260709 12:21:17.639446  1159 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.641309  1159 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:17.642931  1371 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:17.644555  1371 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:17.599226  1256 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.648885  1256 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:17.651293 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:17.620517  1369 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.634115   987 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:17.558467  1114 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:17.662881 31671 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:17.636406  1030 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election won for term 3
W20260709 12:21:17.661736 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.019s	sys 0.033s
W20260709 12:21:17.672015 31716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.000s	sys 0.027s
W20260709 12:21:17.675947 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.019s	sys 0.033s
W20260709 12:21:17.676219 31716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.563s	user 0.001s	sys 0.027s
I20260709 12:21:17.650614  1351 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:17.676520 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 533777us
I20260709 12:21:17.673894  1370 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:17.684221  1370 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
W20260709 12:21:17.691974 31643 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.019s	sys 0.060s
W20260709 12:21:17.695514 31643 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.019s	sys 0.060s
I20260709 12:21:17.666443  1369 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:17.697309 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 539047us
I20260709 12:21:17.701246 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:17.712709 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:17.723387 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:17.728732 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 517141us
I20260709 12:21:17.698572  1293 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.707109  1330 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:17.734104 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.733942 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:17.739267 31669 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:17.739869 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:17.740631 31676 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:17.735064 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:17.742043  1249 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:17.742496  1249 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:17.659770   972 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:17.742903  1249 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:17.743182   972 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:17.745041   972 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.727260  1351 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.748687 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:17.748766 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:17.752624 31669 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:17.746353  1289 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.730978  1312 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.728778  1150 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260709 12:21:17.763131  1379 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:17.763316  1351 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.748937 31792 leader_election.cc:400] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5.
I20260709 12:21:17.725746  1178 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.026s
I20260709 12:21:17.733057  1303 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:17.811456 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:17.812538 31669 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:17.765789 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:17.791816 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 98735.1us
W20260709 12:21:17.754520 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 382354us
I20260709 12:21:17.763183  1293 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:17.817230 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51560.7us
I20260709 12:21:17.747620  1256 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.050s
I20260709 12:21:17.728281  1079 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:17.764629   972 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:17.842388 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:17.845782 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:17.850549 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 198318us
I20260709 12:21:17.764432  1312 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.826998  1102 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.827600  1150 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:17.887308 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:17.891680 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 67092.6us
I20260709 12:21:17.896803  1150 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.899578  1078 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:17.883996  1313 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.791239  1289 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.881819 31675 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:17.904825 31677 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 12:21:17.882360 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 151971us
I20260709 12:21:17.830914 31670 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:17.903276 31669 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:17.831631  1303 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.831193  1377 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:17.882578 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:17.906972 31672 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:21:17.916117 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.901252 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:17.932432 31671 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:21:17.918273 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.841804 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:17.929381 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.944207 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:17.911381  1114 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:17.883677  1102 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.954457 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:17.955258 32373 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:17.955392  1102 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:17.956240 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:17.957134   987 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:17.957662 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:17.959275   931 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.959981 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:17.959805   931 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:17.960067  1075 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:17.960590  1075 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:17.963824 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:17.961758  1075 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:17.911175  1367 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:17.900951  1384 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:17.974046 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:17.977953 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:17.905570  1369 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:17.925863  1256 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:17.887048  1385 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:17.990377 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.975096 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.011554 31749 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:17.922163  1030 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:17.931408  1377 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:18.010144 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:17.954440  1150 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:17.928050  1313 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.010618 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1657) took 1053 ms. Trace:
I20260709 12:21:17.973716 32369 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:17.982723  1379 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:17.998315  1371 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:18.022269 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 332386us
I20260709 12:21:17.999286  1384 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:17.946684  1303 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.027197 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.025684 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:18.023626 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 178128us
I20260709 12:21:18.023605 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:18.011092  1385 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:18.033982  1075 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:18.017832 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:17.998948  1269 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:21:18.064458 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:18.022959 31747 rpcz_store.cc:276] 0709 12:21:16.957237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.964133 (+  6896us) service_pool.cc:224] Handling call
0709 12:21:16.964638 (+   505us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:16.965061 (+   423us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:17.377480 (+412419us) log.cc:844] Serialized 14 byte log entry
0709 12:21:18.010033 (+632553us) log.cc:844] Serialized 14 byte log entry
0709 12:21:18.010215 (+   182us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:18.010284 (+    69us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:18.010338 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.010427 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.010549 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38313,"prepare.queue_time_us":6100,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":2147,"raft.queue_time_us":103082,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":5019,"thread_start_us":985956,"threads_started":3,"wal-append.queue_time_us":516597}
I20260709 12:21:18.014698  1256 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.080119 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:18.080881 31827 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:18.061372  1385 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:18.084585 31824 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:18.064177   987 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:18.065989  1351 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:18.089574 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.091105 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:18.092231  1386 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:21:18.092422 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:18.093698 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.094671 31752 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:18.098901  1403 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:18.099011 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.099578  1403 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.100448 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.102093 31745 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:18.102780 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 209) took 1350 ms. Trace:
I20260709 12:21:18.104996 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:18.106003 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 82494.1us
I20260709 12:21:18.114187 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:18.115458  1102 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:18.115590 31750 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:18.116210  1102 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:21:18.118662  1392 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:21:18.109469  1403 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:18.125378 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:18.128440 31752 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:18.129009 31676 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:18.103170 31748 rpcz_store.cc:276] 0709 12:21:16.752214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.783798 (+ 31584us) service_pool.cc:224] Handling call
0709 12:21:17.273526 (+489728us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0f8c 00007fe8f9cb0ef6 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:17.273750 (+   224us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:17.274142 (+   392us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.274173 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:17.301762 (+ 27589us) log.cc:844] Serialized 26 byte log entry
0709 12:21:17.740581 (+438819us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:17.740634 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:17.740710 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.102448 (+361738us) raft_consensus.cc:1685] finished
0709 12:21:18.102470 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.102711 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7787648,"thread_start_us":377520,"threads_started":1,"wal-append.queue_time_us":545374}
I20260709 12:21:18.123109 31750 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:18.137121  1351 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:18.138784  1388 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 12:21:18.139122  1351 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:18.141570 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.142221  1301 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:18.143045  1351 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:18.137867 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50375.2us
I20260709 12:21:18.135028 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:18.153831  1303 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:18.143800 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 115442us
I20260709 12:21:18.098009  1379 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.144331  1301 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:18.137542  1380 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:18.140798 32373 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:18.145355  1079 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:18.145419   678 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:18.146095 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.145887  1030 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.148036 31827 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:18.148473 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:18.148712 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.150728   931 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:18.152046 31749 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:21:18.151450  1384 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.084055  1382 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:21:18.154086 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:18.068359  1330 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.036s
I20260709 12:21:18.155547 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:18.256435 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 108067us
W20260709 12:21:18.263535 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 108064us
I20260709 12:21:18.156463 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:18.156801  1312 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:18.157177  1351 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:18.158245  1313 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:18.129168 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.161094 31678 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:18.161944  1249 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.164211  1075 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:18.164552  1227 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:18.165475  1194 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:18.166214 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:18.286578 31671 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:18.170006 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:18.172621  1289 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:18.177996 31669 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:18.179064  1290 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:18.179591 31576 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 2 to 5, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 5 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:18.084834  1377 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.184962  1256 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:18.190392 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.191409 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:18.197561 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.186375 31572 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 234) took 8179 ms. Trace:
I20260709 12:21:18.224349 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:18.227897 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.239542 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1700) took 1342 ms. Trace:
I20260709 12:21:18.249835 31792 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:18.273510  1159 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:18.252257 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:18.315271 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:18.243306  1301 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:18.265053 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:18.246152   678 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:21:18.285735 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.287459 31672 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
W20260709 12:21:18.292227 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 138201us
I20260709 12:21:18.310716 31750 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:18.324514 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 72362.6us
I20260709 12:21:18.285075 31747 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
W20260709 12:21:18.316395 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 92192us
I20260709 12:21:18.251528 31675 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:18.316193 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:18.317135 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 201096us
I20260709 12:21:18.274260  1293 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:18.192407  1383 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.328869 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:18.293216  1384 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:18.331367 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 152) took 1028 ms. Trace:
I20260709 12:21:18.254366  1380 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:18.327211 31643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.660s (SENT) [suppressed 13 similar messages]
I20260709 12:21:18.343097  1396 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:18.343510  1405 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:18.343988 31643 leader_election.cc:336] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.705s (SENT)
I20260709 12:21:18.344156  1405 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.344663 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:18.346937  1300 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:18.349401   678 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.349794  1385 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.350646   678 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:18.346971 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:18.356840  1249 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:18.320614  1313 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:18.360822  1313 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.363125  1313 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:18.360863  1078 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260709 12:21:18.364907  1078 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:18.365283  1078 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:18.366812  1382 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:18.368492  1380 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:18.331823   931 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:18.369254   931 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:18.369725   931 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:18.338111  1269 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:18.370548  1269 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:18.370903 32369 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:18.371374 32369 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:18.371582  1269 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.372325 32369 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.373489 32369 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:18.380100  1370 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:18.335404  1377 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:18.384057 31576 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 3 to 4, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 4 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:18.386996 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 137289us
I20260709 12:21:18.387936 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:18.313395 31572 rpcz_store.cc:276] 0709 12:21:10.006577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:10.006766 (+   189us) service_pool.cc:224] Handling call
0709 12:21:18.186103 (+8179337us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:15.443060 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:15.717079 (+274019us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:15.719045 (+  1966us) write_op.cc:435] Acquired schema lock
0709 12:21:15.724441 (+  5396us) tablet.cc:662] Decoding operations
0709 12:21:15.951879 (+227438us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:15.951947 (+    68us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:15.951993 (+    46us) tablet.cc:685] Acquiring locks for 23 operations
0709 12:21:16.182792 (+230799us) tablet.cc:701] Row locks acquired
0709 12:21:16.193288 (+ 10496us) write_op.cc:260] PREPARE: finished
0709 12:21:16.193493 (+   205us) write_op.cc:270] Start()
0709 12:21:16.193630 (+   137us) write_op.cc:276] Timestamp: P: 1783599676193427 usec, L: 0
0709 12:21:16.193671 (+    41us) op_driver.cc:348] REPLICATION: starting
0709 12:21:16.195777 (+  2106us) log.cc:844] Serialized 9422 byte log entry
0709 12:21:16.294718 (+ 98941us) op_driver.cc:464] REPLICATION: finished
0709 12:21:16.599027 (+304309us) write_op.cc:301] APPLY: starting
0709 12:21:16.650549 (+ 51522us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:16.731495 (+ 80946us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:16.731655 (+   160us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:16.762410 (+ 30755us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:16.763152 (+   742us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=23
0709 12:21:16.763200 (+    48us) write_op.cc:312] APPLY: finished
0709 12:21:16.763832 (+   632us) log.cc:844] Serialized 158 byte log entry
0709 12:21:16.797156 (+ 33324us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:16.823617 (+ 26461us) write_op.cc:454] Released schema lock
0709 12:21:16.823713 (+    96us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:16.824328 (+   615us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:16.824363 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":109723,"reported_tablets":32,"child_traces":[["op",{"apply.queue_time_us":149161,"apply.run_cpu_time_us":7965,"apply.run_wall_time_us":279800,"num_ops":23,"prepare.queue_time_us":213100,"prepare.run_cpu_time_us":50876,"prepare.run_wall_time_us":814901,"raft.queue_time_us":1335,"raft.run_cpu_time_us":20274,"raft.run_wall_time_us":146692,"replication_time_us":100855,"thread_start_us":304593,"threads_started":4,"wal-append.queue_time_us":93919,"wal-append.run_cpu_time_us":5756,"wal-append.run_wall_time_us":1501478}]]}
I20260709 12:21:18.390950 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:18.392091 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 49404.9us
I20260709 12:21:18.290788  1290 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:18.342597  1383 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.393316  1290 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.393842 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:18.394686  1290 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:18.394769 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 54332.9us
I20260709 12:21:18.395927  1159 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:18.396507  1159 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:18.397390  1159 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:18.392211  1408 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.076s	user 0.000s	sys 0.002s
I20260709 12:21:18.330390  1293 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:18.399025  1293 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:18.401484  1402 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:18.401576  1387 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:18.402448  1402 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:18.402966  1402 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:18.403098  1194 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:18.403937  1402 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:18.339561 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 74566.4us
I20260709 12:21:18.406347 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:18.389472 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:18.338898 31820 rpcz_store.cc:276] 0709 12:21:17.303403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.376551 (+ 73148us) service_pool.cc:224] Handling call
0709 12:21:17.381805 (+  5254us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:17.404932 (+ 23127us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:17.839167 (+434235us) log.cc:844] Serialized 14 byte log entry
0709 12:21:18.320266 (+481099us) log.cc:844] Serialized 14 byte log entry
0709 12:21:18.331023 (+ 10757us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:18.331077 (+    54us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:18.331123 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.331187 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.331310 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10434,"prepare.queue_time_us":710012,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":19435,"spinlock_wait_cycles":16940416,"thread_start_us":712019,"threads_started":2}
I20260709 12:21:18.385946 31576 catalog_manager.cc:5697] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 3 to 4, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 4 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:18.314258 31825 rpcz_store.cc:276] 0709 12:21:16.897061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.119204 (+222143us) service_pool.cc:224] Handling call
0709 12:21:17.271220 (+152016us) spinlock_profiling.cc:243] Waited 39.2 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:17.271522 (+   302us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:17.693937 (+422415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.693988 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:17.702509 (+  8521us) log.cc:844] Serialized 52 byte log entry
0709 12:21:18.021968 (+319459us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:18.022012 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.022089 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:18.215477 (+193388us) raft_consensus.cc:1685] finished
0709 12:21:18.215510 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:18.239441 (+ 23931us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7432,"raft.queue_time_us":90036,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":33298,"spinlock_wait_cycles":90263296,"thread_start_us":704751,"threads_started":2,"wal-append.queue_time_us":342716}
W20260709 12:21:18.412478 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 79904.3us
I20260709 12:21:18.416570 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:18.415359 31750 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:18.418205 31642 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:18.419277 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:18.419513 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:18.419979 31675 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:18.420150 31672 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:18.420925 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:18.421391  1383 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:18.366192  1078 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.421833 31576 catalog_manager.cc:5697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 2 to 5, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 5 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:18.422854  1178 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:18.428156 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.423489 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.426062  1078 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:18.432597 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:18.438966 31576 catalog_manager.cc:5697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:18.437376 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 43618.6us
I20260709 12:21:18.441061 31750 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:21:18.442557 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.452865 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.453338   987 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:18.453807   987 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:18.454027 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.453933  1399 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:18.455921 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:18.457945 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:18.454301   987 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:18.442515 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:18.467972 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.462168 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:18.463375 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:18.464041 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 74723.8us
I20260709 12:21:18.429422  1380 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:18.472788 31744 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:18.428501  1178 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:18.485489 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.436172  1382 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:18.485852 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:18.497426 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 56393.9us
I20260709 12:21:18.408243  1405 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:18.519814 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 175851us
I20260709 12:21:18.465967  1289 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:18.521639 31671 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:18.495129  1299 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:18.489301  1290 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.467053  1383 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:18.535669 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.486335  1249 raft_consensus.cc:2764] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:21:18.535364 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:18.472570  1150 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:18.472060  1256 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:18.472270  1370 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:21:18.489063  1330 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:18.542094 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 56053.4us
I20260709 12:21:18.551488 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:18.552723 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:18.502305  1387 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:18.472746  1273 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.068s
W20260709 12:21:18.543560 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53223.5us
I20260709 12:21:18.539060 31576 catalog_manager.cc:5697] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 1 to 3, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:18.509179  1351 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:18.545735 31643 leader_election.cc:336] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.781s (SENT)
I20260709 12:21:18.548429 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:18.554360 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:18.547418 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:18.557291 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:18.557634  1385 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:18.557847  1415 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:18.542814  1150 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:18.558881  1289 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:18.559098  1256 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:18.559716 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.560076  1249 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:18.564471  1405 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:18.564870   931 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.571097  1330 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.576789 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:18.534734  1313 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:18.597585 31718 connection.cc:256] RPC call timeout handler was delayed by -1.1817124399999557s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1521s for a response
W20260709 12:21:18.664717 31718 connection.cc:256] RPC call timeout handler was delayed by -1.1007124399998247s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16110000000000002s for a response
I20260709 12:21:18.579442 31824 pending_rounds.cc:85] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 1
I20260709 12:21:18.665438 31824 pending_rounds.cc:107] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 12:21:18.544128  1383 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:18.577709 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:18.586588 31576 catalog_manager.cc:5697] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 2 to 4, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 4 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:18.546481  1403 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.604403 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:18.574735  1387 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:18.605221 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63586.9us
I20260709 12:21:18.612335 31821 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:18.614595  1178 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:18.719589 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44592.6us
I20260709 12:21:18.720508 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:18.588402  1159 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260709 12:21:18.616618 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.618690  1405 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:18.619793  1416 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.082s
I20260709 12:21:18.621276  1400 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:18.575724  1290 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.621872  1404 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:18.624701   987 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.628186 32369 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:21:18.631775  1424 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:21:18.632797 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:18.633994 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:18.598237  1427 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.739846  1404 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:18.665753 31824 raft_consensus.cc:2889] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:18.671459 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 94735.8us
I20260709 12:21:18.604135  1150 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:18.673287 31576 catalog_manager.cc:5697] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 1 to 2, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 2 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:18.808255 31576 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:18.615753 31561 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f6404cb9c2174326b06cbbb0d27d43f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b95aeeaf4c51442b8092cff1de674f2b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:21:18.725003 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 187714us
I20260709 12:21:18.726778 31747 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
W20260709 12:21:18.811058 31561 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 195495us
I20260709 12:21:18.646464  1382 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:18.650811  1415 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.812177  1415 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.812898 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:18.734091 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 156464us
I20260709 12:21:18.813751  1159 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:18.814988  1159 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:18.815379  1159 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:18.816376  1159 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:18.737751 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.819998  1114 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.820794  1114 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.823020  1381 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.823809  1381 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.670279  1383 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:18.826732 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:18.827095 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:18.827945 32378 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:18.828624 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:18.829209 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.831070 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:18.831401 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:18.832309 31751 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:18.720283  1387 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:18.675602 31818 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:18.842420 32378 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:18.719199  1102 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:18.720047  1399 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:18.759495 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:18.844225 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:18.781299 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:18.845603 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.846011  1114 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:18.651146  1030 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.086s
I20260709 12:21:18.732167  1400 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:18.737985  1178 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:18.669543 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:18.870451 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.747004  1079 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:18.796988 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:18.736929  1377 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.831058 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:18.719857  1403 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.836858 32371 raft_consensus.cc:2376] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5. Re-sending same reply.
I20260709 12:21:18.833428 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.854041 32378 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:18.957626 32373 raft_consensus.cc:2410] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
W20260709 12:21:18.957521  1386 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
W20260709 12:21:18.868337 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.870689 31820 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:18.673904  1313 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.871627 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:18.872570 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.909704 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:18.930831 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.836548  1404 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:18.838855  1312 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:18.956192 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:19.034116 31821 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:18.978804 32376 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260709 12:21:18.745529 32377 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:18.869884  1400 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:18.854645 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 134209us
I20260709 12:21:18.984519 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:18.865578 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:19.025247 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 212421us
W20260709 12:21:19.050374 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:18.787621  1194 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:18.959939  1079 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:19.056737  1079 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.976768  1178 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:19.059778  1178 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:18.981078  1277 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:19.062898  1277 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:19.063532  1277 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:18.978384 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:18.996428  1075 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:18.867101  1102 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:19.023530 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:19.095741 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 47259.2us
I20260709 12:21:19.104666 31669 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:19.100662 31671 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:19.108114   987 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:19.108870  1430 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:21:19.109329  1330 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:19.023818  1377 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:18.837033  1427 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:19.120486 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:19.082222 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:19.110795  1420 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.418s	user 0.001s	sys 0.001s
I20260709 12:21:19.125379  1330 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.150130  1178 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:19.189177 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:19.150339 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 361218us
I20260709 12:21:19.152390 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:19.152751 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:18.959205  1405 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:19.192345  1371 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.194504  1371 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:19.108126 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:19.119153   931 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:19.212615 32375 raft_consensus.cc:2410] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 4.
W20260709 12:21:19.152150  1426 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.700s	user 0.003s	sys 0.004s
I20260709 12:21:19.079825  1312 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:19.161027  1114 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.110374  1075 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:19.148142  1381 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:19.087913  1194 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:19.246578 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60482.4us
I20260709 12:21:19.248708 32369 raft_consensus.cc:2410] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:19.249030 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:19.282542 31820 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:19.249428  1428 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.412s	user 0.001s	sys 0.000s
I20260709 12:21:19.249676  1075 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:19.251190  1427 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:19.251765  1423 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.511s	user 0.001s	sys 0.000s
W20260709 12:21:19.259377  1425 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.507s	user 0.001s	sys 0.000s
I20260709 12:21:19.263114  1443 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:19.264670   931 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:19.266085  1438 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.229s	user 0.000s	sys 0.000s
W20260709 12:21:19.267076  1437 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.289s	user 0.001s	sys 0.000s
W20260709 12:21:19.272557 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 120251us
W20260709 12:21:19.239869  1384 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.858s	user 0.008s	sys 0.004s
I20260709 12:21:19.284066 31821 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:19.285938 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:19.197808  1330 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:19.238291 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:19.291846  1301 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.947s	user 0.009s	sys 0.003s
I20260709 12:21:19.286414 31670 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:19.308667 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 162) took 1115 ms. Trace:
W20260709 12:21:19.309295 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:19.300519  1443 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:19.310614 31745 raft_consensus.cc:2410] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260709 12:21:19.310978  1422 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.545s	user 0.000s	sys 0.002s
I20260709 12:21:19.311547  1102 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:19.268854  1379 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.051s
I20260709 12:21:19.290419  1269 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.315727 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:19.317055 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:19.331260 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:19.271328  1114 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:19.322588 31826 rpcz_store.cc:276] 0709 12:21:18.193595 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.193796 (+   201us) service_pool.cc:224] Handling call
0709 12:21:18.243427 (+ 49631us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:18.257098 (+ 13671us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.257120 (+    22us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:18.272347 (+ 15227us) log.cc:844] Serialized 26 byte log entry
0709 12:21:18.502860 (+230513us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:18.502902 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.502996 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:19.304383 (+801387us) raft_consensus.cc:1685] finished
0709 12:21:19.304405 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:19.308599 (+  4194us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3878,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":6982,"thread_start_us":183375,"threads_started":1,"wal-append.queue_time_us":236959}
I20260709 12:21:19.337499 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:19.338667  1381 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:19.339178  1290 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:19.339305 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 71259.7us
W20260709 12:21:19.342406 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:19.354550 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:19.351096 31820 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:19.320330  1382 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:19.355341 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:19.353951 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:19.356284 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:19.359699 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:19.363034 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:19.386999  1435 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:21:19.387313 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:19.388470  1434 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.263s	user 0.001s	sys 0.000s
I20260709 12:21:19.388595 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:19.390071 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:19.386991 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:19.386240 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:19.391426  1429 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.346s	user 0.001s	sys 0.000s
I20260709 12:21:19.393805 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:19.395407  1377 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:19.396844 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:19.322978  1443 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:19.397691 32370 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 3: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:19.346997  1403 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:19.403589 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:19.481289 31826 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:19.481647 31751 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:19.389619  1446 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.482113 31751 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:19.466017  1427 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.466123  1382 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:19.392493  1079 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:19.471127 31820 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:19.471436 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:19.473707 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 118435us
I20260709 12:21:19.474905 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1655) took 1067 ms. Trace:
I20260709 12:21:19.475198 31823 raft_consensus.cc:2376] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4. Re-sending same reply.
I20260709 12:21:19.331702  1451 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:19.324040  1301 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:19.491793 32378 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:19.477831  1290 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:19.493798 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:19.487520 32372 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:19.491736 31751 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:19.498855 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 144367us
I20260709 12:21:19.505609 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:19.352793  1416 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.143s
I20260709 12:21:19.339186  1269 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:19.517393  1440 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.370s	user 0.000s	sys 0.014s
I20260709 12:21:19.563230  1382 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:19.517623  1102 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:19.565270  1102 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:19.566980 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1608) took 1011 ms. Trace:
I20260709 12:21:19.567440 31819 rpcz_store.cc:276] 0709 12:21:18.555184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.574911 (+ 19727us) service_pool.cc:224] Handling call
0709 12:21:18.612966 (+ 38055us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:18.645011 (+ 32045us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.645044 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:18.645699 (+   655us) log.cc:844] Serialized 26 byte log entry
0709 12:21:18.745020 (+ 99321us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:18.996597 (+251577us) log.cc:844] Serialized 14 byte log entry
0709 12:21:19.048728 (+ 52131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:19.089565 (+ 40837us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:19.566720 (+477155us) raft_consensus.cc:1685] finished
0709 12:21:19.566738 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:19.566914 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":103541,"prepare.queue_time_us":194211,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":56013,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":12957,"spinlock_wait_cycles":104087296,"thread_start_us":83499,"threads_started":1,"wal-append.queue_time_us":131845}
I20260709 12:21:19.570178 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:19.520083  1383 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.695s	user 0.010s	sys 0.000s
W20260709 12:21:19.523571  1449 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.204s	user 0.000s	sys 0.000s
I20260709 12:21:19.524006  1454 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:19.500912 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1656) took 1092 ms. Trace:
I20260709 12:21:19.478019  1030 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.488066 32376 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:19.532021 31821 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:19.515837 31673 rpcz_store.cc:276] 0709 12:21:18.407611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.407798 (+   187us) service_pool.cc:224] Handling call
0709 12:21:18.419180 (+ 11382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:18.436961 (+ 17781us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.436994 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:18.466813 (+ 29819us) log.cc:844] Serialized 26 byte log entry
0709 12:21:18.595763 (+128950us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:18.595809 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.595874 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:19.357725 (+761851us) raft_consensus.cc:1685] finished
0709 12:21:19.357756 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:19.474840 (+117084us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29960,"raft.queue_time_us":19343,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":156,"thread_start_us":99698,"threads_started":1,"wal-append.queue_time_us":188100}
I20260709 12:21:19.539098 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:19.490571  1385 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:19.556155 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:19.556234 31751 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:19.560810  1413 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.013s	user 0.002s	sys 0.002s
I20260709 12:21:19.563128 32371 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:19.518806  1445 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:19.571978  1114 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:19.500978  1427 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:19.580439  1150 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.779s	user 0.009s	sys 0.002s
W20260709 12:21:19.581336  1293 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.182s	user 0.010s	sys 0.002s
I20260709 12:21:19.590927 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:19.502871  1446 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:19.595291 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 130300us
W20260709 12:21:20.013350 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 126619us
I20260709 12:21:19.722689 31826 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:19.965474 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 379) took 1004 ms. Trace:
I20260709 12:21:20.031355 32375 rpcz_store.cc:276] 0709 12:21:18.960721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.118969 (+158248us) service_pool.cc:224] Handling call
0709 12:21:19.965143 (+846174us) spinlock_profiling.cc:243] Waited 4.75 ms on lock 0x7b5400595454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:19.965377 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":10926976}
W20260709 12:21:19.654737 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.030323 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.012575 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1678) took 1487 ms. Trace:
W20260709 12:21:19.559602  1289 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.911s	user 0.010s	sys 0.002s
W20260709 12:21:19.795728 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:19.701432 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 383) took 1092 ms. Trace:
I20260709 12:21:20.033082 31747 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:20.008322 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:20.027989 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 674021us
I20260709 12:21:20.016300 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 200) took 1169 ms. Trace:
I20260709 12:21:20.046518 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.028591 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1658) took 1587 ms. Trace:
I20260709 12:21:20.053336 31826 raft_consensus.cc:2410] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 6 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 5.
W20260709 12:21:20.047072 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 508026us
I20260709 12:21:20.047140 31748 rpcz_store.cc:276] 0709 12:21:18.524917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.536297 (+ 11380us) service_pool.cc:224] Handling call
0709 12:21:18.542854 (+  6557us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:18.576870 (+ 34016us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.576904 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:18.595549 (+ 18645us) log.cc:844] Serialized 26 byte log entry
0709 12:21:18.783837 (+188288us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:18.783885 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.783959 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:19.628825 (+844866us) raft_consensus.cc:1685] finished
0709 12:21:19.628858 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.012474 (+383616us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31379,"raft.queue_time_us":34784,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":15016,"thread_start_us":170249,"threads_started":1,"wal-append.queue_time_us":498209}
W20260709 12:21:19.654973  1448 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
W20260709 12:21:19.655431  1387 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.803s	user 0.012s	sys 0.004s
I20260709 12:21:20.054816 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:20.050150 31749 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
W20260709 12:21:19.744048  1400 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.650s	user 0.010s	sys 0.000s
I20260709 12:21:20.049249 32377 rpcz_store.cc:276] 0709 12:21:18.609344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.609526 (+   182us) service_pool.cc:224] Handling call
0709 12:21:18.610037 (+   511us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:19.046519 (+436482us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:19.701136 (+654617us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54000127d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:19.701359 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":310982528}
W20260709 12:21:20.009075  1460 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.471s	user 0.001s	sys 0.000s
W20260709 12:21:19.900039  1404 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.820s	user 0.012s	sys 0.000s
I20260709 12:21:19.955109   678 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:20.050328 31672 rpcz_store.cc:276] 0709 12:21:18.846503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.986423 (+139920us) service_pool.cc:224] Handling call
0709 12:21:18.986863 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:19.028609 (+ 41746us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:19.028642 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:19.079431 (+ 50789us) log.cc:844] Serialized 26 byte log entry
0709 12:21:19.472008 (+392577us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:19.472052 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:19.472132 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:20.015894 (+543762us) raft_consensus.cc:1685] finished
0709 12:21:20.015934 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.016201 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41333,"raft.queue_time_us":115240,"spinlock_wait_cycles":27124992,"thread_start_us":311055,"threads_started":1,"wal-append.queue_time_us":503008}
I20260709 12:21:19.957801  1370 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.050773 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:20.089231 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.089622 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.089999 31820 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:19.966287 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 164) took 1557 ms. Trace:
I20260709 12:21:20.082602 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:19.517393  1381 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260709 12:21:19.614888  1256 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.961s	user 0.003s	sys 0.009s
W20260709 12:21:19.994028  1277 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.930s	user 0.015s	sys 0.000s
I20260709 12:21:20.060007  1289 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:20.051489 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:19.686285  1312 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:19.917991  1078 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.477s	user 0.009s	sys 0.006s
I20260709 12:21:20.047096 31674 rpcz_store.cc:276] 0709 12:21:18.408137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.426762 (+ 18625us) service_pool.cc:224] Handling call
0709 12:21:18.427107 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:18.427502 (+   395us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.427534 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:18.428042 (+   508us) log.cc:844] Serialized 26 byte log entry
0709 12:21:18.734011 (+305969us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:18.734063 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.734157 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:19.500628 (+766471us) raft_consensus.cc:1685] finished
0709 12:21:19.500654 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:19.500825 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":364,"raft.run_cpu_time_us":52,"raft.run_wall_time_us":52,"thread_start_us":300590,"threads_started":1,"wal-append.queue_time_us":420610}
I20260709 12:21:20.052000  1114 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:20.046977 31823 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
W20260709 12:21:20.077631  1380 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.004s	sys 0.011s
I20260709 12:21:19.594720  1371 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.053412 31675 rpcz_store.cc:276] 0709 12:21:18.441636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.441796 (+   160us) service_pool.cc:224] Handling call
0709 12:21:18.471516 (+ 29720us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:18.496312 (+ 24796us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.496344 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:18.537648 (+ 41304us) log.cc:844] Serialized 26 byte log entry
0709 12:21:18.753298 (+215650us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:18.753356 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.753456 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:20.028237 (+1274781us) raft_consensus.cc:1685] finished
0709 12:21:20.028279 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.028505 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13874,"raft.queue_time_us":38419,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":17793,"thread_start_us":164616,"threads_started":1,"wal-append.queue_time_us":448524}
I20260709 12:21:20.045828  1178 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:20.010259  1313 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:20.084568  1402 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.476s	user 0.010s	sys 0.010s
I20260709 12:21:20.052138  1367 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.105729  1277 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.127106  1367 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.129338  1277 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.122502  1178 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.133122  1178 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.083870  1387 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.122800  1312 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.100797  1256 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:20.057821  1445 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.057371  1403 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:20.143863  1387 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.143520  1403 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.088752 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:19.916893  1273 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.174s
I20260709 12:21:20.104142 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.083078  1451 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:19.913574  1431 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.179s
I20260709 12:21:19.615729  1459 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.480s
I20260709 12:21:20.059412  1454 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.146708  1289 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.159660  1289 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.160140  1454 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.108273  1427 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:20.164476  1403 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.165103  1427 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.165856  1427 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.168900  1427 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.085796  1249 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:20.046025  1030 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.172333  1030 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:20.047963  1150 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.093284 31818 rpcz_store.cc:276] 0709 12:21:18.408759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.437201 (+ 28442us) service_pool.cc:224] Handling call
0709 12:21:18.525081 (+ 87880us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:18.841708 (+316627us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:19.965934 (+1124226us) spinlock_profiling.cc:243] Waited 66.2 ms on lock 0x7b54001987d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:19.966207 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":152238976}
I20260709 12:21:20.173861 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.110903 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1609) took 1555 ms. Trace:
I20260709 12:21:20.174173 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:20.175463  1150 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.174592 31822 rpcz_store.cc:276] 0709 12:21:18.555483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.578135 (+ 22652us) service_pool.cc:224] Handling call
0709 12:21:18.578503 (+   368us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:18.610944 (+ 32441us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.611001 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:18.665444 (+ 54443us) log.cc:844] Serialized 26 byte log entry
0709 12:21:19.126272 (+460828us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:19.227205 (+100933us) log.cc:844] Serialized 14 byte log entry
0709 12:21:19.234316 (+  7111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:19.234394 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:20.110648 (+876254us) raft_consensus.cc:1685] finished
0709 12:21:20.110671 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.110838 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37970,"prepare.queue_time_us":95474,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":71223,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":29342,"thread_start_us":336305,"threads_started":1,"wal-append.queue_time_us":580355}
I20260709 12:21:20.176745 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.176862 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 222) took 1682 ms. Trace:
I20260709 12:21:20.177350 31822 raft_consensus.cc:2376] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5. Re-sending same reply.
I20260709 12:21:19.603044  1383 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.112165 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.086688  1400 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:20.189960 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.052306 31819 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:20.112809 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1241) took 2170 ms. Trace:
I20260709 12:21:20.231371 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.232439 31819 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 6 because replica has last-logged OpId of term: 3 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:20.230939 31753 rpcz_store.cc:276] 0709 12:21:17.942056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:17.960856 (+ 18800us) service_pool.cc:224] Handling call
0709 12:21:17.961211 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:17.961632 (+   421us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:17.961663 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:17.962188 (+   525us) log.cc:844] Serialized 26 byte log entry
0709 12:21:18.321681 (+359493us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:18.321725 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.321812 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:19.398453 (+1076641us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:20.112444 (+713991us) raft_consensus.cc:1685] finished
0709 12:21:20.112473 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.112719 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3640,"raft.queue_time_us":11247,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":31051,"spinlock_wait_cycles":5564672,"thread_start_us":358817,"threads_started":1,"wal-append.queue_time_us":492878}
I20260709 12:21:20.233314 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.233721 31819 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:20.234576 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.177472 31750 rpcz_store.cc:276] 0709 12:21:18.494405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.494540 (+   135us) service_pool.cc:224] Handling call
0709 12:21:18.495007 (+   467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:18.519635 (+ 24628us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.519700 (+    65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:18.520374 (+   674us) log.cc:844] Serialized 26 byte log entry
0709 12:21:18.646995 (+126621us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:18.647037 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:18.647125 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:20.165016 (+1517891us) spinlock_profiling.cc:243] Waited 24.2 ms on lock 0x7b24002a9c08. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9b6e9c4 00007fe8f9b6cacb 00007fe8f9d27dc9 00007fe8f9d27a55 00007fe8f9d26fb0 00007fe8fc99d488 00007fe8fc99dbe8 00007fe8fc975398 00007fe8fc97076a 00007fe8fc96fbb6
0709 12:21:20.176374 (+ 11358us) raft_consensus.cc:1685] finished
0709 12:21:20.176412 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.176770 (+   358us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24055,"raft.queue_time_us":24289,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":13818,"spinlock_wait_cycles":55660416,"thread_start_us":110646,"threads_started":1,"wal-append.queue_time_us":329110}
I20260709 12:21:20.235934 31750 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:20.236690 31753 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:20.236881 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 380) took 1214 ms. Trace:
I20260709 12:21:20.197901 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:20.052187  1293 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.113364  1416 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:20.114095  1455 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.615s	user 0.000s	sys 0.004s
W20260709 12:21:20.114105  1444 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.871s	user 0.000s	sys 0.003s
I20260709 12:21:20.088418   678 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:20.119275  1371 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:20.119408  1377 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:20.092691  1404 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.119712  1301 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:20.120221 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 68818.9us
I20260709 12:21:20.238138  1150 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:20.237329 32371 rpcz_store.cc:276] 0709 12:21:19.022373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.107834 (+ 85461us) service_pool.cc:224] Handling call
0709 12:21:20.236279 (+1128445us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b540074e854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:20.236813 (+   534us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":806194048}
I20260709 12:21:20.247586  1150 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:20.236702 32374 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:21:20.248195  1150 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:20.249456  1277 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:20.250847  1377 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:20.251171 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 384) took 1601 ms. Trace:
I20260709 12:21:20.251289  1377 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:20.251469  1454 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:20.182920  1445 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.251684 32374 rpcz_store.cc:276] 0709 12:21:18.650064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.650455 (+   391us) service_pool.cc:224] Handling call
0709 12:21:19.251390 (+600935us) spinlock_profiling.cc:243] Waited 5.25 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:20.236430 (+985040us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:20.248400 (+ 11970us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.250945 (+  2545us) spinlock_profiling.cc:243] Waited 985 ms on lock 0x7b540074e850. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:20.251101 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2277037184}
I20260709 12:21:20.241348  1381 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.253170  1381 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.253557  1416 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.255359  1381 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:20.272480  1194 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.126122 31821 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:20.135603 31751 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
W20260709 12:21:20.122016 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 71307.9us
W20260709 12:21:20.115998  1380 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.004s	sys 0.011s
I20260709 12:21:20.147156 31826 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:20.276137 31826 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:20.276724 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1613) took 1675 ms. Trace:
I20260709 12:21:20.277036 31826 rpcz_store.cc:276] 0709 12:21:18.601252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.275484 (+1674232us) service_pool.cc:224] Handling call
0709 12:21:20.275828 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:20.276478 (+   650us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.276673 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1792}
I20260709 12:21:20.277712  1380 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:20.278270 31826 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:20.278579  1385 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:20.278304  1380 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.279361 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1710) took 1188 ms. Trace:
I20260709 12:21:20.161969  1289 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:20.163374  1453 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.838s	user 0.001s	sys 0.001s
I20260709 12:21:20.107674  1447 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.211251 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:20.280843  1457 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.802s	user 0.005s	sys 0.000s
I20260709 12:21:20.281265  1380 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.281140  1447 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.282032  1330 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.282083 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1614) took 1680 ms. Trace:
I20260709 12:21:20.282876 31826 rpcz_store.cc:276] 0709 12:21:18.601538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.277789 (+1676251us) service_pool.cc:224] Handling call
0709 12:21:20.278091 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:20.281817 (+  3726us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.282005 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:20.283995  1447 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.284148 31826 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:20.284770 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1707) took 1194 ms. Trace:
I20260709 12:21:20.285168 31826 rpcz_store.cc:276] 0709 12:21:19.089906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.283647 (+1193741us) service_pool.cc:224] Handling call
0709 12:21:20.283974 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:20.284522 (+   548us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.284704 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:20.178474 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.286262 31822 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:20.287397 31822 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:20.287959 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1713) took 1185 ms. Trace:
I20260709 12:21:20.288245 31822 rpcz_store.cc:276] 0709 12:21:19.102217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.286916 (+1184699us) service_pool.cc:224] Handling call
0709 12:21:20.287231 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:20.287733 (+   502us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.287909 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:20.191491 31745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1249) took 2001 ms (client timeout 2566 ms). Trace:
W20260709 12:21:20.219410  1439 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.108s	user 0.001s	sys 0.001s
I20260709 12:21:20.099761  1370 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.221060  1290 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:20.220907 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.221486  1353 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:20.290310 31745 rpcz_store.cc:269] 0709 12:21:18.190282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.191066 (+   784us) service_pool.cc:224] Handling call
0709 12:21:20.191192 (+2000126us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b54001ed7d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:20.191412 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2281626368}
I20260709 12:21:20.294132  1353 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.222616 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.181310  1415 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.234400 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1612) took 1651 ms. Trace:
I20260709 12:21:20.184887  1269 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.224730  1383 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:20.225656 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.335304 31747 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:20.139276  1367 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.149492  1312 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.144881  1382 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.338569 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:20.337607 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.339293 31672 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:20.340128 31749 raft_consensus.cc:2368] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5.
I20260709 12:21:20.228322  1102 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.229583 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:20.349557 31820 raft_consensus.cc:2376] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4. Re-sending same reply.
I20260709 12:21:20.190478  1387 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.221587 31746 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:20.114996  1462 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.138s
I20260709 12:21:20.207317  1451 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.362973  1451 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:20.363770  1466 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.170s	user 0.001s	sys 0.000s
I20260709 12:21:20.371163 31818 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:20.373196  1312 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.373399 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:20.256098  1416 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.239436  1446 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.256675  1371 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.250476 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:20.251303 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53564.3us
I20260709 12:21:20.258710 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1605) took 1704 ms. Trace:
I20260709 12:21:20.333572 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:20.346011  1415 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.249945  1078 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.012s
I20260709 12:21:20.117142 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.385004 31823 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:20.385107 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.385637 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1719) took 1276 ms. Trace:
I20260709 12:21:20.256260  1404 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.266184 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:20.385205 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.145267  1464 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:20.267503  1293 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.268674  1377 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.394071  1370 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.394798  1178 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.395334  1178 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.395916  1377 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.274721 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:20.334272 31819 rpcz_store.cc:276] 0709 12:21:18.583014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.233065 (+1650051us) service_pool.cc:224] Handling call
0709 12:21:20.233381 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:20.234130 (+   749us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.234346 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:20.397377  1194 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:20.397607  1178 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.397850 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.398738  1404 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:20.399307 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.400110  1404 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
W20260709 12:21:20.136142  1402 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.517s	user 0.011s	sys 0.010s
I20260709 12:21:20.286343 31821 rpcz_store.cc:276] 0709 12:21:19.091002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.046137 (+955135us) service_pool.cc:224] Handling call
0709 12:21:20.279180 (+233043us) spinlock_profiling.cc:243] Waited 24.4 ms on lock 0x7b54000fcb54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:20.279300 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":56154496}
I20260709 12:21:20.355029  1382 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.220498  1400 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.402081 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:20.212445 31825 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
W20260709 12:21:20.396763  1443 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.904s	user 0.012s	sys 0.000s
I20260709 12:21:20.403258 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:20.403618  1382 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.403364  1443 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.403829 31821 raft_consensus.cc:2376] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5. Re-sending same reply.
I20260709 12:21:20.404690 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.293007 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.393891 31642 leader_election.cc:400] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5.
I20260709 12:21:20.405459 31642 leader_election.cc:403] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:20.405687 31750 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:20.293401 31826 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260709 12:21:20.299672 31748 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:20.351969 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:20.223209  1458 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.455s	user 0.003s	sys 0.000s
I20260709 12:21:20.406585 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1708) took 1316 ms. Trace:
I20260709 12:21:20.336232 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:20.407048 31826 rpcz_store.cc:276] 0709 12:21:19.090331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.286021 (+1195690us) service_pool.cc:224] Handling call
0709 12:21:20.293130 (+  7109us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:20.406198 (+113068us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.406510 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:20.347107 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1606) took 1792 ms. Trace:
W20260709 12:21:20.349841 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 138675us
I20260709 12:21:20.407912 31827 rpcz_store.cc:276] 0709 12:21:18.554552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.579919 (+ 25367us) service_pool.cc:224] Handling call
0709 12:21:18.607625 (+ 27706us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:18.643439 (+ 35814us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.643487 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:18.644206 (+   719us) log.cc:844] Serialized 26 byte log entry
0709 12:21:18.880267 (+236061us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:19.268197 (+387930us) log.cc:844] Serialized 14 byte log entry
0709 12:21:19.268485 (+   288us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:19.268559 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:20.009031 (+740472us) spinlock_profiling.cc:243] Waited 7.43 ms on lock 0x7b180028e100. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:21:20.219038 (+210007us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b2400234c08. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9b6e9c4 00007fe8f9b6cacb 00007fe8f9d27dc9 00007fe8f9d27a55 00007fe8f9d2709d 00007fe8fc95910e 00007fe8fc97088c 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221
0709 12:21:20.346807 (+127769us) raft_consensus.cc:1685] finished
0709 12:21:20.346846 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.347027 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":286111,"prepare.queue_time_us":201109,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":10566,"raft.queue_time_us":41215,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":28413,"spinlock_wait_cycles":28685824,"thread_start_us":428661,"threads_started":2,"wal-append.queue_time_us":402504}
I20260709 12:21:20.349689 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:20.411307 31675 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 3: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:20.358057 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.292227  1290 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:20.292761  1381 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:20.352872  1367 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:20.427280 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.428020 31748 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:20.333262  1454 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.340744  1330 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:20.338805  1445 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.379037 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.375548  1387 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.225598  1159 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:20.385995 31823 rpcz_store.cc:276] 0709 12:21:19.109354 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.116857 (+1007503us) service_pool.cc:224] Handling call
0709 12:21:20.385575 (+268718us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:20.387804 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 137409us
I20260709 12:21:20.383056 31824 rpcz_store.cc:276] 0709 12:21:18.554252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.578893 (+ 24641us) service_pool.cc:224] Handling call
0709 12:21:18.579229 (+   336us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:18.870556 (+291327us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:18.870589 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:18.909095 (+ 38506us) log.cc:844] Serialized 52 byte log entry
0709 12:21:19.168553 (+259458us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:19.249951 (+ 81398us) log.cc:844] Serialized 14 byte log entry
0709 12:21:19.250375 (+   424us) log.cc:844] Serialized 14 byte log entry
0709 12:21:19.250540 (+   165us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:19.250611 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:20.110262 (+859651us) spinlock_profiling.cc:243] Waited 2.71 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:20.258448 (+148186us) raft_consensus.cc:1685] finished
0709 12:21:20.258470 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.258613 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":57357,"prepare.queue_time_us":82680,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":14469,"raft.run_cpu_time_us":506,"raft.run_wall_time_us":19516,"spinlock_wait_cycles":6236416,"thread_start_us":131756,"threads_started":1,"wal-append.queue_time_us":500013}
W20260709 12:21:20.391127 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.396803 31751 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:20.279183  1385 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:20.400529  1404 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:20.465036 31824 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:20.428948  1467 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:21:20.289543 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:20.469554 31822 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:20.292197 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:20.345832 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:20.470793 31672 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:20.471804 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:20.413610 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1603) took 1860 ms. Trace:
I20260709 12:21:20.472488 31825 rpcz_store.cc:276] 0709 12:21:18.553660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:18.579841 (+ 26181us) service_pool.cc:224] Handling call
0709 12:21:20.413444 (+1833603us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b540030edd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:20.413554 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3409792256}
I20260709 12:21:20.473531 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.474117 31825 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:20.417979 31677 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:20.410390 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:20.418057 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:20.376150  1462 consensus_queue.cc:1243] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:21:20.429186 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:20.476002 31676 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:20.383790  1312 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:20.431700 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.476619 31827 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:20.476711 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:20.435381 31820 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:20.374078 31818 raft_consensus.cc:2376] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5. Re-sending same reply.
I20260709 12:21:20.435593 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:20.434106 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:20.478564 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44544.4us
I20260709 12:21:20.428169 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.436626 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.440846 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:20.401582 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.468281 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:20.481320 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:20.439668 31752 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.007s	sys 0.003s
I20260709 12:21:20.477358 31669 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:20.417299  1415 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.432596  1367 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.611385 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:20.615756 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:20.616456 31676 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:20.536562 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.617663 31827 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:20.536149 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.482740 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.619580 31820 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:20.619861 31818 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:20.484447 31749 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
W20260709 12:21:20.621748 31752 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.007s	sys 0.009s
W20260709 12:21:20.614809 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 204500us
W20260709 12:21:20.507946 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 89967us
I20260709 12:21:20.623003 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:20.624356 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:20.507350 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 215210us
I20260709 12:21:20.610689 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.512626 31672 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:20.625715 31747 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:20.555204 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:20.500994 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:20.627132 32377 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
W20260709 12:21:20.597153 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44592.9us
I20260709 12:21:20.633140 31751 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:20.621444 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.615743 31753 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:20.636684 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:20.486516 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:20.587543 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.634227 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:20.638149 32375 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:21:20.657120 31748 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:20.464720  1464 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.698925 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:20.482843  1370 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:20.536511  1468 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.185s	user 0.000s	sys 0.000s
I20260709 12:21:20.698015 31822 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:20.464927  1446 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:20.644959 31642 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:20.555073  1289 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:20.604823  1443 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:20.825359 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:20.603132 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:20.603933  1330 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:20.702311 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.832031 31673 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:20.636889  1471 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:20.604497 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:20.657338 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:20.841816 31675 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:20.711493 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.711220  1464 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.658833 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.506064  1312 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.561168 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:20.610920  1290 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.658941  1383 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:20.636278  1079 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:20.794438 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 78159.2us
I20260709 12:21:20.435531  1381 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:20.795485 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 169846us
I20260709 12:21:20.807250 32372 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:20.554200  1380 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:20.597275  1396 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.114s
I20260709 12:21:20.438438  1454 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.636152 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:20.808838 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.806581 32343 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:20.868921  1353 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.656980  1416 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:20.813659  1194 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:20.825935 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 181062us
W20260709 12:21:20.825282 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 381161us
I20260709 12:21:20.467995  1445 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.692472  1462 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:20.816843 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.932638  1462 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:20.932849 31748 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:20.933107  1462 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:20.827898 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 191380us
I20260709 12:21:20.934222 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:20.852186 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.805382  1178 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:20.864081 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.804555  1370 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:20.871456 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:20.815665  1399 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:20.841140  1102 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:20.795755 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:20.883498 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.883145 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:20.892166 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:20.943858 31751 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:20.871587 32373 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:20.691954  1404 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:20.905238 31675 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:20.817787 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:20.831583  1473 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.061s
I20260709 12:21:20.874245 31745 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:20.909358  1290 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:20.987027 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:20.987799 31675 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:20.891253  1079 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:20.910264  1478 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.189s	user 0.000s	sys 0.000s
I20260709 12:21:20.836030  1446 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.912546  1464 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:20.913321 31749 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1e27b968c91a4906aec537421f481269"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
W20260709 12:21:21.000327 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66203us
I20260709 12:21:20.883589  1383 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.916725  1387 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:20.917096  1477 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:21:20.917565  1385 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:21:20.894843 31746 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:20.923291 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:21.026899 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 103727us
I20260709 12:21:21.028465 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:20.881659  1330 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:20.926913 31670 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:20.929255  1415 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.929040 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:20.931332  1467 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5.
I20260709 12:21:20.932011  1445 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:20.934391 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:20.841848  1471 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:20.943524 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60441.2us
I20260709 12:21:21.038519 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:20.942875 31747 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:20.983022  1249 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:20.984231 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:20.949515  1102 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:20.990166 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:20.920974  1380 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:20.991137 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:20.926680 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 226) took 1436 ms. Trace:
I20260709 12:21:20.943499  1370 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:21.000890 31749 raft_consensus.cc:606] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:21.066816 31749 raft_consensus.cc:614] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:20.914201 32370 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:20.942488  1399 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:20.983808  1416 raft_consensus.cc:2764] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:21:21.010491 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:20.990475  1404 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:21.030792 31678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.000s	sys 0.013s
W20260709 12:21:21.076272 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 65947.5us
I20260709 12:21:21.077579 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:21.039871 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:21.046691 31822 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:21.040069 31819 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
W20260709 12:21:21.068322 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 139448us
I20260709 12:21:21.039850  1396 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:21.041693  1473 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:21.041986  1249 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:21.067001 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 260504us
I20260709 12:21:21.050200  1385 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:21.040588  1471 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:21.086148 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:21.073657  1301 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:21.081285 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1682) took 2029 ms. Trace:
W20260709 12:21:21.072137  1150 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.823s	user 0.024s	sys 0.000s
I20260709 12:21:21.097750 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:21.080403 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.068264  1227 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:21.100385 31752 rpcz_store.cc:276] 0709 12:21:19.051949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.145864 (+ 93915us) service_pool.cc:224] Handling call
0709 12:21:19.203099 (+ 57235us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:19.231844 (+ 28745us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:19.231889 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:19.276048 (+ 44159us) log.cc:844] Serialized 26 byte log entry
0709 12:21:19.847006 (+570958us) spinlock_profiling.cc:243] Waited 3.56 ms on lock 0x7b180048e380. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:21:20.626432 (+779426us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:20.626476 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.626545 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:21.073979 (+447434us) raft_consensus.cc:1685] finished
0709 12:21:21.074002 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:21.081148 (+  7146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":66958,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":3874,"spinlock_wait_cycles":8191232,"thread_start_us":779204,"threads_started":1,"wal-append.queue_time_us":1250479}
I20260709 12:21:21.108431 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:21.116657 31673 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:21.084966  1473 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.085140  1415 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:21.091144  1249 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.096035  1256 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:21.128374 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:21.100601  1467 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:21.131485  1446 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:21.135277  1488 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:21:21.121889 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:21.136881   931 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:21.137166 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:21.137663   931 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:21.138815  1387 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:21.139498  1450 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:21.183228  1330 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.186692  1399 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:21.191416 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:21.065426 31744 rpcz_store.cc:276] 0709 12:21:19.489968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:19.491872 (+  1904us) service_pool.cc:224] Handling call
0709 12:21:19.583116 (+ 91244us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:19.651726 (+ 68610us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:19.651777 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:19.652550 (+   773us) log.cc:844] Serialized 26 byte log entry
0709 12:21:20.103734 (+451184us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:20.115122 (+ 11388us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:20.117463 (+  2341us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:20.896902 (+779439us) raft_consensus.cc:1685] finished
0709 12:21:20.897401 (+   499us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:20.921684 (+ 24283us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1454,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":18277,"thread_start_us":72139,"threads_started":1,"wal-append.queue_time_us":499038}
I20260709 12:21:21.200604 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:21.140419 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1721) took 2030 ms. Trace:
I20260709 12:21:21.108701  1447 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:21.108265 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:21.141450  1159 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.673s	user 0.010s	sys 0.000s
I20260709 12:21:21.095760  1102 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.142138  1273 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:21.142318 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:21.108212  1385 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:21:21.143743 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1717) took 2035 ms. Trace:
W20260709 12:21:21.146883  1486 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:21:21.147184  1396 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:21.147393  1415 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:21.106841  1150 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:21.102454  1312 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:21.149125 31675 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:21.152107 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:21.204869  1447 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:21.206868  1290 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:21.157388  1371 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:21.159142 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:21:21.227346 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 68278.3us
I20260709 12:21:21.162878  1451 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:21:21.163589  1490 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:21:21.166452   678 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:21.167069  1471 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.169860 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.121173  1473 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:21.171821  1496 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:21:21.121393  1445 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.051s
I20260709 12:21:21.174069 31748 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:21:21.273579 31748 raft_consensus.cc:606] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Received request to transfer leadership to TS b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:21.177742 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.179015  1381 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:21.180384  1380 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.183224 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:21.274111 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 176412us
I20260709 12:21:21.274807 31753 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.195356 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:21.201834 31821 rpcz_store.cc:276] 0709 12:21:19.109910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.441315 (+1331405us) service_pool.cc:224] Handling call
0709 12:21:20.441805 (+   490us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:20.468436 (+ 26631us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:20.468467 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:20.482902 (+ 14435us) log.cc:844] Serialized 52 byte log entry
0709 12:21:20.869975 (+387073us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:21.066300 (+196325us) log.cc:844] Serialized 14 byte log entry
0709 12:21:21.076555 (+ 10255us) log.cc:844] Serialized 14 byte log entry
0709 12:21:21.076749 (+   194us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.076817 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:21.140157 (+ 63340us) raft_consensus.cc:1685] finished
0709 12:21:21.140180 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:21.140356 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43720,"prepare.queue_time_us":87446,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":66,"raft.queue_time_us":26838,"raft.run_cpu_time_us":447,"raft.run_wall_time_us":68673,"thread_start_us":514985,"threads_started":2,"wal-append.queue_time_us":497459}
I20260709 12:21:21.209513 31827 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:21.211050 31826 rpcz_store.cc:276] 0709 12:21:19.108808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.437138 (+1328330us) service_pool.cc:224] Handling call
0709 12:21:20.471262 (+ 34124us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:20.602667 (+131405us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:20.602718 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:20.603352 (+   634us) log.cc:844] Serialized 26 byte log entry
0709 12:21:21.046402 (+443050us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:21.070041 (+ 23639us) log.cc:844] Serialized 14 byte log entry
0709 12:21:21.070246 (+   205us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.070322 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:21.143497 (+ 73175us) raft_consensus.cc:1685] finished
0709 12:21:21.143519 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:21.143674 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96670,"prepare.queue_time_us":23464,"prepare.run_cpu_time_us":530,"prepare.run_wall_time_us":754,"raft.queue_time_us":91821,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":14474,"thread_start_us":334729,"threads_started":1,"wal-append.queue_time_us":450556}
W20260709 12:21:21.224176 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 81956.3us
I20260709 12:21:21.155788   931 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.208923  1289 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:21.232582 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.234567  1150 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:21.235463 31674 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:21.236557  1387 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260709 12:21:21.237463 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.231534  1447 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.210060 31670 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
W20260709 12:21:21.288450 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 93256.4us
W20260709 12:21:21.259956 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 174180us
I20260709 12:21:21.249411  1399 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:21.252635  1383 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:21.262508 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.218840  1396 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.265997 31676 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:21.271375  1464 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.269613 31671 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:21.277026 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:21.226778  1290 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.275921  1382 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.289696 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:21.232321  1079 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:21.291462 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:21.294677  1467 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:21.302476  1470 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:21.303089  1470 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:21.303565  1470 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.261559 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:21.300190 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:21.218120  1273 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:21.265234  1312 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:21.269668 32369 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
W20260709 12:21:21.328397 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.296751 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:21.338989 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49354us
W20260709 12:21:21.354677 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44958us
I20260709 12:21:21.410687  1079 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:21.305611  1290 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:21.410871 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 114301us
I20260709 12:21:21.310803 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:21.349756 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:21.388495 32369 pending_rounds.cc:85] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 5
I20260709 12:21:21.297263   678 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.297806  1443 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:21.297091  1396 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.349450  1445 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:21.503824 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:21.355840  1227 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.296797  1367 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:21.387833  1496 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20260709 12:21:21.506739 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 96882.7us
I20260709 12:21:21.350216  1382 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:21.293238  1399 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.356115  1387 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:21.511969 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 211858us
I20260709 12:21:21.515333 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.588853 31673 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:21.350392  1454 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:21.518795 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 176) took 1071 ms. Trace:
I20260709 12:21:21.306192  1464 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:21.498423 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:21:21.600416 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 102161us
I20260709 12:21:21.601446 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:21.514406 32369 pending_rounds.cc:107] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 4.6
W20260709 12:21:21.567456 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 63714.6us
I20260709 12:21:21.607110 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:21.500555 31626 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 87315.7us
I20260709 12:21:21.390383  1312 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.506434 32372 raft_consensus.cc:1217] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Deduplicated request from leader. Original: 3.5->[4.6-4.6]   Dedup: 4.6->[]
I20260709 12:21:21.580219 31822 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:21.356307  1472 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.159s
I20260709 12:21:21.534034  1445 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:21.585146  1079 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:21.603022 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:21.608642 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:21.536758  1443 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:21.626300 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:21.611999 31820 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:21.598352 31818 rpcz_store.cc:276] 0709 12:21:20.447605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.271771 (+824166us) service_pool.cc:224] Handling call
0709 12:21:21.286976 (+ 15205us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:21.287236 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:21.302977 (+ 15741us) log.cc:844] Serialized 14 byte log entry
0709 12:21:21.518384 (+215407us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:21.518442 (+    58us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:21.518495 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.518570 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:21.518734 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1908,"prepare.queue_time_us":16830,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":1610,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":11449,"thread_start_us":215042,"threads_started":1}
I20260709 12:21:21.616186 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.628597 31673 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:21.610983 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:21.598522  1399 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.623653 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:21.597955 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1739) took 1299 ms. Trace:
I20260709 12:21:21.610710 32369 raft_consensus.cc:2889] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:21.634934 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:21.637825 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:21.625439   931 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:21.582026  1227 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.606417  1387 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:21.669293 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1187) took 1106 ms. Trace:
I20260709 12:21:21.646718 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.661562 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 204) took 1325 ms. Trace:
I20260709 12:21:21.660774 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.690869 31674 rpcz_store.cc:276] 0709 12:21:20.335851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.335963 (+   112us) service_pool.cc:224] Handling call
0709 12:21:21.661325 (+1325362us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b54003926d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:21.661473 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1903042944}
I20260709 12:21:21.633877 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:21:21.692601 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58833.6us
I20260709 12:21:21.689339 31675 rpcz_store.cc:276] 0709 12:21:20.562829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.075057 (+512228us) service_pool.cc:224] Handling call
0709 12:21:21.669131 (+594074us) spinlock_profiling.cc:243] Waited 45.8 ms on lock 0x7b54002b0554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:21.669232 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105436544}
I20260709 12:21:21.694808 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:21.662209 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 177) took 1073 ms. Trace:
I20260709 12:21:21.695433 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:21.674363 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.698455 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:21.699290 31824 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:21.695873 31827 rpcz_store.cc:276] 0709 12:21:20.589109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.690143 (+101034us) service_pool.cc:224] Handling call
0709 12:21:21.662032 (+971889us) spinlock_profiling.cc:243] Waited 366 ms on lock 0x7b54000fa854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:21.662149 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":841942400}
W20260709 12:21:21.659369  1404 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.576s	user 0.007s	sys 0.005s
I20260709 12:21:21.702723 31749 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:21.619050  1454 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:21.725003 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 371548us
I20260709 12:21:21.725963 31753 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:21.630185  1445 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:21.736377 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.631870  1289 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:21.726668 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1702) took 1293 ms. Trace:
I20260709 12:21:21.630003  1079 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:21.738868 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 101164us
I20260709 12:21:21.745745 31753 rpcz_store.cc:276] 0709 12:21:20.433269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.635624 (+202355us) service_pool.cc:224] Handling call
0709 12:21:21.182962 (+547338us) spinlock_profiling.cc:243] Waited 6 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:21.726574 (+543612us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13791104}
I20260709 12:21:21.696331 31675 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:21.748160 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.677781  1178 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:21.631610 31670 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:21.681340  1382 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:21.675094  1194 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.736s	user 0.010s	sys 0.003s
I20260709 12:21:21.639081 32372 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
W20260709 12:21:21.689349  1462 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.743s	user 0.013s	sys 0.000s
I20260709 12:21:21.744594 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:21.776487 31747 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:21.687301  1467 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:21.726487  1249 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.537s	user 0.011s	sys 0.000s
I20260709 12:21:21.644025  1446 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.661474  1402 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:21.752282 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:21.808396  1402 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.808873  1402 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:21.809013  1269 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:21.809095 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 57263.1us
I20260709 12:21:21.809623  1269 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.738188 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.810571 31673 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:21.811709 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.812409 31673 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:21.754410 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:21.635131 31819 rpcz_store.cc:276] 0709 12:21:20.291298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.400293 (+108995us) service_pool.cc:224] Handling call
0709 12:21:21.579529 (+1179236us) spinlock_profiling.cc:243] Waited 419 ms on lock 0x7b540043cbd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:21.590653 (+ 11124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":964184960}
I20260709 12:21:21.643755 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 384) took 1163 ms. Trace:
I20260709 12:21:21.815871  1387 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:21.646037  1443 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.815779 32369 rpcz_store.cc:276] 0709 12:21:20.479974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.541064 (+ 61090us) service_pool.cc:224] Handling call
0709 12:21:21.268773 (+727709us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:21.269095 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:21.643386 (+374291us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.643680 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":62405504}
I20260709 12:21:21.649528  1312 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:21.813580 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.803870 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:21.819783 31673 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:21.664762  1464 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:21.757380 31678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.002s	sys 0.009s
I20260709 12:21:21.634999  1451 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:21.670418  1273 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:21.821254  1464 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:21.691591   931 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:21.822037   931 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:21.821898  1464 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.709232  1507 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:21.824786  1507 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.825188  1507 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.828202 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.644557  1513 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:21.828943  1227 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:21.829303 31673 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:21.727210  1377 raft_consensus.cc:2764] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
I20260709 12:21:21.700721  1500 raft_consensus.cc:2764] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260709 12:21:21.644255  1383 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.115s
I20260709 12:21:21.740904  1454 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:21.749635  1400 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:21.832221  1454 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:21.832360  1400 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:21.832777  1447 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.533s	user 0.006s	sys 0.002s
I20260709 12:21:21.833063  1400 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.833266  1447 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.751360  1404 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.834980  1404 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.838369 31671 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:21.799261  1467 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.755726  1079 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.775170  1382 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.841339  1382 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.843562  1382 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:21.843937  1404 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:21.807160  1470 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.503s	user 0.010s	sys 0.000s
I20260709 12:21:21.845585  1470 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.795303 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:21.847754 31745 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:21.799072  1462 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:21.807509  1446 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:21.810073  1269 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.851078  1462 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:21.854267  1473 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.571s	user 0.000s	sys 0.008s
I20260709 12:21:21.810096 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:21.895061  1473 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:21.895852 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 85836.9us
I20260709 12:21:21.898988  1501 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.900345 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:21.901131 31753 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:21.905582 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:21.906944 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:21.754115  1178 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:21.908103 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:21.908941  1178 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.910719  1178 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:21.804205 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 178452us
W20260709 12:21:21.911770 31824 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 173) took 1465 ms (client timeout 1893 ms). Trace:
W20260709 12:21:21.912283 31824 rpcz_store.cc:269] 0709 12:21:20.446514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.500585 (+ 54071us) service_pool.cc:224] Handling call
0709 12:21:21.911556 (+1410971us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x7b54001973d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:21.911682 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1546084224}
I20260709 12:21:21.913579 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:21.913877  1289 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:21.848749  1470 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:21.915457 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:21.662912  1471 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:21.916231  1514 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:21.851081  1446 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:21.855464 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:21.794268  1516 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:21.918547  1514 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:21.919407  1516 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:21.919955  1516 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:21.920563 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:21.859057 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:21.923005 32373 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:21.922892  1385 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:21.861428  1290 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:21.924551  1385 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:21.925103  1385 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:21.925494  1385 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:21.926192  1517 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:21.926759  1290 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:21.824268  1464 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:21.863287  1506 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:21:21.794531  1194 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:21.929932  1517 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:21.932667 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.933631 31672 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:21.936228 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 169) took 1490 ms. Trace:
I20260709 12:21:21.936780 31822 rpcz_store.cc:276] 0709 12:21:20.445409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.104537 (+659128us) service_pool.cc:224] Handling call
0709 12:21:21.579731 (+475194us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:21.616271 (+ 36540us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.935962 (+319691us) spinlock_profiling.cc:243] Waited 475 ms on lock 0x7b540043cbd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:21.936143 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1092134272}
I20260709 12:21:21.937397 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:21.938992 32373 raft_consensus.cc:2410] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 6 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:21.942098  1396 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:21.866047 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:21.953845 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 317036us
I20260709 12:21:21.954782 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:21.934572 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.898432  1473 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:21.868185 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.958556 31673 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:21.959967 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:21.960088 31749 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:21.960582 31673 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:21.960278 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:21.863524  1399 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:21.874795 31825 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:21:21.977125 31825 pending_rounds.cc:85] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 5
I20260709 12:21:21.977491 31825 pending_rounds.cc:107] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:21:21.977891 31825 raft_consensus.cc:2889] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:21.982697   678 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.983428 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:21.984522 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:21.983451   678 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.877067  1454 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.881033  1450 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:21.806228  1249 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.039093 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:22.038789 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:22.004776 32376 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:21.883641 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:21.886404  1451 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:22.065356 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:21.893006 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1735) took 1678 ms. Trace:
I20260709 12:21:21.991887  1330 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:21.915232  1470 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:21:21.807919  1370 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.734s	user 0.011s	sys 0.000s
I20260709 12:21:21.916639  1471 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:21.829264  1513 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:21:21.821275 31678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.002s	sys 0.009s
I20260709 12:21:22.098533 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1667) took 1658 ms. Trace:
I20260709 12:21:21.949833 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1672) took 1246 ms. Trace:
I20260709 12:21:21.950712 31677 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:22.100770 31671 rpcz_store.cc:276] 0709 12:21:20.703618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.305511 (+601893us) service_pool.cc:224] Handling call
0709 12:21:21.305873 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:21.839058 (+533185us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.949518 (+110460us) spinlock_profiling.cc:243] Waited 532 ms on lock 0x7b54002e6b54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:21.949757 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1223879040}
I20260709 12:21:22.102867 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:21.956542 31672 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:21.976666  1446 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.101s
I20260709 12:21:21.958492 31792 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:21.869515  1507 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.991787  1416 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.095s
I20260709 12:21:21.985689  1079 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:21.993378 31718 connection.cc:256] RPC call timeout handler was delayed by -1.4562610950001726s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2202s for a response
I20260709 12:21:21.953137  1353 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:21.957396 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:21.961814  1299 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:22.042219   678 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:21.882207  1269 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:21.957621 31670 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:22.064308 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:22.045092 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:21.953738   931 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:21.889206 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:22.077992 31823 rpcz_store.cc:276] 0709 12:21:20.215084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.443637 (+228553us) service_pool.cc:224] Handling call
0709 12:21:20.480797 (+ 37160us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:20.543343 (+ 62546us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:20.617231 (+ 73888us) log.cc:844] Serialized 14 byte log entry
0709 12:21:20.842772 (+225541us) log.cc:844] Serialized 14 byte log entry
0709 12:21:20.915883 (+ 73111us) spinlock_profiling.cc:243] Waited 20.7 ms on lock 0x7b1800038aa0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:21:21.274566 (+358683us) spinlock_profiling.cc:243] Waited 8.48 ms on lock 0x7b1800038aa0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:21:21.364246 (+ 89680us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:21.364353 (+   107us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:21.364441 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.892718 (+528277us) spinlock_profiling.cc:243] Waited 62.2 ms on lock 0x7b54001969d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:21.892773 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:21.892931 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":666,"prepare.queue_time_us":76292,"prepare.run_cpu_time_us":139,"prepare.run_wall_time_us":2596,"raft.queue_time_us":38376,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":41045,"spinlock_wait_cycles":210099072,"thread_start_us":313803,"threads_started":2,"wal-append.queue_time_us":417457}
I20260709 12:21:22.131732 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:21.907102 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:22.082592 31673 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:22.135174 31673 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:21.993685  1464 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:21.992359  1312 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:21.902294 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:22.146770 31753 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:22.009407  1450 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:22.009356 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:22.105744 31672 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:21:21.764456 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:22.106297 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:22.158066 31827 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:22.099197 31678 rpcz_store.cc:276] 0709 12:21:20.440140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.119008 (+678868us) service_pool.cc:224] Handling call
0709 12:21:21.129884 (+ 10876us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:21.130148 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:21:21.160358 (+ 30210us) log.cc:844] Serialized 14 byte log entry
0709 12:21:22.098017 (+937659us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:22.098093 (+    76us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:22.098163 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:22.098279 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:22.098453 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":27174,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":57,"thread_start_us":903039,"threads_started":1,"wal-append.queue_time_us":648791}
I20260709 12:21:22.107805 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:22.077415  1290 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.167462  1290 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:21.974251 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 228756us
I20260709 12:21:21.880904  1402 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.119246  1382 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:22.090240  1330 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:22.114842 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 156426us
I20260709 12:21:22.009707  1432 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.099s
I20260709 12:21:22.122786 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:22.170358  1290 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:22.109400 31819 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:22.063853  1445 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.096592  1462 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:22.129721 31824 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:22.137055 31673 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:22.098153 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:22.140789 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:22.142160 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 77936.5us
I20260709 12:21:22.132640 31823 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:22.129082 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:22.113890 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:22.119256  1528 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:22.163403 31669 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:22.119028  1381 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:22.163565 31675 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:22.123967  1447 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:22.187526 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 165) took 1743 ms. Trace:
I20260709 12:21:22.199252 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:22.266120 31747 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:22.267112 31825 rpcz_store.cc:276] 0709 12:21:20.444402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.477953 (+ 33551us) service_pool.cc:224] Handling call
0709 12:21:20.573123 (+ 95170us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:21.991549 (+1418426us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:22.187254 (+195705us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54002080d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:22.187458 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2992095488}
W20260709 12:21:22.187717 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 65077.4us
I20260709 12:21:22.200201 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:22.191332 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:22.179136 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:22.268827 31821 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 1.
I20260709 12:21:22.269068 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:22.118564  1299 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:22.201736 31677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1668) took 1734 ms (client timeout 2133 ms). Trace:
I20260709 12:21:22.202122 31676 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:22.198778 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:22.098413  1370 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:22.188231   678 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:22.204006 31827 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:22.191953 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:22.179423 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 71706.8us
W20260709 12:21:22.269835 31677 rpcz_store.cc:269] 0709 12:21:20.467311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.611113 (+143802us) service_pool.cc:224] Handling call
0709 12:21:22.201562 (+1590449us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54001815d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:22.201674 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2586285440}
I20260709 12:21:22.273633 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:22.111200  1400 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:22.171403  1519 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:21:22.270871 31826 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:22.358042   678 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:22.384054   678 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:22.357347 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 83840.6us
I20260709 12:21:22.385751 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:22.194427  1462 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.362200 32378 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:21:22.365265 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 165929us
I20260709 12:21:22.388384 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:22.143760  1194 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.181794 32373 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:22.183573  1330 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:22.179688  1503 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:22.181417  1079 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:22.175770 31670 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:22.191917 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:22.399276 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:22.401147 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:22.171479  1396 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:22.410066 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:22.183265 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 389) took 1263 ms. Trace:
I20260709 12:21:22.200556 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:22.194041  1445 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:22.330844  1383 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.105s
I20260709 12:21:22.129877  1353 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.418372 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:22.421366 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:22.416594 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:22.182197  1402 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:22.284031  1404 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:22.425911 32375 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:22.369801  1299 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:22.474200  1178 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.553s	user 0.007s	sys 0.000s
I20260709 12:21:22.184412  1470 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.216s
I20260709 12:21:22.371003  1303 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.474763  1249 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:22.474982  1303 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:22.475607  1249 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.476125  1249 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.477609  1303 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:22.437292 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:22.366616  1301 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:22.478262  1249 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:22.440325 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:22.440554 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:22.490108 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 49642.2us
W20260709 12:21:22.490660 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.491083 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:22.492317 31791 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:22.183221  1382 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.203612  1312 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:22.367556 32371 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:22.170617  1454 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:22.485482  1467 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.646s	user 0.010s	sys 0.001s
I20260709 12:21:22.458750 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:22.528539 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 69904.4us
I20260709 12:21:22.528082   678 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.319180  1528 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.420008  1503 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:22.467082 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:22.459033  1531 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:22.469753 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:22.363974  1381 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.470584  1404 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:22.281656  1289 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:22.467962 31745 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:22.478586  1301 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:22.160885  1464 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:22.479740  1524 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
I20260709 12:21:22.460284  1079 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:22.480366  1402 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:22.484153  1370 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:22.444304  1500 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:22.237594  1451 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.410861  1194 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.439864  1532 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:22.503118 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:22.522444 31818 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.537s	user 0.010s	sys 0.000s
I20260709 12:21:22.594846 31818 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:22.543641  1528 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:22.191332  1529 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.252s
I20260709 12:21:22.463563 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:22.479641  1445 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:21:22.532332 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.206821  1513 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.544564   678 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:22.547204 31678 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:22.488493 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:22.526768  1383 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:22.530529  1249 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:22.566803 31825 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:22.541960  1178 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:22.576628 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:22.569344  1079 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:22.622853 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 46374.9us
W20260709 12:21:22.555698  1525 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:21:22.543462  1503 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:22.641757 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 512782us
I20260709 12:21:22.594269  1532 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:22.549207  1471 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:22.552584  1404 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:22.606046 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:22.649448 31818 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:22.451202 32372 rpcz_store.cc:276] 0709 12:21:20.919538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:20.927740 (+  8202us) service_pool.cc:224] Handling call
0709 12:21:20.932160 (+  4420us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:21.840731 (+908571us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:22.182925 (+342194us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b54004c4354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:22.183175 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1312631552}
I20260709 12:21:22.566430  1533 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:22.626098 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:22.720188 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 183) took 1061 ms. Trace:
W20260709 12:21:22.608315  1385 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.682s	user 0.011s	sys 0.000s
W20260709 12:21:22.723387  1526 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.237s	user 0.000s	sys 0.001s
I20260709 12:21:22.723275  1385 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:22.185102  1450 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:22.727558  1450 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.597502  1507 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:22.543164  1467 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.737349  1467 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:22.740464  1540 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:22.743839 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:22.745287 31670 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:22.748203  1464 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:22.738201 31672 raft_consensus.cc:606] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Received request to transfer leadership to TS 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:22.750325 31672 raft_consensus.cc:614] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:22.650202 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:22.751703 31821 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:22.720808 31820 rpcz_store.cc:276] 0709 12:21:21.658522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.658706 (+   184us) service_pool.cc:224] Handling call
0709 12:21:21.659122 (+   416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:21.659504 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:21.659844 (+   340us) log.cc:844] Serialized 14 byte log entry
0709 12:21:21.983548 (+323704us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:21.995031 (+ 11483us) log.cc:844] Serialized 26 byte log entry
0709 12:21:21.995213 (+   182us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:21.995273 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:21.995369 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:22.193803 (+198434us) spinlock_profiling.cc:243] Waited 35.8 ms on lock 0x7b4800130158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:21:22.719757 (+525954us) raft_consensus.cc:1685] finished
0709 12:21:22.719908 (+   151us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:22.720097 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":888,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":37,"raft.queue_time_us":12976,"raft.run_cpu_time_us":2400,"raft.run_wall_time_us":201051,"spinlock_wait_cycles":82351360,"thread_start_us":322332,"threads_started":1,"wal-append.queue_time_us":740347}
I20260709 12:21:22.670775 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:22.753145 31750 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:22.557215  1531 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:22.616411  1249 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:22.642566  1516 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.722s	user 0.009s	sys 0.003s
I20260709 12:21:22.623304 31819 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:22.702163 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:22.783315 31818 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:22.617445  1383 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:22.783689 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:22.784705 31820 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:22.643254  1416 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:22.637040  1079 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:22.652562  1532 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:22.646421   931 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:22.749151 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:22.611784  1454 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:22.770588  1249 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:22.787477  1535 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:22.787791 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:22.709255 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:22.726521  1385 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:22.749667  1400 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:22.801887 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:22.834106 31820 raft_consensus.cc:2376] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4. Re-sending same reply.
I20260709 12:21:22.785807  1462 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:22.817205 31821 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:22.788707 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:22.789211  1383 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:22.797329  1529 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:22.789755  1516 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:22.828085 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 379837us
I20260709 12:21:22.840369 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:22.855659 31821 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:22.857065 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:22.858201 31820 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:22.806175 31744 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
W20260709 12:21:22.866526 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:22.822970  1377 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:22.845503  1415 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.015s	sys 0.010s
I20260709 12:21:22.839174  1507 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:22.887195 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 525207us
W20260709 12:21:22.877091 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 384869us
I20260709 12:21:22.861842 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:22.889240 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:22.848951  1529 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:22.915973  1531 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:22.916186  1529 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:22.891484  1537 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:21:22.917742 32374 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:22.897187 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:22.919205  1385 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:22.919397  1507 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.921990  1507 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:22.897482 31821 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:22.820480  1471 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:22.881242  1415 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.016s	sys 0.011s
I20260709 12:21:22.811313  1454 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:22.914913 31745 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:22.919821  1385 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:22.874017  1501 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.974s	user 0.014s	sys 0.000s
I20260709 12:21:22.885236  1382 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:22.934145 31750 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
W20260709 12:21:22.945348  1330 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.506s	user 0.008s	sys 0.004s
W20260709 12:21:22.940343 31747 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.674s	user 0.008s	sys 0.004s
I20260709 12:21:22.977452 31747 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:22.978085 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1713) took 1018 ms. Trace:
I20260709 12:21:22.852769 31818 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:22.980329 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:22.947571 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:22.947688  1548 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:22.949404  1531 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:22.887867  1377 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:22.956405  1528 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:22.901070  1381 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:22.959313  1194 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:22.871649  1353 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:22.964336 31821 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:23.001909 31751 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:22.932394 31820 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:23.003165 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:22.971982 31819 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:22.979241  1501 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:22.897075 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:22.827379  1535 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:22.947103 32372 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:22.981907 31818 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:23.019723 31823 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:22.984001 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:22.961613  1385 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:22.992329 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:23.023044 31822 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:23.004995 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:22.973078 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 302454us
I20260709 12:21:22.984185 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:23.017923 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 120974us
I20260709 12:21:22.978951 31747 rpcz_store.cc:276] 0709 12:21:21.959635 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.959755 (+   120us) service_pool.cc:224] Handling call
0709 12:21:22.199109 (+239354us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:22.978001 (+778892us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37675392}
I20260709 12:21:23.048396 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:23.023665 31819 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:23.050788 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:23.064040 32374 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:22.990581  1249 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.069774  1404 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.070816  1559 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:23.071290  1559 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:23.071633  1559 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:23.072342 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:23.073192 31746 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:22.999096  1548 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:23.074398 31746 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260709 12:21:22.979570  1532 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.081952  1548 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:23.091044  1301 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.096737 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:23.102057 31821 raft_consensus.cc:2368] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5.
W20260709 12:21:23.101805 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 230120us
I20260709 12:21:23.102973 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:23.103296 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:22.983176  1330 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.104374 32369 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:23.105229 31671 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:21:23.106560 31821 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:23.098798 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 114914us
I20260709 12:21:23.102013 31818 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:23.108613 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:23.098699  1501 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:23.103441  1451 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:23.110934 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:23.111331 31818 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:23.116668 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 222) took 1211 ms. Trace:
I20260709 12:21:23.099882 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:23.103792 31822 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:23.111501 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:23.121471 31819 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:23.111074 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:23.109023 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:23.021039  1353 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:23.097276 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:23.114319  1536 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:21:23.119163 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:23.066497  1535 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.136480 31820 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:23.120357 31677 rpcz_store.cc:276] 0709 12:21:21.905727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.367821 (+462094us) service_pool.cc:224] Handling call
0709 12:21:22.368329 (+   508us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:22.368633 (+   304us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:22.793236 (+424603us) log.cc:844] Serialized 14 byte log entry
0709 12:21:23.116281 (+323045us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:23.116355 (+    74us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:23.116410 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.116485 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:23.116608 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":571205,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":59,"raft.queue_time_us":910,"thread_start_us":689155,"threads_started":2}
I20260709 12:21:23.137136 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1628) took 1128 ms. Trace:
I20260709 12:21:23.137501 31820 rpcz_store.cc:276] 0709 12:21:22.009011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.135975 (+1126964us) service_pool.cc:224] Handling call
0709 12:21:23.136303 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.136895 (+   592us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.137079 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.023589  1558 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:23.122318 31821 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:23.139035 31819 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [2.2-3.3]
I20260709 12:21:23.139236 31820 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:23.139814 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1626) took 1147 ms. Trace:
I20260709 12:21:23.117867 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:23.125494 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.126461  1548 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.128466  1560 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:23.120299  1447 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:23.112679 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:23.130263  1530 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:23.131229  1501 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Stepping down as leader of term 2
W20260709 12:21:23.132718  1505 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.544s	user 0.000s	sys 0.005s
I20260709 12:21:23.154610  1432 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:23.134305 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1710) took 1466 ms. Trace:
I20260709 12:21:23.155450 31745 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:23.219558  1432 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:23.101372  1516 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:23.220077  1432 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:23.220221 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:23.139835 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1633) took 1045 ms. Trace:
W20260709 12:21:23.218880 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 48770us
I20260709 12:21:23.222024 31820 rpcz_store.cc:276] 0709 12:21:22.094959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.138235 (+1043276us) service_pool.cc:224] Handling call
0709 12:21:23.138537 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.139655 (+  1118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.139794 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.223613 31820 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:21:23.224236 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1636) took 1109 ms. Trace:
I20260709 12:21:23.224545 31820 rpcz_store.cc:276] 0709 12:21:22.114538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.223055 (+1108517us) service_pool.cc:224] Handling call
0709 12:21:23.223395 (+   340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:23.223983 (+   588us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.224176 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.225375 31820 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:23.140348 31821 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:21:23.124286 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:23.140275 31819 rpcz_store.cc:276] 0709 12:21:21.992502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.138441 (+1145939us) service_pool.cc:224] Handling call
0709 12:21:23.138786 (+   345us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:23.139551 (+   765us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.139763 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.142843 31747 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:23.169904 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1274) took 1273 ms. Trace:
I20260709 12:21:23.226100 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1634) took 1117 ms. Trace:
I20260709 12:21:23.171128 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 187) took 1031 ms. Trace:
I20260709 12:21:23.171655 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 529) took 1174 ms. Trace:
I20260709 12:21:23.226387 31821 rpcz_store.cc:276] 0709 12:21:22.108829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.139702 (+1030873us) service_pool.cc:224] Handling call
0709 12:21:23.140082 (+   380us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:23.225820 (+ 85738us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.226053 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.219620 31748 rpcz_store.cc:276] 0709 12:21:21.668178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.668359 (+   181us) service_pool.cc:224] Handling call
0709 12:21:21.668702 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:22.041732 (+373030us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:22.041776 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:22.041821 (+    45us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:22.041860 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.134030 (+1092170us) spinlock_profiling.cc:243] Waited 81.5 ms on lock 0x7b54002f6cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:23.134081 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:23.134235 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1473,"raft.queue_time_us":136203,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"spinlock_wait_cycles":187447168,"thread_start_us":258450,"threads_started":1}
I20260709 12:21:23.226797 32375 rpcz_store.cc:276] 0709 12:21:21.997618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.997797 (+   179us) service_pool.cc:224] Handling call
0709 12:21:23.171478 (+1173681us) spinlock_profiling.cc:243] Waited 298 ms on lock 0x7b54004272d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:23.171600 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":684477184}
I20260709 12:21:23.226270 31751 rpcz_store.cc:276] 0709 12:21:21.896056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.904657 (+  8601us) service_pool.cc:224] Handling call
0709 12:21:21.905040 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.002408 (+1097368us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.169625 (+167217us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54002134d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:23.169835 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2522031872}
I20260709 12:21:23.194854 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.229949 31753 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:23.223845 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:23.205341 31827 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260709 12:21:23.231593 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1635) took 1119 ms. Trace:
I20260709 12:21:23.232060 31827 rpcz_store.cc:276] 0709 12:21:22.112266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.204765 (+1092499us) service_pool.cc:224] Handling call
0709 12:21:23.205105 (+   340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:23.231313 (+ 26208us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.231533 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.206275 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:23.208985 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:23.190238 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.234501 31818 raft_consensus.cc:1813] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Handling vote request from an unknown peer b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:23.174965 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:23.135329 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:23.142135  1353 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.202970  1377 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:23.200176  1445 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.201927  1301 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:23.138278 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:23.136569  1299 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.665s	user 0.010s	sys 0.002s
I20260709 12:21:23.198041 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1188) took 1480 ms. Trace:
I20260709 12:21:23.235349 31818 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:23.115659  1450 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:23.235811 31823 rpcz_store.cc:276] 0709 12:21:22.139991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.404128 (+264137us) service_pool.cc:224] Handling call
0709 12:21:23.170933 (+766805us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b54000fa854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:23.171068 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1244395392}
I20260709 12:21:23.237674 31819 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:23.237969 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:23.240240 31749 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:23.472210 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:23.240351 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:23.234587 32343 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:23.231490 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:23.474184 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 242790us
W20260709 12:21:23.474414 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 236554us
I20260709 12:21:23.475518 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:23.475592 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:23.477121 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1756) took 1283 ms. Trace:
I20260709 12:21:23.477452 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:23.477649 31819 rpcz_store.cc:276] 0709 12:21:22.193320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.236935 (+1043615us) service_pool.cc:224] Handling call
0709 12:21:23.237336 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.476738 (+239402us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.477036 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:23.330194 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 205964us
I20260709 12:21:23.479241 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:23.480746 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:23.199195  1330 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.247165 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:23.295722 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 184) took 1180 ms. Trace:
I20260709 12:21:23.485059 31820 rpcz_store.cc:276] 0709 12:21:22.114882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.225022 (+1110140us) service_pool.cc:224] Handling call
0709 12:21:23.225243 (+   221us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.225606 (+   363us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.295602 (+ 69996us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.332100 31678 rpcz_store.cc:276] 0709 12:21:21.717459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.159176 (+441717us) service_pool.cc:224] Handling call
0709 12:21:22.167006 (+  7830us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:22.609502 (+442496us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.197749 (+588247us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b54002549d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:23.197960 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":873582848}
W20260709 12:21:23.255980 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:23.487720 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:23.243007 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.395955 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:23.488219 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:23.168987  1529 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:23.403134 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:23.337121 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.160959 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 395) took 1254 ms. Trace:
I20260709 12:21:23.163347  1540 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:23.220932  1273 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.184995  1312 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.214334  1517 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.507850  1312 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:23.507966  1517 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.508361  1312 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.510535  1517 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:23.220366 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.512296 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:23.407868 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1677) took 2486 ms. Trace:
I20260709 12:21:23.513381 31676 rpcz_store.cc:276] 0709 12:21:20.921943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.352215 (+430272us) service_pool.cc:224] Handling call
0709 12:21:21.501318 (+149103us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:22.270675 (+769357us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.407589 (+1136914us) spinlock_profiling.cc:243] Waited 701 ms on lock 0x7b5400254ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:23.407808 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1611210624}
I20260709 12:21:23.514768 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:23.408896 31825 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1749) took 1222 ms (client timeout 1607 ms). Trace:
W20260709 12:21:23.515547 31825 rpcz_store.cc:269] 0709 12:21:22.186362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.384809 (+198447us) service_pool.cc:224] Handling call
0709 12:21:23.408704 (+1023895us) spinlock_profiling.cc:243] Waited 77 ms on lock 0x7b54002080d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:23.408834 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":177030912}
I20260709 12:21:23.515950 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:23.516566 31676 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:23.516286  1385 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.517148 31825 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:21:23.517830 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1763) took 1313 ms. Trace:
I20260709 12:21:23.229103  1516 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.518889  1516 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.518150 31825 rpcz_store.cc:276] 0709 12:21:22.204316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.516510 (+1312194us) service_pool.cc:224] Handling call
0709 12:21:23.516898 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:23.517559 (+   661us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.517773 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.520907  1516 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:23.228402  1289 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.212441  1530 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:23.521571  1289 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.514760 31675 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:23.522178  1530 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.523365  1530 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:23.323686  1443 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:23.523670  1289 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.524061  1443 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:23.524289  1289 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:23.524546  1396 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:23.525190  1396 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:23.525017  1443 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:23.525914 32369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 390) took 1547 ms (client timeout 1530 ms). Trace:
I20260709 12:21:23.180286  1399 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:23.526379 32369 rpcz_store.cc:269] 0709 12:21:21.978903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.984866 (+  5963us) service_pool.cc:224] Handling call
0709 12:21:23.525666 (+1540800us) spinlock_profiling.cc:243] Waited 880 ms on lock 0x7b54005947d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:23.525792 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2023236992}
I20260709 12:21:23.226423  1501 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:23.526695  1396 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.495790  1535 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:23.528563  1535 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:23.528021  1501 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.523670 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.323287  1370 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:23.529939  1370 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:23.330838  1475 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:23.331473  1404 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.221123  1447 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.395329  1560 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.339973 31746 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:23.428684 31670 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:23.159691  1563 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.268s
I20260709 12:21:23.447988 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 189) took 1247 ms. Trace:
I20260709 12:21:23.518461  1385 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.242295  1473 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.221s
I20260709 12:21:23.460598  1353 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:23.443679  1558 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260709 12:21:23.407011 32370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 389) took 1451 ms (client timeout 1697 ms). Trace:
I20260709 12:21:23.338480 32372 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:23.460898  1559 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:23.323932  1451 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.126s
W20260709 12:21:23.462756  1503 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.789s	user 0.006s	sys 0.005s
I20260709 12:21:23.538103  1513 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:23.463570  1383 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.591s	user 0.009s	sys 0.001s
I20260709 12:21:23.339357  1445 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.401922  1548 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:23.241237  1382 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.203s
I20260709 12:21:23.571087  1382 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:23.571492  1382 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:23.572371  1382 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.573657  1382 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:23.395483  1299 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.575271 32378 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:23.400566  1249 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:23.579255  1432 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.335386 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.581357 31751 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:23.483192 31643 leader_election.cc:400] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5.
I20260709 12:21:23.582201 31643 leader_election.cc:403] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:23.489024 31749 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:23.489928 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:23.583837 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 93993.1us
I20260709 12:21:23.584861 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:23.586287 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 103284us
I20260709 12:21:23.588313 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:23.592990 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:23.470310 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1753) took 1277 ms. Trace:
W20260709 12:21:23.452939  1079 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.649s	user 0.011s	sys 0.000s
I20260709 12:21:23.604748  1079 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.607456  1079 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.608785  1269 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:23.495131 31677 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:23.496523 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1617) took 1514 ms. Trace:
I20260709 12:21:23.609697 31824 rpcz_store.cc:276] 0709 12:21:21.982564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.983158 (+   594us) service_pool.cc:224] Handling call
0709 12:21:23.496191 (+1513033us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54003c91d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:23.496456 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":332612864}
I20260709 12:21:23.610150 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.494513 31753 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:23.611519 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 218) took 1711 ms. Trace:
I20260709 12:21:23.497648 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 190) took 1284 ms. Trace:
I20260709 12:21:23.612149 31821 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:23.500900  1381 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:23.496958  1556 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:23.612654  1227 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:23.612857 32374 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:23.613214  1556 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:23.612287 31819 rpcz_store.cc:276] 0709 12:21:22.200902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.481854 (+1280952us) service_pool.cc:224] Handling call
0709 12:21:23.482308 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.482573 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:23.482614 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:23.482719 (+   105us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:23.482770 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.482845 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:23.485634 (+  2789us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.614326 31820 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:21:23.614180  1556 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.614704 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:23.611953 31673 rpcz_store.cc:276] 0709 12:21:21.900204 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.195503 (+295299us) service_pool.cc:224] Handling call
0709 12:21:22.265373 (+ 69870us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:22.265577 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:22.626287 (+360710us) log.cc:844] Serialized 14 byte log entry
0709 12:21:22.962767 (+336480us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:22.963849 (+  1082us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:22.963938 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:22.964037 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:23.611358 (+647321us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b54000474d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:23.611459 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14898,"prepare.queue_time_us":629354,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":55,"raft.queue_time_us":1491,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":13834,"spinlock_wait_cycles":159682816,"thread_start_us":586079,"threads_started":2,"wal-append.queue_time_us":496805}
I20260709 12:21:23.501957 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:23.616829 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40834 (request call id 11) took 1553 ms. Trace:
I20260709 12:21:23.494518  1569 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:23.502455 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.617504  1569 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:23.617969  1569 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:23.618124 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:23.503826  1532 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:23.473327  1528 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.496150 32376 rpcz_store.cc:276] 0709 12:21:21.906366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.906549 (+   183us) service_pool.cc:224] Handling call
0709 12:21:23.160667 (+1254118us) spinlock_profiling.cc:243] Waited 68.8 ms on lock 0x7b54005c30d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:23.160875 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":158278912}
I20260709 12:21:23.533718 31676 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:23.557011  1404 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:23.625348 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.626573 31676 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:23.510456  1312 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.421612  1450 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.541148 31823 rpcz_store.cc:276] 0709 12:21:22.200591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.396606 (+1196015us) service_pool.cc:224] Handling call
0709 12:21:23.447220 (+ 50614us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.447510 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:23.447555 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:23.447608 (+    53us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:23.447654 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.447734 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:23.447898 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:23.458438  1571 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:21:23.493558  1544 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:23.499343  1471 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:23.499763  1446 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:23.529531  1501 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:23.548461 31675 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:23.529564  1535 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.4, Last appended by leader: 4, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.543583  1475 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:23.544498  1454 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.566246  1383 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.567049  1503 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.576102  1445 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:23.594530 31827 rpcz_store.cc:276] 0709 12:21:22.192479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.232875 (+1040396us) service_pool.cc:224] Handling call
0709 12:21:23.469537 (+236662us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.469826 (+   289us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:23.469869 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:23.469929 (+    60us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:23.469975 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.470056 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:23.470226 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.541992  1560 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.596700  1432 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:23.646024  1150 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:23.557351 32370 rpcz_store.cc:269] 0709 12:21:21.956050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.957779 (+  1729us) service_pool.cc:224] Handling call
0709 12:21:23.066552 (+1108773us) spinlock_profiling.cc:243] Waited 5.82 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:23.399552 (+333000us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13393664}
I20260709 12:21:23.591017  1513 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.604126  1078 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:23.611445 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:23.498231  1540 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:23.501237 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.696103 31748 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:23.617189 31752 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:23.506841  1273 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.617249 31672 rpcz_store.cc:276] 0709 12:21:22.063611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.105575 (+ 41964us) service_pool.cc:224] Handling call
0709 12:21:23.616593 (+1511018us) spinlock_profiling.cc:243] Waited 581 ms on lock 0x7b54003917d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdcf853 00007fe901d9d60a 00007fe8fb3237d4 00007fe8fb323762 00007fe8fb323691 00007fe8fb32360c 00007fe8fb3228c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:23.616722 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1335796992}
I20260709 12:21:23.621513 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 243) took 1255 ms. Trace:
I20260709 12:21:23.629673 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:23.634490 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:23.576556  1299 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:23.641047 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 406497us
I20260709 12:21:23.670001  1475 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:23.613286  1227 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:23.644852 31751 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:23.646881 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:23.619818  1353 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:23.619647  1330 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 5
I20260709 12:21:23.662988 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.620198  1531 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:23.612912  1381 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:23.667754 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 496939us
I20260709 12:21:23.722993  1381 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:23.666595 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:23.630823  1573 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.724419  1573 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.724893 31669 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:23.726770  1573 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.566279  1559 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:21:23.645982  1415 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:23.727757  1415 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.729933  1415 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:23.647940  1517 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:23.730612  1517 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:23.731091  1566 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.200s	user 0.000s	sys 0.001s
I20260709 12:21:23.731179  1517 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:23.647652  1370 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.663318  1471 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:23.734957  1471 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:23.736683 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 215) took 2146 ms. Trace:
I20260709 12:21:23.737103 31671 rpcz_store.cc:276] 0709 12:21:21.589992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.122916 (+532924us) service_pool.cc:224] Handling call
0709 12:21:22.123252 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:23.105702 (+982450us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.736383 (+630681us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b54001b2bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:23.736619 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2257673088}
I20260709 12:21:23.632010  1404 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.740509  1404 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:23.697026 31750 rpcz_store.cc:276] 0709 12:21:22.383206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.974669 (+591463us) service_pool.cc:224] Handling call
0709 12:21:22.975043 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:22.975288 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:23.333885 (+358597us) log.cc:844] Serialized 14 byte log entry
0709 12:21:23.621168 (+287283us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:23.621223 (+    55us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:23.621269 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.621333 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:23.621455 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4552,"prepare.queue_time_us":556095,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":4653,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":12877,"thread_start_us":632294,"threads_started":2}
I20260709 12:21:23.741653 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:23.742316 31750 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:23.742846 31678 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:23.743409 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:23.744184  1531 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:23.744624 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.744998  1447 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:23.745754 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:23.700883 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 110299us
I20260709 12:21:23.703877 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:23.748443 31672 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:23.748522 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 82067.3us
I20260709 12:21:23.705153 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:23.749986 31676 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:21:23.750293 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.711795 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1759) took 1517 ms. Trace:
I20260709 12:21:23.711155  1475 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.747743 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:23.769776 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:23.771658 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:23.715896 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.715999 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:23.704684  1445 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:23.744647  1531 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:23.715911  1383 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:23.775745  1451 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:23.720095 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.776922 31670 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:23.777128 32375 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260709 12:21:23.777892 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.778563 31670 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:23.779132 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:23.779515 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:23.779778 31675 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:23.780126 31670 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:23.730540 31748 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
W20260709 12:21:23.734476 32378 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 533) took 1643 ms (client timeout 1985 ms). Trace:
I20260709 12:21:23.783900 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:23.784159 32378 rpcz_store.cc:269] 0709 12:21:22.091452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.460760 (+369308us) service_pool.cc:224] Handling call
0709 12:21:23.734149 (+1273389us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540031d654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:23.734411 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2399566592}
I20260709 12:21:23.784571 32374 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:23.688328 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.691260 31677 raft_consensus.cc:2376] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5. Re-sending same reply.
I20260709 12:21:23.785977 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.786589 31677 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:23.787397 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.788408 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:23.788630 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:23.789040 31677 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:23.789451  1450 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.789742 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:23.750700 31818 rpcz_store.cc:276] 0709 12:21:22.194198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.117323 (+923125us) service_pool.cc:224] Handling call
0709 12:21:23.711559 (+594236us) spinlock_profiling.cc:243] Waited 98.2 ms on lock 0x7b5400447ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:23.711725 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":225866112}
I20260709 12:21:23.690994  1150 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.751232 31750 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:23.738315 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.790413 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:23.749244 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.802742 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:23.783291 32370 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:23.784868  1503 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:23.800432 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.764504 31669 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
W20260709 12:21:23.831846 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.830086 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:23.793462 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.815454 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:23.833588 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.834323 31752 raft_consensus.cc:2368] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 5.
I20260709 12:21:23.790503 31677 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:21:23.819239 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44904.2us
I20260709 12:21:23.821822 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.822695 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:23.823932 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 191) took 1622 ms. Trace:
I20260709 12:21:23.838411 31820 rpcz_store.cc:276] 0709 12:21:22.201229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.486195 (+1284966us) service_pool.cc:224] Handling call
0709 12:21:23.614102 (+127907us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:23.614752 (+   650us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.823730 (+208978us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b540043cbd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:23.823871 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":293154048}
I20260709 12:21:23.818161 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:23.840348 31751 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:23.842098 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:23.843290 31820 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:23.783535  1575 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:21:23.830075 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.796294  1540 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.839334 31750 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:23.851822 31821 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:23.853531 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:23.869781 31824 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:23.849373 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:23.856688 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 54062.1us
I20260709 12:21:23.857223 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:23.871770 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1686) took 1262 ms. Trace:
I20260709 12:21:23.872145 31818 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:23.857688 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.872329 31678 rpcz_store.cc:276] 0709 12:21:22.609329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.870725 (+1261396us) service_pool.cc:224] Handling call
0709 12:21:23.871129 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.871359 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:23.871393 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:23.871452 (+    59us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:23.871501 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.871573 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:23.871699 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:23.873301 32343 leader_election.cc:336] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.738s (SENT)
I20260709 12:21:23.873790 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:23.835858 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:23.831449 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:23.867719 31671 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:23.873270 31672 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 1.
I20260709 12:21:23.805487  1432 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:23.792986  1563 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:23.876116 31672 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
W20260709 12:21:23.883859 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 384355us
I20260709 12:21:23.807500  1454 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:23.876482 31826 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:23.876849 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1685) took 1267 ms. Trace:
I20260709 12:21:23.877033 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:23.791326  1273 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:23.867290 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:23.867543 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 49506.1us
I20260709 12:21:23.890817 31717 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:23.792743  1451 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:23.878063 31751 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:21:23.896500 31751 pending_rounds.cc:85] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 2
I20260709 12:21:23.896844 31751 pending_rounds.cc:107] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:21:23.897173 31751 raft_consensus.cc:2889] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:23.799710 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 388) took 1848 ms. Trace:
I20260709 12:21:23.828809 31675 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:23.796494  1377 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:23.870942 31819 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 1.
I20260709 12:21:23.885260 31671 rpcz_store.cc:276] 0709 12:21:22.609087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.866979 (+1257892us) service_pool.cc:224] Handling call
0709 12:21:23.867531 (+   552us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.876561 (+  9030us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.876792 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.904008  1432 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:23.903375  1377 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:23.903613 31573 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:23.904675 31671 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:23.905356 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1692) took 1045 ms. Trace:
I20260709 12:21:23.905252  1377 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.905826 31671 rpcz_store.cc:276] 0709 12:21:22.860066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.904203 (+1044137us) service_pool.cc:224] Handling call
0709 12:21:23.904506 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.905110 (+   604us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.905303 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.908536 31671 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:23.884327 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1689) took 1274 ms. Trace:
I20260709 12:21:23.867903  1387 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:23.909740  1387 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:23.909574 31672 rpcz_store.cc:276] 0709 12:21:22.610116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.875475 (+1265359us) service_pool.cc:224] Handling call
0709 12:21:23.875784 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.884027 (+  8243us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.884266 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.910297  1387 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:23.887872  1415 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:23.912848  1415 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.913626 31675 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260709 12:21:23.885557 32374 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:23.914526 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1691) took 1061 ms. Trace:
I20260709 12:21:23.795960  1560 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.893502 31820 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:23.921262  1564 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:23.922467  1516 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:23.922955  1516 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:23.898419 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1688) took 1288 ms. Trace:
I20260709 12:21:23.925397 31676 rpcz_store.cc:276] 0709 12:21:22.609847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.875261 (+1265414us) service_pool.cc:224] Handling call
0709 12:21:23.897896 (+ 22635us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.898076 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:23.898102 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:23.898146 (+    44us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:23.898179 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.898237 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:23.898356 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.927102 31676 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:23.856242  1303 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:23.928377  1150 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:23.883118 31827 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:23.790812  1548 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.098s
I20260709 12:21:23.895534  1400 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
I20260709 12:21:23.903352 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:23.929833 32375 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:23.929980  1400 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:23.905050  1432 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:23.885181 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:23.878779  1503 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:21:23.934696 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 49606.3us
I20260709 12:21:23.889256  1454 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:23.809499 31744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 239) took 1536 ms (client timeout 1859 ms). Trace:
I20260709 12:21:23.830680  1470 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:23.790932  1530 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:23.843448  1353 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:23.903352  1273 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260709 12:21:23.938285 32378 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:21:23.861306  1517 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.982817 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:23.983630 32378 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:23.941779 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:23.943583 32375 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:23.807778  1404 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.082s
I20260709 12:21:23.921854 31675 rpcz_store.cc:276] 0709 12:21:22.853613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.912868 (+1059255us) service_pool.cc:224] Handling call
0709 12:21:23.913309 (+   441us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:23.914193 (+   884us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.914458 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:23.956279  1523 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:23.957664  1447 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:23.958817 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 73609.9us
I20260709 12:21:24.031111 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:21:24.031810 32343 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.545s (SENT) [suppressed 17 similar messages]
I20260709 12:21:23.791810  1227 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.033334 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:23.957785 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:23.958914  1383 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:23.962657  1560 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:24.046871 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 90934.5us
I20260709 12:21:23.964349  1559 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:24.047865 32342 leader_election.cc:400] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 5.
I20260709 12:21:24.048178 32342 leader_election.cc:403] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 12:21:23.971845 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:24.049302 31820 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:23.883167  1473 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:23.886159  1450 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:23.977243 31749 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:23.888379  1563 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:23.979954 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:23.977849  1544 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:23.981290 31826 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [3.5-3.5]
I20260709 12:21:23.937081  1402 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:23.936324  1454 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:23.992475 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:23.993472 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:23.987975 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:24.087513 31675 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:24.028196 31818 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:23.946595 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:24.035141 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 77431.3us
I20260709 12:21:24.027316 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:24.034536 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:23.974289  1470 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.048094 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 212) took 2516 ms. Trace:
I20260709 12:21:24.046216  1446 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:23.968631 31827 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:24.095791 31642 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:24.096345 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:24.097004 31670 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:24.097811 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:24.053211 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:24.051949 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:24.055687 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:23.977905  1353 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:24.111004 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:24.111228 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:24.068635 31824 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:24.079921  1569 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:23.991248 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:24.046500  1445 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:24.047319  1312 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:24.086133 31821 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:24.099057 31671 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 4 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:24.099697 31672 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:21:24.054289 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:24.099433  1415 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:24.159395 31819 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
W20260709 12:21:24.104091 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 179595us
I20260709 12:21:24.105108  1451 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:23.946425 31744 rpcz_store.cc:269] 0709 12:21:22.273069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.281685 (+  8616us) service_pool.cc:224] Handling call
0709 12:21:23.809261 (+1527576us) spinlock_profiling.cc:243] Waited 330 ms on lock 0x7b54004051d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:23.809415 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":757998848}
I20260709 12:21:24.116720 32376 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:24.068243  1383 consensus_queue.cc:1243] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 7.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:21:23.982851  1530 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.108533 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:24.189814 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 81354us
I20260709 12:21:24.122090  1579 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 5.
W20260709 12:21:24.109433 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 77651.5us
I20260709 12:21:24.113300 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:24.113783 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:24.114333 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 79882.6us
I20260709 12:21:23.899827 32373 rpcz_store.cc:276] 0709 12:21:21.951054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:21.952733 (+  1679us) service_pool.cc:224] Handling call
0709 12:21:22.064029 (+111296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:22.409397 (+345368us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.799428 (+1390031us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54006dd654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:23.799633 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":234430720}
I20260709 12:21:24.094547 31674 rpcz_store.cc:276] 0709 12:21:21.533944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.114472 (+580528us) service_pool.cc:224] Handling call
0709 12:21:22.114976 (+   504us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:23.165340 (+1050364us) spinlock_profiling.cc:243] Waited 36.7 ms on lock 0x7b18000ebe40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcdcdbfb 00007fe8fcdd864b
0709 12:21:23.165495 (+   155us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:23.165520 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:23.166071 (+   551us) log.cc:844] Serialized 26 byte log entry
0709 12:21:23.535920 (+369849us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:23.535989 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.536091 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:24.033403 (+497312us) raft_consensus.cc:1685] finished
0709 12:21:24.033431 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:24.048007 (+ 14576us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":120501,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":63792,"spinlock_wait_cycles":84333312,"thread_start_us":590669,"threads_started":2,"wal-append.queue_time_us":516721}
I20260709 12:21:24.181951  1078 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.203506 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:24.085608  1402 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:24.091427  1450 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:24.120508 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 72810.7us
I20260709 12:21:24.097574  1330 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5.
I20260709 12:21:24.156731 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:24.122304 31752 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:24.098044  1535 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:24.098722  1470 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:24.043749  1523 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.122385 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:24.056952  1560 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.071939  1559 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.158061  1312 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:24.161768  1531 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.163316  1471 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.165278 31671 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
W20260709 12:21:24.103094 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 167361us
I20260709 12:21:24.101662  1377 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:24.115811  1353 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:24.175678 31751 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:21:24.178401 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:24.179908 31820 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:24.113494  1399 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:24.155439 32370 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:24.191222 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:24.109378 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:24.260262  1404 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:24.120210  1446 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:24.122570  1445 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:24.323071  1445 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:24.323177  1446 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.325625 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:24.211493 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:24.326988 31674 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:24.329210 31819 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:24.329650 31819 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:24.181851  1586 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:24.330775  1377 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:24.331383  1377 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:24.118729  1301 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:24.158795  1443 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.041s
I20260709 12:21:24.332285  1377 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:24.228107 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 114405us
I20260709 12:21:24.334301 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:24.278330 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:24.294242 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59790.6us
W20260709 12:21:24.335402  1591 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 12:21:24.335052  1312 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:24.336835  1583 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.242s	user 0.000s	sys 0.001s
I20260709 12:21:24.337144  1273 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:21:24.338402 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:24.338730  1596 consensus_queue.cc:1175] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b95aeeaf4c51442b8092cff1de674f2b" responder_term: 6 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: [3.6-5.7]" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:21:24.340723  1592 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:21:24.225821 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:24.341737  1443 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:24.342175  1443 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:24.342597  1443 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:24.343473  1569 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:24.345152 31675 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:24.348193 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 48684.2us
I20260709 12:21:24.358065 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:24.359326 31674 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:24.360450 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:24.364861 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:24.367823 31674 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:24.371330 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:24.202826  1447 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:21:24.299379 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:24.227653  1402 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:24.308470 31826 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
W20260709 12:21:24.315205  1382 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.741s	user 0.006s	sys 0.006s
I20260709 12:21:24.120460  1079 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:24.188879  1594 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.113s
I20260709 12:21:24.311265 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:24.229704  1513 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:24.314194 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 123091us
I20260709 12:21:24.391646 31818 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:24.392010 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:24.393302 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:24.156050  1563 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.318562 31673 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:24.321557 31751 raft_consensus.cc:606] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:24.397293 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:24.397744 31751 raft_consensus.cc:614] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:21:24.342617  1578 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.282s	user 0.000s	sys 0.001s
I20260709 12:21:24.298383 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:24.308265  1529 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.811s	user 0.012s	sys 0.000s
I20260709 12:21:24.296402 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:24.260358  1532 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.055s
W20260709 12:21:24.404536 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 108221us
I20260709 12:21:24.344669  1569 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.056s
W20260709 12:21:24.412560 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 48541.8us
I20260709 12:21:24.412914 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:24.413910 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:24.225284  1078 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.384497 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:24.384681  1447 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:24.400195  1513 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:24.389047 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:24.338745  1559 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:24.308323  1399 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:24.389744 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:24.390952 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:24.314838  1404 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:24.354887  1330 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:24.398463 31747 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:21:24.202018  1544 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.377466  1377 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:24.260421  1582 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.117s
I20260709 12:21:24.337183  1387 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.414867 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:24.379066  1178 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260709 12:21:24.387039  1535 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:21:24.380013 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 154257us
I20260709 12:21:24.417876 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:24.396983 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:24.422644 31826 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:24.403406  1556 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:24.393754  1194 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.021s
I20260709 12:21:24.407245  1587 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:24.410848  1396 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:24.419349 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:24.425575 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:24.373065  1451 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.475699 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:24.426966  1078 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:24.427740 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:24.409511 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:24.482187 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:24.432278  1399 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.419827  1442 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:24.428936  1404 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:24.431015  1330 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.433436  1595 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260709 12:21:24.400041  1400 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.024s
I20260709 12:21:24.419620  1576 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:24.422567  1532 consensus_queue.cc:1243] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 3.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:21:24.465701   931 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:24.471504  1523 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:24.478535 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 63755.7us
W20260709 12:21:24.479772 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 62028.5us
W20260709 12:21:24.454218 31745 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.669s	user 0.012s	sys 0.000s
I20260709 12:21:24.486171  1597 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.481544 32376 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:24.432183  1447 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:24.483136 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 57643us
I20260709 12:21:24.483494 31826 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:24.410145  1450 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.487639 31671 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:21:24.488330  1442 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:21:24.488622 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 98996.8us
I20260709 12:21:24.467041  1150 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:24.481190 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:24.537273 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56164.8us
I20260709 12:21:24.508086 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:24.487320  1396 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:24.488543  1604 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:24.494319 31746 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:24.430480  1559 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:21:24.498311  1330 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:24.517176 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1622) took 2531 ms. Trace:
I20260709 12:21:24.531337 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:24.497632  1079 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:24.498493  1530 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:24.538332 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:24.514499  1523 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:21:24.547245  1415 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:24.581136 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:24.513991  1471 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:24.509014 31745 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:24.533236  1442 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:24.545198  1604 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.544416 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:24.537148  1194 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.057s
W20260709 12:21:24.589689 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 45340us
I20260709 12:21:24.590842 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:24.591444 31823 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:24.543264  1532 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:24.507743  1399 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:24.579011 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:24.579398 31826 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:24.581832 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:24.582476 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:24.594193  1396 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.594799  1613 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:24.595036  1445 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.596460  1471 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:24.596657 31670 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:24.587538  1446 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:24.597018  1454 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.598337  1475 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:24.599257  1454 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:24.592015  1604 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:24.603071  1227 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:24.603564  1227 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:24.603869 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:24.605119 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:24.605955  1607 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.175s	user 0.001s	sys 0.000s
I20260709 12:21:24.606824  1582 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:24.607260  1582 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:24.607620  1582 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:24.606300 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:24.608768 31676 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:21:24.610440 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:24.584805  1312 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:24.551680 31822 rpcz_store.cc:276] 0709 12:21:21.985648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.141471 (+1155823us) service_pool.cc:224] Handling call
0709 12:21:23.153155 (+ 11684us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:23.211655 (+ 58500us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:23.211707 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:23.239619 (+ 27912us) log.cc:844] Serialized 26 byte log entry
0709 12:21:23.668363 (+428744us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:23.680032 (+ 11669us) log.cc:844] Serialized 14 byte log entry
0709 12:21:23.680294 (+   262us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:23.680372 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:24.516961 (+836589us) raft_consensus.cc:1685] finished
0709 12:21:24.516983 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:24.517116 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":11442,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":68,"raft.queue_time_us":68604,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":7106,"thread_start_us":225162,"threads_started":1,"wal-append.queue_time_us":549554}
I20260709 12:21:24.517313  1402 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.537648  1597 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.573799  1560 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:24.615996 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:24.617038 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:24.596951  1471 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:24.617893   931 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:24.587608  1602 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.033s
I20260709 12:21:24.620236  1581 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:24.622100  1402 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:24.622906  1559 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:24.624214  1614 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:24.617921  1446 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:24.615547  1301 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:24.618021 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:24.635784 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:24.617394 31792 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:24.606819  1606 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:24.666909 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:24.640188  1471 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:24.641888 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:24.643446 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:24.686084 32373 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:24.644306 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:24.687860  1381 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.964s	user 0.009s	sys 0.002s
I20260709 12:21:24.688355  1381 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.691275  1415 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:24.692924  1353 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.694306 31674 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:24.640743  1079 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.645826  1443 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:24.697446 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 55645.2us
I20260709 12:21:24.644790  1622 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:24.647957  1604 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.648559  1387 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:24.654372 31669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 225) took 2046 ms (client timeout 2444 ms). Trace:
W20260709 12:21:24.715551 31669 rpcz_store.cc:269] 0709 12:21:22.607964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:22.608151 (+   187us) service_pool.cc:224] Handling call
0709 12:21:24.654139 (+2045988us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54002e8454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:24.654298 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2315624192}
I20260709 12:21:24.655261 31678 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
W20260709 12:21:24.668042 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 50761.1us
I20260709 12:21:24.641280 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:24.684724  1102 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.051s
I20260709 12:21:24.719089  1445 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:24.719475 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 78289.3us
I20260709 12:21:24.720227 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:24.697763  1559 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:24.721586  1559 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:24.721827  1581 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:24.696041  1312 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:24.669695  1605 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:21:24.604446  1227 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.670459  1454 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:24.671316  1556 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:24.675036 31744 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:21:24.677594  1586 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:24.678066  1532 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:24.681030  1470 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.683897 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:24.640134  1560 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.699229 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:24.699589 32378 raft_consensus.cc:2410] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:24.699926  1576 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:24.702226 31826 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:24.703392  1443 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:24.705613  1544 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:24.710078 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:24.705446  1381 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:24.712062 31819 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.713764  1513 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:24.668772 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:24.740418 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:24.728657  1383 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:24.742789 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:24.751217 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:24.766521 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:21:24.803074 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:24.830001 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:24.872433 31819 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:24.906193 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 196191us
W20260709 12:21:24.907191 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 140817us
I20260709 12:21:24.728699  1312 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:24.909691  1544 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:24.909963 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:24.910212  1544 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:24.910669 31669 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:24.910487  1576 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.873195  1523 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.082s
I20260709 12:21:24.912345  1617 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:24.912899  1617 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:24.913787  1617 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.830713  1556 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:24.875010 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:24.908730  1560 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.920394 31753 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:24.935813 31823 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:24.937319 31823 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:21:24.935822 31671 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: [5.2-5.2]
I20260709 12:21:24.940250 31671 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
W20260709 12:21:24.942552 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 191421us
I20260709 12:21:24.943331 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:24.944007 31669 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
W20260709 12:21:24.944840 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 202189us
I20260709 12:21:24.948724 31671 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:24.908432  1227 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:24.958176 31819 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:21:24.961546 31820 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:24.903888  1150 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.908066  1383 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:24.934044  1443 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:21:24.971077 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 246728us
I20260709 12:21:24.921130  1614 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:24.916757 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:24.942071  1532 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:24.981165 31671 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:21:24.950784  1586 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:24.915746  1556 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.948165  1353 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:24.972141  1381 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:21:24.975682  1383 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:24.973785  1443 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:24.977854  1614 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:24.966995  1273 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260709 12:21:24.991377  1627 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:24.916428  1559 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:24.992918  1586 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:24.920528  1446 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:25.018748 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 68987.5us
I20260709 12:21:25.000272  1532 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.022068 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:25.021771  1532 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:25.030149 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 83197us
I20260709 12:21:25.031183 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:24.975886  1150 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:24.999958  1381 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:24.985162  1399 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:21:25.000586 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:25.071471  1597 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:25.020674  1586 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:25.319720 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 194) took 1288 ms. Trace:
I20260709 12:21:25.320199 31818 rpcz_store.cc:276] 0709 12:21:24.031712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.901410 (+869698us) service_pool.cc:224] Handling call
0709 12:21:24.905941 (+  4531us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:24.906133 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:24.967669 (+ 61536us) log.cc:844] Serialized 14 byte log entry
0709 12:21:25.319402 (+351733us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:25.319456 (+    54us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:25.319501 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.319564 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:25.319666 (+   102us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":200841,"prepare.queue_time_us":81878,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"thread_start_us":202992,"threads_started":2}
I20260709 12:21:25.322382 31818 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
W20260709 12:21:25.236454 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 205356us
I20260709 12:21:25.335614 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:25.072084 31677 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:21:25.053478  1381 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:25.010177  1622 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:25.009004  1614 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:25.074505  1443 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:24.985921  1544 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.215401 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1218) took 1004 ms. Trace:
I20260709 12:21:25.308128 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 405) took 1355 ms. Trace:
I20260709 12:21:25.293303 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1704) took 1531 ms. Trace:
I20260709 12:21:25.392596 31674 rpcz_store.cc:276] 0709 12:21:24.210992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.371049 (+160057us) service_pool.cc:224] Handling call
0709 12:21:25.215151 (+844102us) spinlock_profiling.cc:243] Waited 320 ms on lock 0x7b54000474d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:25.215325 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":735908096}
I20260709 12:21:25.392946 31675 rpcz_store.cc:276] 0709 12:21:23.761563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.088130 (+326567us) service_pool.cc:224] Handling call
0709 12:21:24.088428 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:24.345717 (+257289us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.292908 (+947191us) spinlock_profiling.cc:243] Waited 256 ms on lock 0x7b5400253ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:25.293200 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":589880704}
I20260709 12:21:25.296561 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35284 (request call id 13) took 1121 ms. Trace:
I20260709 12:21:25.394196 31751 rpcz_store.cc:276] 0709 12:21:24.175272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.175483 (+   211us) service_pool.cc:224] Handling call
0709 12:21:25.296227 (+1120744us) spinlock_profiling.cc:243] Waited 18.2 ms on lock 0x7b54000b8bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdcf853 00007fe901d9d60a 00007fe8fb3237d4 00007fe8fb323762 00007fe8fb323691 00007fe8fb32360c 00007fe8fb3228c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:25.296416 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41800960}
I20260709 12:21:25.309795 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:25.020041  1627 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.344030 31671 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: []
W20260709 12:21:25.283130  1528 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
W20260709 12:21:25.218777  1624 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
W20260709 12:21:25.261749  1585 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.068s	user 0.032s	sys 0.036s
W20260709 12:21:25.297552  1606 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.615s	user 0.010s	sys 0.000s
W20260709 12:21:25.278167  1330 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.702s	user 0.005s	sys 0.005s
I20260709 12:21:25.290462  1514 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:25.396205 31820 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:25.344972  1532 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:25.395815 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 77129.6us
I20260709 12:21:25.392657 32378 rpcz_store.cc:276] 0709 12:21:23.952644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.989964 (+ 37320us) service_pool.cc:224] Handling call
0709 12:21:24.475500 (+485536us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:25.307741 (+832241us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b5400426dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:25.307994 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":515592320}
W20260709 12:21:25.150143  1626 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 12:21:25.208182  1150 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:25.130887  1604 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:25.306517  1594 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:25.606284  1594 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:25.607611  1594 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.609025  1594 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:25.286602 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1725) took 1637 ms. Trace:
I20260709 12:21:25.610114 31746 rpcz_store.cc:276] 0709 12:21:23.649568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.948193 (+298625us) service_pool.cc:224] Handling call
0709 12:21:24.040202 (+ 92009us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:24.552563 (+512361us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.286137 (+733574us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b54004051d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:25.286503 (+   366us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1002504064}
I20260709 12:21:25.306048  1603 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:25.611630  1603 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:25.248533  1555 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.612583  1555 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:25.214380 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:25.347785  1611 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.185s
I20260709 12:21:25.615072 32370 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:25.615689 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 544) took 1268 ms. Trace:
I20260709 12:21:25.616034 32370 rpcz_store.cc:276] 0709 12:21:24.347196 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.373057 (+ 25861us) service_pool.cc:224] Handling call
0709 12:21:25.214164 (+841107us) spinlock_profiling.cc:243] Waited 6.43 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:25.615616 (+401452us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14783232}
I20260709 12:21:25.287015  1560 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:25.372984  1470 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:25.488883  1628 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:25.510251 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1792) took 1146 ms. Trace:
I20260709 12:21:25.617985 31820 rpcz_store.cc:276] 0709 12:21:24.363712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.395623 (+1031911us) service_pool.cc:224] Handling call
0709 12:21:25.395889 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:25.509947 (+114058us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.510181 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:25.289453  1312 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:25.511621 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.403575  1273 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:25.500185  1079 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:25.516108  1606 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:25.509208  1514 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:25.620092  1475 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.021s	user 0.008s	sys 0.004s
I20260709 12:21:25.656036  1540 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:25.658239 31824 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:25.654497  1634 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.024s
I20260709 12:21:25.654769  1514 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.664710  1514 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:25.668186 31822 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:25.678818  1632 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.679117  1634 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:25.679589 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1717) took 2044 ms. Trace:
I20260709 12:21:25.680012 31752 rpcz_store.cc:276] 0709 12:21:23.634744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.834876 (+200132us) service_pool.cc:224] Handling call
0709 12:21:23.835302 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:24.294677 (+459375us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.679330 (+1384653us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b54000b81d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:25.679526 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":645627648}
I20260709 12:21:25.498430  1399 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.142s
I20260709 12:21:25.632352  1273 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:25.516338  1330 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.660425  1353 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.711366  1599 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:25.712592  1637 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:25.667073  1581 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.681528  1560 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.658970  1532 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:25.688532  1383 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.678s	user 0.006s	sys 0.006s
I20260709 12:21:25.680658  1604 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.691812  1079 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:25.685948  1555 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:25.401153  1622 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:25.733021 31672 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:25.679490  1632 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.760965  1627 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:25.767195  1650 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:25.683113  1576 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260709 12:21:25.744225  1640 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260709 12:21:25.772926 31677 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260709 12:21:25.763988  1582 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.155s	user 0.015s	sys 0.001s
I20260709 12:21:25.738372  1599 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:25.728572  1353 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:25.767968  1650 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:25.795605 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1794) took 1183 ms. Trace:
I20260709 12:21:25.796200 31822 rpcz_store.cc:276] 0709 12:21:24.612262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.616791 (+  4529us) service_pool.cc:224] Handling call
0709 12:21:25.795394 (+1178603us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54000fad54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:25.795534 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2373640960}
I20260709 12:21:25.797477 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:25.798065 31822 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:25.742257  1383 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.747654  1532 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.742522  1560 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.747366  1604 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:25.679912  1634 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.736083  1581 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.764374  1586 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:25.787285  1582 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.831640  1582 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:25.835067  1636 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:21:25.769289  1273 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:25.838135  1642 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
W20260709 12:21:25.838524  1404 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.339s	user 0.011s	sys 0.002s
W20260709 12:21:25.826264  1471 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.119s	user 0.005s	sys 0.007s
I20260709 12:21:25.839865 31674 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:25.840942  1618 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:21:25.841609  1415 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.150s	user 0.011s	sys 0.002s
W20260709 12:21:25.843657  1442 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.235s	user 0.010s	sys 0.002s
I20260709 12:21:25.854270  1532 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:25.853883 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 254) took 1675 ms. Trace:
I20260709 12:21:25.887379 31749 rpcz_store.cc:276] 0709 12:21:24.178727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.304109 (+125382us) service_pool.cc:224] Handling call
0709 12:21:24.347832 (+ 43723us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:24.348035 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:24.541027 (+192992us) log.cc:844] Serialized 14 byte log entry
0709 12:21:25.845421 (+1304394us) spinlock_profiling.cc:243] Waited 1.21 ms on lock 0x7b1800418ba0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:21:25.853374 (+  7953us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:25.853458 (+    84us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:25.853525 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.853627 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:25.853802 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13012,"prepare.queue_time_us":241289,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":1275468,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":111,"spinlock_wait_cycles":10985472,"thread_start_us":184768,"threads_started":2,"wal-append.queue_time_us":104191}
W20260709 12:21:25.855731 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 1.27326e+06us
I20260709 12:21:25.889654 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:25.895682 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:25.854957  1643 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:25.836385  1312 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:25.900036 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:25.900959  1645 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.896785 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:25.897322  1602 consensus_queue.cc:1243] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 3.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:21:25.898320 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:25.899161 31748 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:25.904536 31752 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:25.893079  1618 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:25.868927  1404 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.888731  1415 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:25.820785  1079 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.921777 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:25.922755 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:25.907426  1573 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.913841 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:25.883512  1400 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:21:25.873917 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 465724us
I20260709 12:21:25.865419  1471 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:25.930207 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1764) took 3722 ms. Trace:
I20260709 12:21:25.938043 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:25.905936  1330 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:25.931864 31671 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:25.830363  1599 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:25.937242 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:25.941680 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52120.9us
I20260709 12:21:25.893784  1273 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:25.942826 31752 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:25.942842 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:25.938238 31825 rpcz_store.cc:276] 0709 12:21:22.204636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.520311 (+1315675us) service_pool.cc:224] Handling call
0709 12:21:23.520705 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:23.544305 (+ 23600us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:21:23.562928 (+ 18623us) log.cc:844] Serialized 14 byte log entry
0709 12:21:24.087062 (+524134us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:24.087118 (+    56us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:24.087162 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.926596 (+1839434us) spinlock_profiling.cc:243] Waited 19.2 ms on lock 0x7b54001982d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:25.926678 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:25.926848 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38296,"prepare.queue_time_us":21152,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":41,"raft.queue_time_us":53625,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":20543,"spinlock_wait_cycles":44253056,"thread_start_us":484989,"threads_started":1,"wal-append.queue_time_us":539662,"wal-append.run_cpu_time_us":1556,"wal-append.run_wall_time_us":1011300}
I20260709 12:21:25.950212 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:25.950652 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1705) took 2188 ms. Trace:
I20260709 12:21:25.950923 31752 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:25.951665 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:25.952643 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:25.952813 31821 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 197) took 1902 ms (client timeout 2462 ms). Trace:
W20260709 12:21:25.953265 31821 rpcz_store.cc:269] 0709 12:21:24.050722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.122119 (+ 71397us) service_pool.cc:224] Handling call
0709 12:21:25.952629 (+1830510us) spinlock_profiling.cc:243] Waited 928 ms on lock 0x7b540043e9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:25.952755 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2133625600}
I20260709 12:21:25.891319  1442 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:25.957398 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:25.951073 31670 rpcz_store.cc:276] 0709 12:21:23.761833 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.100719 (+338886us) service_pool.cc:224] Handling call
0709 12:21:24.101093 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:24.597169 (+496076us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.950395 (+1353226us) spinlock_profiling.cc:243] Waited 495 ms on lock 0x7b540022a954. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:25.950567 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1139204352}
I20260709 12:21:25.958374 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:25.959829 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:25.956955 31821 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:21:25.981554 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:25.981989 31821 pending_rounds.cc:85] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 5
I20260709 12:21:25.982249 31821 pending_rounds.cc:107] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:21:25.982542 31821 raft_consensus.cc:2889] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:25.983073 31670 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:25.983317 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 202) took 1270 ms. Trace:
I20260709 12:21:25.983696 31821 rpcz_store.cc:276] 0709 12:21:24.713306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.954147 (+1240841us) service_pool.cc:224] Handling call
0709 12:21:25.956478 (+  2331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:25.983076 (+ 26598us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.983262 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:25.984685 31670 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:25.986793 31717 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:25.925482  1643 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.945538  1194 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:25.985147 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 203) took 1271 ms. Trace:
I20260709 12:21:25.990052 31821 rpcz_store.cc:276] 0709 12:21:24.713575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.984429 (+1270854us) service_pool.cc:224] Handling call
0709 12:21:25.984664 (+   235us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:25.984853 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:25.984877 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:25.984918 (+    41us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:25.984954 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.985003 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:25.985099 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:25.937331  1650 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:25.989306  1194 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:25.989403  1269 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.003s
I20260709 12:21:25.992223 31821 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260709 12:21:25.996006  1381 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.597s	user 0.013s	sys 0.000s
I20260709 12:21:25.996589 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:26.006880 31677 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:26.006546  1381 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:25.980450  1330 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:25.947077  1622 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:25.979576 31745 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:21:25.998713  1194 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:26.002218  1400 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:26.002406  1301 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:26.002480  1471 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:25.989806  1529 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:25.979686  1555 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 12:21:26.005776  1377 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:25.956892  1613 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.009433 31820 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:26.000152 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:26.016831  1599 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:26.012327 31752 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:26.001183  1545 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260709 12:21:26.040737 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:25.937944  1573 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:25.988080  1560 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:25.982882  1353 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:26.017249 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1786) took 1665 ms. Trace:
I20260709 12:21:26.045399 31827 rpcz_store.cc:276] 0709 12:21:24.352130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.937101 (+584971us) service_pool.cc:224] Handling call
0709 12:21:24.937372 (+   271us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:24.948269 (+ 10897us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:21:25.006604 (+ 58335us) log.cc:844] Serialized 14 byte log entry
0709 12:21:25.309660 (+303056us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:25.399195 (+ 89535us) log.cc:844] Serialized 26 byte log entry
0709 12:21:25.399332 (+   137us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:25.399365 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.399429 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:26.017024 (+617595us) raft_consensus.cc:1685] finished
0709 12:21:26.017049 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:26.017183 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":120128,"prepare.queue_time_us":39936,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":27,"raft.queue_time_us":360654,"raft.run_cpu_time_us":347,"raft.run_wall_time_us":111919,"thread_start_us":322312,"threads_started":3,"wal-append.queue_time_us":638947}
I20260709 12:21:26.047420 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:25.995110  1383 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:25.994741 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:26.019826 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:26.037868  1614 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.541s	user 0.013s	sys 0.000s
I20260709 12:21:26.021806  1078 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260709 12:21:26.022394  1594 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:26.049032  1613 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.027002 31677 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:21:25.999210  1632 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:26.009416  1604 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:26.059603  1450 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:26.025379  1451 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.005s
I20260709 12:21:26.029251  1400 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:26.031597  1442 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:26.032648  1381 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:26.001422 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 204) took 1287 ms. Trace:
I20260709 12:21:26.072145 31821 rpcz_store.cc:276] 0709 12:21:24.713779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.991600 (+1277821us) service_pool.cc:224] Handling call
0709 12:21:25.991908 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:26.001134 (+  9226us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.001362 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:26.012774 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:26.024957 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:25.926249  1645 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:26.043018  1634 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:26.039047  1609 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260709 12:21:26.079965 32369 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 6 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260709 12:21:26.079660  1400 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:26.049099  1569 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:26.028159  1586 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:26.020674  1631 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.182s	user 0.001s	sys 0.001s
I20260709 12:21:26.056218 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:26.026096 31670 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:26.016255  1330 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.004945  1602 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:26.039283  1529 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:26.027527  1269 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 12:21:26.048506  1540 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:26.062889 32370 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:26.050282 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:26.033778  1603 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.064200  1450 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.049199  1404 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:26.068470  1517 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:26.111397  1540 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20260709 12:21:26.036669  1618 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.094033  1602 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:26.078404 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:26.062114  1614 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.089169 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:26.125026 31821 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:26.094672 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.043865  1643 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:26.098985  1269 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:26.137274  1594 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.100986 31750 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:26.137960  1594 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:26.109117  1603 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:26.140285  1594 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:26.076483  1635 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.186s	user 0.001s	sys 0.001s
I20260709 12:21:26.143891  1594 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.125154 31827 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:26.144726 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 198) took 2060 ms. Trace:
I20260709 12:21:26.100863 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:21:26.145346 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:26.145303 31826 rpcz_store.cc:276] 0709 12:21:24.084410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.904570 (+820160us) service_pool.cc:224] Handling call
0709 12:21:24.904885 (+   315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:24.915373 (+ 10488us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:24.915411 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:24.915900 (+   489us) log.cc:844] Serialized 26 byte log entry
0709 12:21:25.028334 (+112434us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:25.029172 (+   838us) log.cc:844] Serialized 14 byte log entry
0709 12:21:25.315513 (+286341us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:25.315601 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:26.144452 (+828851us) raft_consensus.cc:1685] finished
0709 12:21:26.144485 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:26.144648 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10646,"prepare.queue_time_us":235,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":337499,"raft.run_cpu_time_us":336,"raft.run_wall_time_us":33462,"thread_start_us":397938,"threads_started":2,"wal-append.queue_time_us":906563}
I20260709 12:21:26.147243 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:26.148910 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 205) took 1434 ms. Trace:
I20260709 12:21:26.149171 31827 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:26.049661  1595 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
I20260709 12:21:26.149459 31826 rpcz_store.cc:276] 0709 12:21:24.713985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.147795 (+1433810us) service_pool.cc:224] Handling call
0709 12:21:26.148206 (+   411us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:26.148461 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:26.148499 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:26.148561 (+    62us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:26.148610 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.148682 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:26.148838 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:26.150866  1594 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:26.123379  1618 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:26.152485 31827 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260709 12:21:26.153276  1594 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:26.129104  1602 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:26.162938 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:26.167294  1594 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:26.168613 31821 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:26.176738  1602 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.171747 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:26.179796 31826 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260709 12:21:26.183693 31826 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:26.129019  1540 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:26.187979  1540 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.189507  1450 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:26.192955  1450 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.193918  1330 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:26.190104  1540 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:26.193989  1450 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.196749  1450 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:26.198424  1301 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:26.111400  1529 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:26.204314 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:26.185173  1634 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.207526  1634 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:26.210016 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:26.216490 31573 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 2 to 3, leader changed from <none> to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 3 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:26.219933 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:26.220822 31751 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:21:26.220922 31826 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
W20260709 12:21:26.221400 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.206229 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:26.226961 31748 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:26.186245  1587 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.186270  1595 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:26.225464 31746 raft_consensus.cc:2410] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:21:26.186309  1594 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:26.243896  1645 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:26.244053  1602 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:26.244767  1194 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:26.248528  1516 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:26.250703 32376 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 548) took 1607 ms (client timeout 1985 ms). Trace:
I20260709 12:21:26.252112  1432 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:26.217090  1613 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:26.237437  1450 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.226356 31573 catalog_manager.cc:5697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 3 to 4, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 4 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:26.252769  1614 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:26.260820 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1793) took 1649 ms. Trace:
W20260709 12:21:26.263744 32376 rpcz_store.cc:269] 0709 12:21:24.643355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.643977 (+   622us) service_pool.cc:224] Handling call
0709 12:21:26.250301 (+1606324us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b540031d654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:26.250585 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":687000832}
I20260709 12:21:26.267320 31822 rpcz_store.cc:276] 0709 12:21:24.611832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.802424 (+1190592us) service_pool.cc:224] Handling call
0709 12:21:25.802792 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:25.990501 (+187709us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:25.990533 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:25.990577 (+    44us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:25.990607 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.260530 (+269923us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54002675d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:26.260584 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:26.260753 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":431240448}
W20260709 12:21:26.266454 32344 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.768s	user 0.024s	sys 0.039s
W20260709 12:21:26.269665 32344 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.024s	sys 0.039s
W20260709 12:21:26.269979 31673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1206) took 2937 ms (client timeout 2215 ms). Trace:
W20260709 12:21:26.270947 31673 rpcz_store.cc:269] 0709 12:21:23.328258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.617871 (+289613us) service_pool.cc:224] Handling call
0709 12:21:26.265370 (+2647499us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b5400229a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:26.265493 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1555640192}
I20260709 12:21:26.272766 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:26.274354 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:26.285017 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.932s	user 0.017s	sys 0.055s
W20260709 12:21:26.285439 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.017s	sys 0.055s
W20260709 12:21:26.285732 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 932481us
I20260709 12:21:26.288965 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:26.289495 32378 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
W20260709 12:21:26.292282 31669 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.231s	user 0.036s	sys 0.037s
I20260709 12:21:26.293239 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:26.298223 31669 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.237s	user 0.036s	sys 0.037s
I20260709 12:21:26.300522 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:26.301440 31673 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:26.266090  1432 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:26.260350  1595 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:26.309746 31753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1729) took 2624 ms (client timeout 3233 ms). Trace:
I20260709 12:21:26.312873 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:26.313402 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:26.295250  1516 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:26.288272  1513 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.778s	user 0.021s	sys 0.026s
I20260709 12:21:26.311429 32376 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
W20260709 12:21:26.313033 31753 rpcz_store.cc:269] 0709 12:21:23.680435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.811574 (+131139us) service_pool.cc:224] Handling call
0709 12:21:26.303985 (+2492411us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54004038d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:26.304155 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2500911488}
I20260709 12:21:26.318337 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:26.319859 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:26.320945 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:26.342864 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.315289 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:26.368463 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.856s	user 0.015s	sys 0.036s
W20260709 12:21:26.369125 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.016s	sys 0.037s
I20260709 12:21:26.296491  1620 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.287472  1450 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:26.381655 31751 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
W20260709 12:21:26.390233 31791 connection.cc:256] RPC call timeout handler was delayed by -1.3713715270000648s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2215s for a response
I20260709 12:21:26.392647 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:26.393312 31751 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:26.261273  1587 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.396556  1587 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:26.389820  1432 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.399624 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1718) took 2764 ms. Trace:
W20260709 12:21:26.411491 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.411366 31747 rpcz_store.cc:276] 0709 12:21:23.635023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.837509 (+202486us) service_pool.cc:224] Handling call
0709 12:21:23.837984 (+   475us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:24.431257 (+593273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.399362 (+1968105us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b54000b8bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:26.399561 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1288439552}
I20260709 12:21:26.414141 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:26.416082 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:26.413151 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:26.418352 31747 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:26.420753 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:26.319880  1513 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.823s	user 0.022s	sys 0.026s
I20260709 12:21:26.380304 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:26.423745  1432 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:26.426992 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:26.424280  1194 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:26.433053 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:26.418040  1649 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.441529 31751 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:26.444195  1454 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:26.443387 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:26.417832  1611 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:26.422217  1628 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:26.452147  1556 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.272s	user 0.038s	sys 0.059s
I20260709 12:21:26.455896 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:26.450026 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:26.459864 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:26.473572 31751 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:26.473624 32378 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:26.464138 31748 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
W20260709 12:21:26.466107 31792 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.003s	sys 0.042s
W20260709 12:21:26.478935 31792 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.601s	user 0.003s	sys 0.042s
W20260709 12:21:26.479274 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 601384us
I20260709 12:21:26.435472  1587 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.454425 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:26.472476 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.498534 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:26.512408 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58064.7us
I20260709 12:21:26.483678 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:26.515969 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:26.517591 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:26.485571 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.520220 31751 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:26.521142 31752 raft_consensus.cc:606] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:26.523104 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:26.471824 32369 raft_consensus.cc:2368] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6.
W20260709 12:21:26.476815  1447 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.748s	user 0.015s	sys 0.029s
I20260709 12:21:26.524649 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:26.483055  1556 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.303s	user 0.038s	sys 0.059s
W20260709 12:21:26.529696  1446 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.787s	user 0.011s	sys 0.034s
I20260709 12:21:26.434360  1442 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.049s
W20260709 12:21:26.526186 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.908s	user 0.013s	sys 0.024s
I20260709 12:21:26.484750 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:26.472512  1640 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:26.531870 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:26.536966 31818 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.853s	user 0.005s	sys 0.024s
I20260709 12:21:26.525780  1312 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:26.463600  1649 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:26.556035 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.013s	sys 0.025s
I20260709 12:21:26.555979 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:26.565068 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 80433.2us
I20260709 12:21:26.565883 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:26.550055  1447 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.821s	user 0.015s	sys 0.036s
I20260709 12:21:26.483489  1611 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.556530 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1703) took 2795 ms. Trace:
I20260709 12:21:26.523862  1587 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:26.559284 31818 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.875s	user 0.005s	sys 0.025s
I20260709 12:21:26.559262 31826 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:26.560774 31675 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.008s	sys 0.025s
W20260709 12:21:26.593921 31675 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.008s	sys 0.025s
I20260709 12:21:26.594489 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:26.540006 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:26.567852 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:26.558259  1559 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.011s	sys 0.042s
W20260709 12:21:26.603845 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.606273  1656 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:26.613495  1656 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:26.605085  1330 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:21:26.556035  1446 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.819s	user 0.011s	sys 0.034s
I20260709 12:21:26.558845  1640 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:26.422561  1620 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.628465 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:26.606798  1312 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:26.591099  1556 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:26.596190  1531 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:26.625138  1646 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:26.606994  1559 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.781s	user 0.011s	sys 0.042s
I20260709 12:21:26.614553  1656 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:26.639832 31748 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
W20260709 12:21:26.598364  1273 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.652s	user 0.011s	sys 0.001s
I20260709 12:21:26.634469  1382 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:26.637224  1639 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:26.639142  1640 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:26.655557  1450 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:26.644505  1556 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:26.673637  1618 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.003s
I20260709 12:21:26.720024 32377 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:26.629253  1447 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.022s
I20260709 12:21:26.657770  1595 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:26.638573  1651 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.226s
I20260709 12:21:26.739701  1330 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:26.740504  1330 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:26.752718  1470 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:26.753379  1470 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:26.761443  1517 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:26.762004  1517 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:26.768308  1573 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:26.717279  1647 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.729518  1556 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:26.783465  1660 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.275s	user 0.000s	sys 0.002s
W20260709 12:21:26.797297  1668 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.273s	user 0.001s	sys 0.000s
W20260709 12:21:26.835750  1659 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.177s	user 0.000s	sys 0.001s
I20260709 12:21:26.815063  1647 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:26.854934  1669 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 12:21:26.857295  1649 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:26.850009  1658 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:21:26.852082  1620 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:26.857438  1647 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:26.901643 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:26.905567 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:26.907048 31670 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 2: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:26.908715 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:26.909770 31670 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:26.601372 31676 rpcz_store.cc:276] 0709 12:21:23.761279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.927810 (+166531us) service_pool.cc:224] Handling call
0709 12:21:23.928143 (+   333us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:24.609170 (+681027us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.556237 (+1947067us) spinlock_profiling.cc:243] Waited 680 ms on lock 0x7b54002b0a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:26.556459 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1564639488}
W20260709 12:21:26.911550 31718 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.616s	user 0.011s	sys 0.046s
I20260709 12:21:26.912573 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:26.913287 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 346632us
I20260709 12:21:26.918475 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1791) took 2555 ms. Trace:
I20260709 12:21:26.918208 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:26.923494 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.924054 31748 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
W20260709 12:21:26.924767 31825 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.898s	user 0.015s	sys 0.049s
W20260709 12:21:26.925374 31825 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.899s	user 0.015s	sys 0.049s
I20260709 12:21:26.926066 31826 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:26.927088 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1796) took 2314 ms. Trace:
W20260709 12:21:26.928560 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.927703 31825 rpcz_store.cc:276] 0709 12:21:24.612944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.947906 (+1334962us) service_pool.cc:224] Handling call
0709 12:21:25.986722 (+ 38816us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:25.987068 (+   346us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:25.987096 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:26.011323 (+ 24227us) log.cc:844] Serialized 26 byte log entry
0709 12:21:26.926263 (+914940us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:26.926403 (+   140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.926570 (+   167us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:26.926687 (+   117us) raft_consensus.cc:1685] finished
0709 12:21:26.926709 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:26.926977 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3890,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":5897,"thread_start_us":899536,"threads_started":1,"wal-append.queue_time_us":810028}
W20260709 12:21:26.929800 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.782709 32378 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:26.932987 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:26.919128 31818 rpcz_store.cc:276] 0709 12:21:24.363055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.323187 (+960132us) service_pool.cc:224] Handling call
0709 12:21:25.323453 (+   266us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:25.509627 (+186174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:25.509664 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:25.587169 (+ 77505us) log.cc:844] Serialized 52 byte log entry
0709 12:21:26.087635 (+500466us) spinlock_profiling.cc:243] Waited 20.1 ms on lock 0x7b4800130158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:21:26.593101 (+505466us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:26.593146 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.593232 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:26.917613 (+324381us) raft_consensus.cc:1685] finished
0709 12:21:26.917654 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:26.918181 (+   527us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":231160,"raft.run_cpu_time_us":2911,"raft.run_wall_time_us":491115,"spinlock_wait_cycles":46220672,"thread_start_us":1094719,"threads_started":2,"wal-append.queue_time_us":1088817}
I20260709 12:21:26.934458 31670 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:26.926298 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:26.936070 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.936681 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:26.936965  1301 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.951978  1301 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:26.936766 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.937386  1273 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:21:26.955590 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:26.955681  1273 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.958233  1301 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:26.776635  1312 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.959471 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:26.959872  1273 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:26.939678 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:26.939577  1396 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:21:26.912128 31718 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.011s	sys 0.046s
I20260709 12:21:26.754374  1470 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:26.719484  1382 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:26.935905 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:26.968502 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 673669us
I20260709 12:21:26.962385 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:26.985679  1643 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:26.987951  1649 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.989120  1649 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.990804  1618 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:26.991636 31748 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:26.992198  1618 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:26.993017 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:26.993863 31748 raft_consensus.cc:2376] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6. Re-sending same reply.
I20260709 12:21:26.996699  1604 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:26.820806  1517 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:26.999431  1646 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:26.999781  1604 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:21:27.000836  1517 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:27.001148  1576 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:27.002962 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.001989  1470 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:27.018888  1432 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.536s	user 0.013s	sys 0.005s
I20260709 12:21:27.019006  1529 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.019904 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.019850  1529 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:27.020740 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.020824 32369 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:27.018548 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.022159  1382 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:27.024586  1656 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.027422  1618 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.013576 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:27.028409 31825 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:27.029754  1516 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.707s	user 0.004s	sys 0.013s
I20260709 12:21:27.030424  1596 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.031232 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.032258 31748 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:27.032308 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.033485 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:26.962518  1273 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.050s
I20260709 12:21:27.038872  1587 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.031059  1596 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.034375 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.043432 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:27.044269 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:27.045003  1573 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.045881  1573 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.047331  1604 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:27.048576  1573 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:27.051750 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:27.053103  1656 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.053423 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.056080  1657 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.002079  1576 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:27.054788 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 376092us
I20260709 12:21:27.067723 31825 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:27.070101  1599 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.026s	user 0.009s	sys 0.012s
I20260709 12:21:27.074502 31670 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:21:27.054611  1616 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:27.079075 31676 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:27.067176  1529 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.054423  1609 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260709 12:21:27.089572 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.072408  1573 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:27.096207 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.100009  1377 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260709 12:21:27.100885 32378 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:27.101505 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 240) took 2375 ms. Trace:
I20260709 12:21:27.102377 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:27.104018 31672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1220) took 2890 ms (client timeout 2472 ms). Trace:
I20260709 12:21:27.101972 31671 rpcz_store.cc:276] 0709 12:21:24.725961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.404301 (+678340us) service_pool.cc:224] Handling call
0709 12:21:25.404702 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:25.939964 (+535262us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.100955 (+1160991us) spinlock_profiling.cc:243] Waited 527 ms on lock 0x7b5400046ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:27.101299 (+   344us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1211923840}
W20260709 12:21:27.104676 31672 rpcz_store.cc:269] 0709 12:21:24.213447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.427448 (+214001us) service_pool.cc:224] Handling call
0709 12:21:27.103375 (+2675927us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54002b0554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:27.103700 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2879967488}
I20260709 12:21:27.105525 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1716) took 2609 ms. Trace:
I20260709 12:21:27.105834 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.109033 31671 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:21:27.113091 32370 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.993s	user 0.007s	sys 0.014s
I20260709 12:21:27.113729 32370 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:27.119553 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 409) took 1174 ms. Trace:
I20260709 12:21:27.117673  1573 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:27.118719 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.120821 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:27.109306  1616 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.117010 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:27.126543 32378 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:27.141870 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.126808 32369 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:27.087591  1657 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.125134  1649 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:27.111496  1377 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:27.153853 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.124575  1330 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.105973 31669 rpcz_store.cc:276] 0709 12:21:24.496087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.944702 (+448615us) service_pool.cc:224] Handling call
0709 12:21:24.944988 (+   286us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:25.060137 (+115149us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:25.060171 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:25.061036 (+   865us) log.cc:844] Serialized 26 byte log entry
0709 12:21:26.300453 (+1239417us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:26.300494 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.104915 (+804421us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400229f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:27.104990 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:27.105055 (+    65us) raft_consensus.cc:1685] finished
0709 12:21:27.105067 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.105337 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6190,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":4886,"spinlock_wait_cycles":263835392,"thread_start_us":1238707,"threads_started":1,"wal-append.queue_time_us":1614770}
I20260709 12:21:27.122612  1556 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.127417 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:27.159813 31748 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 3 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:27.133747 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:27.162734  1556 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.162997  1330 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:27.124707 32370 rpcz_store.cc:276] 0709 12:21:25.944739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.946531 (+  1792us) service_pool.cc:224] Handling call
0709 12:21:27.119437 (+1172906us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.158864  1377 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.118424  1640 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.168231 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:27.133778  1616 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:27.132297 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:27.170176 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.171692 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.168351  1377 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.173260 31676 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:27.180853 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.165339  1573 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:27.180346 32375 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.187147 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:27.188704 31673 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:27.190593 31673 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: [5.2-5.2]
W20260709 12:21:27.191263 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.192286 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1652) took 2804 ms. Trace:
I20260709 12:21:27.192389  1576 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.193341 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35284 (request call id 14) took 1171 ms. Trace:
I20260709 12:21:27.224241 31752 rpcz_store.cc:276] 0709 12:21:26.021692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.038477 (+ 16785us) service_pool.cc:224] Handling call
0709 12:21:27.192993 (+1154516us) spinlock_profiling.cc:243] Waited 375 ms on lock 0x7b5400403dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdcf853 00007fe901d9d60a 00007fe8fb3237d4 00007fe8fb323762 00007fe8fb323691 00007fe8fb32360c 00007fe8fb3228c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:27.193130 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":862534784}
I20260709 12:21:27.226097 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:27.194419 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1739) took 1293 ms. Trace:
I20260709 12:21:27.195128 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1787) took 2835 ms. Trace:
W20260709 12:21:27.227250 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.227543 31823 rpcz_store.cc:276] 0709 12:21:24.359348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.938025 (+578677us) service_pool.cc:224] Handling call
0709 12:21:24.938339 (+   314us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:24.996991 (+ 58652us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:24.997023 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:24.997599 (+   576us) log.cc:844] Serialized 52 byte log entry
0709 12:21:25.262411 (+264812us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:25.262467 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.194310 (+1931843us) spinlock_profiling.cc:243] Waited 53.8 ms on lock 0x7b5400197dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:27.194399 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:27.194492 (+    93us) raft_consensus.cc:1685] finished
0709 12:21:27.194505 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.194885 (+   380us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4436,"raft.queue_time_us":54250,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":58862,"spinlock_wait_cycles":123735808,"thread_start_us":240417,"threads_started":1,"wal-append.queue_time_us":481066,"wal-append.run_cpu_time_us":3006,"wal-append.run_wall_time_us":1111789}
I20260709 12:21:27.230679 32371 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:27.226984 31750 rpcz_store.cc:276] 0709 12:21:25.901122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.070863 (+169741us) service_pool.cc:224] Handling call
0709 12:21:26.089557 (+ 18694us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:26.138523 (+ 48966us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.181626 (+1043103us) spinlock_profiling.cc:243] Waited 7.11 ms on lock 0x7b54000b95d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:27.194150 (+ 12524us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16363264}
I20260709 12:21:27.232998  1612 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:27.233203 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:27.233551  1612 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:27.233227 32371 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:21:27.235353  1657 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:27.230207 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.195293 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:27.196609 31824 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1648) took 3166 ms (client timeout 2481 ms). Trace:
I20260709 12:21:27.197762  1640 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:27.198748 31718 leader_election.cc:400] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6.
I20260709 12:21:27.198665 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.237531 31718 leader_election.cc:403] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:27.200881 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:27.236863 31824 rpcz_store.cc:269] 0709 12:21:24.029949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.113056 (+ 83107us) service_pool.cc:224] Handling call
0709 12:21:27.196204 (+3083148us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54001c89d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:27.196402 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3350573440}
I20260709 12:21:27.203013 31671 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:27.184919 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1727) took 1068 ms. Trace:
I20260709 12:21:27.203850 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.208835  1432 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.210397 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:27.210760  1078 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:27.211850 31676 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:27.240880 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1730) took 1043 ms. Trace:
I20260709 12:21:27.238958 31672 rpcz_store.cc:276] 0709 12:21:26.115679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.164323 (+1048644us) service_pool.cc:224] Handling call
0709 12:21:27.164827 (+   504us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.183916 (+ 19089us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:27.183950 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:27.184035 (+    85us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:27.184092 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.184166 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.184300 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.213763 31819 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260709 12:21:27.186031 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52493.8us
I20260709 12:21:27.215850  1649 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:27.217096  1402 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:27.219179  1681 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:27.222021 31820 rpcz_store.cc:276] 0709 12:21:24.388081 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.619236 (+1231155us) service_pool.cc:224] Handling call
0709 12:21:25.619637 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:26.036417 (+416780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.191728 (+1155311us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b54000fc154. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:27.192072 (+   344us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":895971072}
I20260709 12:21:27.198978 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.241405 31676 rpcz_store.cc:276] 0709 12:21:26.197619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.210969 (+1013350us) service_pool.cc:224] Handling call
0709 12:21:27.211588 (+   619us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.240617 (+ 29029us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.240823 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.240356  1078 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:27.247614 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.248930  1432 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.270908  1432 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:27.272008  1078 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.273481  1078 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:27.275658 31827 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 12:21:27.276052 31827 pending_rounds.cc:85] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 5
I20260709 12:21:27.276360 31827 pending_rounds.cc:107] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:21:27.276682 31827 raft_consensus.cc:2889] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:27.283051 31826 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:27.249351 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.284886 32369 raft_consensus.cc:2410] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:27.285624 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.285738 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.287194 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:27.287596 31820 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:27.288370 32374 raft_consensus.cc:2410] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:27.250260 31825 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:27.289021 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:27.251667 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56578.4us
I20260709 12:21:27.248970 31823 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:27.252058  1611 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:27.240490  1656 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:27.253448  1470 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:27.253414 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.257571 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.258656 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:27.256398  1402 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:27.260568 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 94558us
I20260709 12:21:27.264348  1377 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.265215  1517 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6.
W20260709 12:21:27.265941  1663 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.426s	user 0.001s	sys 0.001s
W20260709 12:21:27.267899  1666 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.449s	user 0.001s	sys 0.000s
I20260709 12:21:27.250651  1612 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.297926 31750 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2.
I20260709 12:21:27.299747 31676 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:27.316468 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.277339 31573 catalog_manager.cc:5697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 3 to 4, term changed from 3 to 4, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to <none>, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted, no longer has a pending config: VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194), VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195), VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 4 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:27.286247 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.252707 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:27.270418  1673 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.450s	user 0.000s	sys 0.001s
I20260709 12:21:27.312825 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.313648 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.316112 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.318712 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.291409 31746 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:27.319388 31823 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
I20260709 12:21:27.315923 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.306409  1470 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:27.327458 31673 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:27.345124 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.330025 31825 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:27.332314 31676 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:27.349313 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1230) took 1017 ms. Trace:
W20260709 12:21:27.348059 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 89579.1us
I20260709 12:21:27.349812 31676 rpcz_store.cc:276] 0709 12:21:26.332018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.317336 (+985318us) service_pool.cc:224] Handling call
0709 12:21:27.331721 (+ 14385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.349020 (+ 17299us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.349239 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.344959 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:27.332455 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:27.328280 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 114922us
I20260709 12:21:27.342604 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.342254 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.338382 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 238) took 2613 ms. Trace:
I20260709 12:21:27.331230 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.339524 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1687) took 4729 ms. Trace:
I20260709 12:21:27.358644 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:27.360879 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.353863 31674 rpcz_store.cc:276] 0709 12:21:24.725282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.393728 (+668446us) service_pool.cc:224] Handling call
0709 12:21:25.394038 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:25.878782 (+484744us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.337749 (+1458967us) spinlock_profiling.cc:243] Waited 446 ms on lock 0x7b540022b854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:27.338123 (+   374us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1024841472}
I20260709 12:21:27.356763 31678 rpcz_store.cc:276] 0709 12:21:22.609563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:23.873576 (+1264013us) service_pool.cc:224] Handling call
0709 12:21:24.655025 (+781449us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:24.722573 (+ 67548us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.339080 (+2616507us) spinlock_profiling.cc:243] Waited 781 ms on lock 0x7b54002e8450. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:27.339304 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1796411648}
I20260709 12:21:27.329232 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:21:27.331225  1612 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:27.330891  1661 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.514s	user 0.000s	sys 0.004s
I20260709 12:21:27.329751 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:27.363660 31670 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:27.347359 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1304) took 1023 ms. Trace:
I20260709 12:21:27.340255  1377 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.330210  1470 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:27.327637 32369 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 3.
I20260709 12:21:27.364302 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.367543 31826 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:27.313249  1649 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.368439 31823 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:27.330667  1402 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.369928 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:27.370370 31670 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:27.372876 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 257) took 1481 ms. Trace:
I20260709 12:21:27.373202 31671 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:27.373538  1523 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.374007  1611 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:21:27.390405 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 67976.3us
I20260709 12:21:27.374961  1415 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:27.395056 31673 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:27.395195  1544 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:27.396796 31673 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:27.390105 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.401790 31676 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:21:27.403973  1563 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:27.404572  1523 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.407279 31670 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:27.408803 31670 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:27.409394 31826 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:21:27.410808  1415 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:21:27.412569  1670 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.595s	user 0.001s	sys 0.000s
W20260709 12:21:27.415102 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.375747 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.336452  1594 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 12:21:27.367111  1556 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.376328 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:27.422765 31751 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.422472  1556 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.425627  1556 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.378746 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.427066  1594 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:27.427587  1594 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:27.428565 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.428728  1594 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.429932 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1321) took 1011 ms. Trace:
I20260709 12:21:27.379873 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 242) took 1444 ms. Trace:
I20260709 12:21:27.432561 31752 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:27.431533 31675 rpcz_store.cc:276] 0709 12:21:25.935554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.935695 (+   141us) service_pool.cc:224] Handling call
0709 12:21:25.936066 (+   371us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:25.936448 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:25.936489 (+    41us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:25.937157 (+   668us) log.cc:844] Serialized 52 byte log entry
0709 12:21:26.594497 (+657340us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:26.594557 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.606008 (+ 11451us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:27.266911 (+660903us) spinlock_profiling.cc:243] Waited 48.2 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:27.379325 (+112414us) raft_consensus.cc:1685] finished
0709 12:21:27.379348 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.379677 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12545,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"spinlock_wait_cycles":110869760,"thread_start_us":656857,"threads_started":1,"wal-append.queue_time_us":837186}
I20260709 12:21:27.433794 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:27.368949 31746 rpcz_store.cc:276] 0709 12:21:26.301263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.311894 (+ 10631us) service_pool.cc:224] Handling call
0709 12:21:27.285450 (+973556us) spinlock_profiling.cc:243] Waited 3.16 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:27.325015 (+ 39565us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7263104}
I20260709 12:21:27.434720 31671 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:27.435777 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.436595 31746 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:27.379971 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.438119 31821 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:27.380710 31820 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.381153 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:27.382325  1573 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:27.370455 31672 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:27.440164 31752 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:27.430435 31750 rpcz_store.cc:276] 0709 12:21:26.418860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.428007 (+1009147us) service_pool.cc:224] Handling call
0709 12:21:27.429836 (+  1829us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.387638 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 245) took 1247 ms. Trace:
I20260709 12:21:27.392560  1402 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:27.393054  1530 consensus_queue.cc:1175] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b95aeeaf4c51442b8092cff1de674f2b" responder_term: 6 status { last_received { term: 1 index: 1 } last_committed_idx: 1 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: [5.2-5.2]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:21:27.406186 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:27.389212 31749 rpcz_store.cc:276] 0709 12:21:25.891791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.897834 (+  6043us) service_pool.cc:224] Handling call
0709 12:21:26.214924 (+317090us) spinlock_profiling.cc:243] Waited 5.5 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:26.215150 (+   226us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:26.383362 (+168212us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:26.383395 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:26.383440 (+    45us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:26.383472 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.372329 (+988857us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b5400213ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:27.372420 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.372664 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":398812416}
I20260709 12:21:27.409169 31670 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:27.419442 31791 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:27.419782 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1725) took 1360 ms. Trace:
I20260709 12:21:27.395103 31825 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: [5.2-5.2]
I20260709 12:21:27.430953 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:27.441418 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:27.412309  1415 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.455323  1399 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:27.455747  1399 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.456094  1399 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.446519 31674 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:27.446869 31669 rpcz_store.cc:276] 0709 12:21:26.139073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.178107 (+1039034us) service_pool.cc:224] Handling call
0709 12:21:27.185634 (+  7527us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:27.204956 (+ 19322us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:27.204993 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:27.205654 (+   661us) log.cc:844] Serialized 26 byte log entry
0709 12:21:27.312166 (+106512us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:27.312276 (+   110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.313466 (+  1190us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:27.385951 (+ 72485us) raft_consensus.cc:1685] finished
0709 12:21:27.385975 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.386140 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1614,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":175,"thread_start_us":105979,"threads_started":1,"wal-append.queue_time_us":162858}
I20260709 12:21:27.449682 31678 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:21:27.450816 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.460039 31746 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:27.463384 31676 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:27.438774  1535 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.524492 31820 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
W20260709 12:21:27.520718 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 139687us
I20260709 12:21:27.464702 31672 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:21:27.479130  1657 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:27.521070 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 105694us
I20260709 12:21:27.530282 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:27.530555  1573 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election won for term 4
W20260709 12:21:27.530879 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:27.532630 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:27.532786  1676 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.679s	user 0.002s	sys 0.000s
I20260709 12:21:27.505836 31826 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: []
W20260709 12:21:27.533089 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.532815  1643 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:27.529105 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:27.535465 31751 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
W20260709 12:21:27.541307 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 135205us
I20260709 12:21:27.453874 31677 rpcz_store.cc:276] 0709 12:21:26.059175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.073811 (+ 14636us) service_pool.cc:224] Handling call
0709 12:21:26.091532 (+ 17721us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:26.092028 (+   496us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:26.092075 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:26.097878 (+  5803us) log.cc:844] Serialized 26 byte log entry
0709 12:21:26.593415 (+495537us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:26.593455 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:26.593625 (+   170us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:27.411640 (+818015us) spinlock_profiling.cc:243] Waited 48 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:27.418567 (+  6927us) raft_consensus.cc:1685] finished
0709 12:21:27.418592 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.419607 (+  1015us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2683,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":4140,"spinlock_wait_cycles":110372608,"thread_start_us":468510,"threads_started":1,"wal-append.queue_time_us":667914}
W20260709 12:21:27.536445 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:27.479224 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 48409.6us
I20260709 12:21:27.479046 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:27.484972 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.546746 32375 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:27.505573  1545 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.565s	user 0.002s	sys 0.015s
I20260709 12:21:27.548270 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.547935 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:27.546578 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:27.551620 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.554064 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:27.529050  1634 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.464095  1523 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:27.551568  1545 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.004s	sys 0.019s
I20260709 12:21:27.546168  1643 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.557184 31751 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:27.557516 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.557762 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.521366  1647 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:27.568994 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:27.561837 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.566375 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.555758  1604 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:27.553076  1573 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:27.570773 31752 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:27.557153  1657 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:27.575385 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.575749 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:27.571463 31749 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:27.586408 31751 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
W20260709 12:21:27.582261 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.587049  1617 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:21:27.587481 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.587754  1560 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 12:21:27.605378 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1313) took 1270 ms. Trace:
I20260709 12:21:27.607100 31825 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:27.589385  1656 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:27.592622 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1803) took 1065 ms. Trace:
I20260709 12:21:27.559453 31670 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.599340 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:27.600551  1649 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:27.576767 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 47775.7us
I20260709 12:21:27.567775  1643 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:27.569161  1516 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:27.594610 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1789) took 3233 ms. Trace:
I20260709 12:21:27.484973  1556 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20260709 12:21:27.551743  1415 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260709 12:21:27.588532  1573 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.607023 31745 rpcz_store.cc:276] 0709 12:21:26.334720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.565747 (+1231027us) service_pool.cc:224] Handling call
0709 12:21:27.605310 (+ 39563us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.610333  1387 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.611027 31670 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.611467  1312 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.556898  1611 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 12:21:27.632718  1312 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.614880 31752 raft_consensus.cc:2410] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:21:27.616979 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 43504.3us
I20260709 12:21:27.617750  1645 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:27.636428 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1319) took 1245 ms. Trace:
I20260709 12:21:27.616882 31819 rpcz_store.cc:276] 0709 12:21:24.361713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:24.965919 (+604206us) service_pool.cc:224] Handling call
0709 12:21:27.212970 (+2247051us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.242076 (+ 29106us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.594311 (+352235us) spinlock_profiling.cc:243] Waited 2.25 s on lock 0x7b54001c89d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:27.594526 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5166782848}
I20260709 12:21:27.636888 31752 rpcz_store.cc:276] 0709 12:21:26.391226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.587072 (+1195846us) service_pool.cc:224] Handling call
0709 12:21:27.636360 (+ 49288us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.621706 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.623592 31677 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:27.569860  1523 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.043s
I20260709 12:21:27.625989 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.611919  1656 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:27.631773  1387 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.638775  1681 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:27.656919  1681 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.657394  1681 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:27.618206  1516 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.659631  1516 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:27.659715 31748 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:27.639307 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:27.663043 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:27.663295 31747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1315) took 1326 ms (client timeout 1715 ms). Trace:
W20260709 12:21:27.663801 31747 rpcz_store.cc:269] 0709 12:21:26.336489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.419251 (+ 82762us) service_pool.cc:224] Handling call
0709 12:21:27.611566 (+1192315us) spinlock_profiling.cc:243] Waited 3.22 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:27.663161 (+ 51595us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7395200}
I20260709 12:21:27.664526 32342 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:27.665648 31747 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:21:27.641008  1415 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:21:27.642416 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:27.636334  1645 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.643036  1581 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:27.644488 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.645991 31745 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:27.638971  1579 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.646389  1399 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.651464 31792 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:27.608979 31822 rpcz_store.cc:276] 0709 12:21:26.526722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.526869 (+   147us) service_pool.cc:224] Handling call
0709 12:21:26.527261 (+   392us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:21:26.527701 (+   440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:26.527730 (+    29us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:21:26.538422 (+ 10692us) log.cc:844] Serialized 78 byte log entry
0709 12:21:26.923511 (+385089us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:27.076306 (+152795us) log.cc:844] Serialized 14 byte log entry
0709 12:21:27.090039 (+ 13733us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.098298 (+  8259us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:27.531648 (+433350us) spinlock_profiling.cc:243] Waited 5.46 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:27.592105 (+ 60457us) raft_consensus.cc:1685] finished
0709 12:21:27.592238 (+   133us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.592434 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":163400,"prepare.run_cpu_time_us":111,"prepare.run_wall_time_us":10412,"raft.queue_time_us":8568,"raft.run_cpu_time_us":482,"raft.run_wall_time_us":26656,"spinlock_wait_cycles":12549376,"thread_start_us":501465,"threads_started":2,"wal-append.queue_time_us":304392}
I20260709 12:21:27.666484 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1747) took 1752 ms. Trace:
I20260709 12:21:27.674434 31751 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260709 12:21:27.671387 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 72272.6us
I20260709 12:21:27.717649 31747 rpcz_store.cc:276] 0709 12:21:25.914521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.664727 (+1750206us) service_pool.cc:224] Handling call
0709 12:21:27.665256 (+   529us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:27.666141 (+   885us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.666396 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.680809 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.719866 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.720213 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.720804 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:27.721971 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1736) took 1824 ms. Trace:
W20260709 12:21:27.722232  1694 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:21:27.722738  1312 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:27.724102 31752 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:27.724808 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1748) took 1800 ms. Trace:
I20260709 12:21:27.725162 31752 rpcz_store.cc:276] 0709 12:21:25.924282 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.723319 (+1799037us) service_pool.cc:224] Handling call
0709 12:21:27.723716 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.724519 (+   803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.724728 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:27.726168 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.728081  1387 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:27.728345  1399 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:27.728677  1387 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.726285  1613 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.729290  1387 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.729477  1613 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.677771 31670 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 12:21:27.731276 31670 pending_rounds.cc:85] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 3
I20260709 12:21:27.607931  1535 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.669167 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:27.731556 31670 pending_rounds.cc:107] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:21:27.666574 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:27.735791 31670 raft_consensus.cc:2889] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:27.719858 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 114681us
I20260709 12:21:27.737293 31753 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:27.737447 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.737039 31573 catalog_manager.cc:5697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 1 to 4, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 4 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:21:27.673238 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47905.7us
I20260709 12:21:27.722261 31826 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:21:27.731678 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:27.721117 31822 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:27.736006 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1742) took 1816 ms. Trace:
I20260709 12:21:27.739842 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:27.740020 31745 rpcz_store.cc:276] 0709 12:21:25.901998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.645255 (+1743257us) service_pool.cc:224] Handling call
0709 12:21:27.645712 (+   457us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.718163 (+ 72451us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.718393 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.740692 31818 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 2: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:27.735365 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 83978us
I20260709 12:21:27.666843  1617 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:27.747880 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:27.737051 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:27.744016 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1312) took 1411 ms. Trace:
I20260709 12:21:27.755649 31749 rpcz_store.cc:276] 0709 12:21:26.332415 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.609753 (+1277338us) service_pool.cc:224] Handling call
0709 12:21:27.743955 (+134202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.679672  1645 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:27.682916  1556 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:27.764756  1556 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.765211  1556 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.765290 31642 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:27.766865  1523 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:27.767603  1523 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:27.722457 31744 rpcz_store.cc:276] 0709 12:21:25.897393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:25.903033 (+  5640us) service_pool.cc:224] Handling call
0709 12:21:27.720398 (+1817365us) spinlock_profiling.cc:243] Waited 4.94 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:27.721879 (+  1481us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":11353856}
I20260709 12:21:27.772603 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 262) took 1823 ms. Trace:
I20260709 12:21:27.773066 31744 rpcz_store.cc:276] 0709 12:21:25.949376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.771417 (+1822041us) service_pool.cc:224] Handling call
0709 12:21:27.771904 (+   487us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.772158 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:27.772191 (+    33us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:27.772251 (+    60us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:27.772306 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.772393 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.772531 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.773808  1556 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.774911 31824 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.776319  1556 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:27.776741 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.777916  1535 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.734925  1377 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.779111  1377 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.779452 31792 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:27.781527  1377 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:27.744320  1581 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:27.782660  1560 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:27.783236  1560 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.783629  1560 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:27.785765  1443 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.787422  1613 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:27.788662 31753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1300) took 1689 ms (client timeout 2105 ms). Trace:
W20260709 12:21:27.789172 31753 rpcz_store.cc:269] 0709 12:21:26.101584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.320724 (+219140us) service_pool.cc:224] Handling call
0709 12:21:27.788463 (+1467739us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54001ee1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:27.788593 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3183761280}
I20260709 12:21:27.748396 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1811) took 1215 ms. Trace:
I20260709 12:21:27.790583 31825 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:27.753784 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1242) took 1273 ms. Trace:
I20260709 12:21:27.791114 31825 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:27.791280 31674 rpcz_store.cc:276] 0709 12:21:26.471559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.365595 (+894036us) service_pool.cc:224] Handling call
0709 12:21:27.440157 (+ 74562us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.458227 (+ 18070us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.741740 (+283513us) spinlock_profiling.cc:243] Waited 74.2 ms on lock 0x7b54001d5e50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:27.744335 (+  2595us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":170568576}
I20260709 12:21:27.747287 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:27.792083 31825 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 2, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.792644 32374 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:27.793167 31825 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.669685  1150 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.794262  1301 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:27.794986  1150 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.795339  1301 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:27.795809  1301 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:27.796164 31819 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:27.796705  1301 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.746831 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:27.803181  1301 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.805630 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:27.806841 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:27.751004  1404 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:27.749056  1654 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 12:21:27.746909 31573 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 4 to 5, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 5 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:27.763018 31749 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260709 12:21:27.807677  1404 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.763243 31747 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:27.763865  1656 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.809180 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 260) took 1885 ms. Trace:
I20260709 12:21:27.809643 31747 rpcz_store.cc:276] 0709 12:21:25.924246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.745041 (+1820795us) service_pool.cc:224] Handling call
0709 12:21:27.763027 (+ 17986us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.808816 (+ 45789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.809080 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.782806  1581 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.763900  1616 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:27.810966  1581 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.811945  1544 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:27.812897  1634 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:27.813432  1634 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:27.813983  1544 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:27.814401  1544 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:27.814580  1634 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.815234  1544 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.816519  1544 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:27.817270 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.810979  1616 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.818228  1616 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.811339 31747 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:21:27.819384 31744 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 5. (term mismatch)
I20260709 12:21:27.818792  1643 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.819578 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 266) took 1709 ms. Trace:
I20260709 12:21:27.819770 31744 pending_rounds.cc:85] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 4
I20260709 12:21:27.820102 31744 pending_rounds.cc:107] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 3.5
I20260709 12:21:27.819973 31747 rpcz_store.cc:276] 0709 12:21:26.110489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.810657 (+1700168us) service_pool.cc:224] Handling call
0709 12:21:27.811089 (+   432us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:27.819285 (+  8196us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.819522 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.820468 31744 raft_consensus.cc:2889] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:27.821444  1643 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:27.821620 31826 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:27.822036 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 264) took 1806 ms. Trace:
W20260709 12:21:27.747123  1691 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260709 12:21:27.799140 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.800216 31823 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:27.747692  1470 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.833022  1516 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.834221  1470 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:27.798398 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:27.742898  1579 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.808262 31749 pending_rounds.cc:85] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 5
I20260709 12:21:27.851404 31749 pending_rounds.cc:107] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:21:27.851819 31749 raft_consensus.cc:2889] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:27.811347  1404 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:27.758322 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 91884.4us
I20260709 12:21:27.792496  1596 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.667084  1396 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:27.821445 31747 raft_consensus.cc:1217] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Deduplicated request from leader. Original: 5.4->[6.5-6.5]   Dedup: 6.5->[]
I20260709 12:21:27.861521 31747 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:21:27.798179 31819 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260709 12:21:27.797281 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:27.671588  1415 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:27.680377  1576 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.081s
W20260709 12:21:27.831561  1692 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.212s	user 0.000s	sys 0.001s
I20260709 12:21:27.790835 31818 rpcz_store.cc:276] 0709 12:21:26.533431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.934769 (+401338us) service_pool.cc:224] Handling call
0709 12:21:27.529989 (+595220us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:27.739717 (+209728us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0e9c 00007fe8f9cb0a5b 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:27.748316 (+  8599us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":98173568}
I20260709 12:21:27.835568  1387 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:27.831637 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 62745.4us
I20260709 12:21:27.852928 31824 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260709 12:21:27.852807 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46069.7us
I20260709 12:21:27.857533  1556 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.870375 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:27.871634 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 132823us
I20260709 12:21:27.872646 32374 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:27.813405  1581 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:27.800557  1560 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.863963 31749 pending_rounds.cc:107] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:21:27.863963 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1306) took 1546 ms. Trace:
I20260709 12:21:27.826496 31744 rpcz_store.cc:276] 0709 12:21:26.015915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.774296 (+1758381us) service_pool.cc:224] Handling call
0709 12:21:27.819006 (+ 44710us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.821751 (+  2745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.821976 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.826921  1656 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.875699 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:27.881067 31824 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.878178 31749 raft_consensus.cc:2889] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:27.878257 31747 rpcz_store.cc:276] 0709 12:21:26.317100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.820903 (+1503803us) service_pool.cc:224] Handling call
0709 12:21:27.821337 (+   434us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:27.863667 (+ 42330us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.863888 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.861356  1396 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.863605  1530 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:27.884884 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:27.900821  1617 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:27.901813  1150 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.927640 31573 catalog_manager.cc:5697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 2 to 3, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 3 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:27.928622 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:27.901819 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:27.929373 31573 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 2 to 4, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 4 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:27.903335 31747 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:21:27.903834  1613 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:27.932009 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1307) took 1609 ms. Trace:
I20260709 12:21:27.904562  1612 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:27.937166 31819 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:27.915288  1556 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.905248  1535 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:27.905433  1560 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:27.932034  1579 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.917845 31821 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:27.939479  1556 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.939644  1563 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:27.940296  1563 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:27.867267  1377 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20260709 12:21:27.940627 31821 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:21:27.767424 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:27.879249  1470 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.939507  1579 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.942515  1470 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.942914  1579 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.943632  1596 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.944191  1596 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.932415 31747 rpcz_store.cc:276] 0709 12:21:26.322855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.881947 (+1559092us) service_pool.cc:224] Handling call
0709 12:21:27.903049 (+ 21102us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:27.931567 (+ 28518us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.931943 (+   376us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":242304}
I20260709 12:21:27.944854  1470 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.902379 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:27.945968 31747 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:27.946553 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1325) took 1029 ms. Trace:
I20260709 12:21:27.946861 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:27.946990 31747 rpcz_store.cc:276] 0709 12:21:26.917101 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.945228 (+1028127us) service_pool.cc:224] Handling call
0709 12:21:27.945602 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.946314 (+   712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.946496 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.948556 31744 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:27.949405 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1309) took 1623 ms. Trace:
I20260709 12:21:27.928551 31751 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:27.915591  1396 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:27.941568  1612 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:27.921082 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 261) took 1958 ms. Trace:
I20260709 12:21:27.929517 31818 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:27.950729  1612 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:27.950843 31749 rpcz_store.cc:276] 0709 12:21:25.933615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.756806 (+1823191us) service_pool.cc:224] Handling call
0709 12:21:27.762563 (+  5757us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.891665 (+129102us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.892026 (+   361us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.941669  1617 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:27.951969  1612 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.954303  1612 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:27.954077  1617 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.923601   931 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:27.955431   931 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:21:27.955670 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 161110us
I20260709 12:21:27.956413   931 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.957518  1516 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:27.958033   931 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:27.954432 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:27.954955 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1746) took 2025 ms. Trace:
I20260709 12:21:27.943225 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1324) took 1501 ms. Trace:
I20260709 12:21:27.941255  1563 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:27.949888  1613 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:27.950286 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1310) took 1621 ms. Trace:
I20260709 12:21:27.952924  1377 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:27.970338 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:27.948601 31747 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:27.938850 31819 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:21:27.993054 31749 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:27.917116  1530 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:27.974864  1445 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:27.979835 31827 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:27.965083 31748 rpcz_store.cc:276] 0709 12:21:26.442335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.929286 (+1486951us) service_pool.cc:224] Handling call
0709 12:21:27.929664 (+   378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.942748 (+ 13084us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:27.942782 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:27.942835 (+    53us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:27.942933 (+    98us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.943003 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.943154 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3119,"raft.queue_time_us":9258,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92}
I20260709 12:21:27.946444  1596 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:27.987954  1681 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:27.989252  1563 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:27.990916  1613 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:21:27.960172 31643 leader_election.cc:336] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.985s (SENT)
I20260709 12:21:27.963886 31746 rpcz_store.cc:276] 0709 12:21:25.907898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.645858 (+1737960us) service_pool.cc:224] Handling call
0709 12:21:27.646366 (+   508us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:27.647055 (+   689us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:27.647087 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:27.647775 (+   688us) log.cc:844] Serialized 52 byte log entry
0709 12:21:27.843912 (+196137us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:27.843970 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.844070 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:27.933322 (+ 89252us) raft_consensus.cc:1685] finished
0709 12:21:27.933355 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:27.933548 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3895,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":1923,"thread_start_us":195645,"threads_started":1,"wal-append.queue_time_us":236492}
I20260709 12:21:27.967598 31751 rpcz_store.cc:276] 0709 12:21:26.328589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.927764 (+1599175us) service_pool.cc:224] Handling call
0709 12:21:27.928155 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.950026 (+ 21871us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.950206 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:27.993104 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1326) took 1043 ms. Trace:
I20260709 12:21:28.008435 31747 rpcz_store.cc:276] 0709 12:21:26.950158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.947898 (+997740us) service_pool.cc:224] Handling call
0709 12:21:27.948240 (+   342us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.992834 (+ 44594us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.993045 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:28.013695 31824 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:27.958591  1560 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.021807  1560 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:28.023061 31576 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42070 (request call id 221) took 14285 ms (client timeout 15000 ms). Trace:
I20260709 12:21:28.007313  1445 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:28.024341  1560 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:27.949787 31744 rpcz_store.cc:276] 0709 12:21:26.325779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.893097 (+1567318us) service_pool.cc:224] Handling call
0709 12:21:27.948372 (+ 55275us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:27.949109 (+   737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:27.949342 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:28.000696  1530 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:28.025139 32370 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:21:27.944854  1579 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:28.000531  1556 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:28.030864  1613 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:28.031948  1613 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:28.015138 31573 catalog_manager.cc:5697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 1 to 4, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 4 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:28.000293 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:28.022807  1596 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:28.026321  1681 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:28.027130  1616 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.007871 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:27.999224 31818 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:28.005256  1645 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.056665 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:28.058383 31825 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260709 12:21:28.062049 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:28.062252  1544 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:28.083792  1415 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:28.085189  1415 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:21:28.071275 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:28.087324 31748 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:28.089087  1377 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:28.089751  1377 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.090232  1377 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:28.090479 31748 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:21:28.073578 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:28.023524 31576 rpcz_store.cc:269] 0709 12:21:13.737746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:13.737943 (+   197us) service_pool.cc:224] Handling call
0709 12:21:28.022813 (+14284870us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:20.113828 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:20.285292 (+171464us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:20.285347 (+    55us) write_op.cc:435] Acquired schema lock
0709 12:21:20.285368 (+    21us) tablet.cc:662] Decoding operations
0709 12:21:20.438551 (+153183us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:20.478543 (+ 39992us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:20.484309 (+  5766us) tablet.cc:685] Acquiring locks for 9 operations
0709 12:21:20.831933 (+347624us) tablet.cc:701] Row locks acquired
0709 12:21:20.870706 (+ 38773us) write_op.cc:260] PREPARE: finished
0709 12:21:20.895498 (+ 24792us) write_op.cc:270] Start()
0709 12:21:20.938604 (+ 43106us) write_op.cc:276] Timestamp: P: 1783599680881090 usec, L: 0
0709 12:21:20.943946 (+  5342us) op_driver.cc:348] REPLICATION: starting
0709 12:21:21.042938 (+ 98992us) log.cc:844] Serialized 3654 byte log entry
0709 12:21:21.687144 (+644206us) op_driver.cc:464] REPLICATION: finished
0709 12:21:22.120095 (+432951us) write_op.cc:301] APPLY: starting
0709 12:21:22.128113 (+  8018us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:22.163315 (+ 35202us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:22.163499 (+   184us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:22.164948 (+  1449us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:22.165271 (+   323us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0709 12:21:22.165347 (+    76us) write_op.cc:312] APPLY: finished
0709 12:21:22.165755 (+   408us) log.cc:844] Serialized 71 byte log entry
0709 12:21:22.166038 (+   283us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:22.166215 (+   177us) write_op.cc:454] Released schema lock
0709 12:21:22.166280 (+    65us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:22.385570 (+219290us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:22.400695 (+ 15125us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":618708,"reported_tablets":25,"child_traces":[["op",{"apply.queue_time_us":409159,"apply.run_cpu_time_us":4705,"apply.run_wall_time_us":151762,"num_ops":9,"prepare.queue_time_us":741034,"prepare.run_cpu_time_us":63249,"prepare.run_wall_time_us":1172374,"raft.queue_time_us":331337,"raft.run_cpu_time_us":21463,"raft.run_wall_time_us":315710,"replication_time_us":743002,"thread_start_us":1078774,"threads_started":4,"wal-append.queue_time_us":251596,"wal-append.run_cpu_time_us":26145,"wal-append.run_wall_time_us":1866104}]]}
I20260709 12:21:28.056011  1556 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:28.082894 32369 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:28.093616 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:28.094599 31824 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:28.112921 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:28.081640  1616 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:28.095013 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 94890.8us
I20260709 12:21:28.059091 31825 pending_rounds.cc:85] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 3
I20260709 12:21:28.139191 31825 pending_rounds.cc:107] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260709 12:21:28.139642 31825 raft_consensus.cc:2889] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:28.092574  1645 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:28.062620  1581 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.101014  1445 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:28.105526 32377 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 408) took 1607 ms (client timeout 1586 ms). Trace:
I20260709 12:21:28.109285 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:28.151484 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1753) took 1063 ms. Trace:
I20260709 12:21:28.151876 31820 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:28.111970  1640 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.112594  1643 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.072598  1596 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:21:28.093948 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 175036us
W20260709 12:21:28.158495 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:28.159034 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.137331 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:28.074070  1530 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.152050 31750 rpcz_store.cc:276] 0709 12:21:27.089440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.448714 (+359274us) service_pool.cc:224] Handling call
0709 12:21:28.151251 (+702537us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b54004038d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:28.151417 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":656377600}
W20260709 12:21:28.151510 32377 rpcz_store.cc:269] 0709 12:21:26.497970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.498157 (+   187us) service_pool.cc:224] Handling call
0709 12:21:28.105127 (+1606970us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54000127d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:28.105325 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":475128704}
W20260709 12:21:28.182559 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.063802  1396 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.197350 31825 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:28.163028 31818 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:28.155521 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 199019us
W20260709 12:21:28.204430 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 327117us
I20260709 12:21:28.152540  1399 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260709 12:21:28.194595  1612 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:28.199604  1560 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:21:28.224917 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 72936.7us
W20260709 12:21:28.227587 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 114799us
I20260709 12:21:28.164952  1640 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:28.234354 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 97169.6us
I20260709 12:21:28.170085  1643 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:28.169571  1581 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.231249 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:28.219787  1387 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:28.247931 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:28.250164 31826 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:28.251284 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:28.240377  1387 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:28.266616 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:28.227187  1634 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:21:28.269104 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:28.286907 31820 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:28.244974  1645 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.033s
I20260709 12:21:28.289562 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:28.276077   931 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:21:28.295485 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.275722  1470 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:28.278549  1681 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:28.308516 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60692.4us
I20260709 12:21:28.308669 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:28.314939 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:28.307927  1544 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:28.316009  1544 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:28.316529  1544 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:28.315496 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 110224us
I20260709 12:21:28.323837 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:28.295527  1654 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:28.335527 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:28.307178  1634 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:28.306318  1312 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:28.329566  1645 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.346174 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:28.352463 31820 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
W20260709 12:21:28.353714 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.355438 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:28.332021   931 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:28.337896  1576 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:28.336180  1654 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:28.362246 31822 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:28.321105  1681 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.338474  1634 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:28.365847 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.367514 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:28.338096  1613 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:28.379530 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:28.366883 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52100.9us
I20260709 12:21:28.371457 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:28.380686 31749 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:28.373411 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:28.344192  1530 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:28.375200  1627 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260709 12:21:28.391667  1627 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:28.392081  1627 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:28.392993  1627 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:28.341058  1312 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:28.394886  1312 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:28.376792 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:28.340656  1470 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.028s
I20260709 12:21:28.385109 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:28.394371  1627 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:28.381908 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:28.407110 31674 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:28.407116 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:28.385855 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:28.386258 31672 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260709 12:21:28.387826  1370 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:28.385103 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:28.400391 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:28.360970  1396 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:28.383983 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:28.385012  1586 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.104s
I20260709 12:21:28.406145 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:28.385443 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:28.408056 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:28.408182 31678 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:28.409716  1643 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:28.410174  1150 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:21:28.412720 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 45320.3us
I20260709 12:21:28.448827 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:28.413514 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78087.8us
I20260709 12:21:28.379907  1634 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:28.418856  1576 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:28.419312  1556 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:28.419538   931 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:28.375615  1645 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.426468  1370 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:28.484521 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:28.427436 31671 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:28.390836  1443 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:28.446039 31826 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:28.485942 31678 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:28.340308  1387 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:28.489194 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:28.477284 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:28.483271 31677 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260709 12:21:28.421504  1703 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:21:28.454788 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:28.420581  1700 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:21:28.490520 31823 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:28.448774  1301 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:28.497023 31675 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:28.460083  1640 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:28.491446 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 107588us
W20260709 12:21:28.499071 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.503274 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:28.513566 31820 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:28.460973  1576 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:28.484352  1695 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:21:28.467821  1643 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:28.499455 31827 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:28.467746  1556 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:28.496081  1560 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:21:28.532183 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:28.503983  1415 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:28.499429 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:28.508535 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:28.539275 31825 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:28.540184 31820 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:28.540696 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:28.526814 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:28.542292 31677 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:28.495292 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:28.503599  1581 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:28.509706  1387 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:28.521461 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:28.533002 31823 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:28.519448  1443 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:28.553156 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:28.534271 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:28.556931 31671 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:28.557132 31826 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:28.547560 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:28.459650  1445 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
I20260709 12:21:28.541594 31675 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:28.538774  1415 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:28.563939  1415 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:28.564492  1594 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:28.566764  1727 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:28.567277  1681 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:28.552636 32372 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:28.569083  1443 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.571010  1656 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 12:21:28.572803  1708 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.074s	user 0.001s	sys 0.001s
I20260709 12:21:28.519999  1576 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:28.550871 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.532183  1544 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:28.508509  1301 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.484762  1654 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.558400 31678 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:28.562814 32373 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:28.563320 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:28.574977 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:28.574977 31822 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
I20260709 12:21:28.548504  1720 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:28.554180 31673 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:28.580426 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:28.584426 31674 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:28.574168  1312 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.553351 32370 raft_consensus.cc:2376] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5. Re-sending same reply.
I20260709 12:21:28.619861 32376 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:21:28.592505  1563 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.586s	user 0.000s	sys 0.011s
I20260709 12:21:28.572817  1645 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:28.630504 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 80061.2us
I20260709 12:21:28.623107 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:28.592213  1301 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:28.549961  1078 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
W20260709 12:21:28.637696 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 116582us
I20260709 12:21:28.645488  1415 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:28.651737 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:28.653503 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 265) took 2628 ms. Trace:
I20260709 12:21:28.620105  1594 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:21:28.674240 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.630136  1716 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:28.653934 31753 rpcz_store.cc:276] 0709 12:21:26.020675 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.790002 (+1769327us) service_pool.cc:224] Handling call
0709 12:21:27.790406 (+   404us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:27.863826 (+ 73420us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:21:27.863864 (+    38us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:27.878644 (+ 14780us) log.cc:844] Serialized 52 byte log entry
0709 12:21:28.153375 (+274731us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:28.153416 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:28.153508 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:28.648568 (+495060us) raft_consensus.cc:1685] finished
0709 12:21:28.648591 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:28.648756 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56584,"raft.queue_time_us":51414,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":5036,"thread_start_us":253141,"threads_started":1,"wal-append.queue_time_us":374144}
W20260709 12:21:28.691860 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51646.2us
W20260709 12:21:28.698436 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.653630  1654 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:28.691038  1707 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.268s	user 0.001s	sys 0.001s
I20260709 12:21:28.559368  1370 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.650893  1563 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.053s
I20260709 12:21:28.689356  1716 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:28.712334  1594 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:28.715517  1194 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.019s
I20260709 12:21:28.720904  1301 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.723169  1312 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:28.718783 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 317837us
I20260709 12:21:28.688278  1722 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:21:28.733443 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 248864us
I20260709 12:21:28.701224  1613 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:28.630097  1556 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:28.742785 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:28.732328  1377 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.641s	user 0.010s	sys 0.003s
I20260709 12:21:28.728816  1716 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:28.751500  1470 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:28.754070  1715 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.254s	user 0.000s	sys 0.001s
I20260709 12:21:28.751965  1370 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:28.764237  1556 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:28.748476  1617 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.019s
I20260709 12:21:28.786578 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:28.792634 31670 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:28.789557  1078 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:28.790051 31674 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:28.751137  1377 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:28.794152 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:28.795503  1399 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:28.803622  1399 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.806109  1399 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:28.810073 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:28.811056 32375 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:28.811465 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:28.812880 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:28.814618 32376 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260709 12:21:28.816298 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:28.817174 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.826156 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:28.760514  1579 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.041s
W20260709 12:21:28.840620 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 64910.3us
I20260709 12:21:28.843061 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:28.843988 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:28.845122  1735 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:28.840220  1396 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:28.846127  1734 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:28.848850  1445 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:28.849269  1445 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:28.850208  1445 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:28.851610  1445 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:28.853816  1634 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:28.855254 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61177.7us
I20260709 12:21:28.858871  1586 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:28.861083 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:28.862087 31753 raft_consensus.cc:2410] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 6 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 2.
I20260709 12:21:28.863068 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:28.863153 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:28.863870 31752 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:28.863972 31747 raft_consensus.cc:2376] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 2. Re-sending same reply.
I20260709 12:21:28.864915 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:28.865592 31750 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:28.797627  1078 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:28.866582 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:28.867306 31748 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:28.876819 31749 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:28.801502  1544 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:28.777040  1721 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.063s
W20260709 12:21:28.889751 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.894471 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:28.778221  1470 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:28.893465 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:28.917748 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 73849.8us
I20260709 12:21:28.918426 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:28.923058 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:28.927393 31676 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:28.927160 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:28.928225 32377 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:28.928261 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:28.929870 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:28.955406 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:28.961499 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.962975 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:28.963593 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:28.967236 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48302.6us
I20260709 12:21:28.967407 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:28.968199 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:28.968847 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:28.967904 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:28.966703 31749 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:28.972437 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:28.980590 31747 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:28.986027 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:28.986102 31750 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
W20260709 12:21:28.988183 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.989648 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:28.989892 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:28.991837 31752 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:28.922756  1396 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:28.926118  1415 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:28.932248  1150 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.010289 31750 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 1. Current term is 2. Ops: []
W20260709 12:21:28.994055 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 256291us
I20260709 12:21:28.964774  1613 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20260709 12:21:28.932934  1586 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:28.998531 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:29.021478 31745 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:28.968840  1594 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:29.008296 31824 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:28.976593   931 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:29.017803 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:28.995920 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:29.022289  1594 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:29.033005 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:29.014191  1740 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:29.039650 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:29.043910 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:29.036055  1729 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:29.046988  1729 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:29.047201  1737 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:21:29.047422  1729 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:29.047704  1634 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:29.033004   931 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.049126 31748 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:21:29.031585  1594 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:29.051826 31826 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:29.054446  1078 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.055989  1078 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:29.050338   931 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:29.027828  1194 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:29.060563 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.067145 31747 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
W20260709 12:21:29.069519 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 102107us
I20260709 12:21:29.073302 31673 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:29.015045  1150 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.036248  1645 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:29.074733 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:29.045835  1544 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:29.046406  1740 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:29.065850  1194 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:29.069306  1634 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:29.067447  1586 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:29.042171  1722 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:29.090953 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:29.099503 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 103654us
I20260709 12:21:29.108204 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:29.110340 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:29.091732  1579 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:29.092988  1544 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:29.123248  1645 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:29.095793  1377 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:29.102488  1194 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:29.100670  1740 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:29.130767  1544 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:29.110694  1586 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:29.108925  1722 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:29.155941 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 45668.3us
W20260709 12:21:29.131953 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 64410.7us
I20260709 12:21:29.168861 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:29.164925 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 54588.2us
I20260709 12:21:29.102218  1387 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 12:21:29.170584 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:29.171036 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 431184us
I20260709 12:21:29.172457 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:29.174175 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 278503us
I20260709 12:21:29.174479 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:29.175905 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:29.181442 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:29.113704  1613 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.054s
I20260709 12:21:29.131407  1720 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:29.169777  1586 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:29.191896 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.192333  1301 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:29.193894  1720 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:29.194842  1725 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:29.195592  1150 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:29.153546  1579 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:21:29.211730 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.169340  1470 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:29.200788 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:29.166025  1748 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:29.178784  1640 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:29.220572 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:29.224661 31669 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:29.195871  1723 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.179014  1634 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:29.200623  1730 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.244515  1730 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.201354  1759 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:29.246996  1759 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:29.247417  1759 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:29.182467  1560 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:21:29.237413  1757 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:29.230402  1470 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.224943  1748 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:29.241322  1747 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:21:29.220037  1563 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260709 12:21:29.240963  1579 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:29.279922 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 108117us
W20260709 12:21:29.242880  1745 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.111s	user 0.002s	sys 0.000s
W20260709 12:21:29.248207  1627 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.819s	user 0.012s	sys 0.000s
W20260709 12:21:29.277045 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 104669us
I20260709 12:21:29.280957 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:29.235283  1643 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:21:29.283429 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 115492us
I20260709 12:21:29.305544 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:29.265431  1756 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:29.278055  1762 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:29.236413  1723 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.237703  1645 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.300525  1751 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:29.362601 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 224) took 1152 ms. Trace:
W20260709 12:21:29.320567 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 130519us
W20260709 12:21:29.356563 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 75711.3us
I20260709 12:21:29.279759  1415 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:29.271394  1748 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:29.312964 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:29.321341  1370 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.530s	user 0.009s	sys 0.000s
I20260709 12:21:29.364279 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:29.371204 31669 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:29.363121 31818 rpcz_store.cc:276] 0709 12:21:28.210740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.391379 (+180639us) service_pool.cc:224] Handling call
0709 12:21:28.391790 (+   411us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:28.507680 (+115890us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:28.527322 (+ 19642us) log.cc:844] Serialized 14 byte log entry
0709 12:21:28.798244 (+270922us) spinlock_profiling.cc:243] Waited 4.75 ms on lock 0x7b180020a2e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:21:28.814181 (+ 15937us) spinlock_profiling.cc:243] Waited 1.96 ms on lock 0x7b180020a2e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:21:29.091230 (+277049us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b180020a2e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe900ca958f 00007fe8f9ce8571 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd
0709 12:21:29.180380 (+ 89150us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:29.180927 (+   547us) log.cc:844] Serialized 26 byte log entry
0709 12:21:29.181092 (+   165us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:29.181132 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:29.181219 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:29.362351 (+181132us) raft_consensus.cc:1685] finished
0709 12:21:29.362377 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:29.362535 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":152467,"prepare.queue_time_us":40323,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":270235,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":34289,"spinlock_wait_cycles":57725696,"thread_start_us":353216,"threads_started":2,"wal-append.queue_time_us":807826}
I20260709 12:21:29.271674  1470 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:29.372519 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 56085us
I20260709 12:21:29.390704 31745 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:29.394467 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:29.304294  1704 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:29.321578  1301 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:29.308739  1757 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:29.284108  1656 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.098s
I20260709 12:21:29.361387  1751 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:29.381907  1744 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:21:29.364362  1730 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:29.310938  1752 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:29.380319  1415 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:29.322183  1756 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:29.371152  1387 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:29.371645  1312 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:29.409036 31748 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 5. Ops: [4.5-4.5]
W20260709 12:21:29.412030 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47907.5us
I20260709 12:21:29.360870  1723 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:29.417279 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 226802us
I20260709 12:21:29.424520 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:21:29.411809  1716 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.651s	user 0.011s	sys 0.000s
I20260709 12:21:29.431841  1415 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:29.436457  1757 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.437963 31744 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:29.439021  1767 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.439074 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:29.439695  1767 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.440214  1769 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:29.441881  1706 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:29.442365  1706 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:29.456478 31746 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:29.444751 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:29.425146  1560 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:29.462126 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.420022  1751 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:29.464879 31561 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet f6404cb9c2174326b06cbbb0d27d43f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica b95aeeaf4c51442b8092cff1de674f2b is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 6 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:29.466037  1764 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:29.466692   931 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:29.467270   931 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.443404  1706 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.469570   931 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:29.469842  1706 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:29.473598 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:29.475389 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:29.426023  1752 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:29.484213 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.407238  1704 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.485457 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:29.486227 31822 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
W20260709 12:21:29.471334  1654 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.763s	user 0.005s	sys 0.007s
I20260709 12:21:29.487054 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:29.490234 31827 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: [3.4-5.5]
I20260709 12:21:29.470736  1399 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:29.469076  1640 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:29.481312 31677 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:29.496271 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:29.482026 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:29.496906 31822 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:29.497063 31826 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:29.481709  1764 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:29.478442 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:29.479064 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:29.434137  1756 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:29.430047 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 259524us
I20260709 12:21:29.504024 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:29.439021  1758 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:29.408075  1765 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260709 12:21:29.518725 31819 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:29.519016 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:29.508375 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:29.520296 31821 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:29.510656 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:29.508770 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:29.521561 31673 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:29.511608 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:29.522725 31824 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:29.509984 31825 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:29.513073 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:29.520404 31823 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260709 12:21:29.509459 31678 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:29.416855  1634 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.529512 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:29.493459  1312 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:29.532591 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:29.543479 31826 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 4: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:29.540565 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:29.492331  1752 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:29.550523 31825 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
W20260709 12:21:29.543404 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.541167 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:29.545807 31823 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:29.552353 32374 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:29.509402  1758 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:29.552150 31823 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:29.525651  1616 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:29.456058  1716 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.463513  1769 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20260709 12:21:29.561445 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:29.551451  1640 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:29.542610  1312 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:29.579960 31823 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.551550  1643 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:29.559525  1560 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:29.545812  1563 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
I20260709 12:21:29.635140 31823 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:29.642127 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.578305  1581 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:29.578112  1301 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:21:29.599880  1594 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.549s	user 0.007s	sys 0.003s
I20260709 12:21:29.589279  1312 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:29.581044  1616 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:29.582713  1387 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.587419 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:29.647614  1399 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.629374  1560 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:21:29.705420 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:29.710984 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.650043  1754 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.653031  1470 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:29.685355  1613 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.694432 32373 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:29.652299  1301 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:29.732318 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:29.662717  1764 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.749308  1771 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.749475  1775 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:29.767375 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:29.727088  1544 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.575s	user 0.012s	sys 0.000s
I20260709 12:21:29.662977  1581 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.692116  1594 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.770923 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:29.713424  1387 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.761377  1768 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election won for term 5
W20260709 12:21:29.808874  1777 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 12:21:29.809687  1780 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:29.807364 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:29.723439  1399 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.781996  1771 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.847141  1782 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:29.847998  1776 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.848624  1776 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.850823 32371 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:29.831439  1560 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.850317 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:29.854050 31825 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:29.771540  1613 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.685710  1758 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:29.891176  1784 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:21:29.892174  1758 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:29.822856  1556 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.022s	user 0.007s	sys 0.006s
I20260709 12:21:29.836150  1768 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:29.894188  1768 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.810330  1780 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:29.897531  1781 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.104s	user 0.000s	sys 0.002s
I20260709 12:21:29.898841  1780 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:29.900756  1776 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:29.832370  1563 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.901648  1563 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:29.904701  1759 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.657s	user 0.007s	sys 0.005s
I20260709 12:21:29.905215  1759 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:29.874478  1783 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:21:29.891003  1750 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:29.875577 31753 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:29.785717  1544 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:29.940076  1786 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:29.940913  1786 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.943423  1788 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:29.806530  1778 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:29.961539 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:29.962546 31819 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:29.775064  1643 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.772257  1754 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:29.944001  1722 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.765s	user 0.006s	sys 0.006s
I20260709 12:21:29.786765  1775 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.932443  1750 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:29.974211 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:29.999646  1729 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.952s	user 0.013s	sys 0.001s
I20260709 12:21:30.000281  1729 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:29.999799  1576 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.400s	user 0.015s	sys 0.000s
I20260709 12:21:30.003376  1576 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:30.004868 32369 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:29.847954  1782 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.975731  1716 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:29.812456  1616 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:29.959298  1788 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:30.074033 31823 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:21:30.026571 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.586938 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1846) took 1070 ms. Trace:
I20260709 12:21:30.587416 31823 rpcz_store.cc:276] 0709 12:21:29.516904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.524491 (+  7587us) service_pool.cc:224] Handling call
0709 12:21:29.545678 (+ 21187us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.586595 (+1040917us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.586874 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:30.087318 31747 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.000s	sys 0.008s
I20260709 12:21:30.136022 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:30.070849 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.008s	sys 0.008s
I20260709 12:21:30.593044 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:30.585525 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 270) took 1386 ms. Trace:
W20260709 12:21:30.141811 31822 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.007s	sys 0.009s
W20260709 12:21:30.090158 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:30.596880 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:30.603478 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:30.599076 31822 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.052s	user 0.015s	sys 0.015s
W20260709 12:21:30.594141  1794 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.682s	user 0.012s	sys 0.014s
W20260709 12:21:30.601467 31819 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.614s	user 0.010s	sys 0.001s
W20260709 12:21:30.386853  1194 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.212s	user 0.014s	sys 0.000s
W20260709 12:21:30.383955  1078 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.327s	user 0.016s	sys 0.000s
W20260709 12:21:30.595728  1579 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.008s	sys 0.004s
I20260709 12:21:30.598307 31825 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:30.598505 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.122s	user 0.013s	sys 0.018s
W20260709 12:21:30.460943  1740 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.287s	user 0.013s	sys 0.000s
W20260709 12:21:30.423556  1415 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.947s	user 0.013s	sys 0.000s
W20260709 12:21:30.351847  1586 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.127s	user 0.013s	sys 0.001s
I20260709 12:21:30.612839 31819 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
W20260709 12:21:30.612965  1752 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.045s	user 0.008s	sys 0.005s
I20260709 12:21:30.598832 31670 rpcz_store.cc:276] 0709 12:21:29.199386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.210029 (+ 10643us) service_pool.cc:224] Handling call
0709 12:21:29.227695 (+ 17666us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:29.227978 (+   283us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:29.228017 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:29.239553 (+ 11536us) log.cc:844] Serialized 26 byte log entry
0709 12:21:29.412061 (+172508us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:29.412129 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:29.412267 (+   138us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:30.585239 (+1172972us) raft_consensus.cc:1685] finished
0709 12:21:30.585262 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.585449 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":90692,"threads_started":1,"wal-append.queue_time_us":261279}
I20260709 12:21:30.603224  1581 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:30.600458  1758 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.620079 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1848) took 1102 ms. Trace:
I20260709 12:21:30.599184 31669 raft_consensus.cc:2410] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:30.554260  1774 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:30.625710  1756 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.099s	user 0.003s	sys 0.009s
W20260709 12:21:30.627460  1725 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 12:21:30.627553 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:30.613624  1563 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.197784  1470 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:29.986443  1722 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:30.630129 32373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 420) took 1819 ms (client timeout 1925 ms). Trace:
I20260709 12:21:30.630888 32375 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260709 12:21:30.633452 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1764) took 1148 ms. Trace:
I20260709 12:21:30.636229 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:30.671546 31671 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:21:30.601260  1751 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.102s	user 0.009s	sys 0.005s
W20260709 12:21:30.638495 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 1.16309e+06us
I20260709 12:21:30.009413  1786 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:30.671049 31672 rpcz_store.cc:276] 0709 12:21:29.485511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.485717 (+   206us) service_pool.cc:224] Handling call
0709 12:21:29.508508 (+ 22791us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:29.508859 (+   351us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:29.508887 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:29.525386 (+ 16499us) log.cc:844] Serialized 26 byte log entry
0709 12:21:29.730908 (+205522us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:29.730974 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:29.731080 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:30.633133 (+902053us) raft_consensus.cc:1685] finished
0709 12:21:30.633172 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.633361 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9733,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":9483,"spinlock_wait_cycles":81280,"thread_start_us":162625,"threads_started":1,"wal-append.queue_time_us":1066512}
I20260709 12:21:30.668059  1756 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:30.640151 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:30.670537 32373 rpcz_store.cc:269] 0709 12:21:28.810731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.812563 (+  1832us) service_pool.cc:224] Handling call
0709 12:21:30.629879 (+1817316us) spinlock_profiling.cc:243] Waited 785 ms on lock 0x7b54004272d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:30.630059 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1806092544}
I20260709 12:21:30.675037 31823 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:30.660267  1758 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:30.676391 31823 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:30.676940  1370 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:30.677363 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 230) took 1096 ms. Trace:
I20260709 12:21:30.677752 31823 rpcz_store.cc:276] 0709 12:21:29.581157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.675796 (+1094639us) service_pool.cc:224] Handling call
0709 12:21:30.676177 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.677062 (+   885us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.677302 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.641439 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1766) took 1137 ms. Trace:
I20260709 12:21:30.112506  1729 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.658496  1634 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.685142  1634 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:30.332373  1727 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:30.592758  1720 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:30.144289  1544 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:30.597757 31747 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.182s	user 0.000s	sys 0.013s
I20260709 12:21:30.705477 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 277) took 1785 ms. Trace:
W20260709 12:21:30.611240  1794 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.013s	sys 0.015s
I20260709 12:21:30.650012  1415 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:30.706195 31747 rpcz_store.cc:276] 0709 12:21:28.919872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.075842 (+155970us) service_pool.cc:224] Handling call
0709 12:21:29.076221 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:29.076468 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:29.177509 (+101041us) log.cc:844] Serialized 14 byte log entry
0709 12:21:29.394091 (+216582us) log.cc:844] Serialized 14 byte log entry
0709 12:21:29.415966 (+ 21875us) log.cc:844] Serialized 14 byte log entry
0709 12:21:29.467147 (+ 51181us) spinlock_profiling.cc:243] Waited 4.74 ms on lock 0x7b180031a380. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:21:30.704706 (+1237559us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:30.704826 (+   120us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:30.704914 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.705041 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.705377 (+   336us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":112468,"prepare.queue_time_us":335006,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":116,"raft.queue_time_us":336537,"raft.run_cpu_time_us":578,"raft.run_wall_time_us":41669,"spinlock_wait_cycles":10903936,"thread_start_us":1544515,"threads_started":3}
I20260709 12:21:30.709493  1415 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:30.710427 31675 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:30.650063  1586 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:30.712033  1634 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.142419  1312 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:30.713406 31818 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
W20260709 12:21:30.636642  1579 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.009s	sys 0.004s
W20260709 12:21:30.650296 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.864s	user 0.013s	sys 0.033s
I20260709 12:21:30.636934  1716 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:30.714859 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.928s	user 0.013s	sys 0.033s
I20260709 12:21:30.715072  1716 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:30.715517  1729 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.716056  1729 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:30.648458  1706 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.156s	user 0.012s	sys 0.002s
I20260709 12:21:30.637395  1757 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.654417 31827 rpcz_store.cc:276] 0709 12:21:29.517384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.532161 (+ 14777us) service_pool.cc:224] Handling call
0709 12:21:29.532540 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:29.559180 (+ 26640us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:29.559237 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:29.597895 (+ 38658us) log.cc:844] Serialized 26 byte log entry
0709 12:21:29.929160 (+331265us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:29.969259 (+ 40099us) log.cc:844] Serialized 14 byte log entry
0709 12:21:29.969568 (+   309us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:29.969682 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:30.619835 (+650153us) raft_consensus.cc:1685] finished
0709 12:21:30.619858 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.620004 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":35671,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":50,"raft.queue_time_us":236924,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":32785,"thread_start_us":168490,"threads_started":1,"wal-append.queue_time_us":957751}
I20260709 12:21:30.679657 31823 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:30.647573  1740 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:30.677404  1370 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:30.679401  1758 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.077026  1750 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:30.686818 31825 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:30.652974  1752 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:30.637555  1194 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:30.656651  1576 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.679435 31674 rpcz_store.cc:276] 0709 12:21:29.504517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.518353 (+ 13836us) service_pool.cc:224] Handling call
0709 12:21:29.542384 (+ 24031us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:29.548728 (+  6344us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:29.629400 (+ 80672us) log.cc:844] Serialized 14 byte log entry
0709 12:21:29.734213 (+104813us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:30.594690 (+860477us) log.cc:844] Serialized 26 byte log entry
0709 12:21:30.607613 (+ 12923us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:30.607650 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.607717 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:30.641090 (+ 33373us) raft_consensus.cc:1685] finished
0709 12:21:30.641112 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.641370 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23762,"prepare.queue_time_us":85603,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":6648,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":11069,"thread_start_us":38165,"threads_started":1,"wal-append.queue_time_us":966627}
I20260709 12:21:30.825805 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 235) took 1186 ms. Trace:
I20260709 12:21:30.852576 31820 rpcz_store.cc:276] 0709 12:21:29.639006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.824597 (+1185591us) service_pool.cc:224] Handling call
0709 12:21:30.825049 (+   452us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.825301 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:30.825370 (+    69us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:30.825437 (+    67us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:30.825488 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.825572 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.825722 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.721458 32374 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: []
W20260709 12:21:30.855311 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:30.718516 31753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1771) took 1836 ms (client timeout 1566 ms). Trace:
W20260709 12:21:30.856027 31753 rpcz_store.cc:269] 0709 12:21:28.882538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.882738 (+   200us) service_pool.cc:224] Handling call
0709 12:21:30.718286 (+1835548us) spinlock_profiling.cc:243] Waited 856 ms on lock 0x7b54004056d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:30.718435 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1968980352}
I20260709 12:21:30.856673 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 238) took 1216 ms. Trace:
I20260709 12:21:30.730826 31824 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:30.856801 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:30.829818 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:30.857076 31825 rpcz_store.cc:276] 0709 12:21:29.639890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.112387 (+472497us) service_pool.cc:224] Handling call
0709 12:21:30.856613 (+744226us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:30.813534 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 1.04619e+06us
I20260709 12:21:30.732836  1399 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:30.733083  1748 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.328s	user 0.013s	sys 0.001s
I20260709 12:21:30.857502 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 234) took 1275 ms. Trace:
W20260709 12:21:30.819571 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 140367us
I20260709 12:21:30.861618 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 432) took 1285 ms. Trace:
I20260709 12:21:30.861428 31824 rpcz_store.cc:276] 0709 12:21:29.638554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.730011 (+1091457us) service_pool.cc:224] Handling call
0709 12:21:30.730381 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.857253 (+126872us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.857457 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.862042 32374 rpcz_store.cc:276] 0709 12:21:29.569721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.569884 (+   163us) service_pool.cc:224] Handling call
0709 12:21:30.721010 (+1151126us) spinlock_profiling.cc:243] Waited 4.25 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:30.721260 (+   250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.855040 (+133780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.855242 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9770240}
I20260709 12:21:30.862068 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:30.855225 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 231) took 1273 ms. Trace:
I20260709 12:21:30.865091 31823 rpcz_store.cc:276] 0709 12:21:29.581461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.678464 (+1097003us) service_pool.cc:224] Handling call
0709 12:21:30.679487 (+  1023us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.854947 (+175460us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.855162 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.868634 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 572) took 1107 ms. Trace:
I20260709 12:21:30.868698 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 236) took 1229 ms. Trace:
I20260709 12:21:30.868984 32378 rpcz_store.cc:276] 0709 12:21:29.761756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.761948 (+   192us) service_pool.cc:224] Handling call
0709 12:21:30.829614 (+1067666us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:30.868582 (+ 38968us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3982720}
I20260709 12:21:30.869098 31820 rpcz_store.cc:276] 0709 12:21:29.639312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.853826 (+1214514us) service_pool.cc:224] Handling call
0709 12:21:30.868177 (+ 14351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.868419 (+   242us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:30.868442 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:30.868486 (+    44us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:30.868524 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:30.868575 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:30.868654 (+    79us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:30.871265 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:30.830574  1301 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.047s	user 0.012s	sys 0.001s
I20260709 12:21:30.872341 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:30.874037 31747 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:21:30.834033  1767 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.877022 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:30.787622  1762 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:30.722205  1627 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.829715  1470 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:30.877686 31750 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:30.694341  1771 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:30.728183  1586 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.816335  1775 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:30.717310  1716 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:30.864029  1813 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:21:30.858346  1415 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.798348  1788 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.881191 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:30.821060  1722 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.861498  1579 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:30.879369 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:30.883561  1756 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.881405  1312 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:30.886186 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:30.883512 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:30.863737  1758 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.883162  1803 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.867730  1757 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:30.879122  1706 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:30.927707  1803 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:30.923655  1643 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.928823  1643 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:30.930002  1803 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:30.892379 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:30.906826  1779 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 12:21:30.930923 31753 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:30.930955  1643 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:30.931511  1779 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:30.931902  1779 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:30.932742  1779 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:30.933919  1779 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:30.879312  1301 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:30.878973  1797 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:30.912114  1399 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 12:21:30.947245 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:30.948359 31745 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:30.952255 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:30.883442 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:30.877027  1725 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:30.883003  1810 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260709 12:21:30.901522  1759 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.884150  1774 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:30.955044  1774 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:30.957962  1774 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.881402  1795 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
W20260709 12:21:30.901923 31818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1688) took 1413 ms (client timeout 1540 ms). Trace:
I20260709 12:21:30.687767  1751 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:21:30.959170 31818 rpcz_store.cc:269] 0709 12:21:29.485446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.497970 (+ 12524us) service_pool.cc:224] Handling call
0709 12:21:30.890523 (+1392553us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b540030ded4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:30.899088 (+  8565us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2779602688}
I20260709 12:21:30.960409 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:30.954394  1725 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:30.960795  1751 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:30.903148 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:30.961112  1725 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:30.961700 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:30.961927 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62035.1us
I20260709 12:21:30.911384 31751 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:30.787873  1782 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:30.866080  1720 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:30.963439  1724 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:30.965148  1720 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:30.787113  1594 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.798415  1727 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:30.916224 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:30.967402 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:30.968519 31747 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:30.968597  1727 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:30.983325 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:30.984416 31747 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:30.916811 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:30.899070  1793 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:30.926401 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:30.867857  1748 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:30.992520  1793 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:21:30.992673 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66364us
I20260709 12:21:30.993541 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:30.994485  1594 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:30.994580  1748 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:30.995043  1594 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:30.881445  1800 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:30.872007  1370 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:30.995702  1594 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:30.995975  1800 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:30.996485  1370 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:30.962924 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:30.999138  1720 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:31.001389  1720 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:31.003273  1720 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:31.003798  1312 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:31.004338  1724 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:31.004650  1720 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:31.005052  1720 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:30.993932 31823 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:30.998507  1800 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.994652 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:31.007278  1643 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:31.007949  1643 consensus_queue.cc:1243] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:21:31.010823 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:31.011538 31824 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:30.935307  1788 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:21:31.031330  1815 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
I20260709 12:21:31.032048  1752 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:31.031330  1809 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 12:21:31.034449  1704 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:31.036512  1754 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:31.038472 32369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 564) took 2572 ms (client timeout 2743 ms). Trace:
I20260709 12:21:31.038355 31750 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:31.038951 32369 rpcz_store.cc:269] 0709 12:21:28.465550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.466764 (+  1214us) service_pool.cc:224] Handling call
0709 12:21:31.038179 (+2571415us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b54003f3254. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:31.038357 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3353543424}
I20260709 12:21:31.005965 31744 raft_consensus.cc:2410] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:21:31.040683 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 569) took 1442 ms. Trace:
I20260709 12:21:31.041103 32375 rpcz_store.cc:276] 0709 12:21:29.598409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.598553 (+   144us) service_pool.cc:224] Handling call
0709 12:21:30.630611 (+1032058us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:30.670711 (+ 40100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.040478 (+369767us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54004272d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:31.040619 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2372416384}
I20260709 12:21:31.047341  1740 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:31.039727 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 262) took 2268 ms. Trace:
I20260709 12:21:31.048514 31678 rpcz_store.cc:276] 0709 12:21:28.771434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:28.771597 (+   163us) service_pool.cc:224] Handling call
0709 12:21:28.788645 (+ 17048us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:29.524429 (+735784us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.039350 (+1514921us) spinlock_profiling.cc:243] Waited 710 ms on lock 0x7b54003921d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:31.039622 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1632451968}
W20260709 12:21:31.049228 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.050585 31753 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:30.899521  1627 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:30.901173  1762 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:31.053267 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46281.6us
I20260709 12:21:31.058795 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:31.059737 31678 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:31.047739  1757 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:30.937930  1729 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:31.023185 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:30.924211  1579 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:31.053584  1748 raft_consensus.cc:2764] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:21:31.013154  1586 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:31.216389 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 127067us
I20260709 12:21:31.012601  1756 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.219051  1756 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:31.068514 31674 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
W20260709 12:21:31.223217 31642 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.566s (SENT) [suppressed 14 similar messages]
W20260709 12:21:31.224650 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 160006us
I20260709 12:21:31.225977  1757 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:31.229332  1735 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:31.231184  1735 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:31.232482  1399 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:31.233166  1399 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:31.075953 31824 raft_consensus.cc:2410] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:21:31.237135 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:30.913916  1415 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:31.071177 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:31.238621 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:31.047382  1387 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:31.014269  1576 consensus_queue.cc:1175] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 4 status { last_received { term: 4 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 4 index: 5 } }
I20260709 12:21:31.084666 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:31.276676 31821 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.017s	sys 0.032s
I20260709 12:21:31.239151 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:31.264662 31675 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
W20260709 12:21:31.134500 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 75763.1us
I20260709 12:21:31.263927 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:31.251001 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.060523  1722 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.059831  1721 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.265180 31824 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
W20260709 12:21:31.216354 31793 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.026s	sys 0.045s
W20260709 12:21:31.369118 31793 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.026s	sys 0.045s
I20260709 12:21:31.087455  1782 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:31.370101 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.216904  1754 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.266556 31678 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:31.216513  1752 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:31.275511 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:31.056900 31823 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
W20260709 12:21:31.324530  1445 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 2.093s	user 0.017s	sys 0.043s
I20260709 12:21:31.358373 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:31.278585 31821 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.017s	sys 0.032s
I20260709 12:21:31.278846 31753 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:30.941696  1758 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:31.359676 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:31.396133 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:31.070749  1788 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:31.060575  1194 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:31.397114 31673 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:31.071211  1763 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:31.240633 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.410125 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:31.375613 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:31.411379 32369 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:31.371785 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:31.412943 31674 raft_consensus.cc:2410] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 6 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260709 12:21:31.227414 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.005940  1720 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:31.240800  1681 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.239996  1756 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:31.390643 31826 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:21:31.239523 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:31.452054 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:31.455796 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:31.257613  1399 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:31.283250  1820 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:31.391004  1445 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 2.161s	user 0.019s	sys 0.046s
I20260709 12:21:31.409274 31675 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260709 12:21:31.426873 31750 pending_rounds.cc:85] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 5
I20260709 12:21:31.029326  1797 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:31.375788  1734 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.724s	user 0.015s	sys 0.028s
I20260709 12:21:31.419763 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:31.282866  1586 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:31.526659 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 64160.5us
I20260709 12:21:31.527477 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:31.366950  1722 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:31.529033 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:31.530933 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:31.532832 31669 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:31.530506 32374 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:31.548026 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:31.536335 31819 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.870s	user 0.009s	sys 0.024s
W20260709 12:21:31.549095 31819 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.883s	user 0.009s	sys 0.024s
W20260709 12:21:31.550083 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 326878us
I20260709 12:21:31.532984 31750 pending_rounds.cc:107] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260709 12:21:31.544646 31753 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 6: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:31.552325 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:31.533497 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:31.553321 31747 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:31.559481 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:31.535137 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:31.551131 31750 raft_consensus.cc:2889] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:31.453516  1763 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:31.598418 31669 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:31.529289 31675 pending_rounds.cc:85] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 2
I20260709 12:21:31.635231 31675 pending_rounds.cc:107] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 4.3
I20260709 12:21:31.635636 31675 raft_consensus.cc:2889] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:31.396304  1735 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.658764 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:31.659775 31747 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:31.659720 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:31.661126 31669 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:31.525542  1772 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:31.535831  1764 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.748s	user 0.013s	sys 0.020s
I20260709 12:21:31.641019 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:31.532009  1734 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.906s	user 0.015s	sys 0.029s
I20260709 12:21:31.536357  1727 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:31.654592 31744 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:31.403064  1723 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.151s
W20260709 12:21:31.549886  1786 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.010s	sys 0.024s
I20260709 12:21:31.388952  1721 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:31.452675  1788 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:31.643796 31678 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:31.551131 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:31.663093 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:31.692170 31746 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:31.666829  1764 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.881s	user 0.013s	sys 0.023s
I20260709 12:21:31.546095  1729 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.664489 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:31.712828 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.397333  1782 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:31.391012  1754 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:31.420109  1740 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.408542  1752 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:31.642819  1771 raft_consensus.cc:2764] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:21:31.764632  1613 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:31.765340  1817 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:31.765928  1817 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:31.766324  1817 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:31.768685 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:31.769403  1833 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:21:31.769608 31674 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:31.772569 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:31.774259 31673 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260709 12:21:31.775224  1823 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.245s	user 0.000s	sys 0.000s
I20260709 12:21:31.663342 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:31.527307  1681 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:31.734149  1751 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:31.804018 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1698) took 2314 ms. Trace:
W20260709 12:21:31.746620  1816 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.339s	user 0.000s	sys 0.001s
I20260709 12:21:31.711875 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:31.666280  1808 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:21:31.687220  1786 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.011s	sys 0.025s
I20260709 12:21:31.739463 32369 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:31.740706 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:31.742208 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:31.810524 31669 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:31.671841  1727 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:31.527606  1820 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:31.671180  1772 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:31.813810 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:31.718271  1720 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:31.779744 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.780055 31826 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:31.779945 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:31.544394  1399 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.135s
I20260709 12:21:31.788426 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:31.826125 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:31.670696  1735 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:31.665303 31671 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:21:31.804503 31822 rpcz_store.cc:276] 0709 12:21:29.490012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.497638 (+  7626us) service_pool.cc:224] Handling call
0709 12:21:29.497956 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:30.613764 (+1115808us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:30.613807 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:30.637277 (+ 23470us) log.cc:844] Serialized 26 byte log entry
0709 12:21:31.049999 (+412722us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:31.050036 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.267944 (+217908us) spinlock_profiling.cc:243] Waited 35.8 ms on lock 0x7b54003c96d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:31.268005 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:31.779659 (+511654us) raft_consensus.cc:1685] finished
0709 12:21:31.779683 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:31.803921 (+ 24238us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12359,"raft.queue_time_us":127756,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":5351,"spinlock_wait_cycles":82327808,"thread_start_us":1449398,"threads_started":2,"wal-append.queue_time_us":615025}
I20260709 12:21:31.813649 31675 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:21:31.796873  1762 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:31.817792 31672 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
W20260709 12:21:31.822913 31718 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.031s	sys 0.057s
I20260709 12:21:31.831833 32376 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 6: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:31.836663 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:31.837648 31718 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.031s	sys 0.058s
W20260709 12:21:31.838575 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 557512us
I20260709 12:21:31.839069 31822 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:31.840498 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:31.734874  1729 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:31.864480 31671 raft_consensus.cc:606] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:31.900096 31671 raft_consensus.cc:614] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:31.842943 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:31.901854 31670 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:31.893698 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:31.755684  1740 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:31.842912 32374 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
W20260709 12:21:31.899183 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 623678us
I20260709 12:21:31.844211 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:31.933529 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:31.935156 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 91319us
I20260709 12:21:31.935170 31823 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 2.
I20260709 12:21:31.954999 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:31.955776 31826 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
W20260709 12:21:31.955967 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:31.902977 31822 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:31.958408 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:31.830309  1704 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.827662  1627 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:31.833843  1756 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:31.964578 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:31.970306 31822 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
W20260709 12:21:31.931878  1821 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.555s	user 0.000s	sys 0.002s
I20260709 12:21:31.974781 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:31.994087 31747 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 4 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:21:31.990304 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:32.007004 31823 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:31.976889 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.074412 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.075223 31747 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:32.074604 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1785) took 1202 ms. Trace:
W20260709 12:21:32.075076 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 523979us
I20260709 12:21:32.077876 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:32.079299 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:32.082847 31823 raft_consensus.cc:2410] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
W20260709 12:21:31.962136  1370 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.965s	user 0.008s	sys 0.003s
I20260709 12:21:32.075831 31752 rpcz_store.cc:276] 0709 12:21:30.872519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.877190 (+  4671us) service_pool.cc:224] Handling call
0709 12:21:30.877510 (+   320us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:30.887271 (+  9761us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:30.887291 (+    20us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:30.902868 (+ 15577us) log.cc:844] Serialized 26 byte log entry
0709 12:21:31.012336 (+109468us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:31.017758 (+  5422us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.017905 (+   147us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:32.074387 (+1056482us) raft_consensus.cc:1685] finished
0709 12:21:32.074411 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.074549 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6191,"raft.queue_time_us":6314,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":122,"thread_start_us":85300,"threads_started":1,"wal-append.queue_time_us":451755}
I20260709 12:21:32.096653 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:32.101956 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:32.110996 31749 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:32.111244 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.112229 31753 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:32.112417 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:32.113288 31749 raft_consensus.cc:2410] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:32.113432 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:32.114288 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:31.961638  1810 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.116392 31749 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:32.119256 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:31.962452  1613 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:32.115783 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 239) took 1201 ms. Trace:
I20260709 12:21:32.200181 31820 rpcz_store.cc:276] 0709 12:21:30.914915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.915093 (+   178us) service_pool.cc:224] Handling call
0709 12:21:30.935825 (+ 20732us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.964317 (+ 28492us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:31.083846 (+119529us) log.cc:844] Serialized 14 byte log entry
0709 12:21:31.788484 (+704638us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:31.788567 (+    83us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:31.788648 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.115516 (+326868us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x7b54001e9bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:32.115568 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.115705 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":533099,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":8537,"raft.queue_time_us":165390,"spinlock_wait_cycles":194636160,"thread_start_us":823292,"threads_started":3,"wal-append.queue_time_us":734828}
I20260709 12:21:32.126459  1820 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:31.984242  1756 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:31.964246  1579 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:32.127339 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.127853  1740 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:32.129685 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:32.212297 31818 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.687s	user 0.017s	sys 0.055s
W20260709 12:21:32.212822 31818 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.687s	user 0.017s	sys 0.055s
I20260709 12:21:32.212671 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:32.132771  1797 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:32.117790  1730 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.140571  1800 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:32.140831  1824 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.689s	user 0.001s	sys 0.000s
I20260709 12:21:32.141145  1780 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.115602  1841 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.058s
I20260709 12:21:32.126505  1763 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:32.131573  1681 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:31.991160  1704 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:32.147841  1725 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.186s	user 0.008s	sys 0.004s
I20260709 12:21:32.149718  1803 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:32.149938 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1699) took 2498 ms. Trace:
I20260709 12:21:32.241151 31827 rpcz_store.cc:276] 0709 12:21:29.688869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.855888 (+1167019us) service_pool.cc:224] Handling call
0709 12:21:30.856229 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:30.861164 (+  4935us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:21:30.861196 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:30.861713 (+   517us) log.cc:844] Serialized 26 byte log entry
0709 12:21:31.044121 (+182408us) spinlock_profiling.cc:243] Waited 3.28 ms on lock 0x7b4800130158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:21:31.361039 (+316918us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:31.361081 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.361145 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:32.149493 (+788348us) raft_consensus.cc:1685] finished
0709 12:21:32.149527 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.149847 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4621,"raft.run_cpu_time_us":2536,"raft.run_wall_time_us":178857,"spinlock_wait_cycles":7552768,"thread_start_us":423484,"threads_started":1,"wal-append.queue_time_us":1191426}
W20260709 12:21:32.151644  1779 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.217s	user 0.005s	sys 0.008s
I20260709 12:21:32.160212  1788 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:32.162375 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:32.163443  1830 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.561s	user 0.001s	sys 0.000s
I20260709 12:21:32.165926 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.250650 31752 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:32.166457  1722 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:31.984473  1758 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:32.101447  1727 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.171298  1754 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:32.171988  1613 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.179390 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1792) took 1228 ms. Trace:
I20260709 12:21:32.180239  1810 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:32.121696  1818 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:32.184561 31747 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:32.116829 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:32.259303 31745 rpcz_store.cc:276] 0709 12:21:30.950972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.975690 (+ 24718us) service_pool.cc:224] Handling call
0709 12:21:30.976189 (+   499us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:30.977027 (+   838us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:30.977069 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:30.977737 (+   668us) log.cc:844] Serialized 26 byte log entry
0709 12:21:31.420036 (+442299us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:31.747018 (+326982us) log.cc:844] Serialized 14 byte log entry
0709 12:21:31.753115 (+  6097us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.753218 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:32.179146 (+425928us) raft_consensus.cc:1685] finished
0709 12:21:32.179169 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.179319 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5621,"prepare.queue_time_us":694790,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":5906,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":222,"thread_start_us":718796,"threads_started":2,"wal-append.queue_time_us":1198822}
I20260709 12:21:32.113202  1301 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:32.131717  1721 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:32.131502 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:32.213402 31823 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:32.227655 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:32.231606 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:32.330920 32341 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.124s	sys 0.194s
W20260709 12:21:32.350282 32341 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.124s	sys 0.194s
I20260709 12:21:32.249565  1803 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.341322 31826 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:32.363278 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 231866us
I20260709 12:21:32.236564 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:32.254563 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:32.365553 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 111046us
W20260709 12:21:32.238507  1594 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.242s	user 0.010s	sys 0.006s
I20260709 12:21:32.263365 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:32.375306 31827 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:32.375829 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:32.270887  1842 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:32.379776 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.380714 31747 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:32.263512 31744 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:32.317334 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.233942  1730 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.278723  1613 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:32.280683 31753 raft_consensus.cc:2368] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
I20260709 12:21:32.230973  1800 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:32.382529 31752 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:32.236649  1780 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:21:32.329418  1812 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.012s	sys 0.036s
I20260709 12:21:32.253291  1725 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.263705  1627 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.107s
I20260709 12:21:32.367646  1782 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:32.262367  1841 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.115s
I20260709 12:21:32.358410  1803 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:32.334012  1818 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.415575 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.379118  1842 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:32.394506 31752 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:32.359617  1729 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:32.378353  1735 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:32.425601 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.425797 31744 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 2.
I20260709 12:21:32.502372 31753 raft_consensus.cc:2410] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
W20260709 12:21:32.384923  1757 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.156s	user 0.020s	sys 0.025s
I20260709 12:21:32.377514 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1706) took 1364 ms. Trace:
I20260709 12:21:32.376936  1844 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:32.541815  1750 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:32.376601  1851 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:32.415400 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:32.450481 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:32.547434 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 97051.2us
I20260709 12:21:32.548539 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:32.549437 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:32.549889 31746 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.834s	user 0.014s	sys 0.000s
I20260709 12:21:32.550285 31746 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:32.552748 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 289) took 1001 ms. Trace:
I20260709 12:21:32.553381 31747 raft_consensus.cc:2410] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 7 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:21:32.556414 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40834 (request call id 12) took 1006 ms. Trace:
I20260709 12:21:32.556912 31671 rpcz_store.cc:276] 0709 12:21:31.550481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.609301 (+ 58820us) service_pool.cc:224] Handling call
0709 12:21:32.556164 (+946863us) spinlock_profiling.cc:243] Waited 36.1 ms on lock 0x7b540022b354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdcf853 00007fe901d9d60a 00007fe8fb3237d4 00007fe8fb323762 00007fe8fb323691 00007fe8fb32360c 00007fe8fb3228c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:32.556298 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":82938240}
I20260709 12:21:32.561422 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 285) took 1135 ms. Trace:
I20260709 12:21:32.450920 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:32.561898 31750 rpcz_store.cc:276] 0709 12:21:31.425796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.426027 (+   231us) service_pool.cc:224] Handling call
0709 12:21:31.426515 (+   488us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:31.702582 (+276067us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:31.702656 (+    74us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:31.703272 (+   616us) log.cc:844] Serialized 26 byte log entry
0709 12:21:32.256010 (+552738us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:32.256057 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.256130 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:32.561169 (+305039us) raft_consensus.cc:1685] finished
0709 12:21:32.561191 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.561353 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":474381,"threads_started":1,"wal-append.queue_time_us":688900}
I20260709 12:21:32.564445 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:32.573002 31827 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
W20260709 12:21:32.577420 32403 heartbeater.cc:646] Failed to heartbeat to 127.29.201.254:37495 (0 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.29.201.254:37495 timed out after 15.000s (SENT)
I20260709 12:21:32.500236 31823 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
W20260709 12:21:32.394341  1812 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.012s	sys 0.037s
I20260709 12:21:32.527591 31820 rpcz_store.cc:276] 0709 12:21:31.013027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.211678 (+1198651us) service_pool.cc:224] Handling call
0709 12:21:32.377442 (+165764us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:32.579571 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 128728us
I20260709 12:21:32.383232  1594 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.530145 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:32.538030  1415 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:32.538754 31752 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
W20260709 12:21:32.539256  1850 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.273s	user 0.000s	sys 0.001s
I20260709 12:21:32.540194 31753 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:21:32.563201  1762 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:32.576174  1301 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.576419 31750 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:21:32.576326 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:32.581532 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:32.593583 31823 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:32.580338  1851 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:32.597402 31749 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:32.590545 31641 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.001s	sys 0.013s
I20260709 12:21:32.582089 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1710) took 1525 ms. Trace:
I20260709 12:21:32.551188  1735 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:21:32.527312  1757 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.299s	user 0.021s	sys 0.026s
I20260709 12:21:32.588104 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:32.601176 31820 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:32.562810  1544 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:32.594154 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1858) took 1198 ms. Trace:
I20260709 12:21:32.602257 31823 rpcz_store.cc:276] 0709 12:21:31.401713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.579523 (+1177810us) service_pool.cc:224] Handling call
0709 12:21:32.594074 (+ 14551us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:32.594709 31676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.731s	user 0.036s	sys 0.101s
W20260709 12:21:32.604118 31676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.036s	sys 0.101s
I20260709 12:21:32.598814 31827 rpcz_store.cc:276] 0709 12:21:31.056825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.243173 (+1186348us) service_pool.cc:224] Handling call
0709 12:21:32.581703 (+338530us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.481935 31826 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:32.581359 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:32.604909 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.605336 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1860) took 1075 ms. Trace:
I20260709 12:21:32.605609 31747 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:32.605695 31826 rpcz_store.cc:276] 0709 12:21:31.540532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.212993 (+672461us) service_pool.cc:224] Handling call
0709 12:21:32.605275 (+392282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.606851  1844 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:32.607306  1844 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:32.607702  1844 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:32.606274 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:32.611560  1560 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:32.611720 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:32.612022  1560 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.612411  1560 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:32.612573  1843 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:32.612813 31791 leader_election.cc:400] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
I20260709 12:21:32.613229 31791 leader_election.cc:403] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:21:32.613178  1843 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:32.613838  1843 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:32.614118 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:32.614821  1723 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:32.615337 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 55281.8us
W20260709 12:21:32.598150 31641 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.001s	sys 0.013s
I20260709 12:21:32.583624  1724 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:32.584065  1756 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:32.589704  1762 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:32.592270 31821 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.201s	user 0.003s	sys 0.011s
W20260709 12:21:32.628942 31821 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.238s	user 0.003s	sys 0.011s
W20260709 12:21:32.622390 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:32.626852 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.078s	user 0.005s	sys 0.009s
I20260709 12:21:32.627054 31827 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:32.631639 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1852) took 1371 ms. Trace:
I20260709 12:21:32.632054 31827 rpcz_store.cc:276] 0709 12:21:31.260397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.626446 (+1366049us) service_pool.cc:224] Handling call
0709 12:21:32.626878 (+   432us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.631268 (+  4390us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.631566 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:32.595610  1786 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.703s	user 0.004s	sys 0.008s
W20260709 12:21:32.623126 31644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.837s	user 0.001s	sys 0.013s
I20260709 12:21:32.629910 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:32.623664 31819 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.074s	user 0.011s	sys 0.004s
W20260709 12:21:32.623441  1445 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.904s	user 0.001s	sys 0.012s
I20260709 12:21:32.597831  1301 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:32.600488  1735 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.623682  1851 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:32.634564  1762 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:32.628580  1758 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:32.625628  1841 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:32.637969  1544 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:21:32.623572  1751 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.757s	user 0.011s	sys 0.011s
W20260709 12:21:32.623786  1768 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.591s	user 0.003s	sys 0.013s
W20260709 12:21:32.640375 31819 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.091s	user 0.011s	sys 0.004s
W20260709 12:21:32.640746 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.092s	user 0.005s	sys 0.009s
W20260709 12:21:32.710909 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 1.16288e+06us
I20260709 12:21:32.553366 31746 rpcz_store.cc:276] 0709 12:21:31.551480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.551624 (+   144us) service_pool.cc:224] Handling call
0709 12:21:32.552679 (+1001055us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:32.661967 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:32.713109 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.672590 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.713310 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.714684 31746 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:32.638803 31644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.853s	user 0.001s	sys 0.013s
W20260709 12:21:32.638561  1786 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.775s	user 0.005s	sys 0.009s
W20260709 12:21:32.660674  1855 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260709 12:21:32.755618 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 125782us
W20260709 12:21:32.710485  1751 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.875s	user 0.012s	sys 0.011s
I20260709 12:21:32.638926  1704 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:32.761009 31676 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:21:32.671684  1445 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.965s	user 0.002s	sys 0.013s
I20260709 12:21:32.674839  1788 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:32.682739  1735 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:32.768448 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.684283  1810 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:32.814205 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:32.815570 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:32.722234  1863 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:32.772404 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:32.594651  1842 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:32.689385  1724 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:32.773792 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.822045  1724 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:32.822284 31753 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:32.822564  1724 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:32.823405 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.824075 31753 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:32.825066 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.825405 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:32.825608 31753 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
W20260709 12:21:32.826360  1847 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.279s	user 0.001s	sys 0.000s
I20260709 12:21:32.826893 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:32.828071 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.828504  1594 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:32.828629 31746 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:32.829576 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:32.830018  1720 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.003s	user 0.012s	sys 0.000s
I20260709 12:21:32.830121 31746 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:32.698184  1758 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.831449  1782 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.831934  1782 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.832929  1758 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:32.833235  1301 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:32.826495 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.836234 31753 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:21:32.836736 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 1.23975e+06us
I20260709 12:21:32.698678  1841 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.837051 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:32.837926 31753 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 7 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:32.837852 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:32.764380  1751 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:32.709807  1800 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:32.714179  1730 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:32.766935  1722 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260709 12:21:32.660108 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:32.840229 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 244) took 1873 ms. Trace:
I20260709 12:21:32.781237 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:32.848747 31671 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:32.840983  1863 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:32.849311  1544 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.851676  1863 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:32.696666  1729 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:32.860545  1851 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.861517  1729 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:32.708766  1768 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.676s	user 0.003s	sys 0.013s
I20260709 12:21:32.738670  1634 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.745028  1780 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:21:32.872678  1869 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:32.873354  1869 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.875912 31752 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:21:32.880713  1762 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.883039  1724 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.885440  1724 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:32.871846  1723 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
I20260709 12:21:32.888929 31746 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:21:32.839675  1751 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.838951  1841 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:32.894904 31751 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:32.896100  1797 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:32.896297  1843 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.695165  1560 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:32.898342  1586 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:32.773975 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:32.899035  1634 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:32.858350  1817 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.091s	user 0.012s	sys 0.000s
I20260709 12:21:32.900877  1817 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:32.901927  1634 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:32.838619  1613 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.843969 32370 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:32.903358 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1779) took 1372 ms. Trace:
I20260709 12:21:32.903721 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:32.858775  1725 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:32.639918  1844 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:32.859345  1756 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:32.860086 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 277) took 2250 ms. Trace:
I20260709 12:21:32.860961 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 425) took 1219 ms. Trace:
W20260709 12:21:32.847818 32377 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 568) took 3287 ms (client timeout 3293 ms). Trace:
I20260709 12:21:32.848270 31818 rpcz_store.cc:276] 0709 12:21:30.966915 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.967064 (+   149us) service_pool.cc:224] Handling call
0709 12:21:30.967423 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:30.967853 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:31.396007 (+428154us) log.cc:844] Serialized 14 byte log entry
0709 12:21:32.238849 (+842842us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:32.247545 (+  8696us) log.cc:844] Serialized 26 byte log entry
0709 12:21:32.247763 (+   218us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:32.247817 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.247918 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:32.840030 (+592112us) raft_consensus.cc:1685] finished
0709 12:21:32.840049 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.840177 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34202,"prepare.queue_time_us":563640,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":8643,"raft.queue_time_us":21641,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":18112,"thread_start_us":1129418,"threads_started":2,"wal-append.queue_time_us":922084}
I20260709 12:21:32.846992 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:32.858131  1840 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.897s	user 0.004s	sys 0.000s
I20260709 12:21:32.896687  1797 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:32.771255 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:32.904392 31748 raft_consensus.cc:2368] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
W20260709 12:21:32.942894 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:32.908769 31824 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1714) took 1644 ms (client timeout 1630 ms). Trace:
I20260709 12:21:33.186797 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:33.117393 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 503298us
I20260709 12:21:32.971690 31753 pending_rounds.cc:85] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 1
I20260709 12:21:33.187348 31677 rpcz_store.cc:276] 0709 12:21:30.609361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.609697 (+   336us) service_pool.cc:224] Handling call
0709 12:21:30.610135 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.966279 (+356144us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:30.966334 (+    55us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:30.966393 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:30.966442 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:32.859723 (+1893281us) spinlock_profiling.cc:243] Waited 356 ms on lock 0x7b54001824d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:32.859790 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:32.859996 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":818386688}
I20260709 12:21:32.863698  1729 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:32.882495 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.663161 31747 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:33.663766 31753 pending_rounds.cc:107] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 5.2
I20260709 12:21:33.664067 31753 raft_consensus.cc:2889] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:33.502002 31718 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.000s	sys 0.032s
W20260709 12:21:33.484442 32369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 436) took 2072 ms (client timeout 1975 ms). Trace:
W20260709 12:21:33.665053 31718 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.856s	user 0.000s	sys 0.043s
W20260709 12:21:33.665414 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 856545us
W20260709 12:21:33.503321 31826 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.817s	user 0.000s	sys 0.027s
I20260709 12:21:33.186299 32376 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:33.483573 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 233) took 3901 ms. Trace:
W20260709 12:21:33.411194 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.667568 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:33.668152 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 430) took 1060 ms. Trace:
I20260709 12:21:33.668538 31669 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
W20260709 12:21:33.188551 32377 rpcz_store.cc:269] 0709 12:21:29.560902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:29.561089 (+   187us) service_pool.cc:224] Handling call
0709 12:21:32.847575 (+3286486us) spinlock_profiling.cc:243] Waited 2.37 s on lock 0x7b54004c4354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:32.847746 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5454979968}
W20260709 12:21:33.670552 31824 rpcz_store.cc:269] 0709 12:21:31.263993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.908492 (+1644499us) service_pool.cc:211] Skipping call since client already timed out
0709 12:21:32.908655 (+   163us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:21:33.207823  1560 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:33.299850 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:32.841540  1627 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.831s
I20260709 12:21:33.672228 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:33.673146 31824 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:33.284416 31818 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260709 12:21:33.173259 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.012s	sys 0.009s
W20260709 12:21:33.674782 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.013s	user 0.024s	sys 0.021s
I20260709 12:21:33.668555 32376 rpcz_store.cc:276] 0709 12:21:32.607862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.814103 (+206241us) service_pool.cc:224] Handling call
0709 12:21:33.668087 (+853984us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:33.264020 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:33.245846 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:33.671095 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:33.042424 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1850) took 3524 ms. Trace:
I20260709 12:21:33.186151 32374 rpcz_store.cc:276] 0709 12:21:31.642047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.658928 (+ 16881us) service_pool.cc:224] Handling call
0709 12:21:32.860777 (+1201849us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b540055da54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:32.860900 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":298420480}
I20260709 12:21:33.676548 31669 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:32.981945  1818 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:33.145345  1862 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
W20260709 12:21:33.347186 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 106098us
I20260709 12:21:33.669472 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:33.483814 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1788) took 1716 ms. Trace:
I20260709 12:21:33.676508 31675 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
W20260709 12:21:33.659668 31671 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.810s	user 0.028s	sys 0.022s
W20260709 12:21:33.509877 31827 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.014s	sys 0.012s
W20260709 12:21:33.665221 32369 rpcz_store.cc:269] 0709 12:21:31.412250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.414332 (+  2082us) service_pool.cc:224] Handling call
0709 12:21:33.484177 (+2069845us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54004c5c54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:33.484351 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":448571136}
I20260709 12:21:33.257457 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:33.185484 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 331098us
W20260709 12:21:33.667348 31826 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.981s	user 0.003s	sys 0.031s
I20260709 12:21:32.903733 31678 rpcz_store.cc:276] 0709 12:21:31.531420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.531614 (+   194us) service_pool.cc:224] Handling call
0709 12:21:32.903159 (+1371545us) spinlock_profiling.cc:243] Waited 43.5 ms on lock 0x7b540036f1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:32.903269 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99953536}
I20260709 12:21:33.677227 31819 rpcz_store.cc:276] 0709 12:21:29.517844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.652696 (+1134852us) service_pool.cc:224] Handling call
0709 12:21:30.653386 (+   690us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.653619 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:21:31.549428 (+895809us) log.cc:844] Serialized 14 byte log entry
0709 12:21:33.041517 (+1492089us) log.cc:844] Serialized 14 byte log entry
0709 12:21:33.041856 (+   339us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:33.041931 (+    75us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:33.042010 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.042114 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:33.042320 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78422,"prepare.queue_time_us":1461602,"prepare.run_cpu_time_us":152,"prepare.run_wall_time_us":3923,"raft.queue_time_us":124451,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":19510,"thread_start_us":2321874,"threads_started":3,"wal-append.queue_time_us":1224029}
I20260709 12:21:33.685344 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:33.508059  1851 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.012s	sys 0.016s
W20260709 12:21:33.610024  1861 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 12:21:32.843466  1772 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:33.248943 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:33.748184 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:33.748416 31827 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.908s	user 0.014s	sys 0.028s
W20260709 12:21:33.756989 31670 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1786) took 1990 ms (client timeout 1513 ms). Trace:
I20260709 12:21:33.752244 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:33.248754  1849 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:33.241459  1803 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.738s	user 0.005s	sys 0.006s
W20260709 12:21:33.757536 31670 rpcz_store.cc:269] 0709 12:21:31.766569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.804756 (+ 38187us) service_pool.cc:224] Handling call
0709 12:21:33.756799 (+1952043us) spinlock_profiling.cc:243] Waited 74.7 ms on lock 0x7b540022b354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:33.756914 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":171917184}
W20260709 12:21:33.467742  1786 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.686s	user 0.005s	sys 0.029s
I20260709 12:21:33.189991  1727 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:33.739372 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1868) took 1767 ms. Trace:
W20260709 12:21:33.743052 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44000.5us
I20260709 12:21:33.775043 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:33.241695  1780 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.775892 31819 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 3.
I20260709 12:21:32.905790  1613 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:33.421087  1757 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.256683  1866 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.757952 31824 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:33.784201 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:33.744537  1560 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:33.116707  1817 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:33.670011 31821 rpcz_store.cc:276] 0709 12:21:29.582045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.716774 (+1134729us) service_pool.cc:224] Handling call
0709 12:21:30.717133 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:30.717350 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:31.390834 (+673484us) log.cc:844] Serialized 14 byte log entry
0709 12:21:33.287952 (+1897118us) log.cc:844] Serialized 14 byte log entry
0709 12:21:33.482581 (+194629us) log.cc:844] Serialized 14 byte log entry
0709 12:21:33.483061 (+   480us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:33.483137 (+    76us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:33.483209 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.483305 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:33.483488 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1462826,"prepare.run_cpu_time_us":149,"prepare.run_wall_time_us":2938,"raft.queue_time_us":193164,"raft.run_cpu_time_us":469,"raft.run_wall_time_us":141257,"thread_start_us":2569080,"threads_started":4,"wal-append.queue_time_us":1454497}
I20260709 12:21:33.790812 32376 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260709 12:21:33.792028 31676 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:21:33.770426  1786 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.001s	user 0.011s	sys 0.030s
I20260709 12:21:33.752844  1818 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:21:33.748932 31674 rpcz_store.cc:276] 0709 12:21:31.767309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.770278 (+  2969us) service_pool.cc:224] Handling call
0709 12:21:31.770579 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.861792 (+ 91213us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:31.956820 (+ 95028us) log.cc:844] Serialized 14 byte log entry
0709 12:21:32.453232 (+496412us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:32.453313 (+    81us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:32.453380 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.483466 (+1030086us) spinlock_profiling.cc:243] Waited 75.1 ms on lock 0x7b54001810d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:33.483531 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:33.483718 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":94267,"prepare.queue_time_us":100244,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":51236,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":17462,"spinlock_wait_cycles":172758784,"thread_start_us":495696,"threads_started":1,"wal-append.queue_time_us":783254}
I20260709 12:21:33.783818  1866 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:33.431123  1762 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:33.764323 31669 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:33.291600  1723 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:32.904409 32370 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:33.748178 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:33.758975  1725 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260709 12:21:33.247833  1782 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:33.334481  1735 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:33.747537  1851 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.884s	user 0.018s	sys 0.022s
I20260709 12:21:33.776042 31818 rpcz_store.cc:276] 0709 12:21:31.971860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.188015 (+1216155us) service_pool.cc:224] Handling call
0709 12:21:33.261275 (+ 73260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:33.674417 (+413142us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.739255 (+ 64838us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12544}
I20260709 12:21:33.681242  1841 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20260709 12:21:33.749008 31671 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:33.741954 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:33.751566  1544 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.788156  1877 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 12:21:33.832079  1735 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:33.819641  1723 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:33.834556  1752 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:33.836081  1723 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:33.776942  1876 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.776494  1772 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:33.816444 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 68355.8us
I20260709 12:21:33.839828 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:33.841449 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:33.820719 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 280) took 1103 ms. Trace:
W20260709 12:21:33.822868 31641 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.161s	user 0.024s	sys 0.050s
I20260709 12:21:33.771046  1875 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
W20260709 12:21:33.843073 31641 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.181s	user 0.024s	sys 0.050s
I20260709 12:21:33.804301  1613 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:33.777314  1849 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:33.818259  1843 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:33.797076  1757 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:33.819644  1817 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:33.822531 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 433) took 1012 ms. Trace:
I20260709 12:21:33.830554  1841 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.830348  1782 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:21:33.833864 32371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 438) took 2414 ms (client timeout 2093 ms). Trace:
I20260709 12:21:33.842820 31671 rpcz_store.cc:276] 0709 12:21:32.717565 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.739762 (+ 22197us) service_pool.cc:224] Handling call
0709 12:21:33.820631 (+1080869us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:33.885830 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:33.886786 31671 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:33.773473  1803 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:33.801533  1818 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:33.813714 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:33.842377 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:33.895143 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52837.8us
I20260709 12:21:33.895790 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:33.896628 31671 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:33.846704 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:33.896951 31750 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.020s	sys 0.041s
I20260709 12:21:33.851856 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:33.897416 31819 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:33.898335 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:33.899776 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:33.900729 31669 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:33.853001 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:33.855333  1879 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:21:33.856158 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:33.802217  1560 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:33.911595 31752 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.014s	sys 0.051s
W20260709 12:21:33.913028 31752 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.014s	sys 0.051s
I20260709 12:21:33.701489 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:33.828301  1780 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:33.640805  1878 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:33.612061  1869 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:33.812216  1445 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:21:33.878343 32370 rpcz_store.cc:276] 0709 12:21:32.809956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.903422 (+ 93466us) service_pool.cc:224] Handling call
0709 12:21:33.819855 (+916433us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:33.889273 32369 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:33.857568  1852 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:33.889780  1841 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:33.894126 31676 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:33.896320 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:21:33.897397 31750 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.610s	user 0.020s	sys 0.041s
I20260709 12:21:33.907524 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:33.904722 31818 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:33.913300 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:33.898255 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:33.845789  1544 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:33.863662 31747 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:33.921705 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1804) took 1314 ms. Trace:
I20260709 12:21:33.859187  1613 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:33.904757  1716 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:33.858057  1876 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:33.979892 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:33.899571  1818 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:21:33.992121 31669 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:33.982045 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 280675us
I20260709 12:21:33.991709 31674 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:33.993302 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1796) took 1415 ms. Trace:
I20260709 12:21:33.997336 31747 rpcz_store.cc:276] 0709 12:21:32.578157 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.721386 (+143229us) service_pool.cc:224] Handling call
0709 12:21:33.993178 (+1271792us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:33.998517  1716 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:33.997478 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:33.997658 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.003664 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:34.004490 31818 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:33.992712 31671 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:34.006963 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:34.007555 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.008420 31676 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:33.997291 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:33.981933  1844 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:33.891428 32371 rpcz_store.cc:269] 0709 12:21:31.419825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.470062 (+ 50237us) service_pool.cc:224] Handling call
0709 12:21:33.681126 (+2211064us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:33.833754 (+152628us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29769472}
I20260709 12:21:33.987820  1852 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.014771 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:34.016299 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:33.995577  1869 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.021901 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.001351  1716 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:33.985172  1878 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.023963 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:34.019713 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.029711 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1853) took 2768 ms. Trace:
I20260709 12:21:34.052243 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1851) took 2792 ms. Trace:
W20260709 12:21:34.155792 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46560.7us
I20260709 12:21:33.988644  1884 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:34.028939 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:34.157217 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:34.154856 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1806) took 1546 ms. Trace:
I20260709 12:21:34.156497 31823 rpcz_store.cc:276] 0709 12:21:31.259641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.624828 (+1365187us) service_pool.cc:224] Handling call
0709 12:21:32.628632 (+  3804us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:32.726386 (+ 97754us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:32.726440 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:32.768033 (+ 41593us) log.cc:844] Serialized 26 byte log entry
0709 12:21:33.264480 (+496447us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:33.264526 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.264595 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:34.051972 (+787377us) raft_consensus.cc:1685] finished
0709 12:21:34.051994 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.052156 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":201710,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"thread_start_us":415715,"threads_started":1,"wal-append.queue_time_us":1208178}
I20260709 12:21:34.156435 31826 rpcz_store.cc:276] 0709 12:21:31.261054 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.624916 (+1363862us) service_pool.cc:224] Handling call
0709 12:21:32.625239 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.625469 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:33.748409 (+1122940us) log.cc:844] Serialized 14 byte log entry
0709 12:21:34.029243 (+280834us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:34.029319 (+    76us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:34.029392 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.029481 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.029638 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":916349,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":8600,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":1498,"thread_start_us":1335569,"threads_started":2,"wal-append.queue_time_us":309607}
W20260709 12:21:34.157946 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 129090us
I20260709 12:21:34.031857 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.044991 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1712) took 2983 ms. Trace:
I20260709 12:21:34.162231 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.163287 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.190074  1751 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:34.204936  1751 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.025122  1627 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:34.199702  1907 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:34.202921 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:34.064889  1866 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.063896  1841 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:34.064509  1757 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:34.202940 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.204114 31747 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:34.028724  1735 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:34.231036  1735 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.231575  1735 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:34.232897 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.233547 31819 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:34.234476 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:34.013599  1759 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.235074 31819 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.235451  1759 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.035988  1399 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:34.238252 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.206781 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.240317 31749 raft_consensus.cc:2410] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 5 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:34.142212 31818 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:34.070253  1544 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.162448  1884 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.193289 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.156978 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:21:34.251364 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 94465.6us
I20260709 12:21:34.252293 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:34.076943 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:33.993214 31752 rpcz_store.cc:276] 0709 12:21:32.607230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.876996 (+269766us) service_pool.cc:224] Handling call
0709 12:21:32.877377 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.877635 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:33.241564 (+363929us) log.cc:844] Serialized 14 byte log entry
0709 12:21:33.921057 (+679493us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:33.921130 (+    73us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:33.921201 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.921295 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:33.921624 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":363642,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":74555,"thread_start_us":678692,"threads_started":1}
I20260709 12:21:34.205386  1906 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.212363  1627 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.205389  1751 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:34.212869 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.204973 31823 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
W20260709 12:21:34.213758 31751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 288) took 2672 ms (client timeout 1763 ms). Trace:
I20260709 12:21:34.261291 31746 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 3.
W20260709 12:21:34.261418 31751 rpcz_store.cc:269] 0709 12:21:31.541306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.541528 (+   222us) service_pool.cc:224] Handling call
0709 12:21:34.213534 (+2672006us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b5400064d54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:34.213651 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2824366976}
I20260709 12:21:34.215776 31745 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.200558 31820 rpcz_store.cc:276] 0709 12:21:31.061935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.621683 (+1559748us) service_pool.cc:224] Handling call
0709 12:21:32.638781 (+ 17098us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:33.045134 (+406353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:33.045193 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:33.185142 (+139949us) log.cc:844] Serialized 26 byte log entry
0709 12:21:33.898341 (+713199us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:33.898383 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.898448 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:34.044712 (+146264us) raft_consensus.cc:1685] finished
0709 12:21:34.044742 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.044913 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":727,"raft.queue_time_us":191806,"raft.run_cpu_time_us":56,"raft.run_wall_time_us":56,"thread_start_us":831684,"threads_started":2,"wal-append.queue_time_us":758421}
I20260709 12:21:34.032326  1896 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.026028  1772 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:34.224188 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.020921  1852 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.214934  1907 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:34.229293  1770 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:34.214506  1818 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.067293  1560 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:34.208257 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:34.241752 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.280953 31818 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
W20260709 12:21:34.209545 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 313298us
I20260709 12:21:34.068187  1613 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:34.036194  1803 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:34.067400  1849 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.047793  1876 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:34.211714 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 455295us
I20260709 12:21:34.239193 31747 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:34.297271 31745 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:34.255501 31819 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:34.154598  1869 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:34.255033  1884 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.258076 31675 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:34.057636  1780 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.259394 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.260161  1627 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.264899 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.265081 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.190711  1765 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.063s
I20260709 12:21:34.266888 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.314287 31676 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:34.266820 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:34.265652 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 491492us
W20260709 12:21:34.269575 31716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.007s	sys 0.013s
I20260709 12:21:34.243803  1399 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:34.337776  1399 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:34.274541  1725 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.287921  1874 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:34.338963  1399 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:34.339002  1725 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:34.252511  1905 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:34.340323 31820 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:34.340569  1905 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.341441  1725 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:34.270071  1896 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.342535 31820 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:34.264995  1906 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:34.289040 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.343724 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.344424 31678 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:34.268831  1772 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.345330  1772 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:34.257009  1716 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:34.345963  1759 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.346532  1759 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:34.346902  1759 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.347245  1759 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.304062 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.311628  1613 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:34.315876 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:34.335327 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.343873  1896 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.344835  1906 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.350064 31819 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:34.355147 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 146947us
I20260709 12:21:34.356292 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:34.288373  1844 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:34.322587 31716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.007s	sys 0.013s
W20260709 12:21:34.361132 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.364689 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:34.368832 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:34.361541 31717 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:34.348919  1780 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260709 12:21:34.364490 32342 connection.cc:256] RPC call timeout handler was delayed by -1.27563821800004s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1763s for a response
I20260709 12:21:34.350142  1905 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:34.384114 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:34.364063  1756 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:34.351981  1613 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:34.394264 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:34.395097 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1700) took 4705 ms. Trace:
I20260709 12:21:34.395514 31825 rpcz_store.cc:276] 0709 12:21:29.689404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:30.859506 (+1170102us) service_pool.cc:224] Handling call
0709 12:21:30.862517 (+  3011us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:30.870290 (+  7773us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:30.870311 (+    21us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:30.873254 (+  2943us) log.cc:844] Serialized 52 byte log entry
0709 12:21:31.081516 (+208262us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:31.274466 (+192950us) log.cc:844] Serialized 14 byte log entry
0709 12:21:31.671578 (+397112us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:31.671691 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:34.347592 (+2675901us) spinlock_profiling.cc:243] Waited 2.72 ms on lock 0x7b18003a0fc0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:21:34.394772 (+ 47180us) raft_consensus.cc:1685] finished
0709 12:21:34.394876 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.395032 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17903,"prepare.queue_time_us":452729,"prepare.run_cpu_time_us":6814,"prepare.run_wall_time_us":48972,"raft.queue_time_us":193908,"raft.run_cpu_time_us":323,"raft.run_wall_time_us":71728,"spinlock_wait_cycles":6266752,"thread_start_us":757642,"threads_started":3,"wal-append.queue_time_us":764657}
I20260709 12:21:34.403627  1915 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.404315  1915 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.405018  1907 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:34.407074 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 50380.3us
I20260709 12:21:34.408720 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:34.409739 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:34.410518  1896 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:34.410934 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:34.411284  1896 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:21:34.411897  1896 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:34.412204 31745 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:21:34.412815  1896 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.413676  1797 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:34.414000 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.414304  1797 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:34.414987 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.415329  1878 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.415733 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:34.413300 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:34.427143 31745 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.362303  1765 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:34.371031  1415 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:34.406427  1915 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.417312 31747 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:34.370198  1906 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.370189  1581 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.419991 31751 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:34.379716  1734 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:34.383356  1544 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:34.385557  1780 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.429700 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:34.367713  1754 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.052s
I20260709 12:21:34.427197  1851 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:34.416391 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:34.339104  1874 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:34.468709 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:34.469391 32376 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:34.436026  1907 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.024s
I20260709 12:21:34.469801  1874 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:34.416962  1613 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.470217 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:34.369810  1844 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:34.453344 31745 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:34.424728  1797 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:34.505630  1797 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.508769  1844 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:34.460734 31749 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.434790 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:34.510068 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 75375.6us
I20260709 12:21:34.467468 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:34.510919 31751 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.464066  1780 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:34.443820 31643 connection.cc:256] RPC call timeout handler was delayed by -1.900591653000447s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32930000000000004s for a response
W20260709 12:21:34.511639 31643 connection.cc:256] RPC call timeout handler was delayed by -1.7251885929999844s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1543s for a response
I20260709 12:21:34.511673 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:34.512048 31643 connection.cc:256] RPC call timeout handler was delayed by -1.4056885929999225s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1898s for a response
W20260709 12:21:34.512394 31643 connection.cc:256] RPC call timeout handler was delayed by -1.1311885929999335s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2203s for a response
I20260709 12:21:34.512970 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:34.440213  1770 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260709 12:21:34.475589 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.475158  1581 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.478802  1754 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:34.436378  1765 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:34.478828 32371 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:21:34.479421  1613 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.479918  1818 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:34.439646  1905 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.481124  1869 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:34.437485  1415 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.426964  1817 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 12:21:34.482795 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1861) took 2846 ms. Trace:
I20260709 12:21:34.487926  1851 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:34.419847  1627 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.497589 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1360) took 2264 ms. Trace:
I20260709 12:21:34.558319 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:34.589475 31677 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:34.494151 31826 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:34.501523 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.514509 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:34.516269 31746 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:34.518267 31670 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:34.518036 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:34.522300 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:34.527995 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 151418us
W20260709 12:21:34.594049 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 76087.6us
I20260709 12:21:34.531996  1754 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:34.533908  1772 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.588406 31753 rpcz_store.cc:276] 0709 12:21:32.233336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.838578 (+605242us) service_pool.cc:224] Handling call
0709 12:21:32.838979 (+   401us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:34.109605 (+1270626us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:34.109650 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:34.111249 (+  1599us) log.cc:844] Serialized 52 byte log entry
0709 12:21:34.353783 (+242534us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:34.353817 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.353884 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:34.497329 (+143445us) raft_consensus.cc:1685] finished
0709 12:21:34.497352 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.497521 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":233778,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":58868,"thread_start_us":533797,"threads_started":2,"wal-append.queue_time_us":278464}
I20260709 12:21:34.517067  1443 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:34.535269 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:34.522778  1727 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.745s	user 0.013s	sys 0.000s
I20260709 12:21:34.434967  1852 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:34.535280 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 132265us
I20260709 12:21:34.601787 31673 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:34.524128  1907 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:34.524348  1911 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:34.546797 31827 rpcz_store.cc:276] 0709 12:21:31.636772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.632903 (+996131us) service_pool.cc:224] Handling call
0709 12:21:32.633254 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.840481 (+207227us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:33.757291 (+916810us) log.cc:844] Serialized 14 byte log entry
0709 12:21:34.224654 (+467363us) log.cc:844] Serialized 14 byte log entry
0709 12:21:34.242157 (+ 17503us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:34.242235 (+    78us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:34.242304 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.482523 (+240219us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54001c93d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:34.482573 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.482735 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31818,"prepare.queue_time_us":805385,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":86,"raft.queue_time_us":33726,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":21439,"spinlock_wait_cycles":496634624,"thread_start_us":1345150,"threads_started":2,"wal-append.queue_time_us":430122}
I20260709 12:21:34.605474 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:34.551946 31751 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
W20260709 12:21:34.553798  1890 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.526s	user 0.002s	sys 0.000s
I20260709 12:21:34.606921 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.609117 31751 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:34.612918 31671 raft_consensus.cc:2368] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6.
I20260709 12:21:34.616269 31675 raft_consensus.cc:2368] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5.
I20260709 12:21:34.543759  1905 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.545254  1415 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.600929  1770 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:34.563952  1581 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:34.563771 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:34.622232 32378 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:34.605427  1907 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.622890  1907 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:34.565562  1613 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.566465 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:34.528019  1735 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:34.572398 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.605593  1854 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:34.573179 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:34.574185  1843 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:34.576045  1786 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:34.577073  1759 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.640297  1735 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.641291  1759 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:34.578085 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 324802us
I20260709 12:21:34.642025  1627 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.621901  1770 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.541569 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:34.642916 31669 raft_consensus.cc:2368] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8.
I20260709 12:21:34.643064 31752 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:34.586956 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 577) took 2085 ms. Trace:
I20260709 12:21:34.643246 32370 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:34.465041  1906 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:34.643620 32372 rpcz_store.cc:276] 0709 12:21:32.501594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.540563 (+ 38969us) service_pool.cc:224] Handling call
0709 12:21:33.484631 (+944068us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:33.499665 (+ 15034us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:33.499717 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:33.502887 (+  3170us) log.cc:844] Serialized 26 byte log entry
0709 12:21:33.994012 (+491125us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:33.994069 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:33.994160 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:34.113822 (+119662us) raft_consensus.cc:1685] finished
0709 12:21:34.113854 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.586488 (+472634us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b54004c5c50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:34.586851 (+   363us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":187,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":3218,"spinlock_wait_cycles":2170106752,"thread_start_us":363460,"threads_started":1,"wal-append.queue_time_us":540670}
W20260709 12:21:34.594103 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 79671.3us
I20260709 12:21:34.645545  1906 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:34.595204 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:34.646318 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51175.6us
I20260709 12:21:34.648236 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:34.649127 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:34.600767 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.603305 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.650585  1905 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:34.603346 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.650892  1896 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:34.651332 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.605008 31674 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260709 12:21:34.606123 31819 raft_consensus.cc:2376] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7. Re-sending same reply.
I20260709 12:21:34.652148 31670 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:34.606794 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:34.607295 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.652669 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.652957 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.653018 31821 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:34.653848 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.603953 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.655072  1878 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.655287 31825 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:34.557751  1751 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.642926  1770 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.655718  1878 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.657552  1751 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.553956 31749 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:34.658327 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:34.658957 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:34.617264 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:34.659252 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:34.659606 32376 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:34.659847 31745 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:34.654330 31670 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:21:34.660269 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:34.660769 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.660847 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.661299 31670 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:21:34.661914 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.662010 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:34.607995  1911 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.651609 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:34.662819  1911 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:34.662941 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:34.632174  1765 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:34.655153  1851 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:21:34.664901 31748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1342) took 2688 ms (client timeout 2258 ms). Trace:
I20260709 12:21:34.665560  1852 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.665833 31744 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:34.622030 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 437) took 3206 ms. Trace:
I20260709 12:21:34.667018  1869 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:34.667537  1869 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.667706 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1800) took 2078 ms. Trace:
I20260709 12:21:34.667950  1869 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:34.668143 31744 rpcz_store.cc:276] 0709 12:21:32.589494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.865192 (+275698us) service_pool.cc:224] Handling call
0709 12:21:34.665653 (+1800461us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:34.666256 (+   603us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.667498 (+  1242us) spinlock_profiling.cc:243] Waited 1.68 s on lock 0x7b5400294fd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:34.667626 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3870263552}
I20260709 12:21:34.668797  1613 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:34.669369  1613 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:34.639199  1866 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:34.665343 31748 rpcz_store.cc:269] 0709 12:21:31.976390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.976614 (+   224us) service_pool.cc:224] Handling call
0709 12:21:34.664689 (+2688075us) spinlock_profiling.cc:243] Waited 831 ms on lock 0x7b5400294fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:34.664832 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1910268672}
I20260709 12:21:34.672581 31746 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:34.672989 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:34.673722  1911 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.676316  1911 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:34.672991 31748 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:21:34.679987  1915 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:34.680441  1915 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.680788  1915 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.681370 31748 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:34.686846  1786 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:34.687942  1786 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.688495  1905 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:34.688997  1786 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.689711 31792 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:34.690490  1581 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:34.690868  1581 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:34.691689 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:34.691748  1581 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:34.692888  1581 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.696190  1772 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:34.696480 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.697924 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:34.699867 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:34.701175  1399 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:34.598321 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:34.597071 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.651511  1897 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:34.703182  1897 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:34.704192  1897 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.704674  1874 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:34.704833  1844 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:34.705458  1897 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.705696  1844 consensus_queue.cc:1243] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:21:34.706344  1844 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:34.654196  1754 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.658592 31673 raft_consensus.cc:2368] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:21:34.662149  1906 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:34.530372  1884 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:34.707310  1754 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:34.708092 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 109844us
I20260709 12:21:34.708863 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:34.709049  1884 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:34.707202  1906 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.709860  1906 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:34.666363 32373 rpcz_store.cc:276] 0709 12:21:31.412583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.422420 (+  9837us) service_pool.cc:224] Handling call
0709 12:21:31.660475 (+238055us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:31.696074 (+ 35599us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:32.264427 (+568353us) log.cc:844] Serialized 14 byte log entry
0709 12:21:34.543061 (+2278634us) spinlock_profiling.cc:243] Waited 29.2 ms on lock 0x7b180020e540. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a
0709 12:21:34.548836 (+  5775us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:34.589678 (+ 40842us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:34.601242 (+ 11564us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.604827 (+  3585us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.616254 (+ 11427us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":923881,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":6995,"raft.queue_time_us":829,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":5221,"spinlock_wait_cycles":67156864,"thread_start_us":869337,"threads_started":2,"wal-append.queue_time_us":594937,"wal-append.run_cpu_time_us":1099,"wal-append.run_wall_time_us":1007969}
I20260709 12:21:34.670914  1866 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:34.710785  1876 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:34.711005  1866 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.711315  1876 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.642379 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:34.701361 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:34.705197  1874 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.650676 31823 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:34.706808  1725 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:34.704692  1312 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260709 12:21:34.714545 31748 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260709 12:21:34.635715 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62667.3us
I20260709 12:21:34.723671 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:34.724083 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:34.158519 31750 rpcz_store.cc:276] 0709 12:21:32.608808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:32.888074 (+279266us) service_pool.cc:224] Handling call
0709 12:21:32.888387 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:32.888594 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:21:33.989686 (+1101092us) log.cc:844] Serialized 14 byte log entry
0709 12:21:34.154238 (+164552us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:34.154332 (+    94us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:34.154421 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.154534 (+   113us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:34.154775 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":353023,"prepare.queue_time_us":436855,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":69620,"thread_start_us":800049,"threads_started":2}
I20260709 12:21:34.651328  1896 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:34.646703 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:34.682566 31678 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:21:34.638085 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.725876 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:34.661942 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:34.620748 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.728720 31751 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:34.710563 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:34.729533 31748 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:21:34.731128 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:34.716277 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.662583 31745 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:34.659050 32371 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
W20260709 12:21:34.720244 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 77956us
I20260709 12:21:34.601114  1752 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:34.735903 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:34.737422 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:34.729518 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:34.741406 31751 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:34.717427  1874 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.748809 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:34.751454 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:34.738617 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.753762 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:34.633296  1843 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:21:34.757925 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:34.788812 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 142201us
I20260709 12:21:34.757601 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:34.756729 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:34.715194  1844 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.788352  1914 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:34.797936 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.798784 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.817615  1722 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:34.817909 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.818208  1722 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.747726  1904 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:34.818610 31826 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:34.818964  1904 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:34.819439  1904 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.670222  1613 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.820014  1908 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:34.821209 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:34.734994  1852 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:34.718060  1725 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:34.822818 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:34.791744  1900 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:34.723150  1919 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:34.722299  1560 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:34.814985 31643 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:34.825945 31823 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:34.797632  1844 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.735502  1803 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:34.827781 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.822700 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.833567 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.858513 31824 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:34.838521 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:34.859336 31820 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:34.844846 31819 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3. Ops: []
I20260709 12:21:34.859879 31824 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:34.788592  1849 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:34.798132  1765 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:34.755488  1817 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:34.874981 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 60089.4us
I20260709 12:21:34.853416 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.854318 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.858536 31822 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:34.875918 31826 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:34.876025 31827 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:34.818665  1722 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.827312  1852 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.864511 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.735294  1896 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:34.882978 31823 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:21:34.845589  1842 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.885197 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:34.880437 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:34.841060  1399 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:34.881112 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:34.887012 31825 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:34.887748 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:34.918850 31678 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:34.856964  1900 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:34.938225 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 207175us
I20260709 12:21:34.940503 31818 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:34.862154  1919 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:34.927026  1852 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.943764 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:34.873662  1817 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.946209 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.927578  1399 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:34.946409 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:34.988108 31826 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:34.988361 31822 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:34.948597 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:34.951056 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:34.879421  1808 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:34.952414 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 194903us
I20260709 12:21:34.995288 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:34.996394 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:34.996995 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:34.997606 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:34.850745  1869 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:34.882493  1751 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:34.929422  1911 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:34.850984  1915 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:34.944419  1919 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:34.953979 31792 leader_election.cc:336] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.628s (SENT)
I20260709 12:21:34.946946 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:34.884001  1884 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:34.989440 31825 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:35.010865 31820 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 6
I20260709 12:21:34.937824  1443 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:35.011674 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:35.012398 31825 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:34.990394 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:34.999033 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:34.937719  1716 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:35.000679 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 178625us
W20260709 12:21:35.020154  1922 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260709 12:21:35.020534 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:35.001240 31670 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:34.932998  1921 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:34.944703  1900 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:35.005142 31748 raft_consensus.cc:606] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:35.022840 31748 raft_consensus.cc:614] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:35.023768 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:35.025116  1914 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:35.025179 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:35.025493 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:35.025975  1914 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:35.026185 31675 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:35.028237 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:35.002331  1911 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.031179  1911 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.033398  1915 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:35.033999  1878 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:35.034762 31751 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 6. Ops: []
I20260709 12:21:35.018827  1443 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:35.035842 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:35.036199  1443 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.036612 31675 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:35.037009 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:35.037470 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:35.037693 31751 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:35.000010  1770 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.038165 31753 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:35.038486  1876 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:34.995759  1817 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.039669  1770 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:35.039830 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:35.042709 31791 leader_election.cc:400] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6.
I20260709 12:21:35.043201 31791 leader_election.cc:403] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:35.045584 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 441) took 1805 ms. Trace:
I20260709 12:21:35.046042 32377 rpcz_store.cc:276] 0709 12:21:33.240678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.683276 (+442598us) service_pool.cc:224] Handling call
0709 12:21:34.990192 (+1306916us) spinlock_profiling.cc:243] Waited 3.51 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:35.045518 (+ 55326us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":8068864}
I20260709 12:21:35.047430 31820 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:35.010502 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 56564.6us
I20260709 12:21:35.016065 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:35.054821 31826 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:35.018782  1884 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.056844  1772 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:34.995574  1399 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:35.006593 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:35.060953  1581 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:21:35.062132  1780 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.535s	user 0.004s	sys 0.004s
I20260709 12:21:34.938812  1920 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.117s
I20260709 12:21:34.883466  1843 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:35.073009 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 273750us
I20260709 12:21:35.026117 31748 raft_consensus.cc:2376] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7. Re-sending same reply.
I20260709 12:21:35.089247 31751 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260709 12:21:35.106323 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58316.3us
I20260709 12:21:35.066797  1808 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:35.098227  1765 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.019s
I20260709 12:21:35.097936  1876 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:35.117156  1443 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.103904  1930 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6.
I20260709 12:21:35.123613 32373 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:35.136756  1843 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:35.137373 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.148877 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:35.138067 31677 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:35.111927  1581 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.098129  1905 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:35.213945 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 78375.5us
I20260709 12:21:35.215333 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:35.139963 31751 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:21:35.146221  1765 consensus_queue.cc:1243] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:21:35.138044  1780 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:35.126879  1849 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:35.235327 31672 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:35.203279  1581 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:35.138942  1808 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:35.143088  1876 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:35.205475  1905 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.275240 31750 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 2. Current term is 3. Ops: [2.2-2.2]
W20260709 12:21:35.276342 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 150996us
I20260709 12:21:35.233605  1931 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:35.286898 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:35.279362 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 205151us
I20260709 12:21:35.295670 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:35.299723 31822 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:35.248154  1445 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:35.309535 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:35.251464  1808 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:35.310066 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:35.274652  1923 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:21:35.311782 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:35.254269  1876 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:35.315814 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:35.316115 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:35.319269  1854 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:35.319926  1854 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:35.321265  1932 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:35.321779  1932 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.322135  1932 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:35.325577  1754 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.618s	user 0.008s	sys 0.000s
I20260709 12:21:35.311283  1445 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.326004  1443 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:35.331121  1445 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:35.332664 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1254) took 1478 ms. Trace:
I20260709 12:21:35.333055 31674 rpcz_store.cc:276] 0709 12:21:33.854410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:33.997378 (+142968us) service_pool.cc:224] Handling call
0709 12:21:35.332479 (+1335101us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b54002e7f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:35.332606 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1388857088}
W20260709 12:21:35.335235  1928 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 12:21:35.336654 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1272) took 1270 ms. Trace:
I20260709 12:21:35.337141 31676 rpcz_store.cc:276] 0709 12:21:34.066052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.683187 (+617135us) service_pool.cc:224] Handling call
0709 12:21:34.683520 (+   333us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:34.683889 (+   369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:34.683918 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:34.684577 (+   659us) log.cc:844] Serialized 52 byte log entry
0709 12:21:34.955966 (+271389us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:34.956032 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:34.956135 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:35.336351 (+380216us) raft_consensus.cc:1685] finished
0709 12:21:35.336379 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:35.336578 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10949,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":117,"thread_start_us":270876,"threads_started":1,"wal-append.queue_time_us":462780}
I20260709 12:21:35.339524 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:35.341137  1786 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.652s	user 0.008s	sys 0.000s
I20260709 12:21:35.341303 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:35.342574 31823 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
I20260709 12:21:35.236255  1878 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.046s
W20260709 12:21:35.326623 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.276024  1905 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:35.327024  1920 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.334309 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:35.311064  1931 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.342967 31823 pending_rounds.cc:85] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 2
W20260709 12:21:35.343628 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.347900 31823 pending_rounds.cc:107] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:21:35.328052  1849 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:35.344310 31674 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:35.348181 31823 raft_consensus.cc:2889] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:35.328969 31827 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260709 12:21:35.330487  1897 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.624s	user 0.010s	sys 0.000s
I20260709 12:21:35.353820 32343 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:35.356038 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 462170us
W20260709 12:21:35.359706 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 43660.9us
I20260709 12:21:35.360800 31825 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:35.345350 31675 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:35.345770 32377 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:21:35.344899 31751 raft_consensus.cc:606] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:35.362751 31751 raft_consensus.cc:614] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:35.364557 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:35.365090 31791 leader_election.cc:400] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5.
I20260709 12:21:35.365489 31791 leader_election.cc:403] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260709 12:21:35.362396 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:35.366291 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:35.364638 31826 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
W20260709 12:21:35.374399 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 377426us
I20260709 12:21:35.404986 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:35.367566 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:35.375635 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:35.407979 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:35.351042  1765 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.060s
I20260709 12:21:35.417356 31676 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:21:35.311208  1869 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:35.424429 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:35.424791 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60725.4us
I20260709 12:21:35.431553 31674 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:35.359508  1933 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:35.433310 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:35.435197 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:35.354961  1930 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:21:35.360877  1613 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.018s	sys 0.011s
I20260709 12:21:35.367280  1908 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:35.447923 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1285) took 1073 ms. Trace:
I20260709 12:21:35.448529 31677 rpcz_store.cc:276] 0709 12:21:34.374610 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.596750 (+222140us) service_pool.cc:224] Handling call
0709 12:21:35.447642 (+850892us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b540017c5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:35.447836 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":999773568}
I20260709 12:21:35.438982  1935 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260709 12:21:35.461194  1722 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.527s	user 0.012s	sys 0.011s
I20260709 12:21:35.461565  1935 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:35.462024  1935 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:35.461687  1722 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.441629  1754 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:35.463025  1754 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:35.463886  1754 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:35.464411  1911 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:35.442539  1613 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.022s	sys 0.013s
I20260709 12:21:35.368404  1920 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.446945  1869 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:35.476176 31746 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.005s	sys 0.050s
I20260709 12:21:35.439918  1933 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:35.466496  1911 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:35.498473 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1256) took 1643 ms. Trace:
I20260709 12:21:35.499186 31669 rpcz_store.cc:276] 0709 12:21:33.855105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.000927 (+145822us) service_pool.cc:224] Handling call
0709 12:21:35.498190 (+1497263us) spinlock_profiling.cc:243] Waited 483 ms on lock 0x7b540022b354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:35.498389 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1111871232}
I20260709 12:21:35.433276  1544 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 12:21:35.471606 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1265) took 1567 ms. Trace:
I20260709 12:21:35.507237 31670 rpcz_store.cc:276] 0709 12:21:33.904374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.661980 (+757606us) service_pool.cc:224] Handling call
0709 12:21:34.662220 (+   240us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:35.021524 (+359304us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.471242 (+449718us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b54000465d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:35.471524 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":779043584}
I20260709 12:21:35.440915  1915 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:35.446414  1757 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:35.482234 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 424) took 3938 ms. Trace:
I20260709 12:21:35.441098 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:35.486032 31746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.005s	sys 0.050s
I20260709 12:21:35.475775  1936 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 4. Current term is 5.
I20260709 12:21:35.467926 32371 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260709 12:21:35.524214 32375 rpcz_store.cc:276] 0709 12:21:31.543824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:31.543999 (+   175us) service_pool.cc:224] Handling call
0709 12:21:31.550128 (+  6129us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:31.915781 (+365653us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:31.915830 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:34.560377 (+2644547us) spinlock_profiling.cc:243] Waited 5.26 ms on lock 0x7b2c000eba68. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:21:34.561190 (+   813us) log.cc:844] Serialized 52 byte log entry
0709 12:21:34.878857 (+317667us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:34.932058 (+ 53201us) log.cc:844] Serialized 14 byte log entry
0709 12:21:34.934376 (+  2318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.481813 (+547437us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b5400544f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:35.481883 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:35.481947 (+    64us) raft_consensus.cc:1685] finished
0709 12:21:35.481967 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:35.482156 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2762,"prepare.queue_time_us":64924,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":7095,"raft.queue_time_us":249654,"raft.run_cpu_time_us":431,"raft.run_wall_time_us":46611,"spinlock_wait_cycles":252877056,"thread_start_us":504137,"threads_started":2,"wal-append.queue_time_us":561463}
I20260709 12:21:35.510294  1933 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:35.556082 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1814) took 1124 ms. Trace:
I20260709 12:21:35.556710 31746 rpcz_store.cc:276] 0709 12:21:34.431291 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.673281 (+241990us) service_pool.cc:224] Handling call
0709 12:21:34.673582 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:34.673776 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:21:34.941304 (+267528us) log.cc:844] Serialized 14 byte log entry
0709 12:21:35.555592 (+614288us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:35.555668 (+    76us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:35.555744 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.555839 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:35.555988 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28101,"prepare.queue_time_us":530244,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":3509,"thread_start_us":852845,"threads_started":2}
I20260709 12:21:35.559329 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:35.574610 31672 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:35.539382  1786 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:35.538859  1884 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:35.590420 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 171034us
W20260709 12:21:35.594588 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 153650us
I20260709 12:21:35.596568 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:35.569016  1759 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:35.563186  1915 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:35.593966  1797 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.088s	user 0.014s	sys 0.013s
I20260709 12:21:35.579289  1780 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:35.597787  1817 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.556s	user 0.012s	sys 0.010s
I20260709 12:21:35.619668 31751 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:21:35.637562 31751 raft_consensus.cc:606] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:35.638166 31751 raft_consensus.cc:614] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:21:35.594359  1900 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.572s	user 0.015s	sys 0.008s
I20260709 12:21:35.639199  1900 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:35.596190  1415 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.965s	user 0.008s	sys 0.008s
I20260709 12:21:35.641376  1415 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:35.587544  1757 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:35.647195  1415 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:35.628470 31669 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:21:35.655658 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 293) took 1051 ms. Trace:
I20260709 12:21:35.628886 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:35.633422 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1281) took 1260 ms. Trace:
W20260709 12:21:35.606704  1904 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.779s	user 0.013s	sys 0.004s
I20260709 12:21:35.657855  1770 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:35.657648  1904 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.658879 31827 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
W20260709 12:21:35.659694  1919 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.521s	user 0.007s	sys 0.000s
I20260709 12:21:35.657428 31673 rpcz_store.cc:276] 0709 12:21:34.373255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.603601 (+230346us) service_pool.cc:224] Handling call
0709 12:21:35.633174 (+1029573us) spinlock_profiling.cc:243] Waited 5.7 ms on lock 0x7b5400229f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:35.633348 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13100928}
I20260709 12:21:35.656618 31669 rpcz_store.cc:276] 0709 12:21:34.604014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.585357 (+981343us) service_pool.cc:224] Handling call
0709 12:21:35.609460 (+ 24103us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:35.654939 (+ 45479us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.655527 (+   588us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:35.660246  1919 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.661495 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 294) took 1002 ms. Trace:
I20260709 12:21:35.661860 31673 rpcz_store.cc:276] 0709 12:21:34.659114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.660596 (+1001482us) service_pool.cc:224] Handling call
0709 12:21:35.660934 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:35.661118 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:35.661152 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:35.661205 (+    53us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:35.661255 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.661310 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:35.661439 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:35.663554  1757 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.658068 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:35.666378 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:35.667542  1754 raft_consensus.cc:2764] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:21:35.672050 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:35.673362 31824 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:21:35.677758  1944 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:35.678530 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:35.680501 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:35.644533  1866 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.925s	user 0.007s	sys 0.005s
I20260709 12:21:35.693477 31675 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 4. Current term is 6. Ops: []
I20260709 12:21:35.723543 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:35.724267 31822 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
W20260709 12:21:35.652889  1925 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.306s	user 0.001s	sys 0.001s
I20260709 12:21:35.696609 31826 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:35.703051 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 295) took 1043 ms. Trace:
I20260709 12:21:35.726440 31669 rpcz_store.cc:276] 0709 12:21:34.659392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.660819 (+1001427us) service_pool.cc:224] Handling call
0709 12:21:35.662591 (+  1772us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:35.684720 (+ 22129us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:35.684961 (+   241us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:35.685111 (+   150us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:35.702361 (+ 17250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.702434 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:35.702576 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:35.709714 31673 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
W20260709 12:21:35.729992 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 323148us
I20260709 12:21:35.643743  1613 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:35.720995 31824 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:35.734048 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:35.735764 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:35.739714 31824 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:35.729467 31669 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260709 12:21:35.729453  1844 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.852s	user 0.016s	sys 0.009s
I20260709 12:21:35.740293  1803 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:35.728747  1849 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:21:35.734364 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 369870us
I20260709 12:21:35.746282 31673 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:35.740676 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:35.730870  1929 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.325s	user 0.000s	sys 0.001s
I20260709 12:21:35.752875 31824 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:35.754372 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.644263  1797 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.761190 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:35.761984 31822 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:35.757166 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:35.653517  1817 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.763145 31824 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:35.683204  1754 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:35.758126 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:35.780787 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:35.731571  1944 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:35.777427 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:35.724746 32373 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:35.733021  1866 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.783906 31824 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:35.757215  1786 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:35.772012  1780 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:35.787446 31750 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
W20260709 12:21:35.787642 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 188363us
I20260709 12:21:35.784773 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:35.780151  1946 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 12:21:35.783450 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:35.800846 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:35.825578 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:35.829402 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:35.829823 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:35.830019 31822 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:35.752493  1803 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:35.754740  1844 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:35.803512 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.779718  1754 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:35.844635 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1372) took 1442 ms. Trace:
W20260709 12:21:35.799197  1842 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.863s	user 0.008s	sys 0.003s
I20260709 12:21:35.831079 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:35.841221  1854 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.521s	user 0.009s	sys 0.000s
W20260709 12:21:35.842203  1938 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:21:35.786888  1960 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:35.843555  1931 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:35.852288 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 100246us
W20260709 12:21:35.823330  1953 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:21:35.854699 31824 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:35.854790 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:35.897784 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:35.898087 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:35.898736 31824 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 3: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:35.899583 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:35.864463 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:35.845042 31744 rpcz_store.cc:276] 0709 12:21:34.402573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.669130 (+266557us) service_pool.cc:224] Handling call
0709 12:21:34.669505 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:34.669875 (+   370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:34.669906 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:34.670416 (+   510us) log.cc:844] Serialized 26 byte log entry
0709 12:21:35.004463 (+334047us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:35.004513 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.004612 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:35.844409 (+839797us) raft_consensus.cc:1685] finished
0709 12:21:35.844428 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:35.844569 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":263,"raft.run_cpu_time_us":46,"raft.run_wall_time_us":46,"thread_start_us":305518,"threads_started":1,"wal-append.queue_time_us":603818}
I20260709 12:21:35.890111 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:35.893868 31821 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:35.901006 31822 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:35.901022 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:35.760902  1915 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:35.844841  1911 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:35.844077  1735 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:35.899850 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:35.843573  1948 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:21:35.809465  1786 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:35.911252 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46946.5us
I20260709 12:21:35.891633  1770 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:35.892398  1818 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:35.897653  1896 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.956s	user 0.003s	sys 0.008s
I20260709 12:21:35.910894 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:35.924217 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1881) took 1084 ms. Trace:
I20260709 12:21:35.943675  1770 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:35.944227  1770 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:35.943032 31819 rpcz_store.cc:276] 0709 12:21:34.840181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.860289 (+ 20108us) service_pool.cc:224] Handling call
0709 12:21:34.877858 (+ 17569us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:35.234526 (+356668us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:35.234570 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:35.243776 (+  9206us) log.cc:844] Serialized 26 byte log entry
0709 12:21:35.510063 (+266287us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:35.646892 (+136829us) log.cc:844] Serialized 14 byte log entry
0709 12:21:35.669118 (+ 22226us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.669195 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:35.923925 (+254730us) raft_consensus.cc:1685] finished
0709 12:21:35.923953 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:35.924138 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33064,"prepare.queue_time_us":328920,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":39,"raft.queue_time_us":162167,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":40142,"thread_start_us":716119,"threads_started":3,"wal-append.queue_time_us":399459}
I20260709 12:21:35.945742 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:35.947345 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:35.948258 31819 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:35.926756 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 142048us
I20260709 12:21:35.925149 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:35.922151 31824 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:35.954109 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:35.954774 31824 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:35.955569 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:35.956313 31824 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 6 because replica has last-logged OpId of term: 4 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 4.
I20260709 12:21:35.956995 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:35.886059  1884 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:35.850242  1544 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.079s
I20260709 12:21:35.959296  1627 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:35.959791  1627 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:35.960645  1627 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:35.961911  1627 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:35.962821  1904 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:35.963604  1952 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 12:21:35.963830  1803 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:35.867238  1756 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.076s
I20260709 12:21:35.849807  1754 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:35.890424  1960 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:35.867616  1415 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:35.836234  1780 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.940340 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:35.958326  1884 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:35.930876 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:35.966465 31824 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:21:35.991355 31819 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:35.909657  1772 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:35.912717  1842 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:36.036172  1896 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:35.926797  1951 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:36.048840  1939 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:36.076684  1939 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:36.077119  1896 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.049214 31822 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:36.078943  1842 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:36.079586  1842 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:36.087448  1956 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
I20260709 12:21:35.885612  1854 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:35.964520  1803 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.088s
I20260709 12:21:35.937999  1878 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260709 12:21:36.065421 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:36.059979 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 129188us
I20260709 12:21:36.120108 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:35.929013  1751 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:36.077831  1939 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:36.070106  1808 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.718s	user 0.007s	sys 0.003s
W20260709 12:21:36.058403  1908 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.612s	user 0.006s	sys 0.000s
W20260709 12:21:36.109925  1874 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.310s	user 0.014s	sys 0.000s
I20260709 12:21:35.929042  1900 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:35.893496  1849 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.059296  1544 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
W20260709 12:21:36.110247  1907 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.487s	user 0.007s	sys 0.007s
I20260709 12:21:36.141417 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 288) took 1609 ms. Trace:
I20260709 12:21:36.107597 31822 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:36.064679  1786 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:36.048038  1960 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:36.060719  1945 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:36.060277  1931 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.062763  1884 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:36.120297  1920 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:36.149803  1969 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.036852  1818 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.129650 31744 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:36.147949 31676 rpcz_store.cc:276] 0709 12:21:34.531751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.425907 (+894156us) service_pool.cc:224] Handling call
0709 12:21:35.426360 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:35.433970 (+  7610us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:35.434019 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:35.442142 (+  8123us) log.cc:844] Serialized 26 byte log entry
0709 12:21:35.564025 (+121883us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:35.564134 (+   109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.564276 (+   142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:36.141106 (+576830us) raft_consensus.cc:1685] finished
0709 12:21:36.141136 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:36.141335 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6878,"raft.queue_time_us":11729,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":4459,"thread_start_us":96872,"threads_started":1,"wal-append.queue_time_us":285054}
I20260709 12:21:36.143782  1312 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:21:36.182086 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 257010us
I20260709 12:21:36.146600  1908 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:21:36.166909  1843 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.932s	user 0.011s	sys 0.000s
I20260709 12:21:35.919029  1576 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.220s
I20260709 12:21:36.122655  1878 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.128999  1797 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:36.130478  1896 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:36.124207  1751 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:36.163493 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 252677us
I20260709 12:21:36.182164  1312 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:21:36.161952 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 206388us
I20260709 12:21:36.209892 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.211719 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:36.212771 31822 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:35.918408  1950 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:36.158759  1907 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.160727  1957 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:36.172784  1780 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:36.176514  1770 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.299614  1770 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.035048  1944 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.176884  1842 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.304304  1842 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:36.177809  1876 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.823s	user 0.007s	sys 0.001s
I20260709 12:21:36.305248  1876 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.306483 31819 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:36.307545  1876 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:36.160807  1544 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:36.179013 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1807) took 1363 ms. Trace:
I20260709 12:21:36.183048 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:36.308764  1544 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:36.309286 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 126308us
I20260709 12:21:36.064970  1970 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.093s
I20260709 12:21:36.255368  1443 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:21:36.310931  1443 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:36.311017  1970 consensus_queue.cc:1243] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:21:36.311836  1443 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.312181  1979 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:36.313002  1979 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:36.313292  1443 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:21:36.265939  1949 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.601s	user 0.004s	sys 0.000s
I20260709 12:21:36.314292  1979 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.315380  1950 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 12:21:36.277467  1940 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.634s	user 0.000s	sys 0.001s
W20260709 12:21:36.279038 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:35.965605  1919 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:35.925840  1727 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.215s
I20260709 12:21:36.319576  1727 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Stepping down as leader of term 5
W20260709 12:21:36.281865  1933 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.696s	user 0.011s	sys 0.001s
I20260709 12:21:36.320091  1727 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:36.320551  1933 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.321429  1727 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.161146  1945 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.286550 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.322888  1945 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:36.323949 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.325172 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.325829 31676 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:36.177731  1866 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.177801  1951 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.194651 31820 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 3. (term mismatch)
W20260709 12:21:36.200426  1935 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.738s	user 0.007s	sys 0.004s
W20260709 12:21:36.245355  1581 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.002s	user 0.013s	sys 0.006s
I20260709 12:21:36.327445  1935 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.327581  1581 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.327752 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:36.329011 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:36.329628  1935 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.180881  1817 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.330307 31825 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
W20260709 12:21:36.164373 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 247099us
I20260709 12:21:36.332002  1752 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:36.332216 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:36.332584  1752 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:36.335049 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:36.212080  1751 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:36.335840  1752 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.224921 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:36.339982 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 117499us
I20260709 12:21:36.341907  1930 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:36.181542  1969 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.219723  1843 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:36.342442  1930 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:36.342825  1843 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:21:36.223307  1955 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:21:36.329699  1581 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:36.147451  1874 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.144958  1808 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.332947 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:36.355437 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.356093 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:36.356274 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:36.333247 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:36.356763 31822 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:36.357296 32369 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:36.333736 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:36.203975  1878 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:36.334463 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:36.334648 32374 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:36.308427 31672 rpcz_store.cc:276] 0709 12:21:34.815859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.688498 (+872639us) service_pool.cc:224] Handling call
0709 12:21:35.688881 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:35.722200 (+ 33319us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:35.722241 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:35.725606 (+  3365us) log.cc:844] Serialized 26 byte log entry
0709 12:21:35.889753 (+164147us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:35.889824 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.889922 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:36.178711 (+288789us) raft_consensus.cc:1685] finished
0709 12:21:36.178741 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:36.178933 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43604,"raft.queue_time_us":34992,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":10228,"thread_start_us":127807,"threads_started":1,"wal-append.queue_time_us":381757}
I20260709 12:21:36.360041 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.361003 31672 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:36.361603 31749 raft_consensus.cc:2368] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:21:36.338804  1951 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.215682  1844 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:36.170053  1931 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.344533 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.327184 31820 pending_rounds.cc:85] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 2
I20260709 12:21:36.368304 31820 pending_rounds.cc:107] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:21:36.368647 31820 raft_consensus.cc:2889] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:36.248790  1854 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:36.346200 31717 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.638s (SENT) [suppressed 26 similar messages]
I20260709 12:21:36.347518 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.352092 32376 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
W20260709 12:21:36.348445  1399 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.226s	user 0.008s	sys 0.018s
I20260709 12:21:36.326507 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.377903 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:36.357362 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:36.238587  1445 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.907s	user 0.008s	sys 0.000s
I20260709 12:21:36.342814  1727 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:36.361837 32377 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:36.219154  1980 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:36.344892 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.372203 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:36.373618 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.349246  1920 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:36.245682  1900 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:36.376526 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:36.383148 32369 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260709 12:21:36.384654 31818 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:21:36.385906 31791 leader_election.cc:400] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8.
I20260709 12:21:36.396852 31791 leader_election.cc:403] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:21:36.253626  1818 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:36.388048 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:36.349153  1772 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
W20260709 12:21:36.392228 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47403.4us
I20260709 12:21:36.456061 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.397696 32374 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260709 12:21:36.457058  1907 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.370489  1933 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.457787  1920 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:36.458969  1920 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:36.459470  1958 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.795s	user 0.003s	sys 0.000s
I20260709 12:21:36.461591 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.463017 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:36.463070 31573 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 148) took 19914 ms (client timeout 15000 ms). Trace:
I20260709 12:21:36.464232 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.465365 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.466305 31670 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:21:36.391045  1975 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:21:36.404223 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:36.388741  1905 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.030s	user 0.009s	sys 0.003s
I20260709 12:21:36.484874 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:36.485558 31824 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:36.404790  1981 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:36.404719  1974 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:36.406677  1725 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.413736  1970 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8.
I20260709 12:21:36.414031  1908 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:36.414976 31673 raft_consensus.cc:2410] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:36.413113 31791 leader_election.cc:400] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:21:36.495784 31791 leader_election.cc:403] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260709 12:21:36.421063 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1894) took 1473 ms. Trace:
I20260709 12:21:36.496660 31820 rpcz_store.cc:276] 0709 12:21:34.947458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.048151 (+100693us) service_pool.cc:224] Handling call
0709 12:21:35.048462 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:36.383507 (+1335045us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:36.420804 (+ 37297us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b54003c91d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:36.420998 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2634982784}
I20260709 12:21:36.423723 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.427925 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:36.431142  1843 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.433666  1977 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:36.436693  1817 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:36.433723  1874 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:36.439682  1951 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.442202 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1888) took 1497 ms. Trace:
I20260709 12:21:36.517066 31818 rpcz_store.cc:276] 0709 12:21:34.945253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.945383 (+   130us) service_pool.cc:224] Handling call
0709 12:21:34.945728 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:36.396784 (+1451056us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:36.441939 (+ 45155us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b540037a354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:36.442142 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3308388096}
I20260709 12:21:36.518376 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:36.519100 31818 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:36.519965 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:36.521741 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.443320 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 287) took 1912 ms. Trace:
I20260709 12:21:36.522524 31675 rpcz_store.cc:276] 0709 12:21:34.531372 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.363669 (+832297us) service_pool.cc:224] Handling call
0709 12:21:35.693280 (+329611us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:35.722999 (+ 29719us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:36.443129 (+720130us) spinlock_profiling.cc:243] Waited 329 ms on lock 0x7b540017cfd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:36.443258 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":757284736}
I20260709 12:21:36.521185 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:36.447453 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 264) took 1086 ms. Trace:
I20260709 12:21:36.528163 31827 rpcz_store.cc:276] 0709 12:21:35.361404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.361618 (+   214us) service_pool.cc:224] Handling call
0709 12:21:35.419086 (+ 57468us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:35.659260 (+240174us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:36.447219 (+787959us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b540030edd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:36.447391 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":551042944}
W20260709 12:21:36.529184 31749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1375) took 2110 ms (client timeout 2221 ms). Trace:
I20260709 12:21:36.529397 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:36.529649 31749 rpcz_store.cc:269] 0709 12:21:34.419072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.659045 (+239973us) service_pool.cc:224] Handling call
0709 12:21:36.528979 (+1869934us) spinlock_profiling.cc:243] Waited 1.7 s on lock 0x7b54002134d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:36.529126 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3913208192}
I20260709 12:21:36.530045 31827 raft_consensus.cc:2376] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7. Re-sending same reply.
W20260709 12:21:36.448134 31678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1800) took 1754 ms (client timeout 1638 ms). Trace:
I20260709 12:21:36.529604 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:36.530797 31678 rpcz_store.cc:269] 0709 12:21:34.693805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.797643 (+103838us) service_pool.cc:224] Handling call
0709 12:21:36.447957 (+1650314us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54002e8454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:36.448066 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":233135360}
I20260709 12:21:36.531323 31819 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:21:36.451195 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.452267  1969 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.453869 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 292) took 1850 ms. Trace:
I20260709 12:21:36.459393  1950 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.536870 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.536409 31818 raft_consensus.cc:2376] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8. Re-sending same reply.
I20260709 12:21:36.537683 31749 raft_consensus.cc:2368] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6.
I20260709 12:21:36.467365 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1870) took 1667 ms. Trace:
I20260709 12:21:36.467499  1900 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:36.468781  1852 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.474s	user 0.006s	sys 0.011s
I20260709 12:21:36.468986 31668 tablet_service.cc:1558] Processing DeleteTablet for tablet 2c162dbe81714319bfeccd9d7cb7bfea with delete_type TABLET_DATA_TOMBSTONED (TS b95aeeaf4c51442b8092cff1de674f2b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:43180
I20260709 12:21:36.543969 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:36.544904 32378 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
W20260709 12:21:36.469718  1932 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.147s	user 0.006s	sys 0.006s
I20260709 12:21:36.472612  1935 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.472905 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:36.473408 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:36.549817 31822 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:36.474002 31752 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260709 12:21:36.474449 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.474712 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:36.475183 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:36.475261 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.383911  1581 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:36.396694  1803 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:36.551239 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 76929.2us
I20260709 12:21:36.387379  1930 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.476696 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 265) took 1112 ms. Trace:
I20260709 12:21:36.558861 31825 rpcz_store.cc:276] 0709 12:21:35.363997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.364282 (+   285us) service_pool.cc:224] Handling call
0709 12:21:35.422343 (+ 58061us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:36.330742 (+908399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:36.476429 (+145687us) spinlock_profiling.cc:243] Waited 908 ms on lock 0x7b54002080d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:36.476619 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2087564032}
I20260709 12:21:36.559415 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.401799 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.560372 31669 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:36.486506 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:36.488019 31670 raft_consensus.cc:606] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:36.500609 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:36.247622  1312 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.509034 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:36.507437  1722 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:36.511015 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 83247.9us
I20260709 12:21:36.510807  1842 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260709 12:21:36.568064 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.401142  1980 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.500172  1908 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:36.511754  1844 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:21:36.511946  1725 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:36.512815 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 308) took 1150 ms. Trace:
W20260709 12:21:36.510696 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 97741.3us
I20260709 12:21:36.514178  1874 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:36.444254  1931 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:36.463729 31573 rpcz_store.cc:269] 0709 12:21:16.550104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:16.552392 (+  2288us) service_pool.cc:224] Handling call
0709 12:21:36.462942 (+19910550us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:28.519389 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:28.742616 (+223227us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:28.751344 (+  8728us) write_op.cc:435] Acquired schema lock
0709 12:21:28.752627 (+  1283us) tablet.cc:662] Decoding operations
0709 12:21:28.819130 (+ 66503us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:28.843303 (+ 24173us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:28.864578 (+ 21275us) tablet.cc:685] Acquiring locks for 9 operations
0709 12:21:28.865472 (+   894us) tablet.cc:701] Row locks acquired
0709 12:21:28.865518 (+    46us) write_op.cc:260] PREPARE: finished
0709 12:21:28.865706 (+   188us) write_op.cc:270] Start()
0709 12:21:28.865881 (+   175us) write_op.cc:276] Timestamp: P: 1783599688865639 usec, L: 0
0709 12:21:28.865923 (+    42us) op_driver.cc:348] REPLICATION: starting
0709 12:21:28.866603 (+   680us) log.cc:844] Serialized 3658 byte log entry
0709 12:21:29.906806 (+1040203us) op_driver.cc:464] REPLICATION: finished
0709 12:21:30.923387 (+1016581us) write_op.cc:301] APPLY: starting
0709 12:21:30.924683 (+  1296us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:31.057611 (+132928us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:31.282891 (+225280us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:31.777506 (+494615us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:31.777914 (+   408us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0709 12:21:31.777967 (+    53us) write_op.cc:312] APPLY: finished
0709 12:21:31.778388 (+   421us) log.cc:844] Serialized 71 byte log entry
0709 12:21:32.590804 (+812416us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:32.603017 (+ 12213us) write_op.cc:454] Released schema lock
0709 12:21:32.629316 (+ 26299us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:32.769055 (+139739us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:32.780672 (+ 11617us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":3949173,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":978229,"apply.run_cpu_time_us":60647,"apply.run_wall_time_us":1825553,"num_ops":9,"prepare.queue_time_us":172997,"prepare.run_cpu_time_us":39177,"prepare.run_wall_time_us":547791,"raft.queue_time_us":241925,"raft.run_cpu_time_us":28675,"raft.run_wall_time_us":903886,"replication_time_us":966340,"thread_start_us":1878742,"threads_started":5,"wal-append.queue_time_us":972064,"wal-append.run_cpu_time_us":19143,"wal-append.run_wall_time_us":2975546}]]}
I20260709 12:21:36.517721  1904 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.499697  1905 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.396477  1445 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.526527 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.385345  1399 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.527130 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.598289  1842 consensus_queue.cc:1243] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 7.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:21:36.534128 31677 rpcz_store.cc:276] 0709 12:21:34.603691 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.468735 (+865044us) service_pool.cc:224] Handling call
0709 12:21:35.469095 (+   360us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:35.469312 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:35.469336 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:35.479429 (+ 10093us) log.cc:844] Serialized 26 byte log entry
0709 12:21:35.600517 (+121088us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:35.600554 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:35.600622 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:36.453596 (+852974us) raft_consensus.cc:1685] finished
0709 12:21:36.453629 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:36.453795 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":55615,"threads_started":1,"wal-append.queue_time_us":529013}
I20260709 12:21:36.644321 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.537412 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:36.389127 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:36.538771 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:36.538551 31821 rpcz_store.cc:276] 0709 12:21:34.800047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.813923 (+ 13876us) service_pool.cc:224] Handling call
0709 12:21:34.839536 (+ 25613us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:35.903916 (+1064380us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:36.467131 (+563215us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540043dad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:36.467304 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2424263040}
W20260709 12:21:36.647969 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 258922us
I20260709 12:21:36.648149 31818 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:36.648563 31823 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:36.547221 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:36.549961 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:36.650075 31827 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:36.475605  1906 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.765s	user 0.010s	sys 0.018s
I20260709 12:21:36.450205  1754 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.551512 31746 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 3: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:36.476838 31753 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:36.622802  1312 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:36.550025 31751 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:36.554334 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:36.555289 31748 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:36.665223 31818 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:36.551251 32375 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
W20260709 12:21:36.557555 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 211359us
I20260709 12:21:36.561283 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.565025 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:36.666308 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:36.666620 32377 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:36.666951 32370 raft_consensus.cc:2368] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 5. Current term is 6.
I20260709 12:21:36.588452 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.557142  1932 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.557176  1897 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:36.557480  1613 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:36.587541 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:36.604396 31750 rpcz_store.cc:276] 0709 12:21:35.361922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.362087 (+   165us) service_pool.cc:224] Handling call
0709 12:21:35.362375 (+   288us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:35.800402 (+438027us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:36.512541 (+712139us) spinlock_profiling.cc:243] Waited 425 ms on lock 0x7b5400064854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:36.512748 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":977114496}
I20260709 12:21:36.678498 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.679364 31746 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:36.679705 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:36.607661 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.680735 31750 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:36.564885 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:36.681200 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 73634.6us
I20260709 12:21:36.626199 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:36.541007  1957 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260709 12:21:36.588078  1759 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:36.549582  1852 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.545959  1950 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.624485 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.640812 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.588546  1969 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:36.707857 31822 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:36.708257 31822 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:36.596138 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:36.596720  1951 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:36.641708 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.715629 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:36.643220 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.716482 32373 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:36.716779 31675 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:36.718971 31749 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:36.721649 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.533416 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:36.647954 31677 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:36.722767  1969 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.723573 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:36.724732 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:36.725508 32376 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:36.725365  1969 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.619876  1874 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.726594  1874 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:36.703608  1979 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:21:36.555289 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:36.557682  1935 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.465646  1808 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:36.553941  1945 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.598908 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:36.670535 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.670992 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.671897 31671 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:36.671344 31717 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.672253 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.741030 31748 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:36.673383 31818 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:21:36.673287 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.743250 31751 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:36.687423 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.680150 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.744360 31746 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:36.745159 31752 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 4: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:36.745272 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:36.745997 31746 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:36.554029  1884 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.694113 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:36.702077 31821 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:21:36.663285  1752 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:36.704691 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.706900 32374 raft_consensus.cc:2368] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7.
I20260709 12:21:36.635116  1904 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.689849  1854 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:36.709046 31822 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:21:36.714205  1849 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.007s	sys 0.023s
I20260709 12:21:36.702898  1759 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:36.690471  1983 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 12:21:36.735250 31675 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:36.738123 31749 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:36.740288 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 141471us
I20260709 12:21:36.788223 31751 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:36.746973 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.790872 31753 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:36.685708  1844 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:36.717382 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:36.748667  1951 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:36.795943 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:36.748315 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:36.801525 31746 raft_consensus.cc:1813] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Handling vote request from an unknown peer b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.802018 31746 raft_consensus.cc:2368] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6.
I20260709 12:21:36.803177 31746 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:21:36.667413  1906 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.752545 32372 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:21:36.774178  1849 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.008s	sys 0.025s
I20260709 12:21:36.676003  1908 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:36.815510  1988 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
W20260709 12:21:36.817643  1984 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
I20260709 12:21:36.824023  1945 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.829787  1884 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:36.773006  1735 raft_consensus.cc:2764] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260709 12:21:36.840204  1980 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:36.844393  1803 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:36.849344 31747 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 12:21:36.851437  1986 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.145s	user 0.000s	sys 0.000s
I20260709 12:21:36.853061  1996 tablet_replica.cc:333] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
W20260709 12:21:36.854277  1992 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:21:36.854198 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.789928  1752 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:36.856168  1576 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:36.856210  1752 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.858762  1780 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:36.859304  1939 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:36.859822  1939 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:36.860718  1939 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.862057  1939 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:36.792924  1936 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:36.870076 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 289) took 2338 ms. Trace:
I20260709 12:21:36.897475 31642 leader_election.cc:400] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6.
I20260709 12:21:36.896636 31674 rpcz_store.cc:276] 0709 12:21:34.532039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.432470 (+900431us) service_pool.cc:224] Handling call
0709 12:21:36.451564 (+1019094us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:36.451837 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:36.598288 (+146451us) log.cc:844] Serialized 14 byte log entry
0709 12:21:36.650424 (+ 52136us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:36.650503 (+    79us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:36.650569 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:36.650738 (+   169us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:36.869830 (+219092us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002e8450. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:36.869991 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":231155,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":23879,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":19649,"spinlock_wait_cycles":2342876544,"thread_start_us":197582,"threads_started":2,"wal-append.queue_time_us":201292}
I20260709 12:21:36.898876 31670 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:36.872012 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1729) took 1202 ms. Trace:
I20260709 12:21:36.899951 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.875175 31749 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:21:36.888924 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.904731 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:36.905220 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:36.905644 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.906296 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.890101 31750 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:36.767938 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:36.907964 32375 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:36.699779  1613 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:36.674285  1977 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20260709 12:21:36.771764 31822 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:36.774760 31752 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
W20260709 12:21:36.748849  1985 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.212s	user 0.001s	sys 0.000s
I20260709 12:21:36.778005 31821 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: [6.3-6.3]
W20260709 12:21:36.779201 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74572.1us
I20260709 12:21:36.935385 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.935879 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.937300 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.754273 32377 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
W20260709 12:21:36.785161 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 114713us
I20260709 12:21:36.685601  1443 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.067s
I20260709 12:21:36.767483 32371 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:36.793282 31669 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:36.956444 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:36.957218 31827 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:36.957618 31825 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:36.957764 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.958151 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:36.700999  1851 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20260709 12:21:36.958655 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:36.958932  1851 consensus_queue.cc:1243] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:21:36.958711  1716 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:36.959383 31825 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:36.959537  1851 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:36.959975  1851 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:36.960862  1851 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.962075  1851 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:36.771031  1854 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:36.962813  1963 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:36.963662  1959 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:21:36.906908 31792 leader_election.cc:336] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.636s (SENT)
W20260709 12:21:36.964318 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 57446.7us
I20260709 12:21:36.964774  1808 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:36.965492  1808 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:36.888172  1951 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:36.959569 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:36.966564  1969 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.967064  1969 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.967602 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.968840 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.969954 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.825486  1931 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.103s
I20260709 12:21:36.971312  1931 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:36.971666  1931 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:36.972456  1931 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.973134 31822 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 12:21:36.973706 31827 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 6
W20260709 12:21:36.870972  1987 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.199s	user 0.001s	sys 0.000s
I20260709 12:21:36.899688 31823 rpcz_store.cc:276] 0709 12:21:35.669450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.669664 (+   214us) service_pool.cc:224] Handling call
0709 12:21:35.669957 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:36.649188 (+979231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:36.871754 (+222566us) spinlock_profiling.cc:243] Waited 973 ms on lock 0x7b54001671d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:36.871946 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2237625088}
W20260709 12:21:36.978519 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.979274 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:36.980808 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:36.981529 31823 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:36.903479 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:36.983392 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 79986.2us
I20260709 12:21:36.984290 31642 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:36.984517 31791 leader_election.cc:400] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:21:36.800547  1915 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:36.914310 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:36.915802 31670 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:21:36.868947  1932 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.939157 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:36.942651 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:36.944269 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.876624  1445 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:36.949954 31753 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:36.902486  1884 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:36.951431 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:37.004040 31749 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:36.953009 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:37.004912 31821 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:36.954025  1443 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:21:36.955335 32370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 452) took 1605 ms (client timeout 1636 ms). Trace:
I20260709 12:21:36.886116 31745 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
W20260709 12:21:37.005931 32370 rpcz_store.cc:269] 0709 12:21:35.349695 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.367432 (+ 17737us) service_pool.cc:224] Handling call
0709 12:21:36.955111 (+1587679us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b540074e854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:36.955273 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2616085248}
W20260709 12:21:37.004938 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 65867.7us
I20260709 12:21:36.973498 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:37.010100 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:37.011147 31753 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
W20260709 12:21:37.000272  1751 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.664s	user 0.008s	sys 0.002s
I20260709 12:21:36.976972 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:36.977144 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:36.986004 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:36.870239  1935 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.992039 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:37.018877 31670 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:37.019465  1915 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:37.019873 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:37.020045  1915 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:36.997462  1735 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:36.996657 31825 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:36.879357  1399 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.878748  1905 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:36.955348  1960 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.768s	user 0.010s	sys 0.001s
I20260709 12:21:36.877345  1754 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:37.050659  1754 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.050876  1759 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.051219  1754 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.872645  1904 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:36.753443 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 59470.1us
I20260709 12:21:36.999946 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:36.973158  1990 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.226s	user 0.004s	sys 0.000s
I20260709 12:21:36.878077  1900 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:37.007995  1884 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.888016  1852 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:36.889814  1906 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:37.006295 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.985620  1931 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:36.976828  1950 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:37.009706 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:36.983300  1936 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.076836 31745 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:36.973551 31822 pending_rounds.cc:85] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 4
I20260709 12:21:37.077710 31822 pending_rounds.cc:107] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:21:37.077908 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:37.077986 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:37.010162 31752 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.016498 31669 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:37.017761  1945 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:37.019045  1613 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:36.991039  1808 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.022811 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:36.997258  1979 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:36.983693  1817 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 12:21:37.012912  1981 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:37.023921 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:37.106350 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:37.107227 31821 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.024760 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:36.866621  1878 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.025250 31827 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:36.993808 31820 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:37.113895 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 166756us
I20260709 12:21:37.034353  1996 raft_consensus.cc:2243] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:21:36.976929  1874 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.024907  1312 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:37.007915  1443 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:37.031255 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:37.046514 31748 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
W20260709 12:21:37.040752  1727 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.648s	user 0.010s	sys 0.000s
W20260709 12:21:37.053422 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.055068 31823 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:37.123826 31748 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:37.124919 31823 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:37.026153  1735 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:36.859359  1780 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:37.018910  1911 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260709 12:21:37.078023 31822 raft_consensus.cc:2889] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:37.128198 31823 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:37.083060 32370 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:37.025285  1751 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.095821  1759 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:37.111975 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 89225.8us
I20260709 12:21:37.117663 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:37.098114 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:37.056335  1960 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.134538 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:37.121066 31820 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:37.146260 31827 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:37.141140 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:37.146454 31819 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:37.075233  1920 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260709 12:21:37.097946  1906 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.047698  1951 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.115496  1904 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:37.083840  1936 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.147650 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:37.133744  1735 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:37.114528  1808 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.136744  1981 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:37.148085 32371 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.148869  1817 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:37.148263  1999 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:37.190946  1999 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:37.176476  1443 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.191915  1999 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.192327  1754 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:37.193209  1999 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:37.193478  1936 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:37.194083  1884 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:37.196038  1874 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:37.196979  1960 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:37.198108  1878 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:37.142944  1975 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 7 current term: 7 is higher.
I20260709 12:21:37.189276  1751 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:37.170689  1906 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:37.203361  1808 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:37.205495  1979 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:37.206420  1915 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:37.197014  1780 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.148373  1932 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 12:21:37.214046  1981 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:37.213070  1725 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.592s	user 0.011s	sys 0.000s
I20260709 12:21:37.225852  1786 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.227526  1786 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.230295  1906 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:21:36.928941 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:37.021601 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:37.266508 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:21:37.057763 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:37.270046 31676 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:37.272148 31675 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:37.272214 31678 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:37.269465 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:37.273429 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:37.274395 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 203360us
I20260709 12:21:37.276367 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:37.033001 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:37.278291 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 245245us
I20260709 12:21:37.279735 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:37.282040 31672 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 4: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:37.283948 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:36.993382 31642 leader_election.cc:400] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6.
I20260709 12:21:37.109354 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:37.287406 31642 leader_election.cc:403] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:37.111992 31745 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:37.288508 31749 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:37.288950 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.291695 31745 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:37.292249 31749 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:37.147328 31818 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:37.300864 31818 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
W20260709 12:21:37.287246 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 239294us
I20260709 12:21:37.303143 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:37.302178 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:37.150828 31820 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:37.307576 31820 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:21:37.303875 31818 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:21:37.310207 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:37.310794 31826 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260709 12:21:37.148497 31822 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:21:37.314473 31825 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:37.321727 31669 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:21:37.303810 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:37.324358 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:37.326328 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:37.327567 31673 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:37.335047 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:37.337985 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:37.339032 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:37.339807 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:37.340775 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:37.343401 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 195676us
I20260709 12:21:37.344873 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:37.345120 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:37.346295 32370 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.348915  1727 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:37.352581  1727 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:37.353770  1727 raft_consensus.cc:2706] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Received pre-election callback for term 5 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20260709 12:21:37.354991  1735 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.361634  1817 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:21:37.362609 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.417093 32370 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:37.353972  1399 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 12:21:37.433388 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:37.418543  1817 raft_consensus.cc:2764] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20260709 12:21:37.439167 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 94640.3us
W20260709 12:21:37.443426 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 450155us
I20260709 12:21:37.448992  2009 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:37.442967  1852 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:37.452056  1897 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:37.454094  1907 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:37.454608  1915 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
W20260709 12:21:37.457095 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.459515  1996 raft_consensus.cc:2272] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Raft consensus is shut down!
W20260709 12:21:37.459826 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 158173us
I20260709 12:21:37.460455 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:37.453243 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:37.443858  1735 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:37.475178  2002 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:37.476228  1399 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:37.476791  1944 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:37.479496  1980 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:37.454253  1906 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:21:37.480542 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 141800us
I20260709 12:21:37.470734 31677 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:37.449237  1911 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:37.470548 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 194538us
I20260709 12:21:37.489799 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:37.492084 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:37.446367  1931 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.054s
I20260709 12:21:37.499162 31821 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:37.501979 32371 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:37.467038  1897 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.503187 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:37.504348 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:37.506072 31674 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:37.516950 31747 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:37.508019  1944 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:37.521011 31752 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:37.532850 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1910) took 1169 ms. Trace:
I20260709 12:21:37.533365 31823 rpcz_store.cc:276] 0709 12:21:36.363496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:37.130534 (+767038us) service_pool.cc:224] Handling call
0709 12:21:37.135303 (+  4769us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:37.135633 (+   330us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:37.135673 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:37.141088 (+  5415us) log.cc:844] Serialized 26 byte log entry
0709 12:21:37.467869 (+326781us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:37.509832 (+ 41963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.516546 (+  6714us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:37.516612 (+    66us) raft_consensus.cc:1685] finished
0709 12:21:37.516634 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:37.532633 (+ 15999us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":112968,"threads_started":1,"wal-append.queue_time_us":127041}
W20260709 12:21:37.537554 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.520741  1944 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.521370  1735 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:37.522361 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:37.516182  1399 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.516530  2002 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:37.516008  1931 consensus_queue.cc:1243] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 6.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:21:37.540518  1999 raft_consensus.cc:2764] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:21:37.512131  1907 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:37.557247  1907 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.561787  1931 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.562508  1931 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.563772 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1833) took 1085 ms. Trace:
I20260709 12:21:37.564214 31750 rpcz_store.cc:276] 0709 12:21:36.478514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.687162 (+208648us) service_pool.cc:224] Handling call
0709 12:21:37.563344 (+876182us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54000b86d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:37.563596 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":335725440}
W20260709 12:21:37.523351 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96195.5us
I20260709 12:21:37.523932 31822 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:37.524354  1920 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:37.525131 31675 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:37.525727  1852 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:37.525527  2015 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:37.570106 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:37.575652  2015 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:37.576196  2015 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:37.576926 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:37.526805  2013 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:37.543145  1627 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:37.577574  2015 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.526811 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:37.579176 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52565.6us
I20260709 12:21:37.579671  1904 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:37.580116 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:37.529819  1979 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:37.580627  1627 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.581686  1979 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:37.582168  1979 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:37.582917  1957 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:37.583364  1957 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:37.583468  1979 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.585080  1979 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:37.526993  1980 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260709 12:21:37.530155 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:37.583972  1780 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:37.584556  1957 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.587126  1950 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:37.587611  1957 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.587821  1950 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:37.553534  2002 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.589633  2015 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:37.589269  1950 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.590083  1969 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:37.590518 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60566.5us
I20260709 12:21:37.593005  1996 ts_tablet_manager.cc:1916] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:21:37.577114  1920 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:37.594449  1851 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:37.580484 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:37.595021  1851 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.595139  1920 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:37.595517  1851 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.596761 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:37.596666  1920 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.597851  1980 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:37.541360  1735 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:37.599155  1735 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:37.599637 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:37.600680  1627 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:37.600360  1735 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.601567  1904 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:37.602052  1904 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.598366  1980 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:37.602458  1904 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:37.602555  1844 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:37.602880  1980 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:37.603519  1844 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:37.604375  1842 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:37.605474  1842 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:37.605605 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:37.605979  1842 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:37.595088  2012 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20260709 12:21:37.607714  2012 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:37.608084  2012 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:37.609135  2012 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.609962  1844 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.610188 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1904) took 1255 ms. Trace:
I20260709 12:21:37.610563  2012 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:37.610729  1844 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.612846 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 273) took 1159 ms. Trace:
I20260709 12:21:37.613306 31819 rpcz_store.cc:276] 0709 12:21:36.453069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.532086 (+ 79017us) service_pool.cc:224] Handling call
0709 12:21:37.612320 (+1080234us) spinlock_profiling.cc:243] Waited 496 ms on lock 0x7b540043e9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:37.612618 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1141796224}
I20260709 12:21:37.614952  1399 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:37.615649 32372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 443) took 2958 ms (client timeout 1536 ms). Trace:
I20260709 12:21:37.615446  1842 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:37.616150 32372 rpcz_store.cc:269] 0709 12:21:34.656726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:34.661675 (+  4949us) service_pool.cc:224] Handling call
0709 12:21:37.615205 (+2953530us) spinlock_profiling.cc:243] Waited 2.02 s on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:37.615420 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4652562688}
I20260709 12:21:37.511608  1906 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:37.620525 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.582196 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:37.618664 31826 rpcz_store.cc:276] 0709 12:21:36.355239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.851237 (+495998us) service_pool.cc:224] Handling call
0709 12:21:36.851681 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:37.311388 (+459707us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.609782 (+298394us) spinlock_profiling.cc:243] Waited 458 ms on lock 0x7b540043d5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:37.610111 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1054043392}
I20260709 12:21:37.605906  1443 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.616936  1842 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.626055  1906 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:37.652271 32370 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:21:37.658298 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.639616  1931 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:37.648018  1735 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:37.553937 31572 catalog_manager.cc:5697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:21:37.671273 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 92494.2us
W20260709 12:21:37.673739 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 94180.8us
I20260709 12:21:37.666844  1627 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:37.685814 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.686015 31750 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:37.685058 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:37.698941 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 122188us
W20260709 12:21:37.700446 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.700325 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:37.701390 31746 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:37.702278  1975 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:37.703058  1975 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.705508  1975 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:37.690804  1900 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:37.707087  1752 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.706938  1980 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.713382  1752 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.715093  1980 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:37.715967  1752 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:37.716919 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 134890us
I20260709 12:21:37.717640 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1381) took 1245 ms. Trace:
I20260709 12:21:37.718878 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:37.719317 31753 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:37.720567 31718 leader_election.cc:400] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7.
I20260709 12:21:37.720927 31718 leader_election.cc:403] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:37.721648 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:37.722451 31749 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.722803  1627 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.725153  1627 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:37.725921  1544 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:37.726312  1544 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:37.727181 31748 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:21:37.728312 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:37.728960 31748 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:37.730620 31751 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:37.730813  1919 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:37.731225  1919 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:37.730985 31572 catalog_manager.cc:5697] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 reported cstate change: leader changed from <none> to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 4 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:37.732306  1919 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.733493  1919 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:37.733079 31572 catalog_manager.cc:5697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:37.707960 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:37.738854 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.718094 31744 rpcz_store.cc:276] 0709 12:21:36.472222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.481831 (+  9609us) service_pool.cc:224] Handling call
0709 12:21:37.717259 (+1235428us) spinlock_profiling.cc:243] Waited 887 ms on lock 0x7b54001ed2d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:37.717436 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2039648640}
I20260709 12:21:37.743893 31750 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:37.688396  1852 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:37.751838 31752 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:21:37.760290 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:37.720597 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:37.763607 31826 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 12:21:37.764068 31745 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:37.769009 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:37.775554 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:37.777491 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:37.687737  2023 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.080s
I20260709 12:21:37.780467 31746 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:37.752997  1931 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:37.782739 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:37.781369 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:37.795455 31753 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:37.800879 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:37.802001 31826 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
W20260709 12:21:37.803277  2024 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:21:37.807339  1576 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.115s
W20260709 12:21:37.813225 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47875.6us
I20260709 12:21:37.814844  2015 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:37.807214  1544 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.806186  1844 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:37.827569 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:37.830729 31749 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
W20260709 12:21:37.831681 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.834873 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:37.835536 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:37.839775 31748 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:37.851864 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:37.852723 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 132412us
I20260709 12:21:37.797770  1897 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.857547  1897 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:37.864665 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:37.865253  1851 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.867601  2031 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:37.868937 31749 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:37.869544  2031 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:37.871924 31753 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:37.866430 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:37.871340  1851 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:37.879251 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:37.880281 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44550.9us
W20260709 12:21:37.880614 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.882485 31745 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:37.882385  1727 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:37.882897 31572 catalog_manager.cc:5697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 2 to 4, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 4 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:37.884086 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1835) took 1405 ms. Trace:
I20260709 12:21:37.884572 31751 rpcz_store.cc:276] 0709 12:21:36.479071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.789237 (+310166us) service_pool.cc:224] Handling call
0709 12:21:37.730274 (+941037us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:37.731116 (+   842us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.883866 (+152750us) spinlock_profiling.cc:243] Waited 940 ms on lock 0x7b54001ed2d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:37.883992 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2163022720}
I20260709 12:21:37.889446 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:37.848107  1900 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7.
W20260709 12:21:37.892012 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.897858  1907 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:37.898521  1907 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.900945  1907 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:37.901505  1907 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:37.902259 31717 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:37.902168  1907 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:21:37.903231  1897 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:37.903717  1897 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.904070  1897 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:37.910862  2013 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260709 12:21:37.882500  1884 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:37.911906  1884 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.912019 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:37.912366  1884 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:37.913232  1907 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:37.913650  1907 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:37.914023  1907 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:37.915793  1897 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:37.916446 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1846) took 1434 ms. Trace:
I20260709 12:21:37.917259  1852 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.918655  1897 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:37.916894 31747 rpcz_store.cc:276] 0709 12:21:36.481570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.850225 (+368655us) service_pool.cc:224] Handling call
0709 12:21:36.850510 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:37.539125 (+688615us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:37.916111 (+376986us) spinlock_profiling.cc:243] Waited 559 ms on lock 0x7b54002959d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:37.916361 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1285994368}
I20260709 12:21:37.884986  1844 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:37.830540  2015 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.100s
I20260709 12:21:37.927451 31572 catalog_manager.cc:5697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 3 to 4, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 4 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:37.846616  1904 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:37.944510 31676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1295) took 1646 ms (client timeout 1858 ms). Trace:
I20260709 12:21:37.921454  2023 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260709 12:21:37.960605 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66533.7us
W20260709 12:21:37.961113 31676 rpcz_store.cc:269] 0709 12:21:36.288926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.423399 (+134473us) service_pool.cc:224] Handling call
0709 12:21:37.918553 (+1495154us) spinlock_profiling.cc:243] Waited 771 ms on lock 0x7b54001b35d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:37.935043 (+ 16490us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1773066624}
I20260709 12:21:37.909556  1854 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:37.962754 31572 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 1 to 2, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:37.981696 31826 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
W20260709 12:21:37.981909 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 206439us
I20260709 12:21:37.896742  1906 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:37.983747 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.878840  1950 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.096s
W20260709 12:21:37.986680 31671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1805) took 3243 ms (client timeout 2106 ms). Trace:
I20260709 12:21:37.946907  1443 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
W20260709 12:21:37.990118 31671 rpcz_store.cc:269] 0709 12:21:34.740495 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:35.315418 (+574923us) service_pool.cc:224] Handling call
0709 12:21:37.977111 (+2661693us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54001d6354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:37.983931 (+  6820us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3116084096}
W20260709 12:21:37.992360 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:37.994200 31677 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:21:37.977303  1975 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:37.996619 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 130399us
I20260709 12:21:37.995311 31748 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:21:37.986581  1878 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:37.991784 31572 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 4 to 5, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 5 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:37.984898  1907 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:38.014807 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:38.010901 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:37.992229  1979 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
W20260709 12:21:38.019147 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:38.018903  1752 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:38.065146  1979 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:38.065554  1979 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:38.066709  2013 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:38.019752  1981 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.020798  1921 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260709 12:21:38.024928  1765 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:38.075197 31669 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:21:38.025833 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:38.025974  1443 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:21:38.037904  1884 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.040035 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:37.999598  1851 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
I20260709 12:21:38.040356 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:38.043021  1544 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:38.043931 31572 catalog_manager.cc:5697] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 2 to 6, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 6 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:38.051246 31751 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:38.055814  1906 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:38.061569  2031 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:38.061966 31825 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:38.011570  1854 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:38.014672  1975 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.005375  1878 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:38.073135 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:38.091910 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 51761us
I20260709 12:21:38.024289 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.074285 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:38.074261 31672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1290) took 1786 ms (client timeout 1711 ms). Trace:
I20260709 12:21:38.088315 31642 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:38.078832 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:38.078979 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:38.066725 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:38.083729 32370 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 4: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:38.069309  1931 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:38.089035 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:38.078060 31746 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:38.094208 31821 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:38.094950 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 89548.1us
I20260709 12:21:38.097991  1780 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.098591  1780 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.099686  1735 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:38.100142  1735 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:21:38.093057 31672 rpcz_store.cc:269] 0709 12:21:36.287781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.373284 (+ 85503us) service_pool.cc:224] Handling call
0709 12:21:38.073943 (+1700659us) spinlock_profiling.cc:243] Waited 890 ms on lock 0x7b540017bbd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:38.074166 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2048119680}
I20260709 12:21:38.101024  1780 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:38.066313  1979 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.102236 31745 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:38.104151 31822 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:38.072186  2015 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.024s
I20260709 12:21:38.108567 31572 catalog_manager.cc:5697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 2 to 3, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.067333  2013 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.104206  1844 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.111200 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:38.111532 31747 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:38.111769 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 86016.8us
I20260709 12:21:38.112273  1975 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:38.112985  1727 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:38.113544  1727 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:38.114178  1765 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:38.114854  1765 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:38.114562  1727 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.116609 31753 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:38.116529 32371 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260709 12:21:38.117424 31826 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:21:38.121873 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:38.104532  1627 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.125721  1803 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:38.125917 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:38.126502 31677 pending_rounds.cc:85] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 2
I20260709 12:21:38.126271  1627 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:38.126948 31677 pending_rounds.cc:107] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 6.3
I20260709 12:21:38.127059  2018 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:38.127300 31677 raft_consensus.cc:2889] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:38.127609  2018 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:38.092082  1981 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.128764  1627 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:38.128989 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:38.129745 31670 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:38.130852 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:38.125924 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:38.131702 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 107500us
I20260709 12:21:38.101120  1735 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:38.137712 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:38.139338 31670 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:38.141278  1981 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:38.131071  2031 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:21:38.123549  1752 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:38.135682  1980 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:38.142720  1878 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.144209  1735 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:38.116086  1765 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:38.150321 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:38.111497  1844 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.150862 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:38.151029  1979 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:38.151981  1907 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:38.154768 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.155045  1786 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:38.159941 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:38.138695  1975 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.163416  2013 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:38.163244 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.164258 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:38.165416  2035 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.302s	user 0.002s	sys 0.002s
I20260709 12:21:38.165050  1931 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:21:38.165045  1999 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:38.167016 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:38.132021 31671 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:38.216269 31745 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:38.166673 31572 catalog_manager.cc:5697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 3 to 4, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 4 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.178941 31747 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:38.147408 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 309) took 1256 ms. Trace:
I20260709 12:21:38.188767 31748 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260709 12:21:38.186308 31825 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:38.194736  1980 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:38.139017  1921 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260709 12:21:38.160813  1904 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:38.187878 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:38.197870 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:38.204973 31749 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:38.198343 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:38.209697  1920 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:21:38.210108 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46979.5us
I20260709 12:21:38.207437 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:38.202911 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:38.222797 31822 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:38.218986 31678 rpcz_store.cc:276] 0709 12:21:36.894704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.896718 (+  2014us) service_pool.cc:224] Handling call
0709 12:21:36.897131 (+   413us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:37.273157 (+376026us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:38.147040 (+873883us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b54000474d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:38.147325 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":860868480}
I20260709 12:21:38.225700 31823 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:38.167506  1752 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:38.233951 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:38.234733 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:38.217049  1842 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.569s	user 0.010s	sys 0.005s
I20260709 12:21:38.240484 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:38.224918 32375 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:38.242071 31671 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 8 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:21:38.243387 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:38.217338  1878 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:38.229068 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:38.194106  1852 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.038s
I20260709 12:21:38.224223  1921 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.255288 32370 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:38.257337 31821 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
W20260709 12:21:38.257185 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 102600us
W20260709 12:21:38.257987 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:38.258586 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:38.259361 31821 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:38.261695 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:38.262053 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:38.263491 31821 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:38.258443 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.230127  2049 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.267159 31823 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:38.245303  1443 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:38.241304 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:38.208585  1950 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260709 12:21:38.260833 31572 catalog_manager.cc:5697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 2 to 5, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 5 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.274799 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.273576  1443 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:38.280505 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.280449  1443 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.281092 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:38.281958  1443 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:38.282153  1842 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:38.264891  1904 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:38.283154  1842 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:38.284215  1884 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:38.285250 32372 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:38.268345  1920 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:38.282858  1786 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:38.283156  1904 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:38.288213  1904 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.288579  1904 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:38.291595  1844 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:38.291795 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:38.292639 31825 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:38.293576 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:38.294202 31825 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:38.295333 31572 catalog_manager.cc:5697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 4 to 6, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 6 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.300376  1751 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.302904  1980 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:38.307111  2032 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.321s	user 0.002s	sys 0.000s
I20260709 12:21:38.311429 32374 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:38.313022 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:38.314035 31826 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 3.
I20260709 12:21:38.315941 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:38.264008  1921 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:38.281982 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.317659 31572 catalog_manager.cc:5697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 1 to 4, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 4 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:38.228637  1975 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:38.316905 31824 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:38.349081 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:38.350108 32375 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:38.350274 31669 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
W20260709 12:21:38.351341 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 76691.9us
I20260709 12:21:38.350466  2031 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:38.352316 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.352579  2031 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:38.353922 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:38.355521 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:38.356201 31827 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:38.359133 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:38.360044  2012 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:38.360435  2012 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:38.326736  1950 consensus_queue.cc:1243] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:21:38.364776 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:38.364530  1904 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:38.368525  2031 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:38.369369  1981 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.369704  2031 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:38.369975  1981 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:38.371059 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:38.372426  1981 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:38.373121 31744 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:38.373293 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:38.252483  1907 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.075s
I20260709 12:21:38.376502 31818 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:38.397891 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:38.320501 32371 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:38.417171 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:38.425820 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 61148us
W20260709 12:21:38.422873 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:38.397890 31748 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
W20260709 12:21:38.428979 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 53357.1us
I20260709 12:21:38.429227 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.321482 31572 catalog_manager.cc:5697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 5 to 6, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 6 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:21:38.438969 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:38.439605 31826 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:38.373134  1844 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:21:38.448689 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:38.438079 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:38.452114 31750 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:38.406674  2013 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.453631 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:38.457716 32372 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 6: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:38.462778 31821 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
W20260709 12:21:38.462924 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 181017us
I20260709 12:21:38.459306 31671 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260709 12:21:38.465549 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 113378us
I20260709 12:21:38.468116 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:38.440266  2015 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:38.475260 31672 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:21:38.454972  2002 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:21:38.438314  1884 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.038s
I20260709 12:21:38.328042  1751 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.482753 31821 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:38.480274 31572 catalog_manager.cc:5697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 3 to 5, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 5 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.459813  1907 consensus_queue.cc:1243] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:21:38.327584  2049 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:38.486286 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:38.443747  1852 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 12:21:38.494745 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:38.488261  1627 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 12:21:38.476603  2051 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.476845  1900 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:38.511090 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.515946 31744 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:21:38.486814  1999 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:38.525640 31824 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 3. Current term is 4. Ops: []
I20260709 12:21:38.532059 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.503624  1727 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:38.530265 31572 catalog_manager.cc:5697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.522610  1907 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:38.499559  1854 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.037s
I20260709 12:21:38.527073  1999 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:38.541711 31572 catalog_manager.cc:5697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.562506  2020 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:38.563915  1963 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:38.564381  1963 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:38.564625  1951 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:38.565310  1951 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:38.565449  1963 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:38.566265  2018 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:38.566797  1963 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:38.566411  1951 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.568867  1908 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:38.564407  2045 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:38.571151  2045 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:38.571504  2033 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.700s	user 0.000s	sys 0.003s
I20260709 12:21:38.579486  1904 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:38.571544  2045 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:38.585129 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 96641.5us
I20260709 12:21:38.586088 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.587793 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.589231  2012 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.590132 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:38.603772 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 92041.8us
I20260709 12:21:38.596351 31572 catalog_manager.cc:5697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 2 to 3, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.534732  1780 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:38.536767  1727 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:38.630925 31572 catalog_manager.cc:5697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:38.542749  1900 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:38.511914  2013 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:38.634325 31825 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:38.561815  1907 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:38.590590  2012 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:38.592067  1999 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:38.641198  2039 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
W20260709 12:21:38.641850 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 155503us
I20260709 12:21:38.643404  1900 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:38.634904  2027 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
W20260709 12:21:38.644793  2037 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.547s	user 0.005s	sys 0.000s
I20260709 12:21:38.618042  1844 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:38.630437  1727 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:38.631451  1897 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:38.617882  1752 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:38.606057  1915 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.047s
I20260709 12:21:38.533528  2051 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.660550  1751 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:38.664083  1803 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.538s	user 0.007s	sys 0.003s
I20260709 12:21:38.607412  1996 ts_tablet_manager.cc:1929] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
W20260709 12:21:38.655843 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 216674us
I20260709 12:21:38.677573 31826 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:38.653712  1907 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:38.670845  1921 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.731099  1897 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:38.669155  1919 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.798542 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:38.733016 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:38.739800  1751 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.671490  1975 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:21:38.744166 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:38.744622  1544 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.657s	user 0.009s	sys 0.002s
I20260709 12:21:38.744755  1945 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:38.746866  2051 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:38.750329  2018 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:38.752319  2046 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.224s	user 0.001s	sys 0.000s
I20260709 12:21:38.655262  1981 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
W20260709 12:21:38.759001 31745 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.527s	user 0.009s	sys 0.001s
I20260709 12:21:38.764617 31747 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260709 12:21:38.782764 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:38.783334 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:38.789885  1803 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:21:38.846621 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:38.851616 31827 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:38.852891 31745 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:38.783576 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:38.793231 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:38.799569 31748 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:38.856312  1613 raft_consensus.cc:2764] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:21:38.801774 31750 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
W20260709 12:21:38.804826 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:38.858196 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 176962us
I20260709 12:21:38.858698 32369 raft_consensus.cc:2410] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 7 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:21:38.859839 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:38.809301 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 323013us
W20260709 12:21:38.818111 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:38.856711  1975 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:21:38.862715 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 79250.8us
W20260709 12:21:38.908334 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44522.5us
I20260709 12:21:38.813016  1844 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.821010  1996 log.cc:1199] T 2c162dbe81714319bfeccd9d7cb7bfea P b95aeeaf4c51442b8092cff1de674f2b: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/wals/2c162dbe81714319bfeccd9d7cb7bfea
I20260709 12:21:38.911597 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:38.808573  2015 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:38.924224 31819 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
W20260709 12:21:38.914368 31678 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.001s	sys 0.009s
W20260709 12:21:38.925346 31678 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.001s	sys 0.010s
I20260709 12:21:38.923688  1722 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.820622  1945 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:38.908789  1884 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:38.892190  2051 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:38.945466 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 85787.3us
I20260709 12:21:38.948886 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:38.949388 31820 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:38.920109  1752 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.926620  1904 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:38.821971  1751 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:38.946096 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 314484us
I20260709 12:21:38.913215  1544 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
W20260709 12:21:38.944343  1735 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.743s	user 0.008s	sys 0.003s
I20260709 12:21:38.928352  2049 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:38.935148  1884 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:38.944344  1915 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:38.958892  1996 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:43180 (request call id 88) took 2497 ms. Trace:
I20260709 12:21:38.959551 31827 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260709 12:21:38.960942 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49439.4us
I20260709 12:21:38.847119  1919 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:38.963752  1945 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.967124 31751 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260709 12:21:38.974952 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:38.976198 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:38.932265  1844 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:38.954795  2002 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:38.955027  1979 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:38.956599  1975 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:38.984205 31745 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:38.973022  1884 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:39.008903 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:39.009443 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:39.009958  2066 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:21:39.011611  2054 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:21:38.995332  1803 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.012506  1803 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:39.013744  1977 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:39.015842 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:38.946658  1722 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.009620 32377 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:39.022315  2066 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:39.023792  2013 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:39.023423  2066 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:39.024554  1849 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 12:21:39.025259  1921 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:39.026340  1849 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.026963  1849 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.001896  1901 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260709 12:21:38.962998  1752 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.038816  2015 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:38.990320  1878 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.738s	user 0.010s	sys 0.002s
I20260709 12:21:39.041237 31675 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:21:38.975675  1996 rpcz_store.cc:276] 0709 12:21:36.461768 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.468703 (+  6935us) service_pool.cc:224] Handling call
0709 12:21:36.809161 (+340458us) ts_tablet_manager.cc:1151] Deleting tablet 2c162dbe81714319bfeccd9d7cb7bfea
0709 12:21:36.852613 (+ 43452us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:21:38.012427 (+1159814us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:38.593046 (+580619us) tablet_metadata.cc:665] Metadata flushed
0709 12:21:38.958738 (+365692us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":531409,"tablet-delete.queue_time_us":225442,"thread_start_us":39959,"threads_started":1}
I20260709 12:21:38.956837  2051 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:39.093482 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 91058.3us
I20260709 12:21:39.094002 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:39.103669  1544 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.004s
I20260709 12:21:39.108356  1735 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:39.127971  1735 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:39.128566  1735 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:38.996510  2002 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.092737  1975 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:39.132043  2053 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.560s	user 0.000s	sys 0.003s
I20260709 12:21:39.132375  1842 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.132953  1842 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.093232  1979 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:39.107234  1851 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.156759  2061 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.157542  2061 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.159993  1979 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:39.156801  1851 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:39.167510 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1845) took 1054 ms. Trace:
W20260709 12:21:39.184335 32369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 441) took 2616 ms (client timeout 1937 ms). Trace:
I20260709 12:21:39.185165  1945 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:39.182825 31676 rpcz_store.cc:276] 0709 12:21:38.113498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.647856 (+534358us) service_pool.cc:224] Handling call
0709 12:21:38.648363 (+   507us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:38.648895 (+   532us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:38.648939 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:38.649699 (+   760us) log.cc:844] Serialized 26 byte log entry
0709 12:21:38.974719 (+325020us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:38.974760 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:38.974836 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:39.167293 (+192457us) raft_consensus.cc:1685] finished
0709 12:21:39.167310 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:39.167445 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2440,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":3351,"thread_start_us":324499,"threads_started":1,"wal-append.queue_time_us":484950}
I20260709 12:21:39.189216 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:39.191397 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:39.184965 32369 rpcz_store.cc:269] 0709 12:21:36.568024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.595956 (+ 27932us) service_pool.cc:224] Handling call
0709 12:21:39.183954 (+2587998us) spinlock_profiling.cc:243] Waited 2.17 s on lock 0x7b54004cfed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:39.184235 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5001233280}
I20260709 12:21:39.212430 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:39.214820 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 247300us
I20260709 12:21:39.215794 32370 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:39.201265 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.011s	sys 0.007s
W20260709 12:21:39.219635 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.011s	sys 0.007s
W20260709 12:21:39.225455 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 243711us
I20260709 12:21:39.223631 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:39.231175 31674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.562s	user 0.007s	sys 0.008s
W20260709 12:21:39.231581 31674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.007s	sys 0.008s
W20260709 12:21:39.236984 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 240174us
I20260709 12:21:39.212584  1544 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:39.212268  1931 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:39.242220 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:39.242363 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:39.248186 31823 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260709 12:21:39.249171 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 659071us
I20260709 12:21:39.222486  1627 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:39.259891 31673 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260709 12:21:39.260478 31673 pending_rounds.cc:85] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 3
I20260709 12:21:39.260754 31673 pending_rounds.cc:107] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260709 12:21:39.261042 31673 raft_consensus.cc:2889] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:39.229803  1844 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:39.269728 31750 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:39.282326 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:39.283407 31818 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:39.247406  1751 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:39.291473 31821 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:21:39.326779 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:39.328306 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 316) took 1013 ms. Trace:
I20260709 12:21:39.262775 31561 catalog_manager.cc:5028] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): tablet 2c162dbe81714319bfeccd9d7cb7bfea (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:21:39.294950 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:39.298785 31822 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:39.255447  1627 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.256637  1722 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:39.254060  1904 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:39.273353 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:39.268049  2049 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260709 12:21:39.335608 31818 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:39.346692 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:39.347487 31818 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:39.335798 32374 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:39.328745 31678 rpcz_store.cc:276] 0709 12:21:38.314498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.315566 (+  1068us) service_pool.cc:224] Handling call
0709 12:21:38.315853 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:38.316126 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:38.930379 (+614253us) log.cc:844] Serialized 14 byte log entry
0709 12:21:39.327928 (+397549us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:39.327984 (+    56us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:39.328038 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.328113 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:39.328236 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51207,"prepare.queue_time_us":410155,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"thread_start_us":919950,"threads_started":2}
I20260709 12:21:39.338229 31826 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:39.258483  1981 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.065s
I20260709 12:21:39.289077  1842 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:39.293833  1752 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:39.301338  2002 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:39.373792 31826 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:21:39.376081 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:39.376744 31746 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:39.377015 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:39.377807 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 126261us
I20260709 12:21:39.378600 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:39.379192 31821 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 3 FOLLOWER]: Advancing to term 4
W20260709 12:21:39.371821 31561 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 118447us
I20260709 12:21:39.331840  1975 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:39.333784  1627 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:39.342185  2045 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.758s	user 0.007s	sys 0.014s
I20260709 12:21:39.249497  2079 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.096s
I20260709 12:21:39.288839  2020 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 12:21:39.337798  1803 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:39.399909 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:39.381819  1751 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.414888 32376 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260709 12:21:39.418926 32376 pending_rounds.cc:85] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 6
I20260709 12:21:39.419325 32376 pending_rounds.cc:107] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260709 12:21:39.419643 32376 raft_consensus.cc:2889] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:39.415040 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 191495us
I20260709 12:21:39.374550  2061 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:39.423871 31749 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:39.423995 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.397109  1613 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:39.396730  2080 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.436120 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.438436 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:39.382644  1904 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.450570 31823 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:21:39.439364 31672 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:39.448993 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.455843 31751 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:39.453680 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:39.484388 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.423528  1844 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:39.433537  2012 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.778s	user 0.010s	sys 0.010s
I20260709 12:21:39.426777  1975 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.427335  1920 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.484587 31678 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:39.485620 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:39.425482  2013 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.426283  1906 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.079s
I20260709 12:21:39.494184 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:39.428094  2062 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.497325 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:39.426134  1851 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:39.500870 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 80997.2us
I20260709 12:21:39.394773  2045 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.506532 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.415421  1722 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260709 12:21:39.411129  1780 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.042s
I20260709 12:21:39.432464  1613 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:39.508466 31821 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
W20260709 12:21:39.497247  1999 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.808s	user 0.014s	sys 0.007s
I20260709 12:21:39.564128  1999 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:39.513592 31822 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:39.521489 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:39.567826 31749 raft_consensus.cc:2368] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
I20260709 12:21:39.521828 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:39.522644 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:39.569180 31678 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:39.428818  1751 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:39.530709 31746 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
W20260709 12:21:39.497613  2031 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.119s	user 0.008s	sys 0.015s
I20260709 12:21:39.513763  1443 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.530112  2062 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.521489  1852 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:39.447917  2080 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.504134  1844 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.515344  1920 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.512352  2098 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.089s
I20260709 12:21:39.516525  1842 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.077s
I20260709 12:21:39.515278  1975 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.525480  2084 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.572050 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.818077 31746 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
W20260709 12:21:39.604743  2089 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:21:39.819172 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:39.817889  2084 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.575714 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:39.820833 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:39.603520 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96910.2us
I20260709 12:21:39.646574 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:39.822579 31678 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:39.633671 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:39.824132 31823 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
W20260709 12:21:39.702548 32377 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.684s	user 0.012s	sys 0.010s
I20260709 12:21:39.826117 32377 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:39.835078 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:39.836725 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.838027 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:39.778134  2097 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.198s	user 0.000s	sys 0.000s
I20260709 12:21:39.840411 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:39.766139 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:39.662791  2083 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.210s	user 0.000s	sys 0.001s
I20260709 12:21:39.856416 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:39.604763  2069 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.584s	user 0.000s	sys 0.006s
W20260709 12:21:39.663015  2086 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.279s	user 0.000s	sys 0.001s
I20260709 12:21:39.856796 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:39.605376  2090 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
W20260709 12:21:39.839383  1727 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.155s	user 0.019s	sys 0.015s
I20260709 12:21:39.870992 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.608822  1919 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:39.592931  2064 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.861889 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:39.885928 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 486039us
I20260709 12:21:39.604264  1915 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:21:39.605224  1751 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:39.604151  1901 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.878072 31749 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:39.879246 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:39.886554 31750 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:39.782819  1878 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.566731  1613 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:39.886214 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.904054 31752 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 4.
I20260709 12:21:39.519814  2013 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:39.890261 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.851132  1931 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.836400  1443 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:39.861856  2031 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.922003 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.905230  1901 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:39.924865  2031 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.926940  1901 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:39.927032  2031 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:39.928145 31678 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 4.
I20260709 12:21:39.872057  1981 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.868981  2049 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:39.921375 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.936733 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:39.854745  1980 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 12:21:39.938083 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.902865 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:39.878819  2093 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:39.898818  1735 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.766s	user 0.018s	sys 0.004s
I20260709 12:21:39.891567  1915 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:39.954351 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51652.3us
I20260709 12:21:39.954476 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.955272 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:39.956358 31749 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:39.904920  1852 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:39.885288  1727 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:39.969573 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 203565us
I20260709 12:21:39.988363 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:39.955883  2082 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:39.992023 31752 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:21:39.949954  1981 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.907291  1780 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:21:39.899094  2064 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:39.916147  1878 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:39.968978  1921 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:39.963747  2108 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:39.964113  1852 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:39.967497 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:40.017897 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96208.7us
I20260709 12:21:39.953850  1735 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:40.010991  1908 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:39.968941  1931 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.062116 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:40.011502  2045 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:40.059814 31669 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
W20260709 12:21:40.053993  2060 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:21:40.030045  1842 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.072880  2102 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:21:40.073374 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:39.953876  2093 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.056262  1901 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 12:21:40.030520  1852 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:40.074378 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1850) took 1955 ms. Trace:
I20260709 12:21:40.018242  2107 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:40.083237 31791 leader_election.cc:400] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
W20260709 12:21:40.109076 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 141609us
I20260709 12:21:40.110584 31791 leader_election.cc:403] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:21:40.025198  2100 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:40.066684  2018 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.176s	user 0.023s	sys 0.004s
I20260709 12:21:40.107481 31676 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:40.060559  2108 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:40.074045  2098 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:40.133390  1931 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:40.133641  2098 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:40.123499  2113 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.063s
I20260709 12:21:40.134260  2098 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:40.136227 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53132.7us
I20260709 12:21:40.069372 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:40.144331 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:40.126526 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:40.126236  2108 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.128916 32372 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260709 12:21:40.100155  1906 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:40.130121  1443 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:40.130790  1999 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:40.130820  2090 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:21:40.070216  1915 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.108012  1842 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.132287  1975 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:40.120396  2045 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.124804  2084 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:40.136668  2061 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.115825  2109 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260709 12:21:40.120424  2100 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.071499  1921 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.064s
I20260709 12:21:40.119905  1851 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.006s
I20260709 12:21:40.075802  1908 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:40.130816  2080 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:40.166697  1884 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.143s	user 0.020s	sys 0.000s
I20260709 12:21:40.110116 31674 rpcz_store.cc:276] 0709 12:21:38.118587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.118721 (+   134us) service_pool.cc:224] Handling call
0709 12:21:38.137649 (+ 18928us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:38.192940 (+ 55291us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:38.192986 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:38.668356 (+475370us) spinlock_profiling.cc:243] Waited 8.07 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:21:38.668987 (+   631us) log.cc:844] Serialized 52 byte log entry
0709 12:21:39.231928 (+562941us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:39.242386 (+ 10458us) log.cc:844] Serialized 14 byte log entry
0709 12:21:39.245513 (+  3127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.245612 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:40.065178 (+819566us) raft_consensus.cc:1685] finished
0709 12:21:40.074122 (+  8944us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.074304 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30107,"prepare.queue_time_us":9978,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":19514,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":15518,"spinlock_wait_cycles":18550656,"thread_start_us":562543,"threads_started":1,"wal-append.queue_time_us":611829}
I20260709 12:21:40.146346 32377 raft_consensus.cc:2410] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260709 12:21:40.131261  2095 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.330s	user 0.002s	sys 0.000s
W20260709 12:21:40.161816 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 92525.3us
I20260709 12:21:40.131345  1920 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:40.116037  1844 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:40.171808  1443 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:21:40.172551 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:40.161492  2113 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.155772  1906 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:40.197108 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:40.198658 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:40.237205 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:40.195051 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:40.206151 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:40.202873 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:40.270619 31751 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:40.200151  2114 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.202s	user 0.000s	sys 0.000s
I20260709 12:21:40.179006  1915 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:21:40.281144  2123 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.281782  2123 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:40.283998  2123 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:40.188784  2051 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:40.281175 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 75140.5us
I20260709 12:21:40.162292  1901 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.103s
I20260709 12:21:40.284870  2123 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:40.285372  2123 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:40.182242  1907 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.377s	user 0.013s	sys 0.007s
I20260709 12:21:40.187624  1904 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:40.286178  1907 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.287349  1751 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:40.287907  1751 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.288414  1907 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:40.289997  1751 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:40.189759  2101 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:21:40.201239  2081 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.815s	user 0.000s	sys 0.001s
I20260709 12:21:40.276633  2120 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:40.278396  1950 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:40.285737  2123 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:40.178843  2122 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.187733  1908 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:40.174590  1945 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.989s	user 0.009s	sys 0.000s
I20260709 12:21:40.185351  1842 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:40.202855  2109 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.246240  1980 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.290529  1751 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:40.184727  2100 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:40.331920  2093 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:40.298383  2013 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:40.323475  1950 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:40.268065  1906 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:40.301052 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 138265us
I20260709 12:21:40.183061  2129 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
I20260709 12:21:40.262530  1443 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:40.297426 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:40.191118  1884 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.179366  2045 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.314689 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:40.224100  2061 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:40.329588  1613 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:40.298298  1920 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:40.331044  1751 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.272764  2113 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.197504  2098 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:40.323930  2120 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:40.201388  1852 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:40.605677 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:40.598869 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 450) took 2187 ms. Trace:
I20260709 12:21:40.610473 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:40.613246 31752 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:40.612500 32375 rpcz_store.cc:276] 0709 12:21:38.410377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.410551 (+   174us) service_pool.cc:224] Handling call
0709 12:21:38.460369 (+ 49818us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:38.563814 (+103445us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:38.917927 (+354113us) log.cc:844] Serialized 14 byte log entry
0709 12:21:39.215748 (+297821us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:39.215799 (+    51us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:39.215841 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.200526 (+984685us) spinlock_profiling.cc:243] Waited 12.6 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:40.597582 (+397056us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b5400544f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:40.597638 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.597773 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2863,"prepare.queue_time_us":390496,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":2136,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":4445,"spinlock_wait_cycles":266152064,"thread_start_us":613146,"threads_started":2,"wal-append.queue_time_us":359300}
I20260709 12:21:40.597877 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:40.598307 31822 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:21:40.615603  1920 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:40.619233 31671 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:40.621289 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1317) took 1177 ms. Trace:
I20260709 12:21:40.621663 31671 rpcz_store.cc:276] 0709 12:21:39.444130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.509392 (+1065262us) service_pool.cc:224] Handling call
0709 12:21:40.593967 (+ 84575us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:40.619586 (+ 25619us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.621021 (+  1435us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b54001b35d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:40.621194 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":57324800}
I20260709 12:21:40.614470 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:40.498157 31719 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.799s	user 0.009s	sys 0.015s
I20260709 12:21:40.624581 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:40.634052 31719 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.935s	user 0.009s	sys 0.019s
I20260709 12:21:40.624540  2113 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:40.388908  1908 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.625710 31747 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:40.626408  2049 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.627952  1852 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:40.628798  1751 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:40.323894  2122 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:40.630119 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1320) took 1131 ms. Trace:
I20260709 12:21:40.337540 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:40.631055  1443 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.631419  1878 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:40.565833 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.632561 32370 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:21:40.452984 31750 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.009s	sys 0.013s
W20260709 12:21:40.455796 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.010s	sys 0.010s
W20260709 12:21:40.455426 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.600s	user 0.004s	sys 0.017s
I20260709 12:21:40.525424 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:40.634670 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 337337us
W20260709 12:21:40.632607 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:40.594173 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:40.579427 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:40.568820  1752 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:40.421428  1627 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.014s	user 0.011s	sys 0.002s
I20260709 12:21:40.596521 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:40.674250 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 530006us
I20260709 12:21:40.701887 31746 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:40.697068 31677 rpcz_store.cc:276] 0709 12:21:39.498417 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.304764 (+806347us) service_pool.cc:224] Handling call
0709 12:21:40.630054 (+325290us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:40.716572 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 137316us
I20260709 12:21:40.743007 31826 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
W20260709 12:21:40.717429 31750 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.013s	sys 0.017s
I20260709 12:21:40.700825 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:40.718901 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.894s	user 0.010s	sys 0.018s
I20260709 12:21:40.699549 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:40.674023 31719 connection.cc:256] RPC call timeout handler was delayed by -1.0322005250000075s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 1.5s for a response
I20260709 12:21:40.326910  1727 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:40.424338  2087 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.052s	user 0.000s	sys 0.004s
I20260709 12:21:40.758450 31752 raft_consensus.cc:2410] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
W20260709 12:21:40.720717 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.865s	user 0.006s	sys 0.022s
W20260709 12:21:40.759479 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 924471us
I20260709 12:21:40.672997  2012 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:40.737201 31818 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:21:40.401428  2093 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.618559  1920 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:40.743230 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:40.720261 31823 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:40.401710  1981 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:40.760658 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:40.601511  1950 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:40.568511  1849 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:40.677237  1945 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.698585  2098 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:41.253917  2123 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.859s	user 0.006s	sys 0.004s
W20260709 12:21:40.919946  2087 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.081s	user 0.001s	sys 0.000s
I20260709 12:21:40.758543  1627 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.879711 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1933) took 1627 ms. Trace:
W20260709 12:21:41.306437 31678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1325) took 1735 ms (client timeout 1690 ms). Trace:
I20260709 12:21:41.024381  2127 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:40.818540 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:40.675846  1908 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:40.844722  1884 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:40.337903  1980 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:41.341507 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:41.342350 32377 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:40.881999 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:41.081763  2140 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.197s	user 0.001s	sys 0.002s
I20260709 12:21:41.022985  1906 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:40.767426  1752 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:40.826547 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:41.347626 32375 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:41.087668 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:40.878613  2110 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.714s	user 0.002s	sys 0.000s
I20260709 12:21:41.081700  2138 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.765014  2012 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:40.777546 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:40.721377  2045 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:40.738797  2049 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.774525  1977 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.093220 31823 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:40.895188  2137 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:40.802093  1975 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:40.902767  2082 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:41.308490 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.358309 31826 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:40.903429  1613 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:40.361925  2109 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:41.309433 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 566212us
I20260709 12:21:41.361799 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:40.740603  1751 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:40.823375 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:41.362777 31752 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:40.775193 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:41.085549 31746 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:40.880163  1951 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.945322 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 332) took 1952 ms. Trace:
I20260709 12:21:40.833461  1950 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:40.875840 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1887) took 1448 ms. Trace:
I20260709 12:21:41.350924  2012 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:40.776078  2093 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:40.844504 32372 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:21:40.763656  1915 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:40.808099 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47519.3us
I20260709 12:21:41.352882  2098 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.825661  2120 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:40.801057  2130 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:40.426854  2062 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:41.301874 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1860) took 2076 ms. Trace:
I20260709 12:21:41.356803  2104 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:41.321193 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1930) took 3019 ms. Trace:
W20260709 12:21:41.329941 31678 rpcz_store.cc:269] 0709 12:21:39.571064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.837757 (+266693us) service_pool.cc:224] Handling call
0709 12:21:41.101996 (+1264239us) spinlock_profiling.cc:243] Waited 85 ms on lock 0x7b54002549d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:41.306331 (+204335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":195395968}
I20260709 12:21:41.328504 31827 rpcz_store.cc:276] 0709 12:21:39.252067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.252220 (+   153us) service_pool.cc:224] Handling call
0709 12:21:39.267270 (+ 15050us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:40.615409 (+1348139us) spinlock_profiling.cc:243] Waited 5.57 ms on lock 0x7b18004ad940. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcdcdbfb 00007fe8fcdd864b
0709 12:21:40.615571 (+   162us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:40.615600 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:40.616190 (+   590us) log.cc:844] Serialized 26 byte log entry
0709 12:21:40.750370 (+134180us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:40.750413 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.750484 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:40.879261 (+128777us) raft_consensus.cc:1685] finished
0709 12:21:40.879283 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.879596 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35989,"raft.queue_time_us":63630,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":24262,"spinlock_wait_cycles":12800256,"thread_start_us":208610,"threads_started":2,"wal-append.queue_time_us":260873}
I20260709 12:21:41.378916 31820 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:21:41.332746 31818 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:41.335325 31822 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 12:21:41.362897  2082 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:41.343784 31745 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:41.346421 31670 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:41.317222  1979 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:41.324739  1945 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.359920 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.331740  2123 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.366758 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:41.339979  2127 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.317632 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1861) took 2091 ms. Trace:
I20260709 12:21:41.359210  1975 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:41.364306 31744 rpcz_store.cc:276] 0709 12:21:39.189251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.215439 (+ 26188us) service_pool.cc:224] Handling call
0709 12:21:39.215853 (+   414us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:39.216222 (+   369us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:39.216249 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:39.221038 (+  4789us) log.cc:844] Serialized 52 byte log entry
0709 12:21:39.326170 (+105132us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:39.413488 (+ 87318us) log.cc:844] Serialized 14 byte log entry
0709 12:21:40.893739 (+1480251us) spinlock_profiling.cc:243] Waited 3.96 ms on lock 0x7b18002de3e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287
0709 12:21:40.893924 (+   185us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.894033 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:40.894111 (+    78us) raft_consensus.cc:1685] finished
0709 12:21:40.935277 (+ 41166us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.945213 (+  9936us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19365,"prepare.queue_time_us":164342,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":350116,"raft.run_cpu_time_us":559,"raft.run_wall_time_us":75080,"spinlock_wait_cycles":9112448,"thread_start_us":206523,"threads_started":2,"wal-append.queue_time_us":156208,"wal-append.run_cpu_time_us":34792,"wal-append.run_wall_time_us":1867400}
I20260709 12:21:41.375741 31676 rpcz_store.cc:276] 0709 12:21:39.225518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.225650 (+   132us) service_pool.cc:224] Handling call
0709 12:21:39.238569 (+ 12919us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:40.120968 (+882399us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.880418 (+759450us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b54001810d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:41.301763 (+421345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1980822272}
I20260709 12:21:41.379042 31824 rpcz_store.cc:276] 0709 12:21:38.289276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:38.528226 (+238950us) service_pool.cc:224] Handling call
0709 12:21:38.582436 (+ 54210us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:38.582907 (+   471us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:38.582937 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:38.583496 (+   559us) log.cc:844] Serialized 26 byte log entry
0709 12:21:38.934649 (+351153us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:38.934706 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:38.934787 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:39.436084 (+501297us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:40.017446 (+581362us) raft_consensus.cc:1685] finished
0709 12:21:40.017487 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.883859 (+866372us) spinlock_profiling.cc:243] Waited 53.8 ms on lock 0x7b54001969d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:41.308297 (+424438us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":20257,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":12924,"spinlock_wait_cycles":174788352,"thread_start_us":350768,"threads_started":1,"wal-append.queue_time_us":437047,"wal-append.run_cpu_time_us":24335,"wal-append.run_wall_time_us":2056567}
I20260709 12:21:41.367789 31750 rpcz_store.cc:276] 0709 12:21:39.427170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.925774 (+498604us) service_pool.cc:224] Handling call
0709 12:21:39.926223 (+   449us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:39.926739 (+   516us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:39.926776 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:39.927455 (+   679us) log.cc:844] Serialized 52 byte log entry
0709 12:21:40.744071 (+816616us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:40.769628 (+ 25557us) log.cc:844] Serialized 14 byte log entry
0709 12:21:40.769904 (+   276us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.770005 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:40.875477 (+105472us) raft_consensus.cc:1685] finished
0709 12:21:40.875511 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:40.875742 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1174,"prepare.queue_time_us":26098,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":74,"raft.queue_time_us":14846,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":8125,"spinlock_wait_cycles":59614080,"thread_start_us":816174,"threads_started":1,"wal-append.queue_time_us":738060}
I20260709 12:21:40.827236 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.380141 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.431602 31818 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:41.431360 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.355345  2045 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:41.419890 31823 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:41.422178 32377 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:41.363785 31751 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:41.367745  1950 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:41.424561 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:41.427412 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:41.432327 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.433954 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:41.438951 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1894) took 1215 ms. Trace:
I20260709 12:21:41.439333 31749 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:41.439517 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.378770  2104 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:41.436950 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.432331 31824 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:41.385378  2130 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.441198 31822 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:41.445597 31822 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: [5.3-7.4]
I20260709 12:21:41.445928 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.359565  1977 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.383527  1751 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:41.353191  2138 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:41.419701  2120 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.442864 31818 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:41.451671 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:41.452337 31745 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:41.439281 31752 rpcz_store.cc:276] 0709 12:21:40.224086 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.389840 (+1165754us) service_pool.cc:224] Handling call
0709 12:21:41.438883 (+ 49043us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.418352  2062 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:41.428165 31675 rpcz_store.cc:276] 0709 12:21:39.226105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.226235 (+   130us) service_pool.cc:224] Handling call
0709 12:21:39.239111 (+ 12876us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:39.257536 (+ 18425us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:39.257569 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:39.288573 (+ 31004us) log.cc:844] Serialized 26 byte log entry
0709 12:21:40.617818 (+1329245us) spinlock_profiling.cc:243] Waited 5.3 ms on lock 0x7b1800406a20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:21:40.717001 (+ 99183us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:40.717048 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:40.717118 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:41.102319 (+385201us) raft_consensus.cc:1685] finished
0709 12:21:41.102344 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.317532 (+215188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50348,"raft.queue_time_us":11180,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":34682,"spinlock_wait_cycles":12208640,"thread_start_us":80316,"threads_started":1,"wal-append.queue_time_us":1570951}
I20260709 12:21:41.380141  1915 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:41.368880  2137 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.448436 31824 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:41.448827 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:41.429273  1975 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:41.419924  1844 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:41.379858  2098 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:41.372851  1951 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:41.440379  1950 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:41.456393 31819 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:41.457415 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1898) took 1211 ms. Trace:
I20260709 12:21:41.463585 31744 rpcz_store.cc:276] 0709 12:21:40.246453 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.439754 (+1193301us) service_pool.cc:224] Handling call
0709 12:21:41.457353 (+ 17599us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.459259 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:41.441800  2122 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:41.462149 31752 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:21:41.427282  1979 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.464953 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1895) took 1239 ms. Trace:
I20260709 12:21:41.470902 31752 rpcz_store.cc:276] 0709 12:21:40.225116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.453350 (+1228234us) service_pool.cc:224] Handling call
0709 12:21:41.461871 (+  8521us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.464636 (+  2765us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.464895 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.472175 31827 raft_consensus.cc:2410] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260709 12:21:41.465749 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.467023 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:41.473557 32377 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:41.467732  2122 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:41.467872  2127 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:41.469810 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1931) took 2219 ms. Trace:
I20260709 12:21:41.440012  2045 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.473603 31744 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:21:41.481962 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1896) took 1256 ms. Trace:
I20260709 12:21:41.482479 31744 rpcz_store.cc:276] 0709 12:21:40.225928 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.464124 (+1238196us) service_pool.cc:224] Handling call
0709 12:21:41.473305 (+  9181us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.481583 (+  8278us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.481878 (+   295us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.479876  1979 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:41.460002 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:41.477222 31825 rpcz_store.cc:276] 0709 12:21:39.250315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.250918 (+   603us) service_pool.cc:224] Handling call
0709 12:21:39.267463 (+ 16545us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:39.300842 (+ 33379us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:21:39.486758 (+185916us) log.cc:844] Serialized 14 byte log entry
0709 12:21:39.857364 (+370606us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:39.868038 (+ 10674us) log.cc:844] Serialized 52 byte log entry
0709 12:21:39.868189 (+   151us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:39.868223 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:39.868295 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:40.877866 (+1009571us) spinlock_profiling.cc:243] Waited 9.21 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:41.469463 (+591597us) raft_consensus.cc:1685] finished
0709 12:21:41.469575 (+   112us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.469743 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4950,"prepare.queue_time_us":157276,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":33716,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":25700,"spinlock_wait_cycles":22699136,"thread_start_us":416677,"threads_started":2,"wal-append.queue_time_us":461137}
I20260709 12:21:41.486442 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1902) took 1236 ms. Trace:
I20260709 12:21:41.490928 31752 rpcz_store.cc:276] 0709 12:21:40.250446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.471722 (+1221276us) service_pool.cc:224] Handling call
0709 12:21:41.485707 (+ 13985us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.485973 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:41.486014 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:41.486079 (+    65us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:41.486137 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.486224 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.486371 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:41.487196 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.495908 31824 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:41.498888 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1904) took 1201 ms. Trace:
I20260709 12:21:41.499625 31744 rpcz_store.cc:276] 0709 12:21:40.297411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.483461 (+1186050us) service_pool.cc:224] Handling call
0709 12:21:41.483890 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.489267 (+  5377us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 12:21:41.489313 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:41.489382 (+    69us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:41.489440 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.490000 (+   560us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.498806 (+  8806us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":798}
I20260709 12:21:41.504086 32371 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
W20260709 12:21:41.504380  2152 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
I20260709 12:21:41.504568  1906 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:21:41.505788  2148 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:21:41.507221  2137 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:41.508153  2138 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:41.511332  1980 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:41.513442 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:41.514253 31819 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:41.517342 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.457823  2120 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.457791  2062 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.485129  2127 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.488749  1908 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:41.514981  1627 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:41.532675 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 165173us
I20260709 12:21:41.462919  1751 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:41.563405 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:41.488137  1844 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 12:21:41.755695 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.837266 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:41.843173 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:41.663820 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:41.842736 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 176227us
W20260709 12:21:41.707154 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 929649us
I20260709 12:21:41.795692 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:41.845677 31823 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:41.814949 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:41.810814  2156 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:41.847250 31675 raft_consensus.cc:2368] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
I20260709 12:21:41.848922 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:21:41.795578 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:41.848227 32369 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:41.848658 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:41.857231 31676 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:41.859690  1544 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:41.860010 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:41.860785  2153 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.308s	user 0.000s	sys 0.001s
I20260709 12:21:41.851758  1921 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:41.861358 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:41.861584  1921 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:41.861985 32373 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:41.868438 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.860010 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:41.869194 31823 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:41.869521 31677 raft_consensus.cc:2410] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:21:41.862037  1921 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:41.849560 31643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 2.082s (SENT) [suppressed 4 similar messages]
I20260709 12:21:41.551826  1908 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.551427  1752 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:41.870479  1977 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:41.855851 31674 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:21:41.715083  2138 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.758458  1945 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:41.700598  1897 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:41.729445  2123 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:41.550843  2127 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:41.907683 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:41.881507 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:41.797080  2164 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:41.887408 31674 raft_consensus.cc:606] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:41.909226 31674 raft_consensus.cc:614] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:41.825150  2020 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:41.638777  2137 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:41.766713  2109 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:41.858371  1915 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:41.873369  2098 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:41.620208  1980 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.269s
I20260709 12:21:41.914191  2164 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:41.918336  1751 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:41.919107  2120 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:41.919263  2160 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:41.919523  2164 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:41.919804  2160 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:41.920243  2160 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:41.922797  2020 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.871158  2141 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260709 12:21:41.911908  2138 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.929936  2020 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:41.930881  2020 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:41.931411  2020 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:41.931898  2020 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:41.911496  1897 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:41.870060  2156 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.936889  1897 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:41.915834  1752 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:41.937829  2175 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:21:41.938099  1752 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:41.940227  1752 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:41.954965 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.955721 31825 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:41.957105 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:41.980424  2156 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:41.981766 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:41.982534 31674 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:41.983618 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:41.984252 31674 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:41.985316 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:41.986327 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1788) took 1287 ms. Trace:
I20260709 12:21:41.987449  2174 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:41.988040  2174 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:41.986786 31821 rpcz_store.cc:276] 0709 12:21:40.699061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.699312 (+   251us) service_pool.cc:224] Handling call
0709 12:21:40.699754 (+   442us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:40.744133 (+ 44379us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:41.023486 (+279353us) log.cc:844] Serialized 14 byte log entry
0709 12:21:41.610222 (+586736us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:41.679638 (+ 69416us) log.cc:844] Serialized 26 byte log entry
0709 12:21:41.849105 (+169467us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:41.849150 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.849220 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:41.986102 (+136882us) raft_consensus.cc:1685] finished
0709 12:21:41.986124 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.986266 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28198,"prepare.queue_time_us":683103,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":37471,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":64275,"thread_start_us":482294,"threads_started":2,"wal-append.queue_time_us":641391}
I20260709 12:21:41.988405  2174 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:41.989104 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.989727 31821 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:41.991477 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:41.992350 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.993096  2062 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:41.993387 31821 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
W20260709 12:21:41.993705  1950 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.518s	user 0.004s	sys 0.004s
I20260709 12:21:41.994092  2138 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:41.888383  1908 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:41.994793 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:41.992565  2127 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:41.992348  2082 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.562s	user 0.002s	sys 0.008s
I20260709 12:21:41.951574  2045 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:41.996095  2082 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:41.979070 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 340) took 1229 ms. Trace:
I20260709 12:21:41.997394 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:41.986554 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:41.997200 31744 rpcz_store.cc:276] 0709 12:21:40.749471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.502315 (+752844us) service_pool.cc:224] Handling call
0709 12:21:41.514490 (+ 12175us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:41.607991 (+ 93501us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:41.608035 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:41.639234 (+ 31199us) log.cc:844] Serialized 26 byte log entry
0709 12:21:41.820302 (+181068us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:41.820365 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.820474 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:41.978822 (+158348us) raft_consensus.cc:1685] finished
0709 12:21:41.978841 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:41.979001 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15104,"raft.queue_time_us":36903,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":115602,"thread_start_us":64869,"threads_started":1,"wal-append.queue_time_us":137848}
I20260709 12:21:41.999482 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:42.000655 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:42.001305 31825 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:42.002127 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:42.005795  1975 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.535s	user 0.007s	sys 0.003s
I20260709 12:21:41.987040  2020 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.006299  1975 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.006592 32373 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:42.008028  2020 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.008502  1975 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:42.014447 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1893) took 1911 ms. Trace:
W20260709 12:21:42.019140 31819 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.503s	user 0.007s	sys 0.003s
I20260709 12:21:42.019635 31819 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:41.999127 31674 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:42.015308 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:42.019515 31745 rpcz_store.cc:276] 0709 12:21:40.103260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.452909 (+1349649us) service_pool.cc:224] Handling call
0709 12:21:41.461468 (+  8559us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.461628 (+   160us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:41.734040 (+272412us) log.cc:844] Serialized 14 byte log entry
0709 12:21:42.014070 (+280030us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:42.014122 (+    52us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:42.014179 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.014253 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.014387 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3044,"prepare.queue_time_us":47660,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":65,"thread_start_us":548669,"threads_started":2}
I20260709 12:21:41.942466  1951 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:42.035691 32370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 610) took 1843 ms (client timeout 2082 ms). Trace:
I20260709 12:21:42.031841 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:42.031934 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:42.014492  2045 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.026779  1980 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:42.040936 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:42.042766 31745 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:42.034534 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 446) took 5382 ms. Trace:
I20260709 12:21:42.040961 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1790) took 1220 ms. Trace:
W20260709 12:21:42.041227 32370 rpcz_store.cc:269] 0709 12:21:40.192244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.194727 (+  2483us) service_pool.cc:224] Handling call
0709 12:21:42.035458 (+1840731us) spinlock_profiling.cc:243] Waited 363 ms on lock 0x7b54004c5c54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:42.035592 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":834216192}
I20260709 12:21:42.036762 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1886) took 2620 ms. Trace:
I20260709 12:21:41.994258  1950 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.057401 31752 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:42.058602 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:42.073045 31825 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:41.993709  2062 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:42.057396 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 207843us
I20260709 12:21:42.059566 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:42.077986 31821 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:42.039209  2139 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.229s	user 0.018s	sys 0.026s
I20260709 12:21:42.069924 31820 rpcz_store.cc:276] 0709 12:21:40.820297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.820495 (+   198us) service_pool.cc:224] Handling call
0709 12:21:40.826601 (+  6106us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.379426 (+552825us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.040730 (+661304us) spinlock_profiling.cc:243] Waited 552 ms on lock 0x7b540030ded4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:42.040884 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1268753280}
I20260709 12:21:42.073662 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:42.080471 31819 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:21:42.022812  2093 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:42.070266 32376 rpcz_store.cc:276] 0709 12:21:36.652583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:36.726077 (+ 73494us) service_pool.cc:224] Handling call
0709 12:21:39.243024 (+2516947us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:39.453027 (+210003us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.034282 (+2581255us) spinlock_profiling.cc:243] Waited 2.46 s on lock 0x7b54004cfed0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:42.034455 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5661994752}
I20260709 12:21:41.943821  2137 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.044066  2020 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:42.078976 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:42.039327  2023 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:21:42.084785 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 235931us
I20260709 12:21:42.069927 31753 rpcz_store.cc:276] 0709 12:21:39.416640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.416767 (+   127us) service_pool.cc:224] Handling call
0709 12:21:39.435709 (+ 18942us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:39.867330 (+431621us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:39.867375 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:39.896807 (+ 29432us) log.cc:844] Serialized 52 byte log entry
0709 12:21:40.055891 (+159084us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:40.055937 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.036322 (+1980385us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b54002945d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:42.036372 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:42.036419 (+    47us) raft_consensus.cc:1685] finished
0709 12:21:42.036431 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.036675 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4910,"raft.queue_time_us":23479,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":36987,"spinlock_wait_cycles":980273536,"thread_start_us":103145,"threads_started":1,"wal-append.queue_time_us":262937,"wal-append.run_cpu_time_us":9569,"wal-append.run_wall_time_us":1117515}
I20260709 12:21:42.089860 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.091307 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:42.087988 31821 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:42.096045 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:42.100564 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:42.038383  1931 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:21:42.077699  2062 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:42.111614 31819 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:42.112938 31821 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:42.103224 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:42.115089 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:42.118496 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:42.094152  2023 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.124457 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:42.152992 31753 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:42.087762  2020 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:42.153434 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:42.072484  2045 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.101853  2141 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.157321 31752 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
W20260709 12:21:42.158113 31792 leader_election.cc:336] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.524s (SENT)
I20260709 12:21:42.157224 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.163870 31674 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:42.082274  1980 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.156021  2023 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:42.160151 31749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1900) took 1911 ms (client timeout 1775 ms). Trace:
I20260709 12:21:42.160393 31819 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
W20260709 12:21:42.123760  2012 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.748s	user 0.008s	sys 0.004s
I20260709 12:21:42.115113  2062 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:42.171149 31825 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:21:42.180680 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:42.184188 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 81034.7us
I20260709 12:21:42.219257 31745 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
W20260709 12:21:42.174703 31749 rpcz_store.cc:269] 0709 12:21:40.248336 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.593866 (+345530us) service_pool.cc:224] Handling call
0709 12:21:42.159945 (+1566079us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b5400064854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:42.160072 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1660826368}
I20260709 12:21:42.221581 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:42.221107 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:42.215440 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 122652us
I20260709 12:21:42.222767 31749 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:21:42.215837 31821 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:21:42.225529 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:42.230302 32370 raft_consensus.cc:2368] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
I20260709 12:21:42.230399 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:42.228549 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:42.225143 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.232456 31674 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:42.226121 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.233733 31672 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 8 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:42.233635 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.231467 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:42.228572 31825 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260709 12:21:42.232401 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:42.234674 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:42.234961 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.232955 31749 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:42.235469 32370 raft_consensus.cc:2376] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7. Re-sending same reply.
I20260709 12:21:42.234617 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.236183 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:42.234839 32374 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:42.236601 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:42.225910 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:42.236893 31674 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:42.237663 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.237759 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:42.237355 32377 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:42.237331 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:42.239583 31745 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:42.242411 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.162318  2020 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:42.239886 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:42.125603  1908 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.247364 31672 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:42.165997  2161 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.254943  2161 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.260757 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.166445  2093 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.187422  1977 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:42.153736  1921 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.289075 31674 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:42.290195 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.255780  2082 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:42.259941  1735 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:42.263002 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:42.263373  2173 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
W20260709 12:21:42.264375  2176 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:21:42.215696  1727 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:42.267184  1980 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:42.277109  2093 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.185868  2015 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:42.256332 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:42.291285 32377 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:42.262056 31672 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:42.244956  2012 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.347364 31675 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 8 because replica has last-logged OpId of term: 6 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:21:42.341353 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1318) took 2896 ms. Trace:
W20260709 12:21:42.344576 31746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1414) took 2661 ms (client timeout 1524 ms). Trace:
I20260709 12:21:42.355597 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:42.343292 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:42.357666 31744 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:42.364571 31744 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:42.359824 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:42.362720 31746 rpcz_store.cc:269] 0709 12:21:39.682855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.818885 (+136030us) service_pool.cc:224] Handling call
0709 12:21:42.344353 (+2525468us) spinlock_profiling.cc:243] Waited 882 ms on lock 0x7b54001ee1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:42.344473 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2028403584}
I20260709 12:21:42.361857 31670 rpcz_store.cc:276] 0709 12:21:39.444518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:40.614476 (+1169958us) service_pool.cc:224] Handling call
0709 12:21:40.614880 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.419326 (+804446us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.341072 (+921746us) spinlock_profiling.cc:243] Waited 731 ms on lock 0x7b54002553d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:42.341277 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1681751296}
I20260709 12:21:42.377034 31749 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260709 12:21:42.377933 31672 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:42.383730 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:42.349062  2177 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.175s	user 0.000s	sys 0.000s
I20260709 12:21:42.285054  1977 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.382598 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.388386 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.388430 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.398936 31670 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 5: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:42.362914  1727 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.408214 31678 raft_consensus.cc:2368] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
I20260709 12:21:42.339046  2045 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:42.408897 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.417411 31672 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:42.401420  1977 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:42.174453  2141 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.425359  2141 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.426767  1727 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:42.429502 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:42.351615  1950 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.395759  2062 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.342888  2182 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:42.432052  2182 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.432659  2182 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:42.433290  2062 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.434068  2062 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:42.435859 32372 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:42.438771 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1954) took 1118 ms. Trace:
I20260709 12:21:42.439453 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:42.439329 31822 rpcz_store.cc:276] 0709 12:21:41.320615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.447439 (+126824us) service_pool.cc:224] Handling call
0709 12:21:41.458715 (+ 11276us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:41.754734 (+296019us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:41.831393 (+ 76659us) log.cc:844] Serialized 14 byte log entry
0709 12:21:42.085813 (+254420us) log.cc:844] Serialized 14 byte log entry
0709 12:21:42.086104 (+   291us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:42.100413 (+ 14309us) log.cc:844] Serialized 26 byte log entry
0709 12:21:42.100584 (+   171us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:42.100621 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.100730 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:42.438459 (+337729us) raft_consensus.cc:1685] finished
0709 12:21:42.438488 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.438699 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9741,"prepare.queue_time_us":81548,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":70,"raft.queue_time_us":60785,"raft.run_cpu_time_us":349,"raft.run_wall_time_us":36810,"spinlock_wait_cycles":20224,"thread_start_us":477353,"threads_started":3,"wal-append.queue_time_us":235877}
I20260709 12:21:42.438855  1921 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.440232 31670 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:42.442199 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:42.445956 31669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1326) took 2843 ms (client timeout 1532 ms). Trace:
I20260709 12:21:42.461031 31672 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:42.395956  2157 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:42.375962  2015 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:42.461186 31669 rpcz_store.cc:269] 0709 12:21:39.602836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.813479 (+210643us) service_pool.cc:224] Handling call
0709 12:21:42.445724 (+2632245us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b540036f1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:42.445855 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":438348160}
I20260709 12:21:42.448052  2049 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:42.464025 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 63284.3us
I20260709 12:21:42.468425  2020 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.471251  2020 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.476594  2015 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:42.480206 31745 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:42.496063 31669 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
W20260709 12:21:42.497042 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 311140us
W20260709 12:21:42.463873 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 208363us
I20260709 12:21:42.500947 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.467350 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:42.502436 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:42.504362 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:42.512761 31747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1421) took 2613 ms (client timeout 1688 ms). Trace:
W20260709 12:21:42.513197 31747 rpcz_store.cc:269] 0709 12:21:39.906089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:39.923728 (+ 17639us) service_pool.cc:224] Handling call
0709 12:21:42.512540 (+2588812us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x7b54002f76d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:42.512686 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1541560192}
W20260709 12:21:42.516142  2115 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 2.353s	user 0.007s	sys 0.005s
W20260709 12:21:42.519273 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 52044.1us
I20260709 12:21:42.524163 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.505651  2015 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:42.527251 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:42.506872  1921 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:42.529316  2015 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.512315  2137 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:42.518827  2174 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.530s	user 0.013s	sys 0.000s
I20260709 12:21:42.525624  2161 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.535643 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:42.533161 31670 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:42.466553  2157 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.540426 31792 leader_election.cc:400] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
I20260709 12:21:42.553537 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:42.569720 31792 leader_election.cc:403] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:42.570794  2049 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
I20260709 12:21:42.564201 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:42.567961 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 466344us
I20260709 12:21:42.582856 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.583830 31670 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
W20260709 12:21:42.586743 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51180us
I20260709 12:21:42.525553  2190 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:42.572227  2159 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:42.541848  2141 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:42.534137  2020 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:42.591686 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.541090  2192 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:42.562358  1945 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:42.561285  2012 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:42.605008  2188 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:21:42.605419 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:42.605921  1907 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260709 12:21:42.607699 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:42.536266  2015 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:42.595006  2160 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.674s	user 0.009s	sys 0.005s
I20260709 12:21:42.610502 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:42.554931  1897 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:42.611317 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:42.628465 32377 raft_consensus.cc:2376] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8. Re-sending same reply.
I20260709 12:21:42.616724 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:42.629411 31670 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
W20260709 12:21:42.608950 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 493926us
I20260709 12:21:42.594777  2161 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:42.614763 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 61356.4us
I20260709 12:21:42.584021  2093 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:42.629050 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.635824 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:42.638906 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:42.647207 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.604038  2020 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.637153 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:42.648567 31821 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:42.648758 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:42.644579 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.649104 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:42.650111 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.618749  1735 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:42.608166  2192 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:42.581564  2174 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.657492 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.645258  1907 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.658950 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:42.611195  2130 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.661422 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:42.628922  2012 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:42.654709  1921 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:42.651780  2020 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:42.668617 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.655334  2137 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.663861 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:42.672134 31819 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:42.664521 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:42.640467  2161 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:42.662950  1907 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.618539  1950 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 12:21:42.657987  1735 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.610809  2191 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.063s
I20260709 12:21:42.618767  2159 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.654870  2093 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:42.626922  1945 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.655843  2062 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:42.661240  2064 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:42.667639  2113 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:42.661958  2192 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:42.674960  1765 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:42.640765  1897 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.688730 32343 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.636566  2160 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.665042  2201 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:42.679132  2107 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:42.675352  2123 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.021s
I20260709 12:21:42.681353  2161 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:42.686060  1735 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:42.699960 31821 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:21:42.688195  2130 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:21:42.719518 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 46907.8us
I20260709 12:21:42.719579 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:42.690109  2113 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:42.696271  2093 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 8 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:42.695227  2062 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:21:42.726754 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 76709.8us
W20260709 12:21:42.713874  2189 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 12:21:42.687083  2012 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
I20260709 12:21:42.680444  2137 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.720666 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:42.735397 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 77980.2us
I20260709 12:21:42.713636  2187 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:42.745715  2187 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.745788 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.746301  2187 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:42.738742 32376 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8. Ops: [7.7-7.7]
I20260709 12:21:42.700562  2201 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:42.718717  2107 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.713243  2108 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:42.742265 32342 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:42.686992  2197 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:21:42.758737  2206 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:42.728617  2113 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:42.721864  2130 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.722821  1765 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:42.732646  1627 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:42.731156  2156 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:42.761121 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:42.752251  2018 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:42.770349  2107 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:42.789281 31643 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:42.772578  2184 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:42.766719  2201 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:42.808935 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:42.778275  2130 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:42.781229  2197 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:42.741852  2023 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.809793 31821 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:42.809798  2018 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.781373  2157 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:42.855901 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:42.812003 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:42.859771 31819 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:42.813692  2210 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.809098  2209 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:42.794482  2156 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:42.814993 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:42.875519 31818 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:42.877735 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:42.878481 32374 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:42.879465 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:42.880352 32369 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:42.881280 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:42.882845  2202 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.112s	user 0.001s	sys 0.000s
I20260709 12:21:42.883136  1945 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:42.883781  1945 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:42.883908  2171 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.967s	user 0.004s	sys 0.003s
I20260709 12:21:42.884193  1945 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.884577  1945 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:42.885376 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1957) took 1547 ms. Trace:
I20260709 12:21:42.886169 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:42.886750  2198 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
W20260709 12:21:42.866562 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 120855us
I20260709 12:21:42.888036 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 301) took 1212 ms. Trace:
I20260709 12:21:42.888571 31820 rpcz_store.cc:276] 0709 12:21:41.676221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.160694 (+484473us) service_pool.cc:224] Handling call
0709 12:21:42.161055 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:42.189675 (+ 28620us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:42.189728 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:42.190353 (+   625us) log.cc:844] Serialized 26 byte log entry
0709 12:21:42.655120 (+464767us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:42.655172 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.663747 (+  8575us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:42.887742 (+223995us) raft_consensus.cc:1685] finished
0709 12:21:42.887779 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.887973 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1200,"raft.queue_time_us":17312,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"thread_start_us":368955,"threads_started":1,"wal-append.queue_time_us":428954}
I20260709 12:21:42.890659 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:42.890971  1907 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.891371 31820 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:42.894932  2174 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.804774  2120 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.093s
I20260709 12:21:42.890668 32374 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:42.885763 31824 rpcz_store.cc:276] 0709 12:21:41.338214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.496818 (+158604us) service_pool.cc:224] Handling call
0709 12:21:41.497367 (+   549us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:41.497819 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:41.497857 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:41.498504 (+   647us) log.cc:844] Serialized 26 byte log entry
0709 12:21:41.780487 (+281983us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:41.780543 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:41.780625 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:42.772754 (+992129us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:42.885057 (+112303us) raft_consensus.cc:1685] finished
0709 12:21:42.885080 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:42.885309 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10977,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":78,"spinlock_wait_cycles":25539968,"thread_start_us":135762,"threads_started":1,"wal-append.queue_time_us":390968}
I20260709 12:21:42.897274  2023 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.882539 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:42.901684 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.904646 32373 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:42.903054 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.910009 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:42.910936 31825 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:42.911341  1897 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:42.912102  1897 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:42.912508  1897 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:42.912395 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:42.912942  1897 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:42.914052 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1332) took 1058 ms. Trace:
I20260709 12:21:42.914440 31675 rpcz_store.cc:276] 0709 12:21:41.855976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.877202 (+ 21226us) service_pool.cc:224] Handling call
0709 12:21:42.913821 (+1036619us) spinlock_profiling.cc:243] Waited 438 ms on lock 0x7b5400229a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:42.913964 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1008068864}
I20260709 12:21:42.917270 31670 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260709 12:21:42.920045  2187 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.920648 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:42.925598 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:42.888216  2200 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:21:42.951704 31670 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:42.931083 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:42.936201 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:42.811519  2191 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:42.947798 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:42.806114  1950 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:42.814574  2197 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:21:42.818910  2195 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 12:21:42.969744  2045 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:42.973809  1727 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.546s	user 0.012s	sys 0.000s
I20260709 12:21:42.810425  2062 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.862409  2207 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:42.984577  1727 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.969975  1443 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:42.992543 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:42.997392 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 94409.5us
I20260709 12:21:42.963917  2049 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:42.971089  2157 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:42.997759 31819 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:42.980492  1950 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:42.882501  1977 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.008414 31672 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260709 12:21:43.015569 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:43.021539 31748 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:43.016784 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:43.019325 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:42.994163  2187 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:43.020257 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:43.020514 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:42.984526  2045 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:42.861286  2093 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.979143  2206 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 12:21:42.951026  2210 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:43.011801  2182 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.578s	user 0.008s	sys 0.006s
I20260709 12:21:43.040830 31747 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:42.999634  1443 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:43.014236 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1891) took 2911 ms. Trace:
I20260709 12:21:43.003448  2157 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:43.048172 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:43.021273  2160 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.868038  2209 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:43.067836 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 113513us
I20260709 12:21:42.995640  2211 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:43.043778  2197 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:42.979609  2191 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.026579  1950 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:43.059875 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:43.014384 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 257800us
I20260709 12:21:43.075162 31676 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:21:43.063093 32370 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:43.039873  2049 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:43.063143 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:43.056033 31751 rpcz_store.cc:276] 0709 12:21:40.102592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.439069 (+1336477us) service_pool.cc:224] Handling call
0709 12:21:41.454300 (+ 15231us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:41.454497 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:21:41.464773 (+ 10276us) log.cc:844] Serialized 14 byte log entry
0709 12:21:42.691270 (+1226497us) spinlock_profiling.cc:243] Waited 6.3 ms on lock 0x7b18004bd960. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:21:43.013821 (+322551us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:43.013879 (+    58us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:43.013939 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:43.014012 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:43.014163 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":10111,"prepare.run_cpu_time_us":74,"prepare.run_wall_time_us":2153,"raft.queue_time_us":6543,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":7783,"spinlock_wait_cycles":14484864,"thread_start_us":322326,"threads_started":1,"wal-append.queue_time_us":1589147}
I20260709 12:21:43.075214 31745 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:21:43.076207 31826 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.013437  2108 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:43.070092 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:43.064248  2157 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:43.009742  2159 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:42.870002  2156 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.061373  1443 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:43.079928 31752 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:42.981624  1765 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.081637 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:43.083249 31819 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:43.087247 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:43.088634 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:43.089202 31819 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:21:43.069036  2209 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:43.077332  2049 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:43.091084  2049 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:43.071231  2137 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:43.104918 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 157234us
I20260709 12:21:43.110210 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:43.077940  1921 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:43.130968 31826 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:43.131688  2138 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:43.135797 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:43.136646 31677 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:43.132239 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:43.138100 31826 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.144734 31745 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:21:43.149577 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:43.150975  2108 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.156142 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:43.157307 32371 raft_consensus.cc:2368] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8.
I20260709 12:21:43.157569 31819 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:43.163573  2045 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:43.063624  2182 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.064962  2107 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.204830 31826 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:43.208549 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:43.220077 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:43.224524 31826 raft_consensus.cc:2368] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7.
W20260709 12:21:43.162235  2015 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.545s	user 0.007s	sys 0.004s
I20260709 12:21:43.104624  2211 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:43.205943  2107 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:43.245040 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 139695us
I20260709 12:21:43.265709 31751 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:21:43.267072 31751 raft_consensus.cc:606] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:43.270102 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:43.270730  1735 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.544s	user 0.007s	sys 0.003s
I20260709 12:21:43.271236  1735 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.111239  2206 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.157s
I20260709 12:21:43.419488 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:43.758935 31818 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:43.477345 31678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1329) took 1914 ms (client timeout 1681 ms). Trace:
I20260709 12:21:43.277331 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:43.749894 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 639688us
I20260709 12:21:43.345355 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:43.763635 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:43.764007 31823 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:43.226517  2192 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.528s	user 0.008s	sys 0.000s
I20260709 12:21:43.760519 31824 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.272604 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 481) took 1038 ms. Trace:
W20260709 12:21:43.765683 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.272537 31671 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:21:43.760295 31678 rpcz_store.cc:269] 0709 12:21:41.562869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.563036 (+   167us) service_pool.cc:224] Handling call
0709 12:21:43.476949 (+1913913us) spinlock_profiling.cc:243] Waited 793 ms on lock 0x7b54002b0554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:43.477223 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1822810368}
I20260709 12:21:43.766165 32372 rpcz_store.cc:276] 0709 12:21:42.234195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.234366 (+   171us) service_pool.cc:224] Handling call
0709 12:21:43.272387 (+1038021us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54004984d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:43.272541 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":459809536}
I20260709 12:21:43.765364 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:43.346596 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1866) took 1121 ms. Trace:
I20260709 12:21:43.757170 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:43.770136 31821 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:43.756805 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:43.756299 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:43.750990 31717 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:43.232560  2210 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:43.229785  2182 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:43.768435 31669 rpcz_store.cc:276] 0709 12:21:42.225786 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.524727 (+298941us) service_pool.cc:224] Handling call
0709 12:21:42.537448 (+ 12721us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:42.539581 (+  2133us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:42.539629 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:42.540267 (+   638us) log.cc:844] Serialized 26 byte log entry
0709 12:21:42.664208 (+123941us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:42.664270 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:43.346018 (+681748us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b540036f6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:43.346089 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:43.346156 (+    67us) raft_consensus.cc:1685] finished
0709 12:21:43.346179 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:43.346512 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2510,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":4175,"spinlock_wait_cycles":3678080,"thread_start_us":88505,"threads_started":1,"wal-append.queue_time_us":165446}
I20260709 12:21:43.259469  1727 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:43.407262  2201 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.549s	user 0.006s	sys 0.004s
W20260709 12:21:43.718892  1443 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.618s	user 0.010s	sys 0.000s
W20260709 12:21:43.717248  2157 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.613s	user 0.008s	sys 0.000s
W20260709 12:21:43.799170  2130 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.937s	user 0.009s	sys 0.003s
W20260709 12:21:43.274850  2020 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.595s	user 0.008s	sys 0.003s
W20260709 12:21:43.287930  2199 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.542s	user 0.004s	sys 0.000s
I20260709 12:21:43.800312 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:43.406901  1897 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.805150 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:43.728287  2209 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.638s	user 0.011s	sys 0.000s
I20260709 12:21:43.803376 31819 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:43.227447  2161 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.456565  1977 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:43.789840  1950 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.703s	user 0.006s	sys 0.003s
I20260709 12:21:43.806416 31818 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:43.273603  1735 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:43.760891  2191 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:43.803735  2093 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.671s	user 0.010s	sys 0.021s
I20260709 12:21:43.789474 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:43.815527 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:43.816857 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:43.784585  2137 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:43.785055  2108 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:43.260193  2015 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.802223  2157 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.786053  2079 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.039s
I20260709 12:21:43.419816  2049 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.274936  1945 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.803247 31823 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
W20260709 12:21:43.813390  2113 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.018s	user 0.008s	sys 0.003s
I20260709 12:21:43.802958  2061 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:43.814026  2107 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:43.837435 31675 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: []
W20260709 12:21:43.838305 31827 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1951) took 2517 ms (client timeout 2155 ms). Trace:
I20260709 12:21:43.342330  1921 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.802474  2130 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.840279 32378 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:43.841291 31821 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:43.785320  2192 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:43.842867 31827 rpcz_store.cc:269] 0709 12:21:41.313366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:41.426513 (+113147us) service_pool.cc:224] Handling call
0709 12:21:43.830849 (+2404336us) spinlock_profiling.cc:243] Waited 31.5 ms on lock 0x7b54001982d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:43.831053 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":72357248}
W20260709 12:21:43.825966  2093 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.693s	user 0.011s	sys 0.021s
I20260709 12:21:43.858153 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 341) took 1044 ms. Trace:
I20260709 12:21:43.858530 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 314) took 1225 ms. Trace:
I20260709 12:21:43.859020 31822 rpcz_store.cc:276] 0709 12:21:42.633169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.640112 (+  6943us) service_pool.cc:224] Handling call
0709 12:21:43.858357 (+1218245us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54004461d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:43.858476 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2755330816}
I20260709 12:21:43.859686 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 624) took 1048 ms. Trace:
I20260709 12:21:43.860066 32370 rpcz_store.cc:276] 0709 12:21:42.811567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.811773 (+   206us) service_pool.cc:224] Handling call
0709 12:21:43.859501 (+1047728us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b540055da54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:43.859631 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":442923264}
I20260709 12:21:43.860553 31822 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: [7.7-8.8]
I20260709 12:21:43.861585 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1987) took 1058 ms. Trace:
I20260709 12:21:43.862296 31822 rpcz_store.cc:276] 0709 12:21:42.803050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.859887 (+1056837us) service_pool.cc:224] Handling call
0709 12:21:43.860297 (+   410us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:43.860952 (+   655us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:43.861523 (+   571us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:43.866107  2015 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:43.805837  2201 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.869184 31748 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:43.811833  2020 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.808609  1443 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.339411  2104 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.843695  2079 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:43.879387  2079 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.879722  2161 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:43.881479  2079 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:43.825275  1950 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.848896  2219 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:43.882812  2219 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:43.883641  2219 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.884603  2219 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.886850 31678 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:43.888015  2157 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:43.888566  1950 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:43.889148 31670 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:43.849592  1897 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:43.850031  1544 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:43.578712  2211 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.845882 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1865) took 1620 ms. Trace:
I20260709 12:21:43.890283  2192 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:43.890206 31673 rpcz_store.cc:276] 0709 12:21:42.225250 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.494879 (+269629us) service_pool.cc:224] Handling call
0709 12:21:42.507130 (+ 12251us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:42.527199 (+ 20069us) raft_consensus.cc:1534] Early marking committed up to index 1
0709 12:21:42.527230 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:42.536078 (+  8848us) log.cc:844] Serialized 26 byte log entry
0709 12:21:42.723758 (+187680us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:42.723814 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:42.723910 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:43.287139 (+563229us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:43.841001 (+553862us) raft_consensus.cc:1685] finished
0709 12:21:43.841023 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:43.845801 (+  4778us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19613,"raft.queue_time_us":26884,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":14131,"spinlock_wait_cycles":25265536,"thread_start_us":168869,"threads_started":1,"wal-append.queue_time_us":191544}
I20260709 12:21:43.892009  2130 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:43.892604 31673 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:43.892796 32369 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:43.893347 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1884) took 1285 ms. Trace:
W20260709 12:21:43.851657 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62283.1us
I20260709 12:21:43.855545 31821 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:21:43.893690 31673 rpcz_store.cc:276] 0709 12:21:42.608058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.892040 (+1283982us) service_pool.cc:224] Handling call
0709 12:21:43.892434 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.893094 (+   660us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:43.893277 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:43.894215 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1979) took 1095 ms. Trace:
I20260709 12:21:43.894601 31821 rpcz_store.cc:276] 0709 12:21:42.798466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.854865 (+1056399us) service_pool.cc:224] Handling call
0709 12:21:43.855209 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.893985 (+ 38776us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:43.894162 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:43.895885 31821 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:21:43.897058  1544 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:43.858604 31671 rpcz_store.cc:276] 0709 12:21:42.813624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.912080 (+ 98456us) service_pool.cc:224] Handling call
0709 12:21:43.857927 (+945847us) spinlock_profiling.cc:243] Waited 359 ms on lock 0x7b54002b1454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:43.858085 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":826176256}
I20260709 12:21:43.842914 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:43.898288 31819 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:43.898103  1544 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.899515 31819 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: [3.6-5.7]
I20260709 12:21:43.900076  1627 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:43.900686  1627 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.900951 31819 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:21:43.832103  2223 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:43.820550  2209 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.878667  2049 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:43.897222 31821 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:21:43.904543  1951 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:43.905550  1951 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:43.906733  2209 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:43.906551  1951 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.907521  2197 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.907982  1951 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:43.842566  2107 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:43.845275 31826 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:43.908862 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1425) took 1878 ms. Trace:
I20260709 12:21:43.930837 31750 rpcz_store.cc:276] 0709 12:21:42.030074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.363879 (+333805us) service_pool.cc:224] Handling call
0709 12:21:42.387351 (+ 23472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:42.451723 (+ 64372us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:42.451793 (+    70us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:43.341157 (+889364us) spinlock_profiling.cc:243] Waited 3.44 ms on lock 0x7b18004bd420. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a
0709 12:21:43.341993 (+   836us) spinlock_profiling.cc:243] Waited 1.9 ms on lock 0x7b2c00001158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:21:43.357852 (+ 15859us) log.cc:844] Serialized 26 byte log entry
0709 12:21:43.646427 (+288575us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:43.796164 (+149737us) log.cc:844] Serialized 14 byte log entry
0709 12:21:43.796370 (+   206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:43.796448 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:43.857355 (+ 60907us) spinlock_profiling.cc:243] Waited 3.89 ms on lock 0x7b18004bd420. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:21:43.908642 (+ 51287us) raft_consensus.cc:1685] finished
0709 12:21:43.908667 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:43.908802 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13303,"prepare.queue_time_us":149531,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":55,"raft.queue_time_us":33874,"raft.run_cpu_time_us":327,"raft.run_wall_time_us":9353,"spinlock_wait_cycles":21252992,"thread_start_us":209548,"threads_started":1,"wal-append.queue_time_us":441894}
I20260709 12:21:43.939215  2107 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:43.945425  1945 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:43.945701  2018 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:43.948518 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.950598  1945 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:43.950809  2201 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:43.951365  1945 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.953598  1945 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:43.955063 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 932547us
I20260709 12:21:43.956063 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:43.957733 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:43.940291 31818 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260709 12:21:43.960457 31677 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:43.833529  2108 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:43.967214 31824 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:43.968508 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:43.973044 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:43.986397 31671 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:43.973513 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:43.987403 31673 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.988472 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:43.847252  2137 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.083s
I20260709 12:21:43.989152 31745 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 4.
I20260709 12:21:43.844370  2113 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:43.992002 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:43.975353 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:43.979598 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 228736us
I20260709 12:21:43.967955  1897 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:43.990656 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:43.993266 32370 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:43.991541 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:43.991797 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:43.968137  2201 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:44.001770 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:44.001772 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:44.004544 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:44.007109 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:44.007931 31818 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:44.009361 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:44.010210 31825 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:44.011726 31748 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:44.012346 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:44.012928 31824 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:43.978978  2174 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.016019 31751 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:21:44.020592 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:44.023082 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:44.023785 31674 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:44.024151 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 236135us
I20260709 12:21:44.024325 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:44.026051 31718 connection.cc:256] RPC call timeout handler was delayed by -1.3498015529999066s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1815s for a response
I20260709 12:21:44.026786 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:44.037122 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 251543us
I20260709 12:21:44.040139 31745 pending_rounds.cc:85] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 1
I20260709 12:21:44.040535 31745 pending_rounds.cc:107] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260709 12:21:44.040800 31745 raft_consensus.cc:2889] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:44.039291 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:43.977164  1627 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:44.041447 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:44.059907 31752 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:44.047317 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:44.062768 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:44.068343 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:44.043583 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:44.071110 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:44.079043 31791 leader_election.cc:400] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
I20260709 12:21:44.082701 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:44.083606 31751 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:44.087970 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:44.088655 31752 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:44.016630  2157 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:44.090219 32374 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260709 12:21:44.093964 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 319) took 1197 ms. Trace:
I20260709 12:21:44.094187 31643 leader_election.cc:400] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8.
I20260709 12:21:44.101598 31643 leader_election.cc:403] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:21:44.106057 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54852.4us
I20260709 12:21:44.106580 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:44.107210 31746 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.107992 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:44.108374 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:44.109241 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:44.109972 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1881) took 1502 ms. Trace:
I20260709 12:21:44.109256 31822 rpcz_store.cc:276] 0709 12:21:42.896408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.863348 (+966940us) service_pool.cc:224] Handling call
0709 12:21:43.863703 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.863930 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:43.979181 (+115251us) log.cc:844] Serialized 14 byte log entry
0709 12:21:44.090684 (+111503us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:44.093527 (+  2843us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:44.093612 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.093721 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:44.093886 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":224821,"threads_started":2}
W20260709 12:21:44.112687 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45283.4us
I20260709 12:21:44.110529 31677 rpcz_store.cc:276] 0709 12:21:42.607276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.150986 (+543710us) service_pool.cc:224] Handling call
0709 12:21:43.151298 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.968582 (+817284us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.109736 (+141154us) spinlock_profiling.cc:243] Waited 809 ms on lock 0x7b54002b0a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:44.109911 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1860333312}
I20260709 12:21:44.092852 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:44.110503 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:44.096223 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:44.100906 32374 pending_rounds.cc:85] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 5
I20260709 12:21:44.118112 32374 pending_rounds.cc:107] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:21:44.118408 32374 raft_consensus.cc:2889] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:44.119917 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:44.120539 32374 raft_consensus.cc:2376] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7. Re-sending same reply.
I20260709 12:21:44.121588 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:44.121893  1931 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:44.028774  2073 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:44.122921 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:44.123015  2073 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:44.123618 31748 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:44.123953  2206 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:44.028262  1613 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.125747 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:44.122349 32374 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:44.014415  2201 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:44.130800 31753 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260709 12:21:44.164446 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 56194.7us
I20260709 12:21:44.165750 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:44.166718 31822 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:44.123484  1931 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:44.168486 32375 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:44.170037 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:44.170439 31820 raft_consensus.cc:2368] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9.
I20260709 12:21:44.012712  1897 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:44.184406 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 211082us
I20260709 12:21:44.184527 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:44.185853 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:44.186196 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:44.188575 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:44.060071  2204 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.175170 32373 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:44.089572  2223 raft_consensus.cc:2764] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260709 12:21:44.181947 31748 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:44.199486 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:44.183434  2094 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:44.208221 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:44.208458  2073 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:44.209093 32374 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:44.211743  2130 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:44.212639 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:44.213361 31822 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:44.215914 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:44.216665 32373 raft_consensus.cc:2368] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7.
I20260709 12:21:44.216462  1897 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:44.195852  1921 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:44.217537 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:44.120216  2079 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.099s
W20260709 12:21:44.219347 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 108901us
I20260709 12:21:44.208259  2094 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.219877  2079 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:44.220281  2094 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:44.220422  2079 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:44.221475  2079 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:44.224241  2130 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:44.077692  1908 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:44.078047  2049 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.247177  2049 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:44.247913  2130 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:44.250109  2049 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:44.246354 32374 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:44.088901  2160 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:44.077404  1945 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:44.090026  2093 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:44.196624 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:44.225467 31671 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:44.099990  2209 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:44.229010 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:44.232575 31673 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:44.098322  2064 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:44.283406 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 623) took 1478 ms. Trace:
I20260709 12:21:44.028460  2174 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:44.237010 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:44.288776 32369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 629) took 1409 ms (client timeout 1775 ms). Trace:
W20260709 12:21:44.239076 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53324.5us
W20260709 12:21:44.289312 32369 rpcz_store.cc:269] 0709 12:21:42.878979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.882208 (+  3229us) service_pool.cc:224] Handling call
0709 12:21:44.288541 (+1406333us) spinlock_profiling.cc:243] Waited 992 ms on lock 0x7b54004312d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:44.288706 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2282597248}
I20260709 12:21:44.290020 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:44.262573 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:44.291611 31753 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:44.264251 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:44.267279  1443 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:44.267712 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 627) took 1389 ms. Trace:
I20260709 12:21:44.270077 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:44.295625 32378 rpcz_store.cc:276] 0709 12:21:42.878161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.882792 (+  4631us) service_pool.cc:224] Handling call
0709 12:21:43.749347 (+866555us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.851658 (+102311us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.267423 (+415765us) spinlock_profiling.cc:243] Waited 390 ms on lock 0x7b54004984d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:44.267613 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":897326336}
I20260709 12:21:44.198443  2020 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:44.225219  2213 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:44.225541  2098 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260709 12:21:44.273298 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 76755.5us
I20260709 12:21:44.276140 31748 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:44.271893  2223 raft_consensus.cc:2764] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:21:44.327286 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:44.328004 31748 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:44.277994 32373 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260709 12:21:44.279767 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 336) took 1922 ms. Trace:
I20260709 12:21:44.329600 31670 rpcz_store.cc:276] 0709 12:21:42.356848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.021883 (+665035us) service_pool.cc:224] Handling call
0709 12:21:43.058669 (+ 36786us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.889575 (+830906us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.279455 (+389880us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b5400046ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:44.279692 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1909481344}
I20260709 12:21:44.279932 31746 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:44.323616 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:44.182682 31674 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:44.270236 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:44.297277 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:44.299015 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:44.191908  1613 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:44.173756  2157 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:44.306692 31824 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:44.283975 32377 rpcz_store.cc:276] 0709 12:21:42.804677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.805032 (+   355us) service_pool.cc:224] Handling call
0709 12:21:42.855241 (+ 50209us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:42.884870 (+ 29629us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:42.885179 (+   309us) log.cc:844] Serialized 14 byte log entry
0709 12:21:42.992705 (+107526us) log.cc:844] Serialized 14 byte log entry
0709 12:21:43.954462 (+961757us) spinlock_profiling.cc:243] Waited 2.18 ms on lock 0x7b18003016c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:21:43.954678 (+   216us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:43.954733 (+    55us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:43.954792 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.123053 (+168261us) spinlock_profiling.cc:243] Waited 1.95 ms on lock 0x7b18003016c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:21:44.283071 (+160018us) spinlock_profiling.cc:243] Waited 29.3 ms on lock 0x7b54004272d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:44.283138 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:44.283326 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":995749,"prepare.queue_time_us":3016,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":84,"raft.queue_time_us":961969,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":19989,"spinlock_wait_cycles":76989568,"thread_start_us":106059,"threads_started":1,"wal-append.queue_time_us":178844,"wal-append.run_cpu_time_us":1966,"wal-append.run_wall_time_us":1099286}
W20260709 12:21:44.300182 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 315151us
I20260709 12:21:44.268586  2182 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:44.302172 31822 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 5: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:44.285801 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:44.332835 31752 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:44.338506 31677 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:44.339956 31752 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:21:44.341645 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:44.270968  1627 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:44.350340 31677 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
W20260709 12:21:44.351464 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 81308.3us
I20260709 12:21:44.327103 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:44.279839  2093 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.349910 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
W20260709 12:21:44.299970  2123 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.547s	user 0.005s	sys 0.023s
I20260709 12:21:44.294984  1751 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8.
W20260709 12:21:44.406409 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58356.2us
I20260709 12:21:44.486435 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:44.488750 31821 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.048s	sys 0.098s
I20260709 12:21:44.357688 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:44.489128 31821 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.048s	sys 0.098s
I20260709 12:21:44.489415 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:44.300107  2209 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:44.490079 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:44.490892 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:44.446412 31752 raft_consensus.cc:606] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:44.491971 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:44.357604 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:44.492674 31753 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:44.447229 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:44.448675 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:44.495623 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 98106.6us
I20260709 12:21:44.496315 31670 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:44.443322 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 101804us
I20260709 12:21:44.449575 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:44.498220 31674 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:44.499424 31824 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:44.235378  2204 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:44.484411 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:44.505072 31748 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:44.448828 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1874) took 2219 ms. Trace:
I20260709 12:21:44.505754 31676 rpcz_store.cc:276] 0709 12:21:42.229766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.919303 (+689537us) service_pool.cc:224] Handling call
0709 12:21:42.919633 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.075661 (+156028us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.448558 (+1372897us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b540017c5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:44.448748 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":346585984}
I20260709 12:21:44.352380  2061 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:21:44.484800  2123 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.704s	user 0.005s	sys 0.023s
I20260709 12:21:44.507251 31671 raft_consensus.cc:2368] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8.
I20260709 12:21:44.505148  1897 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:44.484539  2122 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:44.508564  2018 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:44.509631  1944 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:44.443856  2187 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:44.448649  2219 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.073s
I20260709 12:21:44.511909 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:44.521965 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:44.545955 31818 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:44.524701 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "5206fd0087404d2da964a5d985a04ff4"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:21:44.546957 31752 raft_consensus.cc:606] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Received request to transfer leadership to TS 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:44.489719  2108 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:44.286976  1765 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:44.528862 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:44.358790 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:44.530673 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 203653us
W20260709 12:21:44.560263 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 339310us
I20260709 12:21:44.525400 31674 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:44.501080 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:44.510532  2062 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:44.529654 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:44.538272  2015 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:44.322398  2213 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.547653 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:44.538131 32371 raft_consensus.cc:2368] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7.
I20260709 12:21:44.326043  2098 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:44.486884  2093 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:44.566627 31643 leader_election.cc:400] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7.
I20260709 12:21:44.588235 31643 leader_election.cc:403] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:21:44.564350 31746 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:44.529947  2113 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:44.558655 31670 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:44.567581 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:44.562697 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1986) took 1759 ms. Trace:
I20260709 12:21:44.567237 31824 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:44.563966 31748 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:44.358067  2182 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:44.296762  2064 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:44.323139  2156 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.856s	user 0.016s	sys 0.008s
I20260709 12:21:44.533860  2160 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.570830 32378 raft_consensus.cc:2410] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 9 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:21:44.565433 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1883) took 1957 ms. Trace:
I20260709 12:21:44.534304  1897 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:44.533322  1944 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:44.523459  1977 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:21:44.538890  2122 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:21:44.552479  1951 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.623s	user 0.010s	sys 0.000s
I20260709 12:21:44.519850  2131 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:21:44.543779  2018 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:44.513125  2234 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.063s
W20260709 12:21:44.599532  2156 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.133s	user 0.016s	sys 0.016s
I20260709 12:21:44.590826  2098 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:44.594218 31669 rpcz_store.cc:276] 0709 12:21:42.607801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.822296 (+1214495us) service_pool.cc:224] Handling call
0709 12:21:43.837551 (+ 15255us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.837785 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:44.044636 (+206851us) log.cc:844] Serialized 14 byte log entry
0709 12:21:44.565001 (+520365us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:44.565049 (+    48us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:44.565104 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.565170 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:44.565289 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78659,"prepare.queue_time_us":285721,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":76303,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":5849,"thread_start_us":621081,"threads_started":2,"wal-append.queue_time_us":529066}
I20260709 12:21:44.595613 31827 rpcz_store.cc:276] 0709 12:21:42.802824 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.870802 (+1067978us) service_pool.cc:224] Handling call
0709 12:21:43.875629 (+  4827us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.875864 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:44.082029 (+206165us) log.cc:844] Serialized 14 byte log entry
0709 12:21:44.561564 (+479535us) log.cc:844] Serialized 14 byte log entry
0709 12:21:44.562312 (+   748us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:44.562358 (+    46us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:44.562404 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.562485 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:44.562612 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":335752,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":85,"raft.queue_time_us":8633,"spinlock_wait_cycles":4269824,"thread_start_us":670050,"threads_started":2}
I20260709 12:21:44.535250  2094 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:44.598197 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:44.571820  2161 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:44.598491  2062 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:44.607947 31748 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:44.620066  2160 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:44.580617  2061 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:44.627740 31677 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:44.603626 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:44.630089  1897 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:44.776429 31746 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
W20260709 12:21:44.631088  2243 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
W20260709 12:21:44.631810  2107 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.690s	user 0.011s	sys 0.001s
I20260709 12:21:44.600195  2015 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:44.635236  2123 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:44.636865 31676 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:44.592463 31674 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:44.620417  1944 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:44.573921  1443 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:21:44.693804 31824 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:44.628335  1627 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7.
I20260709 12:21:44.620080  2201 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:44.723658 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 211806us
W20260709 12:21:44.792452 31750 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.012s	sys 0.016s
I20260709 12:21:44.718290 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:44.601606 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1864) took 2367 ms. Trace:
I20260709 12:21:44.594597 31718 leader_election.cc:400] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7.
I20260709 12:21:44.813731 31718 leader_election.cc:403] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260709 12:21:44.602170  1931 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.091s
I20260709 12:21:44.762688 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:44.813656 31672 rpcz_store.cc:276] 0709 12:21:42.224583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.478657 (+254074us) service_pool.cc:224] Handling call
0709 12:21:42.478989 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:43.021242 (+542253us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.591872 (+1570630us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b54002549d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:44.592162 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1216936320}
I20260709 12:21:44.815245 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:44.583564  2213 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:44.818145 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 223654us
W20260709 12:21:44.773774 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 170217us
I20260709 12:21:44.819036 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:44.786352 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:44.821620 31744 raft_consensus.cc:2368] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8.
I20260709 12:21:44.788956 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:44.729403  2131 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:44.729178  2187 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:44.803978 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1978) took 2006 ms. Trace:
W20260709 12:21:44.790025  2248 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.278s	user 0.000s	sys 0.001s
I20260709 12:21:44.823493 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:44.775475  2174 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:44.806906 31746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
W20260709 12:21:44.813112 31750 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.012s	sys 0.016s
W20260709 12:21:44.788983  2206 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s	user 0.011s	sys 0.015s
I20260709 12:21:44.625176  2209 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:44.766894  1613 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:44.788473  1765 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:44.823554 31823 rpcz_store.cc:276] 0709 12:21:42.797686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.846314 (+1048628us) service_pool.cc:224] Handling call
0709 12:21:43.846952 (+   638us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.910309 (+ 63357us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:44.050173 (+139864us) log.cc:844] Serialized 14 byte log entry
0709 12:21:44.324995 (+274822us) log.cc:844] Serialized 14 byte log entry
0709 12:21:44.325213 (+   218us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:44.325259 (+    46us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:44.325324 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.803547 (+478223us) spinlock_profiling.cc:243] Waited 63.1 ms on lock 0x7b54004466d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:44.803763 (+   216us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:44.803917 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28639,"prepare.queue_time_us":331181,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":80,"raft.queue_time_us":122359,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":32543,"spinlock_wait_cycles":167277312,"thread_start_us":340468,"threads_started":2,"wal-append.queue_time_us":516682}
I20260709 12:21:44.786922  2113 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:44.843135 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:44.796618  2061 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:44.870106 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:44.873418 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 193665us
I20260709 12:21:44.791891  2107 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:44.875073 32370 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:21:44.799716  2073 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.029s
W20260709 12:21:44.840497  2206 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.011s	sys 0.015s
I20260709 12:21:44.878448 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:44.877929 32378 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:44.884373 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:44.887045 31747 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:44.824601  2131 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:44.892814 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1990) took 1129 ms. Trace:
I20260709 12:21:44.846700  2190 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:44.874507  1613 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:44.914453 32372 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:44.825241  2201 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7.
I20260709 12:21:44.893607 31821 rpcz_store.cc:276] 0709 12:21:43.763127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.909323 (+146196us) service_pool.cc:224] Handling call
0709 12:21:43.909601 (+   278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.909789 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:44.489604 (+579815us) log.cc:844] Serialized 14 byte log entry
0709 12:21:44.891388 (+401784us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:44.891481 (+    93us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:44.891559 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.891674 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:44.892724 (+  1050us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":301309,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"thread_start_us":980812,"threads_started":3,"wal-append.queue_time_us":379406}
I20260709 12:21:44.897217 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:44.938308 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:44.939061 31748 raft_consensus.cc:2368] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7.
I20260709 12:21:44.897542 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:44.803306  2123 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:44.799220  2018 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:44.921322 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 132437us
I20260709 12:21:44.792964  2182 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:44.939790 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:44.956496 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:44.792398  2122 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:44.840312  2253 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.075s
W20260709 12:21:45.020501 31753 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.517s	user 0.011s	sys 0.000s
I20260709 12:21:45.021111 31753 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
W20260709 12:21:44.898409  2252 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.260s	user 0.000s	sys 0.002s
I20260709 12:21:44.886243  2157 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:44.893276  1951 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.059s
I20260709 12:21:44.938063  2206 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:21:45.023595 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 68191.7us
W20260709 12:21:45.045560 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 224437us
I20260709 12:21:45.048597 31746 raft_consensus.cc:606] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:45.049676 31669 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 12:21:44.963660  2250 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.342s	user 0.000s	sys 0.001s
W20260709 12:21:45.009178  2251 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.383s	user 0.001s	sys 0.001s
I20260709 12:21:44.890235  1627 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260709 12:21:44.953435  1613 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:45.007546  2093 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:45.061811  2255 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.278s	user 0.001s	sys 0.000s
I20260709 12:21:45.165076 31746 raft_consensus.cc:614] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:21:45.202744 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1923) took 1163 ms. Trace:
I20260709 12:21:45.203224 31745 rpcz_store.cc:276] 0709 12:21:44.039483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.039598 (+   115us) service_pool.cc:224] Handling call
0709 12:21:44.039949 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:44.084451 (+ 44502us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:44.084478 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:44.085025 (+   547us) log.cc:844] Serialized 26 byte log entry
0709 12:21:44.514906 (+429881us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:44.514934 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.514982 (+    48us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:45.202471 (+687489us) raft_consensus.cc:1685] finished
0709 12:21:45.202493 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.202672 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":321954,"threads_started":1,"wal-append.queue_time_us":523838}
W20260709 12:21:45.183972  2245 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.502s	user 0.001s	sys 0.001s
I20260709 12:21:45.168093  2187 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:45.234845  2264 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:45.235560  2264 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:45.236765  2264 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:45.238116 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1922) took 1199 ms. Trace:
I20260709 12:21:45.238565 31750 rpcz_store.cc:276] 0709 12:21:44.039060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.039178 (+   118us) service_pool.cc:224] Handling call
0709 12:21:44.085439 (+ 46261us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:44.125480 (+ 40041us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:44.125514 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:44.171216 (+ 45702us) log.cc:844] Serialized 26 byte log entry
0709 12:21:44.828449 (+657233us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:44.828506 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.828603 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:45.234398 (+405795us) raft_consensus.cc:1685] finished
0709 12:21:45.234425 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.234614 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39666,"raft.queue_time_us":110874,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":121,"thread_start_us":560454,"threads_started":1,"wal-append.queue_time_us":977180}
W20260709 12:21:45.140213  2256 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.261s	user 0.000s	sys 0.001s
W20260709 12:21:45.139546  2263 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.241s	user 0.001s	sys 0.000s
I20260709 12:21:45.073204  2131 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:45.219944  2123 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:45.211047  2254 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.384s	user 0.000s	sys 0.001s
I20260709 12:21:45.215516  2204 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:45.211982  1613 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:45.213577  2259 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.277s	user 0.000s	sys 0.000s
W20260709 12:21:45.204658  2246 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.488s	user 0.001s	sys 0.000s
I20260709 12:21:45.167397  1897 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:45.247937 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1989) took 1485 ms. Trace:
I20260709 12:21:45.248595 31819 rpcz_store.cc:276] 0709 12:21:43.762844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.901809 (+138965us) service_pool.cc:224] Handling call
0709 12:21:43.902104 (+   295us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:43.902476 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:43.902506 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:43.907688 (+  5182us) log.cc:844] Serialized 26 byte log entry
0709 12:21:44.299470 (+391782us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:44.299516 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.299585 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:45.247645 (+948060us) raft_consensus.cc:1685] finished
0709 12:21:45.247676 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.247865 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":789,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":264,"thread_start_us":391240,"threads_started":1,"wal-append.queue_time_us":684360}
I20260709 12:21:45.210229  1627 raft_consensus.cc:2764] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:21:45.220273  2160 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:45.247889  1897 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:45.219589  2107 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:45.026072  2174 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.165s
I20260709 12:21:45.165720  2157 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:44.899667  2015 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.067416  2113 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:45.275750 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:45.207265  2206 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:45.161581 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:45.305532 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 144096us
I20260709 12:21:45.305673 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:45.306339 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:44.889515 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.752s	user 0.004s	sys 0.009s
W20260709 12:21:45.307192 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.169s	user 0.004s	sys 0.009s
I20260709 12:21:45.268766 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:44.951447  2062 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:45.308880 31748 raft_consensus.cc:2410] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 8 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
W20260709 12:21:45.267688  2250 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:21:45.299928 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:45.314486 31824 raft_consensus.cc:2410] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 8 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260709 12:21:45.299975 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:45.054979  2061 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:45.315725 31819 raft_consensus.cc:2410] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 8 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:45.041868  1765 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.267299  2157 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:21:45.319063  2258 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.477s	user 0.000s	sys 0.001s
I20260709 12:21:45.319479  2157 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:45.320936  2157 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:45.322278  1897 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:45.323863  1897 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:45.304296 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:45.259971 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 303550us
I20260709 12:21:45.326066 31821 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260709 12:21:45.307619 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 1.21473e+06us
I20260709 12:21:45.326471 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:45.325683 31822 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:45.312103 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:45.329321 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:45.329591 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:45.331584 31718 leader_election.cc:400] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9.
I20260709 12:21:45.334532  2279 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:45.347724  2279 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.348233  2279 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:45.349009  2174 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.349633  2174 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:45.339061 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1918) took 1351 ms. Trace:
I20260709 12:21:45.304316  2264 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:45.339857  1627 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:45.323169  2098 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.594s	user 0.011s	sys 0.000s
I20260709 12:21:45.351518  2098 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.353724  2098 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:45.334105  2107 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:45.327389 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:45.354732  2277 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:45.355744 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:45.355945  2123 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:45.356472  2174 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:45.356818  2123 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:45.357105  2267 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:45.357285  2123 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:45.357563  2267 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.357952  2267 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:45.350826 31749 rpcz_store.cc:276] 0709 12:21:43.987863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.987970 (+   107us) service_pool.cc:224] Handling call
0709 12:21:43.988323 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:43.988708 (+   385us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:43.988732 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:44.013001 (+ 24269us) log.cc:844] Serialized 26 byte log entry
0709 12:21:44.527623 (+514622us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:44.527682 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:44.527772 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:45.338735 (+810963us) raft_consensus.cc:1685] finished
0709 12:21:45.338762 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.338982 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2389,"raft.queue_time_us":1231,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":10481,"thread_start_us":445232,"threads_started":1,"wal-append.queue_time_us":597529}
I20260709 12:21:45.359238 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:45.356570  2269 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:45.360646  2269 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.360970  2269 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:45.361660  2234 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:45.362068  2234 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.362679  2234 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:45.364290 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:45.365161  2271 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:45.365593  2271 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.365950  2271 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:45.324949  2253 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:45.369201 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:45.345302 31718 leader_election.cc:403] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:45.314558  1931 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:45.344259 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:45.371989 31822 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:45.377724  2253 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.340894  2274 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:45.394578 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:45.376402 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45007.5us
I20260709 12:21:45.404881 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:45.379376  2190 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:45.406888 31824 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:45.408196 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:45.392570  2098 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:45.358170  2123 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 6, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:45.417645 32370 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:21:45.380911  2093 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.034s
I20260709 12:21:45.406574  2253 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:45.434341 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:45.430949 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:45.393520  1627 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.022s
I20260709 12:21:45.406152  2274 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:45.436385 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:45.438083 31822 raft_consensus.cc:2368] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8.
I20260709 12:21:45.418805  1931 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:45.414812  2280 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:45.438095 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:45.403280  2275 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 12:21:45.440358 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:45.450085 31821 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:45.452147 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:45.452351 32369 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:45.453148 31819 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:45.466745 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:45.467636 31819 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:45.467258 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:45.458792 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 89661.3us
I20260709 12:21:45.471853 31791 leader_election.cc:400] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
I20260709 12:21:45.391675  2206 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:45.438881  2274 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:45.449641  1931 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:45.447678  2123 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:45.479291 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:45.497946 31819 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:45.483527  1944 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.667s	user 0.007s	sys 0.003s
I20260709 12:21:45.486954 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 355) took 1397 ms. Trace:
I20260709 12:21:45.509533 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1899) took 1164 ms. Trace:
I20260709 12:21:45.509562 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:45.472465 31791 leader_election.cc:403] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:45.493458 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:45.493793 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1870) took 3266 ms. Trace:
I20260709 12:21:45.509925 31751 rpcz_store.cc:276] 0709 12:21:44.089986 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.092156 (+  2170us) service_pool.cc:224] Handling call
0709 12:21:44.092448 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:44.221685 (+129237us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:21:44.222056 (+   371us) log.cc:844] Serialized 14 byte log entry
0709 12:21:44.685216 (+463160us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:44.685310 (+    94us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:44.685395 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.486719 (+801324us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b54004033d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:45.486767 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.486885 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37291,"prepare.queue_time_us":475,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":429,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":24247,"spinlock_wait_cycles":296560000,"thread_start_us":458858,"threads_started":1,"wal-append.queue_time_us":582774}
I20260709 12:21:45.512020 31675 rpcz_store.cc:276] 0709 12:21:42.227756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:42.916611 (+688855us) service_pool.cc:224] Handling call
0709 12:21:42.916978 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.843814 (+926836us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.493538 (+1649724us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x7b540022a954. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:45.493730 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2116356352}
I20260709 12:21:45.414275  2113 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:45.520634 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 48988.8us
W20260709 12:21:45.463097 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:45.503260 31827 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:45.507594 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260709 12:21:45.487475  2213 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.641s	user 0.011s	sys 0.000s
I20260709 12:21:45.510048 31669 rpcz_store.cc:276] 0709 12:21:44.345121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.323427 (+978306us) service_pool.cc:224] Handling call
0709 12:21:45.323810 (+   383us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:45.324031 (+   221us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:45.324056 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:45.324538 (+   482us) log.cc:844] Serialized 26 byte log entry
0709 12:21:45.432398 (+107860us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:45.432435 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.432500 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:45.509298 (+ 76798us) raft_consensus.cc:1685] finished
0709 12:21:45.509323 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.509460 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":107519,"threads_started":1,"wal-append.queue_time_us":176943}
I20260709 12:21:45.438488  2107 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9.
I20260709 12:21:45.533353 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 1998) took 1035 ms. Trace:
I20260709 12:21:45.515002 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:45.524013  2093 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:45.530776  2213 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.509881  1931 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:45.549064 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:45.567765 31827 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:45.551370 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1882) took 2943 ms. Trace:
I20260709 12:21:45.568920 31678 rpcz_store.cc:276] 0709 12:21:42.607556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.767291 (+1159735us) service_pool.cc:224] Handling call
0709 12:21:43.767572 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:43.887288 (+119716us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.551058 (+1663770us) spinlock_profiling.cc:243] Waited 119 ms on lock 0x7b540017cfd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:45.551298 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":273863936}
I20260709 12:21:45.514685  1944 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:45.555233 31673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1895) took 1474 ms (client timeout 1717 ms). Trace:
W20260709 12:21:45.572770 31673 rpcz_store.cc:269] 0709 12:21:44.080361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.298926 (+218565us) service_pool.cc:224] Handling call
0709 12:21:45.555034 (+1256108us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400046fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:45.555167 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2569161088}
W20260709 12:21:45.556092 31671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1892) took 1533 ms (client timeout 1989 ms). Trace:
W20260709 12:21:45.574122 31671 rpcz_store.cc:269] 0709 12:21:44.022584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.297000 (+274416us) service_pool.cc:224] Handling call
0709 12:21:45.555891 (+1258891us) spinlock_profiling.cc:243] Waited 174 ms on lock 0x7b54001824d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:45.556032 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":399547648}
I20260709 12:21:45.533927 31818 rpcz_store.cc:276] 0709 12:21:44.497597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.566518 (+ 68921us) service_pool.cc:224] Handling call
0709 12:21:45.532418 (+965900us) spinlock_profiling.cc:243] Waited 2.6 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:45.532682 (+   264us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:45.532955 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:45.532993 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:45.533048 (+    55us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:45.533094 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.533169 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.533289 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5990656}
I20260709 12:21:45.560401 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:45.571048 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:45.590220 31818 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:45.590795 31818 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:45.520237  2182 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:45.473431  2190 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:45.652202 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 242930us
I20260709 12:21:45.654645 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:45.623862 31670 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:21:45.679509 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:45.662429 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:45.653524 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:45.703473 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:45.592519 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:45.713315  2236 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:45.715495 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:45.567199  2201 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:45.723809  2201 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:45.578311  2282 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 12:21:45.727768  2201 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:45.717635 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:45.727989 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 630) took 2814 ms. Trace:
I20260709 12:21:45.729100 31678 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:45.728797 32376 rpcz_store.cc:276] 0709 12:21:42.913812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.861917 (+948105us) service_pool.cc:224] Handling call
0709 12:21:43.862280 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:44.014013 (+151733us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:44.014058 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:44.901576 (+887518us) spinlock_profiling.cc:243] Waited 2.28 ms on lock 0x7b2c000eba68. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:21:44.902063 (+   487us) log.cc:844] Serialized 26 byte log entry
0709 12:21:45.255385 (+353322us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:45.255423 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.255493 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:45.727713 (+472220us) raft_consensus.cc:1685] finished
0709 12:21:45.727731 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.727927 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":270411,"raft.queue_time_us":151813,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":7048,"spinlock_wait_cycles":5238656,"thread_start_us":232265,"threads_started":1,"wal-append.queue_time_us":439256}
I20260709 12:21:45.734488 31751 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:21:45.561477  2093 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:45.735853 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:45.641134 31818 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.741340 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:45.735889 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:45.652467 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1897) took 1315 ms. Trace:
I20260709 12:21:45.742005 31818 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:45.613690  2295 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7.
I20260709 12:21:45.744170 31673 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:45.589212  2073 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:45.755700 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:45.743875 31676 rpcz_store.cc:276] 0709 12:21:44.337483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.895120 (+557637us) service_pool.cc:224] Handling call
0709 12:21:44.938990 (+ 43870us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:44.939285 (+   295us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:45.436012 (+496727us) log.cc:844] Serialized 14 byte log entry
0709 12:21:45.652093 (+216081us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:45.652154 (+    61us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:45.652211 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:45.652278 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:45.652397 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":563333,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":89979,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":41999,"spinlock_wait_cycles":27654272,"thread_start_us":572210,"threads_started":2,"wal-append.queue_time_us":247680}
I20260709 12:21:45.755697 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:45.558506  2271 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.615669  1944 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:45.753032  2279 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:45.762460 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 70499.5us
I20260709 12:21:45.765012 31825 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
W20260709 12:21:45.764940 31826 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 1992) took 2001 ms (client timeout 1664 ms). Trace:
I20260709 12:21:45.763540 31823 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
W20260709 12:21:45.768043 31826 rpcz_store.cc:269] 0709 12:21:43.763608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.792006 (+ 28398us) service_pool.cc:224] Handling call
0709 12:21:45.764684 (+1972678us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b540037a854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:45.764860 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50687360}
I20260709 12:21:45.773952 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:45.779731 31827 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:45.776121 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:21:45.782135 31718 leader_election.cc:336] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.937s (SENT)
W20260709 12:21:45.778218 31791 connection.cc:256] RPC call timeout handler was delayed by -2.5843537159998959s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1532s for a response
W20260709 12:21:45.782671 31791 connection.cc:256] RPC call timeout handler was delayed by -2.4502537159996791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
W20260709 12:21:45.783056 31791 connection.cc:256] RPC call timeout handler was delayed by -2.43855371599966s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1694s for a response
W20260709 12:21:45.783927 31791 connection.cc:256] RPC call timeout handler was delayed by -2.3431537159999607s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18000000000000002s for a response
W20260709 12:21:45.784169 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:45.784204 31791 connection.cc:256] RPC call timeout handler was delayed by -2.2513537159998123s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1902s for a response
W20260709 12:21:45.784710 31791 connection.cc:256] RPC call timeout handler was delayed by -2.0803537159999905s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
W20260709 12:21:45.785068 31791 connection.cc:256] RPC call timeout handler was delayed by -2.06775371599997s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2106s for a response
W20260709 12:21:45.785362 31791 connection.cc:256] RPC call timeout handler was delayed by -1.9246537159997388s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22650000000000003s for a response
W20260709 12:21:45.785661 31791 connection.cc:256] RPC call timeout handler was delayed by -1.9237537159997373s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22660000000000002s for a response
W20260709 12:21:45.785995 31791 connection.cc:256] RPC call timeout handler was delayed by -1.2865537160000713s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.29740000000000005s for a response
I20260709 12:21:45.773150  1944 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:45.764424  2223 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:45.764298  2236 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:45.789341 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:45.790738 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:45.774955  2079 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.008s	user 0.012s	sys 0.012s
I20260709 12:21:45.792400 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:45.787983  1944 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:45.775686  1897 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:45.800417 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:45.799355 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:45.614063  2182 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:45.772727  2260 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.782846  2295 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:45.868518  2267 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.510s	user 0.008s	sys 0.002s
I20260709 12:21:45.853551 31818 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
W20260709 12:21:45.855875 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 73762.3us
I20260709 12:21:45.857450  2299 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:45.860728  1944 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:45.799227  2213 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:45.861789 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:45.886435 31746 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:45.863101 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:45.766028  2201 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:21:45.864843 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:45.865149  2182 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:45.799619  2300 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:45.893455  2201 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:45.867735  2279 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:45.893966  2201 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:45.868652  2295 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.894955  2201 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:45.794695  2079 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.035s	user 0.013s	sys 0.012s
I20260709 12:21:45.870839 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:45.896212  2201 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:45.904131  2271 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:45.904923  2208 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:45.905407  2208 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:45.905629 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:45.906090  2261 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:45.904206  2161 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:45.918242  2234 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.555s	user 0.008s	sys 0.000s
I20260709 12:21:45.880434 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:45.790064  1613 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:45.884524 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:45.890024 31678 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:45.796993  2296 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:45.784502  2093 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.851218  2157 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:45.910543  2161 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:45.922937  2182 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:45.923449  2253 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:45.928632  2269 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.567s	user 0.013s	sys 0.000s
I20260709 12:21:45.886677  2267 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.931200  2277 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:45.932262  2160 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:45.854807 31676 raft_consensus.cc:2368] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8.
I20260709 12:21:45.944859 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:45.956035 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:45.941731  2182 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:45.961243 31818 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:45.849849  2260 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:45.982940  2182 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:45.959360  1907 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:45.983595  1907 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:45.984651  1907 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.985708  1907 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:45.963287 31822 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:45.987004 32376 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:45.986573  1931 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:45.954736  2160 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:21:45.965929 32371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 457) took 1971 ms (client timeout 1880 ms). Trace:
I20260709 12:21:45.972322  2267 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:45.951817  2277 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:45.938824  2234 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.976346  2104 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:45.975996  2274 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.906314  2208 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.953359  2269 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:45.990692 31746 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:45.962482  2093 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:45.939514  1613 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:45.943399  2296 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:46.003130 31822 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260709 12:21:46.005285 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 212937us
I20260709 12:21:46.019479 31819 raft_consensus.cc:2368] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6.
I20260709 12:21:46.099282 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:46.061229 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:46.035805 31826 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:21:46.106244  2064 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:46.114696 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1806) took 2091 ms. Trace:
I20260709 12:21:46.115483  2260 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:46.116135 31747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1916) took 2129 ms (client timeout 1643 ms). Trace:
W20260709 12:21:46.116533 31747 rpcz_store.cc:269] 0709 12:21:43.987131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.987245 (+   114us) service_pool.cc:224] Handling call
0709 12:21:46.115957 (+2128712us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b54002963d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:46.116069 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2053105024}
I20260709 12:21:46.118175 31745 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260709 12:21:46.118405  2313 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:21:46.119982  2234 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:46.121850  2269 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:46.102684  2267 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:46.122601  2253 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:46.123291  2261 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:46.115301 31825 rpcz_store.cc:276] 0709 12:21:44.022882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.023122 (+   240us) service_pool.cc:224] Handling call
0709 12:21:45.764827 (+1741705us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:45.766737 (+  1910us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.114445 (+347708us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b540037a850. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:46.114596 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4004871552}
W20260709 12:21:46.062711  2257 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.297s	user 0.004s	sys 0.000s
W20260709 12:21:46.103868 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60158.8us
I20260709 12:21:46.027878 31824 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:46.039752  2208 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:45.988852 32371 rpcz_store.cc:269] 0709 12:21:43.994656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.995586 (+   930us) service_pool.cc:224] Handling call
0709 12:21:45.965672 (+1970086us) spinlock_profiling.cc:243] Waited 412 ms on lock 0x7b5400646c54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:45.965832 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":947562368}
W20260709 12:21:46.141723 32372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 456) took 2163 ms (client timeout 1937 ms). Trace:
W20260709 12:21:46.142223 32372 rpcz_store.cc:269] 0709 12:21:43.978201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:43.978349 (+   148us) service_pool.cc:224] Handling call
0709 12:21:46.141512 (+2163163us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b54004cfed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:46.141651 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":830063360}
I20260709 12:21:46.129253  2261 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:46.127580 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:46.011969  1613 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:46.020664  2160 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:46.131901  2066 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.149379 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:46.150478  2066 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:46.153154 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:46.154565 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:46.155028 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:46.155978 31748 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:46.157207 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:46.159725 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:46.160565 31744 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:46.162163 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:46.163288 31744 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:46.130344 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:46.163803 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:46.164680 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:46.167305 31753 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:46.168705  2160 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:46.127254  2296 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:46.171389 31747 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8. Ops: [5.3-7.4]
I20260709 12:21:46.171136 31749 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8. Ops: [5.6-7.7]
I20260709 12:21:46.173800 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:46.174995 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44714.9us
I20260709 12:21:46.175315 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:46.176026 32343 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:46.015273  2190 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.135s
I20260709 12:21:46.180539 31751 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:21:46.180719 31823 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:46.011262  2161 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:46.185171 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:46.184792 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:46.187143 31749 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:46.186758 31747 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:46.181195  1908 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:46.174894  2104 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.191203  1908 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:46.192612  1908 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:46.192374  2104 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:46.184319  2261 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.010s
I20260709 12:21:46.195899  2104 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:46.181986  2296 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:46.074884  2277 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:46.231462  2277 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:46.196056 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:46.202903 31748 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:46.240191  2094 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:46.241886 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1890) took 2219 ms. Trace:
I20260709 12:21:46.242306 31670 rpcz_store.cc:276] 0709 12:21:44.021999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:44.593029 (+571030us) service_pool.cc:224] Handling call
0709 12:21:44.593354 (+   325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:45.706535 (+1113181us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.241608 (+535073us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54002e6b54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:46.241821 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2369308032}
W20260709 12:21:46.245254 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:46.248092 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 144184us
I20260709 12:21:46.249408 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:46.250190  2296 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:46.184544  1931 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:46.181543  2182 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:46.254251 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:46.257500 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:46.259477  2182 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:46.263298 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:46.272055 31820 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:46.277192 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:46.278453 31678 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:46.280660 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:46.281419 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:46.282449 31718 leader_election.cc:400] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8.
I20260709 12:21:46.282922 31718 leader_election.cc:403] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:21:46.283890 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:46.184955  2184 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:46.290580 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:46.293417 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:46.293489 31820 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:46.295142 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:46.296038 31669 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:46.296384  2211 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:46.297848 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:46.293421 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:46.296870  2211 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:46.300436 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:46.184931  1944 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.113s
I20260709 12:21:46.292956  2296 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:46.294108  2201 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:46.300230 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:46.303970 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:46.322299 31676 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:46.299870  2211 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:46.300046  2269 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:46.314215 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:46.329486 31822 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:21:46.332790 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:46.333001 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:46.329365  1908 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:46.306208  1931 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8.
I20260709 12:21:46.337703 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:46.344338 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:46.346055 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:46.340544 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:46.336014  2312 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:46.469547 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 112804us
W20260709 12:21:46.374248 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:46.336119  1907 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:46.336977  2253 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:46.361354 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:46.324126  2201 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:46.340533  2223 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:46.669032 31822 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
W20260709 12:21:46.679977 31642 connection.cc:256] RPC call timeout handler was delayed by -1.7897778449996622s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1643s for a response
W20260709 12:21:46.680426 31642 connection.cc:256] RPC call timeout handler was delayed by -1.7726778449996345s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16620000000000001s for a response
W20260709 12:21:46.687834 31642 connection.cc:256] RPC call timeout handler was delayed by -1.7708778449996316s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1664s for a response
I20260709 12:21:46.473176  1951 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:21:46.702507 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 223500us
W20260709 12:21:46.678515 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 319057us
W20260709 12:21:46.679366  2093 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.667s	user 0.003s	sys 0.007s
W20260709 12:21:46.688983  2318 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.214s	user 0.000s	sys 0.000s
I20260709 12:21:46.610491  2094 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.254s
W20260709 12:21:46.770128 31642 connection.cc:256] RPC call timeout handler was delayed by -1.6835778449999452s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1761s for a response
I20260709 12:21:46.689220  2223 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:46.773674 31642 connection.cc:256] RPC call timeout handler was delayed by -1.6781778449999365s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1767s for a response
W20260709 12:21:46.775584 31642 connection.cc:256] RPC call timeout handler was delayed by -1.6709778449999249s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1775s for a response
I20260709 12:21:46.704623  2269 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
W20260709 12:21:46.782935 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:46.724395  2184 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:46.792733 31642 connection.cc:256] RPC call timeout handler was delayed by -1.6232778449998477s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18280000000000002s for a response
I20260709 12:21:46.719653  1951 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:46.771158  2160 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.602s	user 0.011s	sys 0.000s
W20260709 12:21:46.797633 31642 connection.cc:256] RPC call timeout handler was delayed by -1.5611778449997473s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1897s for a response
I20260709 12:21:46.771664  2277 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:46.807299 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:46.807842 31642 connection.cc:256] RPC call timeout handler was delayed by -1.4927778449996367s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19730000000000003s for a response
W20260709 12:21:46.808920 31642 connection.cc:256] RPC call timeout handler was delayed by -1.477477844999612s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.199s for a response
W20260709 12:21:46.809690 31642 connection.cc:256] RPC call timeout handler was delayed by -1.4504778449995683s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.202s for a response
W20260709 12:21:46.810206 31642 connection.cc:256] RPC call timeout handler was delayed by -1.3397778449998441s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2143s for a response
W20260709 12:21:46.810788 31642 connection.cc:256] RPC call timeout handler was delayed by -1.3109180569999808s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
I20260709 12:21:46.777323  1908 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:46.723584  1945 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.076s
W20260709 12:21:46.815585 31642 connection.cc:256] RPC call timeout handler was delayed by -1.2632778449997204s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22280000000000003s for a response
W20260709 12:21:46.817682 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46754.4us
W20260709 12:21:46.822961 31642 connection.cc:256] RPC call timeout handler was delayed by -1.1894180569997843s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17020000000000002s for a response
W20260709 12:21:46.823830 31642 connection.cc:256] RPC call timeout handler was delayed by -1.1412185910003245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17400000000000002s for a response
I20260709 12:21:46.805905  1951 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:46.776661  1907 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
W20260709 12:21:46.814689  2208 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.665s	user 0.010s	sys 0.000s
I20260709 12:21:46.794152  1931 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260709 12:21:46.813273  2161 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:46.915194 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:46.918432 32377 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20260709 12:21:46.848664 31820 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.553s	user 0.010s	sys 0.000s
I20260709 12:21:46.927433 31820 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
W20260709 12:21:46.931538 31643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.896s (SENT) [suppressed 18 similar messages]
W20260709 12:21:46.936010 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 92244.9us
I20260709 12:21:46.937368 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:46.339071  2234 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:21:46.940578 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 579302us
I20260709 12:21:46.347506  2049 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:46.468668  2233 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:46.852926 31672 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.6, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.050591 31791 leader_election.cc:400] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8.
I20260709 12:21:47.056855 31791 leader_election.cc:403] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
W20260709 12:21:47.050693 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 119209us
I20260709 12:21:46.468878  2271 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:46.805299  2184 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.062456  2184 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:47.063004  2330 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
I20260709 12:21:47.065073  2312 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:46.346776  1897 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.703s
I20260709 12:21:47.065800  2312 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.068346  2271 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.069046  2271 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:46.351584  2157 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:46.814751  2277 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:47.106964 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 284430us
I20260709 12:21:47.142699 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:47.054960  2094 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:47.090041  2093 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:47.175508  2049 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.185503  2049 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:47.188587  1945 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:47.189206  1945 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:47.193252  2211 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.858s	user 0.011s	sys 0.000s
I20260709 12:21:47.193859  2211 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:47.197721 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55188.2us
I20260709 12:21:47.198686 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 334) took 1699 ms. Trace:
I20260709 12:21:47.213024 31826 rpcz_store.cc:276] 0709 12:21:45.514950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.779994 (+265044us) service_pool.cc:224] Handling call
0709 12:21:45.844892 (+ 64898us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:46.133333 (+288441us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.198245 (+1064912us) spinlock_profiling.cc:243] Waited 191 ms on lock 0x7b5400446bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:47.198579 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":438195456}
I20260709 12:21:47.046887  2208 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.109s
I20260709 12:21:47.220325 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:47.224249 31820 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:21:47.075450  2253 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.106s
I20260709 12:21:47.227821 32377 raft_consensus.cc:2368] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7.
I20260709 12:21:47.175920  2234 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.244804 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:47.085879  2233 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:47.255110  1613 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.084s	user 0.011s	sys 0.000s
I20260709 12:21:47.217953  2273 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:47.217181  2201 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.190205  1945 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.198709  2094 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.313377  2332 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:47.331748  2332 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:47.332772  2332 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.273272  2273 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:47.343539 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1832) took 1071 ms. Trace:
I20260709 12:21:47.344110 31818 rpcz_store.cc:276] 0709 12:21:46.272233 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.272641 (+   408us) service_pool.cc:224] Handling call
0709 12:21:46.272931 (+   290us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:46.273354 (+   423us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:46.300552 (+ 27198us) log.cc:844] Serialized 14 byte log entry
0709 12:21:46.471917 (+171365us) log.cc:844] Serialized 14 byte log entry
0709 12:21:46.670130 (+198213us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:46.686832 (+ 16702us) log.cc:844] Serialized 26 byte log entry
0709 12:21:46.686991 (+   159us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:46.687027 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.687102 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:47.073543 (+386441us) spinlock_profiling.cc:243] Waited 7.05 ms on lock 0x7b1800176200. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:21:47.343202 (+269659us) raft_consensus.cc:1685] finished
0709 12:21:47.343243 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:47.343453 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":202234,"prepare.queue_time_us":28843,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":2704,"raft.queue_time_us":224690,"raft.run_cpu_time_us":422,"raft.run_wall_time_us":129569,"spinlock_wait_cycles":16209152,"thread_start_us":38565,"threads_started":1,"wal-append.queue_time_us":1024404}
I20260709 12:21:47.347187 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1824) took 1861 ms. Trace:
I20260709 12:21:47.347710 31827 rpcz_store.cc:276] 0709 12:21:45.485587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.568643 (+ 83056us) service_pool.cc:224] Handling call
0709 12:21:47.346917 (+1778274us) spinlock_profiling.cc:243] Waited 59.8 ms on lock 0x7b540043dfd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:47.347109 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":137544576}
W20260709 12:21:47.348954 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 411707us
I20260709 12:21:47.256839  2234 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.328572  1945 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:47.277541  1613 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.273758  1931 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.064s
I20260709 12:21:47.335414  2324 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:47.371558 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.385577 31746 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:47.390697 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 340) took 1210 ms. Trace:
I20260709 12:21:47.391173 31825 rpcz_store.cc:276] 0709 12:21:46.180129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.182765 (+  2636us) service_pool.cc:224] Handling call
0709 12:21:46.184743 (+  1978us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:46.185102 (+   359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:46.185129 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:46.185585 (+   456us) log.cc:844] Serialized 26 byte log entry
0709 12:21:47.055619 (+870034us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b1800350a40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:21:47.277748 (+222129us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:47.277810 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.277890 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:47.384485 (+106595us) raft_consensus.cc:1685] finished
0709 12:21:47.384508 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:47.390593 (+  6085us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":175,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":101,"spinlock_wait_cycles":3245952,"thread_start_us":221860,"threads_started":1,"wal-append.queue_time_us":1177730}
I20260709 12:21:47.374042  2331 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:21:47.410315 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 197873us
I20260709 12:21:47.379247  2157 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:47.383833  2325 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.389283  2277 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:47.407027 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:47.430722  2295 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:47.431387  2295 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:47.431923  2295 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.432974  2066 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:47.433642  2066 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:21:47.434193  2295 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:47.436056  2223 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.642s	user 0.008s	sys 0.003s
I20260709 12:21:47.436437 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 464) took 2006 ms. Trace:
I20260709 12:21:47.378209  2324 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:47.433983 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.437288 32370 rpcz_store.cc:276] 0709 12:21:45.430079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.431057 (+   978us) service_pool.cc:224] Handling call
0709 12:21:46.144100 (+713043us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:46.144524 (+   424us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:46.144556 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:46.145033 (+   477us) log.cc:844] Serialized 26 byte log entry
0709 12:21:46.184714 (+ 39681us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:46.339630 (+154916us) log.cc:844] Serialized 14 byte log entry
0709 12:21:46.340314 (+   684us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.340401 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:46.340446 (+    45us) raft_consensus.cc:1685] finished
0709 12:21:46.340459 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:47.436224 (+1095765us) spinlock_profiling.cc:243] Waited 705 ms on lock 0x7b54004cfed0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:47.436370 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":142769,"prepare.queue_time_us":598348,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":5067,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":9709,"spinlock_wait_cycles":1621201664,"thread_start_us":181162,"threads_started":2,"wal-append.queue_time_us":47253}
W20260709 12:21:47.441216 31643 leader_election.cc:336] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.664s (SENT)
I20260709 12:21:47.446383 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:21:47.420876  2325 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.453518  2325 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:47.453930 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.454360 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:47.454613 31824 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1811) took 2021 ms (client timeout 2120 ms). Trace:
I20260709 12:21:47.455181 31677 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:21:47.455071 31824 rpcz_store.cc:269] 0709 12:21:45.433173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.433327 (+   154us) service_pool.cc:224] Handling call
0709 12:21:47.454413 (+2021086us) spinlock_profiling.cc:243] Waited 593 ms on lock 0x7b54003c96d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:47.454555 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1362805120}
I20260709 12:21:47.456467 31821 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:21:47.457516  2324 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:47.457955 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1829) took 1187 ms. Trace:
I20260709 12:21:47.413067  2273 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.459794 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:47.460024 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:47.460511 31819 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1826) took 1974 ms (client timeout 2021 ms). Trace:
I20260709 12:21:47.460585 31676 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:47.460666 31827 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:47.462971  2333 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.463538  2333 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.458333 31821 rpcz_store.cc:276] 0709 12:21:46.271091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.271258 (+   167us) service_pool.cc:224] Handling call
0709 12:21:47.456081 (+1184823us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.456849 (+   768us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:47.457773 (+   924us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54003c96d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:47.457898 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2724156288}
I20260709 12:21:47.465967  2333 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.468393 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.469034 31820 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:47.429765  2312 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:47.474522 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.462461 31827 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
W20260709 12:21:47.460974 31819 rpcz_store.cc:269] 0709 12:21:45.486194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.514751 (+ 28557us) service_pool.cc:224] Handling call
0709 12:21:47.460329 (+1945578us) spinlock_profiling.cc:243] Waited 483 ms on lock 0x7b54001c93d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:47.460457 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1109910784}
I20260709 12:21:47.421236  2157 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.448668  2233 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:47.485627 31820 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:21:47.471664  2211 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:47.436651  2223 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.493016 32377 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:47.481683  2094 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.481405  2232 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:47.482944  2018 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.495003 32370 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:47.482916  2253 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.486514  1931 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.495164 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:47.486289  2334 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:47.482683  2204 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.014s
W20260709 12:21:47.506444 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:47.488081  2201 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260709 12:21:47.489207  1443 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 12:21:47.542110 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:47.543884 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:47.493176  2337 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.021s
I20260709 12:21:47.546576 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:47.549418 31677 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:47.536067  2234 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:47.509075  1613 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.509853  1907 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.045s
I20260709 12:21:47.541541 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:47.536784  2295 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.500979  2232 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.536244  1443 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:47.502547  2018 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.536605  2324 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:47.502496  2253 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.505589  1931 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.565264 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:47.576802 31678 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:47.540434  2208 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.019s
I20260709 12:21:47.569983  1443 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:47.571924  2338 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.575057  2232 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:47.574855  2324 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:47.581887  2339 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20260709 12:21:47.582782  2338 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:47.590189 31643 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.567s (SENT)
W20260709 12:21:47.600992 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 250647us
I20260709 12:21:47.601912 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:47.602552  2338 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:47.603137  2327 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:47.507550  2334 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.553922  2094 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.025s
I20260709 12:21:47.604911 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:47.580081  2131 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:47.610877 31820 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
W20260709 12:21:47.621069 31792 connection.cc:256] RPC call timeout handler was delayed by -1.0297648490000029s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1875s for a response
I20260709 12:21:47.571033  2295 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.622954 31820 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:47.587288  2018 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:47.624315 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.624924 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:47.628099 31678 raft_consensus.cc:2368] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6.
I20260709 12:21:47.628458 31820 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:47.604118  2253 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.629303 31673 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:47.626485 31676 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:21:47.632925 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:47.633217 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:47.634055 31677 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:47.634224 31674 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:47.632936 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.636047 31825 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:47.638485 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:47.624574  1443 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.622735  2234 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.623230  2325 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.643429  1443 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:47.642367 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:47.644937 31673 raft_consensus.cc:2368] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8.
I20260709 12:21:47.645187 31820 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:47.646413 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.648176 31820 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:47.647653 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:47.648622 31678 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:21:47.648698 31671 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:47.649180 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.649749 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:47.651059 31673 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:47.651959 31642 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:47.652136 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 158065us
I20260709 12:21:47.654592 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:47.657327 31752 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 5.
I20260709 12:21:47.655575 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:47.658804 31749 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:47.636727 31672 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:47.663419 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:47.632599  2208 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:47.670913 31747 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:47.676724 31749 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:47.685935 31670 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:21:47.668587 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:47.642707  2157 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.611680  2327 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.706053  1443 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260709 12:21:47.706128 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54274us
I20260709 12:21:47.706322 31676 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:21:47.709455 31825 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:21:47.712529 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:47.717679 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:47.718317  2342 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:21:47.719509 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.720093 31825 raft_consensus.cc:2376] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7. Re-sending same reply.
I20260709 12:21:47.720813 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:47.721318 31825 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:21:47.721983  2093 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.519s	user 0.009s	sys 0.000s
I20260709 12:21:47.722656  2093 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8.
W20260709 12:21:47.724737  1951 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.879s	user 0.008s	sys 0.003s
I20260709 12:21:47.727200 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:47.729611  2333 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:47.730013  2333 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:47.730152 31820 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:47.731178  1931 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.731530  2334 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.716063  2208 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:47.721987 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.686589  2223 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:47.653559  2325 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.732697 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:47.739660  2327 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.758558 31820 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:47.759907  2325 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:47.733043 31792 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:47.761933 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:47.653961  2234 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.762125 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:47.725198  1951 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.770448  2234 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:47.770900  2324 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.771828 31820 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:47.773140 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:47.773973 31820 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:47.775602 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.776512 31820 raft_consensus.cc:2376] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8. Re-sending same reply.
I20260709 12:21:47.773676  2273 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:47.779902 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1441) took 1625 ms. Trace:
I20260709 12:21:47.780330 31746 rpcz_store.cc:276] 0709 12:21:46.154187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.154327 (+   140us) service_pool.cc:224] Handling call
0709 12:21:47.779686 (+1625359us) spinlock_profiling.cc:243] Waited 1.23 s on lock 0x7b54004033d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:47.779838 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2829108480}
I20260709 12:21:47.781689  1951 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:47.764117  2269 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.969s	user 0.014s	sys 0.000s
I20260709 12:21:47.782343  2269 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:47.785076 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.785697 31825 raft_consensus.cc:2376] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7. Re-sending same reply.
I20260709 12:21:47.785825  2197 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.786598 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.786446  2197 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.788415  2333 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.788061 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:47.791280 31820 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:47.792994  1931 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.793292 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:47.793586  1931 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.793022 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:47.795089 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:47.795356 31747 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:47.795739 31752 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:47.795742  2273 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:47.796634 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:47.764907  2296 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.447s	user 0.002s	sys 0.012s
I20260709 12:21:47.798857  2296 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:47.795957 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.799717 31821 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:47.800192 31820 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:47.797233  2273 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.803776  2234 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:47.804261  2234 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:47.804682  2234 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:47.805514 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 72554.9us
I20260709 12:21:47.806103  2267 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:47.806345 31792 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:47.808843  2253 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:47.809469  2253 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.810539  2347 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:47.811146  2355 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:47.811745  2355 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.813308  2219 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:47.813874  2219 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.816304  2049 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:21:47.816577  2219 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.817086  2219 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:47.817526  2219 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.712955  2348 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:47.818519 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.819324 31818 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:47.819011  2348 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.820190 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:47.803205  2333 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:47.820768 31818 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:47.821269  2348 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.821650 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.732741  2208 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.822261 31818 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:47.823943 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:47.824750 31750 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
W20260709 12:21:47.825337 31750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1448) took 1652 ms (client timeout 2040 ms). Trace:
W20260709 12:21:47.825726 31750 rpcz_store.cc:269] 0709 12:21:46.173052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.823683 (+1650631us) service_pool.cc:224] Handling call
0709 12:21:47.825274 (+  1591us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:47.826478  1931 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.827293  1931 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:47.827625 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:47.827791  1931 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:47.776528  2274 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:47.804461 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:47.830011 31791 leader_election.cc:336] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.524s (SENT)
I20260709 12:21:47.830569 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:47.831291  2267 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:47.739121  1897 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:47.832099  2267 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:47.832297  1897 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:47.833528  1897 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.835845  1897 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:47.835945 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:47.838207  2156 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:47.838308 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:47.838860  2156 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:47.839295  2271 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.840214  2062 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:47.841171  2062 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:47.841387 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:47.840005  1977 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:47.839314  2156 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:47.844192 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:47.845521 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:47.846989 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.842566  2062 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:47.849259 31826 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:47.734962 31677 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:21:47.854031 31752 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:47.787428 31825 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:47.854790 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:47.852720 32372 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
W20260709 12:21:47.845916  2320 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.143s	user 0.002s	sys 0.000s
I20260709 12:21:47.863011 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:47.837821  1931 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:47.974020 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.033218 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:48.034152 31745 raft_consensus.cc:2410] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 12:21:48.034710 32372 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
W20260709 12:21:48.014158 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 151067us
I20260709 12:21:48.035455 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:48.016355  2325 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.002575 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:47.982481 31751 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:21:48.038060  2325 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:48.039377 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:48.019904 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:47.943570 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 225745us
I20260709 12:21:48.040863 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.040995 31669 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:48.042104 31751 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:48.043503 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.044468 31642 leader_election.cc:400] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8.
I20260709 12:21:48.044922 31642 leader_election.cc:403] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Cancelling election due to peer responding with higher term
I20260709 12:21:48.045084 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:47.945293 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:48.046240 31751 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:48.046806 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:48.022399  2190 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:47.987340 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:47.960197 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:48.024442  2064 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.206s	user 0.005s	sys 0.001s
W20260709 12:21:47.966923 31717 leader_election.cc:336] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.717s (SENT)
W20260709 12:21:48.051131  2064 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.243s	user 0.006s	sys 0.001s
I20260709 12:21:48.051342 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:47.971916 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:48.009160  2208 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:48.016791  2131 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:48.030818 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:47.985591 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.007368  1951 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:48.032349 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 341) took 1832 ms. Trace:
I20260709 12:21:47.985193 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:47.965404 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:48.033461 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:48.045679  2325 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:47.961205 31671 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:47.980427 31748 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:21:47.980337 31749 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:48.036476 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:48.037242 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:48.015872  2273 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:47.908464  2347 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:47.967789 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:48.018285  2018 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:48.106047  2018 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:48.095278 31752 raft_consensus.cc:2376] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7. Re-sending same reply.
W20260709 12:21:48.095978 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 51850us
W20260709 12:21:48.097931 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62436.9us
I20260709 12:21:48.099210  2295 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:48.079945 31823 rpcz_store.cc:276] 0709 12:21:46.199362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.199539 (+   177us) service_pool.cc:224] Handling call
0709 12:21:46.199894 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:46.200323 (+   429us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:46.200358 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:46.200835 (+   477us) log.cc:844] Serialized 26 byte log entry
0709 12:21:46.324111 (+123276us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:46.324182 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:46.324306 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:47.845277 (+1520971us) spinlock_profiling.cc:243] Waited 8.7 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:48.031637 (+186360us) raft_consensus.cc:1685] finished
0709 12:21:48.031695 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:48.032101 (+   406us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":103,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":20929,"spinlock_wait_cycles":20015360,"thread_start_us":99576,"threads_started":1,"wal-append.queue_time_us":248194}
I20260709 12:21:47.975943 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:48.104303  2219 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:47.869699  2261 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.236s
W20260709 12:21:48.106835 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 57896.3us
I20260709 12:21:48.106539  2018 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:48.106835  1951 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:48.001067 31820 raft_consensus.cc:2376] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9. Re-sending same reply.
I20260709 12:21:47.963760 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:48.098510 31747 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:48.038079 31825 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:48.119238 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:48.120828 31824 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:48.120620  2064 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 7. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8.
I20260709 12:21:48.121819 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:47.981071 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:48.054412  2131 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:48.134012  1945 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:48.134807 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.136466 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.136879  2253 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:48.137262 31748 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
W20260709 12:21:48.137694 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 84224.4us
I20260709 12:21:48.138468  2273 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:48.004438 31791 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.688s (SENT)
I20260709 12:21:48.145239 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:48.147233 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.149798 31671 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260709 12:21:48.159241 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 155257us
I20260709 12:21:48.161329  2260 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:48.161070  2234 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:48.162045  2260 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:48.164819 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:48.167476  1931 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:48.169823 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:48.169934 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:48.163594  2260 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:48.171370 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 135283us
I20260709 12:21:48.171780 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:48.172003 31642 leader_election.cc:400] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7.
I20260709 12:21:48.173691 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:48.179665 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.159202 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:48.176286 31678 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260709 12:21:48.176760 31669 raft_consensus.cc:2410] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:21:48.176921 31823 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:48.180639 31678 pending_rounds.cc:85] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 5
I20260709 12:21:48.174614 32377 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:48.175302 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:48.186656 31820 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260709 12:21:48.187404 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:48.188691 31820 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260709 12:21:48.166400  2234 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:48.196493 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:48.191651 31820 pending_rounds.cc:85] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 3
I20260709 12:21:48.202364 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.179001  2295 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:48.174599  2066 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260709 12:21:48.164803 31643 leader_election.cc:336] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.894s (SENT)
I20260709 12:21:48.208544 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:48.203228  2325 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:48.199600 31678 pending_rounds.cc:107] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 6.6
I20260709 12:21:48.212920 31678 raft_consensus.cc:2889] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:48.207113 31820 pending_rounds.cc:107] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260709 12:21:48.213480 31820 raft_consensus.cc:2889] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:48.174594  1908 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:21:48.215368 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.211992  2325 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:48.225922  2325 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:48.215456 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 248930us
I20260709 12:21:48.231178 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:48.232141 31819 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:48.219164 32377 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:21:48.199348  2273 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:48.256177 31718 leader_election.cc:400] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6.
I20260709 12:21:48.223150 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:48.237461 31827 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:21:48.252113 31745 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:48.254339 31751 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:48.247697 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1837) took 1032 ms. Trace:
I20260709 12:21:48.245064 31744 raft_consensus.cc:2410] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:21:48.261044 31718 leader_election.cc:403] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260709 12:21:48.243832 31677 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:48.251251 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:48.245954 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:48.269374 31824 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:21:48.257256 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:48.252672 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:48.210229  2295 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.271945 31673 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:48.237914  2131 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:48.290407 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:48.291962 31674 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:48.293021 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:48.294245 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:48.287794 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:48.288486 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:48.289446 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:48.295604 31669 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
W20260709 12:21:48.303314 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50702.5us
I20260709 12:21:48.303849 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:48.304495  2347 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:48.305752 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:48.307406 31823 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:48.310134 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:48.312367 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.313048 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:48.313203  1951 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:48.313997 31747 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:21:48.246943  2325 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:48.256492  1897 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.063s
I20260709 12:21:48.286597  2219 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6.
I20260709 12:21:48.263863 31822 rpcz_store.cc:276] 0709 12:21:47.215243 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.219822 (+  4579us) service_pool.cc:224] Handling call
0709 12:21:47.387175 (+167353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:47.387422 (+   247us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:47.412526 (+ 25104us) log.cc:844] Serialized 14 byte log entry
0709 12:21:48.043234 (+630708us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b18000d50a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:21:48.211883 (+168649us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:48.247013 (+ 35130us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:48.247137 (+   124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.247252 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:48.247467 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":24942,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":4543,"raft.queue_time_us":18522,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":36694,"spinlock_wait_cycles":4018944,"thread_start_us":108476,"threads_started":1,"wal-append.queue_time_us":811349}
I20260709 12:21:48.322101 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:48.278071  2295 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:48.222338  2018 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:48.340826 31825 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:48.341769 31670 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:48.346987 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:48.350479 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 65617us
I20260709 12:21:48.373320 32377 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:48.372303 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:48.377013 31671 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7. Ops: [6.5-6.5]
W20260709 12:21:48.378132 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 213371us
I20260709 12:21:48.380967  2064 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:48.382069  2064 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:48.385183 31753 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:48.385043 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:48.374850 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 69257.2us
I20260709 12:21:48.390198 31669 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:21:48.410216 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:47.447726 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:48.391654  2219 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:48.423228 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:48.430059 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 85687.5us
I20260709 12:21:48.414567  2234 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.428618 31821 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:48.439453 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:48.441977 31818 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.571s	user 0.011s	sys 0.004s
I20260709 12:21:48.421006  2219 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.442183  2130 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:48.420420  1951 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.442939 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:48.442739  1977 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.442477 31818 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:48.455119 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:48.415534  2131 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.023s
I20260709 12:21:48.460466 31823 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 8 index: 7. (term mismatch)
I20260709 12:21:48.463599 31823 pending_rounds.cc:85] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 6
I20260709 12:21:48.468179 31823 pending_rounds.cc:107] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 7.7
I20260709 12:21:48.468425 31823 raft_consensus.cc:2889] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:48.465729 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58183.5us
I20260709 12:21:48.456342  2219 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:48.457089  2130 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:48.476228 31750 raft_consensus.cc:2410] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260709 12:21:48.477227  2156 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.613s	user 0.011s	sys 0.008s
I20260709 12:21:48.477700 31826 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260709 12:21:48.478475 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47436.5us
I20260709 12:21:48.478420  1977 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:48.489804  2018 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:48.491995  1977 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:48.496507 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:48.496989  2064 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:48.497499 31818 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:48.497824  2064 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:21:48.499662  2371 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:21:48.500615 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:48.501055 31672 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:21:48.501962  1977 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:48.502877  1977 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.481501  1951 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:48.503381  1977 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:48.505609  1951 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:48.506801 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:48.508000 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:48.508946 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:48.509915 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:48.511013  2108 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:48.511396  2108 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:48.511757  2108 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:48.515949 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:48.516526  2190 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.517133  2190 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:48.517593 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:48.519291  2066 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:48.519742  2066 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.520146  2066 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:48.520238  2330 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:48.520720  2330 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:48.523559 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 351) took 3713 ms. Trace:
I20260709 12:21:48.525068 31825 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:48.524006 31675 rpcz_store.cc:276] 0709 12:21:44.810529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:45.512939 (+702410us) service_pool.cc:224] Handling call
0709 12:21:45.526066 (+ 13127us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:45.716426 (+190360us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:45.743171 (+ 26745us) log.cc:844] Serialized 14 byte log entry
0709 12:21:46.143754 (+400583us) spinlock_profiling.cc:243] Waited 8.9 ms on lock 0x7b1800407680. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:21:47.256262 (+1112508us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b1800407680. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:21:47.504472 (+248210us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:47.504564 (+    92us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:47.504633 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.523119 (+1018486us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54000474d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:48.523178 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:48.523386 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41035,"prepare.queue_time_us":26596,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":63905,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":9644,"spinlock_wait_cycles":338008192,"thread_start_us":233674,"threads_started":1,"wal-append.queue_time_us":1842854}
I20260709 12:21:48.530768  2333 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:48.531512  2333 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:48.533766  2333 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:48.534322  2333 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:48.538574 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:48.543864 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:48.571830 31676 raft_consensus.cc:2410] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:21:48.549343 31825 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:21:48.545121  2325 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:48.459039  2234 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:48.536552  2208 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:48.535068  1931 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:21:48.562742 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45293us
I20260709 12:21:48.543700 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:48.571574 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:48.546252  1897 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:48.521929  2330 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:48.577255  1951 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:21:48.628434 31642 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:48.607000  2208 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.615595  2347 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:21:48.607982 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:48.597458 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:48.644117 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:48.613698  1897 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.606586  1931 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:48.607091  2333 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:48.619686 32372 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:48.602775  2156 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:48.659804 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 56491.8us
I20260709 12:21:48.639256  2208 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:48.664952  2370 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.191s	user 0.001s	sys 0.000s
I20260709 12:21:48.665561 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:48.666163 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:48.666837  2326 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:48.667909  2156 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:48.668676  2381 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:48.667923  2234 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:48.670717  2296 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:21:48.670805 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 132319us
I20260709 12:21:48.671264 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:48.654594  1897 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:48.656771  2330 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:48.690011 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:48.658396  1931 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:48.665442  2160 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:48.670581  2355 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:48.695101 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 184464us
I20260709 12:21:48.706008 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.634927  2190 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:48.700323 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:48.709313 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.709581 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.710090 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:48.710160 31753 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:48.683490  2381 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:48.720146 31747 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:48.719560 31746 raft_consensus.cc:2376] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7. Re-sending same reply.
I20260709 12:21:48.727298  2197 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:48.727365  2273 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:48.728127  1443 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:48.728487  2273 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:48.728986  1443 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:21:48.729633  2273 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:48.732318 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:48.733628 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:48.734160 31747 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:48.737046 31820 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260709 12:21:48.742640 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 51167.8us
I20260709 12:21:48.746189 31744 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:48.747130 31748 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
W20260709 12:21:48.749218 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 205640us
I20260709 12:21:48.717697  2130 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:48.756256  2377 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:21:48.756672 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.757443 31751 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:48.757766 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:48.758962 31746 raft_consensus.cc:2410] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 9 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:21:48.720094  2381 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:48.677156  2064 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:48.763695 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 351) took 1044 ms. Trace:
I20260709 12:21:48.765861 31825 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:48.773664 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 56278.6us
I20260709 12:21:48.741911 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2031) took 1012 ms. Trace:
I20260709 12:21:48.757445  2130 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:48.765661 31819 rpcz_store.cc:276] 0709 12:21:47.718878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.233090 (+514212us) service_pool.cc:224] Handling call
0709 12:21:48.233641 (+   551us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:48.234198 (+   557us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:48.234244 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:48.323983 (+ 89739us) log.cc:844] Serialized 52 byte log entry
0709 12:21:48.457846 (+133863us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:21:48.762447 (+304601us) spinlock_profiling.cc:243] Waited 7.7 ms on lock 0x7b180044b960. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf
0709 12:21:48.762612 (+   165us) log.cc:844] Serialized 14 byte log entry
0709 12:21:48.763077 (+   465us) log.cc:844] Serialized 14 byte log entry
0709 12:21:48.763308 (+   231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.763386 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:48.763496 (+   110us) raft_consensus.cc:1685] finished
0709 12:21:48.763509 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:48.763637 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":179218,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":137,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":133,"spinlock_wait_cycles":17719040,"thread_start_us":242458,"threads_started":2,"wal-append.queue_time_us":330499}
I20260709 12:21:48.808744 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:48.749572  2108 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:48.810089  2104 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:21:48.817975 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 152574us
I20260709 12:21:48.821777 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:48.823827 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:48.824695 31752 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:48.829066  2190 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:48.802536  2160 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260709 12:21:48.769074  2219 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:48.835822  2219 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:48.837568 31819 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:21:48.840111 31823 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:48.849129 31823 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:48.844112 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:48.789817 31827 rpcz_store.cc:276] 0709 12:21:47.729708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.286503 (+556795us) service_pool.cc:224] Handling call
0709 12:21:48.308189 (+ 21686us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:48.308610 (+   421us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:21:48.415400 (+106790us) log.cc:844] Serialized 14 byte log entry
0709 12:21:48.708052 (+292652us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:48.708105 (+    53us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:48.708159 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.708236 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:48.741639 (+ 33403us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":296670,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":24764,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":23469,"thread_start_us":351213,"threads_started":3,"wal-append.queue_time_us":279211}
I20260709 12:21:48.834306  2296 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:48.858273 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 182868us
I20260709 12:21:48.859153 31747 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:48.859058 32342 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:48.867717 31818 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
W20260709 12:21:48.856218  2377 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:21:48.870272 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:48.887702  2160 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:21:48.888096 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 97987.9us
I20260709 12:21:48.888963  2347 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:48.889271 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67615.3us
I20260709 12:21:48.889657  2347 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:48.890431 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:48.891821  2108 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:48.892194  2347 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:48.893869 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:48.894737 32371 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:48.894850 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:48.895499 31820 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:48.865047  2066 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:48.900769  2066 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:48.902081  2323 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.704s	user 0.000s	sys 0.008s
I20260709 12:21:48.889032 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:48.872231  2234 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:21:48.914183 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63878.9us
I20260709 12:21:48.866573  2296 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:48.911077 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2013) took 1286 ms. Trace:
I20260709 12:21:48.873572 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:48.883489  2104 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:48.912710  2234 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:48.922851  2384 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:21:48.923126  2393 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:21:48.925045  2392 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:21:48.927199  2355 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:48.927726  2219 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:48.931321 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 131424us
I20260709 12:21:48.932041  2277 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:48.979341  2381 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:48.981853  2381 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:48.916170 31826 rpcz_store.cc:276] 0709 12:21:47.624714 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.850224 (+225510us) service_pool.cc:224] Handling call
0709 12:21:48.477117 (+626893us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:48.602966 (+125849us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:48.902860 (+299894us) spinlock_profiling.cc:243] Waited 610 ms on lock 0x7b54001c8ed0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:48.910984 (+  8124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1402849536}
I20260709 12:21:48.984627 31820 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:21:48.939028 31825 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:48.855948  2333 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:48.943773 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58433.6us
I20260709 12:21:48.949154  2234 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:48.991623 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:48.993186 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:48.994168 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:48.933166  2108 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:48.995164  2156 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:48.995589  2156 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:48.995939  2156 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:48.996059  2262 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:48.996678  2262 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:48.997624  2262 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:48.972250  2355 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:48.972645  1977 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:48.910156  2066 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.055s
I20260709 12:21:48.973268  2219 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.056286  1977 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:48.976694  2295 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.603s	user 0.010s	sys 0.000s
I20260709 12:21:48.994031  2197 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:21:48.970028 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.001891  1931 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.024884  2326 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:49.031158  2190 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:49.016089  2253 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:49.034996  2160 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:49.037225  2268 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:49.002830  2018 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:49.037122  2130 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
W20260709 12:21:49.038892 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44807.1us
I20260709 12:21:49.040659  2398 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:21:49.041507 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 80188.1us
I20260709 12:21:49.042140  2184 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:48.968581  1945 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.034s
I20260709 12:21:49.042162  2331 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:49.042331  2277 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:49.061946 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 354) took 1210 ms. Trace:
I20260709 12:21:49.096553  2393 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:49.108965  2208 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.039847  2108 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:49.030503  1897 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.105455 31745 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260709 12:21:49.108768  2397 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:21:49.056859  2219 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:49.120594 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 231653us
I20260709 12:21:49.058029  1951 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:49.128909 31822 rpcz_store.cc:276] 0709 12:21:47.851210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:48.514324 (+663114us) service_pool.cc:224] Handling call
0709 12:21:48.514832 (+   508us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:48.631423 (+116591us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:48.631465 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:48.660259 (+ 28794us) log.cc:844] Serialized 26 byte log entry
0709 12:21:48.764295 (+104036us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:48.764334 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:49.052458 (+288124us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54003ca0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:49.052544 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:49.052631 (+    87us) raft_consensus.cc:1685] finished
0709 12:21:49.052664 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:49.061835 (+  9171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1013,"raft.run_cpu_time_us":252,"raft.run_wall_time_us":23513,"spinlock_wait_cycles":244481920,"thread_start_us":89211,"threads_started":1,"wal-append.queue_time_us":208269}
I20260709 12:21:49.030434  2234 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.146761 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:49.155079 31825 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:49.154500 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:49.156241 31824 raft_consensus.cc:2376] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8. Re-sending same reply.
I20260709 12:21:49.159046 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:49.158942 31823 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.131729  2393 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:49.160162 31823 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:49.160625  2398 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:49.160898 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:49.161516  1951 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.079720  2295 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.159471 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:49.059129 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 195984us
I20260709 12:21:49.163105 31821 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:49.131924  2184 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:21:49.153258 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.132947  2325 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:49.166762 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:49.159159 32371 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:49.168372 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:49.164609 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:49.177034  2392 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.177749 31826 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:49.182919 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:49.160923  2393 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.185365  2365 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:49.189915 31821 raft_consensus.cc:2410] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 7 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
W20260709 12:21:49.191598 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 165998us
I20260709 12:21:49.192937 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:49.199038 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 162442us
W20260709 12:21:49.208977 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 977787us
I20260709 12:21:49.171681  2325 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.182946  2400 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:49.214572 31821 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:49.173770  2104 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260709 12:21:49.189383  1931 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:49.216711 31826 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:49.217743 31825 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260709 12:21:49.182500  2392 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.229158 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:49.231359 31821 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:21:49.231195  2394 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.045s
I20260709 12:21:49.232307  2394 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:49.232702  2394 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:49.233180 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:49.236234  2295 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:49.236665  2234 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:49.236778 31718 leader_election.cc:400] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 6. Current term is 7.
I20260709 12:21:49.236937  2295 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:49.238122 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:49.223026  2108 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.239220  2399 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:49.226166  2328 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.015s
I20260709 12:21:49.242442 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:49.243517  2399 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:49.244179  2399 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:49.246471  2399 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.246917  2379 raft_consensus.cc:2955] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Committing config change with OpId 7.7: config changed from index 3 to 7, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:49.250455 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:49.251492  1897 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:49.252626  1443 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:49.253242  1443 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:49.255033 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:49.255085 31747 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260709 12:21:49.255966 31746 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:49.256426  2156 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.257094 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:49.257874 31753 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 5: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:49.258193  2392 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:49.258782 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:49.230089  2184 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.259567 31750 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:49.260519 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:49.262359 31752 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:49.229926  2277 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.233768  2394 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.262575 31823 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:49.254205  1443 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.279443 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:49.281117 31753 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:49.276976  2156 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:49.282409 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:49.283241 31753 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:49.284008 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:49.284454 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:21:49.285189  2382 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.491s	user 0.004s	sys 0.000s
I20260709 12:21:49.285517  2404 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:49.286057  2404 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:49.286432  2404 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:49.286511 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:49.287544  2355 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:49.290441 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:49.292368 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:49.285431 31753 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:49.274863  2394 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:49.280115 31746 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:49.300930 31753 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:49.302386 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:49.303176 31819 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:49.305148 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 140624us
I20260709 12:21:49.286827  2197 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:49.314265 31823 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:49.319952 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:49.321591 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:49.322791  2327 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.322973 31752 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260709 12:21:49.323364  2327 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.325954  2333 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:49.327468  2325 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:49.333006  2296 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.334986 31827 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:21:49.290086  1977 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
W20260709 12:21:49.373046 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 64455.7us
I20260709 12:21:49.373795 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:49.376752 31746 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
W20260709 12:21:49.380947 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 169717us
I20260709 12:21:49.385326 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:49.386127 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:49.386945 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:49.387599 31751 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:49.325942  2327 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:49.403978 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:49.388859 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:49.406500 31825 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:49.317458  2066 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.411299 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:49.415084  2407 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:21:49.415717 31825 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260709 12:21:49.416425  2253 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:49.417073 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:49.416971  2253 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.417833 31825 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:49.349483  2197 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:49.403615  2355 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:49.415706 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:49.373400  1951 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.424160 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:49.427207 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:49.429965 31818 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:49.431442 31750 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:49.411389  2392 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 12:21:49.434535 31819 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:21:49.438762 31823 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:21:49.440587 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:49.441229 32378 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:49.443516 31819 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: [5.3-8.4]
I20260709 12:21:49.444986 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:49.422559  1897 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.354900  2333 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.464811 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:49.422622  2066 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.467921 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:49.469045 31717 connection.cc:256] RPC call timeout handler was delayed by -1.2331796960002066s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22280000000000003s for a response
I20260709 12:21:49.470698 32369 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:49.485496 31818 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:49.471038  2413 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:49.492317 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.418458  2104 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.036s
I20260709 12:21:49.472674  2253 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:49.433101  2355 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:49.473680  2406 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:21:49.474090 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:49.476781 32378 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:49.487207 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:49.471407  2404 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.535537 31819 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:49.536852 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:49.538213 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:49.537849  2064 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:49.541915 31819 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
W20260709 12:21:49.537894 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.439334  1951 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.470772  2392 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260709 12:21:49.560237 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 147997us
I20260709 12:21:49.501343  2296 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:49.565372 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:49.573843 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:49.574579 31820 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:49.582152 31818 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:21:49.584111 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:49.585239 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:49.585309 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:49.483934  1897 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.588644 32370 raft_consensus.cc:2368] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:21:49.533885  2104 consensus_queue.cc:1243] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 5.2 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 1
I20260709 12:21:49.590613 31818 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:49.559944  2325 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:49.588619 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:49.597311 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 170937us
W20260709 12:21:49.601898 31717 leader_election.cc:336] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.989s (SENT)
W20260709 12:21:49.603674 31572 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42072 (request call id 239) took 23410 ms (client timeout 15000 ms). Trace:
I20260709 12:21:49.607033  2408 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:49.607558  2408 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:49.607925  2408 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:49.576630  2398 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:49.603817  2396 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:21:49.622866  2414 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
W20260709 12:21:49.624274 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.619256 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:49.502133  2197 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:49.629483 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 93658us
I20260709 12:21:49.628718 31823 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:49.629418 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:49.631388 31670 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:49.632149 31818 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:21:49.615873  2396 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:49.612151  2328 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:49.639649 31820 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:49.641305 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:49.641949 31677 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 12:21:49.659600 31677 pending_rounds.cc:85] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 7
I20260709 12:21:49.659911 31677 pending_rounds.cc:107] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 12:21:49.660002 31669 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:21:49.660169 31677 raft_consensus.cc:2889] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:49.642045 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:49.641644  2396 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.642655 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:49.661451 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:49.643347 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:49.662184 31670 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:49.644563 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:49.646339  2404 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:49.647055 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51981.7us
W20260709 12:21:49.604173 31572 rpcz_store.cc:269] 0709 12:21:26.193822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:26.194068 (+   246us) service_pool.cc:224] Handling call
0709 12:21:27.287928 (+1093860us) spinlock_profiling.cc:243] Waited 55.1 ms on lock 0x7b54002828d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3f090 00007fe8fcdddf3a 00007fe902b93e9c 00007fe902c9204e 00007fe9011532a4 00007fe901153232 00007fe901153161 00007fe9011530dc 00007fe901152392 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:49.603411 (+22315483us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:39.612171 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:40.013737 (+401566us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:40.030218 (+ 16481us) write_op.cc:435] Acquired schema lock
0709 12:21:40.059218 (+ 29000us) tablet.cc:662] Decoding operations
0709 12:21:40.195576 (+136358us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:40.195711 (+   135us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:40.195754 (+    43us) tablet.cc:685] Acquiring locks for 19 operations
0709 12:21:40.197958 (+  2204us) tablet.cc:701] Row locks acquired
0709 12:21:40.198019 (+    61us) write_op.cc:260] PREPARE: finished
0709 12:21:40.198276 (+   257us) write_op.cc:270] Start()
0709 12:21:40.198484 (+   208us) write_op.cc:276] Timestamp: P: 1783599700198182 usec, L: 0
0709 12:21:40.198518 (+    34us) op_driver.cc:348] REPLICATION: starting
0709 12:21:40.238740 (+ 40222us) log.cc:844] Serialized 7563 byte log entry
0709 12:21:41.863628 (+1624888us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b54002828d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd0e28 00007fe90080b581 00007fe90080ad58 00007fe90080f9f1 00007fe90080f9a9 00007fe90080f939 00007fe90080f901 00007fe90080eb2d 0000000000388634 0000000000388479
0709 12:21:42.011231 (+147603us) op_driver.cc:464] REPLICATION: finished
0709 12:21:42.307824 (+296593us) write_op.cc:301] APPLY: starting
0709 12:21:42.307963 (+   139us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:42.335454 (+ 27491us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:42.335504 (+    50us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:42.338156 (+  2652us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:42.361792 (+ 23636us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=19
0709 12:21:42.381645 (+ 19853us) write_op.cc:312] APPLY: finished
0709 12:21:42.416856 (+ 35211us) log.cc:844] Serialized 132 byte log entry
0709 12:21:42.417149 (+   293us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:42.417418 (+   269us) write_op.cc:454] Released schema lock
0709 12:21:42.417531 (+   113us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:42.418136 (+   605us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:42.418183 (+    47us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":383407,"reported_tablets":43,"spinlock_wait_cycles":126839168,"child_traces":[["op",{"apply.queue_time_us":235779,"apply.run_cpu_time_us":6581,"apply.run_wall_time_us":132705,"num_ops":19,"prepare.queue_time_us":303871,"prepare.run_cpu_time_us":57310,"prepare.run_wall_time_us":2332307,"raft.queue_time_us":538697,"raft.run_cpu_time_us":9519,"raft.run_wall_time_us":102681,"replication_time_us":1812521,"spinlock_wait_cycles":31156608,"thread_start_us":607735,"threads_started":4,"wal-append.queue_time_us":645132,"wal-append.run_cpu_time_us":100353,"wal-append.run_wall_time_us":2535829}]]}
I20260709 12:21:49.654299  2328 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:49.648583 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:49.673007 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:49.649703 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:49.675657  2381 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:49.649946  2418 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:21:49.651813  1897 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:49.559580  2413 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.667161 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:49.671599  2328 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:49.674371 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:49.692283 31672 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:49.706883 32375 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:49.707566 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:49.722939 31676 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:49.717353 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:49.662231  2325 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:49.727821  2425 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.735098 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:49.728191 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53923.2us
I20260709 12:21:49.728979 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:49.732026 31673 pending_rounds.cc:85] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 7
I20260709 12:21:49.737722 31673 pending_rounds.cc:107] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 8.8
W20260709 12:21:49.737376 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 63440.4us
I20260709 12:21:49.700448  2398 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:49.741914 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 112454us
I20260709 12:21:49.742501  2426 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:49.743160  2426 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:49.743599  2426 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:49.754092 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:49.754716 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 193705us
I20260709 12:21:49.709873  2277 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:49.754258 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:49.764312 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:49.738042 31673 raft_consensus.cc:2889] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:49.753438  2426 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:49.783947 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 171953us
W20260709 12:21:49.803458 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48115.7us
I20260709 12:21:49.733945  2425 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.828027 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:49.831044 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:49.832276  2066 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:49.716383  2219 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 1, Time since last communication: 0.056s
I20260709 12:21:49.713780  2253 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 2, Time since last communication: 0.058s
I20260709 12:21:49.832921  2066 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:21:49.833772  2253 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:49.834252  2253 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:49.835186  2253 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.836620  2253 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:49.714066  2399 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260709 12:21:49.755671  2325 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:49.848032 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:49.729803  1897 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.102s
I20260709 12:21:49.881542 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:49.882859 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 55182.5us
I20260709 12:21:49.883915 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:49.888415 31672 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:49.764400  2398 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:49.889735 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62325.8us
I20260709 12:21:49.892334 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:49.895030  2234 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:49.895571  2234 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:49.895992  2234 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:49.896021  2424 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:21:49.896353  2425 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:49.899943  2426 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:49.828825  2389 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:49.903023  2389 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.906512 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:49.907797 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:49.849855  2333 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:49.879096  2296 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:49.929528  2296 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:49.930069  2296 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:49.931304  2331 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:49.931748  2331 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:21:49.932802  2379 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.684s	user 0.012s	sys 0.000s
I20260709 12:21:49.932665  2331 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:49.939858 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:49.940893 32378 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:49.942503 31676 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:21:49.910699 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:49.911320  2430 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
I20260709 12:21:49.911794 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:49.912933  2417 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
W20260709 12:21:49.913578  2412 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.319s	user 0.001s	sys 0.000s
I20260709 12:21:49.913502  2234 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.919204  2443 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:49.924737  2398 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:49.906165  1951 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:49.927079 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:49.991981 32375 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:49.927991 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:49.928109  2413 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:49.928843 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:49.950889 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:49.950187  2331 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:49.929952  2408 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.992347 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:49.951341  2438 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
W20260709 12:21:49.975754 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 65155.5us
I20260709 12:21:49.911375  2427 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.177s
I20260709 12:21:50.020067 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:50.021063  2408 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:50.000403 32377 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 6: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:50.000948 31672 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:21:50.001516 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 50713us
I20260709 12:21:50.027580  2440 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:49.769708  2327 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.111s
I20260709 12:21:50.028296  2440 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.028754  2327 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:50.029178  2327 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:50.029799  2389 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:50.030107  2327 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:50.007177 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55641.1us
I20260709 12:21:50.033572  2427 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:50.034283  2427 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:21:50.006577  2394 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.704s	user 0.007s	sys 0.006s
W20260709 12:21:50.004614 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.034924  2066 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:21:50.036213  2425 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:50.037878 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:50.038679 31677 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:50.035324  2427 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.039451 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.040918  2425 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:50.041682  2434 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:50.042135  2434 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:50.043537  2410 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.443s	user 0.003s	sys 0.000s
I20260709 12:21:50.043040  2434 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:50.043427 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.044878  2434 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:50.045478 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.046343  2330 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:50.046896  2398 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:50.047150  2330 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:50.048826 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.050832  2425 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:50.052222  2234 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:49.934795  2399 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.118s
I20260709 12:21:50.054437  2325 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.055258 31746 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 9. Current term is 10. Ops: []
I20260709 12:21:50.056695 32375 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:50.057157 31746 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:49.997720  2333 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:50.058655  2333 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:50.059755  2333 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:49.952414  2273 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.110s
I20260709 12:21:50.062428 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.063340 31746 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:50.010824  1951 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:50.066557  1951 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:50.067235 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.067574  1951 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:50.070935 31746 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
W20260709 12:21:50.071054 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.071758 31746 pending_rounds.cc:85] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 5
I20260709 12:21:50.072098 31746 pending_rounds.cc:107] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 3.6
I20260709 12:21:50.072410 31746 raft_consensus.cc:2889] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:50.072788 31746 pending_rounds.cc:107] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260709 12:21:50.073081 31746 raft_consensus.cc:2889] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:50.079105 31746 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:21:50.079882 31751 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:50.079962 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:50.080839 31749 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:50.079174 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.083112 31745 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:50.084985 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:50.085661 31818 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:50.079129 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:50.093384 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:50.093542 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:50.093739 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:50.095016 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:50.095307  2273 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:50.095709  2273 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:50.095860  1897 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:50.096122  2273 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:50.096410  1897 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:50.096424 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:50.114459 31818 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
W20260709 12:21:50.116642 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 311317us
I20260709 12:21:49.935359  2426 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.100s
I20260709 12:21:50.121414 31671 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:50.126960 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:50.126799 31671 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:50.046176  2328 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:50.127872 31827 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 5 index: 2. (index mismatch)
W20260709 12:21:50.190711 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.128013 31824 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 12:21:50.208755 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 477591us
W20260709 12:21:50.210987 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 87648.6us
I20260709 12:21:50.177814 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:50.216784 31818 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:50.215343 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:50.102885  2440 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:50.224715 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:50.224241 31671 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.225119  1443 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.008s
I20260709 12:21:50.231778 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:50.234978 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:50.239449 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 354382us
W20260709 12:21:50.239895 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.241567  1443 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:50.242086  1443 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:50.245682 31752 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:50.248612 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:50.249207 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.246294 31752 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:50.252596 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.201375 31571 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 5 to 6, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 6 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:50.256323 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.258548 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.259410 32375 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:50.258072 31752 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:50.268163 31752 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:50.271400 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:50.272842 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:50.274824 31671 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:50.277182 31670 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:50.270434  2398 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:50.279117 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:50.280102 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:50.281098  2389 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:50.281530  2389 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:50.281941  2389 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:50.284699  2409 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:50.285386  2409 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:50.232373  2447 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.064s
I20260709 12:21:50.249408  2253 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:50.289227 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:50.291033 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:50.291738  2413 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:50.292769  2452 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.293196 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.292634 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:50.293502  2452 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.294112 31669 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260709 12:21:50.295001 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:50.295797 31747 raft_consensus.cc:2368] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:21:50.296787 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:50.294467 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.300277 31751 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:50.300485  2234 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:50.301023  2234 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:50.301427  2234 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:50.301493 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:50.302309 31751 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:50.291170 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:50.315119 31746 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:50.315430 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 89730.8us
I20260709 12:21:50.317695 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:50.293859 31748 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:50.318725  2447 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:21:50.304536 32377 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:50.319419  2444 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:50.320013  2444 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:50.305182 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:50.320418  2444 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:50.298271 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.321465  2328 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:50.322249 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63787.4us
I20260709 12:21:50.322942 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.306479  2018 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:50.331336  2425 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:50.305943 31571 catalog_manager.cc:5697] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 3 to 5, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 5 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:50.331774  2425 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:50.332002  2446 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:50.332252  2425 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:50.332428  2446 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:50.333389  2446 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:50.319569  2447 consensus_queue.cc:1243] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 10.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:21:50.333400 31571 catalog_manager.cc:5697] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 2 to 5, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 5 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:21:50.289841  2296 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.193897  1897 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:50.319576 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.337716  2296 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:50.338122 31748 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:50.339356 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.323649 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.305392 32369 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: [6.7-7.8]
I20260709 12:21:50.344738 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.265358  2325 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.059s
I20260709 12:21:50.326016 31670 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260709 12:21:50.267251  2440 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:50.328619  2452 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:50.329205  2379 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:43180 (request call id 89) took 3502 ms. Trace:
W20260709 12:21:50.330736 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.292690 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:50.363077 31745 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:50.305672 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:50.363950 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58372.8us
I20260709 12:21:50.361141 31748 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:50.366580 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:50.367610 31748 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:50.360916  2325 consensus_queue.cc:1243] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:21:50.362403  2379 rpcz_store.cc:276] 0709 12:21:46.826529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:46.826719 (+   190us) service_pool.cc:224] Handling call
0709 12:21:47.105749 (+279030us) log.cc:844] Serialized 508 byte log entry
0709 12:21:47.744274 (+638525us) spinlock_profiling.cc:243] Waited 9.72 ms on lock 0x7b5400327998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd917bc 00007fe8fcd942c0 00007fe8fcd94248 00007fe8fcd941c8 00007fe8fcd94183 00007fe8fcd9344c 00007fe902c402f6 00007fe902c3aaa8 00007fe8fc97cb41 00007fe8fc96ff2a
0709 12:21:48.623612 (+879338us) spinlock_profiling.cc:243] Waited 14.3 ms on lock 0x7b5400327920. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4e142 00007fe8fcd2eeaa 00007fe8fcd35d53 00007fe8fcd35cd9 00007fe8fcd35c69 00007fe8fcd35c31 00007fe8fcd34dfd 0000000000388634 0000000000388479 00007fe8f9ce8475
0709 12:21:50.329083 (+1705471us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":19304,"mutex_wait_us":2048,"raft.queue_time_us":86703,"raft.run_cpu_time_us":4766,"raft.run_wall_time_us":1119435,"spinlock_wait_cycles":55316608,"thread_start_us":495594,"threads_started":2,"wal-append.queue_time_us":315300,"wal-append.run_cpu_time_us":7222,"wal-append.run_wall_time_us":1405367}
I20260709 12:21:50.286339  2409 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.371447  2328 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:50.371546  2066 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:50.372146  2328 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.373039  2389 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.374740  2408 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:50.375281  2330 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:50.337750 31571 catalog_manager.cc:5697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 4 to 5, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 5 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:50.355651 31749 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:50.356146 31746 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:50.358933  2446 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:50.372548  2066 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:50.273562  1443 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.348582 31747 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
W20260709 12:21:50.300267  2381 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.624s	user 0.005s	sys 0.008s
I20260709 12:21:50.360623  2296 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:21:50.391127 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 118342us
I20260709 12:21:50.391628 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.393326 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:50.401502 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:50.388276 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 125626us
I20260709 12:21:50.415026 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.406379  2108 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.375850  2219 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:50.405798  2381 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:50.416658 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.346308  2262 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.124s
I20260709 12:21:50.374483  2328 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:50.421948 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.388444  2160 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:50.387053  2330 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:50.399749 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.429679  1443 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:50.430364 31751 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:50.402179  2066 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:50.442253 31746 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:50.443616 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.404222  2325 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:50.427202  1931 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:50.404682  2197 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 12:21:50.390337  2434 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.044s
I20260709 12:21:50.430275  2253 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:50.445339 31571 catalog_manager.cc:5697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 4 to 5, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 5 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:21:50.453161 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:50.461475 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 172430us
I20260709 12:21:50.462599 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:50.465871 31751 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:50.470302 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1859) took 1034 ms. Trace:
I20260709 12:21:50.443327  2330 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:50.486259 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.488570 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:50.489523 31746 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:50.470738 31819 rpcz_store.cc:276] 0709 12:21:49.435746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.558270 (+122524us) service_pool.cc:224] Handling call
0709 12:21:49.573515 (+ 15245us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:49.638264 (+ 64749us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:49.710482 (+ 72218us) log.cc:844] Serialized 14 byte log entry
0709 12:21:49.873332 (+162850us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:49.878981 (+  5649us) log.cc:844] Serialized 26 byte log entry
0709 12:21:49.879134 (+   153us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:49.879166 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.462298 (+583132us) spinlock_profiling.cc:243] Waited 48.7 ms on lock 0x7b54003c96d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:50.462347 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:50.462402 (+    55us) raft_consensus.cc:1685] finished
0709 12:21:50.470104 (+  7702us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:50.470244 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20083,"prepare.queue_time_us":78359,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":33715,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":14060,"spinlock_wait_cycles":111957248,"thread_start_us":89200,"threads_started":1,"wal-append.queue_time_us":342011}
I20260709 12:21:50.494354 31821 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:50.434305  2108 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:50.464675  2452 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.464569  2296 raft_consensus.cc:2764] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:21:50.448496  2389 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:50.468849  2219 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:50.466851  2398 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.441874  2190 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.051s
W20260709 12:21:50.515179 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 52727.7us
I20260709 12:21:50.485497  2064 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:50.467399  2399 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:50.534056 31827 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260709 12:21:50.537784 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 115967us
I20260709 12:21:50.514580  1951 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:50.460970  2325 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:50.587468 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.532660  2434 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
W20260709 12:21:50.592492 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55469.6us
I20260709 12:21:50.594944 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:50.601008 31750 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:21:50.585920 31571 catalog_manager.cc:5697] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 3 to 4, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 4 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:21:50.554327  2389 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:50.528688  2452 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.634100 31748 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:50.635533 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.636600 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:50.637446 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 220853us
I20260709 12:21:50.637732 31748 raft_consensus.cc:2410] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:21:50.636961 31791 leader_election.cc:400] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:21:50.638293 31791 leader_election.cc:403] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
W20260709 12:21:50.639708 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:50.634845 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 205024us
I20260709 12:21:50.598471  2434 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:50.512676  2066 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:50.598224  2464 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:50.643554 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:50.643143 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.538836  2398 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.646507 31571 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 4 to 5, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 5 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:50.539340  2273 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:50.627758  2389 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:50.651906 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.650835 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:50.598345  1931 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.047s
I20260709 12:21:50.597496  2064 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.635085  2445 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.642498  1951 raft_consensus.cc:2764] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:21:50.642840  2160 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:50.646268  1897 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:21:50.650511  2296 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:50.654860  2389 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:50.643719  2197 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.022s
I20260709 12:21:50.664710  2066 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:50.641510  2437 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.230s
I20260709 12:21:50.701267 31745 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
W20260709 12:21:50.685451 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48639.5us
W20260709 12:21:50.689492  2466 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:21:50.690521  2398 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:50.692067  2325 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:50.692677  2457 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
W20260709 12:21:50.683339  2463 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:21:50.688097  1897 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:50.655423 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet f6404cb9c2174326b06cbbb0d27d43f2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 12)
I20260709 12:21:50.650275  2469 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.040s
I20260709 12:21:50.710186 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:50.742761 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:50.744427 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:50.666863  2445 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:50.759814 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 159474us
I20260709 12:21:50.762619 31671 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260709 12:21:50.763346 31671 pending_rounds.cc:85] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 3
I20260709 12:21:50.763645 31671 pending_rounds.cc:107] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260709 12:21:50.763958 31671 raft_consensus.cc:2889] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:50.761799 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:50.778071 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:21:50.780026 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49068.4us
I20260709 12:21:50.734561  2325 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:21:50.685690  2296 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:50.683682  2234 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:50.733371  2398 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:50.783306 31746 raft_consensus.cc:2955] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Committing config change with OpId 7.7: config changed from index 3 to 7, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:50.788545 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.737063  1897 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:50.726060  2472 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
W20260709 12:21:50.793762  2462 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.277s	user 0.000s	sys 0.001s
I20260709 12:21:50.685171  2434 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.794512  2471 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260709 12:21:50.790879 31561 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 285958us
I20260709 12:21:50.754451  2108 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:50.788430  2457 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:21:50.786984  2469 consensus_queue.cc:1243] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 5.3 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 2
I20260709 12:21:50.789042  2464 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:50.820405 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 355419us
W20260709 12:21:50.824101 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46154.6us
I20260709 12:21:50.793784  2064 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:50.850565 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63487.6us
W20260709 12:21:50.870996 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:50.877068  2452 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:50.878857 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:21:50.849823 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61344.2us
I20260709 12:21:50.871606 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:50.822953  1931 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:50.860551  2325 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:50.899744 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:50.893285 31748 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: [5.6-8.7]
I20260709 12:21:50.909140 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:50.870594  2464 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:50.851332  2471 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:50.948357 31792 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:50.795181  2160 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:50.947765 31669 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.504s	user 0.014s	sys 0.032s
I20260709 12:21:50.905709  2452 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:50.904650  2325 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:50.972592 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.930135  2398 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:50.985821 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:50.954792  2208 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:50.977618  2453 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.291s	user 0.000s	sys 0.001s
I20260709 12:21:50.955031  2474 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:21:50.956573  2434 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:50.957293  1977 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:50.970135  2464 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:51.020792 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 143922us
I20260709 12:21:50.951768  2108 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 12:21:50.978240  2471 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:50.987129  2325 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:50.987684  2398 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:51.010066  2456 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.260s	user 0.000s	sys 0.001s
W20260709 12:21:51.029294 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.012s	sys 0.016s
I20260709 12:21:50.944489  1897 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:50.991000  2480 raft_consensus.cc:2764] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260709 12:21:51.081120 32369 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:21:51.072628 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:51.063968 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.109093 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2039) took 1937 ms. Trace:
I20260709 12:21:51.018666  2474 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:21:51.259191 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.018s	sys 0.028s
I20260709 12:21:51.123869 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 488) took 1029 ms. Trace:
I20260709 12:21:51.317737 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 384) took 1097 ms. Trace:
W20260709 12:21:51.346025 31818 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.850s	user 0.013s	sys 0.040s
I20260709 12:21:51.354682 32375 rpcz_store.cc:276] 0709 12:21:50.094384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.115365 (+ 20981us) service_pool.cc:224] Handling call
0709 12:21:50.115784 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.259806 (+144022us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.123561 (+863755us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b5400427f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:51.123767 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":329853696}
W20260709 12:21:51.357447 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 452408us
I20260709 12:21:51.016855  2469 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.284s
W20260709 12:21:51.024652  2018 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.009s	sys 0.013s
W20260709 12:21:51.122902 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 768547us
W20260709 12:21:51.118103 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.353998 31822 rpcz_store.cc:276] 0709 12:21:49.171270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.178256 (+  6986us) service_pool.cc:224] Handling call
0709 12:21:49.193310 (+ 15054us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:49.377856 (+184546us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:49.918042 (+540186us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b1800486f40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf
0709 12:21:49.918247 (+   205us) log.cc:844] Serialized 14 byte log entry
0709 12:21:50.267104 (+348857us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:50.286330 (+ 19226us) log.cc:844] Serialized 26 byte log entry
0709 12:21:50.286515 (+   185us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:50.286784 (+   269us) log.cc:844] Serialized 14 byte log entry
0709 12:21:50.286957 (+   173us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.287035 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.108750 (+821715us) raft_consensus.cc:1685] finished
0709 12:21:51.108791 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.108995 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65135,"prepare.queue_time_us":118176,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":6696,"raft.queue_time_us":57177,"raft.run_cpu_time_us":335,"raft.run_wall_time_us":8416,"spinlock_wait_cycles":24427904,"thread_start_us":584674,"threads_started":3,"wal-append.queue_time_us":482256}
W20260709 12:21:51.358857 31818 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.863s	user 0.013s	sys 0.041s
I20260709 12:21:51.082404 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:51.373811 32378 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:51.372437 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:51.082904  2461 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.316s	user 0.000s	sys 0.001s
W20260709 12:21:51.047253  2450 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.292s	user 0.001s	sys 0.000s
I20260709 12:21:51.354837 31671 rpcz_store.cc:276] 0709 12:21:50.220244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.807934 (+587690us) service_pool.cc:224] Handling call
0709 12:21:50.811009 (+  3075us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:50.811478 (+   469us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.811519 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:50.823710 (+ 12191us) log.cc:844] Serialized 26 byte log entry
0709 12:21:50.973420 (+149710us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:51.003486 (+ 30066us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.003792 (+   306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.003895 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.317440 (+313545us) raft_consensus.cc:1685] finished
0709 12:21:51.317471 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.317652 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":25804,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":1781,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":47416,"spinlock_wait_cycles":15602944,"thread_start_us":100276,"threads_started":1,"wal-append.queue_time_us":226524}
I20260709 12:21:51.015791  2483 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:21:51.074743  2459 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.339s	user 0.000s	sys 0.001s
W20260709 12:21:51.311326  2478 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.333s	user 0.001s	sys 0.000s
W20260709 12:21:51.126348  2444 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.805s	user 0.008s	sys 0.002s
W20260709 12:21:51.083565  2425 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.751s	user 0.013s	sys 0.000s
W20260709 12:21:51.119922  2479 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
W20260709 12:21:51.385793 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.385797  1443 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.377262  1897 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:51.072425  2453 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
W20260709 12:21:51.082353  2475 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.324s	user 0.001s	sys 0.001s
I20260709 12:21:51.259635  2398 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:51.084077  2491 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.077425  2208 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.109407  2234 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:51.081996  2445 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:51.376269  2018 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.920s	user 0.017s	sys 0.020s
I20260709 12:21:51.122741  2476 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:51.084213 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 475) took 1798 ms. Trace:
I20260709 12:21:51.312131  2477 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:51.410787 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:51.035084  2484 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:51.359308  2489 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.317271  2408 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.047075  1977 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:51.426517  1443 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:51.120669  2482 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
W20260709 12:21:51.430346 31745 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.025s	sys 0.037s
W20260709 12:21:51.471712 31745 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.025s	sys 0.037s
I20260709 12:21:51.472853  1977 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:51.473627  2497 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:51.470490  2408 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:51.472898  1443 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:51.477571  2408 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:51.480499  2396 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:51.481833  2330 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.970s	user 0.010s	sys 0.002s
I20260709 12:21:51.457621 32371 rpcz_store.cc:276] 0709 12:21:49.286031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.286205 (+   174us) service_pool.cc:224] Handling call
0709 12:21:51.083957 (+1797752us) spinlock_profiling.cc:243] Waited 931 ms on lock 0x7b54004272d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:51.084129 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2140709632}
I20260709 12:21:51.461504  2504 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:51.491484  2504 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:51.491657  1931 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260709 12:21:51.492156  2219 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.952s	user 0.013s	sys 0.000s
I20260709 12:21:51.492722  2219 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:51.493902 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1987) took 1139 ms. Trace:
I20260709 12:21:51.377992  2452 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 8 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:51.380702  2498 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.386590  2296 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:51.436945 32377 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:21:51.391441  2490 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:51.444521 32378 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:51.414768  2502 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:51.437680  2480 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:51.462801 31792 leader_election.cc:400] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:21:51.546656 31792 leader_election.cc:403] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:51.429921  2425 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:51.446295  2476 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:51.470566 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 388) took 1390 ms. Trace:
I20260709 12:21:51.493975  2504 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:51.494488 31746 rpcz_store.cc:276] 0709 12:21:50.354592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.491024 (+136432us) service_pool.cc:224] Handling call
0709 12:21:50.519827 (+ 28803us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:50.532764 (+ 12937us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:50.596339 (+ 63575us) log.cc:844] Serialized 14 byte log entry
0709 12:21:50.747569 (+151230us) log.cc:844] Serialized 14 byte log entry
0709 12:21:50.747789 (+   220us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:50.782962 (+ 35173us) log.cc:844] Serialized 508 byte log entry
0709 12:21:50.783132 (+   170us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:50.895702 (+112570us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.027667 (+131965us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.027762 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.493671 (+465909us) raft_consensus.cc:1685] finished
0709 12:21:51.493692 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.493841 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3208,"prepare.queue_time_us":128756,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":11619,"raft.queue_time_us":212121,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":40037,"thread_start_us":239484,"threads_started":2,"wal-append.queue_time_us":251599}
I20260709 12:21:51.157172  2436 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.342s
I20260709 12:21:51.503692 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:51.428947  2444 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:51.431983  2464 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:51.469431  2389 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.776s	user 0.014s	sys 0.000s
I20260709 12:21:51.463083  2484 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:51.446476  2208 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:51.550280 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.470789  2477 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:51.482344  2330 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:51.558492 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1953) took 1337 ms. Trace:
I20260709 12:21:51.562796 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1865) took 1992 ms. Trace:
I20260709 12:21:51.552120 31744 rpcz_store.cc:276] 0709 12:21:50.076757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.076889 (+   132us) service_pool.cc:224] Handling call
0709 12:21:50.077355 (+   466us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:50.077822 (+   467us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.077864 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:50.078448 (+   584us) log.cc:844] Serialized 26 byte log entry
0709 12:21:50.261659 (+183211us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:50.261699 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.261761 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.466659 (+1204898us) raft_consensus.cc:1685] finished
0709 12:21:51.466682 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.466826 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3180,"raft.run_cpu_time_us":50,"raft.run_wall_time_us":50,"thread_start_us":163799,"threads_started":1,"wal-append.queue_time_us":370759}
I20260709 12:21:51.569291 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:51.575141 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.443812  2491 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:51.563786  2497 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:51.555387  2476 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:51.582855 31820 rpcz_store.cc:276] 0709 12:21:49.570402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.698834 (+128432us) service_pool.cc:224] Handling call
0709 12:21:49.699317 (+   483us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:49.739229 (+ 39912us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:49.887938 (+148709us) log.cc:844] Serialized 14 byte log entry
0709 12:21:50.260812 (+372874us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:50.330938 (+ 70126us) log.cc:844] Serialized 26 byte log entry
0709 12:21:50.331090 (+   152us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:50.331125 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.331189 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.562503 (+1231314us) raft_consensus.cc:1685] finished
0709 12:21:51.562525 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.562719 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75087,"prepare.queue_time_us":124683,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":49,"raft.queue_time_us":40622,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":22824,"thread_start_us":290529,"threads_started":1,"wal-append.queue_time_us":407438}
I20260709 12:21:51.592270 31822 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:21:51.470544  2489 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:51.596565 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:51.583380  2486 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:21:51.592418 31820 pending_rounds.cc:85] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 2
I20260709 12:21:51.600781 31820 pending_rounds.cc:107] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 5.3
I20260709 12:21:51.601203 31820 raft_consensus.cc:2889] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:51.582532 31669 rpcz_store.cc:276] 0709 12:21:50.220684 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.984701 (+764017us) service_pool.cc:224] Handling call
0709 12:21:51.009488 (+ 24787us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:51.009760 (+   272us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:51.029900 (+ 20140us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.549140 (+519240us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.549369 (+   229us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:51.549437 (+    68us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:51.549511 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.549605 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.558429 (+  8824us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5670,"prepare.queue_time_us":20717,"prepare.run_cpu_time_us":116,"prepare.run_wall_time_us":16565,"raft.queue_time_us":28617,"raft.run_cpu_time_us":318,"raft.run_wall_time_us":19653,"thread_start_us":511453,"threads_started":2,"wal-append.queue_time_us":464928}
W20260709 12:21:51.601823 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 140647us
I20260709 12:21:51.598434 31744 raft_consensus.cc:2410] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:21:51.576889  2484 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:51.574615  2504 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:21:51.606616 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:51.607335 31822 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:51.607345 32378 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
W20260709 12:21:51.590334  2160 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.028s	sys 0.046s
I20260709 12:21:51.544245  2498 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:51.614135 32371 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9. Ops: [6.6-8.7]
W20260709 12:21:51.606190  2066 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.903s	user 0.012s	sys 0.003s
I20260709 12:21:51.615550 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:51.629652  2503 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:21:51.631232  2330 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:51.633052 31751 pending_rounds.cc:85] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 8
I20260709 12:21:51.633427 31751 pending_rounds.cc:107] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 12:21:51.633710 31751 raft_consensus.cc:2889] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:51.591552  2394 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:51.594360  2445 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.614926 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:51.574892  2389 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:51.627336 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:51.555630  2480 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:51.551865  2490 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:51.611454  2160 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.527s	user 0.029s	sys 0.047s
W20260709 12:21:51.616797  2446 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.212s	user 0.008s	sys 0.005s
I20260709 12:21:51.608947  2409 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:51.657248  2394 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:51.657840  2464 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:51.658799  2474 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:51.628543  2444 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:51.660504 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 377) took 1197 ms. Trace:
I20260709 12:21:51.663431 31676 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:21:51.578012  2499 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.063s
W20260709 12:21:51.671525 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 54221.9us
I20260709 12:21:51.678121 31676 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:21:51.680029 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 93624.9us
I20260709 12:21:51.676409 31818 rpcz_store.cc:276] 0709 12:21:50.462919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.463143 (+   224us) service_pool.cc:224] Handling call
0709 12:21:50.491569 (+ 28426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:51.373621 (+882052us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:51.373667 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:51.383876 (+ 10209us) log.cc:844] Serialized 26 byte log entry
0709 12:21:51.566741 (+182865us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:51.566777 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.566845 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.660293 (+ 93448us) raft_consensus.cc:1685] finished
0709 12:21:51.660319 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.660449 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4019,"raft.queue_time_us":137389,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":10126,"thread_start_us":1004397,"threads_started":2,"wal-append.queue_time_us":230487}
I20260709 12:21:51.694739  2425 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:51.696632  2393 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:51.697517  2512 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:51.698186  2512 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:51.700536  2480 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:51.701315  2512 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:51.702701  2394 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:51.703215  2474 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:51.704622  2489 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:51.705720 31753 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:51.709177 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:51.710095 31753 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:51.609200  2452 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:51.666484  2497 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:51.711359 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1952) took 1507 ms. Trace:
I20260709 12:21:51.711826 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:51.691210 31676 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:51.717298 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:51.711767 31670 rpcz_store.cc:276] 0709 12:21:50.204193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.404945 (+200752us) service_pool.cc:224] Handling call
0709 12:21:50.429724 (+ 24779us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:50.459982 (+ 30258us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.460022 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:50.469142 (+  9120us) log.cc:844] Serialized 26 byte log entry
0709 12:21:50.795121 (+325979us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:50.795159 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.795226 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.711096 (+915870us) raft_consensus.cc:1685] finished
0709 12:21:51.711117 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.711275 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20328,"raft.queue_time_us":22694,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":28873,"thread_start_us":261476,"threads_started":1,"wal-append.queue_time_us":909367}
I20260709 12:21:51.746840 31676 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:51.660856  2445 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:51.825079 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.688906  2443 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.833655 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:51.712970  2108 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:51.838948 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 127213us
W20260709 12:21:51.745781  2325 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.629s	user 0.007s	sys 0.006s
W20260709 12:21:51.844934 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:51.675498  2409 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:51.857426 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:51.833227  2396 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.862452 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:51.877468 31676 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:21:51.878150 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1945) took 1759 ms. Trace:
I20260709 12:21:51.747895  2480 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.117s
W20260709 12:21:51.882370  2516 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.194s	user 0.000s	sys 0.001s
W20260709 12:21:51.884074  2451 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.766s	user 0.003s	sys 0.000s
I20260709 12:21:51.878427 31676 rpcz_store.cc:276] 0709 12:21:50.118510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.118680 (+   170us) service_pool.cc:224] Handling call
0709 12:21:50.216116 (+ 97436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:51.663307 (+1447191us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x7b5400356c20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:21:51.877935 (+214628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.878098 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55405440}
I20260709 12:21:51.891776 31670 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:21:51.675467  2477 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:51.848860  2325 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:51.871888  2066 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:51.910812 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1991) took 1484 ms. Trace:
I20260709 12:21:51.851259  2108 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:51.848954  2443 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:51.918834  2505 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.298s	user 0.001s	sys 0.000s
I20260709 12:21:51.919238 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:51.911778 31749 rpcz_store.cc:276] 0709 12:21:50.426794 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.760160 (+333366us) service_pool.cc:224] Handling call
0709 12:21:50.789876 (+ 29716us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.790168 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:50.903528 (+113360us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.887586 (+984058us) spinlock_profiling.cc:243] Waited 25.3 ms on lock 0x7b18004bdd80. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:21:51.909696 (+ 22110us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.910331 (+   635us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:51.910387 (+    56us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:51.910441 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.910538 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.910732 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44030,"prepare.queue_time_us":80832,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":51,"raft.queue_time_us":173704,"raft.run_cpu_time_us":257,"raft.run_wall_time_us":46210,"spinlock_wait_cycles":58191744,"thread_start_us":119418,"threads_started":3}
W20260709 12:21:51.904841  2434 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.883s	user 0.010s	sys 0.005s
I20260709 12:21:51.870329  2396 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:51.918608  2443 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:51.936966 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:51.946681 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:51.950133 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1954) took 1729 ms. Trace:
I20260709 12:21:51.950802 31671 rpcz_store.cc:276] 0709 12:21:50.221046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.436927 (+1215881us) service_pool.cc:224] Handling call
0709 12:21:51.457449 (+ 20522us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:51.462404 (+  4955us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:51.462433 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:51.544873 (+ 82440us) log.cc:844] Serialized 26 byte log entry
0709 12:21:51.620619 (+ 75746us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:51.640429 (+ 19810us) log.cc:844] Serialized 14 byte log entry
0709 12:21:51.640762 (+   333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.640868 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.949793 (+308925us) raft_consensus.cc:1685] finished
0709 12:21:51.949824 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.950037 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":19585,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":4398,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":14325,"thread_start_us":40998,"threads_started":1,"wal-append.queue_time_us":153861}
W20260709 12:21:51.954917  2517 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 12:21:51.956642 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1955) took 1730 ms. Trace:
I20260709 12:21:51.960390  2445 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:51.964408  2325 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:51.967244 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:51.968894 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:51.969640 31822 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 4.
I20260709 12:21:51.942871  2491 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:51.971830  2234 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:51.972559  2234 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:51.939571  2490 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:51.908677  2480 consensus_queue.cc:1243] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:21:51.942158 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:51.942487 31670 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:51.976195 31670 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:21:51.934764  2471 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.880s	user 0.010s	sys 0.004s
I20260709 12:21:51.894728  2208 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:51.957157 31678 rpcz_store.cc:276] 0709 12:21:50.226299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.487606 (+1261307us) service_pool.cc:224] Handling call
0709 12:21:51.488084 (+   478us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:51.488566 (+   482us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:51.488600 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:51.489160 (+   560us) log.cc:844] Serialized 26 byte log entry
0709 12:21:51.578484 (+ 89324us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:51.578556 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:51.578723 (+   167us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:51.956367 (+377644us) raft_consensus.cc:1685] finished
0709 12:21:51.956394 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:51.956565 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":6494,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":82,"thread_start_us":88796,"threads_started":1,"wal-append.queue_time_us":158338}
I20260709 12:21:51.992388 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:51.984863 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:51.988251 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 117309us
I20260709 12:21:51.993458 31749 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:21:51.994105 31820 raft_consensus.cc:2410] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:21:51.931025  2389 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:21:51.979518 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:51.995400 31822 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:51.988253 31670 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:52.017812  2398 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.573s	user 0.013s	sys 0.000s
I20260709 12:21:52.018044 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.018472  2398 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.019344 31822 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:52.019838  2480 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:51.920625  2066 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.021123 31671 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:52.047266 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.048166 31749 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 5.
I20260709 12:21:52.049274 31678 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:21:52.031458 31670 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:52.040700 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:52.052336 31820 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:52.021247 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.059587  2396 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.060102  2066 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.061602 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.061923  2355 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.073025 31672 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 2. (term mismatch)
I20260709 12:21:52.073457 31672 pending_rounds.cc:85] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 1
I20260709 12:21:52.073662 31672 pending_rounds.cc:107] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260709 12:21:52.073894 31672 raft_consensus.cc:2889] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:52.074218 31672 pending_rounds.cc:107] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 4.3
W20260709 12:21:52.074311  1897 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.640s	user 0.007s	sys 0.005s
I20260709 12:21:52.074477 31672 raft_consensus.cc:2889] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:52.074986 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.054728 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.077122 31753 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:52.077771 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.078881 31746 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:52.078749 31749 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:52.080739 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.081147 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.082099 31744 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:52.090067 31749 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:21:51.989199  2471 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.086349 31820 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:52.075491  2328 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.094554 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.100028 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.101658 31820 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:52.103216 31749 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:52.105067 31670 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:21:52.105643 31670 pending_rounds.cc:85] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 7
I20260709 12:21:52.105993 31670 pending_rounds.cc:107] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:21:52.106316 31670 raft_consensus.cc:2889] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:52.092836  2398 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:52.109905 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.053751  2480 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:52.107896 31718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 2.821s (SENT) [suppressed 24 similar messages]
I20260709 12:21:52.111513 31822 raft_consensus.cc:2410] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:21:52.111873 31753 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:21:52.098378  2394 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:52.116384 31752 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:21:52.080579  2476 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.101159  2328 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.123927 31670 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:52.133205 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.117177  2234 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.134568 31822 raft_consensus.cc:2410] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:21:52.139446  2498 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:52.151675 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.153344 31822 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:52.184808 31744 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:52.186095  2498 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:52.186743  2498 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.211246 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.212082  2471 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:52.212260 31674 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1948) took 2091 ms (client timeout 1979 ms). Trace:
I20260709 12:21:52.211117  2476 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:52.212635 31674 rpcz_store.cc:269] 0709 12:21:50.120447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.120558 (+   111us) service_pool.cc:224] Handling call
0709 12:21:52.211097 (+2090539us) spinlock_profiling.cc:243] Waited 66.5 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0e9c 00007fe8f9cb0a5b 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:52.212124 (+  1027us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":152863488}
I20260709 12:21:52.220886 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:52.221794 31827 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:52.225301  2480 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:52.236099 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.237231 31751 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
W20260709 12:21:52.238237  2484 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.615s	user 0.009s	sys 0.004s
I20260709 12:21:52.239180  2484 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.240796 31818 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:52.244357  2484 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:52.243046  2498 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:52.248384  2419 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 2.589s	user 0.000s	sys 0.006s
I20260709 12:21:52.261515 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.262303 31744 raft_consensus.cc:2376] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8. Re-sending same reply.
I20260709 12:21:52.264214  2498 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.265456  2393 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:52.266327 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.266242  2393 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.268239 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.268675  2393 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:52.269184 31675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1949) took 2142 ms (client timeout 1535 ms). Trace:
I20260709 12:21:52.264923  2498 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:52.269734 31675 rpcz_store.cc:269] 0709 12:21:50.126316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.182499 (+ 56183us) service_pool.cc:224] Handling call
0709 12:21:52.267993 (+2085494us) spinlock_profiling.cc:243] Waited 13 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:52.269104 (+  1111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":29997312}
I20260709 12:21:52.270862 31822 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:21:52.273257 31827 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:21:52.275081 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.276410 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.276970 31674 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:52.277820  2511 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.276980  2498 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:52.278764  2511 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.278981 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.280663 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.281478 31678 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:21:52.281845 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.283149 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.284711 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.285547 31753 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:52.280750 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.288048 31744 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:21:52.288448 31753 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:52.282342 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.285928 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:52.271822  2064 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.292201 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.292162 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.293144 31675 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:52.294147 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.295020 31822 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:52.296133 31671 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:52.298571 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.299342 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.299801 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.300226 31744 raft_consensus.cc:2410] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:21:52.300453 31678 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:52.301059 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:52.303310 31821 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: [4.3-6.4]
I20260709 12:21:52.298571 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.306133 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:52.308697 31821 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:52.309656 31821 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:52.307453  2534 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260709 12:21:52.296192  2064 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.311363  2452 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:52.312090  2452 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:52.312434  2452 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:52.312767  2452 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:52.313210  2064 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.311141 31822 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:52.305140 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:52.293776  2498 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:52.322785 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:52.327716 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:52.297591  2389 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:21:52.323053 31674 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:52.330544 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:52.332693 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:52.330447  2476 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.335345  2476 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.337988  2476 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:52.338179 31671 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:52.338810  2208 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:52.339298 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.339387  2208 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:52.340596 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.310809  2484 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:52.341372 31822 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:52.341506  2484 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.340600  2208 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.345018  2484 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.346602  2493 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:52.347009 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.332388  2445 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:52.348407 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.348455  2445 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:52.349092 31671 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
W20260709 12:21:52.349402 31717 leader_election.cc:336] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.812s (SENT)
I20260709 12:21:52.349696 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 384) took 1865 ms. Trace:
I20260709 12:21:52.350068 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.350677 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1943) took 2246 ms. Trace:
I20260709 12:21:52.351197 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.351096 31672 rpcz_store.cc:276] 0709 12:21:50.104722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.104889 (+   167us) service_pool.cc:224] Handling call
0709 12:21:50.126272 (+ 21383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:52.075033 (+1948761us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:52.350425 (+275392us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b540017d9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:52.350603 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4476141440}
I20260709 12:21:52.352404 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.353190 31672 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:52.360486  2409 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:52.325224  2444 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:52.363077  2444 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:52.363682  2444 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:52.366621 31717 leader_election.cc:336] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.979s (SENT)
W20260709 12:21:52.369181  2331 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 10 current term: 10 is higher.
I20260709 12:21:52.349160  2445 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:52.331037  2389 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:52.390072  2389 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:21:52.391839 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1990) took 1965 ms. Trace:
I20260709 12:21:52.392264 31745 rpcz_store.cc:276] 0709 12:21:50.426474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.706084 (+279610us) service_pool.cc:224] Handling call
0709 12:21:50.706557 (+   473us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.706875 (+   318us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:50.804044 (+ 97169us) log.cc:844] Serialized 14 byte log entry
0709 12:21:52.391345 (+1587301us) spinlock_profiling.cc:243] Waited 5.82 ms on lock 0x7b18004bd780. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287
0709 12:21:52.391456 (+   111us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:52.391510 (+    54us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:52.391559 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:52.391642 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:52.391773 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2628,"prepare.queue_time_us":40222,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":1953,"raft.queue_time_us":21983,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":102,"spinlock_wait_cycles":13389184,"thread_start_us":853643,"threads_started":3,"wal-append.queue_time_us":752583}
I20260709 12:21:52.354251 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.396617 31678 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:52.398813 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:52.399870 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 67236.6us
W20260709 12:21:52.401583 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 71107.8us
I20260709 12:21:52.350106 31824 rpcz_store.cc:276] 0709 12:21:50.484374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.536081 (+ 51707us) service_pool.cc:224] Handling call
0709 12:21:50.597625 (+ 61544us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:50.629668 (+ 32043us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:50.629697 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:50.629742 (+    45us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:50.629770 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:52.349510 (+1719740us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b54002080d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:52.349551 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:52.349649 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":26881920}
W20260709 12:21:52.386274  2454 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.869s	user 0.003s	sys 0.000s
I20260709 12:21:52.403504 31674 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:52.407027 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:52.409904 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:52.415107 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:52.347095  2493 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:52.416604 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:52.414223 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:52.406008  2484 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:52.375269  2511 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:52.394362  1897 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:52.420789 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:52.430251 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:52.408152  2427 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:52.331141  2498 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.432096 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:52.410775  2409 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:52.416369  2464 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:21:52.438041  2493 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.438191  2452 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.439069  2498 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:52.433445  2427 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:52.428651  2325 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:52.419063  2526 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:52.443810 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:52.424427  2476 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.419241  2413 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 12:21:52.458053  2476 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.429113  2443 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:52.461064  2443 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:52.461134  2434 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.461601  2443 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:52.461786  2434 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.419504  2333 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.030s
I20260709 12:21:52.464666  2480 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:52.465349  2444 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.465416  2480 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.433863  2546 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:52.413620  2064 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
W20260709 12:21:52.452412 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.467900  2511 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.452805 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.468870 31824 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:52.442950  2484 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:52.419421  2393 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.028s
I20260709 12:21:52.436266  2547 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:52.435854  2355 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.468928 31671 raft_consensus.cc:2410] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 7 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260709 12:21:52.478479 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:52.440688  2464 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:52.452490  2493 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:52.452793  2470 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.477420  2325 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:52.486788  2409 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:52.489497 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:52.443945  2534 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260709 12:21:52.464973  2333 consensus_queue.cc:1243] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 8.4 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:21:52.464733  2413 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:52.433351  1897 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.519716 31753 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: [5.7-7.8]
I20260709 12:21:52.519707 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.450907  2427 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:52.468461  2389 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.496734 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.522140 31821 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:52.490669  2546 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:52.524106 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:52.492460  2476 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:52.487272  2064 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:52.499708  2444 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:52.501257  2445 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.538780 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.539484 31672 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:52.508111  2480 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:52.508968  2448 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 2.391s	user 0.003s	sys 0.000s
I20260709 12:21:52.511215 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1956) took 2272 ms. Trace:
I20260709 12:21:52.512729 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 492) took 1126 ms. Trace:
I20260709 12:21:52.544512 31669 rpcz_store.cc:276] 0709 12:21:50.239056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.609884 (+1370828us) service_pool.cc:224] Handling call
0709 12:21:51.635142 (+ 25258us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:51.665560 (+ 30418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:51.665612 (+    52us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:51.676969 (+ 11357us) log.cc:844] Serialized 52 byte log entry
0709 12:21:52.079153 (+402184us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:52.091852 (+ 12699us) log.cc:844] Serialized 14 byte log entry
0709 12:21:52.096522 (+  4670us) log.cc:844] Serialized 14 byte log entry
0709 12:21:52.096778 (+   256us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:52.096886 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:52.507439 (+410553us) spinlock_profiling.cc:243] Waited 1.86 ms on lock 0x7b18003eb960. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:21:52.510968 (+  3529us) raft_consensus.cc:1685] finished
0709 12:21:52.510996 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:52.511151 (+   155us) inbound_call.cc:177] Queueing success response
0709 12:21:52.514997 (+  3846us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b18003eb960. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f
0709 12:21:52.526030 (+ 11033us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b4800080058. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
Metrics: {"mutex_wait_us":10149,"prepare.queue_time_us":33588,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":79,"raft.queue_time_us":5683,"raft.run_cpu_time_us":4146,"raft.run_wall_time_us":441185,"spinlock_wait_cycles":13273472,"thread_start_us":248992,"threads_started":1,"wal-append.queue_time_us":397604}
I20260709 12:21:52.512946 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:52.513630  2399 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
W20260709 12:21:52.516502 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.518162 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.535419  2484 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260709 12:21:52.551154  2484 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:52.551532  2484 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:52.552522  2484 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.553651  2484 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:52.452414  2208 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.064s
I20260709 12:21:52.492159  2464 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:52.530802 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.531751  2064 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:52.532248 31753 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:21:52.532634 31749 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: [5.6-8.7]
I20260709 12:21:52.468487  2511 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.561159 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 382) took 2864 ms. Trace:
I20260709 12:21:52.561257 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.521165  2190 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.535063 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.537253 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.495211  2409 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:52.507493 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.539995 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.544595 32377 rpcz_store.cc:276] 0709 12:21:51.386364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.386787 (+   423us) service_pool.cc:224] Handling call
0709 12:21:51.390450 (+  3663us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:51.538309 (+147859us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:52.512421 (+974112us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b54004bb254. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:52.512631 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105596672}
I20260709 12:21:52.534238  2547 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:21:52.549270 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.495754  2470 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.492466  2355 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.520780 31752 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:52.532199  2234 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:52.551391 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.559657 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:52.563997 31678 raft_consensus.cc:2376] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6. Re-sending same reply.
I20260709 12:21:52.581349 32372 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:52.567582 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 398) took 1496 ms. Trace:
I20260709 12:21:52.567272 31747 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:52.569887  2511 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.581966 31748 rpcz_store.cc:276] 0709 12:21:51.070725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.071530 (+   805us) service_pool.cc:224] Handling call
0709 12:21:51.071968 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:51.078514 (+  6546us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:51.078549 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:51.079179 (+   630us) log.cc:844] Serialized 26 byte log entry
0709 12:21:52.276804 (+1197625us) spinlock_profiling.cc:243] Waited 2.65 ms on lock 0x7b1800383d40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:21:52.297018 (+ 20214us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:52.324158 (+ 27140us) log.cc:844] Serialized 14 byte log entry
0709 12:21:52.324480 (+   322us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:52.324586 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:52.559143 (+234557us) raft_consensus.cc:1685] finished
0709 12:21:52.567327 (+  8184us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:52.567500 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6100,"prepare.queue_time_us":28743,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"raft.queue_time_us":76422,"raft.run_cpu_time_us":220,"raft.run_wall_time_us":5503,"spinlock_wait_cycles":15911552,"thread_start_us":20001,"threads_started":1,"wal-append.queue_time_us":1313732}
I20260709 12:21:52.570520  2480 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:52.574410 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.576210 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.576892 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:52.531267  2413 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.587482 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.579146 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.555243  2476 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:52.568065 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.590711 31749 raft_consensus.cc:2376] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6. Re-sending same reply.
I20260709 12:21:52.591043 31753 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:52.561661 31673 rpcz_store.cc:276] 0709 12:21:49.697337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.709285 (+ 11948us) service_pool.cc:224] Handling call
0709 12:21:49.731771 (+ 22486us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:49.840952 (+109181us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:49.840987 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:49.879022 (+ 38035us) log.cc:844] Serialized 26 byte log entry
0709 12:21:50.103797 (+224775us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:50.405733 (+301936us) log.cc:844] Serialized 14 byte log entry
0709 12:21:50.411164 (+  5431us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:50.411280 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:52.508345 (+2097065us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:52.554425 (+ 46080us) raft_consensus.cc:1685] finished
0709 12:21:52.560978 (+  6553us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:52.561105 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27745,"prepare.queue_time_us":422513,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":5758,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":5074,"spinlock_wait_cycles":2771200,"thread_start_us":480922,"threads_started":2,"wal-append.queue_time_us":228196}
I20260709 12:21:52.589321 31744 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:52.575071  2409 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:52.599041 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1882) took 1011 ms. Trace:
I20260709 12:21:52.580063  2498 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:52.607200 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:52.613018 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.614120 31827 rpcz_store.cc:276] 0709 12:21:51.587994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.274070 (+686076us) service_pool.cc:224] Handling call
0709 12:21:52.274397 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:52.308512 (+ 34115us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:52.308552 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:52.321324 (+ 12772us) log.cc:844] Serialized 26 byte log entry
0709 12:21:52.479307 (+157983us) raft_consensus.cc:1645] Marking committed up to 1
0709 12:21:52.479353 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:52.479428 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:52.598794 (+119366us) raft_consensus.cc:1685] finished
0709 12:21:52.598821 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:52.598983 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4000,"raft.queue_time_us":11963,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":16294,"thread_start_us":71526,"threads_started":1,"wal-append.queue_time_us":263336}
I20260709 12:21:52.594492 31752 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:52.578238  2190 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.575129  2526 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:52.651505 31749 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:52.616640  2389 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:52.617412 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.657469 31677 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:52.541935  2325 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.623943  2327 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:52.627381  2452 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:52.628010  2064 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.632515 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.671551 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.584586  2547 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:21:52.659405 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 83211.5us
I20260709 12:21:52.674677 31747 pending_rounds.cc:85] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 3
I20260709 12:21:52.675207 31747 pending_rounds.cc:107] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 8.4
I20260709 12:21:52.647933 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:21:52.669382 31677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 1940) took 2580 ms (client timeout 1812 ms). Trace:
I20260709 12:21:52.677284 31749 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
W20260709 12:21:52.676084 31677 rpcz_store.cc:269] 0709 12:21:50.078409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.111868 (+ 33459us) service_pool.cc:224] Handling call
0709 12:21:52.507306 (+2395438us) spinlock_profiling.cc:243] Waited 7.11 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:52.658477 (+151171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":16342400}
W20260709 12:21:52.678550 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.598421  2355 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.675519 31747 raft_consensus.cc:2889] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:52.691813 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 105862us
I20260709 12:21:52.691836 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.692996 31677 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:21:52.694519 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:52.680735  2064 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:52.657110  2498 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.698680  2190 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:52.699302  2399 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.700412  2413 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.700966  2526 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.714955  2464 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.668654  2443 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.703917 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.716421 31827 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:52.686949  2480 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:52.683523  2476 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:52.713845  2399 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:52.720602 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45355us
I20260709 12:21:52.722911 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:52.725148 32372 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:21:52.688005  2355 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 12:21:52.736670 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:52.741315 31677 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:21:52.722124  2445 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.742084 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:52.748757 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 54335.9us
I20260709 12:21:52.724973  2480 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:52.749329 32377 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:52.753883 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:52.753168  2554 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:52.755720  2554 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:52.757409 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:52.761690  2555 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:52.748662  2547 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.764612 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:52.766937  2555 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:52.768229  2498 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.768376  2559 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:21:52.761514  2554 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:52.769490  2480 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.773387  2553 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:21:52.774539 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:52.744105  2413 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:21:52.775884 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.775242 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:52.776938 32370 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:52.755657 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:52.782964 31674 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:21:52.783679 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:52.742854  2497 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260709 12:21:52.792429 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.793431 31827 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:52.762514  2556 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:52.796492 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.785374  2498 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:21:52.781322  2445 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.781234  2551 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.751426  2427 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.800844 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.813210 31674 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:21:52.799678  2556 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:52.816718 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 48239.2us
I20260709 12:21:52.802416  2333 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:52.819520 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.807407  2399 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:52.809595  2409 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.828640  2427 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:52.829551 31673 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:52.829457  2409 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.810732  2445 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.829995  2333 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:52.833122 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.833828 31751 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 8
I20260709 12:21:52.834074  2333 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:52.818023 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:52.803186  2498 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:52.824484 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.824592 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:52.774909  2476 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.842162 31751 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:52.843552 32377 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:52.815147  2551 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.845722  2190 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:21:52.807464  2296 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:52.847620 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.847548 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.850221 31827 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:52.850708 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.849515 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.851949 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.854336  2554 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.849339 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.862303 31749 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:52.857062 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.845320  2498 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:21:52.871649 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.872589 31674 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:52.851374  2296 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:52.877658 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.878374 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.887197 31749 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:52.888190 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.888684 31749 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:52.890599 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.891290 31674 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:52.870000  2445 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:52.892079 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 72318.5us
W20260709 12:21:52.894093 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.899705 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.877585  2443 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.904053  2296 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.904143 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.907397 31749 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:52.908603 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.909202 31749 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:21:52.909874  2399 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.910545 31749 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:21:52.910434  2399 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.913992  2409 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.914580  2409 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.916792  2409 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:52.919279 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:52.920060 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:52.920288 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:52.921479 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:52.922598  2413 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:52.923393  2500 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:52.923549  2413 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:52.923849  2500 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:52.924145  2413 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:52.924319  2500 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:52.925108  2413 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.927512 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.922494  2440 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:52.928546  2440 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:52.929405 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.929445  2440 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.932401 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:52.932636 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:52.937004  2394 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:52.937583  2394 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:52.938534  2394 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:52.941273  2273 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.923756 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.942127 31675 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:52.939013 31744 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:52.949254 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:52.950737 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:52.948982 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:52.952795 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:52.954006 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:52.955310 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:52.959111 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:52.959932 31827 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:52.962656 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.963477 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:52.963784 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:52.964287 31677 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:52.965170  2465 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 2.318s	user 0.004s	sys 0.001s
I20260709 12:21:52.966094  2379 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:52.966602  2379 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:52.969184  2379 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:52.892925  2464 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:52.971511 31670 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:21:52.996176 31677 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:53.000064  2554 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.996176  2464 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:53.001430 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 1984) took 2653 ms. Trace:
I20260709 12:21:53.001370  2464 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.001873 31750 rpcz_store.cc:276] 0709 12:21:50.347661 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:50.485620 (+137959us) service_pool.cc:224] Handling call
0709 12:21:50.486117 (+   497us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:50.601455 (+115338us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.001143 (+2399688us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b5400213ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:53.001350 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":263567616}
I20260709 12:21:53.003767  2464 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:53.005471 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:52.998251 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.006157 31674 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:53.007270 31677 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:53.008745 31677 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:53.014915  2551 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:52.960747  2556 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:52.961426  2511 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:21:52.957746  2452 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:52.957571  2484 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:52.960976  2208 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:53.020233  2208 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.020849  2208 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:52.961318  2546 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:53.026449 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:53.027560 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:52.970448  2234 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:21:53.039280 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:52.974006  2409 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.069s
I20260709 12:21:53.032894  2511 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:53.042115 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:53.033442  2484 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:53.061856 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:53.040480  2452 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:53.040663  2234 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:53.057003  2273 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:53.097024 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.097986 31674 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:53.078114  2452 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.078352  2484 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:53.109143 31752 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:21:53.116586 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:53.097245  2526 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:53.070451  2561 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.050s
I20260709 12:21:53.113942  2452 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:53.146598 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:53.160745 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44256.7us
I20260709 12:21:53.161769 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:53.096967  2500 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.143544  2476 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260709 12:21:53.160218  2480 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:53.149410  2526 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.172142  2476 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:53.196933  2571 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:21:53.197185  2328 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:53.199750  2574 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:21:53.120216  2511 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.177701  2480 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:53.148820  2234 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:53.232848 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 71190.3us
I20260709 12:21:53.203239  2476 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:53.265407 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:53.225308  2480 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:53.228297  2561 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:53.230685  2389 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:53.276471 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 50722.6us
I20260709 12:21:53.228780  2556 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:21:53.295612 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:53.304333 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:53.309082 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.294850  2389 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.319208  2389 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:53.320417  2262 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:53.321056  2262 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:21:53.321539  2580 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
I20260709 12:21:53.322089  2262 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.226006  2484 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.323573  2262 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:53.327185  2576 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20260709 12:21:53.335940 31671 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:21:53.291759  2556 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:53.320848  2579 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.149s	user 0.000s	sys 0.000s
W20260709 12:21:53.351341 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 86112.7us
I20260709 12:21:53.346882  2556 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:53.364128  2500 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:53.368583  2526 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:53.344435  2546 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
W20260709 12:21:53.366317  2578 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.221s	user 0.001s	sys 0.000s
I20260709 12:21:53.350859  2273 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.020s
W20260709 12:21:53.384375 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 101234us
W20260709 12:21:53.389279  2498 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.511s	user 0.009s	sys 0.001s
I20260709 12:21:53.401216 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:53.353114  2594 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:21:53.417475 31749 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:21:53.373467  2589 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:53.473254 31749 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: []
W20260709 12:21:53.472010  2591 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
I20260709 12:21:53.460747  2589 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.474925  2593 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.393520  2355 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.080s
I20260709 12:21:53.415728  2498 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.416586  2555 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 12:21:53.499955 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:53.562000 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:53.563647 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:53.485522  2589 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:53.563982  2586 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:53.565420 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:53.580380 32370 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:53.559183  2066 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:53.581192 32376 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:21:53.581398  2066 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:53.581878  2546 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:53.571035 31643 connection.cc:256] RPC call timeout handler was delayed by -1.4930180329997711s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18180000000000002s for a response
I20260709 12:21:53.572777 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:53.582356  2066 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.573134 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:53.478103  2600 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.095s
I20260709 12:21:53.575811 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:53.577266 31751 raft_consensus.cc:2368] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11.
I20260709 12:21:53.574210 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:53.493338  2593 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.583396  2328 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:53.579926  2586 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:53.595991 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.583410  2066 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:53.599030 31746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 1995) took 2149 ms (client timeout 2427 ms). Trace:
W20260709 12:21:53.608685 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:53.599649 31746 rpcz_store.cc:269] 0709 12:21:51.445457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:51.568921 (+123464us) service_pool.cc:224] Handling call
0709 12:21:53.595042 (+2026121us) spinlock_profiling.cc:243] Waited 491 ms on lock 0x7b54004047d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:53.595180 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1130324736}
I20260709 12:21:53.582363  2546 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.630705 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:53.632598  2593 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:53.633419  2593 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.642357  2605 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:53.643165  2605 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.645660  2498 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:53.645970  2605 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:53.648476  2208 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.627s	user 0.014s	sys 0.000s
I20260709 12:21:53.648929  2208 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.595997  2328 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.626904  2484 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:53.651502  2484 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.653149  2328 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:53.654006  2328 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:53.617620  2607 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.655750  2607 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.658202  2546 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:53.622835  2555 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.660598  2555 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.607558  2586 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:53.634011  2593 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:53.626192  2592 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.246s	user 0.000s	sys 0.001s
I20260709 12:21:53.681351 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:53.654273  2484 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:53.675724  2603 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20260709 12:21:53.688444 31819 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:53.682868 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:53.679966  2601 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.195s	user 0.001s	sys 0.000s
I20260709 12:21:53.689965 31825 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:53.658499  2607 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:53.679455  2546 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:53.700402 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 117292us
I20260709 12:21:53.700628 31827 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:21:53.686049  2590 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
W20260709 12:21:53.703127 31643 connection.cc:256] RPC call timeout handler was delayed by -1.2971219010000823s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2404s for a response
W20260709 12:21:53.704319 31643 connection.cc:256] RPC call timeout handler was delayed by -1.2019665509997139s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2129s for a response
I20260709 12:21:53.668346  2476 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.705448  2546 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.711144  2476 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:53.711318  2546 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:53.704865  2602 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:21:53.674011  2328 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.698503  2561 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:53.720870 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 368634us
I20260709 12:21:53.715756  2555 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:53.723677 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:53.735769 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:53.737382 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:53.707718  2393 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:53.684327  2570 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 12:21:53.752585  2500 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:53.753237 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:53.754381 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:53.755208  2444 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:53.755422 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:53.757910 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.757617 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:53.760426  2500 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:53.760267  2393 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.762147 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:53.762982  2500 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:53.764524 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:53.765360  2500 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:53.728439  2447 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 4, Time since last communication: 0.038s
I20260709 12:21:53.766693  2500 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Stepping down as leader of term 7
W20260709 12:21:53.766955  2604 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 12:21:53.767159  2500 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:53.730309  2208 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:53.767621  2593 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:53.768476  2500 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.768587  2561 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.769999 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:53.770543 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:53.771669 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.712481  2262 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.060s
I20260709 12:21:53.774132  2608 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:21:53.781909 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:53.781222  2613 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:53.775677  2598 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:53.777066  2476 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:53.775888  2208 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:53.778053  2500 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:53.779824  2555 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:53.799340  2393 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:53.800485  2434 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:53.786432  2296 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:53.800987  2452 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.631s	user 0.008s	sys 0.004s
I20260709 12:21:53.805878  2208 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260709 12:21:53.813966  2444 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.014s
I20260709 12:21:53.828411  2444 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:53.828198  2556 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.828951  2444 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:53.813701 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:53.808168  2613 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:53.845101 31825 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
W20260709 12:21:53.845366 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.845784 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:53.855026 31818 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:53.807281  2608 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:53.847024 31821 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:53.829965  2444 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.838984  2586 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.850186 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:53.812045  2328 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:53.851961  2589 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.859855 31820 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:21:53.836608  2452 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.862071 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:53.862277  2444 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:53.853667  2434 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.862735 31819 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:53.863072  2434 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.864099  2586 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:53.852893  2480 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.555s	user 0.005s	sys 0.006s
I20260709 12:21:53.865339  2434 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:53.865767  2480 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.868363  2480 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:53.868569  2589 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:53.870141  2499 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:53.870563  2499 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:53.871376  2499 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.858270 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:53.853315  2476 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:53.835757  2234 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:53.873622 31823 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260709 12:21:53.873942  2234 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.875772  2452 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:53.876531  2582 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.632s	user 0.001s	sys 0.000s
I20260709 12:21:53.876757  2234 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:53.885763 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.886489 31671 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:53.884544  2608 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.887590 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:53.888204 31671 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:53.889979 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:53.890847 31819 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:53.891709 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:53.892885 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:53.810891  2066 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260709 12:21:53.895306  2066 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:53.895650  2066 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:53.895936  2399 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:53.896530  2066 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.897651 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:53.897816  2066 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:53.898877 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.899480 31678 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:53.900090 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.900382 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.900688 31672 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:53.901021 31678 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:53.877660  2470 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:53.877849 31826 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:53.903719 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:53.904222  2598 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:53.904408 31826 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:53.880210 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:53.881572 31820 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:21:53.905157 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:53.806383  2555 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:53.906371  2555 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:53.882863 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:53.911175 31818 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:53.883005 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:53.883665  2556 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:53.913708  2556 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.914228  2556 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.883796 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.884299 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:53.884507 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.915459 31674 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:53.915761 31822 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:53.915868 31676 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:21:53.873633  2476 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.885046 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.916442 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.916908 31669 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:53.906169 31823 raft_consensus.cc:2410] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 8 because replica has last-logged OpId of term: 6 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:21:53.885226 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.885385 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:53.918143 31825 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:21:53.887542 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1520) took 1104 ms. Trace:
I20260709 12:21:53.893628 31819 raft_consensus.cc:2410] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 6 because replica has last-logged OpId of term: 5 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260709 12:21:53.922585 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.920231 31752 rpcz_store.cc:276] 0709 12:21:52.783136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.935369 (+152233us) service_pool.cc:224] Handling call
0709 12:21:52.935703 (+   334us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:53.114925 (+179222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:53.887275 (+772350us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b54002f5dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:53.887483 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":397986688}
I20260709 12:21:53.923586 31669 raft_consensus.cc:2410] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:21:53.924554 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.925164 31669 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:53.926057 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.927109 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:53.927966 32372 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:21:53.928315 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:53.922433 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:53.929172 31677 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
W20260709 12:21:53.938078 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.939796 31669 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:53.922289  2108 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:53.941965 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:53.938135 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:53.944496 32371 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:21:53.944372  2555 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:53.922333  2476 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:53.932826  2605 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:53.947710  2555 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:53.949189  2556 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:53.950618  2587 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.768s	user 0.002s	sys 0.000s
I20260709 12:21:53.941965 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:53.954357 31669 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:53.954377 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:53.956732 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:53.957952 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:53.953495 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 276109us
I20260709 12:21:53.958990 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:53.960043 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:53.961833 31671 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:53.969906  2464 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:53.970098  2556 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:53.970616  2556 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.970976  2613 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:53.971087  2556 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:53.970479  2464 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:53.971899  2464 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:21:53.973032 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.068s	user 0.001s	sys 0.027s
W20260709 12:21:53.973364 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.071s	user 0.002s	sys 0.027s
W20260709 12:21:53.973624 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 1.07104e+06us
I20260709 12:21:53.976058  2325 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:53.976548  2325 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:53.976953  2325 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:53.953461  2605 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:53.980440  2605 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:53.981550 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:53.983954 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:53.984687 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:53.986351 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:53.999452 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:54.001097 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:54.002966 31717 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:54.008872 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:53.998466  2605 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:54.007570  2296 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:54.017766 31748 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:54.016613  2605 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:54.007133  2570 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:54.009553  2399 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:54.007587  2555 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:54.022919 31747 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: [5.7-7.8]
W20260709 12:21:54.059688 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74383.1us
I20260709 12:21:54.010250  2554 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:54.024497 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:21:54.028420 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 223845us
I20260709 12:21:54.009189  2613 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:54.016497  2108 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:54.005255  2553 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20260709 12:21:54.007061  2160 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.015s
I20260709 12:21:54.008462  2552 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20260709 12:21:54.072114 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:54.093873 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.094732 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:21:54.062388  2559 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.293s	user 0.010s	sys 0.023s
I20260709 12:21:54.022655  2570 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:54.032990  2605 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:54.089955 31671 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
W20260709 12:21:54.066500 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.096613 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:54.024276  2399 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:54.059762  2555 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:54.107429 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:54.078518 32372 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:54.109871 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:54.110805 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:54.064330  2613 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:54.062225  2554 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:54.124002 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:54.126058 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:54.132035 31671 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:21:54.104280  2559 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.335s	user 0.010s	sys 0.023s
I20260709 12:21:54.135789  2511 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:54.135488  2613 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:54.141453 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:54.148005 31671 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:54.149139 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:54.150008 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:54.153229 31671 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:21:54.082935  2108 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:54.153553 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:54.064433  2234 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.067s
I20260709 12:21:54.121539  2399 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:54.131021  2555 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:54.104215  2527 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:54.105643  2570 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:54.068590  2447 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.083s
I20260709 12:21:54.133268  2554 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:54.081245  2328 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.077s
I20260709 12:21:54.082492  2500 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 1, Time since last communication: 0.076s
I20260709 12:21:54.135990  2477 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:54.151666  2552 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.164891  2561 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:54.167554  2616 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:54.220955 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1405) took 1605 ms. Trace:
I20260709 12:21:54.221606 31673 rpcz_store.cc:276] 0709 12:21:52.615180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.657077 (+ 41897us) service_pool.cc:224] Handling call
0709 12:21:54.220628 (+1563551us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b540036f1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:54.220856 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":361118976}
I20260709 12:21:54.229424  2499 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:54.243400 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:54.271303 31673 raft_consensus.cc:1813] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Handling vote request from an unknown peer 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:54.272274 31673 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 8 because replica has last-logged OpId of term: 7 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 12:21:54.273581 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:54.287200 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:54.335738 31673 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
W20260709 12:21:54.186019  2413 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.256s	user 0.013s	sys 0.024s
I20260709 12:21:54.357159 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:54.182296  2561 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:54.181464  2477 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:54.184492  2393 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:21:54.438261 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:54.360914  2413 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.430s	user 0.014s	sys 0.024s
I20260709 12:21:54.178822  2552 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.446524  2296 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:54.448217  2566 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.491s	user 0.016s	sys 0.000s
I20260709 12:21:54.449294  2616 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:21:54.450160 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 79196.1us
I20260709 12:21:54.360671  2483 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.339059  2527 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:54.486711 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1396) took 1957 ms. Trace:
I20260709 12:21:54.487365 31670 rpcz_store.cc:276] 0709 12:21:52.529597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.529775 (+   178us) service_pool.cc:224] Handling call
0709 12:21:52.543939 (+ 14164us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:52.972002 (+428063us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.486354 (+1514352us) spinlock_profiling.cc:243] Waited 427 ms on lock 0x7b540022b354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:54.486608 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":982259584}
I20260709 12:21:54.365741  2608 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:54.489476 31678 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.015s	sys 0.008s
W20260709 12:21:54.491168 31678 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.016s	sys 0.008s
W20260709 12:21:54.493750 31669 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.539s	user 0.013s	sys 0.000s
I20260709 12:21:54.493975 31671 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:54.495857 31671 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:21:54.494391 31669 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:54.377247  2452 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.368383  2561 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:54.503362 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 347821us
W20260709 12:21:54.473039  2583 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.004s	sys 0.035s
I20260709 12:21:54.516376 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:54.520918 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:54.524766 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:54.524981 32372 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:21:54.527024 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:54.522285  2583 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.616s	user 0.004s	sys 0.035s
I20260709 12:21:54.530164 31670 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:21:54.528136 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:54.539059 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:54.541133 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:54.353734  2511 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.133s
W20260709 12:21:54.517480  2440 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.016s	sys 0.013s
I20260709 12:21:54.544689 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:54.532334 31669 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:21:54.559648 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:54.524111  2527 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:54.561662 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 537226us
I20260709 12:21:54.565539 31670 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:21:54.568677 31670 pending_rounds.cc:85] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 7
I20260709 12:21:54.569001 31670 pending_rounds.cc:107] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:21:54.569324 31670 raft_consensus.cc:2889] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:54.570922 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:54.537288  2499 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:54.529093  2234 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:54.558197  2440 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.017s	sys 0.013s
I20260709 12:21:54.361116  2108 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.579873 32376 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9. Ops: [6.6-8.7]
W20260709 12:21:54.574942 31751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1519) took 1814 ms (client timeout 2099 ms). Trace:
I20260709 12:21:54.580152 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:54.583259 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:54.596009 32372 raft_consensus.cc:2410] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260709 12:21:54.559553  2547 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.793s	user 0.019s	sys 0.063s
I20260709 12:21:54.515075  2493 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.047s
I20260709 12:21:54.503014  2296 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.602213 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:54.522970  2471 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:21:54.609305 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 497) took 1914 ms. Trace:
W20260709 12:21:54.595455 31751 rpcz_store.cc:269] 0709 12:21:52.760712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.849292 (+ 88580us) service_pool.cc:224] Handling call
0709 12:21:54.574739 (+1725447us) spinlock_profiling.cc:243] Waited 725 ms on lock 0x7b54000b90d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:54.574880 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1667532672}
I20260709 12:21:54.613029 32375 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:21:54.613168 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:54.614498 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:54.615326 31825 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:21:54.561242  2434 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.578543  2477 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.665618  2434 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.616587 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:54.519045  2608 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.618872 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:54.579900  2527 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.523209  2452 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.671382  2452 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:54.671605  2527 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:54.672155  2452 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:54.672487  2527 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:54.609881 32369 rpcz_store.cc:276] 0709 12:21:52.694971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.695115 (+   144us) service_pool.cc:224] Handling call
0709 12:21:52.712063 (+ 16948us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:52.723410 (+ 11347us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:52.723439 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:52.739483 (+ 16044us) log.cc:844] Serialized 52 byte log entry
0709 12:21:52.815663 (+ 76180us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:52.815722 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:52.815823 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:54.447545 (+1631722us) spinlock_profiling.cc:243] Waited 3.83 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:54.608931 (+161386us) raft_consensus.cc:1685] finished
0709 12:21:54.608963 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.609239 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3251,"raft.queue_time_us":4611,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":12736,"spinlock_wait_cycles":8806656,"thread_start_us":59863,"threads_started":1,"wal-append.queue_time_us":120257}
I20260709 12:21:54.672971  2527 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:54.600018  2234 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.673879  2527 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:54.672703  2452 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.675802  2234 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:54.654757  2440 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:21:54.617412  2160 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.022s
W20260709 12:21:54.655588 32371 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.710s	user 0.008s	sys 0.005s
W20260709 12:21:54.655874 32341 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.016s	sys 0.013s
W20260709 12:21:54.682963 32341 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.016s	sys 0.013s
W20260709 12:21:54.608795  2547 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.842s	user 0.020s	sys 0.063s
W20260709 12:21:54.656093  2066 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.758s	user 0.009s	sys 0.009s
I20260709 12:21:54.671499  2608 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:54.659783 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:54.663359 32374 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:21:54.664042 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:54.693596 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:54.664942 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:54.559085  2389 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:54.560580  2355 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:21:54.673390  2464 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.701s	user 0.011s	sys 0.000s
I20260709 12:21:54.676812  2452 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:54.572537  2498 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.687059 32371 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:21:54.710570  2498 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.703469  2066 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:54.500541  2483 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.578341  2600 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.683966  2440 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:54.656388  2296 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:54.697227 31825 raft_consensus.cc:2410] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 8.
I20260709 12:21:54.699380 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:54.615680  2499 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:54.711673  2160 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Stepping down as leader of term 6
W20260709 12:21:54.643424  2618 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:21:54.701624  2413 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:54.564739  2583 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:54.685091  2234 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.753598 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:54.716092 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:54.716995 31644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.023s	sys 0.051s
W20260709 12:21:54.755182 31644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.024s	sys 0.051s
I20260709 12:21:54.753118 32378 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:54.756742 32378 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:21:54.758144 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.735085 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:54.736385 31676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.013s	sys 0.027s
I20260709 12:21:54.707732  2464 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.735982 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:54.759470 32376 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
W20260709 12:21:54.759784 31676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.013s	sys 0.028s
I20260709 12:21:54.707894  2547 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:21:54.716704  2447 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.018s	sys 0.019s
I20260709 12:21:54.761379 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:54.739516 31641 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.013s	sys 0.027s
W20260709 12:21:54.772794 31641 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.014s	sys 0.027s
W20260709 12:21:54.762907 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.763538  2471 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:54.760017 32376 pending_rounds.cc:85] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 6
I20260709 12:21:54.775187 32376 pending_rounds.cc:107] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260709 12:21:54.775521 32376 raft_consensus.cc:2889] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:21:54.777114 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:54.783180 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:54.751849  2605 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.617s	user 0.008s	sys 0.004s
W20260709 12:21:54.752936  2325 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.774s	user 0.013s	sys 0.000s
W20260709 12:21:54.797726 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 44234.2us
W20260709 12:21:54.751540  2556 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.780s	user 0.012s	sys 0.000s
I20260709 12:21:54.798856 31677 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:21:54.799736 31669 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:21:54.754184  2434 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:54.739688  2452 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:54.771906 32378 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.795130 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:54.812927 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:54.739394  2160 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:54.708858  2355 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.834072 31676 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:54.742226  2498 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:54.748611  2413 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:54.709242  2389 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:54.771936  2447 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.019s	sys 0.019s
I20260709 12:21:54.838447 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:54.798878  2476 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.853s	user 0.011s	sys 0.000s
I20260709 12:21:54.797937  2639 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.085s
I20260709 12:21:54.878846  2499 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.867120  2434 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:54.858170  2637 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 12:21:54.881322 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:54.881430  2296 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:21:54.882154 31670 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:54.878892  2483 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:54.856065  2547 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:54.838145  2556 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.868661  2413 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:54.885569  2556 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:54.886691 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:54.887316  2526 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:54.887387 31669 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:54.887904  2526 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:54.888813  2526 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.890028  2526 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:54.890552  2613 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.726s	user 0.011s	sys 0.001s
I20260709 12:21:54.891053  2613 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.891795 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 388) took 1224 ms. Trace:
I20260709 12:21:54.892198 31827 rpcz_store.cc:276] 0709 12:21:53.667450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.677085 (+  9635us) service_pool.cc:224] Handling call
0709 12:21:53.677500 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.702818 (+ 25318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.891549 (+1188731us) spinlock_profiling.cc:243] Waited 21.4 ms on lock 0x7b5400196ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:54.891736 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":49122688}
I20260709 12:21:54.799793  2621 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.069s
I20260709 12:21:54.894850  2434 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:54.895936  2434 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.882167  2497 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:21:54.903286  2497 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:54.846721  2440 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.904640  2440 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:54.904264  2497 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.802539  2608 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:54.773077  2234 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.911305  2608 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:54.911918  2608 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:54.913055  2234 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:54.915496  2497 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
W20260709 12:21:54.916896  2399 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.743s	user 0.009s	sys 0.000s
I20260709 12:21:54.917445  2399 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.918076  2389 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:54.918769 31752 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:54.770054  2583 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.741153  2600 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.902189  2413 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.920604  2583 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:54.920872  2600 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:54.848444  2471 consensus_queue.cc:1243] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:21:54.877378  2355 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:54.860374 32378 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:54.834489  2325 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.905153  2447 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:21:54.857264 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 62238.5us
I20260709 12:21:54.833168  2605 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.802544  1897 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:54.878961  2634 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
I20260709 12:21:54.922259 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:54.908866  2527 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.938699 31669 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
W20260709 12:21:54.933534 31675 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.006s	sys 0.020s
W20260709 12:21:54.940518 31675 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.007s	sys 0.020s
W20260709 12:21:54.948017 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:54.929725  2511 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.980798  2234 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.981477  2234 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.982656 32371 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:54.983069 32371 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:54.984046  2234 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:54.986449 31670 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:21:54.986847 31670 pending_rounds.cc:85] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 5
I20260709 12:21:54.987067 31670 pending_rounds.cc:107] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:21:54.987380 31670 raft_consensus.cc:2889] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:54.988090 31746 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 5. Current term is 7. Ops: []
I20260709 12:21:54.988453 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:54.989156 31670 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:54.989995 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:54.949651  2389 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:54.947003  2583 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260709 12:21:54.954988  2594 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.956346  2464 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:54.957934  2605 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:54.959089  2554 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.779s	user 0.009s	sys 0.002s
I20260709 12:21:54.915561  2561 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:54.959553  2555 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.784s	user 0.005s	sys 0.005s
I20260709 12:21:54.927914  2552 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:54.960150  2547 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.960692 32378 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:21:54.961323  2413 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:54.963253 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:55.027917 31749 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9. Ops: []
W20260709 12:21:55.028117 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64943.4us
I20260709 12:21:55.029116 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:54.963725  2570 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.784s	user 0.008s	sys 0.001s
I20260709 12:21:54.963829  2452 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.938738  2160 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.964488  2325 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:54.930373  2476 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.936453  2613 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:54.975030  2619 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:21:54.977850  2596 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.516s	user 0.000s	sys 0.008s
I20260709 12:21:54.979318  2636 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:54.929380  2447 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:54.950610 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:54.990862 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.099231 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.099946 31746 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:55.012778  2594 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.997710 31670 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:55.003553 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:54.983853 32371 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:54.986287  2234 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260709 12:21:54.980100  2511 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.980125  2393 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:55.119957  2393 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:55.032230 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.033702 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.122460 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.123106 31745 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:55.123849 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.124594 31745 raft_consensus.cc:2410] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:21:55.125305 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.125833  2160 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:55.039067 31753 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260709 12:21:55.127528  2619 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:55.127619 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:55.128309  2619 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:55.042912 31750 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: []
W20260709 12:21:55.043898 31790 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.020s	sys 0.042s
W20260709 12:21:55.129328 31790 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.020s	sys 0.042s
I20260709 12:21:55.130966 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:55.027619  2130 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 6 current term: 6 is higher.
I20260709 12:21:55.133597 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:55.010707  2633 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260709 12:21:55.010215  2389 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:54.947871 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1983) took 1063 ms. Trace:
I20260709 12:21:55.090595  2477 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:21:55.178175 31675 rpcz_store.cc:276] 0709 12:21:53.883956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.902906 (+ 18950us) service_pool.cc:224] Handling call
0709 12:21:53.903210 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.903434 (+   224us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:54.218620 (+315186us) log.cc:844] Serialized 14 byte log entry
0709 12:21:54.946206 (+727586us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:54.946263 (+    57us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:54.946317 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:54.947680 (+  1363us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:54.947813 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":637431,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":50,"raft.queue_time_us":6241,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"thread_start_us":1039108,"threads_started":2}
I20260709 12:21:55.103628 31752 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:21:55.103173 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:55.180022 31669 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:55.023301  2554 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:55.041953  2464 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:21:55.108173 31642 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.000s	sys 0.015s
I20260709 12:21:55.102453  2608 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:55.025007  2555 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:55.208217 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:55.108557 31678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.000s	sys 0.014s
W20260709 12:21:55.209074 31678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.000s	sys 0.015s
I20260709 12:21:55.102241  2447 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:55.102043  2636 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:55.046020  2452 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:55.037151  2547 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:55.221702 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:55.222019 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:55.222347 31826 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:55.055140  2413 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.064s
W20260709 12:21:55.228734 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.010s	sys 0.014s
I20260709 12:21:55.126401 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:55.142568 31673 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.022s	sys 0.021s
W20260709 12:21:55.229930 31673 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.023s	sys 0.021s
W20260709 12:21:55.142961 31674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.927s	user 0.016s	sys 0.047s
I20260709 12:21:55.177245 31670 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
W20260709 12:21:55.230998 31674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.041s	user 0.016s	sys 0.048s
W20260709 12:21:55.231910 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.655s	user 0.010s	sys 0.015s
I20260709 12:21:55.231712 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:55.232221 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 662834us
I20260709 12:21:55.231776 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1989) took 1337 ms. Trace:
I20260709 12:21:55.233292 31674 rpcz_store.cc:276] 0709 12:21:53.894348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.930959 (+ 36611us) service_pool.cc:224] Handling call
0709 12:21:53.931313 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:53.931602 (+   289us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:54.189263 (+257661us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.231413 (+1042150us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:55.231466 (+    53us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:55.231516 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.231584 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.231719 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":722366,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":52,"thread_start_us":1290116,"threads_started":2}
I20260709 12:21:55.235036 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:55.235867 31674 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:55.186902 31642 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.000s	sys 0.016s
W20260709 12:21:55.237644 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 755427us
W20260709 12:21:55.238536 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:55.239076 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.242022 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:55.243067 31748 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:55.243772 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.244953 31749 raft_consensus.cc:2368] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8.
I20260709 12:21:55.041330  2570 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:55.213115 31823 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 7. (term mismatch)
I20260709 12:21:55.217950 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:55.246778 31818 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:21:55.246976 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:55.247874 31823 pending_rounds.cc:85] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 6
I20260709 12:21:55.248461 31823 pending_rounds.cc:107] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 8.7
I20260709 12:21:55.248811 31823 raft_consensus.cc:2889] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:55.206548  2234 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:55.251874 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.256318 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:55.220640 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.220417 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.222846 31824 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:55.118995 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.221520  2327 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:55.266004 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:55.266590 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:55.267594 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:55.268261 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.268843 31745 raft_consensus.cc:1813] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Handling vote request from an unknown peer 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:55.269439 31745 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:21:55.272575 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:55.273463 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.275487 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:55.275768 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:55.242353  2500 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:55.276077 31824 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:55.281615 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.275103 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:55.277622 31674 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:55.282714 31744 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:55.278821 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:55.284523 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:55.287213 32378 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:55.287971 31745 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:55.283859 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:55.295323 31745 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:21:55.296135 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:55.327695 31823 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:55.328616 31745 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:55.298137 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:55.298920 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 488759us
I20260709 12:21:55.258349  2477 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.250413  2527 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:55.273452 31747 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.018s	sys 0.022s
W20260709 12:21:55.342998 31747 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.679s	user 0.019s	sys 0.022s
I20260709 12:21:55.343297 31824 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:21:55.332826 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:55.275321  2327 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:55.360180 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:55.360858 31824 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:55.360181 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:55.357548 31823 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:55.362531 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:55.360842 31818 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:21:55.365061 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:55.329547 31745 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:55.367409 31745 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:21:55.367727 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 69693.4us
I20260709 12:21:55.381709 31824 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:21:55.359151  2440 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:55.366794  2130 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.297621  2613 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.342146  2477 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:55.383723  2649 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:55.384809 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.386056 31670 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:55.385417 31753 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:21:55.298509  2399 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:55.273344  2234 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:55.390177 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.390964 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:55.323429  2333 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.392338 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:55.393337  2605 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 12:21:55.399945  2605 consensus_queue.cc:1243] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:21:55.400000  2608 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:55.396428  2413 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.383613  2130 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:55.403118  2333 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:55.403249  2413 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:55.404821  2130 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:55.405215  2333 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:55.405540  2413 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:55.416035  2413 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:55.390478  2355 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:55.411119 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:21:55.423484 31751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.014s	sys 0.016s
I20260709 12:21:55.416493  2234 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:55.417493  2648 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:21:55.407469  2636 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:55.418692 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:55.354617  2108 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.017s	sys 0.021s
I20260709 12:21:55.421464 31753 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:55.412137  2643 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:55.357548 32378 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:55.424347 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:55.425314 31826 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:55.437201 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:55.426185 31751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.015s	sys 0.016s
I20260709 12:21:55.384665  2511 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:55.425736 31671 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.849s	user 0.013s	sys 0.028s
W20260709 12:21:55.439715 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:55.448537 31671 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.872s	user 0.013s	sys 0.029s
I20260709 12:21:55.383651  2613 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:55.440598 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:55.441802 31826 raft_consensus.cc:2410] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260709 12:21:55.439385 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 409133us
I20260709 12:21:55.425612  2333 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:21:55.465456 31745 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260709 12:21:55.436486  2253 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.011s	sys 0.021s
I20260709 12:21:55.423777  2654 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:21:55.441932  2499 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.007s	sys 0.018s
I20260709 12:21:55.471079 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 398) took 1462 ms. Trace:
I20260709 12:21:55.472632 31791 leader_election.cc:400] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11.
I20260709 12:21:55.445617  2476 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:55.479070 31671 rpcz_store.cc:276] 0709 12:21:54.003926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.496877 (+492951us) service_pool.cc:224] Handling call
0709 12:21:54.497239 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.497505 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:21:54.576419 (+ 78914us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.465969 (+889550us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:55.466046 (+    77us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:55.466118 (+    72us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.466214 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.466384 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58496,"prepare.queue_time_us":156833,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":16652,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":2395,"thread_start_us":872453,"threads_started":1}
W20260709 12:21:55.470019  2253 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.012s	sys 0.022s
I20260709 12:21:55.471151 31745 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:21:55.507954 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:55.508275 31671 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:21:55.395603  2649 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:55.463306  2636 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:55.484010 31791 leader_election.cc:403] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:55.425757  2413 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:55.469228  2651 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:21:55.490887 31672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.000s	sys 0.022s
W20260709 12:21:55.523123 31672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.000s	sys 0.022s
W20260709 12:21:55.478816  2499 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.007s	sys 0.019s
W20260709 12:21:55.513516 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.505632 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:55.519181 32374 pending_rounds.cc:85] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 5
I20260709 12:21:55.524334 31826 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:55.524514 32374 pending_rounds.cc:107] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260709 12:21:55.524919 32374 raft_consensus.cc:2889] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:55.502921  2498 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:21:55.525018 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.514902 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.537670 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:55.538743 31745 raft_consensus.cc:2368] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9.
W20260709 12:21:55.528764  2655 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:21:55.528999  2234 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11.
W20260709 12:21:55.531155  2652 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20260709 12:21:55.531733  2645 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
W20260709 12:21:55.528813 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 56342.1us
I20260709 12:21:55.533303 32372 raft_consensus.cc:1813] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Handling vote request from an unknown peer 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:55.515488  2511 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.499070  2654 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:55.542294 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1416) took 1625 ms. Trace:
I20260709 12:21:55.506914  2594 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:55.623273  2568 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:55.554306 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:55.623741  2568 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:55.617462  2393 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:55.624402 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 70211.3us
I20260709 12:21:55.624645  2568 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:55.625217 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:55.625762  2568 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:55.484395 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:55.556324 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:55.626482 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:55.629238 31826 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:55.630151 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:55.630795 31826 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
W20260709 12:21:55.630908 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 146633us
I20260709 12:21:55.631755 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:55.632354 31826 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:55.604085 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1415) took 1687 ms. Trace:
I20260709 12:21:55.634079 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:55.634546 31673 rpcz_store.cc:276] 0709 12:21:53.916286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.445800 (+529514us) service_pool.cc:224] Handling call
0709 12:21:54.498547 (+ 52747us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:54.498985 (+   438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.499026 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:54.499640 (+   614us) log.cc:844] Serialized 26 byte log entry
0709 12:21:54.899037 (+399397us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b4800080058. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:21:55.230526 (+331489us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:55.230593 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.230733 (+   140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:55.603746 (+373013us) raft_consensus.cc:1685] finished
0709 12:21:55.603771 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.604007 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23385,"raft.queue_time_us":68027,"raft.run_cpu_time_us":2281,"raft.run_wall_time_us":324889,"spinlock_wait_cycles":27613952,"thread_start_us":658176,"threads_started":1,"wal-append.queue_time_us":768939}
I20260709 12:21:55.635720 31791 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:55.637645 31642 leader_election.cc:400] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9.
I20260709 12:21:55.638193 31642 leader_election.cc:403] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:55.603636  2447 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:55.552412  2498 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:55.645445 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:55.657333 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.584384  2570 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:55.610122 32369 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:55.611052  2666 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:55.549724  2328 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:55.523361  2452 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:55.614181 31745 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:21:55.615046  2561 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:55.614223 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:55.618432  2477 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:55.636301  2234 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:21:55.725481 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:55.849726 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.856441 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:55.860495 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 246330us
I20260709 12:21:55.916965 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:55.530481  2658 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 12:21:55.863155 31792 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.041s	sys 0.079s
W20260709 12:21:55.887409 31819 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.646s	user 0.028s	sys 0.067s
W20260709 12:21:55.921304 31792 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.046s	sys 0.080s
W20260709 12:21:55.921454 31819 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.029s	sys 0.067s
I20260709 12:21:55.894860 31673 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 6. Current term is 7. Ops: [4.4-6.5]
I20260709 12:21:55.900161 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:55.622860 31672 rpcz_store.cc:276] 0709 12:21:53.916576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.471996 (+555420us) service_pool.cc:224] Handling call
0709 12:21:54.472360 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.472724 (+   364us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:54.950712 (+477988us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.541323 (+590611us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.541696 (+   373us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.541853 (+   157us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:55.541903 (+    50us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:55.541969 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.542066 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:55.542220 (+   154us) inbound_call.cc:177] Queueing success response
0709 12:21:55.621784 (+ 79564us) spinlock_profiling.cc:243] Waited 62.6 ms on lock 0x7b18003875e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:21:55.622891 (+  1107us) spinlock_profiling.cc:243] Waited 66.2 ms on lock 0x7b18003875e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
Metrics: {"mutex_wait_us":44832,"prepare.queue_time_us":269695,"prepare.run_cpu_time_us":128,"prepare.run_wall_time_us":132,"raft.queue_time_us":124992,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":21133,"spinlock_wait_cycles":328089600,"thread_start_us":977122,"threads_started":2}
I20260709 12:21:55.924489 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:55.626297  2393 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:55.928421 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:55.645144  2636 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:55.933560 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:55.863209  2655 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.306s	user 0.000s	sys 0.000s
I20260709 12:21:55.845775  2333 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9.
I20260709 12:21:55.940044 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:55.947011 31672 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:55.946875 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:55.941893 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:55.943774 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:55.953151 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:55.642961  2594 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.852917  2355 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:55.887442  2583 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:21:55.856174  2500 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.555s	user 0.010s	sys 0.000s
I20260709 12:21:55.928138  2476 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.048s
I20260709 12:21:55.930876  2477 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 12:21:55.963124  2498 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:55.964777  2108 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:21:55.965269  2389 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:55.970906  2327 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:55.973306 31672 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:21:56.009414  2646 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:55.922175  2452 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:55.909129  2570 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:55.640188  2511 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:55.938879  2636 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:56.010403 31747 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.061s	sys 0.072s
I20260709 12:21:55.959357  1897 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:21:55.984630  2651 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:55.961227 31826 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:55.974211  2355 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:55.996119  2639 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:55.957763 31673 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:21:55.997521  2296 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:21:56.039541 31753 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.053s	sys 0.077s
W20260709 12:21:56.138054 31753 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.670s	user 0.054s	sys 0.078s
I20260709 12:21:56.138170 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1426) took 1193 ms. Trace:
I20260709 12:21:55.845072  2527 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.182s
W20260709 12:21:56.042878 31822 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.065s	sys 0.176s
I20260709 12:21:56.016597  2333 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:56.069590 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.092819 31672 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
W20260709 12:21:56.107030 31747 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.763s	user 0.061s	sys 0.072s
I20260709 12:21:55.984993  2234 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260709 12:21:56.113631 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:56.043332  2613 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:55.974210  2594 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:56.139215 31822 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.662s	user 0.065s	sys 0.176s
I20260709 12:21:56.092458  2452 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.153694 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2006) took 1201 ms. Trace:
I20260709 12:21:56.197127 31672 rpcz_store.cc:276] 0709 12:21:54.952584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.092283 (+1139699us) service_pool.cc:224] Handling call
0709 12:21:56.092641 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.153242 (+ 60601us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.153614 (+   372us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.027477  2500 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:56.160432 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:56.207080 31826 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:56.033941  2393 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:56.152180  2655 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:21:56.169581 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:56.170810  2413 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:56.171464  2333 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.043916  2108 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:56.103870  2646 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:56.158209  2651 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:56.182476 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:56.094395  2399 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.110440  2636 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:56.183152  2583 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:56.162461  2639 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:56.179457 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 415) took 1517 ms. Trace:
I20260709 12:21:56.185848  2355 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:56.132385  2570 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:56.188593  2511 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:56.108174  2649 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:56.042757  2668 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:56.092242  2273 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.138772 31673 rpcz_store.cc:276] 0709 12:21:54.944608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.956831 (+1012223us) service_pool.cc:224] Handling call
0709 12:21:55.957266 (+   435us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.137783 (+180517us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.138093 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:56.179356  2662 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.241s	user 0.000s	sys 0.001s
I20260709 12:21:56.201242 31672 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:21:56.152807  1897 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:56.156731  2208 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:56.231549 31744 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.003s	sys 0.018s
I20260709 12:21:56.203649  2452 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:56.222918  2296 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:56.233171 32378 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.021s	sys 0.054s
I20260709 12:21:56.028124  2654 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:56.243106 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1904) took 1111 ms. Trace:
I20260709 12:21:56.295859 31822 rpcz_store.cc:276] 0709 12:21:55.131587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.135855 (+  4268us) service_pool.cc:224] Handling call
0709 12:21:55.217039 (+ 81184us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.217241 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:55.477412 (+260171us) log.cc:844] Serialized 14 byte log entry
0709 12:21:56.242733 (+765321us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:56.242787 (+    54us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:56.242832 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.242920 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.243042 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50271,"prepare.queue_time_us":312355,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":55653,"raft.run_wall_time_us":7094,"spinlock_wait_cycles":4797696,"thread_start_us":974431,"threads_started":2,"wal-append.queue_time_us":777932}
I20260709 12:21:56.243106 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 1982) took 2359 ms. Trace:
I20260709 12:21:56.230952  2327 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:56.254539 31744 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.004s	sys 0.018s
W20260709 12:21:56.241330  2253 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.011s	sys 0.046s
I20260709 12:21:56.237022 31747 rpcz_store.cc:276] 0709 12:21:54.661767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.662756 (+   989us) service_pool.cc:224] Handling call
0709 12:21:54.663038 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.663227 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:21:55.343632 (+680405us) log.cc:844] Serialized 14 byte log entry
0709 12:21:56.179063 (+835431us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:56.179129 (+    66us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:56.179187 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.179260 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.179394 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25580,"prepare.queue_time_us":772659,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":18433,"raft.queue_time_us":29525,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":88,"thread_start_us":1489356,"threads_started":2,"wal-append.queue_time_us":839588}
I20260709 12:21:56.241488  2583 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:56.235083  2568 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.608s	user 0.012s	sys 0.001s
I20260709 12:21:56.257368 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2009) took 1291 ms. Trace:
I20260709 12:21:56.302902 31826 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:21:56.247480  2671 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.248978  2234 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:56.253573  2447 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:56.240097  2399 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.291303 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:56.292661 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.320623 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:56.258339 32378 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.021s	sys 0.054s
I20260709 12:21:56.247614  2389 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:21:56.305546 31672 rpcz_store.cc:276] 0709 12:21:54.966259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.198258 (+1231999us) service_pool.cc:224] Handling call
0709 12:21:56.200913 (+  2655us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:56.256979 (+ 56066us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.257296 (+   317us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:56.253825  2619 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.125s	user 0.005s	sys 0.009s
I20260709 12:21:56.304942  2660 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:21:56.331676 31790 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.005s	sys 0.025s
W20260709 12:21:56.332237 31790 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.005s	sys 0.025s
I20260709 12:21:56.312804  2413 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:21:56.308589 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.309406 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:56.312703  2234 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:56.311468 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.339818 31822 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 6: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:56.340941 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:56.231336  2393 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:56.255633  2273 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.257769  2208 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.320832 31820 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:56.298601 31678 rpcz_store.cc:276] 0709 12:21:53.883731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:53.901714 (+ 17983us) service_pool.cc:224] Handling call
0709 12:21:53.902027 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:53.932557 (+ 30530us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:54.507624 (+575067us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.209610 (+701986us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:55.214934 (+  5324us) log.cc:844] Serialized 26 byte log entry
0709 12:21:55.215133 (+   199us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:55.219674 (+  4541us) log.cc:844] Serialized 14 byte log entry
0709 12:21:55.219970 (+   296us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:55.220045 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:56.242906 (+1022861us) raft_consensus.cc:1685] finished
0709 12:21:56.242924 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.243042 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11716,"prepare.queue_time_us":718363,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":64,"raft.queue_time_us":33328,"raft.run_cpu_time_us":400,"raft.run_wall_time_us":15166,"spinlock_wait_cycles":31676032,"thread_start_us":1132326,"threads_started":2,"wal-append.queue_time_us":879458}
W20260709 12:21:56.321657 31825 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.878s	user 0.005s	sys 0.025s
I20260709 12:21:56.291848  2511 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:21:56.307796  2253 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.011s	sys 0.046s
I20260709 12:21:56.267926  2296 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.326206 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 411) took 1312 ms. Trace:
I20260709 12:21:56.400630 31672 rpcz_store.cc:276] 0709 12:21:55.013921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.325329 (+1311408us) service_pool.cc:224] Handling call
0709 12:21:56.325663 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.325852 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:56.325874 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:56.325935 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:56.325970 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.326027 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.326150 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.333715  2555 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:56.269934  2551 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:56.326267  2447 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.337749 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.407183 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.355630  2333 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:56.318238  2108 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.358110  2583 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:56.316144  2651 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.358515  2273 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:56.317346  2671 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:56.359014 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:56.361001  2399 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:56.363572 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1524) took 1333 ms. Trace:
I20260709 12:21:56.415182 31749 rpcz_store.cc:276] 0709 12:21:55.030363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.031893 (+  1530us) service_pool.cc:224] Handling call
0709 12:21:56.363385 (+1331492us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54001edcd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:56.363513 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":283365248}
I20260709 12:21:56.365739  2554 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:56.320653  1897 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:56.367344  2678 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:21:56.368180  2234 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.341786  2413 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.370919 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:56.428264 31669 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
W20260709 12:21:56.429064 31669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1432) took 1433 ms (client timeout 1756 ms). Trace:
W20260709 12:21:56.429544 31669 rpcz_store.cc:269] 0709 12:21:54.995900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.370606 (+1374706us) service_pool.cc:224] Handling call
0709 12:21:56.428973 (+ 58367us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.375821 31827 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:21:56.431305 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1906) took 1299 ms. Trace:
I20260709 12:21:56.431792 31827 rpcz_store.cc:276] 0709 12:21:55.149900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.150203 (+   303us) service_pool.cc:224] Handling call
0709 12:21:56.375235 (+1225032us) spinlock_profiling.cc:243] Waited 2.33 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:56.375529 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.430948 (+ 55419us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.431229 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5357952}
I20260709 12:21:56.433286 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.434177 31827 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:21:56.377146 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 526) took 1719 ms. Trace:
I20260709 12:21:56.378515 31821 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:56.436079 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1905) took 1304 ms. Trace:
W20260709 12:21:56.380944 31825 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.006s	sys 0.026s
I20260709 12:21:56.438331 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1902) took 1307 ms. Trace:
I20260709 12:21:56.399938  2511 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:56.384527 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 405185us
I20260709 12:21:56.312584  2327 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:56.398531 31747 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:56.317095  2527 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.402438 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:56.404703 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.409722 31826 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:56.384203  2594 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:56.385021  2666 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:21:56.313310 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.676s	user 0.043s	sys 0.073s
W20260709 12:21:56.515865 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.879s	user 0.044s	sys 0.073s
W20260709 12:21:56.516281 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 880546us
I20260709 12:21:56.412740 31824 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:56.413559 31752 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:56.402339 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 475784us
I20260709 12:21:56.332403  2619 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:56.417574 31818 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.012s	sys 0.028s
W20260709 12:21:56.567178 31818 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.013s	sys 0.029s
I20260709 12:21:56.421556 31749 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:56.406293  2551 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:56.434808 32372 rpcz_store.cc:276] 0709 12:21:54.657269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.659417 (+  2148us) service_pool.cc:224] Handling call
0709 12:21:56.376806 (+1717389us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b54005947d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:56.377059 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1935957248}
I20260709 12:21:56.381096 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:56.569466 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1916) took 1355 ms. Trace:
I20260709 12:21:56.570144 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:56.436538 31821 rpcz_store.cc:276] 0709 12:21:55.131845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.135934 (+  4089us) service_pool.cc:224] Handling call
0709 12:21:56.377972 (+1242038us) spinlock_profiling.cc:243] Waited 2.77 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:56.378252 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.435728 (+ 57476us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.436006 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6369152}
I20260709 12:21:56.571115 31678 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:56.464377 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:56.510910 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2031) took 1412 ms. Trace:
I20260709 12:21:56.514470 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.515543 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:56.412765 31820 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:21:56.410665  2447 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:56.520066 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67884.6us
I20260709 12:21:56.546166 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2026) took 1485 ms. Trace:
I20260709 12:21:56.551137 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1534) took 1502 ms. Trace:
W20260709 12:21:56.550650 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:56.549957 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 66693.4us
I20260709 12:21:56.442421  2570 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:56.550287 31672 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:56.438839 31825 rpcz_store.cc:276] 0709 12:21:55.131046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.134592 (+  3546us) service_pool.cc:224] Handling call
0709 12:21:55.134899 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.135102 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:55.442256 (+307154us) log.cc:844] Serialized 14 byte log entry
0709 12:21:56.437825 (+995569us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:56.437911 (+    86us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:56.437991 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.438089 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.438252 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":394281,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"thread_start_us":1301950,"threads_started":2}
I20260709 12:21:56.447012  2639 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:56.589398 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:56.716372 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:21:56.643731 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.653967 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:56.585419 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2032) took 1481 ms. Trace:
I20260709 12:21:56.689010 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:56.792953 31749 rpcz_store.cc:276] 0709 12:21:55.104344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.420854 (+1316510us) service_pool.cc:224] Handling call
0709 12:21:56.421308 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.585020 (+163712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.585331 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.793661 31827 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:56.794266 31826 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:56.789728 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 200476us
I20260709 12:21:56.789578 32377 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
W20260709 12:21:56.795750 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.795852 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:56.571599 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:56.791733 31678 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:56.790897 31747 rpcz_store.cc:276] 0709 12:21:55.098218 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.342437 (+1244219us) service_pool.cc:224] Handling call
0709 12:21:56.398233 (+ 55796us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.456213 (+ 57980us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.510839 (+ 54626us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:56.797178 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 225765us
I20260709 12:21:56.795296 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:56.797981 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:56.791699 31752 rpcz_store.cc:276] 0709 12:21:55.048942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.180075 (+131133us) service_pool.cc:224] Handling call
0709 12:21:56.358847 (+1178772us) spinlock_profiling.cc:243] Waited 9.15 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:56.551074 (+192227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21049088}
I20260709 12:21:56.799037 31678 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:56.690023  2333 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.791273 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:21:56.803622 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:56.802702 31717 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:56.598048 31791 leader_election.cc:400] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8.
I20260709 12:21:56.738576 32376 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:21:56.760917 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:56.588248 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:56.812067 31820 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1909) took 1625 ms (client timeout 1613 ms). Trace:
I20260709 12:21:56.812402 32373 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
W20260709 12:21:56.805234 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:56.812518 31820 rpcz_store.cc:269] 0709 12:21:55.165300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.206123 (+ 40823us) service_pool.cc:224] Handling call
0709 12:21:56.273511 (+1067388us) spinlock_profiling.cc:243] Waited 5.8 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:56.790241 (+516730us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13331200}
W20260709 12:21:56.811408 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 213572us
I20260709 12:21:56.551959  2561 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:56.808105  2554 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:56.571470 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.547750  2689 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:56.819378 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:56.829054 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.820370 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.803771 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:21:56.801792  2691 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.214s	user 0.000s	sys 0.001s
I20260709 12:21:56.833981 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:56.836746 31672 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:21:56.647883  2253 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:56.830667 31826 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:21:56.726112  2696 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.838232 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:56.760821  2681 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.829793 31821 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:56.855645 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.856611 31821 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:56.857800 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:21:56.833070  2679 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.387s	user 0.000s	sys 0.001s
I20260709 12:21:56.810030 31753 rpcz_store.cc:276] 0709 12:21:55.060999 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.439569 (+378570us) service_pool.cc:224] Handling call
0709 12:21:55.439926 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.440212 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:56.138622 (+698410us) log.cc:844] Serialized 14 byte log entry
0709 12:21:56.482402 (+343780us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:56.545611 (+ 63209us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:56.545705 (+    94us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.545830 (+   125us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.546072 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3515,"prepare.queue_time_us":74560,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":6687,"raft.queue_time_us":18614,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":35402,"thread_start_us":1013232,"threads_started":2,"wal-append.queue_time_us":509348}
I20260709 12:21:56.841073 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:56.671439  2499 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.164s
I20260709 12:21:56.849902 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.846683 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:56.849974 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.869791 31826 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:56.811849  2639 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.814100  2399 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.817616  2570 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:21:56.547664  2527 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.891271 31749 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:56.864961 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:56.848182 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:56.851282  2325 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.870802 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.831357  2208 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:56.911484 31826 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:56.876430 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:56.910416 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 112427us
I20260709 12:21:56.915350 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1929) took 1315 ms. Trace:
I20260709 12:21:56.915723 31821 rpcz_store.cc:276] 0709 12:21:55.599582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.876016 (+1276434us) service_pool.cc:224] Handling call
0709 12:21:56.915285 (+ 39269us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.696768  2682 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.165s
I20260709 12:21:56.835342  2583 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:21:56.852240  2452 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.584s	user 0.007s	sys 0.003s
I20260709 12:21:56.566004  2666 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:56.852473  2683 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.350s	user 0.000s	sys 0.001s
I20260709 12:21:56.646123  2636 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.905462 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:56.908569 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:56.932420 31825 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 10 because replica has last-logged OpId of term: 6 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:21:56.836624  2694 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:56.862957 31672 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:21:56.935130 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 96789.6us
I20260709 12:21:56.911960 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 529) took 1014 ms. Trace:
I20260709 12:21:56.889322 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 410) took 1984 ms. Trace:
I20260709 12:21:56.935904 32375 rpcz_store.cc:276] 0709 12:21:55.897100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.898393 (+  1293us) service_pool.cc:224] Handling call
0709 12:21:55.898927 (+   534us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:55.899280 (+   353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.899337 (+    57us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:55.923395 (+ 24058us) log.cc:844] Serialized 26 byte log entry
0709 12:21:56.293117 (+369722us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:56.293155 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.293224 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:56.911756 (+618532us) raft_consensus.cc:1685] finished
0709 12:21:56.911778 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.911905 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":305895,"threads_started":1,"wal-append.queue_time_us":483923}
I20260709 12:21:56.834789  2273 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:56.937454 31677 rpcz_store.cc:276] 0709 12:21:54.904639 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.531354 (+626715us) service_pool.cc:224] Handling call
0709 12:21:55.531739 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:55.550582 (+ 18843us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:55.583602 (+ 33020us) log.cc:844] Serialized 14 byte log entry
0709 12:21:56.325848 (+742246us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:56.329092 (+  3244us) log.cc:844] Serialized 26 byte log entry
0709 12:21:56.329285 (+   193us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:56.329321 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.329388 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:56.888963 (+559575us) raft_consensus.cc:1685] finished
0709 12:21:56.888996 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.889225 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18365,"prepare.queue_time_us":48824,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"raft.queue_time_us":67211,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":4105,"thread_start_us":417904,"threads_started":1,"wal-append.queue_time_us":704817}
I20260709 12:21:56.940514 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:56.817085  2333 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:56.839804  2689 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.920908 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:56.911159 31820 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:56.860183  2253 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:56.833280  2685 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:56.947645 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2091) took 1025 ms. Trace:
I20260709 12:21:56.948101 31820 rpcz_store.cc:276] 0709 12:21:55.922752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.847254 (+924502us) service_pool.cc:224] Handling call
0709 12:21:56.947584 (+100330us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.933419 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1921) took 1508 ms. Trace:
I20260709 12:21:56.934140 31678 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:56.948944 31821 rpcz_store.cc:276] 0709 12:21:55.424651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.932473 (+1507822us) service_pool.cc:224] Handling call
0709 12:21:56.932795 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.933002 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:56.933041 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:56.933093 (+    52us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:56.933135 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.933197 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.933309 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.949584 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:56.950295 31678 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:56.950464 31821 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 5. (term mismatch)
I20260709 12:21:56.950842 31821 pending_rounds.cc:85] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 4
I20260709 12:21:56.951052 31821 pending_rounds.cc:107] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 7.5
I20260709 12:21:56.951272 31821 raft_consensus.cc:2889] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:56.951195 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:56.877144  2393 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:56.936872 31669 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
W20260709 12:21:56.916640 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 78903.5us
I20260709 12:21:56.959295 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:56.934432 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:56.938529 31826 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:21:56.961797 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1920) took 1569 ms. Trace:
I20260709 12:21:56.932468 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2012) took 1927 ms. Trace:
I20260709 12:21:56.942113 31677 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:56.888108  2702 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:56.962348 31826 rpcz_store.cc:276] 0709 12:21:55.392241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.929613 (+1537372us) service_pool.cc:224] Handling call
0709 12:21:56.929967 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.961439 (+ 31472us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.961721 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.962952 31673 rpcz_store.cc:276] 0709 12:21:55.005049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.249358 (+1244309us) service_pool.cc:224] Handling call
0709 12:21:56.249820 (+   462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.250076 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:21:56.471465 (+221389us) log.cc:844] Serialized 14 byte log entry
0709 12:21:56.932113 (+460648us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:56.932170 (+    57us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:56.932221 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.932288 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.932406 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":550952,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":45,"raft.queue_time_us":1966,"thread_start_us":584912,"threads_started":2}
I20260709 12:21:56.964376 31826 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: [6.5-6.5]
I20260709 12:21:56.965396 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2096) took 1041 ms. Trace:
I20260709 12:21:56.841341  2555 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.067s
I20260709 12:21:56.951997 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2095) took 1028 ms. Trace:
I20260709 12:21:56.956669 31672 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:21:56.965332 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:56.859328  2700 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.105s
W20260709 12:21:56.925048 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 90958.8us
I20260709 12:21:56.977476 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:56.978300 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:56.978682 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:56.966111 31826 rpcz_store.cc:276] 0709 12:21:55.924211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.963539 (+1039328us) service_pool.cc:224] Handling call
0709 12:21:56.964028 (+   489us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:56.965021 (+   993us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.965322 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.968350 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:56.969137 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:56.980299 31672 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:56.972304 31717 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:56.981410 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:56.967939 31821 rpcz_store.cc:276] 0709 12:21:55.923948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.949851 (+1025903us) service_pool.cc:224] Handling call
0709 12:21:56.950155 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:56.951790 (+  1635us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.951953 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.981663 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:56.944654  2273 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.875737  2681 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:56.976468 31825 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:57.010955  2273 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:57.011432 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2094) took 1087 ms. Trace:
I20260709 12:21:57.012027 31825 rpcz_store.cc:276] 0709 12:21:55.923653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.948277 (+1024624us) service_pool.cc:224] Handling call
0709 12:21:56.976282 (+ 28005us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.011134 (+ 34852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.011369 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.012584  2681 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:57.013177  2681 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:56.976596 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:57.014597 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:57.015789 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:57.016863 31827 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:56.959795  2675 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.283s	user 0.001s	sys 0.001s
I20260709 12:21:56.908334  2639 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.934834  2694 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:56.985981 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:57.027585 31677 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:56.957710  2327 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:56.986923  2333 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:56.989135  2636 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:56.989557  2690 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:56.924835  2325 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:56.990903  2706 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:56.992266  2710 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:56.993570 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:56.996845  2705 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:57.036525 31827 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:21:56.999480 31821 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260709 12:21:57.039674 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1533) took 1991 ms. Trace:
I20260709 12:21:56.999363  2253 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:56.947304  2499 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:57.008724  2689 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:56.960995  2671 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:56.866748  2696 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:57.010161  2693 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
I20260709 12:21:57.062605  2696 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:57.062700  2639 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:57.063637  2447 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:57.030519  2694 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:57.074913  2554 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:57.019713 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:56.945029  2413 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:57.021626 31826 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: [5.3-9.5]
W20260709 12:21:57.079542 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59975.2us
I20260709 12:21:57.079842 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2097) took 1155 ms. Trace:
W20260709 12:21:57.023324 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51103.9us
I20260709 12:21:57.080402 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:57.080420 31826 rpcz_store.cc:276] 0709 12:21:55.924474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.020667 (+1096193us) service_pool.cc:224] Handling call
0709 12:21:57.021220 (+   553us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:21:57.079389 (+ 58169us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.079765 (+   376us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.082069 31826 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:57.083814 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:57.086064 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1430) took 2136 ms. Trace:
I20260709 12:21:57.086582 31670 rpcz_store.cc:276] 0709 12:21:54.949850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.324766 (+374916us) service_pool.cc:224] Handling call
0709 12:21:57.085755 (+1760989us) spinlock_profiling.cc:243] Waited 24.3 ms on lock 0x7b54000474d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:57.085972 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55884160}
I20260709 12:21:57.088049 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 409) took 2183 ms. Trace:
I20260709 12:21:57.088227 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:57.089085 31670 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:57.088666 31671 rpcz_store.cc:276] 0709 12:21:54.904348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.509186 (+604838us) service_pool.cc:224] Handling call
0709 12:21:57.086016 (+1576830us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.086325 (+   309us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.086363 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.086428 (+    65us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:57.086487 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.086596 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.087855 (+  1259us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b54000474d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:57.087984 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3625416064}
I20260709 12:21:57.024648 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.091071 31749 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:21:57.091697 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:57.092490 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.094188 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 412) took 2072 ms. Trace:
I20260709 12:21:57.094746 31671 rpcz_store.cc:276] 0709 12:21:55.022324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.093209 (+2070885us) service_pool.cc:224] Handling call
0709 12:21:57.093503 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.093738 (+   235us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.093778 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.093853 (+    75us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:57.093902 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.093976 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.094118 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.095938 31676 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:21:57.096836 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 413) took 1252 ms. Trace:
I20260709 12:21:57.097188 31671 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:21:57.097267 31676 rpcz_store.cc:276] 0709 12:21:55.844236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.094653 (+1250417us) service_pool.cc:224] Handling call
0709 12:21:57.095594 (+   941us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.096500 (+   906us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.096763 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.098325 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 414) took 1253 ms. Trace:
I20260709 12:21:57.099309 31671 rpcz_store.cc:276] 0709 12:21:55.844631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.096495 (+1251864us) service_pool.cc:224] Handling call
0709 12:21:57.096887 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.097995 (+  1108us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.098255 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.101547 31671 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:57.102377 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 415) took 1257 ms. Trace:
I20260709 12:21:56.889473  2551 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.102806 31671 rpcz_store.cc:276] 0709 12:21:55.844927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.100914 (+1255987us) service_pool.cc:224] Handling call
0709 12:21:57.101311 (+   397us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.102057 (+   746us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.102310 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.103355  2551 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:57.105289 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 417) took 1246 ms. Trace:
I20260709 12:21:57.105752 31671 rpcz_store.cc:276] 0709 12:21:55.858985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.104071 (+1245086us) service_pool.cc:224] Handling call
0709 12:21:57.104476 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.104853 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.104890 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.104943 (+    53us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:57.104994 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.105071 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.105217 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:56.922966  2399 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:56.986497  2699 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:21:56.933794  2452 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.029435  2570 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:57.045766  2705 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.003074  2527 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:57.042593 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2099) took 1117 ms. Trace:
I20260709 12:21:57.042399 31744 rpcz_store.cc:276] 0709 12:21:55.048716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.283870 (+235154us) service_pool.cc:224] Handling call
0709 12:21:55.284797 (+   927us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:55.359520 (+ 74723us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.359552 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:55.384441 (+ 24889us) log.cc:844] Serialized 52 byte log entry
0709 12:21:55.706447 (+322006us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b18004bee60. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:21:56.312912 (+606465us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:56.334562 (+ 21650us) log.cc:844] Serialized 14 byte log entry
0709 12:21:56.334896 (+   334us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.334996 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:57.032087 (+697091us) raft_consensus.cc:1685] finished
0709 12:21:57.039442 (+  7355us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.039609 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":73020,"prepare.queue_time_us":24800,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":67,"raft.queue_time_us":85469,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":11096,"spinlock_wait_cycles":56567808,"thread_start_us":597083,"threads_started":1,"wal-append.queue_time_us":1008483}
I20260709 12:21:57.024197  2555 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:57.025636  2498 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:21:57.030824  2687 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.377s	user 0.000s	sys 0.002s
W20260709 12:21:57.055553 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 107800us
I20260709 12:21:57.057471  2619 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:57.056546 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:57.027457  2709 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.150717 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:57.150980 31678 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:57.116494 31676 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:21:57.119410 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2101) took 1152 ms. Trace:
W20260709 12:21:57.117216 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 80227.7us
I20260709 12:21:57.038662 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.045446  2681 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:21:57.151826 31826 rpcz_store.cc:276] 0709 12:21:55.967457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.081696 (+1114239us) service_pool.cc:224] Handling call
0709 12:21:57.081944 (+   248us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.082375 (+   431us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.119311 (+ 36936us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.142880 31821 rpcz_store.cc:276] 0709 12:21:55.925084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.998725 (+1073641us) service_pool.cc:224] Handling call
0709 12:21:56.999095 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.042183 (+ 43088us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.042514 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.153203 31747 raft_consensus.cc:2410] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 10 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:21:56.961165  2685 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.155050 31822 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:57.047935  2706 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:57.058876  2702 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:57.048430  2499 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:57.144767 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.056425 31673 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
W20260709 12:21:57.147284 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66970.2us
I20260709 12:21:57.153968 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:57.123126  2476 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:57.156378 31749 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:57.148723  2707 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:57.157676 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.186563 31745 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:21:57.187253 31747 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 5: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:57.173050 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:57.175591 31744 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:57.174616 32373 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 12:21:57.140102  2705 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:57.192399 32372 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:57.157420 31670 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
W20260709 12:21:57.182680 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 98948.8us
I20260709 12:21:57.036154  2690 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.137290  2253 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.035s
I20260709 12:21:57.196650 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:57.196895 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.188151 32373 pending_rounds.cc:85] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 7
I20260709 12:21:57.143080  2413 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260709 12:21:57.194883 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:57.197736 32373 pending_rounds.cc:107] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 12:21:57.200098 32373 raft_consensus.cc:2889] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:57.043650  2710 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:57.159029  2555 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:21:57.198599 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:57.206805 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:57.208928 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:57.211861 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.213269  2499 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:57.214572  2499 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:57.215471  2671 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:57.215921  2671 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:57.216305  2671 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:57.224808 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 513) took 1706 ms. Trace:
I20260709 12:21:57.224874 31826 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:57.194885  2707 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:57.232052 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:57.180666  2681 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 12:21:57.225472 32374 rpcz_store.cc:276] 0709 12:21:55.518129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.518400 (+   271us) service_pool.cc:224] Handling call
0709 12:21:55.518896 (+   496us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:55.525981 (+  7085us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:55.526028 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:55.572879 (+ 46851us) log.cc:844] Serialized 52 byte log entry
0709 12:21:56.304848 (+731969us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:56.699193 (+394345us) log.cc:844] Serialized 14 byte log entry
0709 12:21:56.699493 (+   300us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.699574 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:57.224524 (+524950us) raft_consensus.cc:1685] finished
0709 12:21:57.224542 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.224707 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":77536,"prepare.run_cpu_time_us":68,"prepare.run_wall_time_us":69,"raft.queue_time_us":13525,"raft.run_cpu_time_us":385,"raft.run_wall_time_us":36674,"thread_start_us":770693,"threads_started":2,"wal-append.queue_time_us":575528}
I20260709 12:21:57.213970  2706 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.203490  1897 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 12:21:57.236601  2685 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:57.237185  2690 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:57.205185  2476 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:57.212090  2711 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:57.238240  2710 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:57.225181  2682 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:21:57.229615  2702 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:21:57.280305 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 71464.3us
I20260709 12:21:57.271873  2554 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:57.274367  2717 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:57.275413  2452 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:57.275321  2666 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:57.276695  2713 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.260393 31744 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 8. Ops: []
W20260709 12:21:57.281407 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 45647.6us
I20260709 12:21:57.293643 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2033) took 2114 ms. Trace:
I20260709 12:21:57.235277  2707 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:57.231673  2715 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:21:57.289420 31791 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.786s (SENT) [suppressed 10 similar messages]
W20260709 12:21:57.289817 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 130116us
I20260709 12:21:57.294011 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:57.294039 31744 rpcz_store.cc:276] 0709 12:21:55.178860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.255487 (+2076627us) service_pool.cc:224] Handling call
0709 12:21:57.260148 (+  4661us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.293351 (+ 33203us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.293571 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.303452  2399 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:57.255625  2498 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:57.309592 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.164179  2709 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.256497  2711 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:21:57.313235 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.282398  2474 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:57.282539  2554 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.288622  2717 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:57.219023  2570 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:57.278124  2476 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:57.427755 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:57.442492 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.352275 31744 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:57.338678 31750 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:57.443637 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2035) took 2229 ms. Trace:
I20260709 12:21:57.444075 31744 rpcz_store.cc:276] 0709 12:21:55.214267 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.310471 (+2096204us) service_pool.cc:224] Handling call
0709 12:21:57.352027 (+ 41556us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.443224 (+ 91197us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.443573 (+   349us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:57.447170 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 136032us
I20260709 12:21:57.291023  2702 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:57.449261 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:57.452427 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 186910us
W20260709 12:21:57.453392 31750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1539) took 2393 ms (client timeout 2186 ms). Trace:
W20260709 12:21:57.453995 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 108080us
I20260709 12:21:57.455583 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:57.455329 31750 rpcz_store.cc:269] 0709 12:21:55.050103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.129660 (+ 79557us) service_pool.cc:224] Handling call
0709 12:21:57.309430 (+2179770us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:57.443853 (+134423us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28504832}
I20260709 12:21:57.461750 31750 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:21:57.459214 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:57.462707 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2038) took 2226 ms. Trace:
W20260709 12:21:57.356931  2668 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.102s	user 0.008s	sys 0.006s
I20260709 12:21:57.463306 31750 rpcz_store.cc:276] 0709 12:21:55.236545 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.460322 (+2223777us) service_pool.cc:224] Handling call
0709 12:21:57.460649 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.462372 (+  1723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.462622 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.371881  2718 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:57.471048 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:57.445318  2474 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:57.445542  2554 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:57.342573  2715 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:57.515463  2715 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:57.440773  2705 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:57.472926  2681 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:57.367211  2499 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.125s
I20260709 12:21:57.447024  2711 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:57.467088  1897 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:57.461813  2452 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:57.353156  2399 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.120s
I20260709 12:21:57.505698 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:57.553022 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47470.2us
I20260709 12:21:57.554924 32377 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 12:21:57.470541  2668 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:57.337757  2273 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:57.509027 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 215076us
I20260709 12:21:57.370893  2727 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.143s
I20260709 12:21:57.550055  2499 consensus_queue.cc:1243] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:21:57.561146 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:57.558472  1897 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:57.525552 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:57.375942  2498 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.678795  2707 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.531356  2476 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:57.531946 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.311391  2713 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:57.376633  2694 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:57.686288  2695 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.739s	user 0.004s	sys 0.004s
I20260709 12:21:57.685598 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.687791  2694 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:57.535074 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2100) took 1609 ms. Trace:
I20260709 12:21:57.688550 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.689596  2713 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:57.537092  2709 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:57.507231  2682 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:57.542260 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2102) took 1574 ms. Trace:
I20260709 12:21:57.688884 31825 rpcz_store.cc:276] 0709 12:21:55.925370 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.012829 (+1087459us) service_pool.cc:224] Handling call
0709 12:21:57.013190 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.013463 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:57.079184 (+ 65721us) log.cc:844] Serialized 14 byte log entry
0709 12:21:57.284497 (+205313us) log.cc:844] Serialized 14 byte log entry
0709 12:21:57.284814 (+   317us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:57.285405 (+   591us) log.cc:844] Serialized 26 byte log entry
0709 12:21:57.285625 (+   220us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:57.312351 (+ 26726us) log.cc:844] Serialized 14 byte log entry
0709 12:21:57.312665 (+   314us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.312758 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:57.534762 (+222004us) raft_consensus.cc:1685] finished
0709 12:21:57.534793 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.534992 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27824,"prepare.queue_time_us":130307,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":95,"raft.queue_time_us":30409,"raft.run_cpu_time_us":1576,"raft.run_wall_time_us":26850,"spinlock_wait_cycles":14989056,"thread_start_us":189832,"threads_started":2,"wal-append.queue_time_us":306143}
I20260709 12:21:57.690464 31826 rpcz_store.cc:276] 0709 12:21:55.967859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.152764 (+1184905us) service_pool.cc:224] Handling call
0709 12:21:57.153062 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.229242 (+ 76180us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.541944 (+312702us) spinlock_profiling.cc:243] Waited 71.5 ms on lock 0x7b540043dfd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:57.542180 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":164526336}
I20260709 12:21:57.691382 31825 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:21:57.691730 31826 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:57.692015 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 393) took 1358 ms. Trace:
I20260709 12:21:57.689860 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.692286 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 394) took 1358 ms. Trace:
I20260709 12:21:57.692346 31825 rpcz_store.cc:276] 0709 12:21:56.333428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.690824 (+1357396us) service_pool.cc:224] Handling call
0709 12:21:57.691142 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.691753 (+   611us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.691961 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.692764 31826 rpcz_store.cc:276] 0709 12:21:56.333724 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.691329 (+1357605us) service_pool.cc:224] Handling call
0709 12:21:57.691582 (+   253us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.692078 (+   496us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.692237 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.693404 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:57.661010  2464 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 2.410s	user 0.005s	sys 0.055s
I20260709 12:21:57.520586  2717 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:57.694104  2464 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 2.450s	user 0.006s	sys 0.056s
I20260709 12:21:57.694670  2717 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:57.695017  2464 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:57.695536 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.696143  2464 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.697024 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 392) took 1364 ms. Trace:
I20260709 12:21:57.509434  2729 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 12:21:57.693895 31826 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
W20260709 12:21:57.700006 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.574781  2452 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:57.565209 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:57.559191  2681 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:57.614421 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2014) took 1252 ms. Trace:
I20260709 12:21:57.613770 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:57.552522  2474 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:57.623111 32371 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:57.627048  2273 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:57.629017 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2098) took 1704 ms (client timeout 1811 ms). Trace:
I20260709 12:21:57.629674  2554 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.632028 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:57.633338 31824 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1932) took 2032 ms (client timeout 1801 ms). Trace:
W20260709 12:21:57.634298 31827 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2103) took 1666 ms (client timeout 2189 ms). Trace:
I20260709 12:21:57.643571 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:57.662331 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:57.676385 31670 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:21:57.677109 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:57.527647  2702 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:57.640080  2444 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.682999 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 416) took 2998 ms. Trace:
W20260709 12:21:57.684034  2735 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:21:57.699245 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.493973 31750 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260709 12:21:57.700387 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 396) took 1366 ms. Trace:
W20260709 12:21:57.799628 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.799542 31826 rpcz_store.cc:276] 0709 12:21:56.334309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.693520 (+1359211us) service_pool.cc:224] Handling call
0709 12:21:57.693738 (+   218us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.700139 (+  6401us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.700333 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:57.801039 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 236430us
I20260709 12:21:57.801450 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 397) took 1466 ms. Trace:
W20260709 12:21:57.800137 31822 rpcz_store.cc:269] 0709 12:21:55.924765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.343299 (+418534us) service_pool.cc:224] Handling call
0709 12:21:57.628792 (+1285493us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b5400447fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:57.628947 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1389780736}
I20260709 12:21:57.801936 31826 rpcz_store.cc:276] 0709 12:21:56.334587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.800607 (+1466020us) service_pool.cc:224] Handling call
0709 12:21:57.800933 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.801131 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.801158 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.801198 (+    40us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:57.801235 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.801287 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.801398 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.803881 31751 rpcz_store.cc:276] 0709 12:21:54.684533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.684781 (+   248us) service_pool.cc:224] Handling call
0709 12:21:54.702255 (+ 17474us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:54.920143 (+217888us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:54.920173 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:54.920805 (+   632us) log.cc:844] Serialized 52 byte log entry
0709 12:21:55.442069 (+521264us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:55.442169 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.369177 (+927008us) spinlock_profiling.cc:243] Waited 217 ms on lock 0x7b54004038d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:56.369330 (+   153us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:57.681251 (+1311921us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b24002a9c98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9b6e9c4 00007fe8f9b6cacb 00007fe8f9d27dc9 00007fe8f9d27a55 00007fe8f9d26fb0 00007fe8fc99d488 00007fe8fc99dbe8 00007fe8fc975398 00007fe8fc97076a 00007fe8fc96fbb6
0709 12:21:57.682715 (+  1464us) raft_consensus.cc:1685] finished
0709 12:21:57.682741 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.682929 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1123,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"spinlock_wait_cycles":997577728,"thread_start_us":520726,"threads_started":1,"wal-append.queue_time_us":635180}
I20260709 12:21:57.803295 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 399) took 1468 ms. Trace:
I20260709 12:21:57.802953 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:57.889424 31822 rpcz_store.cc:276] 0709 12:21:56.335158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.802422 (+1467264us) service_pool.cc:224] Handling call
0709 12:21:57.802766 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.802949 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.802979 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.803021 (+    42us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:57.803052 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.803110 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.803236 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.890666  2694 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:57.890847  2709 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:57.891146  2694 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:57.891520 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 88642.1us
I20260709 12:21:57.891403  2709 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.892429  2709 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:21:57.893054 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 279401us
I20260709 12:21:57.891393 31822 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:57.894450  2476 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:57.896576  2732 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260709 12:21:57.800199 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:57.898281 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:57.814342 31827 rpcz_store.cc:269] 0709 12:21:55.968230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.795039 (+826809us) service_pool.cc:224] Handling call
0709 12:21:57.634125 (+839086us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b5400197dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:57.634239 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":502048512}
I20260709 12:21:57.898540 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:57.899382 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:21:57.813313 31824 rpcz_store.cc:269] 0709 12:21:55.600521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.182205 (+581684us) service_pool.cc:224] Handling call
0709 12:21:57.633087 (+1450882us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b54000fc654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:57.633250 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":408015744}
I20260709 12:21:57.887017 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:57.901225 31677 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:21:57.901386 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1934) took 1495 ms. Trace:
I20260709 12:21:57.902201  2682 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:21:57.888469 32378 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:21:57.697484  2646 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:57.901954 31824 rpcz_store.cc:276] 0709 12:21:56.406449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.900371 (+1493922us) service_pool.cc:224] Handling call
0709 12:21:57.900673 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.900901 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.900945 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.901010 (+    65us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:57.901062 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.901145 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.901312 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.903206  2646 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.903748  2646 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.901978 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.905797  2646 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:57.905963  2670 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:57.800437  2452 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.906488  2670 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:57.906916  2452 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:57.907538  2670 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.908663  2670 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:57.782980  2707 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:57.768594  2705 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:57.905913 31746 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:57.891687  2694 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:21:57.915427 31669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2019) took 1552 ms (client timeout 1801 ms). Trace:
W20260709 12:21:57.915899 31669 rpcz_store.cc:269] 0709 12:21:56.363064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.463890 (+100826us) service_pool.cc:224] Handling call
0709 12:21:57.915251 (+1451361us) spinlock_profiling.cc:243] Waited 40.5 ms on lock 0x7b54003917d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:57.915361 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":93115136}
I20260709 12:21:57.917392 31669 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260709 12:21:57.895243 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 401) took 1559 ms. Trace:
I20260709 12:21:57.783640  1897 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.918849 31822 rpcz_store.cc:276] 0709 12:21:56.335715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.890855 (+1555140us) service_pool.cc:224] Handling call
0709 12:21:57.891194 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.895034 (+  3840us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.895190 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.920336 31822 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260709 12:21:57.700122  2551 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.596s	user 0.016s	sys 0.007s
I20260709 12:21:57.921135 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1942) took 1368 ms. Trace:
I20260709 12:21:57.921378 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.921723 31822 rpcz_store.cc:276] 0709 12:21:56.553247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.919690 (+1366443us) service_pool.cc:224] Handling call
0709 12:21:57.919980 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.920883 (+   903us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.921066 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.890157  2471 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:57.899901 31827 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:21:57.931140 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1933) took 1528 ms. Trace:
I20260709 12:21:57.931720 31827 rpcz_store.cc:276] 0709 12:21:56.402861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.899335 (+1496474us) service_pool.cc:224] Handling call
0709 12:21:57.899607 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.930820 (+ 31213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.931075 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:57.935294 31672 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:57.938290 31826 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 9. Current term is 10. Ops: []
I20260709 12:21:57.888077 32374 raft_consensus.cc:2410] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:21:57.889760 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.906375  2646 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:21:57.923100 31746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1529) took 2877 ms (client timeout 2277 ms). Trace:
W20260709 12:21:57.942004 31746 rpcz_store.cc:269] 0709 12:21:55.045623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.139538 (+ 93915us) service_pool.cc:224] Handling call
0709 12:21:57.806056 (+2666518us) spinlock_profiling.cc:243] Waited 81.5 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:21:57.923028 (+116972us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":187382528}
I20260709 12:21:57.942495 31670 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:57.702170 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:57.697412 31821 rpcz_store.cc:276] 0709 12:21:56.333079 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.152885 (+819806us) service_pool.cc:224] Handling call
0709 12:21:57.153213 (+   328us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:57.153615 (+   402us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.153646 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:57.160999 (+  7353us) log.cc:844] Serialized 52 byte log entry
0709 12:21:57.279626 (+118627us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:21:57.666495 (+386869us) log.cc:844] Serialized 14 byte log entry
0709 12:21:57.667076 (+   581us) log.cc:844] Serialized 14 byte log entry
0709 12:21:57.675743 (+  8667us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.675941 (+   198us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:57.696787 (+ 20846us) raft_consensus.cc:1685] finished
0709 12:21:57.696815 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.696969 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17360,"prepare.queue_time_us":426594,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":21412,"raft.run_cpu_time_us":445,"raft.run_wall_time_us":22859,"spinlock_wait_cycles":1920,"thread_start_us":427935,"threads_started":2,"wal-append.queue_time_us":114189}
I20260709 12:21:57.928987 31747 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:57.831283 31671 rpcz_store.cc:276] 0709 12:21:56.361682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.107297 (+745615us) service_pool.cc:224] Handling call
0709 12:21:57.107862 (+   565us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.108262 (+   400us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.108297 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:57.108842 (+   545us) log.cc:844] Serialized 26 byte log entry
0709 12:21:57.282745 (+173903us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:57.282800 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.282868 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:57.614045 (+331177us) raft_consensus.cc:1685] finished
0709 12:21:57.614075 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.614255 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2791,"thread_start_us":159783,"threads_started":1,"wal-append.queue_time_us":358121}
I20260709 12:21:57.945619 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:57.939939 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:57.941437 31751 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:57.941536 31744 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 8. Ops: []
W20260709 12:21:57.944321 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 242308us
I20260709 12:21:57.944413 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 395) took 1610 ms. Trace:
I20260709 12:21:57.944232 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:57.950580 31746 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:57.955794 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:57.949004 31825 rpcz_store.cc:276] 0709 12:21:56.334011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.693271 (+1359260us) service_pool.cc:224] Handling call
0709 12:21:57.693522 (+   251us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.693717 (+   195us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:21:57.694047 (+   330us) log.cc:844] Serialized 14 byte log entry
0709 12:21:57.944089 (+250042us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:57.944140 (+    51us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:21:57.944190 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.944248 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.944353 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6321,"prepare.queue_time_us":180,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":50,"thread_start_us":243333,"threads_started":1}
I20260709 12:21:57.958168 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.921095  2718 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.958958 31751 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:21:57.958130 31671 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 9. Current term is 10. Ops: []
I20260709 12:21:57.920843  2325 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:57.768013  2399 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:57.937836  2681 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:57.930706  2471 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:21:57.961653  2671 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.745s	user 0.014s	sys 0.000s
I20260709 12:21:57.999445  2671 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.001832  2671 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:57.979287 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1438) took 1144 ms. Trace:
I20260709 12:21:57.930855  2444 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.003401 31676 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:21:58.002558 31678 rpcz_store.cc:276] 0709 12:21:56.834697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.176535 (+341838us) service_pool.cc:224] Handling call
0709 12:21:57.200103 (+ 23568us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.200434 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.200462 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:57.205412 (+  4950us) log.cc:844] Serialized 26 byte log entry
0709 12:21:57.451835 (+246423us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:57.451890 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:57.451981 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:57.979072 (+527091us) raft_consensus.cc:1685] finished
0709 12:21:57.979094 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:57.979226 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":577,"thread_start_us":213225,"threads_started":1,"wal-append.queue_time_us":320042}
I20260709 12:21:57.972259 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:57.950934  2474 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:57.967540 31746 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:21:57.960340  2273 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:21:57.960179  2707 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:21:57.992142 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:57.996798  2711 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:57.952920  2570 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:58.046759  2471 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:57.957609  2646 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:58.047925  2471 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:58.049371  2646 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:58.004370  2444 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:58.044504 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:58.051069 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:58.051458 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:57.952251  2551 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:58.053510 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:58.050482 32371 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 8. (term mismatch)
I20260709 12:21:58.056061 32371 pending_rounds.cc:85] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 7
I20260709 12:21:58.056413 32371 pending_rounds.cc:107] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 7.8
I20260709 12:21:58.056748 32371 raft_consensus.cc:2889] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:21:58.007117 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:58.065413 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 681) took 3344 ms. Trace:
W20260709 12:21:58.065958 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58990.3us
I20260709 12:21:58.066335 32371 rpcz_store.cc:276] 0709 12:21:54.713266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.714904 (+  1638us) service_pool.cc:224] Handling call
0709 12:21:54.833820 (+118916us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:54.982522 (+148702us) spinlock_profiling.cc:243] Waited 72.7 ms on lock 0x7b54004275a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:21:57.571151 (+2588629us) spinlock_profiling.cc:243] Waited 869 ms on lock 0x7b58009911d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd310aa 00007fe8fcdb3447 00007fe8fcdcbfe7 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4
0709 12:21:58.057331 (+486180us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.057655 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2166099712}
W20260709 12:21:58.067399 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.029032 31670 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:21:57.967964  2476 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:21:58.073336  2555 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.860s	user 0.010s	sys 0.002s
W20260709 12:21:58.064998  2553 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 8 current term: 8 is higher.
I20260709 12:21:58.053536  2273 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:57.977881  2355 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.104s
I20260709 12:21:57.966660  2718 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:58.086073 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:58.088222 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:57.977854  2727 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.106s
I20260709 12:21:58.088881 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:58.090423 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:58.092013 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:58.098964 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:58.099499 31822 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:58.100412 31747 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:58.005571  2554 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:57.982692  2399 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:58.094538 31676 raft_consensus.cc:2368] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10.
I20260709 12:21:58.052379  2551 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:58.094424 31825 raft_consensus.cc:2376] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5. Re-sending same reply.
I20260709 12:21:58.079305  2570 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:58.120926 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:58.122411 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:58.123827 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:57.981604  2253 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.113s
I20260709 12:21:58.094265  2273 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:21:58.141954 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:58.151296 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:58.092636  2555 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:21:58.151916 31825 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:21:58.109369 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 204303us
I20260709 12:21:58.153548 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:58.154454 31747 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:58.155491 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:58.101598 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:58.089660  2476 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:58.156855 31791 leader_election.cc:400] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10.
I20260709 12:21:58.157493  2476 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:58.157510 31791 leader_election.cc:403] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:21:58.161011  2690 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 10
W20260709 12:21:58.121734  2659 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 2.571s	user 0.000s	sys 0.012s
I20260709 12:21:58.156116  2666 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.005755  2681 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:58.200601 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:58.089774 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 189998us
I20260709 12:21:58.166038 31747 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:21:58.215934 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:58.216104 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:58.094759  2731 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260709 12:21:58.218041 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:58.225049 31747 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:58.226060 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:58.231745 31747 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:58.232870 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:58.233660 31747 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:58.240085 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:58.221940  2717 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.527s	user 0.008s	sys 0.004s
I20260709 12:21:58.243674 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:58.254117 31747 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:21:58.259112 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2044) took 2973 ms. Trace:
I20260709 12:21:58.259635 31747 rpcz_store.cc:276] 0709 12:21:55.285554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.253437 (+2967883us) service_pool.cc:224] Handling call
0709 12:21:58.253867 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.258608 (+  4741us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.259015 (+   407us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:58.261546 31747 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260709 12:21:58.262698 31670 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:58.267488 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:58.269236 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:58.270069 31676 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:21:58.281828 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2045) took 2991 ms. Trace:
I20260709 12:21:58.282598 31747 rpcz_store.cc:276] 0709 12:21:55.290580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.260645 (+2970065us) service_pool.cc:224] Handling call
0709 12:21:58.261083 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.261982 (+   899us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.281687 (+ 19705us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:58.284144 31747 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:58.284946 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2047) took 2930 ms. Trace:
W20260709 12:21:58.285837  2749 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:21:58.285632 31747 rpcz_store.cc:276] 0709 12:21:55.354341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.283624 (+2929283us) service_pool.cc:224] Handling call
0709 12:21:58.283960 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.284707 (+   747us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.284893 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:58.286914  2753 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.076s	user 0.001s	sys 0.000s
W20260709 12:21:58.288030  2762 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:21:58.288338 31747 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:58.289312 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2051) took 2807 ms. Trace:
I20260709 12:21:58.289633 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1937) took 1836 ms. Trace:
I20260709 12:21:58.289752 31747 rpcz_store.cc:276] 0709 12:21:55.481759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.287361 (+2805602us) service_pool.cc:224] Handling call
0709 12:21:58.287974 (+   613us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.288918 (+   944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.289238 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:58.295259  2748 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.199s	user 0.000s	sys 0.001s
I20260709 12:21:58.218371  2666 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:58.296099 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1443) took 1384 ms. Trace:
I20260709 12:21:58.296599 31673 rpcz_store.cc:276] 0709 12:21:56.911965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.978433 (+ 66468us) service_pool.cc:224] Handling call
0709 12:21:58.295842 (+1317409us) spinlock_profiling.cc:243] Waited 77.1 ms on lock 0x7b54002553d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:58.296021 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":177237248}
I20260709 12:21:58.297518  2666 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:58.297856 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:58.299732  2711 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:58.300096  2727 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:58.300801  2727 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:58.301630  2750 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:21:58.305295 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 423) took 1716 ms. Trace:
I20260709 12:21:58.305737 31751 rpcz_store.cc:276] 0709 12:21:56.588958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.304438 (+1715480us) service_pool.cc:224] Handling call
0709 12:21:58.304783 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.304962 (+   179us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:58.304986 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:58.305026 (+    40us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:21:58.305064 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.305124 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.305233 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:58.307252 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1550) took 1467 ms. Trace:
I20260709 12:21:58.307951  2452 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:58.309118  2393 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:21:58.309590  2393 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 4 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:58.311414 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2030) took 3249 ms. Trace:
I20260709 12:21:58.327885 31745 rpcz_store.cc:276] 0709 12:21:55.062180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.981366 (+919186us) service_pool.cc:224] Handling call
0709 12:21:55.982213 (+   847us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.187038 (+1204825us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.311142 (+1124104us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54002f62d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:58.311345 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2769179392}
I20260709 12:21:58.311841 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 413) took 4291 ms. Trace:
I20260709 12:21:58.329422 31748 rpcz_store.cc:276] 0709 12:21:54.020183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:54.020303 (+   120us) service_pool.cc:224] Handling call
0709 12:21:54.020625 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.243469 (+1222844us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.311578 (+3068109us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b5400064854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:58.311747 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2810919808}
I20260709 12:21:58.290015 31824 rpcz_store.cc:276] 0709 12:21:56.452970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.903722 (+1450752us) service_pool.cc:224] Handling call
0709 12:21:57.904076 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.924759 (+ 20683us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:57.941789 (+ 17030us) log.cc:844] Serialized 14 byte log entry
0709 12:21:58.118421 (+176632us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:58.164984 (+ 46563us) log.cc:844] Serialized 26 byte log entry
0709 12:21:58.219875 (+ 54891us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:58.219929 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.220653 (+   724us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:58.289434 (+ 68781us) raft_consensus.cc:1685] finished
0709 12:21:58.289454 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.289577 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22010,"prepare.queue_time_us":27753,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":20551,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":6459,"thread_start_us":113590,"threads_started":1,"wal-append.queue_time_us":245529}
I20260709 12:21:58.319629 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:58.332360 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:58.323998 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:58.307675 31749 rpcz_store.cc:276] 0709 12:21:56.840011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.092108 (+252097us) service_pool.cc:224] Handling call
0709 12:21:58.307049 (+1214941us) spinlock_profiling.cc:243] Waited 62.8 ms on lock 0x7b5400065754. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:58.307194 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":144455936}
I20260709 12:21:58.335592 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:58.336010 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:58.336604 31670 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:58.336884 31673 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:58.339082 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:58.326948 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:58.340131  2752 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:21:58.335053  2452 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:58.345193 31749 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:21:56.569818 31818 rpcz_store.cc:276] 0709 12:21:55.213753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.366201 (+152448us) service_pool.cc:224] Handling call
0709 12:21:55.422075 (+ 55874us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:55.422365 (+   290us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:21:55.634943 (+212578us) log.cc:844] Serialized 14 byte log entry
0709 12:21:56.569103 (+934160us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:56.569156 (+    53us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:21:56.569218 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.569290 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:56.569411 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1335,"prepare.queue_time_us":107354,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":47,"raft.queue_time_us":1472,"thread_start_us":1133479,"threads_started":2}
I20260709 12:21:58.346760 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:58.322574  2737 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.176s	user 0.000s	sys 0.000s
I20260709 12:21:58.347306 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1553) took 1495 ms. Trace:
I20260709 12:21:58.363044 31749 rpcz_store.cc:276] 0709 12:21:56.852264 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.344572 (+1492308us) service_pool.cc:224] Handling call
0709 12:21:58.344945 (+   373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:58.346964 (+  2019us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.347236 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:58.347487 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:58.355085 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:58.352192 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:58.357592 31673 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:58.366595 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:58.368139 31676 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:58.343508 31751 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:58.346213  2646 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:58.369305 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1544) took 1543 ms. Trace:
W20260709 12:21:58.313433  2751 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
W20260709 12:21:58.323232  2758 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:21:58.321485  2511 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:58.372649 31748 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: []
I20260709 12:21:58.369841 31751 rpcz_store.cc:276] 0709 12:21:56.825670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.342845 (+1517175us) service_pool.cc:224] Handling call
0709 12:21:58.343262 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.368941 (+ 25679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.369231 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:58.380095 31751 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:21:58.317472  2399 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:58.317574  2727 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:58.344602  2444 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:58.385088  2476 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:58.385581  2561 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:58.391147  2729 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.384428 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1554) took 1531 ms. Trace:
I20260709 12:21:58.359571  2253 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
W20260709 12:21:58.407476 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.310607  2694 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.373602  2733 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:58.413369 31748 rpcz_store.cc:276] 0709 12:21:56.853194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.356179 (+1502985us) service_pool.cc:224] Handling call
0709 12:21:58.372472 (+ 16293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.384063 (+ 11591us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.384348 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:58.379058  1897 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:58.414391  2561 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:58.310484  2393 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:58.418256 31751 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:58.396672  2444 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:58.419777  2547 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:21:58.419972  2733 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:58.428285  2694 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:58.402001  2727 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.429018 31822 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:58.442389 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 400) took 2107 ms. Trace:
I20260709 12:21:58.400677  2705 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.443364 31826 rpcz_store.cc:276] 0709 12:21:56.335451 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.804028 (+1468577us) service_pool.cc:224] Handling call
0709 12:21:57.886612 (+ 82584us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.941358 (+ 54746us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.442015 (+500657us) spinlock_profiling.cc:243] Waited 46.4 ms on lock 0x7b540037a354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:58.442306 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":106818432}
W20260709 12:21:58.446584 32369 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 510) took 3184 ms (client timeout 1590 ms). Trace:
I20260709 12:21:58.423923  1897 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:21:58.447345 32369 rpcz_store.cc:269] 0709 12:21:55.259342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.259571 (+   229us) service_pool.cc:224] Handling call
0709 12:21:58.443181 (+3183610us) spinlock_profiling.cc:243] Waited 343 ms on lock 0x7b54004c4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:58.443324 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":789643648}
I20260709 12:21:58.434908  2444 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:58.436312  2733 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:58.451138  2393 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:21:58.447057  2551 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:58.420717  2729 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.451049  1897 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:58.478937  2709 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.586s	user 0.012s	sys 0.000s
I20260709 12:21:58.479694  2709 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:21:58.479739  2763 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:21:58.485967  2709 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:21:58.469000  2681 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:58.450138  2727 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.492287  2727 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:58.450364  2705 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.495761  2646 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:58.496389  2729 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:58.496611  2718 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.494557  2705 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:58.371496 31674 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:21:58.376328 31676 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:58.591907  2702 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.002s
I20260709 12:21:58.595240  2731 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.596563  2731 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.554354 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:58.598901 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:58.600379  2498 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:21:58.601926  2681 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:21:58.601980  2646 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.603812  2646 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:58.543534 31669 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8. Ops: []
W20260709 12:21:58.612759 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:58.613652 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.559154 31821 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 10. Ops: []
W20260709 12:21:58.615272 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:58.615788 31675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1428) took 3670 ms (client timeout 1786 ms). Trace:
W20260709 12:21:58.616287 31675 rpcz_store.cc:269] 0709 12:21:54.945345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.615270 (+3669925us) service_pool.cc:211] Skipping call since client already timed out
0709 12:21:58.615531 (+   261us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:21:58.619092  2729 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.620107 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2037) took 3396 ms. Trace:
I20260709 12:21:58.341351 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:58.620227  2729 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.620682 31744 rpcz_store.cc:276] 0709 12:21:55.223579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.458000 (+2234421us) service_pool.cc:224] Handling call
0709 12:21:57.469317 (+ 11317us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.950977 (+481660us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.619407 (+668430us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b54003e3ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:58.619856 (+   449us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1085470976}
I20260709 12:21:58.621403  2727 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.622329  2727 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.624327  2729 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:58.625046  2727 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:58.603421 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49650us
I20260709 12:21:58.547482 31671 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:21:58.614243 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2024) took 2207 ms. Trace:
W20260709 12:21:58.558342 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.626461 31745 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:21:58.544195 31673 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:58.545981 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2056) took 2034 ms. Trace:
I20260709 12:21:58.641352 31672 rpcz_store.cc:276] 0709 12:21:56.406821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:56.981145 (+574324us) service_pool.cc:224] Handling call
0709 12:21:58.613771 (+1632626us) spinlock_profiling.cc:243] Waited 953 ms on lock 0x7b54002b0554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:58.614020 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2192309120}
I20260709 12:21:58.631942 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2043) took 3357 ms. Trace:
I20260709 12:21:58.630576 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:58.545395 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2029) took 1081 ms. Trace:
I20260709 12:21:58.710072 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1552) took 1830 ms. Trace:
W20260709 12:21:58.716558 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.652519 31744 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:58.651911 31746 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:58.766564 31745 rpcz_store.cc:276] 0709 12:21:56.851275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.625260 (+1773985us) service_pool.cc:224] Handling call
0709 12:21:58.625838 (+   578us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:58.648434 (+ 22596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.681314 (+ 32880us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:58.652319  2551 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:58.647593 31750 rpcz_store.cc:276] 0709 12:21:55.274452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.465525 (+2191073us) service_pool.cc:224] Handling call
0709 12:21:57.465920 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:57.810875 (+344955us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.631343 (+820468us) spinlock_profiling.cc:243] Waited 10 ms on lock 0x7b54002954d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:58.631746 (+   403us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23055744}
I20260709 12:21:58.785955  2399 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:21:58.786835  2399 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:58.788398  2399 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:58.790712  2727 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:58.783245 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:58.791666 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.773785  2646 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:58.715155 31674 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:58.775592  2551 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:58.779707 31676 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:58.732100  2729 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.051s
I20260709 12:21:58.793713  2727 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.645747 31747 rpcz_store.cc:276] 0709 12:21:56.511138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.291055 (+1779917us) service_pool.cc:224] Handling call
0709 12:21:58.292025 (+   970us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:21:58.292956 (+   931us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:58.293023 (+    67us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:21:58.294574 (+  1551us) log.cc:844] Serialized 78 byte log entry
0709 12:21:58.543517 (+248943us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:58.544513 (+   996us) log.cc:844] Serialized 14 byte log entry
0709 12:21:58.545055 (+   542us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.545274 (+   219us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:58.545374 (+   100us) raft_consensus.cc:1685] finished
0709 12:21:58.545389 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.545635 (+   246us) inbound_call.cc:177] Queueing success response
0709 12:21:58.637561 (+ 91926us) spinlock_profiling.cc:243] Waited 977 us on lock 0x7b1800355fc0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
Metrics: {"prepare.queue_time_us":36043,"prepare.run_wall_time_us":69,"raft.queue_time_us":36913,"raft.run_cpu_time_us":380,"raft.run_wall_time_us":381,"spinlock_wait_cycles":24897152,"thread_start_us":247787,"threads_started":1,"wal-append.queue_time_us":204704}
I20260709 12:21:58.812160  2666 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:21:58.803334  2551 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:58.815629 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 414306us
I20260709 12:21:58.659147  2694 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.152s
I20260709 12:21:58.799052 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:58.802975 31676 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:58.766856 31677 rpcz_store.cc:276] 0709 12:21:57.464177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.902824 (+438647us) service_pool.cc:224] Handling call
0709 12:21:57.919490 (+ 16666us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.920001 (+   511us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:21:58.119729 (+199728us) log.cc:844] Serialized 14 byte log entry
0709 12:21:58.225929 (+106200us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:58.226398 (+   469us) log.cc:844] Serialized 26 byte log entry
0709 12:21:58.226523 (+   125us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:58.226554 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.226616 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:58.544795 (+318179us) raft_consensus.cc:1685] finished
0709 12:21:58.544848 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.545194 (+   346us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":757759,"prepare.queue_time_us":59500,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":5138,"raft.run_cpu_time_us":434,"raft.run_wall_time_us":753413,"thread_start_us":304403,"threads_started":2,"wal-append.queue_time_us":293105}
I20260709 12:21:58.835373  2709 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:58.836187  2709 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.838666 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:58.839591 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 125803us
W20260709 12:21:58.845346  2688 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.918s	user 0.007s	sys 0.000s
I20260709 12:21:58.830864  2690 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10.
I20260709 12:21:58.852949 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:58.859380 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 144761us
I20260709 12:21:58.861191  2500 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:58.866695 31676 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:58.876489  2728 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.293s	user 0.004s	sys 0.002s
I20260709 12:21:58.876547 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1903) took 3745 ms. Trace:
W20260709 12:21:58.880199  2765 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.417s	user 0.002s	sys 0.000s
I20260709 12:21:58.878749 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1566) took 1261 ms. Trace:
I20260709 12:21:58.880962 31751 rpcz_store.cc:276] 0709 12:21:57.569679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.430183 (+860504us) service_pool.cc:224] Handling call
0709 12:21:58.430511 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:58.430764 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:21:58.431068 (+   304us) log.cc:844] Serialized 14 byte log entry
0709 12:21:58.801150 (+370082us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:58.807562 (+  6412us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:58.823367 (+ 15805us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.828001 (+  4634us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.831242 (+  3241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4289,"prepare.queue_time_us":830,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":9616,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":202,"thread_start_us":326845,"threads_started":1,"wal-append.queue_time_us":276826}
W20260709 12:21:58.883448 31823 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1918) took 3592 ms (client timeout 2127 ms). Trace:
W20260709 12:21:58.883984 31823 rpcz_store.cc:269] 0709 12:21:55.290405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.362253 (+ 71848us) service_pool.cc:224] Handling call
0709 12:21:58.882845 (+3520592us) spinlock_profiling.cc:243] Waited 2.62 s on lock 0x7b54000fbc54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:21:58.883073 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6018786176}
I20260709 12:21:58.879520 31819 rpcz_store.cc:276] 0709 12:21:55.131333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.135569 (+  4236us) service_pool.cc:224] Handling call
0709 12:21:55.212374 (+ 76805us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:55.922081 (+709707us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:21:55.928095 (+  6014us) log.cc:844] Serialized 14 byte log entry
0709 12:21:56.302234 (+374139us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:56.302916 (+   682us) log.cc:844] Serialized 52 byte log entry
0709 12:21:56.303075 (+   159us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:56.303110 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:56.303178 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:58.844242 (+2541064us) spinlock_profiling.cc:243] Waited 4.24 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:21:58.875828 (+ 31586us) raft_consensus.cc:1685] finished
0709 12:21:58.875957 (+   129us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:58.876263 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":266,"prepare.queue_time_us":7192,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":9930,"raft.queue_time_us":198361,"raft.run_cpu_time_us":234,"raft.run_wall_time_us":5443,"spinlock_wait_cycles":9750784,"thread_start_us":1053776,"threads_started":2,"wal-append.queue_time_us":908199}
I20260709 12:21:58.860283  2731 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:58.913516 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:21:58.915823 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57349.8us
I20260709 12:21:58.917382 31677 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:21:58.922448  2444 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.926820  2740 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:58.927872  2740 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.929033  2296 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:58.930208 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:58.900034 31674 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:58.936368 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 306142us
W20260709 12:21:58.934767  2589 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.724s	user 0.010s	sys 0.008s
I20260709 12:21:58.938819 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:58.946996  2512 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:21:58.944005  2589 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.850s	user 0.011s	sys 0.008s
W20260709 12:21:58.968053  2273 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.746s	user 0.009s	sys 0.009s
W20260709 12:21:58.968536 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:58.968669  2273 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:58.971199 31748 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:21:58.971766  2709 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:58.972935  2474 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:58.973470 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:58.973470  2474 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:58.974264  2474 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:58.974563  2500 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:58.974787 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:58.975134  2500 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:58.975587  2500 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:58.982185  2273 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:58.990929  1897 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.000634 31673 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 9. Current term is 10. Ops: []
I20260709 12:21:59.001519  2444 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:59.002427 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.003887 31673 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
I20260709 12:21:59.007472  2393 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:21:58.996935  2273 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.008720  2273 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:58.989893  2512 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260709 12:21:59.010648  2444 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:21:59.012997 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45739.7us
I20260709 12:21:59.022747 31674 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:58.996856 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:59.024477 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 68511.2us
I20260709 12:21:59.026137 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:59.028532 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.029815 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 537) took 1880 ms. Trace:
I20260709 12:21:59.030848 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:59.031009 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:21:58.968122  2706 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.564s	user 0.002s	sys 0.015s
I20260709 12:21:59.031818 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:58.984766  2740 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:59.032727 31822 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:21:59.032773 31823 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 9. Current term is 10. Ops: []
I20260709 12:21:58.986568  2727 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:59.031170 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:59.034914 32377 rpcz_store.cc:276] 0709 12:21:57.149274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.150393 (+  1119us) service_pool.cc:224] Handling call
0709 12:21:57.163452 (+ 13059us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.555338 (+391886us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.029240 (+1473902us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b5400546354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:59.029602 (+   362us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":899670272}
I20260709 12:21:59.059408 31824 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:21:59.072551  1897 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:59.074795  2499 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.075088 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:59.075685 31673 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:59.033816 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:59.076309  2703 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
W20260709 12:21:59.076988  2561 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.643s	user 0.018s	sys 0.009s
I20260709 12:21:59.077734 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:59.079180  2696 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:59.079895  2668 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:59.082931  2705 raft_consensus.cc:2955] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Committing config change with OpId 7.7: config changed from index 3 to 7, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:59.089563 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:59.090888  2273 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:21:59.112643 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 243867us
I20260709 12:21:59.113083  2273 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:21:59.113538  2234 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:59.113642 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:21:59.114842  2273 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:21:59.117053 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.117930 31670 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.647s	user 0.014s	sys 0.005s
I20260709 12:21:59.118383 31670 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:21:59.118775  2727 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:59.119689  2740 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.119462  2733 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.120298  2740 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:59.099059  2772 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
I20260709 12:21:59.104153  2702 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.129554 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:59.114898  2561 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.129904  2702 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.131376 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.132115  2561 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:59.132469  2702 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:59.128427  2660 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:59.127058 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.133236  2660 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:59.133437  2703 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:21:59.133951  2660 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:59.093408  2681 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:59.134801 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.135809 31746 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 7: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:21:59.136730 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.137537 31746 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:21:59.127908 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50313.1us
I20260709 12:21:59.122151 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:59.118789 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.134948 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:59.122839 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:59.119052 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2042) took 1088 ms. Trace:
W20260709 12:21:59.097898  2706 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.723s	user 0.003s	sys 0.016s
I20260709 12:21:59.142390 31670 rpcz_store.cc:276] 0709 12:21:58.031078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.264420 (+233342us) service_pool.cc:224] Handling call
0709 12:21:59.118959 (+854539us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.125054 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.128662 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:59.153306 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.129506 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:21:59.128975 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.142664  2646 raft_consensus.cc:2764] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:21:59.157286 31750 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:21:59.157969  2696 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.158900 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.159430 31752 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:59.160547  2696 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:21:59.161245  2776 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.262s	user 0.001s	sys 0.001s
I20260709 12:21:59.162894 31747 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:21:59.163022 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1555) took 2308 ms. Trace:
I20260709 12:21:59.163961  2719 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:59.164422  2474 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:59.167214  2742 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.076s	user 0.000s	sys 0.004s
I20260709 12:21:59.169099  2713 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:59.169656  2713 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:59.169312  2551 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.170934 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:59.171851  2732 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:21:59.172102  2551 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:59.172365  2732 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:59.172523 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:59.154995 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:59.173558  2709 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:59.174026  2709 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.173627  2732 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.125765  2718 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:59.174975 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:59.176234  2771 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.705s	user 0.001s	sys 0.000s
I20260709 12:21:59.174430  2709 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:59.180317 31678 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:21:59.181782  2499 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.183024  2715 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:59.183207  2160 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:21:59.183516  2715 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.183799  2160 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:59.183996  2715 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:59.137952  2668 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.185550  2668 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:59.186363 31821 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:21:59.187574 31827 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:21:59.156061 31753 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:21:59.189543  2325 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.170930  2713 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.190092  2325 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.176152 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:59.199774 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:59.202813 31676 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:21:59.121821  2733 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:59.203585 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2041) took 1251 ms. Trace:
I20260709 12:21:59.203975 31676 rpcz_store.cc:276] 0709 12:21:57.952202 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.375820 (+423618us) service_pool.cc:224] Handling call
0709 12:21:58.376187 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.203264 (+827077us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.203527 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:21:59.208024 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 79592.9us
I20260709 12:21:59.163445 31749 rpcz_store.cc:276] 0709 12:21:56.854097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.364088 (+1509991us) service_pool.cc:224] Handling call
0709 12:21:58.398555 (+ 34467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:58.398940 (+   385us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:58.398979 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:58.399617 (+   638us) log.cc:844] Serialized 26 byte log entry
0709 12:21:58.722923 (+323306us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:58.723023 (+   100us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:58.723151 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:59.162495 (+439344us) raft_consensus.cc:1685] finished
0709 12:21:59.162515 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:59.162844 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":91920,"threads_started":1,"wal-append.queue_time_us":421081}
I20260709 12:21:59.216586  2732 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:59.217536 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.236095 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.241885 31744 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:21:59.198695  2474 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:59.185042  2499 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:59.246081 31752 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
W20260709 12:21:59.248042 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 72027.8us
I20260709 12:21:59.164431  2719 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:59.207820  2729 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:59.256704 31674 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:21:59.184993  2160 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.250276  2234 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:59.250751  2733 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:21:59.287603 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2040) took 1341 ms. Trace:
I20260709 12:21:59.251756  2729 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:21:59.250315  2393 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:59.291010  2740 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:59.292656 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:55.399309 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:21:59.311986  2690 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:21:59.313375  2666 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:21:59.313907  2666 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:59.315146  2666 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.316671  2666 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:59.294893  2296 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:59.295634 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:21:59.296615 32343 connection.cc:256] RPC call timeout handler was delayed by -2.4368027400000756s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1859s for a response
I20260709 12:21:59.297158 31673 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:21:59.255679  2500 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.313650 31573 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 5 to 6, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 6 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:21:59.332479  2500 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:59.333403  2500 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:59.333935  2500 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:59.337278  2740 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:59.339675 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:59.300521  2108 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:21:59.341315 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2043) took 1130 ms. Trace:
I20260709 12:21:59.283492  2499 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:59.303278 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1558) took 1751 ms. Trace:
I20260709 12:21:59.308764  2512 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:21:59.350502  1897 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:21:59.352165  1897 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:59.345237 31748 rpcz_store.cc:276] 0709 12:21:57.551829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.420276 (+868447us) service_pool.cc:224] Handling call
0709 12:21:58.555672 (+135396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:58.971617 (+415945us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.302990 (+331373us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b54002134d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:59.303214 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":954437376}
I20260709 12:21:59.327392 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:59.304033 31674 rpcz_store.cc:276] 0709 12:21:57.945826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.370974 (+425148us) service_pool.cc:224] Handling call
0709 12:21:58.371351 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.257290 (+885939us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.287491 (+ 30201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:21:59.288239  2234 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.303056  2733 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:21:59.357636 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2045) took 1063 ms. Trace:
I20260709 12:21:59.359727 31676 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260709 12:21:59.358621 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 281300us
I20260709 12:21:59.361001 31678 rpcz_store.cc:276] 0709 12:21:58.294048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.957173 (+663125us) service_pool.cc:224] Handling call
0709 12:21:59.022949 (+ 65776us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.180785 (+157836us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.357384 (+176599us) spinlock_profiling.cc:243] Waited 157 ms on lock 0x7b54002535d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:21:59.357579 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":361116928}
I20260709 12:21:59.341768 31669 rpcz_store.cc:276] 0709 12:21:58.211154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.863279 (+652125us) service_pool.cc:224] Handling call
0709 12:21:58.863736 (+   457us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:58.889982 (+ 26246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:58.890066 (+    84us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:58.916417 (+ 26351us) log.cc:844] Serialized 26 byte log entry
0709 12:21:59.102648 (+186231us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:59.102714 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.102807 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:59.341032 (+238225us) raft_consensus.cc:1685] finished
0709 12:21:59.341054 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:59.341245 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31258,"raft.queue_time_us":34877,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":10679,"thread_start_us":149179,"threads_started":1,"wal-append.queue_time_us":323142}
W20260709 12:21:59.376688 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 84084.4us
I20260709 12:21:59.360308  2234 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:59.380245 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53052.5us
I20260709 12:21:59.380841 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:59.363566  2733 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:59.387398 31672 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:21:59.388818  2709 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.390888  2729 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:21:59.396029 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.399534  2325 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:59.401235 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:21:59.404466 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.404306 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:59.386590  2613 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:21:59.411151  1897 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.411654  2660 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.392931 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:59.412029  2499 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.413618 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:59.353644  2727 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
W20260709 12:21:59.420334 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 327370us
W20260709 12:21:59.424587 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44815.6us
I20260709 12:21:59.381618  2296 consensus_queue.cc:1243] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 8.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:21:59.391999  2512 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:21:59.427490 31678 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:59.413872 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 301545us
I20260709 12:21:59.392958  2474 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:59.394381 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:21:59.429489 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.386564  2710 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.110s
I20260709 12:21:59.448964 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1462) took 1282 ms. Trace:
W20260709 12:21:59.454599 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60361.2us
I20260709 12:21:59.455539 31673 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
W20260709 12:21:59.461033 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 347469us
I20260709 12:21:59.385380  2500 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.440085  2512 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:21:59.463025 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:59.450690 31671 rpcz_store.cc:276] 0709 12:21:58.166119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.713637 (+547518us) service_pool.cc:224] Handling call
0709 12:21:59.200692 (+487055us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:21:59.200885 (+   193us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.201324 (+   439us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:21:59.201750 (+   426us) log.cc:844] Serialized 14 byte log entry
0709 12:21:59.448150 (+246400us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:21:59.448277 (+   127us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:59.448376 (+    99us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.448568 (+   192us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:59.448837 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8122,"prepare.queue_time_us":163,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"spinlock_wait_cycles":5116160,"thread_start_us":230213,"threads_started":1}
I20260709 12:21:59.475376  2568 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:59.476240  2568 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.478271  2325 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:21:59.478379 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:21:59.478986  2325 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.479539  2325 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:59.481884  2670 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:21:59.482503  2670 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:59.482908  2740 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:21:59.484088  2651 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 12:21:59.481566  2296 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:59.486233  2296 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:59.486891  2296 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.487345  2512 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.490099  2234 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.483547  2670 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.493822  2296 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:59.494333  2108 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:21:59.450161  2732 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260709 12:21:59.496628  2589 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.497146 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.497337  2589 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.497988  2234 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:59.498835 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:21:59.501852 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.514881 31573 catalog_manager.cc:5697] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 4 to 7, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 7 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:21:59.516412 31792 connection.cc:256] RPC call timeout handler was delayed by -1.8244710479998503s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16310000000000002s for a response
I20260709 12:21:59.502954  2696 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:21:59.519093  2696 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.519784  2660 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:59.449945  2399 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.073s
I20260709 12:21:59.521814  2696 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:59.506714 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.490468  2613 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:21:59.522681 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:59.517956 31573 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:21:59.524096  2718 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:59.524549  2718 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.524940  2718 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:59.508831  2393 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.513616 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 686) took 1872 ms. Trace:
I20260709 12:21:59.507102  2729 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:59.536504  2499 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:59.510985  2709 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:59.526928 31792 connection.cc:256] RPC call timeout handler was delayed by -1.1246532140003183s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21320000000000003s for a response
W20260709 12:21:59.548463 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 156633us
I20260709 12:21:59.550616 31678 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:21:59.554116 31669 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:21:59.534060  2715 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:59.536764 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 69772.4us
I20260709 12:21:59.523423  2613 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:59.532507 32375 rpcz_store.cc:276] 0709 12:21:57.641456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:57.641646 (+   190us) service_pool.cc:224] Handling call
0709 12:21:57.642123 (+   477us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:57.642541 (+   418us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:57.642569 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:57.661537 (+ 18968us) log.cc:844] Serialized 26 byte log entry
0709 12:21:59.179496 (+1517959us) spinlock_profiling.cc:243] Waited 1.8 ms on lock 0x7b1800373a20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:21:59.453430 (+273934us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:59.453489 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.453631 (+   142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:21:59.513089 (+ 59458us) raft_consensus.cc:1685] finished
0709 12:21:59.513113 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:21:59.513405 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":139,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":15567,"spinlock_wait_cycles":4131584,"thread_start_us":241729,"threads_started":1,"wal-append.queue_time_us":1849382}
I20260709 12:21:59.537247  2710 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:21:59.523367  2399 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 2, Time since last communication: 0.014s
I20260709 12:21:59.488786  2512 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:21:59.544854 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 72242.4us
I20260709 12:21:59.506229  2160 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:59.550987 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:59.550823  2794 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.013s
I20260709 12:21:59.583778  2794 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:59.584287  2794 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:21:59.555078 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:21:59.567888  2710 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:21:59.567982  2613 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.548097  2733 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.556617  2719 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:21:59.572273  2589 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:59.574023 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:59.575120 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:59.575543 32343 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:21:59.575664 31822 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260709 12:21:59.576380  2713 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:59.579852 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:59.580874 31826 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:21:59.582844 31823 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:59.585633  2794 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.595011 31792 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:59.559126  2660 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:59.638393 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:59.595036  2399 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:21:59.595824  2681 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:21:59.647145 31824 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:59.656324 31674 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:59.654106 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:21:59.663586 31827 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:21:59.661705 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:21:59.634696  2696 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260709 12:21:59.656925  2399 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:21:59.689913 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.695786 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:21:59.697072 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 106877us
I20260709 12:21:59.661000  2719 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:21:59.704906 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 129918us
I20260709 12:21:59.709884 31819 raft_consensus.cc:2410] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:21:59.655442  2668 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.713745 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:59.684892  2709 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:21:59.718772 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 143342us
I20260709 12:21:59.716502 31822 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
W20260709 12:21:59.721480 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.704622  2794 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:59.733299 31821 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 10. Ops: []
I20260709 12:21:59.698172  2660 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:59.741509 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.743043 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62349.6us
I20260709 12:21:59.754954  2789 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.155s
I20260709 12:21:59.784281 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:59.776784  2793 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:21:59.792670 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:59.783460 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:59.776032  2654 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.147s
I20260709 12:21:59.794245  2499 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:21:59.775940  2803 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:21:59.734099  2234 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
I20260709 12:21:59.777809  2660 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:59.795701  2682 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:21:59.785066 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:59.788226 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:59.796270  2682 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:59.779302 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.791218  2568 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:59.797371  2682 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:21:59.782197 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:21:59.798612 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:21:59.775835  2790 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:21:59.799470  2790 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.798843  2399 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.799836  2790 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:21:59.779111 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59481.2us
I20260709 12:21:59.775561 31825 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:21:59.776635 31751 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: []
W20260709 12:21:59.734323  2801 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:21:59.794162 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:59.794195 32375 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:21:59.782248 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:59.778429 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:59.821839 31675 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:21:59.821547 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:59.837527 31824 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:21:59.820648 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:59.841815 32374 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:59.741107  2796 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:21:59.749747  2740 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:21:59.806703  2682 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:59.794879  2499 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:21:59.778280  2668 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:21:59.856911 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 171560us
I20260709 12:21:59.859424 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:59.734901  2325 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:21:59.902859 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:21:59.920869 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:59.921310 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:59.922263 31673 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:59.956676 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:21:59.958179 31827 raft_consensus.cc:2410] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:21:59.923410 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.959115 31752 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
W20260709 12:21:59.913738  2705 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.714s	user 0.012s	sys 0.003s
I20260709 12:21:59.846654  2715 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:21:59.961517 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:21:59.901417 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:59.929881 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.855010  2733 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:21:59.840382  2651 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:21:59.854753  2740 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:21:59.855855  2710 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:21:59.981215 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:21:59.969331 31744 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 4. Current term is 5. Ops: []
I20260709 12:21:59.903815  2499 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:21:59.962885 31822 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:21:59.976264 31673 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:21:59.922430  2660 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:21:59.983762 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.960232 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:21:59.978376  1897 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:21:59.905396  2793 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.110s
W20260709 12:22:00.001572 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 141461us
I20260709 12:21:59.982120 31824 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:21:59.987112 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:21:59.997496 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:21:59.861107  2500 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.121s
I20260709 12:22:00.004585  2234 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:00.012271 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:00.019912 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:00.020716 31673 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:22:00.022400 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:00.020187 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 81229.8us
W20260709 12:22:00.027542 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 291790us
W20260709 12:22:00.053148 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 150362us
I20260709 12:22:00.077553 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:21:59.974503  2796 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.070s
I20260709 12:22:00.088778 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:00.143435 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 66454.3us
I20260709 12:22:00.143501 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:00.145663 31674 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:00.144703 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:00.085090  2782 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 12:22:00.160408  2784 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:22:00.160622  2690 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:22:00.160928 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 158025us
W20260709 12:22:00.161752  2805 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:22:00.161821  2568 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:00.163266  2817 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
W20260709 12:22:00.166214 31823 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.523s	user 0.008s	sys 0.000s
I20260709 12:22:00.166971 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:00.168395 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:00.169955 31823 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:00.170780 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:00.171898 32370 raft_consensus.cc:2368] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10.
I20260709 12:22:00.172384  2694 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:22:00.173200  2804 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.261s	user 0.000s	sys 0.001s
I20260709 12:22:00.173089  2694 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:00.175797  2434 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.176182 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2044) took 1921 ms. Trace:
I20260709 12:22:00.176555  2434 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:00.176626 31677 rpcz_store.cc:276] 0709 12:21:58.254612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:58.943508 (+688896us) service_pool.cc:224] Handling call
0709 12:21:59.036589 (+ 93081us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:59.373911 (+337322us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.373945 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:21:59.382105 (+  8160us) log.cc:844] Serialized 52 byte log entry
0709 12:21:59.632895 (+250790us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:21:59.649639 (+ 16744us) log.cc:844] Serialized 14 byte log entry
0709 12:21:59.650220 (+   581us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.650302 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:00.175924 (+525622us) raft_consensus.cc:1685] finished
0709 12:22:00.175946 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.176093 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":270,"prepare.queue_time_us":16389,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":54,"raft.queue_time_us":244800,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":10050,"thread_start_us":464029,"threads_started":2,"wal-append.queue_time_us":318564}
W20260709 12:22:00.179729  2820 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 12:22:00.181387 31677 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:22:00.175859  2694 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:00.186090  2718 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.661s	user 0.002s	sys 0.008s
W20260709 12:22:00.195642 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 95806.9us
W20260709 12:22:00.156759  2802 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.205s	user 0.001s	sys 0.000s
I20260709 12:22:00.020584  2651 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:00.201007  2651 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:00.207374  2325 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:00.212172  2512 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.642s	user 0.010s	sys 0.000s
I20260709 12:22:00.152508  2740 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:00.215049 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:00.215869  2740 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:00.033036  2660 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:00.220765 31674 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:00.220927  2660 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:00.175639 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:00.226028 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 64297.8us
I20260709 12:22:00.227285 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:00.255077 31824 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:00.257972 32375 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
W20260709 12:22:00.260413 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:00.279062 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64094.3us
I20260709 12:22:00.282977 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:00.175767  2807 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.237s	user 0.000s	sys 0.001s
I20260709 12:22:00.292614 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:00.284097 32370 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:22:00.280095 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:00.158609  2789 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:00.242897  2568 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:00.260648  2296 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:22:00.317332 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 217941us
I20260709 12:22:00.315723 32373 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:00.281116  2651 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:00.319634 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:00.330075 31744 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
W20260709 12:22:00.335753 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 178460us
I20260709 12:22:00.343186 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:00.363987 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:00.432473 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 89393.8us
W20260709 12:22:00.432749 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 87344us
I20260709 12:22:00.234870  2718 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:00.281445  2703 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.062s
I20260709 12:22:00.322882  2296 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:00.465008 32374 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.612s	user 0.009s	sys 0.005s
W20260709 12:22:00.429046  2803 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.634s	user 0.009s	sys 0.003s
I20260709 12:22:00.322435  2434 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:00.293361  2666 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.154s
I20260709 12:22:00.494757 31749 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:22:00.495116 31752 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:00.495555 32374 raft_consensus.cc:2410] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:22:00.334215  2681 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:00.510589 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:00.513370 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 316004us
W20260709 12:22:00.512884 31822 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.506s	user 0.012s	sys 0.000s
I20260709 12:22:00.515734 31822 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:00.342209  2789 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:00.500152  2803 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:00.535120 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 236384us
I20260709 12:22:00.537225 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:00.524158  2809 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:00.541153  2824 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:22:00.543809  2718 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:00.547008 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2065) took 1016 ms. Trace:
I20260709 12:22:00.547497 31678 rpcz_store.cc:276] 0709 12:21:59.531076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.564812 (+ 33736us) service_pool.cc:224] Handling call
0709 12:21:59.594612 (+ 29800us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:59.644780 (+ 50168us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.644827 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:59.650966 (+  6139us) log.cc:844] Serialized 26 byte log entry
0709 12:21:59.965418 (+314452us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:59.965470 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.965550 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:00.546765 (+581215us) raft_consensus.cc:1685] finished
0709 12:22:00.546786 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.546940 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":493,"raft.queue_time_us":9815,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":5777,"thread_start_us":257393,"threads_started":1,"wal-append.queue_time_us":592479}
I20260709 12:22:00.518064  2727 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:00.565178  2825 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:00.565824  2825 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:00.566252  2825 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:00.567869  2827 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.140s	user 0.000s	sys 0.000s
W20260709 12:22:00.570710  2816 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:22:00.570935  2296 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:00.572142  2296 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:00.572693 32372 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:22:00.566085 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:00.574931  2646 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:00.530731  2690 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:00.550496 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:00.575434  2646 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:00.585878 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 152087us
I20260709 12:22:00.517287  2568 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
W20260709 12:22:00.601714 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 64585.7us
I20260709 12:22:00.586709  2666 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:00.583025  2690 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:00.629266  2790 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.829s	user 0.012s	sys 0.003s
I20260709 12:22:00.638458 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:00.534365  2681 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:00.637300 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 186136us
W20260709 12:22:00.642673 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 362625us
W20260709 12:22:00.629611  2719 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.921s	user 0.013s	sys 0.000s
I20260709 12:22:00.610931  2551 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:00.651012 32369 raft_consensus.cc:2410] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:22:00.612941  2706 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.040s
W20260709 12:22:00.667691  2794 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.934s	user 0.012s	sys 0.000s
W20260709 12:22:00.668285 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.649264  2551 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 5 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:00.673763 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:00.674890 32369 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:00.684662 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:00.663211  2703 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.649852  2790 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:00.699106 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:00.701020 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 91844.1us
I20260709 12:22:00.702759 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:00.705562 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:00.706097 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:00.706450 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.718786 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:00.716957 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:00.724341 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:00.679850  2837 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:00.728428 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:00.729853 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 418148us
W20260709 12:22:00.710101  2499 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.692s	user 0.012s	sys 0.000s
I20260709 12:22:00.726975 32376 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:00.704221  2654 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:00.737305 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:00.744364 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2062) took 1219 ms. Trace:
W20260709 12:22:00.733014  2828 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 12:22:00.769972 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:00.737426 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:00.793645 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:00.793396 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:00.769641 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 67046.7us
W20260709 12:22:00.737613 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 80410.2us
I20260709 12:22:00.802428 31744 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:00.767534 31676 rpcz_store.cc:276] 0709 12:21:59.525040 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.789474 (+264434us) service_pool.cc:224] Handling call
0709 12:21:59.789959 (+   485us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.790337 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:00.242500 (+452163us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.743883 (+501383us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:00.743963 (+    80us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:00.744038 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.744136 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.744284 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":120312,"prepare.queue_time_us":399868,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":62,"raft.queue_time_us":107784,"raft.run_cpu_time_us":93,"raft.run_wall_time_us":93,"thread_start_us":686624,"threads_started":3,"wal-append.queue_time_us":387376}
I20260709 12:22:00.795751 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:00.806007 31747 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:00.733199  2713 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.092s	user 0.011s	sys 0.004s
I20260709 12:22:00.848348 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:00.852541 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:00.806993  2666 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.845404 31792 leader_election.cc:400] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10.
W20260709 12:22:00.899200  2826 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 12:22:00.989500 31792 leader_election.cc:403] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:00.769253  2837 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:00.705706  2690 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:00.853364  2838 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:01.003506  2837 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:01.003769  2838 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:01.004371 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 198013us
I20260709 12:22:01.005638 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:01.005940  2838 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:01.006779  2651 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:00.905372 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:01.007288  2651 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.008688 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:01.009517  2651 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:00.904691  2434 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:00.903179  2833 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.221s	user 0.000s	sys 0.001s
I20260709 12:22:00.906577 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:00.716024  2703 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.010257  2434 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.012249  2434 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:01.012302  2703 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:00.852773 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:00.922123 31750 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:00.915609 31753 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:00.934067 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1972) took 1137 ms. Trace:
W20260709 12:22:00.938205 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.013937 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:00.928550 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2069) took 1069 ms. Trace:
I20260709 12:22:01.013501 31819 rpcz_store.cc:276] 0709 12:21:59.796874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.000012 (+203138us) service_pool.cc:224] Handling call
0709 12:22:00.016577 (+ 16565us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:22:00.147118 (+130541us) raft_consensus.cc:1534] Early marking committed up to index 2
0709 12:22:00.156623 (+  9505us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.279109 (+122486us) raft_consensus.cc:1539] Triggering prepare for 4 ops
0709 12:22:00.327884 (+ 48775us) log.cc:844] Serialized 104 byte log entry
0709 12:22:00.328125 (+   241us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:00.328497 (+   372us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.328740 (+   243us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.328840 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:00.933810 (+604970us) raft_consensus.cc:1685] finished
0709 12:22:00.933833 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.934000 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":10548,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":121,"raft.queue_time_us":69202,"raft.run_cpu_time_us":380,"raft.run_wall_time_us":21974,"spinlock_wait_cycles":16475008,"thread_start_us":118425,"threads_started":1,"wal-append.queue_time_us":770378}
I20260709 12:22:01.014758 31673 rpcz_store.cc:276] 0709 12:21:59.859097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.099577 (+240480us) service_pool.cc:224] Handling call
0709 12:22:00.151854 (+ 52277us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:00.335122 (+183268us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:00.486268 (+151146us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.727489 (+241221us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:00.794998 (+ 67509us) log.cc:844] Serialized 52 byte log entry
0709 12:22:00.807491 (+ 12493us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:00.807775 (+   284us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.807996 (+   221us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.808079 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:00.928249 (+120170us) raft_consensus.cc:1685] finished
0709 12:22:00.928280 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.928472 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":117332,"prepare.queue_time_us":107064,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":90,"raft.queue_time_us":170833,"raft.run_cpu_time_us":534,"raft.run_wall_time_us":87788,"thread_start_us":177271,"threads_started":1,"wal-append.queue_time_us":391262}
I20260709 12:22:00.895375 31824 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
W20260709 12:22:00.934779 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 228459us
I20260709 12:22:00.971158 31745 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:22:01.016672 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:01.017485 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:01.018148 31824 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:00.779740  2809 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:00.919255 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:01.017485 31819 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:00.976199 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1968) took 1215 ms. Trace:
I20260709 12:22:00.737938  2796 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:22:00.970872  2682 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.046s	user 0.002s	sys 0.012s
I20260709 12:22:01.021497  2809 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:00.807853  2803 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:22:00.926329  2822 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.747s	user 0.000s	sys 0.001s
W20260709 12:22:00.914512  2740 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.698s	user 0.012s	sys 0.000s
I20260709 12:22:00.700106  2646 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:00.845675  2839 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.155s	user 0.000s	sys 0.001s
I20260709 12:22:00.847517 31751 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:00.914273 31744 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260709 12:22:00.892556  2649 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:00.916919  2705 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:00.995136 31822 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:00.922971  2789 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:00.919175  2681 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:00.922545  2790 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:00.765853  2499 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.039336  2682 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.047546 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:01.047660  2682 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.048900  2499 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:00.995307 32369 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:01.050006 32378 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:00.999830 31827 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:22:00.999905  2850 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.050910  2850 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.051491 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:01.052652 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:01.053133  2850 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:01.000479  2660 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:00.916332  2474 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:01.000972  2666 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.003165  2690 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:00.779346  2836 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:22:01.018347 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:00.983179  2793 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.019068 31820 rpcz_store.cc:276] 0709 12:21:59.720080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.720228 (+   148us) service_pool.cc:224] Handling call
0709 12:21:59.721441 (+  1213us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:59.732421 (+ 10980us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.732455 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:59.814241 (+ 81786us) log.cc:844] Serialized 26 byte log entry
0709 12:22:00.149514 (+335273us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:00.149553 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.149624 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:00.935278 (+785654us) raft_consensus.cc:1685] finished
0709 12:22:00.935300 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:00.935447 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":159807,"raft.queue_time_us":5034,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":9467,"thread_start_us":197540,"threads_started":1,"wal-append.queue_time_us":779237}
I20260709 12:22:01.027582  2809 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:01.028215 31821 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 6. (term mismatch)
I20260709 12:22:01.028079  2796 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.031834 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:01.030705  2706 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.004s
I20260709 12:22:01.029042  2803 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:01.035472  2713 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:01.043465 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2066) took 1512 ms. Trace:
I20260709 12:22:01.056160 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:01.053211 31827 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:01.033608  2434 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.013s
I20260709 12:22:01.056892 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:01.043179  2703 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 12:22:01.075642 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:01.050339  2682 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:01.101405  2512 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:01.101064 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2068) took 1256 ms. Trace:
I20260709 12:22:01.103113 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:01.102859 31670 rpcz_store.cc:276] 0709 12:21:59.844952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.040569 (+195617us) service_pool.cc:224] Handling call
0709 12:22:00.124818 (+ 84249us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:00.423561 (+298743us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.423614 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:00.479927 (+ 56313us) log.cc:844] Serialized 26 byte log entry
0709 12:22:00.651554 (+171627us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:00.651598 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.651683 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:01.100860 (+449177us) raft_consensus.cc:1685] finished
0709 12:22:01.100877 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.101004 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72052,"raft.queue_time_us":168312,"spinlock_wait_cycles":33611136,"thread_start_us":105397,"threads_started":1,"wal-append.queue_time_us":511586}
I20260709 12:22:01.036970  2651 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.017s
I20260709 12:22:01.077626 31821 pending_rounds.cc:85] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 5
I20260709 12:22:01.105948 31821 pending_rounds.cc:107] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 7.6
W20260709 12:22:01.083588 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 65309.9us
I20260709 12:22:01.085510 31820 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:01.096457  2681 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:01.107122 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:01.107287  2681 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:01.108387  2681 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:01.109727  2649 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:01.056034  2474 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:01.106386 31821 raft_consensus.cc:2889] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:01.110972  2474 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.110276  2649 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:01.112047  2682 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:01.112323  2649 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:01.118057  2296 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.545s	user 0.011s	sys 0.002s
I20260709 12:22:01.061880  2789 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260709 12:22:01.057070 32343 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:01.119364  2727 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Stepping down as leader of term 9
W20260709 12:22:01.119721 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62739.6us
I20260709 12:22:01.120072  2727 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:01.120352  2789 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.120551 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:01.120913  2789 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.122026  2703 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:01.121403  2727 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:01.095377 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63662.5us
I20260709 12:22:01.079641  2809 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:01.122978  2499 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:01.123461  2499 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:01.123901  2499 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:01.124156  2850 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:01.124194 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 437) took 1379 ms. Trace:
I20260709 12:22:01.058027  2838 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
I20260709 12:22:01.125566  2842 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:01.125821  2727 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:01.079836  2793 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.126683 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:01.127470 31745 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:01.124804 31671 rpcz_store.cc:276] 0709 12:21:59.744476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.849510 (+105034us) service_pool.cc:224] Handling call
0709 12:21:59.901106 (+ 51596us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:21:59.986999 (+ 85893us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:00.222883 (+235884us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.431887 (+209004us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:00.486075 (+ 54188us) log.cc:844] Serialized 52 byte log entry
0709 12:22:00.486268 (+   193us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:00.486529 (+   261us) log.cc:844] Serialized 14 byte log entry
0709 12:22:00.500743 (+ 14214us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.500831 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:01.123938 (+623107us) raft_consensus.cc:1685] finished
0709 12:22:01.123961 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.124126 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":113882,"prepare.queue_time_us":212013,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":85,"raft.queue_time_us":258191,"raft.run_cpu_time_us":465,"raft.run_wall_time_us":25582,"spinlock_wait_cycles":5541632,"thread_start_us":296095,"threads_started":2,"wal-append.queue_time_us":399136}
I20260709 12:22:01.127720 32373 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:22:01.101166  2837 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:01.128584  2793 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:01.128820  2837 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:01.129258  2837 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:01.103327  2796 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:01.044502  2705 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:01.124812  2850 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:01.131196  2825 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.564s	user 0.010s	sys 0.001s
I20260709 12:22:01.131076  2705 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.131704  2825 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.122452  2703 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:01.131910  2850 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.132676  2703 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:01.095180 31669 rpcz_store.cc:276] 0709 12:21:59.531427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.592367 (+ 60940us) service_pool.cc:224] Handling call
0709 12:21:59.639880 (+ 47513us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:59.650097 (+ 10217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.650150 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:59.659180 (+  9030us) log.cc:844] Serialized 26 byte log entry
0709 12:21:59.965781 (+306601us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:21:59.965826 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.965909 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:00.925687 (+959778us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:01.038436 (+112749us) raft_consensus.cc:1685] finished
0709 12:22:01.038458 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.043404 (+  4946us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9944,"raft.run_cpu_time_us":116,"raft.run_wall_time_us":4951,"spinlock_wait_cycles":91543808,"thread_start_us":250890,"threads_started":1,"wal-append.queue_time_us":375392}
I20260709 12:22:01.135512 31744 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:01.130254  2837 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.136891  2809 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.144310  2809 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:01.158367 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:01.159176 32369 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:01.139745 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1963) took 1542 ms. Trace:
W20260709 12:22:01.138790  2814 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.040s	user 0.002s	sys 0.000s
I20260709 12:22:01.141425 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 441) took 1178 ms. Trace:
I20260709 12:22:01.145560 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:01.167165 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:01.146221  2733 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.178162  2793 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:01.147933 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:01.178824  2434 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:01.178228 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.150045 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.179893 31671 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:01.162586 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 533) took 1384 ms. Trace:
I20260709 12:22:01.152040 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.153214 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.176831 31752 rpcz_store.cc:276] 0709 12:21:59.962737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.963216 (+   479us) service_pool.cc:224] Handling call
0709 12:22:01.141153 (+1177937us) spinlock_profiling.cc:243] Waited 498 ms on lock 0x7b54004038d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:01.141303 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1146416896}
I20260709 12:22:01.180828 32378 rpcz_store.cc:276] 0709 12:21:59.778389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.787772 (+  9383us) service_pool.cc:224] Handling call
0709 12:22:01.162349 (+1374577us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b54004272d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:01.162493 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2881815808}
I20260709 12:22:01.181177 31676 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:22:01.137454  2837 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:01.153451  2713 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:01.154899  2846 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:22:01.095619  2740 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:01.163213 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 536) took 1382 ms. Trace:
I20260709 12:22:01.146788  2842 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:01.149257 31670 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:01.095881  2803 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:01.140863  2666 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:01.184041 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.168322 31825 rpcz_store.cc:276] 0709 12:21:59.597276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.638265 (+ 40989us) service_pool.cc:224] Handling call
0709 12:21:59.659791 (+ 21526us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:21:59.804077 (+144286us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.139553 (+1335476us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b54000fc650. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:01.139689 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25773440}
W20260709 12:22:01.186877 31643 connection.cc:256] RPC call timeout handler was delayed by -4.1932206289998248s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1966s for a response
I20260709 12:22:01.192909 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:01.196733 32375 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:01.193987 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:01.195861 32372 rpcz_store.cc:276] 0709 12:21:59.780896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.784562 (+  3666us) service_pool.cc:224] Handling call
0709 12:22:01.163044 (+1378482us) spinlock_profiling.cc:243] Waited 767 ms on lock 0x7b540055da54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:01.163152 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1763654912}
I20260709 12:22:01.199374 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.200486 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:01.165418  2706 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.178431  2733 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:01.224679 31669 raft_consensus.cc:2410] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 10 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:22:01.227795 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.228866 31671 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
W20260709 12:22:01.245805 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45448.6us
I20260709 12:22:01.246726 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:01.248538 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:01.250304 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.251215 31670 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:22:01.254590 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:01.255762 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:01.258088 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 252494us
I20260709 12:22:01.259187 31792 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:01.204468  2654 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:01.178870  2793 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.260571 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:01.267797 31675 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:01.270370 31674 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:22:01.249559  2791 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.192610  2790 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:22:01.253116  2796 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.205780  2740 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.283541 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.286123 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:01.283458 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2075) took 1126 ms. Trace:
I20260709 12:22:01.288473 31750 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:01.289908 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:01.260578  2434 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:01.262737  2809 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:01.293264 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.215765  2706 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:01.299444 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:01.307766 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.296793 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:01.300976 31670 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:01.261090 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:01.296404 31677 rpcz_store.cc:276] 0709 12:22:00.157541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.183378 (+ 25837us) service_pool.cc:224] Handling call
0709 12:22:00.183707 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:00.317365 (+133658us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.317406 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:00.339098 (+ 21692us) log.cc:844] Serialized 26 byte log entry
0709 12:22:00.731259 (+392161us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:00.731298 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:00.731363 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:01.283126 (+551763us) raft_consensus.cc:1685] finished
0709 12:22:01.283155 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.283372 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":115124,"raft.queue_time_us":128000,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":25708,"thread_start_us":230519,"threads_started":1,"wal-append.queue_time_us":579343}
I20260709 12:22:01.268165  2719 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:01.275583  2789 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:01.272730  2825 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:01.328167 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.319402 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.332472 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:01.371454 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 110422us
I20260709 12:22:01.370149 31677 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260709 12:22:01.307045  2731 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:01.378028 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:01.318279  2666 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:01.390497 31672 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:01.390517 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:01.390661 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:01.300359  2796 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.300550  2791 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.407373 31751 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:01.412923 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:01.325304  2649 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.415088  2809 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:01.378360  2719 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:01.388721  2646 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:01.386518  2731 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:01.372522  2844 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:01.441486 31675 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:01.454353 32369 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
W20260709 12:22:01.478451 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 369367us
I20260709 12:22:01.481873 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:01.483460 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:01.484694 31675 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:01.486809 32369 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:01.489108 31673 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
W20260709 12:22:01.492574 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 229919us
I20260709 12:22:01.441067  2646 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:01.447994  2731 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:01.514458 31825 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 10. Ops: []
I20260709 12:22:01.456857  2568 raft_consensus.cc:2764] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:22:01.488926  2666 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.517083  2694 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.407778  2703 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.490530  2296 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:01.517642 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.518419  2646 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:01.546594  2740 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:01.554056 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:01.565326 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:01.565657 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:01.566040 32369 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:01.567852 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:01.555753 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 302730us
W20260709 12:22:01.571048 31643 connection.cc:256] RPC call timeout handler was delayed by -3.2482721850001326s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
W20260709 12:22:01.571496 31643 connection.cc:256] RPC call timeout handler was delayed by -3.1513503759997548s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16180000000000003s for a response
W20260709 12:22:01.571874 31643 connection.cc:256] RPC call timeout handler was delayed by -2.6374503759998333s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2189s for a response
I20260709 12:22:01.572705 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.573570 32375 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:01.573568 31675 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 7: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:22:01.514761 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 193425us
I20260709 12:22:01.574337 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:01.575146 32369 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 9
W20260709 12:22:01.611709 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.545084  2733 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:22:01.627952 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.549576  2713 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:01.635896 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 538) took 1791 ms. Trace:
I20260709 12:22:01.636906 31827 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
W20260709 12:22:01.567024  2792 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.545s	user 0.020s	sys 0.040s
I20260709 12:22:01.636545 32376 rpcz_store.cc:276] 0709 12:21:59.844167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.844299 (+   132us) service_pool.cc:224] Handling call
0709 12:22:01.635600 (+1791301us) spinlock_profiling.cc:243] Waited 823 ms on lock 0x7b54004c4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:01.635801 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1893154176}
I20260709 12:22:01.644466 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:01.563828  2857 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:01.640421 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.632160 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:01.649911 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.655220 31675 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:01.632827 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.642618  2740 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.653522 31824 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260709 12:22:01.656327 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.658288  2740 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.658018 31677 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:22:01.661264 31677 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:22:01.661414  2740 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:01.544891  2666 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.654834 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.687716 32370 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
W20260709 12:22:01.643677  2792 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.712s	user 0.020s	sys 0.041s
I20260709 12:22:01.657794 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:01.672036 32369 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:22:01.638311  2791 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:01.631281  2861 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:01.651893  2434 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:01.705070 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:01.649415  2713 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.564818  2694 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:01.709837 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 75607.5us
I20260709 12:22:01.717983 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:01.719086 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:01.719970 31827 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:01.561002  2719 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.734543 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.735740 31669 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
W20260709 12:22:01.742337 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.730953 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:01.743673 31824 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:01.687916  2793 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:01.734015 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 78724.4us
W20260709 12:22:01.726783 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 172824us
I20260709 12:22:01.732040 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1964) took 2087 ms. Trace:
I20260709 12:22:01.695359  2740 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:01.711277  2860 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.162s	user 0.000s	sys 0.000s
I20260709 12:22:01.703698  2706 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:01.709220  2809 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.761497 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:01.746328 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:01.625962  2296 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.754105 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:01.767566 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:01.768671 31827 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:01.748293  2856 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.369s	user 0.000s	sys 0.001s
I20260709 12:22:01.718345  2702 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:01.798094 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:01.747758  2660 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.690s	user 0.011s	sys 0.003s
I20260709 12:22:01.800556 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:01.802167 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:01.756335  2727 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:01.715729  2681 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:01.811479 31824 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:01.826320 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:01.828274 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:01.818220 31750 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.519s	user 0.006s	sys 0.001s
I20260709 12:22:01.829250 31750 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:22:01.765076  2842 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:01.698454  2857 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.820339 31827 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 6.
I20260709 12:22:01.746203 31826 rpcz_store.cc:276] 0709 12:21:59.644342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.657747 (+ 13405us) service_pool.cc:224] Handling call
0709 12:21:59.682051 (+ 24304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:59.703148 (+ 21097us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.703199 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:59.713676 (+ 10477us) log.cc:844] Serialized 26 byte log entry
0709 12:21:59.856562 (+142886us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:21:59.988396 (+131834us) log.cc:844] Serialized 14 byte log entry
0709 12:22:01.712651 (+1724255us) spinlock_profiling.cc:243] Waited 5.91 ms on lock 0x7b180008d880. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8fc98bd70 00007fe8fc97f293 00007fe8fc974cea 00007fe8fc97523a 00007fe8fcddfa2a 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:22:01.712950 (+   299us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.713061 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:01.713123 (+    62us) raft_consensus.cc:1685] finished
0709 12:22:01.731748 (+ 18625us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:01.731957 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1535690,"prepare.queue_time_us":134277,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":78,"raft.queue_time_us":2642,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":14711,"spinlock_wait_cycles":13598976,"thread_start_us":142388,"threads_started":1,"wal-append.queue_time_us":263894}
I20260709 12:22:01.773809  2649 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:01.812763  2820 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:22:01.774433  2863 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:01.804268  2796 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:01.876255 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:01.813261  2706 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:01.759574  2713 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.832607  2794 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:01.834226  2727 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:01.716493  2861 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:01.857545  2869 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.294s	user 0.000s	sys 0.001s
I20260709 12:22:01.772921  2740 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.900070 31824 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:22:01.860769  2733 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:01.901988 31826 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:01.902766 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 70964.4us
W20260709 12:22:01.903005 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 156915us
I20260709 12:22:01.800071  2809 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:01.881529  2860 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:22:01.830873  2660 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10.
I20260709 12:22:01.909335 31827 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:22:01.910969 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:01.897166 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 93044.4us
I20260709 12:22:01.901429  2727 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:01.931595 31824 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:22:01.857239  2877 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260709 12:22:01.933626 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:01.875337  2681 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:01.908457  2733 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:01.909201  2872 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:01.901775  2702 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.028s
W20260709 12:22:01.933156  2858 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.355s	user 0.000s	sys 0.001s
I20260709 12:22:01.950124 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:01.958526 31669 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:01.918865  2649 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:01.889364  2881 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.094s
I20260709 12:22:01.962751 31824 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260709 12:22:01.890009  2434 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:01.946610  2733 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:01.962152  2794 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:01.925235  2863 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.051s
I20260709 12:22:01.968293  2666 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:01.913949  2870 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 12:22:01.942997  2660 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.034s
W20260709 12:22:02.036445 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 110707us
I20260709 12:22:01.911911  2876 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.085s
I20260709 12:22:02.083249 31819 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:22:01.965811  2706 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:02.057399  2883 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.225s	user 0.001s	sys 0.000s
I20260709 12:22:02.088449 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:02.074651  2790 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:02.073253 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2079) took 1222 ms. Trace:
W20260709 12:22:02.026429  2837 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.831s	user 0.008s	sys 0.006s
I20260709 12:22:02.019272  2850 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:01.977841  2649 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:02.064960  2890 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260709 12:22:02.095341 31748 rpcz_store.cc:276] 0709 12:22:00.850476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.850677 (+   201us) service_pool.cc:224] Handling call
0709 12:22:00.851120 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:00.975317 (+124197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:00.975363 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:00.998930 (+ 23567us) log.cc:844] Serialized 26 byte log entry
0709 12:22:01.295569 (+296639us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:01.295626 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.295720 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:02.072949 (+777229us) raft_consensus.cc:1685] finished
0709 12:22:02.072981 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:02.073174 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":123625,"raft.queue_time_us":129956,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":14787,"thread_start_us":261293,"threads_started":1,"wal-append.queue_time_us":426640}
I20260709 12:22:02.051887  2703 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:02.078588  2690 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.003s	user 0.007s	sys 0.002s
I20260709 12:22:02.133531  2740 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:02.134227  2740 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.134747  2740 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:02.137780  2740 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:02.068698  2713 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:02.083786  2296 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:02.140797 31822 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:02.095054  2790 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:02.103289  2512 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:02.249300  2887 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:22:02.064010  2880 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.352s
W20260709 12:22:02.251556  2731 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.712s	user 0.011s	sys 0.002s
I20260709 12:22:02.251673  2880 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.252065  2905 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.252310  2880 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:02.252727  2905 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:02.254905  2790 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:02.256218  2706 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:02.260165  2646 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.635s	user 0.008s	sys 0.005s
I20260709 12:22:02.260738  2646 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:02.263762  2861 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:02.265802  2842 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:02.267400  2809 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:02.269572  2894 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:22:02.069926  2794 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:02.270938  2899 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:02.271600  2899 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:02.272015  2899 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:02.273790 31675 raft_consensus.cc:2368] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10.
W20260709 12:22:02.285121 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:02.285784 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:02.303179 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:02.313138 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 395852us
W20260709 12:22:02.251559  2803 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.036s	user 0.008s	sys 0.004s
I20260709 12:22:02.410209 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:02.423548 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:02.424559 31676 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
W20260709 12:22:02.429019 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 368743us
I20260709 12:22:02.451181 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:02.312145  2842 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:02.473330 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:02.252200  2731 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:02.522442 31818 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.587s	user 0.008s	sys 0.003s
I20260709 12:22:02.501224 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 538) took 1465 ms. Trace:
I20260709 12:22:02.522966 31818 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:02.523312 32375 rpcz_store.cc:276] 0709 12:22:01.035716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.412612 (+376896us) service_pool.cc:224] Handling call
0709 12:22:02.488103 (+1075491us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b5400595454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:02.501123 (+ 13020us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":349094656}
W20260709 12:22:02.470542  2499 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.327s	user 0.020s	sys 0.005s
I20260709 12:22:02.274611  2512 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:22:02.547178 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 116585us
I20260709 12:22:02.482090  2850 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:02.484705  2842 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:02.430271  2803 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:02.575233  2789 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.582769  2789 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:02.583914 32378 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:02.543159  2905 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:02.577611 32342 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.638s (SENT) [suppressed 7 similar messages]
I20260709 12:22:02.584957  2850 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:02.587198  2850 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:02.588536  2727 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.645s	user 0.007s	sys 0.002s
I20260709 12:22:02.589062  2727 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:02.589924  2733 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.555s	user 0.008s	sys 0.004s
I20260709 12:22:02.590443  2733 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:02.592002  2731 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:02.593747  2727 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:02.542461  2646 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:02.598619 31669 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.634s	user 0.011s	sys 0.000s
I20260709 12:22:02.560432  2838 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:02.601269  2733 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:02.583225  2803 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:02.607645  2789 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:02.608695  2842 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:02.609338  2649 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:02.615149 31669 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:02.626482 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:02.629899 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:02.631893 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:02.645792  2646 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:02.647135 31819 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.008s	sys 0.008s
W20260709 12:22:02.558661 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:02.693302 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 45617.5us
W20260709 12:22:02.742179 31819 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.008s	sys 0.008s
I20260709 12:22:02.544571  2499 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:02.728613  2913 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:22:02.692730  2880 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:02.730062 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.906s	user 0.014s	sys 0.006s
W20260709 12:22:02.762135 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.015s	sys 0.006s
W20260709 12:22:02.730933 31826 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.005s	sys 0.007s
I20260709 12:22:02.721645 31676 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
W20260709 12:22:02.763008 31826 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.005s	sys 0.007s
I20260709 12:22:02.743356 31674 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
W20260709 12:22:02.745424 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 113645us
I20260709 12:22:02.750707 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:02.622238  2838 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:02.743439 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:02.762593 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 948409us
I20260709 12:22:02.768312 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:02.772948 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:02.778508 32369 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:02.775802  2703 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:02.780356  2812 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 2.992s	user 0.000s	sys 0.025s
I20260709 12:22:02.780220  2703 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:02.782001 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 525) took 3278 ms. Trace:
I20260709 12:22:02.783072  2703 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:02.783082  2740 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:02.760716  2649 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:02.766526  2500 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:02.798171  2500 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:02.800503  2500 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:02.801122  2500 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:02.787258  2740 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:02.803143  2740 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:02.691768  2850 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.084s
I20260709 12:22:02.773525  2512 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:02.807415  2512 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:02.808100  2512 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:02.808619  2912 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:02.782419 32377 rpcz_store.cc:276] 0709 12:21:59.503326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.503514 (+   188us) service_pool.cc:224] Handling call
0709 12:21:59.503986 (+   472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:59.504638 (+   652us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.504671 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:59.505270 (+   599us) log.cc:844] Serialized 26 byte log entry
0709 12:21:59.674270 (+169000us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:21:59.674316 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.674395 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:02.744723 (+3070328us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:02.781629 (+ 36906us) raft_consensus.cc:1685] finished
0709 12:22:02.781655 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:02.781911 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5038,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":152,"spinlock_wait_cycles":38574336,"thread_start_us":168520,"threads_started":1,"wal-append.queue_time_us":282085}
I20260709 12:22:02.790721  2649 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:02.810490  2512 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:02.812119  2649 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:02.813347 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1483) took 1242 ms. Trace:
I20260709 12:22:02.813834 31675 rpcz_store.cc:276] 0709 12:22:01.571023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.656017 (+ 84994us) service_pool.cc:224] Handling call
0709 12:22:02.813109 (+1157092us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x7b5400391cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:02.813269 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1411736320}
I20260709 12:22:02.772154  2914 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:02.795538 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:02.816560 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:02.744902  2646 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:02.829376  2838 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:02.832248  2914 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:02.838423 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2081) took 1592 ms. Trace:
I20260709 12:22:02.839715 31678 rpcz_store.cc:276] 0709 12:22:01.246042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.260117 (+ 14075us) service_pool.cc:224] Handling call
0709 12:22:01.273222 (+ 13105us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.449726 (+176504us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:22:01.449755 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:01.449808 (+    53us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:01.449845 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.838160 (+1388315us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b5400254ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:02.838210 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:02.838356 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16419,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":7280,"spinlock_wait_cycles":405204864}
W20260709 12:22:02.840642 31744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2084) took 1738 ms (client timeout 1699 ms). Trace:
W20260709 12:22:02.841073 31744 rpcz_store.cc:269] 0709 12:22:01.102255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.102813 (+   558us) service_pool.cc:224] Handling call
0709 12:22:02.840408 (+1737595us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b54002134d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:02.840560 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":72210304}
W20260709 12:22:02.847167  2806 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 3.060s	user 0.013s	sys 0.006s
W20260709 12:22:02.849098  2831 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 2.662s	user 0.008s	sys 0.009s
W20260709 12:22:02.850054 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:02.851039 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.852154  2914 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:02.852972 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:02.854514 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:02.855436 31748 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:22:02.847247  2160 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:02.855978  2160 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:02.856601  2912 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:02.857059  2912 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:02.856809  2160 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:02.857445  2912 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:02.858168  2160 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:02.858932  2911 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.859476  2911 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:02.860960  2818 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 2.680s	user 0.009s	sys 0.020s
I20260709 12:22:02.861938  2911 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:02.862562 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 527) took 3358 ms. Trace:
I20260709 12:22:02.862914  2911 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.855465 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:02.863034 32371 rpcz_store.cc:276] 0709 12:21:59.504210 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.505918 (+  1708us) service_pool.cc:224] Handling call
0709 12:21:59.506349 (+   431us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:21:59.506896 (+   547us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:21:59.506941 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:21:59.507519 (+   578us) log.cc:844] Serialized 26 byte log entry
0709 12:21:59.729200 (+221681us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:21:59.729246 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:21:59.729318 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:02.859997 (+3130679us) spinlock_profiling.cc:243] Waited 27.1 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:02.862139 (+  2142us) raft_consensus.cc:1685] finished
0709 12:22:02.862168 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:02.862378 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3131,"raft.run_cpu_time_us":58,"raft.run_wall_time_us":58,"spinlock_wait_cycles":62275840,"thread_start_us":221258,"threads_started":1,"wal-append.queue_time_us":506441}
I20260709 12:22:02.866008  2825 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:02.866540  2825 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:02.867466  2825 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:02.868643  2899 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.596s	user 0.012s	sys 0.000s
I20260709 12:22:02.869233  2899 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:02.870273 31677 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:02.870479 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.864110  2705 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:02.871582  2803 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.872071  2803 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:02.874244  2803 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:02.875521  2877 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:02.875958  2877 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:02.876462  2499 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:02.876686 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.876847  2877 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:02.878109 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.878876 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:02.883857 31748 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:02.837703  2905 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:02.895799  2905 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:02.896255  2646 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:02.864387  2911 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:02.896893  2905 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:02.898622  2911 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:02.908397  2899 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:02.908982 31747 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:02.910204 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:02.911594  2713 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.896279 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:02.912009  2694 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:02.912240  2713 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:02.911593 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:02.908385 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:02.914238 31747 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:02.910220 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2079) took 1683 ms. Trace:
I20260709 12:22:02.915128 31673 rpcz_store.cc:276] 0709 12:22:01.227013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.227147 (+   134us) service_pool.cc:224] Handling call
0709 12:22:01.262880 (+ 35733us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.489645 (+226765us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.909990 (+1420345us) spinlock_profiling.cc:243] Waited 223 ms on lock 0x7b5400046fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:02.910151 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":512618368}
I20260709 12:22:02.916765  2877 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:02.917969 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 445) took 1980 ms. Trace:
I20260709 12:22:02.918336 31753 rpcz_store.cc:276] 0709 12:22:00.937199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.013783 (+ 76584us) service_pool.cc:224] Handling call
0709 12:22:01.014035 (+   252us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:02.917135 (+1903100us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:02.917166 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:02.917210 (+    44us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:02.917266 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.917758 (+   492us) spinlock_profiling.cc:243] Waited 1.9 s on lock 0x7b54004033d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:02.917801 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:02.917917 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4376207744}
I20260709 12:22:02.922943 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:02.923782 31746 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:02.924680 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1977) took 2013 ms. Trace:
I20260709 12:22:02.925117 31825 rpcz_store.cc:276] 0709 12:22:00.911281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.204238 (+292957us) service_pool.cc:224] Handling call
0709 12:22:01.217595 (+ 13357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.519140 (+301545us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.924449 (+1405309us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54001969d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:02.924620 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":682100480}
I20260709 12:22:02.914290 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1971) took 3117 ms. Trace:
I20260709 12:22:02.927462  1897 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:02.927203  2551 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:02.927956  1897 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:02.928653  2551 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:02.930068  1897 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:02.931562 31676 raft_consensus.cc:2410] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:22:02.926438 31821 rpcz_store.cc:276] 0709 12:21:59.796548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:59.999446 (+202898us) service_pool.cc:224] Handling call
0709 12:22:00.014422 (+ 14976us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.111624 (+1097202us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.914044 (+1802420us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54004470d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:02.914226 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2330468736}
I20260709 12:22:02.931073  2794 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:02.924746  2838 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:02.944902 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 367320us
I20260709 12:22:02.918006  2803 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:22:02.943748 32370 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.009s	sys 0.018s
I20260709 12:22:02.957054 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:02.957406 32370 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.545s	user 0.010s	sys 0.018s
I20260709 12:22:02.966427 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:02.946205  2500 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:02.971971 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:02.976019  2819 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 2.974s	user 0.000s	sys 0.018s
I20260709 12:22:02.976128 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:02.977669 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:02.979024  2296 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:02.981823  2841 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 2.173s	user 0.011s	sys 0.026s
I20260709 12:22:02.987712 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:02.992476 31824 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.867s	user 0.027s	sys 0.046s
I20260709 12:22:02.974814 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1996) took 1311 ms. Trace:
W20260709 12:22:02.993006 31824 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.868s	user 0.027s	sys 0.046s
I20260709 12:22:02.992690 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:02.993669 31672 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:02.943259  2794 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:02.996482 31677 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:02.997718 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:03.000252 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:02.996613 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:03.001616 31821 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:22:02.997642 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:03.002517 31678 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:02.948426  2649 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:03.031378 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:03.031654 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:02.993224 31819 rpcz_store.cc:276] 0709 12:22:01.663457 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:02.110027 (+446570us) service_pool.cc:224] Handling call
0709 12:22:02.139121 (+ 29094us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:02.139424 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:02.742996 (+603572us) log.cc:844] Serialized 14 byte log entry
0709 12:22:02.974402 (+231406us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:02.974461 (+    59us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:02.974510 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.974581 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:02.974743 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":444395,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":4621,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":144,"thread_start_us":829353,"threads_started":2}
I20260709 12:22:03.028650 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:03.032433 31821 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:03.033233 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:03.033835 31819 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:03.033891 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:03.035164 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:03.035846 31821 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:03.036604 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:03.037165 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.037583 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:03.033234 32377 raft_consensus.cc:2410] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 11 because replica has last-logged OpId of term: 9 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:22:03.039848 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:02.959591  2803 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:03.042943 32371 raft_consensus.cc:1813] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Handling vote request from an unknown peer 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:03.042325 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:03.043556 32371 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 12:22:03.045397 31821 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
W20260709 12:22:03.043493 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 382340us
I20260709 12:22:03.047873 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:03.048633 31753 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:03.048741 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:03.055018 31821 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:22:03.054898 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:03.057348 31746 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:03.055253 31748 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:03.067318 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:03.068480  2803 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:03.069008 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:03.070286 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:03.066431 31746 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:22:03.071224 32375 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:22:03.072535 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:03.075537 32371 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:22:03.081766 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 260931us
I20260709 12:22:03.043588 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:03.086716  1897 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:22:03.089931 31746 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260709 12:22:03.088541 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.091183  1897 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:03.092176 31819 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:22:03.096083  1897 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:03.101540 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:03.072332  2740 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:03.102905 31821 raft_consensus.cc:2368] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10.
I20260709 12:22:03.103922  2740 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:03.104208 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.074589  2912 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:03.108263 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:03.108960 31748 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:03.084427 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 317679us
I20260709 12:22:03.097675  2803 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:03.112360 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:03.114603 31819 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:03.095533  2682 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:03.122956 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:03.132334 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 381717us
I20260709 12:22:03.126396 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:03.089833  2694 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.025s
I20260709 12:22:03.095387  2842 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260709 12:22:03.095144  2731 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.026s
I20260709 12:22:03.150523 31747 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:22:03.133296 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:03.120508  2803 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.164433 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:03.173485 31746 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:03.167441 31791 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:03.179018 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.182452 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:03.189702  2929 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:03.145712  2500 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:03.192610  2500 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:03.145835  2666 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:03.197674 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.163067  2877 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.181320  2803 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:03.179616  2842 consensus_queue.cc:1243] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:22:03.205324 31747 raft_consensus.cc:2368] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10.
I20260709 12:22:03.144775  2682 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:03.183799  2914 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:03.220973 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:03.205117 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 71918.9us
I20260709 12:22:03.228751 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:03.196592  2927 raft_consensus.cc:2764] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:22:03.231675 32377 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:22:03.227450 31749 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:03.224121  2649 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:03.230441  2740 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:03.252599 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.201016  2666 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:03.254947 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:03.256613 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:03.249539 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 67171.4us
I20260709 12:22:03.219473  2877 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:03.274012 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.274765 31747 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:22:03.276674 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.277537 31749 raft_consensus.cc:1813] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Handling vote request from an unknown peer 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:03.278252 31749 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:03.280331 31675 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:22:03.254436  2740 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:03.289803 31819 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:03.250208  2928 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:03.300483 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:03.423628 31678 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:22:03.301496 31745 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:03.303646 31672 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
W20260709 12:22:03.304589 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 49776.5us
I20260709 12:22:03.273010  2930 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:03.300623 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.271992  2927 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:03.316942  2853 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 2.034s	user 0.009s	sys 0.010s
I20260709 12:22:03.321825  2929 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:03.326581 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.327005 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:03.450124 31675 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
W20260709 12:22:03.318006  2920 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:22:03.301534  2740 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:03.455265 31746 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:22:03.456363 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:03.457279 31678 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:03.460697 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:03.461577 31744 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:03.465619 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:03.466564 31819 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 8
W20260709 12:22:03.336519 31818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 413) took 2532 ms (client timeout 2148 ms). Trace:
I20260709 12:22:03.342054 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:03.328697  2815 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 3.133s	user 0.005s	sys 0.011s
W20260709 12:22:03.470176 31818 rpcz_store.cc:269] 0709 12:22:00.804095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.846145 (+ 42050us) service_pool.cc:224] Handling call
0709 12:22:03.336248 (+2490103us) spinlock_profiling.cc:243] Waited 919 ms on lock 0x7b5400197dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:03.336416 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2114706304}
W20260709 12:22:03.483310 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:03.390486 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 253941us
W20260709 12:22:03.486130 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.400622 31753 raft_consensus.cc:2410] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
W20260709 12:22:03.373133  2937 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
W20260709 12:22:03.417351 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.519706 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.520874 31747 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:03.521342 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:03.447141 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:03.528239 31672 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:22:03.453110 31748 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
W20260709 12:22:03.331705 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.543818 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.547473 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.551827 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:03.591187 32373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 554) took 2682 ms (client timeout 1819 ms). Trace:
W20260709 12:22:03.591832 32373 rpcz_store.cc:269] 0709 12:22:00.908494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.918966 (+ 10472us) service_pool.cc:224] Handling call
0709 12:22:03.590827 (+2671861us) spinlock_profiling.cc:243] Waited 109 ms on lock 0x7b540071d4d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:03.591033 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":250168704}
I20260709 12:22:03.489334 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.602139 31746 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:03.517187  2877 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:03.448832  2930 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:03.682446 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:03.683430 31747 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:03.519656  2912 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:03.517251  2682 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:03.733433 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.975366 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.978214 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.875267  2930 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:03.928442  2551 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.999s	user 0.010s	sys 0.006s
W20260709 12:22:03.940805  2909 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 12:22:03.986027 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1992) took 2329 ms. Trace:
I20260709 12:22:03.986546 31827 rpcz_store.cc:276] 0709 12:22:01.662099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.916885 (+254786us) service_pool.cc:224] Handling call
0709 12:22:01.917219 (+   334us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:01.948515 (+ 31296us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:01.948566 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:01.969817 (+ 21251us) log.cc:844] Serialized 52 byte log entry
0709 12:22:02.369228 (+399411us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:02.774554 (+405326us) log.cc:844] Serialized 14 byte log entry
0709 12:22:03.134951 (+360397us) log.cc:844] Serialized 14 byte log entry
0709 12:22:03.135167 (+   216us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:03.135258 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:03.593636 (+458378us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b4800130158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:22:03.771612 (+177976us) spinlock_profiling.cc:243] Waited 8.81 ms on lock 0x7b18004caec0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:22:03.985707 (+214095us) raft_consensus.cc:1685] finished
0709 12:22:03.985739 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:03.985945 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4830,"prepare.queue_time_us":225477,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":75,"raft.queue_time_us":153862,"raft.run_cpu_time_us":19645,"raft.run_wall_time_us":1548103,"spinlock_wait_cycles":104813440,"thread_start_us":1010040,"threads_started":3,"wal-append.queue_time_us":774811}
I20260709 12:22:03.989742 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:03.991856 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:03.950615 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1984) took 2938 ms (client timeout 1806 ms). Trace:
I20260709 12:22:03.993850 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:03.993948 31822 rpcz_store.cc:269] 0709 12:22:01.012673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.047261 (+ 34588us) service_pool.cc:224] Handling call
0709 12:22:03.950327 (+2903066us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b540043e9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:03.950512 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2512526720}
I20260709 12:22:03.995038 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 431) took 1114 ms. Trace:
I20260709 12:22:03.996459 31827 rpcz_store.cc:276] 0709 12:22:02.880238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.993492 (+1113254us) service_pool.cc:224] Handling call
0709 12:22:03.994934 (+  1442us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:03.998478 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.000201 31827 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:04.001024 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53381.1us
W20260709 12:22:04.004261 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:03.949507 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:03.948710 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:03.973003 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:04.008985 31749 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:04.013041 31573 catalog_manager.cc:5697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 4 to 6, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 6 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:04.021862 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 545) took 1936 ms. Trace:
W20260709 12:22:04.068845 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 174722us
I20260709 12:22:04.066030  2825 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:04.081216 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.081727 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:04.082141 31818 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:04.081135  2825 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.081310 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:04.082798 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 429) took 1203 ms. Trace:
I20260709 12:22:04.082470 32375 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:04.083449 31818 rpcz_store.cc:276] 0709 12:22:02.879797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.623252 (+743455us) service_pool.cc:224] Handling call
0709 12:22:04.082718 (+459466us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":272640}
I20260709 12:22:04.084468 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.084823 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:04.085361 31822 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:04.085575 32375 raft_consensus.cc:2368] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10.
I20260709 12:22:04.131803 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:04.140095 31792 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.937s	user 0.059s	sys 0.081s
W20260709 12:22:04.140542 31792 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.059s	sys 0.081s
W20260709 12:22:04.140861 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 941652us
I20260709 12:22:04.152845 32375 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:04.155174 31744 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.668s	user 0.008s	sys 0.003s
I20260709 12:22:04.155627 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2008) took 1096 ms. Trace:
I20260709 12:22:04.155792 31744 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
W20260709 12:22:04.156385 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.156466 31822 rpcz_store.cc:276] 0709 12:22:03.058851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.996108 (+937257us) service_pool.cc:224] Handling call
0709 12:22:04.155448 (+159340us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.155193 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:04.157622 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:04.158290 31822 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 10
W20260709 12:22:04.159283 31672 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.630s	user 0.011s	sys 0.000s
I20260709 12:22:04.159789 31672 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:04.161767 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2125) took 1573 ms. Trace:
I20260709 12:22:04.161921 31747 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:04.162349 31827 rpcz_store.cc:276] 0709 12:22:02.588206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.998003 (+1409797us) service_pool.cc:224] Handling call
0709 12:22:04.161693 (+163690us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.161037 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:04.164095 31672 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:22:04.022490 32370 rpcz_store.cc:276] 0709 12:22:02.085813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:02.086209 (+   396us) service_pool.cc:224] Handling call
0709 12:22:02.113472 (+ 27263us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:02.142688 (+ 29216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:02.142746 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:02.181392 (+ 38646us) log.cc:844] Serialized 26 byte log entry
0709 12:22:02.958185 (+776793us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:02.958249 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.958358 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:04.019443 (+1061085us) raft_consensus.cc:1685] finished
0709 12:22:04.019471 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:04.021756 (+  2285us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":200259,"raft.queue_time_us":122,"raft.run_cpu_time_us":2049,"raft.run_wall_time_us":37272,"thread_start_us":545949,"threads_started":1,"wal-append.queue_time_us":851060}
I20260709 12:22:04.013042  2861 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.167820 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1579) took 1280 ms. Trace:
I20260709 12:22:04.168329 31745 rpcz_store.cc:276] 0709 12:22:02.887244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:02.912694 (+ 25450us) service_pool.cc:224] Handling call
0709 12:22:02.912992 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:03.424743 (+511751us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.167479 (+742736us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b54000f2dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:04.167752 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":893006720}
I20260709 12:22:04.178754 31819 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:22:04.179539 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 428) took 1300 ms. Trace:
I20260709 12:22:04.179737 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:04.180099 31819 rpcz_store.cc:276] 0709 12:22:02.879577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.465268 (+585691us) service_pool.cc:224] Handling call
0709 12:22:04.179473 (+714205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.180574 31818 raft_consensus.cc:2376] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10. Re-sending same reply.
I20260709 12:22:04.181238 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 435) took 1085 ms. Trace:
I20260709 12:22:04.181620 31818 rpcz_store.cc:276] 0709 12:22:03.095615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.153174 (+1057559us) service_pool.cc:224] Handling call
0709 12:22:04.181168 (+ 27994us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.081780  2500 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:04.184406 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.185426 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.190770 31819 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:04.183681 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:04.190860 31827 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260709 12:22:04.191318 32375 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:22:04.188448 31822 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:04.192240 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2006) took 1139 ms. Trace:
I20260709 12:22:04.193709 31822 rpcz_store.cc:276] 0709 12:22:03.053057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.157372 (+1104315us) service_pool.cc:224] Handling call
0709 12:22:04.192171 (+ 34799us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.195325 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.195914 31822 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:04.196367 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2002) took 1146 ms. Trace:
I20260709 12:22:04.197553 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:04.166559  2929 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.198774 32375 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 9. Ops: []
I20260709 12:22:04.200125 31822 rpcz_store.cc:276] 0709 12:22:03.049697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.195054 (+1145357us) service_pool.cc:224] Handling call
0709 12:22:04.196317 (+  1263us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:04.200788 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:04.201999 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.203671 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.206600 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 1999) took 1326 ms. Trace:
I20260709 12:22:04.203678 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:04.207647 31819 rpcz_store.cc:276] 0709 12:22:02.880472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.184836 (+1304364us) service_pool.cc:224] Handling call
0709 12:22:04.206540 (+ 21704us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.208758 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 427) took 1329 ms. Trace:
I20260709 12:22:04.209192 31827 rpcz_store.cc:276] 0709 12:22:02.879314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.183116 (+1303802us) service_pool.cc:224] Handling call
0709 12:22:04.208502 (+ 25386us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.199939  2945 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:04.214804 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:04.215524 32375 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: [8.8-8.8]
I20260709 12:22:04.215876 31747 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:04.182125  2861 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:04.226701 31745 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:22:04.227046 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:04.204360  2682 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:04.204003  2690 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:04.211668  2906 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:04.236413  2945 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.243099 31672 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:04.244117 31672 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 7 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:04.245272  2803 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.015s	user 0.011s	sys 0.002s
W20260709 12:22:04.246251  2740 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.704s	user 0.011s	sys 0.000s
I20260709 12:22:04.209293  2929 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.245853  2803 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:04.247592 31820 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.722s	user 0.008s	sys 0.002s
I20260709 12:22:04.256547 31820 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
W20260709 12:22:04.257272 31820 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2118) took 1671 ms (client timeout 2205 ms). Trace:
W20260709 12:22:04.257719 31820 rpcz_store.cc:269] 0709 12:22:02.585787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.518485 (+932698us) service_pool.cc:224] Handling call
0709 12:22:04.257195 (+738710us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.245239 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:04.259384 31819 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:04.260464 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2007) took 1203 ms. Trace:
I20260709 12:22:04.260946 31819 rpcz_store.cc:276] 0709 12:22:03.056685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.236246 (+1179561us) service_pool.cc:224] Handling call
0709 12:22:04.260389 (+ 24143us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.269155 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1998) took 2605 ms. Trace:
I20260709 12:22:04.269886 31820 rpcz_store.cc:276] 0709 12:22:01.664000 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.268050 (+2604050us) service_pool.cc:224] Handling call
0709 12:22:04.268477 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:04.268713 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:04.268753 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:04.268813 (+    60us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:04.268859 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.268930 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:04.269079 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.272076 31820 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:22:04.273113 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2129) took 1683 ms. Trace:
I20260709 12:22:04.273607 31820 rpcz_store.cc:276] 0709 12:22:02.589512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.271326 (+1681814us) service_pool.cc:224] Handling call
0709 12:22:04.271758 (+   432us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:04.272787 (+  1029us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.273043 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.262727 31672 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.246721  2740 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:04.281912 31672 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 9
I20260709 12:22:04.276082  2930 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.283036 32371 raft_consensus.cc:2368] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
I20260709 12:22:04.237087  2690 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.283383  2930 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.293732  2790 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:04.236760  2682 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:04.299932 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:04.300843 31745 raft_consensus.cc:2410] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:22:04.296280 31819 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 7. Current term is 9. Ops: []
I20260709 12:22:04.302250 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2112) took 1904 ms. Trace:
I20260709 12:22:04.302562 31819 rpcz_store.cc:276] 0709 12:22:02.397629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.277000 (+1879371us) service_pool.cc:224] Handling call
0709 12:22:04.296028 (+ 19028us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:04.301997 (+  5969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.302197 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.308575 31818 raft_consensus.cc:2410] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 4.
I20260709 12:22:04.312731 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:04.313534 31745 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:04.261902  2906 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:04.315296 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.307461  2500 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.316412 31749 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:22:04.306722  2825 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:04.321671 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:04.330230 31745 raft_consensus.cc:2376] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10. Re-sending same reply.
I20260709 12:22:04.330828 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1606) took 1006 ms. Trace:
W20260709 12:22:04.315438 31643 connection.cc:256] RPC call timeout handler was delayed by -2.0308843120001256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1567s for a response
W20260709 12:22:04.331523 31643 connection.cc:256] RPC call timeout handler was delayed by -2.0191843120001067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15800000000000003s for a response
W20260709 12:22:04.332149 31643 connection.cc:256] RPC call timeout handler was delayed by -2.0020843120000791s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15990000000000001s for a response
I20260709 12:22:04.331480 31749 rpcz_store.cc:276] 0709 12:22:03.324435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.008028 (+683593us) service_pool.cc:224] Handling call
0709 12:22:04.008721 (+   693us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:04.330419 (+321698us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.330750 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:04.341866 31643 connection.cc:256] RPC call timeout handler was delayed by -1.9111843119999321s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17s for a response
I20260709 12:22:04.341116 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:04.342298 31643 connection.cc:256] RPC call timeout handler was delayed by -1.8364843120002661s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17830000000000001s for a response
W20260709 12:22:04.342749 31643 connection.cc:256] RPC call timeout handler was delayed by -1.8337843120002617s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1786s for a response
W20260709 12:22:04.343094 31643 connection.cc:256] RPC call timeout handler was delayed by -1.8238843120002457s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1797s for a response
W20260709 12:22:04.344044 31643 connection.cc:256] RPC call timeout handler was delayed by -1.7860843120001846s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1839s for a response
W20260709 12:22:04.344440 31643 connection.cc:256] RPC call timeout handler was delayed by -1.7698843120001584s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1857s for a response
I20260709 12:22:04.344129 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:04.345013 31674 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:04.350754  2857 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:04.352026 31643 connection.cc:256] RPC call timeout handler was delayed by -1.7635843120001482s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1864s for a response
I20260709 12:22:04.354920 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1507) took 1025 ms. Trace:
I20260709 12:22:04.356948 31672 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:04.355610 31678 rpcz_store.cc:276] 0709 12:22:03.329381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.484601 (+155220us) service_pool.cc:224] Handling call
0709 12:22:03.549702 (+ 65101us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:04.013837 (+464135us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:04.013869 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:04.144169 (+130300us) log.cc:844] Serialized 26 byte log entry
0709 12:22:04.314462 (+170293us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:04.314516 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.314602 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:04.354593 (+ 39991us) raft_consensus.cc:1685] finished
0709 12:22:04.354643 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:04.354833 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":396592,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":106814,"thread_start_us":134953,"threads_started":1,"wal-append.queue_time_us":160450}
W20260709 12:22:04.355191 31643 connection.cc:256] RPC call timeout handler was delayed by -1.7356843120001031s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1895s for a response
W20260709 12:22:04.358163 31643 connection.cc:256] RPC call timeout handler was delayed by -1.6582843119999779s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19810000000000003s for a response
W20260709 12:22:04.358578 31643 connection.cc:256] RPC call timeout handler was delayed by -1.6474843119999605s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19930000000000003s for a response
W20260709 12:22:04.358959 31643 connection.cc:256] RPC call timeout handler was delayed by -1.5889843120003206s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20579999999999998s for a response
W20260709 12:22:04.359454 31643 connection.cc:256] RPC call timeout handler was delayed by -1.5187843120002071s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2136s for a response
W20260709 12:22:04.361183 31643 connection.cc:256] RPC call timeout handler was delayed by -1.4566843120001067s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22050000000000003s for a response
W20260709 12:22:04.361565 31643 connection.cc:256] RPC call timeout handler was delayed by -1.2415843120002137s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2444s for a response
I20260709 12:22:04.362593 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:04.360327  2857 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.367764  2927 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:04.381808  2893 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 2.298s	user 0.011s	sys 0.039s
I20260709 12:22:04.382560 31674 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:22:04.369851 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:04.383961  2945 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:04.386961  2929 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:04.387327  2927 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:04.389173 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:04.391963 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2087) took 3064 ms. Trace:
I20260709 12:22:04.394532 31752 rpcz_store.cc:276] 0709 12:22:01.327165 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.327301 (+   136us) service_pool.cc:224] Handling call
0709 12:22:01.327628 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:01.624819 (+297191us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:01.624859 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:01.649058 (+ 24199us) log.cc:844] Serialized 26 byte log entry
0709 12:22:01.841025 (+191967us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:01.841078 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.841161 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:04.380901 (+2539740us) spinlock_profiling.cc:243] Waited 5.92 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:04.391616 (+ 10715us) raft_consensus.cc:1685] finished
0709 12:22:04.391640 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:04.391869 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1456,"raft.queue_time_us":135702,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":4606,"spinlock_wait_cycles":13620480,"thread_start_us":307058,"threads_started":2,"wal-append.queue_time_us":275680}
I20260709 12:22:04.396037 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:04.392467 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:04.396984 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:04.393237 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:04.384979  2930 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:04.339679  2500 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.393937 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:04.392850  2718 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:04.395610  2857 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.398386  2803 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:04.410671 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 430) took 1530 ms. Trace:
I20260709 12:22:04.395354  2929 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.403982 31753 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:04.407140 31749 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:04.404670  2682 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:04.393260  2912 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.411293 31820 rpcz_store.cc:276] 0709 12:22:02.880027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.274684 (+1394657us) service_pool.cc:224] Handling call
0709 12:22:04.275050 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:04.275318 (+   268us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:04.360244 (+ 84926us) log.cc:844] Serialized 14 byte log entry
0709 12:22:04.410278 (+ 50034us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:04.410330 (+    52us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:04.410378 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.410441 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:04.410577 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":94906,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":112,"thread_start_us":111554,"threads_started":2}
I20260709 12:22:04.402616  2666 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:04.413009  2907 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 2.347s	user 0.015s	sys 0.034s
I20260709 12:22:04.413023 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:04.421489 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:04.386528  2906 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:04.403620  2927 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:04.418087  2912 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.413784  2500 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:04.439924 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:04.440842 31678 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:04.441821 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:04.417552  2929 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:04.438815 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:04.442858  2690 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:04.443580 31674 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:04.444217 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1991) took 2787 ms. Trace:
I20260709 12:22:04.444582 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:04.446593 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:04.447654 31678 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:04.444675 31826 rpcz_store.cc:276] 0709 12:22:01.656399 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.914990 (+258591us) service_pool.cc:224] Handling call
0709 12:22:01.915487 (+   497us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:01.924948 (+  9461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:01.924996 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:01.932924 (+  7928us) log.cc:844] Serialized 52 byte log entry
0709 12:22:02.053501 (+120577us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:02.763618 (+710117us) log.cc:844] Serialized 14 byte log entry
0709 12:22:02.765936 (+  2318us) log.cc:844] Serialized 14 byte log entry
0709 12:22:02.777056 (+ 11120us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:02.777168 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:04.412365 (+1635197us) spinlock_profiling.cc:243] Waited 26.2 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:04.443949 (+ 31584us) raft_consensus.cc:1685] finished
0709 12:22:04.443971 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:04.444121 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12289,"prepare.queue_time_us":493067,"prepare.run_cpu_time_us":116,"prepare.run_wall_time_us":15782,"raft.queue_time_us":151564,"raft.run_cpu_time_us":418,"raft.run_wall_time_us":20820,"spinlock_wait_cycles":60211968,"thread_start_us":739776,"threads_started":2,"wal-append.queue_time_us":113778}
I20260709 12:22:04.450671  2929 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:04.419682 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.421028 31745 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:04.426671 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:04.453127 31672 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:22:04.428051 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:04.454615 31823 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 416) took 3649 ms (client timeout 1914 ms). Trace:
W20260709 12:22:04.455156 31823 rpcz_store.cc:269] 0709 12:22:00.804904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:00.845904 (+ 41000us) service_pool.cc:224] Handling call
0709 12:22:04.427907 (+3582003us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:22:04.454539 (+ 26632us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2984704}
I20260709 12:22:04.456409 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:04.409173  2718 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:04.458709  2500 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:04.459194  2500 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:04.459648  2500 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:04.459681  2718 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:04.460446 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:04.460952  2718 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:04.461730 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:04.409811  2949 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:04.464133 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.464944 31823 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:04.465579  2857 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:04.465777 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.453156  2713 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:04.467485 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2121) took 1880 ms. Trace:
I20260709 12:22:04.467900 31818 rpcz_store.cc:276] 0709 12:22:02.586932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.189557 (+1602625us) service_pool.cc:224] Handling call
0709 12:22:04.467230 (+277673us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400379454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:04.467419 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":234274560}
I20260709 12:22:04.469148 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.469830 31818 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:04.472091  2681 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:04.466514 31823 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:22:04.473783 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:04.474565 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.475102 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1497) took 1846 ms. Trace:
I20260709 12:22:04.475525 31823 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:22:04.476647 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.479259  2874 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:04.479477 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.479775  2874 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:04.480201 31818 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:22:04.479787 31823 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:04.480728  2874 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.481222 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.474660 31826 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:04.481509 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.454253 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:04.477380 31820 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:04.482656 31674 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:04.475549 31676 rpcz_store.cc:276] 0709 12:22:02.628271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:02.785093 (+156822us) service_pool.cc:224] Handling call
0709 12:22:04.474897 (+1689804us) spinlock_profiling.cc:243] Waited 70.2 ms on lock 0x7b540017c5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:04.475041 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":161427840}
I20260709 12:22:04.495215 31673 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:22:04.430763 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:04.491523 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.462486 31672 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:04.500128 32343 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:04.506493 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 547) took 1450 ms. Trace:
I20260709 12:22:04.507001 32377 rpcz_store.cc:276] 0709 12:22:03.055740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.055946 (+   206us) service_pool.cc:224] Handling call
0709 12:22:03.057068 (+  1122us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:03.232154 (+175086us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.506187 (+1274033us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54000127d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:04.506418 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":395371392}
I20260709 12:22:04.507812 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:04.512271 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.512564 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:04.513792 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:04.514438 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.515872 31818 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:04.509651 31717 connection.cc:256] RPC call timeout handler was delayed by -2.3516922190001424s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1647s for a response
W20260709 12:22:04.516513 31717 connection.cc:256] RPC call timeout handler was delayed by -2.2103922190003686s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1804s for a response
W20260709 12:22:04.516916 31717 connection.cc:256] RPC call timeout handler was delayed by -2.1293922190002377s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1894s for a response
W20260709 12:22:04.517449 31717 connection.cc:256] RPC call timeout handler was delayed by -1.6248052319997441s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19890000000000002s for a response
I20260709 12:22:04.517740  2713 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:04.518158  2803 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:04.523108  2803 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:04.524228  2803 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.509938  2870 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:04.515147 31826 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:04.528342  2870 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:04.528841 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.529914 31826 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:22:04.520138 31717 connection.cc:256] RPC call timeout handler was delayed by -1.5633265910000773s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20510000000000003s for a response
I20260709 12:22:04.529484  2870 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.532348  2296 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:04.532958  2296 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:04.520625 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.539877 31823 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
W20260709 12:22:04.521848 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 100443us
I20260709 12:22:04.508800  2874 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:04.543150 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.541905  2500 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:04.571739 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.578786 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:04.580981 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:04.516814 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:04.583074 31753 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 6. Current term is 7. Ops: []
W20260709 12:22:04.583395 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66691.8us
I20260709 12:22:04.584321 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:04.586988  2713 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:04.587730  2713 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:04.595692 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:04.585187  2949 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:04.597322 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:04.597707 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:04.600672 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:04.599356 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:04.601889 31753 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:04.602757 31745 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:04.603549 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:04.605440  2713 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:04.612653  2864 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 2.971s	user 0.033s	sys 0.036s
I20260709 12:22:04.614817  2955 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:04.615335 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 553) took 3707 ms. Trace:
I20260709 12:22:04.615890  2500 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:04.615788 32374 rpcz_store.cc:276] 0709 12:22:00.908199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.013419 (+105220us) service_pool.cc:224] Handling call
0709 12:22:01.013770 (+   351us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:01.033813 (+ 20043us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:01.033848 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:01.046536 (+ 12688us) log.cc:844] Serialized 26 byte log entry
0709 12:22:01.386525 (+339989us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:01.699671 (+313146us) log.cc:844] Serialized 14 byte log entry
0709 12:22:01.700049 (+   378us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:01.700184 (+   135us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:04.611986 (+2911802us) spinlock_profiling.cc:243] Waited 3.29 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:04.615002 (+  3016us) raft_consensus.cc:1685] finished
0709 12:22:04.615105 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:04.615268 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38836,"prepare.queue_time_us":148766,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":144082,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":30396,"spinlock_wait_cycles":7562624,"thread_start_us":570324,"threads_started":2,"wal-append.queue_time_us":448154}
I20260709 12:22:04.618643 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:04.620043 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:04.622206 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:04.623682 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:04.624915 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:04.624917  2949 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:04.626842  2927 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:04.627257  2927 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:04.627594  2927 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:04.628058  2296 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:04.629527 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.630403  2296 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:04.631660 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:04.625645 31750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 446) took 3654 ms (client timeout 1638 ms). Trace:
I20260709 12:22:04.633088 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:04.633319 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:04.633397 31750 rpcz_store.cc:269] 0709 12:22:00.971109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.013678 (+ 42569us) service_pool.cc:224] Handling call
0709 12:22:04.625429 (+3611751us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b5400000d54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:04.625545 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":629821184}
I20260709 12:22:04.634040  2160 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:04.636284 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2005) took 1583 ms. Trace:
I20260709 12:22:04.636765 31826 rpcz_store.cc:276] 0709 12:22:03.052754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.635386 (+1582632us) service_pool.cc:224] Handling call
0709 12:22:04.635743 (+   357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:04.635944 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:04.635972 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:04.636017 (+    45us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:04.636059 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.636116 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:04.636224 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.638558 31826 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:04.639210 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2010) took 1579 ms. Trace:
W20260709 12:22:04.639969 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.639609 31826 rpcz_store.cc:276] 0709 12:22:03.059519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.637786 (+1578267us) service_pool.cc:224] Handling call
0709 12:22:04.638381 (+   595us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:04.638971 (+   590us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.639149 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.643992 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.645869 31826 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:04.648093 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:04.650063 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:04.665493 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:04.671487  2793 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:04.634478  2160 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.666821 31826 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:04.678660 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:04.692765  2955 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:04.694798 31753 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:04.646667  2500 raft_consensus.cc:2764] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260709 12:22:04.697309 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2003) took 1645 ms. Trace:
I20260709 12:22:04.702992 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 544) took 3076 ms. Trace:
I20260709 12:22:04.681181  2160 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:04.680027  2943 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:04.709118 32378 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 540) took 3672 ms (client timeout 1894 ms). Trace:
I20260709 12:22:04.703431 32369 rpcz_store.cc:276] 0709 12:22:01.626236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.705443 (+ 79207us) service_pool.cc:224] Handling call
0709 12:22:01.706158 (+   715us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:02.779183 (+1073025us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.702621 (+1923438us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400646c54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:04.702924 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2465667968}
I20260709 12:22:04.701237 31818 rpcz_store.cc:276] 0709 12:22:03.052089 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.581109 (+1529020us) service_pool.cc:224] Handling call
0709 12:22:04.589109 (+  8000us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:04.592840 (+  3731us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:04.592870 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:04.599032 (+  6162us) log.cc:844] Serialized 26 byte log entry
0709 12:22:04.696924 (+ 97892us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:04.696968 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.697046 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:04.697092 (+    46us) raft_consensus.cc:1685] finished
0709 12:22:04.697103 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:04.697235 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":146,"spinlock_wait_cycles":5420416,"thread_start_us":81896,"threads_started":1,"wal-append.queue_time_us":72583}
I20260709 12:22:04.701647  2651 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:04.711115 32375 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
W20260709 12:22:04.709813 32378 rpcz_store.cc:269] 0709 12:22:01.036294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.193713 (+157419us) service_pool.cc:224] Handling call
0709 12:22:04.708786 (+3515073us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b5400499154. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:04.709013 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3163258112}
I20260709 12:22:04.713609 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:04.715152 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:04.677455  2793 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.710806  2943 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.710553  2500 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260709 12:22:04.728024 31818 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:22:04.717319  2706 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:04.729956 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2012) took 1610 ms. Trace:
W20260709 12:22:04.735107 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 87103.6us
W20260709 12:22:04.759974 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:04.734807 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 64606.7us
I20260709 12:22:04.726616  2943 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:04.726223  2718 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:22:04.763077 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46156.7us
I20260709 12:22:04.760375 31818 rpcz_store.cc:276] 0709 12:22:03.119440 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.719426 (+1599986us) service_pool.cc:224] Handling call
0709 12:22:04.725135 (+  5709us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:04.729694 (+  4559us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.729907 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:04.759771  2706 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.763296 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:04.767540 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.780964  2160 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:04.794025 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.769366  2706 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:04.765472  2651 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:04.796661 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:04.808102  2793 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:04.809746 31752 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:22:04.798846  2903 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
W20260709 12:22:04.811112 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:04.817224 32374 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:04.814659 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:04.822355 32378 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:04.823534 31818 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:22:04.824380 32369 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260709 12:22:04.821067  2870 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.827699  2160 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:04.827943  2870 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.827775  2651 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:04.823159  2713 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:04.828505 31818 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:04.833627 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:04.834488 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 67159.7us
I20260709 12:22:04.830650  2870 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:04.849287 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2088) took 3558 ms. Trace:
I20260709 12:22:04.853266 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:04.851485 31671 rpcz_store.cc:276] 0709 12:22:01.285133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.285267 (+   134us) service_pool.cc:224] Handling call
0709 12:22:01.285594 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:01.285809 (+   215us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:01.649992 (+364183us) log.cc:844] Serialized 14 byte log entry
0709 12:22:04.842966 (+3192974us) spinlock_profiling.cc:243] Waited 2.95 ms on lock 0x7b18004c0000. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287
0709 12:22:04.843054 (+    88us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:04.843088 (+    34us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:04.843125 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.843271 (+   146us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:04.843386 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68163,"prepare.queue_time_us":127101,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":117541,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":11408,"spinlock_wait_cycles":6792576,"thread_start_us":542731,"threads_started":3,"wal-append.queue_time_us":261802,"wal-append.run_cpu_time_us":2602,"wal-append.run_wall_time_us":1243907}
I20260709 12:22:04.846886  2955 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:04.859588 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 96399.6us
I20260709 12:22:04.859383 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:04.863312 31818 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:04.863433 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:04.837863  2713 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.878121 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:04.837867  2965 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260709 12:22:04.858765  2970 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:04.886080  2970 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:04.885872  2975 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:04.886514  2970 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:04.886957  2970 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:04.886914 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 120597us
I20260709 12:22:04.895076  2976 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:04.898104  2976 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:04.898670  2976 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:04.898378 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:04.896250 31818 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:04.917701 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:04.917901  2977 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:04.918488  2861 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:04.918687  2793 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:04.918583  2977 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:04.919122  2861 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:04.919484  2793 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:04.919685  2861 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:04.919533  2977 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:04.920500  2793 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:04.920702  2861 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:04.924880 31822 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:04.925961  2713 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:04.926199 31822 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:04.927424  2949 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:04.928169  2949 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:04.916973 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:04.969897 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 106585us
W20260709 12:22:04.970808 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53970.1us
W20260709 12:22:04.971534 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 106506us
I20260709 12:22:04.982798  2984 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.005s
I20260709 12:22:04.982120  2943 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:04.992396  2943 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:04.978813  2985 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:04.985164  2950 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:04.982203  2955 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:04.970108  2713 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 12:22:04.996384  2927 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.041014  2927 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.014572  2985 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:05.019250  2979 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260709 12:22:05.042464  2985 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.017546  2950 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.044004  2950 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.043776  2977 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.044479 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2000) took 2163 ms. Trace:
W20260709 12:22:05.042445  2987 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.073s	user 0.001s	sys 0.000s
I20260709 12:22:05.044903 31819 rpcz_store.cc:276] 0709 12:22:02.880678 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.303675 (+1422997us) service_pool.cc:224] Handling call
0709 12:22:04.472131 (+168456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:04.472387 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:04.472719 (+   332us) log.cc:844] Serialized 14 byte log entry
0709 12:22:04.594125 (+121406us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:04.594165 (+    40us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:04.594235 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:04.594309 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:05.044277 (+449968us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b5400379450. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:05.044411 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1149,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":34,"raft.queue_time_us":1084,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":4629,"spinlock_wait_cycles":369251584,"thread_start_us":121024,"threads_started":1,"wal-append.queue_time_us":214038}
I20260709 12:22:05.046141  2977 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:05.043871  2985 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:05.036412  2474 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:05.017195  2794 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:05.065716  2970 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.069510  2970 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.075302 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:05.038013  2706 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.089504 31819 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:05.099561  2499 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.100312  2499 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.108551  2976 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:05.111400  2733 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:05.111853  2733 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:05.112552  2976 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:05.109674  2706 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.112643  2950 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.112700  2733 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.113893  2733 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:05.115120  2950 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.157085  2740 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:05.222831 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.247545  2912 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.252030 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:05.252806 31675 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
W20260709 12:22:05.253319 31675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1508) took 1706 ms (client timeout 1921 ms). Trace:
W20260709 12:22:05.253680 31675 rpcz_store.cc:269] 0709 12:22:03.546910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.985805 (+1438895us) service_pool.cc:224] Handling call
0709 12:22:05.253251 (+267446us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:05.271746 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.272876  2971 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:05.273365 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:05.274127  2971 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:22:05.275014 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.275473  2971 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:05.275964  2971 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:05.277451 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:05.278347  2985 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:05.278774  2985 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.278786 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:05.279255  2985 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.279781  2976 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:05.280232  2976 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:05.280685  2976 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:05.284623  2971 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.287389  2971 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.287755  2985 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.288636  2976 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.290199  2985 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.290812  2500 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:05.295873  2500 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:05.296991  2976 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.290460 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:05.296836  2500 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.298377  2499 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.298415  2500 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.298771  2976 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:05.298519 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:05.312023 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.312537 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 707) took 1095 ms. Trace:
I20260709 12:22:05.313308 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:05.314325  2971 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:05.314797  2971 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:05.315183  2971 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:05.316107 31821 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2123) took 2728 ms (client timeout 1993 ms). Trace:
W20260709 12:22:05.316565 31821 rpcz_store.cc:269] 0709 12:22:02.587553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:03.068771 (+481218us) service_pool.cc:224] Handling call
0709 12:22:05.315825 (+2247054us) spinlock_profiling.cc:243] Waited 32.7 ms on lock 0x7b54003ca5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:05.316010 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75273984}
I20260709 12:22:05.319087 31821 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 10. Ops: []
I20260709 12:22:05.319557 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.323627 31825 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
I20260709 12:22:05.325973 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.326980  2955 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:05.327617  2955 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:05.313024 32375 rpcz_store.cc:276] 0709 12:22:04.216737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.243956 (+ 27219us) service_pool.cc:224] Handling call
0709 12:22:04.710850 (+466894us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:04.711656 (+   806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.312283 (+600627us) spinlock_profiling.cc:243] Waited 451 ms on lock 0x7b5400499150. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:05.312455 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1036665088}
I20260709 12:22:05.327944  2955 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.328402  2955 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:05.327950 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:05.328114  2500 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:05.331477  2861 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:05.332191  2985 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:05.333076  2985 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:05.333498  2985 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:05.331614  2976 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:05.339660  2955 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.342048  2955 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.341802  2985 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.342365  2861 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:05.343096  2861 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.343577  2971 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.344405  2985 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:05.344708 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.345604  2971 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.345628  2861 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:05.336252  2976 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:05.356370 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 1994) took 3693 ms. Trace:
I20260709 12:22:05.357015 31824 rpcz_store.cc:276] 0709 12:22:01.662884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:01.981659 (+318775us) service_pool.cc:224] Handling call
0709 12:22:02.056256 (+ 74597us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:02.056716 (+   460us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:03.001552 (+944836us) log.cc:844] Serialized 14 byte log entry
0709 12:22:03.251872 (+250320us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:03.257306 (+  5434us) log.cc:844] Serialized 26 byte log entry
0709 12:22:03.257491 (+   185us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:03.257532 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:03.257598 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:05.240168 (+1982570us) spinlock_profiling.cc:243] Waited 2.87 ms on lock 0x7b18004caaa0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:22:05.355957 (+115789us) raft_consensus.cc:1685] finished
0709 12:22:05.356081 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:05.356273 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96750,"prepare.queue_time_us":526865,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":156602,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":34365,"spinlock_wait_cycles":6598528,"thread_start_us":1056204,"threads_started":3,"wal-append.queue_time_us":227709}
I20260709 12:22:05.354776  2976 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.363111 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:05.366488  2976 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.361907 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:05.368688 31825 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:22:05.371558 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:05.373179 31824 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:05.375110 31825 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:05.377869 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:05.381549 31824 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:22:05.382658  2499 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:05.384904 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.386480  2474 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260709 12:22:05.387408  2861 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:05.383427  2912 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.005s
I20260709 12:22:05.382966  2976 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.397925 31643 leader_election.cc:400] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10.
I20260709 12:22:05.400338 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2001) took 2519 ms. Trace:
W20260709 12:22:05.401510 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 723752us
I20260709 12:22:05.403034 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.550518  2976 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:05.551059 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 153308us
I20260709 12:22:05.551700 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 437) took 1069 ms. Trace:
I20260709 12:22:05.552088 31820 rpcz_store.cc:276] 0709 12:22:04.483042 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.402737 (+919695us) service_pool.cc:224] Handling call
0709 12:22:05.551633 (+148896us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:05.441277  2568 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:22:05.438241 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.556260 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:05.558007 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:05.559139 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:05.502017 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.559619 31670 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:05.559913 31678 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:05.499464 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 456) took 2547 ms. Trace:
I20260709 12:22:05.560989 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:05.561067 31677 rpcz_store.cc:276] 0709 12:22:02.952335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:02.952545 (+   210us) service_pool.cc:224] Handling call
0709 12:22:02.952995 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:02.998356 (+ 45361us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.499128 (+2500772us) spinlock_profiling.cc:243] Waited 43.3 ms on lock 0x7b54001d6354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:05.499381 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":99580672}
I20260709 12:22:05.561761 31675 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:22:05.562695 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:05.502218 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.563199 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:05.563364 31676 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:05.546213  2713 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.552175 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:05.564402 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:05.564327  2713 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.564870 31673 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:05.565088 31672 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:22:05.506331  2985 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:05.566088  2985 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.566968  2713 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:05.546046 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.568099 32342 connection.cc:256] RPC call timeout handler was delayed by -2.0349196150000353s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
I20260709 12:22:05.566216 31823 rpcz_store.cc:276] 0709 12:22:02.880897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.573572 (+1692675us) service_pool.cc:224] Handling call
0709 12:22:04.573897 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:04.574145 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:04.574843 (+   698us) log.cc:844] Serialized 14 byte log entry
0709 12:22:04.647574 (+ 72731us) log.cc:844] Serialized 14 byte log entry
0709 12:22:04.667656 (+ 20082us) log.cc:844] Serialized 14 byte log entry
0709 12:22:05.303992 (+636336us) spinlock_profiling.cc:243] Waited 20.9 ms on lock 0x7b18004ca860. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:22:05.399733 (+ 95741us) spinlock_profiling.cc:243] Waited 12.8 ms on lock 0x7b18004ca860. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:22:05.399921 (+   188us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:05.399987 (+    66us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:05.400050 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.400128 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:05.400277 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7934,"prepare.queue_time_us":33160,"prepare.run_cpu_time_us":228,"prepare.run_wall_time_us":5298,"raft.queue_time_us":19759,"raft.run_cpu_time_us":344,"raft.run_wall_time_us":22432,"spinlock_wait_cycles":77373056,"thread_start_us":71898,"threads_started":1,"wal-append.queue_time_us":122532}
W20260709 12:22:05.568459 32342 connection.cc:256] RPC call timeout handler was delayed by -1.8985428070000125s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19210000000000002s for a response
W20260709 12:22:05.569185 32342 connection.cc:256] RPC call timeout handler was delayed by -1.2645431949999875s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2425s for a response
I20260709 12:22:05.569223 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:05.569640 32342 connection.cc:256] RPC call timeout handler was delayed by -1.0661033979995409s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16920000000000002s for a response
I20260709 12:22:05.384539  2499 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.565359 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:05.559195 31674 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:22:05.571022  2551 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:05.570550 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.571734  2500 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:05.572139 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:05.572257  2955 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:05.572726 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.573277  2568 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.573472  2955 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:05.571046  2499 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:05.573891 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:05.574607  2955 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:05.574842  2985 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:05.575131 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:05.575259  2955 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:05.570310 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:05.575706  2955 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:05.575924 31823 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:05.560050 31670 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:05.576114  2870 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:05.576683 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.576735  2870 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.577255  2955 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.577888 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.578476  2955 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.578892 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.579324 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:05.571888  2551 consensus_queue.cc:1243] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 8.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:22:05.580170  2474 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:05.572206  2500 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.587500 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:05.587882  2500 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:05.588151 32375 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:05.583305  2568 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:05.583828 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 598630us
I20260709 12:22:05.584353  2912 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:05.592367 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:05.593931 31820 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:05.582569  2870 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:05.387969  2861 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:05.599604  2861 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:05.603452 31820 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:22:05.608395  2568 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:05.585913  2976 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.613685 31670 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.616274 31670 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:05.626156 31573 catalog_manager.cc:5697] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 5 to 6, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 6 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:05.628208 31573 catalog_manager.cc:5697] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 2 to 5, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 5 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:05.629920 31573 catalog_manager.cc:5697] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 3 to 6, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 6 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:05.631798 31573 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:05.638819 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2004) took 2586 ms. Trace:
I20260709 12:22:05.639256 31827 rpcz_store.cc:276] 0709 12:22:03.052431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.623514 (+1571083us) service_pool.cc:224] Handling call
0709 12:22:04.623934 (+   420us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:04.624397 (+   463us) raft_consensus.cc:1534] Early marking committed up to index 3
0709 12:22:04.624734 (+   337us) log.cc:844] Serialized 14 byte log entry
0709 12:22:05.633571 (+1008837us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:05.634195 (+   624us) log.cc:844] Serialized 52 byte log entry
0709 12:22:05.634395 (+   200us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:05.634442 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.634532 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:05.638589 (+  4057us) raft_consensus.cc:1685] finished
0709 12:22:05.638608 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:05.638759 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":928007,"prepare.queue_time_us":5198,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":4932,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"thread_start_us":80372,"threads_started":1,"wal-append.queue_time_us":136129}
I20260709 12:22:05.639910 31573 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 3 to 7, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 7 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:05.641196 31823 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
W20260709 12:22:05.642946 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.643303 31675 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:05.645114 31752 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:05.646471 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.647136 31752 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:05.647442 31751 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:05.646386 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.648304 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.648389 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.648867 31753 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 7: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:05.649114 31747 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:05.649742 31745 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:22:05.649899 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.650759 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.651458 31747 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:22:05.652944 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.654098 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.655793 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.656827 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.657562 31752 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:05.658097 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.660812  2979 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:05.661219  2979 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:05.661085  2500 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.649130 31751 raft_consensus.cc:2368] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
I20260709 12:22:05.637144  2499 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:05.665076 31717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.782s	user 0.015s	sys 0.030s
W20260709 12:22:05.724043 31717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.841s	user 0.015s	sys 0.030s
W20260709 12:22:05.724370 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 841191us
I20260709 12:22:05.725569 32375 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:05.662122  2979 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.727419  2979 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.731647  2967 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:05.649745 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.668510  2910 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 12:22:05.671243 31571 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 235) took 38335 ms (client timeout 15000 ms). Trace:
W20260709 12:22:05.679800 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 107668us
W20260709 12:22:05.680927 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 129781us
I20260709 12:22:05.740029  2910 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:05.740859  2910 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:05.741261 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:05.683102 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 119729us
I20260709 12:22:05.742256  2296 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:05.742508  2681 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:05.743184  2681 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:05.743593  2681 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:05.741771 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.739768 31571 rpcz_store.cc:269] 0709 12:21:27.335751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:27.336391 (+   640us) service_pool.cc:224] Handling call
0709 12:21:27.650194 (+313803us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b5400002210. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb10b2 00007fe902c2a759 00007fe902bc2a76 00007fe902b93d58 00007fe902c9204e 00007fe9011532a4 00007fe901153232 00007fe901153161 00007fe9011530dc
0709 12:21:27.740793 (+ 90599us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b5400002210. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb1640 00007fe902b88846 00007fe902c9290e 00007fe9011532a4 00007fe901153232 00007fe901153161 00007fe9011530dc 00007fe901152392 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:05.670937 (+37930144us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:21:51.494590 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:21:51.632896 (+138306us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:21:51.632951 (+    55us) write_op.cc:435] Acquired schema lock
0709 12:21:51.632972 (+    21us) tablet.cc:662] Decoding operations
0709 12:21:51.658861 (+ 25889us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:21:51.658950 (+    89us) write_op.cc:641] Partition lock acquired for write op
0709 12:21:51.658983 (+    33us) tablet.cc:685] Acquiring locks for 7 operations
0709 12:21:51.659849 (+   866us) tablet.cc:701] Row locks acquired
0709 12:21:51.659894 (+    45us) write_op.cc:260] PREPARE: finished
0709 12:21:51.660084 (+   190us) write_op.cc:270] Start()
0709 12:21:51.660244 (+   160us) write_op.cc:276] Timestamp: P: 1783599711660015 usec, L: 0
0709 12:21:51.660266 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:21:51.735228 (+ 74962us) log.cc:844] Serialized 2857 byte log entry
0709 12:21:52.213410 (+478182us) op_driver.cc:464] REPLICATION: finished
0709 12:21:52.303045 (+ 89635us) write_op.cc:301] APPLY: starting
0709 12:21:52.303151 (+   106us) tablet.cc:1367] starting BulkCheckPresence
0709 12:21:52.303364 (+   213us) tablet.cc:1370] finished BulkCheckPresence
0709 12:21:52.307344 (+  3980us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:21:52.335478 (+ 28134us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:21:52.335778 (+   300us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0709 12:21:52.335832 (+    54us) write_op.cc:312] APPLY: finished
0709 12:21:52.336157 (+   325us) log.cc:844] Serialized 59 byte log entry
0709 12:21:52.336347 (+   190us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:52.336454 (+   107us) write_op.cc:454] Released schema lock
0709 12:21:52.336513 (+    59us) write_op.cc:341] FINISH: Updating metrics
0709 12:21:52.336937 (+   424us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:21:52.336963 (+    26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":375591,"reported_tablets":33,"spinlock_wait_cycles":58193024,"child_traces":[["op",{"apply.queue_time_us":89306,"apply.run_cpu_time_us":4136,"apply.run_wall_time_us":33884,"num_ops":7,"prepare.queue_time_us":443642,"prepare.run_cpu_time_us":18681,"prepare.run_wall_time_us":449146,"raft.queue_time_us":257067,"raft.run_cpu_time_us":26883,"raft.run_wall_time_us":69662,"replication_time_us":552985,"thread_start_us":707047,"threads_started":4,"wal-append.queue_time_us":114693,"wal-append.run_cpu_time_us":53790,"wal-append.run_wall_time_us":1395524}]]}
I20260709 12:22:05.740285 31643 leader_election.cc:400] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
I20260709 12:22:05.746085 31643 leader_election.cc:403] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:05.726552 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:05.748775 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.749733  2794 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:05.750423  2794 raft_consensus.cc:2764] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
W20260709 12:22:05.686373 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 95020.5us
I20260709 12:22:05.706497  2861 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.707403  2964 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.708799 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:05.710475 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:05.710675 32376 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:05.742388 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:05.711637 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:05.712845 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:05.750563  2949 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
I20260709 12:22:05.759851 32377 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:05.713964 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:05.750965 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:05.760568  2861 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:05.715114 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:05.750555 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.717566 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:05.762493  2682 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:05.762960  2682 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.763345  2682 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:05.763598 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:05.751399 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:05.719698 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:05.767127 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.763055  2949 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:05.768523 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:05.768683  2949 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:05.769222 31821 raft_consensus.cc:2368] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10.
W20260709 12:22:05.769955 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.770392 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:05.771020 31821 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:05.771714  2996 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.772053  2949 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.772341 31821 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:05.772297  2996 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.773010  2949 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.771533  2984 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:05.752480  2967 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.023s
I20260709 12:22:05.775120  2949 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:05.775678  2971 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:05.775955  2967 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:05.776085  2971 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:05.770912 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.753249  2968 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 12:22:05.777305  2949 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:05.777217  2971 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.778373  2971 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.753674  2794 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20260709 12:22:05.779855  2794 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:05.780187  2794 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:05.755388 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.781428 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.782068 31751 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:05.782003  2968 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:05.783085 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.781004  2794 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.755584  2976 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:22:05.776605  2967 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:05.756691 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.784873  2955 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:05.757412 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:05.785280  2955 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:05.742100  2910 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.756378 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2015) took 1115 ms. Trace:
I20260709 12:22:05.784233  2794 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:05.786464  2955 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.786990  2551 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:05.787606  2955 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.786741 31822 rpcz_store.cc:276] 0709 12:22:04.641181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.949267 (+308086us) service_pool.cc:224] Handling call
0709 12:22:04.949817 (+   550us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:05.630208 (+680391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:05.630239 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:05.630905 (+   666us) log.cc:844] Serialized 52 byte log entry
0709 12:22:05.756056 (+125151us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:05.756095 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.756160 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:05.756202 (+    42us) raft_consensus.cc:1685] finished
0709 12:22:05.756211 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:05.756309 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":679835,"raft.queue_time_us":720546,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":74,"thread_start_us":124800,"threads_started":1,"wal-append.queue_time_us":116848}
I20260709 12:22:05.758896 32374 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:05.789112 31753 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:05.783138  2968 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.759290 31671 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:05.790992  2910 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:05.753046 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:05.794286 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:05.759438 32369 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:05.794878 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:05.795258  2296 consensus_queue.cc:1175] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5206fd0087404d2da964a5d985a04ff4" responder_term: 9 status { last_received { term: 9 index: 8 } last_committed_idx: 8 last_received_current_leader { term: 9 index: 8 } }
I20260709 12:22:05.795595 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.759559 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:05.796855 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:05.797529 32369 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:05.797708 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:05.798233  2943 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:05.798561 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:05.798940 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.799196  2943 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.799651  2943 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:05.799984 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:05.801076 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:05.801178  2942 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:05.802075  2975 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:05.785089  2967 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.774730  2996 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:05.802891 31674 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:05.758024  2964 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.807519  2996 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:05.808127  2968 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:05.808760  2964 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:05.810262  2809 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:05.811031  2296 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:05.808959 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.812359  2996 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:05.808821  2857 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:05.813258  2942 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:05.813846  2857 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.814172 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:05.823086 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.823653 31823 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.824548 32374 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:22:05.825309 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.826589 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.814235  2857 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.827291 31748 raft_consensus.cc:2376] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7. Re-sending same reply.
I20260709 12:22:05.828716 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.829345 31748 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:05.829464  2964 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.830039  2964 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:05.755877 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2011) took 2645 ms. Trace:
I20260709 12:22:05.814656  2942 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.813189 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:05.815816  2967 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:05.816056  2949 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:05.816202 31674 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:05.816784 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.848524 31751 raft_consensus.cc:2376] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11. Re-sending same reply.
I20260709 12:22:05.817231 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:05.849818 31672 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:05.850210 31670 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260709 12:22:05.757439 32342 leader_election.cc:336] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.638s (SENT)
I20260709 12:22:05.850826 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.819051 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.820379 31819 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:05.820900 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.751708 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:05.821740 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.822315 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.824600 31643 leader_election.cc:400] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10.
I20260709 12:22:05.862051 31643 leader_election.cc:403] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:22:05.863762 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 106348us
W20260709 12:22:05.865410 32342 leader_election.cc:336] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.694s (SENT)
W20260709 12:22:05.827016 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:05.811678  2809 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:05.811549  2296 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:05.832569 32375 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:22:05.811528  2794 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.834612 31826 rpcz_store.cc:276] 0709 12:22:03.110693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.678710 (+1568017us) service_pool.cc:224] Handling call
0709 12:22:04.679184 (+   474us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:22:05.632193 (+953009us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:05.632223 (+    30us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:22:05.632982 (+   759us) log.cc:844] Serialized 78 byte log entry
0709 12:22:05.754709 (+121727us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:05.755029 (+   320us) log.cc:844] Serialized 14 byte log entry
0709 12:22:05.755382 (+   353us) log.cc:844] Serialized 14 byte log entry
0709 12:22:05.755569 (+   187us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.755643 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:05.755680 (+    37us) raft_consensus.cc:1685] finished
0709 12:22:05.755692 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:05.755820 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":952418,"prepare.queue_time_us":53308,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":61,"raft.queue_time_us":991480,"raft.run_cpu_time_us":214,"raft.run_wall_time_us":214,"thread_start_us":121297,"threads_started":1,"wal-append.queue_time_us":118867}
I20260709 12:22:05.836486 31822 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:05.837500 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.641655 31573 catalog_manager.cc:5697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 3 to 5, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 5 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:05.878177 31748 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:05.840242 31671 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:05.785933 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.846832 31823 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:05.878665 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:05.817656 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.860260 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 328298us
I20260709 12:22:05.879880 31747 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:22:05.880316 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 94480.3us
I20260709 12:22:05.881116 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.869217 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.882254 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.869341 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.833163  2955 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:22:05.874364 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49930.8us
I20260709 12:22:05.885743 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:05.832532  2964 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:05.874815 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 123177us
I20260709 12:22:05.875880 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.888051 31752 raft_consensus.cc:2376] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7. Re-sending same reply.
I20260709 12:22:05.889578 31717 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:05.890877 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:05.885877 31573 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:05.894107 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:05.894937 31823 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:05.896059 31823 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 12:22:05.897017 32342 leader_election.cc:336] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.794s (SENT)
I20260709 12:22:05.883687 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.884253 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:05.873126  2975 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:05.898324 31675 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:05.899485 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.906383 31822 raft_consensus.cc:2368] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
I20260709 12:22:05.907707 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:05.908643 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:05.909372 31822 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:05.909579 32375 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:05.910197 31826 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:22:05.901906 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:05.891623 31671 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:05.872761  2809 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:05.898788 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:05.873634  2861 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10.
I20260709 12:22:05.871609  2713 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:05.904425 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:05.922055 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:05.913120 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.925041 31751 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:22:05.913749 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:05.913306 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:05.915563 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.927865 31674 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:22:05.922083 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:05.916383 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.917086 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.930877 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.912886 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.926409 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.914722 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.933228 31751 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:05.923532 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:05.935518 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.936348 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.936407 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:05.937188 31749 raft_consensus.cc:2368] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:22:05.928906 32373 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:05.934263 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.938088 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:05.885161  2568 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:05.891297  2971 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:05.942823 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:05.888540  2949 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:05.902878  2651 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:05.947263 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:05.903638  2955 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:05.959256 31752 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:22:05.956396 31671 raft_consensus.cc:2368] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10.
I20260709 12:22:05.961385 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:05.912721  2551 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:05.963114 31672 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:05.963267 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:05.914433  2996 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:05.885977  2794 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:05.921140  2861 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:05.967473 31670 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:05.967662 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:05.998651 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.914786  2984 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:22:05.934950  2399 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:05.993539  2551 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.011281  2551 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.923154  2966 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260709 12:22:05.969955 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.943058  2976 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:05.970710 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:05.971513 31792 leader_election.cc:336] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.819s (SENT)
I20260709 12:22:05.943238  2985 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:05.994673 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:05.996933 31752 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:22:05.999434 31674 raft_consensus.cc:2368] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:22:05.991430  2971 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:06.039314  2971 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.039840  2971 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:05.958165  2568 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:06.046476  2568 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:05.995213  2996 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:05.950417  2884 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:22:06.006038 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2014) took 1365 ms. Trace:
W20260709 12:22:06.007301 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44169.2us
I20260709 12:22:06.015169 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:06.069495 31818 rpcz_store.cc:276] 0709 12:22:04.640803 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:04.932013 (+291210us) service_pool.cc:224] Handling call
0709 12:22:04.970234 (+ 38221us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:05.629075 (+658841us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:05.754127 (+125052us) log.cc:844] Serialized 14 byte log entry
0709 12:22:05.873487 (+119360us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:05.884109 (+ 10622us) log.cc:844] Serialized 26 byte log entry
0709 12:22:05.884308 (+   199us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:05.884373 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:05.884470 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:06.005782 (+121312us) raft_consensus.cc:1685] finished
0709 12:22:06.005802 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:06.005975 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":658343,"prepare.queue_time_us":117833,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":700501,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":8415,"thread_start_us":243604,"threads_started":2,"wal-append.queue_time_us":204317}
W20260709 12:22:06.069541 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 162380us
I20260709 12:22:06.017014 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:06.072942 31749 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:22:05.917887  2809 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:06.019665 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 85497us
I20260709 12:22:06.078886 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:06.024480 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:05.994050  2965 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:06.029744  2681 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.030755  2718 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:05.913213  2296 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.034188  2794 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:06.035060  2955 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.004108  2713 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:06.055583 31751 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:06.056910 31670 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:06.033947  2682 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.048866  2943 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.068153 31753 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:22:06.123998 31821 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
W20260709 12:22:06.071877 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 100402us
W20260709 12:22:06.125170 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:05.998016 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 79933.6us
I20260709 12:22:06.114845  2863 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260709 12:22:06.084846 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 148505us
I20260709 12:22:06.104141 31642 leader_election.cc:400] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
I20260709 12:22:06.144222 31642 leader_election.cc:403] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:22:06.013062  2500 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:06.145685 31643 leader_election.cc:400] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
I20260709 12:22:06.145777 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:06.146108 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:06.147449  2499 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:06.119714  2979 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:06.152240  2979 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:06.153192  2979 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.153879  2718 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.119537  2880 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
W20260709 12:22:06.113739 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 89373.8us
I20260709 12:22:06.156090 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:06.158041  2706 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:05.968091  2651 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:06.158584  2706 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:06.159583  2706 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:06.161808  2880 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:06.162740  2968 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:06.072273  2996 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:06.115232  2861 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:22:06.084105 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 160621us
I20260709 12:22:06.130869 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:06.146379 31643 leader_election.cc:403] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:22:06.179250 31818 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:06.154497  2718 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:06.181051 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:06.181897  2682 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:06.182919  2500 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:06.183521  2943 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:06.183772  2949 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
I20260709 12:22:06.122059  2713 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:06.184638  2713 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:06.184580  2949 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:06.185761  2949 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:06.186249  2949 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.186802  2984 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:06.187232  2984 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:06.188079  2984 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.118157  2965 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:06.174335 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 93889.9us
I20260709 12:22:06.177250 31820 raft_consensus.cc:2368] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
I20260709 12:22:06.206836 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:06.209090 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:06.210068 31821 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:06.192507  2949 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:06.227388  2965 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:06.242544 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:06.243769  3019 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:06.244283  3019 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:06.244731  3019 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:06.246587  2844 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:06.247144  2844 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:06.248234  2844 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.267570  3015 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:06.278573  3015 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.287676 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:06.288925 32369 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
W20260709 12:22:06.305160 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.166994 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:06.311118 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 130126us
I20260709 12:22:06.312904 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:06.312594 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 105912us
I20260709 12:22:06.313417 31643 leader_election.cc:400] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
W20260709 12:22:06.313987 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 135744us
I20260709 12:22:06.315949 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:06.317873 31672 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:06.318794 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:06.346750 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:06.347419 31677 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:06.349440 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:06.311784 31824 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:06.345300 31676 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:06.325987 31671 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:22:06.327795 31672 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260709 12:22:06.343245 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:06.320297 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:06.376085 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:06.379194 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54762.2us
I20260709 12:22:06.404371 31669 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:06.405665 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:06.407232 32373 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
W20260709 12:22:06.403417 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 60336us
W20260709 12:22:06.410877 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 90704.9us
I20260709 12:22:06.360342 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:06.379246 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.426376 31672 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:22:06.415586 31753 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:22:06.416224 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:06.417567 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:06.430941 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:06.321486  2984 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:06.432004 31669 raft_consensus.cc:2410] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 9 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:22:06.446663 31749 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:06.450922 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:06.390031 32399 debug-util.cc:398] Leaking SignalData structure 0x7b0800284920 after lost signal to thread 30503
W20260709 12:22:06.453503 32399 debug-util.cc:398] Leaking SignalData structure 0x7b0800270660 after lost signal to thread 30506
I20260709 12:22:06.345378  2999 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.454142 31750 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 12:22:06.454689 32399 debug-util.cc:398] Leaking SignalData structure 0x7b0800284600 after lost signal to thread 31610
I20260709 12:22:06.465062 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:06.465687 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:06.317970  2943 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:06.456303 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:06.457351 31747 raft_consensus.cc:2410] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 7.
I20260709 12:22:06.360970  2985 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:06.496112 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:06.496995 31748 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:06.318609  2844 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:06.463557 31751 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 12:22:06.510253 31751 pending_rounds.cc:85] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 9
I20260709 12:22:06.510547 31751 pending_rounds.cc:107] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:22:06.509125 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:06.510844 31751 raft_consensus.cc:2889] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:06.464063 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:06.466159 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:06.466401 31750 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 7.
I20260709 12:22:06.403090  2884 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.469282 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:06.459482 31672 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:06.412508  2681 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:06.466180 32399 debug-util.cc:398] Leaking SignalData structure 0x7b08001d41e0 after lost signal to thread 31702
I20260709 12:22:06.360288  2499 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.136s
I20260709 12:22:06.416651  2968 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:06.423527  2474 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:06.509415 31676 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:22:06.359372  2861 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:22:06.508559 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46424.6us
I20260709 12:22:06.517406 31669 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:06.524339 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:06.521394 31672 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
W20260709 12:22:06.521690 32399 debug-util.cc:398] Leaking SignalData structure 0x7b08003cb340 after lost signal to thread 31777
I20260709 12:22:06.522223 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:06.530155 32399 debug-util.cc:398] Leaking SignalData structure 0x7b08001a3e40 after lost signal to thread 31778
I20260709 12:22:06.462875  2966 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:06.531385 32399 debug-util.cc:398] Leaking SignalData structure 0x7b08003f6560 after lost signal to thread 31851
I20260709 12:22:06.531647 31678 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: [7.5-7.5]
W20260709 12:22:06.531737 32399 debug-util.cc:398] Leaking SignalData structure 0x7b0800352660 after lost signal to thread 31852
I20260709 12:22:06.533004 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:06.533120 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:06.533382 32399 debug-util.cc:398] Leaking SignalData structure 0x7b0800085920 after lost signal to thread 32402
I20260709 12:22:06.533836 31676 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:06.534415 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:06.535087 32399 debug-util.cc:398] Leaking SignalData structure 0x7b0800104a00 after lost signal to thread 2893
W20260709 12:22:06.507682  3001 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:22:06.506974  2434 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:06.506716  2870 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:06.345472  3015 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:06.468820  2975 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20260709 12:22:06.555888 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:06.571736 31750 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:06.436659  2794 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.096s
I20260709 12:22:06.504686  2793 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.041s
I20260709 12:22:06.561407 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:06.587978 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:06.556900 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:06.531033  2474 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.502710  2943 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.554409  2434 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.602897  2434 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:06.571943  2912 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:06.542718  2938 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:06.585311  2857 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.758s	user 0.007s	sys 0.013s
I20260709 12:22:06.462865 31745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:22:06.606163 31745 raft_consensus.cc:606] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:06.606523 31745 raft_consensus.cc:614] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:06.497445  2999 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.590299 31672 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:06.566144  2681 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:06.596514 31677 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:06.546716  2861 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:06.525758  2884 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.601703  2474 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:06.580977  2985 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.544718  2966 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.601197  2682 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:06.598664  3016 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.656426 31821 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:22:06.605821  2938 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:06.616362  2976 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:06.658514  2971 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.618s	user 0.009s	sys 0.008s
I20260709 12:22:06.643023  2912 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:06.728718 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 259473us
I20260709 12:22:06.732800  2399 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:06.734043  2880 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:06.734645  2880 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:06.735833  2880 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.736606 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:06.737547  2880 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:06.738246  2399 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:06.673969  2938 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:06.611439  2857 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.768474 31821 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:06.636126  2942 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.101s
I20260709 12:22:06.675035  2955 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:06.794166 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:06.659657  2985 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:06.795037  3013 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.000s	sys 0.013s
W20260709 12:22:06.797144 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57159us
W20260709 12:22:06.789963  2979 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.003s	sys 0.018s
I20260709 12:22:06.675233  3016 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:06.825089  2962 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
W20260709 12:22:06.795243  2296 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.004s	sys 0.015s
I20260709 12:22:06.743799  2971 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:06.884039  3013 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.000s	sys 0.013s
W20260709 12:22:06.890002  2979 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.006s	sys 0.021s
W20260709 12:22:06.913062  2296 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.721s	user 0.004s	sys 0.022s
I20260709 12:22:06.910787  2911 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:06.821244  2955 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:06.838164  2968 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.089s
W20260709 12:22:06.925101 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 507576us
I20260709 12:22:06.983342 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:06.949779 31673 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:22:06.923128  2682 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:06.977433  2996 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.786s	user 0.005s	sys 0.011s
W20260709 12:22:06.956189  3026 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:22:06.957883  3031 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.293s	user 0.000s	sys 0.001s
W20260709 12:22:06.955145  3024 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
W20260709 12:22:06.976496  3017 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.410s	user 0.001s	sys 0.001s
W20260709 12:22:06.953754  3022 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.311s	user 0.000s	sys 0.001s
I20260709 12:22:06.883805  2912 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:06.952389  3049 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:06.956108  2857 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:06.953016  2999 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:06.986472  3015 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:06.987800  2965 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.074656  2499 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:22:07.077086 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 80646.5us
I20260709 12:22:07.071120 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:07.084883 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 272570us
W20260709 12:22:07.085966 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 261779us
W20260709 12:22:07.075870  3025 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:22:07.122517 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:07.085559 31822 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.751s	user 0.013s	sys 0.010s
I20260709 12:22:07.021222  2996 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:07.181205 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:07.075601  2911 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:07.192101 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:07.194123 31822 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.860s	user 0.019s	sys 0.013s
I20260709 12:22:07.110971  2943 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:06.999046  3039 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.195s
W20260709 12:22:07.167536  2713 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.982s	user 0.018s	sys 0.001s
W20260709 12:22:07.301031 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.712s	user 0.003s	sys 0.035s
I20260709 12:22:07.299835 32374 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 8: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:07.249943 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:07.316147 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.004s	sys 0.035s
I20260709 12:22:07.146072  2884 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:07.294806  2793 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:07.314081 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 246617us
W20260709 12:22:07.296582  2949 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.052s	user 0.016s	sys 0.000s
I20260709 12:22:07.320187 31678 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:22:07.323457 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:07.327931 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2106) took 1015 ms. Trace:
W20260709 12:22:07.198851  2809 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.123s	user 0.015s	sys 0.010s
I20260709 12:22:07.225611  2975 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.102913  2861 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:07.130336  2998 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.949s	user 0.013s	sys 0.015s
I20260709 12:22:07.279325  2857 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:07.312024  3031 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
W20260709 12:22:07.376091 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 788111us
W20260709 12:22:07.384769 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 150573us
I20260709 12:22:07.199020  3049 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:07.150319  2500 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.967s	user 0.021s	sys 0.000s
I20260709 12:22:07.295389  3037 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:07.316965 31677 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:07.199399 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:07.202777  2965 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:07.309937  2863 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:07.265651  2980 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.115s
I20260709 12:22:07.421051 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 454) took 1537 ms. Trace:
I20260709 12:22:07.404284 31670 rpcz_store.cc:276] 0709 12:22:06.312963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.313123 (+   160us) service_pool.cc:224] Handling call
0709 12:22:06.406822 (+ 93699us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.407028 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:06.578385 (+171357us) log.cc:844] Serialized 14 byte log entry
0709 12:22:07.327363 (+748978us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:07.327454 (+    91us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:07.327534 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.327651 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.327840 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37811,"prepare.queue_time_us":405808,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":5460,"raft.queue_time_us":398222,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":62183,"thread_start_us":881441,"threads_started":3,"wal-append.queue_time_us":562793}
I20260709 12:22:07.407327 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:07.214123  2979 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 3, Time since last communication: 0.173s
I20260709 12:22:07.213557  2968 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.173s
I20260709 12:22:07.449836 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:07.449570 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:07.475776 31745 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:07.389484  2966 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:07.406545  2998 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.225s	user 0.018s	sys 0.020s
I20260709 12:22:07.239778  2296 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.147s
I20260709 12:22:07.213219  2794 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.176s
W20260709 12:22:07.463783  2718 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.283s	user 0.012s	sys 0.012s
I20260709 12:22:07.170835  2682 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:07.448208 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:07.496706 31827 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:07.439815  3047 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:22:07.407757 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:07.412576 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1523) took 1411 ms. Trace:
I20260709 12:22:07.406188  2975 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:07.417325 31822 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:07.495910  2943 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:07.313056  2713 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:07.400740  3046 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:07.469149 31823 rpcz_store.cc:276] 0709 12:22:05.883894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.896694 (+ 12800us) service_pool.cc:224] Handling call
0709 12:22:05.896998 (+   304us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:05.911998 (+ 15000us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:05.912032 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:05.932559 (+ 20527us) log.cc:844] Serialized 26 byte log entry
0709 12:22:06.101376 (+168817us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:06.101427 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.101504 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.420828 (+1319324us) raft_consensus.cc:1685] finished
0709 12:22:07.420849 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.420988 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":5918,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":6012,"thread_start_us":129790,"threads_started":1,"wal-append.queue_time_us":554869}
I20260709 12:22:07.302019  2911 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:07.405920 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2037) took 1073 ms. Trace:
I20260709 12:22:07.560745 31820 rpcz_store.cc:276] 0709 12:22:06.332121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.332442 (+   321us) service_pool.cc:224] Handling call
0709 12:22:06.332858 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.640991 (+308133us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:06.641278 (+   287us) log.cc:844] Serialized 14 byte log entry
0709 12:22:06.655979 (+ 14701us) log.cc:844] Serialized 14 byte log entry
0709 12:22:06.844598 (+188619us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:06.844676 (+    78us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:06.844755 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.405629 (+560874us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b54000fc654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:07.405678 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.405825 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":75832,"prepare.run_cpu_time_us":167,"prepare.run_wall_time_us":15098,"raft.queue_time_us":615834,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":139,"spinlock_wait_cycles":653326720,"thread_start_us":188240,"threads_started":1,"wal-append.queue_time_us":32307}
I20260709 12:22:07.411517  2500 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
I20260709 12:22:07.406814  2857 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:07.411875  3037 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:07.474459  2965 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:07.475508 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:07.443586 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1521) took 1482 ms. Trace:
W20260709 12:22:07.482540 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 232035us
W20260709 12:22:07.484803 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 77550.7us
W20260709 12:22:07.579866 32342 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 2.020s (SENT) [suppressed 44 similar messages]
I20260709 12:22:07.580422 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:07.373245  3038 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.171s
I20260709 12:22:07.421519  2874 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 12:22:07.400877  2793 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:07.422940  2949 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:07.424746  2861 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.051s
I20260709 12:22:07.407370  2809 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:07.508399 31673 rpcz_store.cc:276] 0709 12:22:06.000984 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.001150 (+   166us) service_pool.cc:224] Handling call
0709 12:22:06.005676 (+  4526us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.990122 (+984446us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.412291 (+422169us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b54000465d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:07.412509 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2170514176}
W20260709 12:22:07.594040  3048 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.480s	user 0.000s	sys 0.002s
I20260709 12:22:07.416709  3015 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.413285  2870 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:07.420630  2884 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.439990  3065 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.144s
I20260709 12:22:07.506338  3049 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:07.543296 31745 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
W20260709 12:22:07.541529  3042 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.318s	user 0.001s	sys 0.000s
I20260709 12:22:07.579500 31753 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:07.579537 31675 rpcz_store.cc:276] 0709 12:22:05.961151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.961524 (+   373us) service_pool.cc:224] Handling call
0709 12:22:05.994489 (+ 32965us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:05.994880 (+   391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:05.994909 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:06.008670 (+ 13761us) log.cc:844] Serialized 26 byte log entry
0709 12:22:06.517290 (+508620us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:06.517393 (+   103us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.517521 (+   128us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.443245 (+925724us) raft_consensus.cc:1685] finished
0709 12:22:07.443271 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:07.443494 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12914,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":6952,"thread_start_us":457614,"threads_started":1,"wal-append.queue_time_us":645052}
I20260709 12:22:07.584704 32370 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:07.586685 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:07.509032  2718 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:07.581729 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:07.606508  2964 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:22:07.632521 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50879us
I20260709 12:22:07.634137 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:07.556007  2955 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:07.550037  3077 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:07.566715  2943 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:07.603755  2500 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:07.599340  2793 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:07.626011  2966 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:07.591334  3038 consensus_queue.cc:1243] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 9.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:22:07.590502  2965 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.600526  2942 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:07.620601 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:07.625945 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:07.626076 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 218395us
W20260709 12:22:07.679237 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53432.9us
I20260709 12:22:07.679466 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:07.627761 31818 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:07.680004 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:07.604363  2794 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.547916  3070 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.684525  2794 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:07.684665  3070 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:07.686864  3066 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:22:07.687520  3068 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260709 12:22:07.687637  3066 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:07.688706  3066 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:07.690397 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:07.690768  2942 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:07.691061 31673 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:07.691707  2942 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:07.692642 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:07.591272  2681 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:22:07.696473  3049 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:07.696873  3049 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:07.697208  3049 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:07.609195  2996 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:07.624305  3043 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.314s	user 0.001s	sys 0.001s
I20260709 12:22:07.569059  3046 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:07.622051  3039 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:07.621240  2870 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:07.624264  2968 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.413036 31824 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 8: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:07.628364  3065 consensus_queue.cc:1243] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 3
I20260709 12:22:07.628175  2861 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.634756  2964 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:07.633862  2874 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:07.641920  2296 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:07.645290  2955 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.650794  2910 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:07.664121  3064 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.214s	user 0.000s	sys 0.001s
I20260709 12:22:07.683943  3075 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:07.686961  3070 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:07.641103 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:07.629910  2884 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:07.661371  3040 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.625005  3015 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:07.661553  2966 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:07.671178  2500 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:07.760782 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 123083us
I20260709 12:22:07.711400  3039 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:07.673983  2965 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:07.814790  2965 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:07.651481  3077 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:07.782714 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:07.817058 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:07.783516 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 163211us
I20260709 12:22:07.817746  3077 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:07.785739 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 144750us
I20260709 12:22:07.819321 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:07.790760  3040 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:07.792448  3087 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:22:07.795181  2964 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:07.796175  2971 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:07.801723 31671 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:07.806392  2975 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:07.760419  2296 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:07.736066  2794 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:07.761270  2910 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:07.736400  2979 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 12:22:07.818677  3039 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:07.737875  2874 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:07.848825 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:07.739198  2861 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:07.873545 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:07.856913 31750 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 10 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:22:07.739297  2968 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:07.862914  2984 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.431s	user 0.015s	sys 0.000s
W20260709 12:22:07.854923  2967 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.994s	user 0.013s	sys 0.011s
I20260709 12:22:07.812232  3046 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:07.766306  2955 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:07.874891 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:07.854754  2681 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:07.862136  2296 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:07.894091 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:07.857987  2911 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:07.865509  2910 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:07.896476 31744 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:22:07.856943  3057 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260709 12:22:07.886924  3036 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.092s	user 0.006s	sys 0.000s
I20260709 12:22:07.924173 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:07.889556  3074 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
I20260709 12:22:07.911921 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:07.913787 31745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:22:07.892678  3037 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:07.929951 31746 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:07.906167  3076 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.923095  3046 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:07.912312 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:07.920538 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:07.887296  2942 raft_consensus.cc:2764] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election decision vote started in defunct term 7: won
I20260709 12:22:07.930347 31745 raft_consensus.cc:606] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:07.930987  2434 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.317s	user 0.018s	sys 0.003s
I20260709 12:22:07.892227  2964 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:07.893190  3062 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:07.891561  3075 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:07.896148  2949 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:07.778517  2943 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:07.929970  2681 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:07.913854 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:07.899654  2975 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:07.904484  2809 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:07.928939 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:07.991925 31670 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
W20260709 12:22:07.913414 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96440.7us
I20260709 12:22:07.914366 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:07.952880 31751 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:07.928263  3063 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:07.953846 31745 raft_consensus.cc:614] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:07.954663 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:07.999305 31826 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:07.925726  3038 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.056s
W20260709 12:22:07.999485 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 124696us
I20260709 12:22:08.000245 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:07.923430 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:07.931192  3019 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.686s	user 0.021s	sys 0.000s
I20260709 12:22:07.922051  2718 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:07.921886  2979 consensus_queue.cc:1243] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:22:07.875414  2966 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:07.958666  3090 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.257s	user 0.000s	sys 0.001s
I20260709 12:22:07.960675 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:08.079285 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:07.961263 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:07.961408  2474 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.288s	user 0.008s	sys 0.008s
I20260709 12:22:07.962949 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:07.906194  3059 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:07.917692  2999 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:08.069489  2938 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.268s	user 0.011s	sys 0.013s
I20260709 12:22:07.952248  3037 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:07.991918 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 78192us
I20260709 12:22:08.097621 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:07.947199 31792 leader_election.cc:336] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.760s (SENT)
W20260709 12:22:08.098418 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 151289us
I20260709 12:22:07.876554  2713 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:07.990662  3062 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:08.000500 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:08.042217 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2036) took 1717 ms. Trace:
W20260709 12:22:08.042863 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49877.8us
W20260709 12:22:08.107868 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 107434us
I20260709 12:22:08.108933 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:08.042078 31826 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:22:07.994503  2651 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:07.991108  2681 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:08.111357 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:08.084566 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:08.112375 31791 leader_election.cc:400] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:22:08.112746 31791 leader_election.cc:403] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:08.109496 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:07.954294  3076 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:08.083770 31673 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:07.979266 31669 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:07.978922  2434 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.143410 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:08.089457  2966 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:08.042119  2949 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:08.107460 31819 rpcz_store.cc:276] 0709 12:22:06.325002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.325181 (+   179us) service_pool.cc:224] Handling call
0709 12:22:06.325627 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:06.325950 (+   323us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:06.325994 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:06.408194 (+ 82200us) log.cc:844] Serialized 26 byte log entry
0709 12:22:06.556001 (+147807us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:06.556047 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:06.556138 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.848571 (+1292433us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b2400218858. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9b6e9c4 00007fe8f9b6cacb 00007fe8f9d27dc9 00007fe8f9d27a55 00007fe8f9d2709d 00007fe8fc95910e 00007fe8fc97088c 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221
0709 12:22:08.039633 (+191062us) raft_consensus.cc:1685] finished
0709 12:22:08.039657 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.042129 (+  2472us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":31248768,"thread_start_us":88592,"threads_started":1,"wal-append.queue_time_us":211839}
I20260709 12:22:08.150715 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:08.152335 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:08.113044 31824 raft_consensus.cc:2410] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 11 because replica has last-logged OpId of term: 10 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 8.
I20260709 12:22:08.085752 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:08.112059 31826 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:08.128504 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:08.095222 31751 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:22:08.145291 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:08.156102 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:08.074663  3046 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 12:22:08.099361  2971 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.095463  3015 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:07.996969  2975 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:08.158808 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56753.1us
I20260709 12:22:08.114403  2682 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260709 12:22:08.151302  3090 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:22:08.096647  3093 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:08.100975  2979 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:08.192548  2979 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.193163 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:08.161082 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:08.167325 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:08.194782 32372 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:08.168213 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:08.165280 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:08.168929 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:08.169744  3067 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260709 12:22:08.170951  3101 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:22:08.091156  2474 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.172950 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:08.144200  2809 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:08.144553  2681 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:08.161193 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:08.100656  2999 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.146036  2651 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:08.075636  3019 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:08.243305 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 59230.9us
I20260709 12:22:08.146234  3059 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:22:08.041330  3063 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.148607  2942 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:08.185113 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:08.161847  2500 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:08.166597  2568 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:08.144413  3103 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:08.179440  2844 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.661s	user 0.011s	sys 0.012s
I20260709 12:22:08.197305 32378 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:08.100660  2938 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:08.289268 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 124024us
I20260709 12:22:08.174363  3057 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:08.166265  2911 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260709 12:22:08.216559 31819 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:08.217042 31752 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:08.199337  3096 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.175107  2949 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.205868  2861 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:08.297637 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:08.233202  2910 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.112156  3075 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.175376  2971 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.195331  3093 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.251149 32373 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
W20260709 12:22:08.262918 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 87074.6us
W20260709 12:22:08.215605  3083 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.286s	user 0.000s	sys 0.001s
W20260709 12:22:08.237527  2880 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.499s	user 0.019s	sys 0.000s
I20260709 12:22:08.269735 32376 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:08.271253  3113 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.282353 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2042) took 1092 ms. Trace:
I20260709 12:22:08.239596  3067 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:08.236661  3084 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.283785 31823 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:08.293452 31673 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:22:08.299775 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:08.325848 31821 rpcz_store.cc:276] 0709 12:22:07.189625 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.189853 (+   228us) service_pool.cc:224] Handling call
0709 12:22:07.193000 (+  3147us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:07.193515 (+   515us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:07.304578 (+111063us) log.cc:844] Serialized 14 byte log entry
0709 12:22:07.571659 (+267081us) log.cc:844] Serialized 14 byte log entry
0709 12:22:07.577875 (+  6216us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:07.578533 (+   658us) log.cc:844] Serialized 26 byte log entry
0709 12:22:07.578811 (+   278us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:07.579369 (+   558us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.580915 (+  1546us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:07.957941 (+377026us) spinlock_profiling.cc:243] Waited 5.31 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:08.220308 (+262367us) spinlock_profiling.cc:243] Waited 5.8 ms on lock 0x7b18004cab00. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:22:08.282031 (+ 61723us) raft_consensus.cc:1685] finished
0709 12:22:08.282063 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.282268 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19431,"prepare.queue_time_us":124255,"prepare.run_cpu_time_us":133,"prepare.run_wall_time_us":7177,"raft.queue_time_us":7407,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":35656,"spinlock_wait_cycles":270353152,"thread_start_us":156285,"threads_started":1,"wal-append.queue_time_us":300278}
I20260709 12:22:08.332814 31819 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:08.334597 31819 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260709 12:22:08.301959 31792 leader_election.cc:336] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 2.022s (SENT)
I20260709 12:22:08.305402 31669 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:08.309401 31670 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:08.288937  2870 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.545s	user 0.010s	sys 0.003s
I20260709 12:22:08.284791  2500 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.337981 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:08.340215 31670 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 11: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:08.340225 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:08.280216  2296 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.342317  3084 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.343425 31821 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:08.346433 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 565) took 1154 ms. Trace:
I20260709 12:22:08.345288 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:08.350443 31673 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:08.204726  3062 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.346972 32371 rpcz_store.cc:276] 0709 12:22:07.192187 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.192361 (+   174us) service_pool.cc:224] Handling call
0709 12:22:07.192697 (+   336us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:07.193094 (+   397us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:07.193125 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:07.193672 (+   547us) log.cc:844] Serialized 26 byte log entry
0709 12:22:07.514446 (+320774us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:07.514507 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:07.514603 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:08.346184 (+831581us) raft_consensus.cc:1685] finished
0709 12:22:08.346209 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.346356 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4049,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":10493,"thread_start_us":143880,"threads_started":1,"wal-append.queue_time_us":574083}
I20260709 12:22:08.354624  2971 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:08.343621  2870 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.320122  3093 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:08.392827  3102 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.248s	user 0.001s	sys 0.001s
W20260709 12:22:08.393651  3030 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.736s	user 0.004s	sys 0.000s
I20260709 12:22:08.302317  3096 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:08.409019  3108 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.206s	user 0.001s	sys 0.000s
I20260709 12:22:08.410516  3092 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.411556  3092 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:08.416563 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.379680  3067 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:08.421239 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.339349  3113 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.345642  2500 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:08.418818  2793 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.744s	user 0.009s	sys 0.005s
W20260709 12:22:08.418713  3049 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.721s	user 0.015s	sys 0.000s
W20260709 12:22:08.431792 31792 leader_election.cc:336] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.872s (SENT)
I20260709 12:22:08.424439  2971 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.458398 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:08.466161 31669 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:22:08.478557 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:08.479640 31669 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:08.483079 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:08.433550  3049 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.485606 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:08.487591 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:08.487870 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:08.488336 31751 raft_consensus.cc:2376] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9. Re-sending same reply.
I20260709 12:22:08.430886  2793 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.494580 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:08.495095  3129 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.495879  3103 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:08.481876  3109 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.193s	user 0.001s	sys 0.001s
I20260709 12:22:08.498600 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:08.500056 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:08.495934  3129 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.503398  2999 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:08.503486 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:08.523492 31669 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:08.523887 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:08.525056 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:08.504750  3119 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.508520  3049 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:08.509696 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2107) took 2196 ms. Trace:
I20260709 12:22:08.510664  3056 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:08.512516 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2127) took 1946 ms. Trace:
I20260709 12:22:08.484745  3063 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:08.516659  3084 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:08.517761  2980 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:08.518131  3096 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:08.512899  2964 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.533s	user 0.011s	sys 0.000s
I20260709 12:22:08.518811  2949 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:08.519296 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:08.520843 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:08.470093  2971 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.534346 31678 rpcz_store.cc:276] 0709 12:22:06.566337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.566783 (+   446us) service_pool.cc:224] Handling call
0709 12:22:06.584335 (+ 17552us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:07.403449 (+819114us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.512218 (+1108769us) spinlock_profiling.cc:243] Waited 735 ms on lock 0x7b54001815d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:08.512448 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1691639168}
I20260709 12:22:08.528890 31674 rpcz_store.cc:276] 0709 12:22:06.313690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.313844 (+   154us) service_pool.cc:224] Handling call
0709 12:22:06.405082 (+ 91238us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:06.405309 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:06.513138 (+107829us) log.cc:844] Serialized 14 byte log entry
0709 12:22:08.509163 (+1996025us) spinlock_profiling.cc:243] Waited 3.7 ms on lock 0x7b1800158d40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287
0709 12:22:08.509281 (+   118us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:08.509338 (+    57us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:08.509400 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.509489 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.509629 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":318592,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":337656,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":91679,"spinlock_wait_cycles":8517376,"thread_start_us":626803,"threads_started":3,"wal-append.queue_time_us":550136}
I20260709 12:22:08.536480 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:22:08.537768 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:08.518496 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:08.551164 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:08.551616 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:08.496850  3104 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:08.552845 32371 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:08.555580 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:08.554420 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:08.556674 31750 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 11: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:08.527280  2984 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:08.534094  3056 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:08.541114  2912 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.564054 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:08.604552 31750 raft_consensus.cc:2376] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10. Re-sending same reply.
I20260709 12:22:08.562337 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:08.605057 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 566) took 1412 ms. Trace:
I20260709 12:22:08.562366  2964 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:08.607158 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:08.581424  3014 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.590909  2474 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:08.581609  3122 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:08.584216  2938 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:08.598783  3084 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:08.614454 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:08.615079 31749 raft_consensus.cc:1813] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Handling vote request from an unknown peer 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:08.615568 31749 raft_consensus.cc:2368] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10.
I20260709 12:22:08.615353  3014 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.616420 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:08.583101 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:08.617208  2706 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:08.617869 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:08.618040  2706 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:08.579803  3085 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:08.540795  3119 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.619112 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:08.619704 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2116) took 2253 ms. Trace:
I20260709 12:22:08.620343 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:08.620951  3119 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:08.621134  2857 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:08.621456 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:08.622042 31753 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:08.618257  2967 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:08.623102  2967 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:08.609493  2912 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.623991  2967 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.625062  2967 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:08.626111  2912 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:08.583846  3018 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 2.011s	user 0.002s	sys 0.003s
I20260709 12:22:08.549626  2999 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 3, Time since last communication: 0.050s
I20260709 12:22:08.588366  2884 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:08.586531  2979 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:08.585129  2998 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:08.600965  3113 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:08.637476  3100 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.671s	user 0.009s	sys 0.004s
I20260709 12:22:08.602128 32377 raft_consensus.cc:2376] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11. Re-sending same reply.
I20260709 12:22:08.551854  3103 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:08.592295 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:08.638491  3103 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:08.639719 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:08.639492  3103 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.641017 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:08.605563 32373 rpcz_store.cc:276] 0709 12:22:07.192502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.291893 (+ 99391us) service_pool.cc:224] Handling call
0709 12:22:07.311034 (+ 19141us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:08.313701 (+1002667us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.604719 (+291018us) spinlock_profiling.cc:243] Waited 940 ms on lock 0x7b5400544f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:08.604975 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2161149696}
I20260709 12:22:08.606915 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:08.642679 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:08.590860 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:08.593161  3129 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:08.617379  3039 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.728s	user 0.011s	sys 0.000s
I20260709 12:22:08.646807  3093 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.647754 31676 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:22:08.648234 31749 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:08.647361  3039 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.650311  3039 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:08.652030 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:08.652820 32370 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:08.565770 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:08.548595  2980 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:08.610466 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:08.611302 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:08.611680 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:08.655356 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 43805.8us
I20260709 12:22:08.655407 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:08.598021 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:08.614667  3084 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.599114  2500 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:08.660698 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:08.612924  3122 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.620085 31671 rpcz_store.cc:276] 0709 12:22:06.359711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:06.359994 (+   283us) service_pool.cc:224] Handling call
0709 12:22:08.613140 (+2253146us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b540022b354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:08.613261 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2689730816}
I20260709 12:22:08.497237  3067 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:08.581207  3121 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 12:22:08.622051 31678 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:08.560818  3096 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:08.596962  2651 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:08.604485  2984 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.636142 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:08.608295  2964 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.656757 31670 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:08.618250  3014 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:08.653812 32371 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:08.659233  2966 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:08.560438  3046 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:08.660349  3093 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:08.671036 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:08.671126  3122 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:08.675063 31675 raft_consensus.cc:2410] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:22:08.688225 31748 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
W20260709 12:22:08.676645 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.634581  2998 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:08.661861 31747 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:08.580240  3126 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.091s
I20260709 12:22:08.681725 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:08.694149 31670 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:08.682435 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:08.673825  3096 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.679189  2964 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:08.683347 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:08.683696 31671 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:08.737497 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2054) took 1042 ms. Trace:
I20260709 12:22:08.738237 31678 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:08.738580 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:08.737210 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 54844.4us
I20260709 12:22:08.690343 31669 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:08.650910  2681 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.760949 31747 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:22:08.749919  3015 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:08.691535  3046 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:08.782541 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:08.737208  3096 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:08.739450 31671 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:08.787101 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:08.683646  3056 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.787918 31747 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:08.672878 31750 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:22:08.738158 31826 rpcz_store.cc:276] 0709 12:22:07.695340 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.198006 (+502666us) service_pool.cc:224] Handling call
0709 12:22:08.198429 (+   423us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:08.285385 (+ 86956us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:08.285436 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:08.286271 (+   835us) log.cc:844] Serialized 26 byte log entry
0709 12:22:08.389109 (+102838us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:08.389160 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.623782 (+234622us) spinlock_profiling.cc:243] Waited 78.2 ms on lock 0x7b54003c91d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:08.623824 (+    42us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:08.694100 (+ 70276us) raft_consensus.cc:1685] finished
0709 12:22:08.694123 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.737407 (+ 43284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":179853696,"thread_start_us":102357,"threads_started":1,"wal-append.queue_time_us":139269}
W20260709 12:22:08.790007 31744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1620) took 2966 ms (client timeout 1760 ms). Trace:
I20260709 12:22:08.701192 31745 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:08.783397 31678 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:08.742301 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:08.784150 31824 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:08.704227  2857 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:08.780359  3015 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:08.782521  2966 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:08.811483 31670 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:22:08.817983 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:08.814029 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:08.785583  2793 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:08.822219 31669 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
W20260709 12:22:08.790578 31744 rpcz_store.cc:269] 0709 12:22:05.823290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:05.825063 (+  1773us) service_pool.cc:224] Handling call
0709 12:22:08.789781 (+2964718us) spinlock_profiling.cc:243] Waited 2.07 s on lock 0x7b54001ee1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:08.789938 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4761685376}
I20260709 12:22:08.822844 31671 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:08.824522 31748 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:08.825042 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:08.827522  3134 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:22:08.809206  2857 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.812716  3015 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:08.692370  2980 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.822851  3056 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:08.845734 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:08.823864  2971 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:08.859822 32370 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:08.852043  2980 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:08.825745  2966 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.828020  3046 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.858816  3084 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:08.864959  3091 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.077s	user 0.005s	sys 0.007s
I20260709 12:22:08.866200  2968 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:08.872514  3079 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:08.873245  3079 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:08.874891 32378 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:22:08.798292  2998 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.874513  3079 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:08.920158  3135 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:22:08.923560  3122 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:08.925738  3124 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.252s	user 0.000s	sys 0.001s
I20260709 12:22:08.926075  3122 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:08.926831  3079 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:08.930518  3096 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:08.966053 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 477) took 1715 ms. Trace:
I20260709 12:22:08.967374 31678 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:08.968899 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:08.967221 31672 rpcz_store.cc:276] 0709 12:22:07.250707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.252572 (+  1865us) service_pool.cc:224] Handling call
0709 12:22:07.252986 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:07.253329 (+   343us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:07.253354 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:08.292893 (+1039539us) spinlock_profiling.cc:243] Waited 45.8 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:22:08.303264 (+ 10371us) log.cc:844] Serialized 26 byte log entry
0709 12:22:08.420723 (+117459us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:08.420776 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:08.420864 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:08.925174 (+504310us) spinlock_profiling.cc:243] Waited 7.37 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:08.964932 (+ 39758us) raft_consensus.cc:1685] finished
0709 12:22:08.965043 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:08.965608 (+   565us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2159,"raft.queue_time_us":217402,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":20935,"spinlock_wait_cycles":122240640,"thread_start_us":76594,"threads_started":1,"wal-append.queue_time_us":235744}
I20260709 12:22:08.971225 31678 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260709 12:22:08.973026 31675 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:22:08.973570 31675 pending_rounds.cc:85] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 4
I20260709 12:22:08.973774 31678 pending_rounds.cc:85] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 7
I20260709 12:22:08.973922 31675 pending_rounds.cc:107] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 8.5
I20260709 12:22:08.974251 31672 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:22:08.975526 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:08.971210 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:08.977484 31675 raft_consensus.cc:2889] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:08.979619 31675 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: [7.8-7.8]
I20260709 12:22:08.983752 31675 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:08.974220 31678 pending_rounds.cc:107] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260709 12:22:08.987080 31678 raft_consensus.cc:2889] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:08.988722 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:08.988088 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:08.991598 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:08.992170 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:09.003787 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.010854  2500 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:09.011943  2500 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:09.013743  2500 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:08.910925  2681 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:09.024184  2971 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:09.024917  2971 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:09.025209  2857 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:09.025483  2971 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:09.027005  2966 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:09.030220 31751 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:22:09.070763  2998 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:09.072598  2964 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:09.076009 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.075500  3095 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 12:22:09.089979  3096 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:09.109727 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:09.110898 31675 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:09.115166 31673 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:09.125842 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2050) took 1805 ms. Trace:
I20260709 12:22:09.126523 31818 rpcz_store.cc:276] 0709 12:22:07.319777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.492124 (+172347us) service_pool.cc:224] Handling call
0709 12:22:07.492524 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:07.680227 (+187703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.124974 (+1444747us) spinlock_profiling.cc:243] Waited 135 ms on lock 0x7b54003c9bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:09.125509 (+   535us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":310537088}
I20260709 12:22:09.141490 31673 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:09.138482  2964 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:09.158056 31675 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:09.160574 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:09.171813 32378 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:09.177397 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:09.183369 31673 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:09.185527  2985 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:09.188966 31678 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:22:09.178191  2971 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:09.201176 31675 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:09.203714  3016 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:09.206209 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.214983 31678 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:22:09.216109  2975 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:09.216728 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.218886 31673 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:09.219375  2984 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:09.228542  3092 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:09.240960 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:09.245240  2434 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:09.242956 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 568) took 1949 ms. Trace:
I20260709 12:22:09.244264 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:09.247488  2296 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:09.247639  3056 raft_consensus.cc:2764] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:22:09.245980 32375 rpcz_store.cc:276] 0709 12:22:07.293923 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.294041 (+   118us) service_pool.cc:224] Handling call
0709 12:22:07.294366 (+   325us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:07.294602 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:07.533322 (+238720us) log.cc:844] Serialized 14 byte log entry
0709 12:22:08.864233 (+1330911us) spinlock_profiling.cc:243] Waited 24.9 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:09.241577 (+377344us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b18004dab80. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a
0709 12:22:09.242091 (+   514us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:09.242195 (+   104us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:09.242343 (+   148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.242426 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:09.242733 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":273725,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":973,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"spinlock_wait_cycles":65798656,"thread_start_us":283989,"threads_started":2,"wal-append.queue_time_us":197093}
I20260709 12:22:09.251955 32370 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: [9.8-9.8]
I20260709 12:22:09.253839 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:09.254985 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.255808 31745 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:09.249131 31748 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:09.250974 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:09.259407 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.260233 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:09.263703 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.264709 31751 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:09.264810  2651 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:09.265271  2651 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:09.266502  2651 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:09.267800  2651 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:09.270789 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:09.270848  2955 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:09.271634 32376 raft_consensus.cc:1813] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Handling vote request from an unknown peer 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:09.272162 32376 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:09.273221  2706 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:09.274621  2706 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:09.276214 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:09.276834 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.277267  3056 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:09.280215  2857 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:09.280725  2857 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:09.281183  2857 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:09.282047  3076 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:22:09.282743 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.283824 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.284718 31825 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:22:09.287171 31673 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:09.287726 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2140) took 1372 ms. Trace:
I20260709 12:22:09.288142 31677 rpcz_store.cc:276] 0709 12:22:07.915514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:07.991633 (+ 76119us) service_pool.cc:224] Handling call
0709 12:22:08.042640 (+ 51007us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:08.318900 (+276260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:08.318934 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:08.318998 (+    64us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:08.319046 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.287305 (+968259us) spinlock_profiling.cc:243] Waited 276 ms on lock 0x7b540017d9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:09.287367 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:09.287546 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":634716416}
W20260709 12:22:09.299469 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.308128  3040 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:09.310706  3019 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:09.304107 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:09.314360 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:09.298870  2955 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:09.321662  3120 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.846s	user 0.000s	sys 0.012s
W20260709 12:22:09.302517 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58519.4us
I20260709 12:22:09.329259 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:09.328310  2499 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:09.331413 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.331845  2499 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:09.333417 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:09.335260  2499 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:09.340488 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:09.341286 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:09.342098 32370 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:09.343786 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 462) took 1907 ms. Trace:
I20260709 12:22:09.344256 31824 rpcz_store.cc:276] 0709 12:22:07.435933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.194781 (+758848us) service_pool.cc:224] Handling call
0709 12:22:08.195109 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:08.795924 (+600815us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.343257 (+547333us) spinlock_profiling.cc:243] Waited 589 ms on lock 0x7b54000fa854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:09.343586 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1354157952}
I20260709 12:22:09.330657  3015 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:09.346832 31670 raft_consensus.cc:2368] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11.
I20260709 12:22:09.348001 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:09.348891 32375 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 12:22:09.359009 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:09.360900 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.359855 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:09.359041 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.362908 32378 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:09.363020 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:09.364651 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:09.363797  2971 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:09.366937  2910 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:09.367661  3056 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:09.368707 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.371795 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 573) took 1204 ms. Trace:
I20260709 12:22:09.372339 32377 rpcz_store.cc:276] 0709 12:22:08.167255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.167390 (+   135us) service_pool.cc:224] Handling call
0709 12:22:09.371358 (+1203968us) spinlock_profiling.cc:243] Waited 400 ms on lock 0x7b54006473d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:09.371564 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":920276352}
I20260709 12:22:09.377405 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.379861 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.382936 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:09.383801 32372 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:09.385429  2857 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:09.382303 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1529) took 1085 ms. Trace:
W20260709 12:22:09.391119 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 115123us
I20260709 12:22:09.392395 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.392831 32370 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:09.390738 31676 rpcz_store.cc:276] 0709 12:22:08.296873 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.297352 (+   479us) service_pool.cc:224] Handling call
0709 12:22:09.375698 (+1078346us) spinlock_profiling.cc:243] Waited 349 ms on lock 0x7b5400229f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:09.381971 (+  6273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":802648320}
I20260709 12:22:09.400185 31745 raft_consensus.cc:2410] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:22:09.399555 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.405328 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:09.406899 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.399788  2955 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.396998  2971 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:09.412624 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.413807 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.412210 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.414570 31749 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:09.410285 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.416059 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.420007 31824 raft_consensus.cc:2410] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:22:09.421676 31825 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:09.429019 31820 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:09.420532 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:09.431552 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.434281 31818 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:09.438021 31675 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:09.436728  3073 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.439628  2874 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:09.440160  3073 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:09.441262  2874 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:09.444908 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.445513 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:09.446244 31824 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:09.447220 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:09.447347  3015 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:09.449031 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:09.449250 31821 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:09.449941  3019 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:09.451273 31823 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:09.453315 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.433660 31822 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 5.
I20260709 12:22:09.457144 31749 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:09.446451 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.452630 31826 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
W20260709 12:22:09.460539 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.423197  2955 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:09.461602 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.462421 31823 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
W20260709 12:22:09.472103 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.431032  2971 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:09.473160 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.474653  2971 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:09.465263  3019 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:09.476238 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.474413 31825 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:09.483165 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:09.483587 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 80402.7us
I20260709 12:22:09.484158 31749 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:09.484766 31791 leader_election.cc:400] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10.
I20260709 12:22:09.485138 31791 leader_election.cc:403] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:09.486083  2499 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:09.487526 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:09.486464  3056 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:09.495777  3056 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:09.496201  3056 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:09.497306 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:09.498087 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.491909 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:09.503015 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.503165 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.503463 32377 raft_consensus.cc:2410] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 12 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260709 12:22:09.504050 31748 raft_consensus.cc:2410] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 9 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:22:09.492482 32375 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:09.505034  2955 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:09.493296 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:09.483180 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:09.506834 32378 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:09.495123 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:09.498478 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.509845 31747 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:09.490897  2910 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10.
I20260709 12:22:09.498829 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.494246 31753 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:09.498055 31746 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:09.493937 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 82863.7us
W20260709 12:22:09.514935 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55664.5us
I20260709 12:22:09.504637 32372 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:09.504820 31744 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 9 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 12:22:09.516122 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:09.516454  3080 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:09.519106 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:09.519702 31744 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:22:09.519819  2955 raft_consensus.cc:2764] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:22:09.520422 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:09.520581 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:09.521366 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.521540 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.503952 31749 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:09.515164 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:09.526749 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:09.533252 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.537447 31820 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:09.540405 31818 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:09.537863 31820 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:09.550547 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.557854 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.558130 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.562229 31748 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:09.537166  3073 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:09.570745  3060 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:09.553349  3015 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:09.570751 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:09.571133  2870 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:09.580744 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:09.572961 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.581787 31750 raft_consensus.cc:2410] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:22:09.587096 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:09.587917 31753 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:22:09.588501 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:09.575534 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:09.577706 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.566905 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.591256 31820 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:09.592485 31820 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:09.579239 31749 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 11: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:09.591318 31825 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:09.595968 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:09.596462 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:09.592520  3015 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:09.530939  2706 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:09.598487 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78245.7us
I20260709 12:22:09.606468 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:09.611856 31824 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:09.599398 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:09.622300 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.623070 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.603052  3015 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.625960 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.619297  3080 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:09.630365  3080 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:09.630831  3080 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:09.609243  2857 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:09.641512 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 45957.3us
W20260709 12:22:09.645335 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 80428.9us
I20260709 12:22:09.646371 31746 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:09.608632  2706 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:09.639837  3015 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:09.655287 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 136290us
I20260709 12:22:09.656509 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:09.661608 31823 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:09.659356 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:09.662585 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.663085 31825 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:09.646800  2499 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:09.665824 31824 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
W20260709 12:22:09.666226 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 178766us
W20260709 12:22:09.673059 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.658877  3073 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:09.669483 32342 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:09.674351 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.661736  2651 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:09.679719  2651 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:09.680191  2651 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:09.656368  2296 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.026s
I20260709 12:22:09.683223 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 479) took 1244 ms. Trace:
I20260709 12:22:09.683521  2296 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:09.684098  2296 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:09.685276  2296 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:09.687170 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:09.688090 31674 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:09.689420 31675 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:09.654026  2870 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.018s
I20260709 12:22:09.665683  2499 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.692804 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:09.705473  3038 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:09.705739  3019 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:09.706198  3019 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:09.703917 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 104890us
I20260709 12:22:09.708964 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:09.683714 31671 rpcz_store.cc:276] 0709 12:22:08.439225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.972638 (+533413us) service_pool.cc:224] Handling call
0709 12:22:09.412382 (+439744us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:22:09.414161 (+  1779us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:09.460119 (+ 45958us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:09.460158 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:09.477982 (+ 17824us) log.cc:844] Serialized 26 byte log entry
0709 12:22:09.550547 (+ 72565us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:09.550594 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:09.550714 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:09.682735 (+132021us) raft_consensus.cc:1685] finished
0709 12:22:09.682755 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:09.683007 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3566,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":67257,"spinlock_wait_cycles":3286400,"thread_start_us":72112,"threads_started":1,"wal-append.queue_time_us":120906}
I20260709 12:22:09.708137 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:09.693306  2296 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:09.699038  2499 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:09.719973 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61717us
I20260709 12:22:09.724076 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:09.711212  3073 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:09.728159 31824 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:09.739019 31827 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:09.739526  3099 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:09.739741 31827 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:09.740082  3099 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:09.738096 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:09.742537 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:09.745771  3056 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:09.731885  3019 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:09.747445  3019 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:09.747771  2706 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:09.748895 31818 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:09.749215 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:09.748171  3056 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:09.753883 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 82699.7us
I20260709 12:22:09.751343  2955 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:09.756168  2857 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:09.756759  2857 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:09.757704  3040 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:09.757835  2857 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:09.758862 31752 raft_consensus.cc:606] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:09.758975  3079 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:09.759788  2874 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:09.761904  3079 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:09.761596 31827 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:09.762315  3079 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:09.762717 31827 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:09.763691 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 71599us
I20260709 12:22:09.763427  3079 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:09.764739  3065 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:09.765167  3065 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:09.766098  3065 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:09.742184  3099 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:09.759399 31752 raft_consensus.cc:614] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:09.770834  2975 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.771528  2975 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:09.754972 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 89004.9us
I20260709 12:22:09.775538 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:09.776659 31573 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 149) took 22480 ms (client timeout 15000 ms). Trace:
I20260709 12:22:09.778959 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35284 (request call id 24) took 1242 ms. Trace:
I20260709 12:22:09.779423 31752 rpcz_store.cc:276] 0709 12:22:08.536109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:08.536291 (+   182us) service_pool.cc:224] Handling call
0709 12:22:09.778684 (+1242393us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54003e3ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdcf853 00007fe901d9d60a 00007fe8fb3237d4 00007fe8fb323762 00007fe8fb323691 00007fe8fb32360c 00007fe8fb3228c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:09.778852 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2779527424}
I20260709 12:22:09.756524 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:09.782773 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:09.783268 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.783414 31825 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:09.783730  2998 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:22:09.783730 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.785732 31791 leader_election.cc:400] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9.
I20260709 12:22:09.785920  2998 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:09.786211  3065 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:09.787153 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:09.787662  3056 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:09.787199  2998 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:09.788374  3056 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:09.785640  2955 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:09.798614 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.800132 31824 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
W20260709 12:22:09.808324  3153 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
W20260709 12:22:09.810079 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 77907.2us
I20260709 12:22:09.807617 31576 catalog_manager.cc:5697] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 3 to 6, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 6 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:09.810474 31823 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:09.811635 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:09.836376 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.821784 31818 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:09.814332 31820 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260709 12:22:09.858543 31820 pending_rounds.cc:85] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 6
I20260709 12:22:09.837595 31827 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 12:22:09.848673 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 117729us
I20260709 12:22:09.848359 31576 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 4 to 5, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 5 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:09.790985  3079 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:09.853927 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:09.853302 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61881.4us
I20260709 12:22:09.797498  2910 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:22:09.863379 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 60102.7us
I20260709 12:22:09.862468 31820 pending_rounds.cc:107] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260709 12:22:09.865350 31820 raft_consensus.cc:2889] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:09.866892 31576 catalog_manager.cc:5697] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 2 to 7. New cstate: current_term: 7 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:09.868692 31576 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:09.870869 31821 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:22:09.870138 31576 catalog_manager.cc:5697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:09.875488 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:09.888818 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:09.884748 31576 catalog_manager.cc:5697] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:09.892068 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 80495.4us
W20260709 12:22:09.896296 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.815330  3056 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:09.890898 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 83462.2us
I20260709 12:22:09.876677  2955 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:09.880661  2706 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:09.906803 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:09.907521  3073 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:09.907912  3127 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:09.887042  3086 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.022s
W20260709 12:22:09.908804 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 133500us
I20260709 12:22:09.903606 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:09.922590 31825 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
W20260709 12:22:09.777324 31573 rpcz_store.cc:269] 0709 12:21:47.254964 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:47.255785 (+   821us) service_pool.cc:224] Handling call
0709 12:22:09.732324 (+22476539us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:06.590192 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:06.989193 (+399001us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:06.999570 (+ 10377us) write_op.cc:435] Acquired schema lock
0709 12:22:06.999602 (+    32us) tablet.cc:662] Decoding operations
0709 12:22:07.608106 (+608504us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:07.617006 (+  8900us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:07.622822 (+  5816us) tablet.cc:685] Acquiring locks for 11 operations
0709 12:22:07.851784 (+228962us) tablet.cc:701] Row locks acquired
0709 12:22:07.851864 (+    80us) write_op.cc:260] PREPARE: finished
0709 12:22:07.852119 (+   255us) write_op.cc:270] Start()
0709 12:22:07.852318 (+   199us) write_op.cc:276] Timestamp: P: 1783599727852032 usec, L: 0
0709 12:22:07.852347 (+    29us) op_driver.cc:348] REPLICATION: starting
0709 12:22:07.853253 (+   906us) log.cc:844] Serialized 4400 byte log entry
0709 12:22:08.535935 (+682682us) op_driver.cc:464] REPLICATION: finished
0709 12:22:08.859055 (+323120us) write_op.cc:301] APPLY: starting
0709 12:22:08.862175 (+  3120us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:08.916568 (+ 54393us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:08.919683 (+  3115us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:08.925849 (+  6166us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:08.926191 (+   342us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=11
0709 12:22:08.926234 (+    43us) write_op.cc:312] APPLY: finished
0709 12:22:08.926709 (+   475us) log.cc:844] Serialized 83 byte log entry
0709 12:22:08.929570 (+  2861us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:08.933620 (+  4050us) write_op.cc:454] Released schema lock
0709 12:22:08.933737 (+   117us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:08.934121 (+   384us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:08.934141 (+    20us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8365066,"reported_tablets":27,"child_traces":[["op",{"apply.queue_time_us":283238,"apply.run_cpu_time_us":4729,"apply.run_wall_time_us":81785,"num_ops":11,"prepare.queue_time_us":509260,"prepare.run_cpu_time_us":57537,"prepare.run_wall_time_us":1492277,"raft.queue_time_us":198008,"raft.run_cpu_time_us":23925,"raft.run_wall_time_us":401596,"replication_time_us":577158,"thread_start_us":706920,"threads_started":4,"wal-append.queue_time_us":258243}]]}
W20260709 12:22:09.928965 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:09.930320 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.920574 31576 catalog_manager.cc:5697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 6 to 9, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 9 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:09.945252 31576 catalog_manager.cc:5697] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 3 to 6, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 6 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:09.932332  3060 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:09.939334  2874 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:09.968680 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 54886.3us
W20260709 12:22:09.970570 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.971773 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:09.942355  3046 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:09.974711 31576 catalog_manager.cc:5697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 3 to 7, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 7 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:09.944473  2910 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:09.986889 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44338.7us
W20260709 12:22:09.988472 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:09.995635 31576 catalog_manager.cc:5697] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:22:09.996446 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47076.6us
I20260709 12:22:09.972667  2874 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:10.005767 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:09.980193  2975 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:10.005033 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 98601.1us
I20260709 12:22:10.013800 31818 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:09.981618  3127 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:10.018327 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.004251  2874 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:09.981034  2651 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.026880  2910 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.029774  2499 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:10.038455  2706 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:10.039074  2706 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:10.003834  3163 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:10.058765  2706 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:10.063266 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:10.067328 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:10.070878  3163 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:10.070977 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:10.071846  3076 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:10.072242  2651 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:10.071995  3163 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:10.073544 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.073913 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:10.074939  3167 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:22:10.075326 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53832.7us
I20260709 12:22:10.077338  3079 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:22:10.056699  3073 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:10.079077  3073 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:10.079481  3073 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:10.064858  2706 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:10.013221  3080 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:10.057236 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:10.084198  3080 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:10.084247 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 63479.7us
I20260709 12:22:10.085533  3058 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:10.063858  3015 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.085918 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:10.086813  2955 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:10.061914  3019 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:10.087229  2955 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:10.060823  3164 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:10.072535  3076 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:10.087723  2955 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:10.089938  3127 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.061187  3162 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:10.103471  2499 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:10.103816  3162 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:10.086042  3058 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:10.104915  3162 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:10.108722  2955 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.111254  3015 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:10.112155  3015 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:10.113672  3015 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:10.097556  2857 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:10.100929  3156 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.083s	user 0.002s	sys 0.000s
I20260709 12:22:10.101735 32372 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:22:10.114907 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51737.5us
I20260709 12:22:10.115602  3159 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:10.118126 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:10.057593 31576 catalog_manager.cc:5697] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:10.080895 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:10.096737  3099 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.143151  3015 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:10.162081 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 156406us
I20260709 12:22:10.162367 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:10.163722 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 45708.1us
I20260709 12:22:10.168898 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:10.173080 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 92247.7us
W20260709 12:22:10.170090 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 198396us
I20260709 12:22:10.163606  3015 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:10.110272  3058 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:10.176332 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:10.185372 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:10.186419 31825 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:22:10.180150 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:10.186439 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 228074us
I20260709 12:22:10.168668  2955 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:10.191362 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:10.192710 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:10.175218  3079 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:10.175585 31576 catalog_manager.cc:5697] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 1 to 7. New cstate: current_term: 7 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:10.183115  3159 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:22:10.201246 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.194933 32342 leader_election.cc:400] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11.
I20260709 12:22:10.166695  2857 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:10.233469 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 45204.2us
I20260709 12:22:10.235786 31791 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:10.196285  2998 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election won for term 8
W20260709 12:22:10.241895 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 156061us
W20260709 12:22:10.245870 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53371.6us
I20260709 12:22:10.248517 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:10.175658  2975 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.056s
W20260709 12:22:10.251566 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.237944  3079 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:10.254710  3079 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.258689  3165 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:10.262720  3040 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:10.263233  3099 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:10.267165  3046 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:10.267946  3058 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:10.237064  2955 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:10.276783  2874 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:10.281487 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:10.282099 32372 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:10.283552 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:10.285439 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 49805.5us
I20260709 12:22:10.272215 31576 catalog_manager.cc:5697] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:10.343598 31671 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:10.250133  3159 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:10.345317 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50987us
W20260709 12:22:10.347416 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51793.7us
W20260709 12:22:10.346385 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.263250  3040 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:10.358929  3040 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:10.359066 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:10.360487 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:10.361619  2651 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:10.362347  2651 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:10.362844  2651 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:10.296204  2998 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:10.363986  2651 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:10.365059  2998 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.374773  3056 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:10.271677  3065 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.379026  3065 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.380195  2975 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.380721  2975 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.381321  3065 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:10.384899 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:10.385602 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:10.386556 31750 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:10.387156  3065 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:10.386987 31576 catalog_manager.cc:5697] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:10.388458  3167 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:10.388988  3167 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:10.388444 31576 catalog_manager.cc:5697] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 1 to 4, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 4 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:10.389909  3167 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.392135 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:10.394838 31751 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:10.269903  2857 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:10.387017 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:10.412751 31671 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:10.278620  3046 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:10.416551 31576 catalog_manager.cc:5697] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 3 to 5, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 5 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:10.420208 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:10.417124 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 70118us
I20260709 12:22:10.424726 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:10.428871 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:10.391353  3167 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:10.439713 31672 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:10.424321 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:10.445750 31677 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:10.412454  3172 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:10.442847 31678 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:10.418167  2975 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:10.418351 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 169934us
W20260709 12:22:10.444466 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51200.1us
I20260709 12:22:10.423738 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:10.423396 31751 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
W20260709 12:22:10.457548 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 112847us
I20260709 12:22:10.458529  3179 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:10.459453 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:10.459566  3164 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:10.460574  2857 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:10.484071 31674 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:10.461437  3165 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:10.443333 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 56407.9us
I20260709 12:22:10.465343 31752 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:10.469700 31669 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:10.422353  3065 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:10.419138  3046 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:10.467154 31670 raft_consensus.cc:2376] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11. Re-sending same reply.
I20260709 12:22:10.423847  2955 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.452471  3160 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:10.485693 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:10.530370 31753 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:10.491902 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:10.531189 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 64815.1us
I20260709 12:22:10.443915  2499 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:10.523041 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:10.544305 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 81713.2us
I20260709 12:22:10.545315 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:10.549242 31821 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:10.552001 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:10.552354 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:10.431027  3073 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.552999 31746 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:10.541394 31672 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:10.525452  3162 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:10.555428 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:10.563663 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:10.561959 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:10.520219  3065 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:10.573076 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:10.558182 31576 catalog_manager.cc:5697] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 2 to 6, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 6 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:10.491564  3157 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:10.588887 31669 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:22:10.593935 31677 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
W20260709 12:22:10.596975 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 86796.8us
W20260709 12:22:10.607556 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 119620us
I20260709 12:22:10.621692 31746 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:10.589196  3065 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:10.594094  3086 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:10.602480  3163 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:10.556449  2499 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:10.548575  2857 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.656333 32372 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
W20260709 12:22:10.659229 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67806.8us
W20260709 12:22:10.665120 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 92124.3us
I20260709 12:22:10.585773  3076 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.061s
W20260709 12:22:10.669603 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 61120.5us
I20260709 12:22:10.572010  2955 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:10.573673  2296 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.082s
I20260709 12:22:10.539752  3165 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.708164 31675 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:10.641669  3086 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:10.642568  3179 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:10.715039 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:10.579486  3162 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:10.668208  2975 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:10.692231  3192 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.692669  3172 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:10.709378  3176 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:10.735657 31674 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:10.609163  3157 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.736325 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:10.741569  3177 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:10.718408  3086 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:10.746987 31746 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:22:10.563217  3159 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:10.762249 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 230410us
I20260709 12:22:10.763440 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:10.609576  3160 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:10.778992 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 87239.1us
I20260709 12:22:10.787662 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:10.792598  3168 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:22:10.743397  2975 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:10.744493  3076 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:10.750128  3176 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:10.743578  2874 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:10.823518 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:10.833432 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:10.720299  3046 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:10.801872  3177 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:10.853029 31677 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:10.854058 31745 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11. Ops: [10.5-10.5]
I20260709 12:22:10.820590  3176 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:10.862991 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 148023us
I20260709 12:22:10.863413 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:10.869791 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:10.863724 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:10.864197 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:10.864699  3195 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:10.744825  3192 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.747779  3172 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:10.897087 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:10.898715 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:10.898851  2874 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.899501  2874 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.898919 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:10.801283  3191 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.073s
I20260709 12:22:10.853724  3073 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:10.863040  2955 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:10.864708  3163 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:10.883020  3195 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:10.922390 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:10.916365  3058 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.648s	user 0.009s	sys 0.009s
I20260709 12:22:10.909132  3076 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:10.924292 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:10.937835 31670 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 6: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:10.914821  3046 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:10.908583  3189 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:10.914588  2651 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:10.939961 31678 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
W20260709 12:22:10.949324 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:10.949735 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 186466us
I20260709 12:22:10.950955  3209 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260709 12:22:10.951435  3209 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:10.951634  3169 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.692s	user 0.000s	sys 0.004s
I20260709 12:22:10.951835  3209 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:10.930995  2857 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:10.925374  3195 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:10.950253  3204 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:10.931555  2955 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:10.986017  3185 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.233s	user 0.000s	sys 0.001s
I20260709 12:22:10.952872  3073 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:10.944422  2874 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:10.981084 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:11.032671 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 131410us
I20260709 12:22:10.985941 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:10.867110  3179 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:10.958297  2499 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:10.980904 31669 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:22:11.055399 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:10.959897  3056 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.584s	user 0.010s	sys 0.002s
W20260709 12:22:11.026680  3174 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.299s	user 0.001s	sys 0.000s
I20260709 12:22:10.931236  3163 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.070019  3163 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:11.066931  3205 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.071405  3205 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.073823  3205 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:11.074719 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:10.940761  2910 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.090s
I20260709 12:22:11.076063  3194 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:11.076201 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:11.076617  3194 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:11.076886 31824 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:11.077657 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:11.078559 31827 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:11.078838  3182 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.309s	user 0.001s	sys 0.000s
I20260709 12:22:11.080978 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:11.081723 31823 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:11.082284 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:11.082998 32370 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:10.937608  2975 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.057375 31751 raft_consensus.cc:606] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:11.028051  2857 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.060690 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:11.037366  3040 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.678s	user 0.006s	sys 0.005s
W20260709 12:22:11.066941  3187 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.327s	user 0.000s	sys 0.002s
I20260709 12:22:11.067253  3211 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:11.070513  3216 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:11.059959  2874 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:10.983480  3058 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:11.045251 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:11.056109 31744 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 6.
I20260709 12:22:11.037669  3162 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:11.148119 31746 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 9 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 12:22:11.125288 31747 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:11.042508  3099 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:11.126906 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:11.141656 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:11.148132 31670 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:11.036162  3204 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.143582 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:11.153316 31818 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 5.
I20260709 12:22:11.077812 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:11.154363 31824 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:10.959026  3189 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.047765  2651 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.090s
I20260709 12:22:11.083793  3056 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:11.162696 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:11.168663 32376 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:11.169425 32375 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:11.064618  3165 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:11.179347 31824 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:11.181811 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:11.057811  3073 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:11.184942 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 260848us
I20260709 12:22:11.147204  2870 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.142082  3211 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:11.186391 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:11.053112  2955 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.141767  3157 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:11.162420  2910 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.195233 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:11.197496 31824 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 10
I20260709 12:22:11.141593  2857 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.207947 31744 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:11.201790 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:11.209460 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:11.211062 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:11.144526  3040 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.168457  3099 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.043644  3177 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.220031 31825 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:11.198884  3160 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:11.181638  2998 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:22:11.198994  3211 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:11.166787  2874 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.226434 31821 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:11.226992 31823 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:11.195142  3162 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 12:22:11.168797  3216 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.077682  3194 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.256688 31825 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
W20260709 12:22:11.264633 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 115242us
I20260709 12:22:11.265831 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:11.271447 31669 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:22:11.226572  3192 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:11.275995 31753 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.706s	user 0.012s	sys 0.003s
I20260709 12:22:11.210384  2870 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.211982  2910 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.281912 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:11.267287  3198 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
W20260709 12:22:11.268252  3015 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.070s	user 0.013s	sys 0.000s
I20260709 12:22:11.258648  3172 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:11.270859  3221 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:11.291703 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 335892us
I20260709 12:22:11.294165 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:11.292145 31753 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:11.223842 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:11.301613 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 77836.9us
I20260709 12:22:11.281682  2296 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:11.292987  3221 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.307024  3203 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.307032  3228 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260709 12:22:11.287542  3015 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.309260 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:11.311902 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:11.326001 31824 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
W20260709 12:22:11.366132 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.289796  3204 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:11.323266  3065 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.630s	user 0.006s	sys 0.008s
I20260709 12:22:11.308467  3188 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:11.387938 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 494) took 1003 ms. Trace:
I20260709 12:22:11.337292 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:11.323945  3221 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:11.401578 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2180) took 1007 ms. Trace:
I20260709 12:22:11.404700 31753 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 8. (term mismatch)
I20260709 12:22:11.428311 31753 pending_rounds.cc:85] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 7
I20260709 12:22:11.428617 31753 pending_rounds.cc:107] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 10.8
I20260709 12:22:11.428889 31753 raft_consensus.cc:2889] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:11.323700  3192 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:11.426826 31671 rpcz_store.cc:276] 0709 12:22:10.394299 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.419420 (+ 25121us) service_pool.cc:224] Handling call
0709 12:22:10.443565 (+ 24145us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:10.536919 (+ 93354us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:10.536951 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:10.537527 (+   576us) log.cc:844] Serialized 26 byte log entry
0709 12:22:10.639386 (+101859us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:10.639426 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.639495 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:11.401320 (+761825us) raft_consensus.cc:1685] finished
0709 12:22:11.401350 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:11.401510 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15117,"raft.queue_time_us":163600,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":21188,"thread_start_us":125175,"threads_started":2,"wal-append.queue_time_us":359760}
I20260709 12:22:11.385547  3165 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:11.419152 31752 rpcz_store.cc:276] 0709 12:22:10.384814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.390782 (+  5968us) service_pool.cc:224] Handling call
0709 12:22:10.391172 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:10.491340 (+100168us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.387688 (+896348us) spinlock_profiling.cc:243] Waited 74 ms on lock 0x7b54004051d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:11.387881 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":170139520}
I20260709 12:22:11.331593  3228 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.384680  3220 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:11.324890  2296 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.386641  3157 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.398262  3213 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:11.443562  3180 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.261s	user 0.000s	sys 0.001s
W20260709 12:22:11.443934  3196 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.269s	user 0.000s	sys 0.001s
W20260709 12:22:11.454304 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.454905 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:11.477658 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:11.455714 31676 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:11.478451 31671 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
W20260709 12:22:11.416326 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 105359us
I20260709 12:22:11.430856 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:11.337965  3203 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.437235  3188 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:11.499603 32370 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:11.452397  3228 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:11.452917  3220 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:11.466182  3189 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:11.501178 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 163979us
I20260709 12:22:11.503813 31744 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:11.509043 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 481) took 1012 ms. Trace:
I20260709 12:22:11.459532  3165 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:11.425678  3065 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.446177  3076 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:11.479255 31670 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:11.528072 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:11.475237  2857 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:11.539669 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:11.543835 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:11.550510 31824 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:11.508430  3040 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:11.536675 31826 rpcz_store.cc:276] 0709 12:22:10.497121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.497247 (+   126us) service_pool.cc:224] Handling call
0709 12:22:10.550284 (+ 53037us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:10.550492 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:10.650311 (+ 99819us) log.cc:844] Serialized 14 byte log entry
0709 12:22:11.226425 (+576114us) spinlock_profiling.cc:243] Waited 2.75 ms on lock 0x7b1800350d40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:22:11.508702 (+282277us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:11.508752 (+    50us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:11.508800 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.508866 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:11.508987 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":35886,"prepare.queue_time_us":130386,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":122266,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":13870,"spinlock_wait_cycles":6321408,"thread_start_us":304089,"threads_started":3}
I20260709 12:22:11.477684  3157 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.563493 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:11.566563 32370 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:11.561810 32375 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:11.567754  3233 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.567955  3199 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:11.568540  3065 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:11.492499  3192 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.055s
W20260709 12:22:11.569844  3230 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:22:11.561101  2910 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:11.572837 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:11.492985  3213 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:11.601410 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 169346us
I20260709 12:22:11.531982  3217 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:11.646479 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:11.647159  3237 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:11.647807 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:11.647811  3181 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.007s	user 0.003s	sys 0.002s
W20260709 12:22:11.648586  3214 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:22:11.664985 32371 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:11.670717 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:11.671550 32375 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
W20260709 12:22:11.650203 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:11.552346  2874 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:11.576207  2955 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:11.560847  3238 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:11.669653 32370 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:11.663216  3015 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:11.682349 31824 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:11.683851 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:11.684672 31824 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:22:11.691339  3197 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.685920  3238 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:11.691890 31677 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:22:11.692307  3223 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.692015  3197 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.692852  3223 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.693802  3215 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.674271  3217 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.700640 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:11.698854 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 135447us
I20260709 12:22:11.702937 32372 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:11.732173 31678 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 8 index: 4. (index mismatch)
I20260709 12:22:11.732791  3203 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:11.692061  3238 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:11.734771  3213 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:11.659942  3233 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.738014  3233 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:11.700471  3223 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:11.741119  3217 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:11.747663  3167 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.283s	user 0.016s	sys 0.000s
I20260709 12:22:11.748571  3199 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:11.749222  3199 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:11.753665  3209 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.801s	user 0.008s	sys 0.003s
I20260709 12:22:11.754763  3165 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.756239 31748 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
W20260709 12:22:11.674463  3176 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.757s	user 0.005s	sys 0.007s
I20260709 12:22:11.766544 31826 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:11.765785 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 63939.6us
W20260709 12:22:11.797840 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 71721.3us
I20260709 12:22:11.799571 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:11.804780 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:11.667534  3172 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:11.763093  3046 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.785s	user 0.009s	sys 0.005s
I20260709 12:22:11.819624 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:11.819711 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2178) took 1426 ms. Trace:
I20260709 12:22:11.819665 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2181) took 1400 ms. Trace:
I20260709 12:22:11.667922  3216 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:11.726697  2870 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:11.826321 31673 rpcz_store.cc:276] 0709 12:22:10.393718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.393852 (+   134us) service_pool.cc:224] Handling call
0709 12:22:10.426872 (+ 33020us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:10.742084 (+315212us) spinlock_profiling.cc:243] Waited 3.18 ms on lock 0x7b180044be40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcdcdbfb 00007fe8fcdd864b
0709 12:22:10.742235 (+   151us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:10.936550 (+194315us) log.cc:844] Serialized 14 byte log entry
0709 12:22:11.290893 (+354343us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:11.291436 (+   543us) log.cc:844] Serialized 26 byte log entry
0709 12:22:11.295344 (+  3908us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:11.295386 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.295460 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:11.763516 (+468056us) raft_consensus.cc:1685] finished
0709 12:22:11.819475 (+ 55959us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:11.819624 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":85638,"prepare.queue_time_us":477959,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":118271,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":23825,"spinlock_wait_cycles":7323008,"thread_start_us":492611,"threads_started":3,"wal-append.queue_time_us":728436}
I20260709 12:22:11.827297 31674 rpcz_store.cc:276] 0709 12:22:10.419028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.533860 (+114832us) service_pool.cc:224] Handling call
0709 12:22:10.570282 (+ 36422us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:10.743146 (+172864us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.819400 (+1076254us) spinlock_profiling.cc:243] Waited 159 ms on lock 0x7b54001d5954. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:11.819601 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":366073088}
I20260709 12:22:11.834812 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:11.694538  3197 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:11.795502  3079 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:11.840705  3079 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:11.841229 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:11.838038 31748 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:22:11.843159 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:11.843598 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:11.754175  3209 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.850556 31824 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:11.853649 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:11.853885 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 83524.1us
I20260709 12:22:11.694507  3215 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.859232 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:11.832429  2296 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:11.862795 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:11.863286 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:11.863886 31748 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:11.866187 31670 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:11.844880  3192 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:11.867362 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:11.869866 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:11.851486  3216 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:11.853893  3015 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:11.860140  3207 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:11.871524 31824 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:11.872601 31678 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:22:11.881412 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:11.873486 31673 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:11.873522 31822 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:11.879474 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:11.873941 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:11.871791 31791 leader_election.cc:400] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10.
I20260709 12:22:11.875872 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2166) took 1342 ms. Trace:
I20260709 12:22:11.700222  3076 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.870869  3193 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:11.675400  3188 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:11.871353  3086 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.083s	user 0.009s	sys 0.006s
I20260709 12:22:11.685874  3220 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.882368 31672 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:11.826928  3176 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.892840 31674 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:11.893536 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:11.854452  3233 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 12:22:11.899936 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:11.893004 31749 rpcz_store.cc:276] 0709 12:22:10.525140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.525984 (+   844us) service_pool.cc:224] Handling call
0709 12:22:10.555500 (+ 29516us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:10.580414 (+ 24914us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:10.580446 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:10.604217 (+ 23771us) log.cc:844] Serialized 26 byte log entry
0709 12:22:10.808587 (+204370us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:10.808627 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:10.816132 (+  7505us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:11.867460 (+1051328us) raft_consensus.cc:1685] finished
0709 12:22:11.867490 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:11.867648 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24538,"raft.queue_time_us":25061,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":25683,"thread_start_us":104658,"threads_started":1,"wal-append.queue_time_us":835183}
I20260709 12:22:11.863389  2870 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 9.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.029s
I20260709 12:22:11.921172 31678 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:11.921350 31824 raft_consensus.cc:2410] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:22:11.923218 31822 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 8. (term mismatch)
I20260709 12:22:11.923656 31822 pending_rounds.cc:85] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 7
I20260709 12:22:11.923967 31822 pending_rounds.cc:107] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 9.8
I20260709 12:22:11.924283 31822 raft_consensus.cc:2889] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:11.901170  3207 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:11.891139  3205 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 12:22:11.935707 31672 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:11.938993 31826 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:11.939363 31826 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:11.922761  3157 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:11.927644  3046 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:11.918044  3086 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.967552  3099 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:11.883591  3079 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:11.962453 31826 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:11.980572 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:11.980865 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:11.981298 31748 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:11.915803  3193 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:12.013473 31827 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:11.964676  3046 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:12.018584 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 146924us
I20260709 12:22:11.974906  3203 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.019762 31826 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:12.025473 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.032263 31827 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260709 12:22:12.015919  3172 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.038928  3172 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:12.041494  3195 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.010s	user 0.012s	sys 0.000s
I20260709 12:22:11.969858  3228 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:12.036396  3250 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
W20260709 12:22:12.038380  3249 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:22:12.041410  3172 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:12.042062 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:12.037535  3251 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:22:12.023921  3157 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:12.038380  3252 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:22:12.047485 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:12.022164  3079 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:12.053292 31744 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:12.055748  3228 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:12.057945  3207 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.042063  3195 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:12.091403  3227 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.200s	user 0.001s	sys 0.000s
I20260709 12:22:12.051164  3209 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:12.028450  3203 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:12.058971  3221 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.605s	user 0.012s	sys 0.000s
I20260709 12:22:12.094103 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.034657  3159 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
I20260709 12:22:12.107225 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:12.108315 32378 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:22:12.073200  2296 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:12.115224 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:12.116257 32375 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 10: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:12.100468 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.086280  3172 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:12.142773  3172 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.121910 31748 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:12.122908  3209 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:12.143867  3209 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:12.144335  3209 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:12.126430  3073 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:12.127708 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.099887  3215 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:12.146263 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.150182 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.159162 31746 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:12.129649  3203 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:12.130342  3242 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.299s	user 0.000s	sys 0.001s
I20260709 12:22:12.131446  3086 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:12.071234  3157 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.133002  3193 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:12.121829  3237 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:12.136034 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 94066.5us
W20260709 12:22:12.136286  3211 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.882s	user 0.012s	sys 0.000s
I20260709 12:22:12.060290  3046 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:12.098876  3197 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.137595 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 503) took 1100 ms. Trace:
W20260709 12:22:12.137689  3225 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.717s	user 0.000s	sys 0.003s
I20260709 12:22:12.139665  3176 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:12.141537  3195 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:12.142110 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.191007  2910 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:12.139724 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.189687 31744 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:12.189877 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:12.190392 31674 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
W20260709 12:22:12.215682 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 76049.1us
I20260709 12:22:12.192052 31826 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 4. (term mismatch)
I20260709 12:22:12.115801  3233 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:12.197450 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.199532 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:12.219509 32376 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:12.110014  3221 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.121662  2296 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:12.206750 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:12.223173 32371 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:12.203840 31669 rpcz_store.cc:276] 0709 12:22:11.037485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.056011 (+ 18526us) service_pool.cc:224] Handling call
0709 12:22:12.137379 (+1081368us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b540017cfd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:12.137512 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":281291520}
I20260709 12:22:12.223783 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:12.225003 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:12.205600 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:12.216838 31826 pending_rounds.cc:85] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 3
I20260709 12:22:12.217286 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:12.226786 31826 pending_rounds.cc:107] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260709 12:22:12.217427  3073 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:12.218132 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.220599 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:12.223364  3192 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.237038 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.227088 31826 raft_consensus.cc:2889] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:12.227909 31744 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:12.230412  3216 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:12.219251  3233 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:12.230962 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:12.218895  3046 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:12.232373 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.203639  3211 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.232474 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:12.228389  3073 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:12.232489 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.225572  3195 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20260709 12:22:12.182964  3238 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.237125 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35284 (request call id 25) took 1842 ms. Trace:
I20260709 12:22:12.214532  3215 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:12.231446  3172 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:12.242672 31818 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:12.209757  3197 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:12.246344 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.246388 31751 rpcz_store.cc:276] 0709 12:22:10.416016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.416188 (+   172us) service_pool.cc:224] Handling call
0709 12:22:12.236962 (+1820774us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b54000b86d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdcf853 00007fe901d9d60a 00007fe8fb3237d4 00007fe8fb323762 00007fe8fb323691 00007fe8fb32360c 00007fe8fb3228c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:12.237059 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1356340992}
I20260709 12:22:12.250679 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:12.240360  3233 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:12.243216  3073 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:12.255101 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:12.216557  3199 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:12.256508 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:12.256449 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:12.258252  3197 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:12.258844  3015 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:12.259085 32377 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 10. (term mismatch)
I20260709 12:22:12.283651 32377 pending_rounds.cc:85] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 9
I20260709 12:22:12.283812 31749 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:12.283951 32377 pending_rounds.cc:107] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 9.10
I20260709 12:22:12.284274 32377 raft_consensus.cc:2889] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:12.259215  3206 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.199s	user 0.000s	sys 0.007s
I20260709 12:22:12.265105 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:12.268687  3211 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:12.268898 31671 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:12.269726 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:12.271611 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:12.273553 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.286355 31825 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
W20260709 12:22:12.274463 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58424.2us
I20260709 12:22:12.226593  2910 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.280102 31819 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:12.243264  3179 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:12.285305 31824 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:12.288062 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.289018 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:12.291050  3195 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:12.291508  3195 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:12.291911  3195 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:12.292419  3238 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:12.295122 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.240605  3192 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.299706  3199 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:12.310940 31822 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260709 12:22:12.314117  3219 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.027s	user 0.005s	sys 0.000s
I20260709 12:22:12.316726  3205 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:22:12.318746 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.321060 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.321194 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.321821 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2162) took 1850 ms. Trace:
I20260709 12:22:12.321847 31751 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:22:12.320397  3209 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:12.322338 31750 rpcz_store.cc:276] 0709 12:22:10.471780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.471919 (+   139us) service_pool.cc:224] Handling call
0709 12:22:10.520386 (+ 48467us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:10.567126 (+ 46740us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:10.567161 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:10.633242 (+ 66081us) log.cc:844] Serialized 52 byte log entry
0709 12:22:10.775388 (+142146us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:10.989783 (+214395us) log.cc:844] Serialized 14 byte log entry
0709 12:22:11.038553 (+ 48770us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.038678 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.224944 (+1186266us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:12.321529 (+ 96585us) raft_consensus.cc:1685] finished
0709 12:22:12.321627 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.321766 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17100,"prepare.queue_time_us":366881,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":135083,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":23711,"spinlock_wait_cycles":24751872,"thread_start_us":280116,"threads_started":2,"wal-append.queue_time_us":211326}
I20260709 12:22:12.324638  3209 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:12.325819  3060 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:12.327028 31818 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:12.328029 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:12.332134 31670 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:12.335271 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:12.336359 31669 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:12.338117 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.338965 31753 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:12.297524  3172 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:12.349777 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.276010  3215 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.357080 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.358084 31750 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 12:22:12.329156  3211 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:12.357514 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:12.359412 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.368191 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 486) took 1563 ms. Trace:
I20260709 12:22:12.370039 31820 rpcz_store.cc:276] 0709 12:22:10.804646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.804836 (+   190us) service_pool.cc:224] Handling call
0709 12:22:10.822048 (+ 17212us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:10.846667 (+ 24619us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:10.846710 (+    43us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:10.880588 (+ 33878us) log.cc:844] Serialized 52 byte log entry
0709 12:22:11.150387 (+269799us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:11.152870 (+  2483us) log.cc:844] Serialized 14 byte log entry
0709 12:22:11.164188 (+ 11318us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:11.164270 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:12.124006 (+959736us) spinlock_profiling.cc:243] Waited 49 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:12.367741 (+243735us) raft_consensus.cc:1685] finished
0709 12:22:12.367887 (+   146us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:12.368101 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10536,"prepare.queue_time_us":1689,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":22284,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":33388,"spinlock_wait_cycles":112605056,"thread_start_us":229258,"threads_started":1,"wal-append.queue_time_us":375032}
I20260709 12:22:12.373301  3015 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:12.374608  3015 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:12.384969 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57043.6us
I20260709 12:22:12.386487 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:12.386920 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:12.359194  3172 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:12.388145  3060 raft_consensus.cc:2764] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election decision vote started in defunct term 8: won
I20260709 12:22:12.388859  3172 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.393685  3195 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.396237 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:12.397425 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:12.379618 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:12.398492 31672 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:12.399015 31669 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 10 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 12:22:12.403843 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.402087 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.406924 31676 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:12.311040  3179 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.415482  3221 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:12.345803  3237 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:12.423034 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.416039  3192 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:12.428192 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:12.429773 31818 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 6.
I20260709 12:22:12.430351 31674 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:12.399735  3157 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:12.413327  3223 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.056s
I20260709 12:22:12.469074  3220 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:12.424721  2975 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260709 12:22:12.425084  2874 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.022s
I20260709 12:22:12.471534 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2189) took 1079 ms. Trace:
I20260709 12:22:12.471932 31671 rpcz_store.cc:276] 0709 12:22:11.391746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.528570 (+136824us) service_pool.cc:224] Handling call
0709 12:22:11.546881 (+ 18311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:12.285468 (+738587us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:12.471269 (+185801us) spinlock_profiling.cc:243] Waited 722 ms on lock 0x7b540017c5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:12.471458 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1659801984}
I20260709 12:22:12.476029  3179 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:12.476612 31826 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:22:12.428103  3162 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:12.478113  3162 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:12.450866  3207 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:12.416998  3058 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:12.462718  3238 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:12.435555 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.478575  3162 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:12.479389  3176 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.003s
I20260709 12:22:12.482610  3176 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:12.464704  3205 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:12.466037  3194 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:12.466218  2910 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:12.468209  3215 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:12.435912  3237 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:12.457149  2296 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:12.490921 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53021.3us
W20260709 12:22:12.499284 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 210294us
I20260709 12:22:12.502964 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2094) took 1334 ms. Trace:
I20260709 12:22:12.509114 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.508584 31823 rpcz_store.cc:276] 0709 12:22:11.149069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:11.181528 (+ 32459us) service_pool.cc:224] Handling call
0709 12:22:12.483267 (+1301739us) spinlock_profiling.cc:243] Waited 44.5 ms on lock 0x7b540037a854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:12.483455 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":102236928}
W20260709 12:22:12.516571 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 81082us
I20260709 12:22:12.517460 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.518592 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.483489  3058 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:12.520831 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.483171  3176 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:12.521991 32378 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 11 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:22:12.539302 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.544715 31750 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:22:12.525007 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:12.531316 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.546490 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.535784  3192 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:12.547623  3076 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:12.583134 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:12.551172 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.583849 31674 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:12.581913 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:12.551641 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.552286  3199 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:12.552150  3073 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:12.524684  2870 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:12.520526  3233 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:12.539832  3223 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:12.554917  3237 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.539602  3215 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.017s
I20260709 12:22:12.538686  3058 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:12.561968  3216 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:12.563459  3176 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:12.566578  3195 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:12.584873 31670 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:12.585618 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.631716 31818 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:12.632488 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:12.630283 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44734.4us
I20260709 12:22:12.588816 31675 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:12.592747 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 194833us
W20260709 12:22:12.593748 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 206922us
I20260709 12:22:12.595953 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:12.598474 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:12.636350 31669 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:12.636442 31678 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 12: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:12.636765 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:12.600103 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:12.637426 32377 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:12.637837 31672 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:12.592444  3267 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:12.593547  2870 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:12.607391 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:12.598608  3223 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:12.616317 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:12.647977 31824 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:12.619433 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:12.589576  3076 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:12.583074  3192 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:12.587996  2706 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:12.588249  3199 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:12.649999 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:12.622066 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:12.633342 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:12.652926 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:12.653328 31825 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:22:12.583961  2975 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:12.637115 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:12.655958 31752 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:12.656250 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:12.588083  3207 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260709 12:22:12.635028 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:12.662146 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:12.586956 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:12.589040  3197 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:12.619185  3238 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.656001 31821 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 5 FOLLOWER]: Advancing to term 7
I20260709 12:22:12.659416 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.670564 32370 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:12.670827 31747 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:12.659170 31674 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:22:12.638535  3172 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:12.595082  3221 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260709 12:22:12.660786 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.676609 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:12.650285  3223 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:12.646808  2870 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:12.682119 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:12.648115  3267 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:12.683430 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:12.684115 32371 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:12.684841 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:12.686161 31822 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:12.656267  3199 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:12.652567  3076 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:12.691128 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:12.661626  3176 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:12.638967  3165 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260709 12:22:12.656795  3192 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:12.674508  3216 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:12.646315  3211 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.037s
I20260709 12:22:12.677121  3209 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Leader election won for term 6
I20260709 12:22:12.699055 31744 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:12.671533  2706 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:12.673033  3197 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:12.789047 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 136203us
I20260709 12:22:12.689651  3193 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:12.790997 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:12.794000 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:12.683593  3172 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:12.686801  3267 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:12.800711 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.801424 31825 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:12.788046  3220 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:12.737133  3216 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:12.819862 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:12.735505  3056 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:12.805424 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:12.679581  3238 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:12.807802 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 184381us
I20260709 12:22:12.809340 31823 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:22:12.798457  3205 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:12.801826 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:12.828519 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 166457us
I20260709 12:22:12.806988  3221 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:12.811312 31824 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:13.006603  3221 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:12.879839 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:22:13.007655 31752 raft_consensus.cc:606] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:13.007720  3221 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.820881  3172 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:12.885969 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:13.009027  3221 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:12.846158  3209 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:12.843792  3215 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:12.908012  3276 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.211s	user 0.001s	sys 0.000s
I20260709 12:22:12.820307  3220 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:12.812378  3233 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:12.809196  3197 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:12.836321  3216 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:12.952370 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 330411us
W20260709 12:22:13.037212 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 450341us
W20260709 12:22:12.963368 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:12.965600 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:12.985755  3285 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:12.986195  3164 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:12.988487 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:12.989187  3076 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:12.990281  3259 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:12.992113 31675 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:22:12.947629  2706 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:12.999258  3279 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.202s	user 0.000s	sys 0.007s
I20260709 12:22:13.003782  3056 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:12.842559 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:13.056605 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 214126us
I20260709 12:22:13.057823 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:13.010372 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 51727.7us
I20260709 12:22:13.060613 31670 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:13.026250 32371 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:12.958426  3193 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:13.043289 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 77781.8us
I20260709 12:22:13.064433 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:13.022260 31672 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
W20260709 12:22:13.051357 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 257457us
W20260709 12:22:13.037061  3273 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.298s	user 0.001s	sys 0.003s
I20260709 12:22:13.069715 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:13.072727 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:13.074555 31670 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260709 12:22:13.047093  3165 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.042024  3164 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:13.046140  3073 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:13.087772 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:13.089814 31824 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:13.096647 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:13.011090  3205 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.101456 31825 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:13.051261  3287 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:13.067245  3289 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:13.071344  3285 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:13.067507  3195 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.062682  3274 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:13.112231 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:13.113281 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.064328  3259 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.114411 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:13.111752  3289 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:13.118871  3233 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:13.119190  3287 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:13.120820  3076 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:13.071691  3193 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.148319  3193 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:13.048905  3199 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.152213  3199 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:13.152982 31673 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:22:13.125756  2870 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.102758  3060 raft_consensus.cc:2764] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:22:13.157042  2870 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:13.159938  3209 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:13.122500  3194 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.615s	user 0.008s	sys 0.000s
I20260709 12:22:13.166007  3259 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:13.133150 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:13.136039 31821 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
I20260709 12:22:13.113770  3267 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:13.141402 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.142510  3205 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:13.136991  3195 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.136798  3289 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:13.114444  3056 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.093209  3298 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:13.092803  3286 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
W20260709 12:22:13.204325  3162 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.722s	user 0.003s	sys 0.010s
I20260709 12:22:13.205626 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:13.192485  3211 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.201615  3199 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:22:13.166546  3058 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.538s	user 0.002s	sys 0.007s
I20260709 12:22:13.201117 32371 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:13.179761  3274 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.215207 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:13.118966  3159 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:22:13.191970  3086 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.018s	sys 0.045s
I20260709 12:22:13.224795 31753 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260709 12:22:13.174376  3278 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.146s	user 0.000s	sys 0.000s
I20260709 12:22:13.204423  3079 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.105202  3165 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.256930 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:13.217531  3298 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:13.305001 31745 raft_consensus.cc:2376] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11. Re-sending same reply.
I20260709 12:22:13.137118  3220 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.300266  3281 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:13.309280  3192 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.505s	user 0.011s	sys 0.000s
I20260709 12:22:13.257059  3286 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:13.248450  3162 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:13.324867 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 239016us
I20260709 12:22:13.251658  3211 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.287616  3058 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.209426  3164 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:13.334331 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 239313us
I20260709 12:22:13.325461  3238 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:13.331990 32371 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
W20260709 12:22:13.339514 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 251811us
W20260709 12:22:13.335286  3176 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.634s	user 0.010s	sys 0.001s
I20260709 12:22:13.348229 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:13.307320  3079 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:13.349438 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 311522us
I20260709 12:22:13.351876 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:13.328837  3298 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:13.357606 31745 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:13.355118 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:13.362524  3291 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:13.365479 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:13.355916 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:13.368690 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.005s	sys 0.009s
W20260709 12:22:13.377533 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.006s	sys 0.010s
I20260709 12:22:13.375221 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:13.373626 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:13.379238 31825 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:13.381861 32371 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260709 12:22:13.382275 32371 pending_rounds.cc:85] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 8
I20260709 12:22:13.382519 32371 pending_rounds.cc:107] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 12:22:13.382426 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:13.382903 32371 raft_consensus.cc:2889] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:13.386061 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:13.386974 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:13.336757  3192 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.392422 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:13.363214  3291 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:13.396046 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:13.353943  3299 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:22:13.399919 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.354020  3233 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.363786  3199 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:13.367158  3164 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:13.351051  3086 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.046s
I20260709 12:22:13.359628  3238 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260709 12:22:13.389796  3176 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:13.416997  3176 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:13.417430  3176 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:13.403119  3299 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:13.418771  3299 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:13.419118 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:13.419829 31819 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:13.423062  3162 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:13.372650  3159 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 8.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.040s
I20260709 12:22:13.423985  3159 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:13.424408  3159 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:13.424090  3291 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.408028 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:13.415664 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:13.430163 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:13.422760  3286 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.398687  3177 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:13.413209 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:13.437999 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2182) took 1194 ms. Trace:
I20260709 12:22:13.346995  3281 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.447072 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:13.440914 31744 rpcz_store.cc:276] 0709 12:22:12.234584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:12.240396 (+  5812us) service_pool.cc:224] Handling call
0709 12:22:13.428475 (+1188079us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b54001ed2d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:13.428608 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1034140032}
I20260709 12:22:13.424322  3274 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:13.400781  3295 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:13.408277  3194 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:13.456936 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:13.402889  3288 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:13.425869  3296 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:22:13.329212  3287 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.432721  2910 raft_consensus.cc:2764] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260709 12:22:13.429698  3086 consensus_queue.cc:1243] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:22:13.443845  3162 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:13.491175 31824 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:13.441845  3199 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:13.447535  3267 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:13.469419  3295 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:13.443651  3283 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 12:22:13.502835 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:13.410379  3233 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.516422 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:13.458449  3221 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.560735 31827 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:13.592252 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:13.489749  3177 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:13.595856 31669 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:13.502524 31825 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:13.472616  3274 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.125s
I20260709 12:22:13.678231 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:13.658798 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 97350.9us
I20260709 12:22:13.685492 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:13.687268 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 257160us
I20260709 12:22:13.706528 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:13.691773 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:13.510529  3194 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:13.669883  3172 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.660s	user 0.011s	sys 0.001s
W20260709 12:22:13.676055  3310 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.217s	user 0.001s	sys 0.004s
I20260709 12:22:13.590512  3162 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:13.675154  3223 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.979s	user 0.012s	sys 0.000s
I20260709 12:22:13.728302 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:13.728336 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.745850 31827 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:13.746364 31827 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:13.746582 31825 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:13.631677  3177 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:13.670761  3216 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.612s	user 0.010s	sys 0.001s
W20260709 12:22:13.739554 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56418.1us
I20260709 12:22:13.651389  3281 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:13.560094  3193 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.189s
I20260709 12:22:13.668908  3079 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:13.651279  2910 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:13.591073  3295 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:13.677381  3211 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:13.651875  3165 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:13.679402  3058 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:13.757982 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 391821us
I20260709 12:22:13.753378  3177 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:13.762917 31827 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:13.744776  3188 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.610s	user 0.042s	sys 0.101s
W20260709 12:22:13.779188 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.691247  3283 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.782248  3046 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:13.726157  2706 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:13.807214 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:13.659524  3305 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.106s
I20260709 12:22:13.688536  3221 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.744854  3321 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:13.750777  3311 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.659046  3304 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.107s
I20260709 12:22:13.820868 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:13.748889  3194 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.822680 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:13.745149  3086 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.066s
W20260709 12:22:13.826100 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 134417us
I20260709 12:22:13.745136  3172 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.678061  3199 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.739383  3176 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.832598 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:13.819695  3188 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.686s	user 0.042s	sys 0.101s
I20260709 12:22:13.827814 31827 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:13.822172  3211 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:13.834347  3073 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:13.753245  3223 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.758471  3216 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.837278 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 740) took 1173 ms. Trace:
I20260709 12:22:13.823520  3046 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:13.843652 31671 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:13.846515 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 56467.9us
I20260709 12:22:13.831490  3267 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:13.851783 31676 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 12:22:13.850263 32377 rpcz_store.cc:276] 0709 12:22:12.663528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:12.663671 (+   143us) service_pool.cc:224] Handling call
0709 12:22:12.686050 (+ 22379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:13.067694 (+381644us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:13.067723 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:13.099609 (+ 31886us) log.cc:844] Serialized 26 byte log entry
0709 12:22:13.324491 (+224882us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:13.324526 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.324592 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:13.837020 (+512428us) raft_consensus.cc:1685] finished
0709 12:22:13.837042 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:13.837212 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":48693,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":48824,"thread_start_us":550009,"threads_started":2,"wal-append.queue_time_us":252033}
I20260709 12:22:13.825640  3283 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.888316 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2214) took 1302 ms. Trace:
I20260709 12:22:13.889295 31677 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:13.836926  3209 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.018s
I20260709 12:22:13.824527  2910 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260709 12:22:13.849675  3058 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:13.936862  3058 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:13.936654  3299 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.937752  3058 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.939015  3299 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:13.900938  3267 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.944512  3194 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:13.941612  3295 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.930426 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:13.947013  3330 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.107s	user 0.000s	sys 0.000s
I20260709 12:22:13.845592  3162 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:13.918300  3285 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.764s	user 0.011s	sys 0.000s
I20260709 12:22:13.948658  3285 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:13.949540 32371 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:22:13.951784  3238 raft_consensus.cc:993] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:22:13.899446 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:13.952401  3238 raft_consensus.cc:1081] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Signalling peer 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:22:13.953547 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:13.837119  3311 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:13.955504  3275 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.277s	user 0.003s	sys 0.002s
I20260709 12:22:13.955780  3283 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:13.902094  3318 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:22:13.956301  3311 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:13.849292  2706 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:13.957628  2706 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:13.959087  3285 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:13.959640  3285 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:13.960538 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:13.960745  3015 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:13.900310  3221 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:13.847653  3308 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:13.964617  3302 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:13.903026 31673 rpcz_store.cc:276] 0709 12:22:12.585481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:12.658094 (+ 72613us) service_pool.cc:224] Handling call
0709 12:22:12.680492 (+ 22398us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:13.153440 (+472948us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.888006 (+734566us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b54002549d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:13.888245 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1085990784}
I20260709 12:22:13.975418  3302 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.849781  3323 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:13.977246  3323 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:13.977943  3302 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:13.904659  3307 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20260709 12:22:13.906446  3325 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:22:13.978325  3323 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:13.979468 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:13.909507  3223 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:13.910488  3197 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.863s	user 0.010s	sys 0.002s
I20260709 12:22:13.912823  3216 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:13.914842 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:13.929298 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:13.931229 31747 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:13.931437  3289 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.716s	user 0.006s	sys 0.005s
I20260709 12:22:13.854486  3321 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:13.931885 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:13.933176  3294 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:13.886865  3305 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:13.934523 31746 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:14.002915 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:13.934466  3046 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:13.965171 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:13.965746 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:14.007226 31678 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:22:13.966085 31669 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:22:13.962182  3015 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:14.010000 31673 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:14.010454  3015 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:14.010983  3015 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:13.902094 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45556.8us
I20260709 12:22:14.009980 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.015202 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:14.015357 31747 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:14.016271 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.016885 31747 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:14.017841 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.018756  3086 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:14.019186  3086 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:14.019574  3086 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:14.020671 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:13.983472 31670 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:14.023739  3046 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:14.025454  3223 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:14.025939  3223 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:14.026319  3223 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:13.984012 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 243494us
I20260709 12:22:13.985954 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:14.028967 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.029445 31826 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:13.988806 31827 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 9 index: 5. (term mismatch)
I20260709 12:22:14.030148 31827 pending_rounds.cc:85] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 4
I20260709 12:22:13.993119 31751 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:14.030409 31827 pending_rounds.cc:107] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 10.5
I20260709 12:22:14.030706 31827 raft_consensus.cc:2889] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:13.995208 32377 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:13.996721 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:13.930522  3298 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.558s	user 0.004s	sys 0.005s
I20260709 12:22:14.000165 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:13.960578  3285 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.052830 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.009462 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:13.993696  3267 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:13.995246  3058 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:14.078895 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.079504 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.083376 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:14.079437 31818 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 9 index: 7. (index mismatch)
I20260709 12:22:14.084007 31819 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:22:14.085130 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:14.081985 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:14.078924 31827 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:14.087917 31678 raft_consensus.cc:2410] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:22:14.086784 31827 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:14.088861 31821 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:22:14.087741 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:14.089437 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:14.089975 32372 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:14.053617 31791 connection.cc:256] RPC call timeout handler was delayed by -1.4181513000003179s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2048s for a response
W20260709 12:22:14.092278 31791 connection.cc:256] RPC call timeout handler was delayed by -1.3209513000001607s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2156s for a response
W20260709 12:22:14.092629 31791 connection.cc:256] RPC call timeout handler was delayed by -1.2930513000001156s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2187s for a response
W20260709 12:22:14.092971 31791 connection.cc:256] RPC call timeout handler was delayed by -1.244451300000037s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22410000000000002s for a response
I20260709 12:22:14.094786 31671 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 11: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:22:14.086882 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 163770us
I20260709 12:22:13.978519  3159 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.097386 31565 maintenance_manager.cc:419] P ac4fa664fb214bda87cd96abb2116a4b: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033379
I20260709 12:22:14.098037  3159 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:14.099349  3015 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:14.102527 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.105074 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.107111 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:14.109973 31745 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:14.117094 31826 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:14.117938 31824 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: [8.4-8.4]
I20260709 12:22:14.118228 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:13.996642  3197 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.020714  3203 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.083s
I20260709 12:22:14.088533  3305 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.130061 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.132232 31826 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: [7.5-9.6]
I20260709 12:22:13.999785  3289 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.119745  3295 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:14.138036 31827 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.139159 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:14.139966 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.147090 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.136308 32372 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:14.153756 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 6 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:14.156162 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:14.060930  3298 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.162467 31747 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:14.123674  3311 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:14.163573 31827 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:14.141752  3267 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.174364 31671 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:14.179524 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:14.180449 31826 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:14.185213 31747 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11. Ops: [8.5-10.6]
W20260709 12:22:14.176208  3282 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.176s	user 0.000s	sys 0.017s
I20260709 12:22:14.235260 31747 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260709 12:22:14.170480  3320 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:14.249035 31827 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 12:22:14.259672  3344 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 12:22:14.260270  3320 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:14.263019  3331 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:22:14.264034  3197 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:14.235102  3282 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.235s	user 0.000s	sys 0.022s
W20260709 12:22:14.237231 31718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.627s (SENT) [suppressed 4 similar messages]
W20260709 12:22:14.239145  3333 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.137s	user 0.000s	sys 0.001s
I20260709 12:22:14.182890  3267 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.282203 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:14.283892 31671 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:22:14.284502 32378 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 587) took 2095 ms (client timeout 1627 ms). Trace:
W20260709 12:22:14.284969 32378 rpcz_store.cc:269] 0709 12:22:12.189355 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:12.189504 (+   149us) service_pool.cc:224] Handling call
0709 12:22:14.280796 (+2091292us) spinlock_profiling.cc:243] Waited 304 ms on lock 0x7b54006c0554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:14.284427 (+  3631us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":698686720}
I20260709 12:22:14.274686  3199 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.276945  3299 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.295332 31826 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:14.298856  3289 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:14.299786  3248 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Stepping down as leader of term 5
I20260709 12:22:14.300266  3248 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:14.304729  3233 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:14.319162 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 212127us
I20260709 12:22:14.261420  3320 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.319583 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.320223 31747 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:14.320504 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:14.321069 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:14.322155 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:14.151481  3321 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:14.323048  3346 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:22:14.331166 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:14.149689  3305 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.301270  3248 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 5.2, Last appended by leader: 2, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.332878  3248 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 5 FOLLOWER]: Advancing to term 6
I20260709 12:22:14.333911  3305 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:14.334805  3340 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:14.335779 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.337209 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.337906 31825 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:14.348380  3267 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:14.335805  3340 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:14.350569  3340 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:14.351768  3340 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:14.357168 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:14.362057  3177 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.540s	user 0.004s	sys 0.004s
I20260709 12:22:14.363579  3015 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:14.367727  3298 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:14.318605  3199 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.370605  3194 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:14.371287  3199 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:14.372007  3311 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:14.373193  3311 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:14.372758 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:14.373705  3311 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:14.376367  3324 raft_consensus.cc:2764] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260709 12:22:14.376693  3336 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:14.377575  3323 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:14.378611  3323 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:14.379148  3323 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:14.326297  3299 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.383961  3299 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:14.390092  3311 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.390564  3233 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.391209  3233 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.392501  3283 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:14.392627  3282 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:14.393102  3283 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.393446  3282 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.396320  3282 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:14.397300  3311 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:14.397833  3340 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:14.393440  3233 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:14.399657 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.403225 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:14.406687 31819 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:14.437824 31747 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:14.442514  3336 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:14.443696  3336 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:14.436571  3177 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.448432 32375 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:14.380141  3323 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.457705 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:14.458046 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:14.458578 31819 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:14.458649 31749 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:14.354171  3223 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.462808  3086 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.465252  3086 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:14.467315  3323 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:14.474416 31818 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 12 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 12:22:14.447778  3177 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:14.492435 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.493348 31819 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:14.454375 31825 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:41204
I20260709 12:22:14.499303 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.440066  3324 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
W20260709 12:22:14.501359 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 264158us
I20260709 12:22:14.502308 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.504231 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.507007  3283 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:14.513764 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:14.518232 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:14.520922 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 113396us
I20260709 12:22:14.521056 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:14.523947  3164 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:14.524474  3164 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.524904  3164 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:14.527616 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.537243 31820 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:14.537237 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.528826 32374 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:14.558743 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:14.559535 31820 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:14.539808 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:14.540607 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:14.573789 32378 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:14.546550  3223 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:14.585429 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.519883  3324 consensus_queue.cc:1243] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 11.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:22:14.586344 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:14.555980 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:14.557276 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:14.582973 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.595206 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:14.604647 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:14.607522 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67800.1us
I20260709 12:22:14.628331 31820 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:14.614758 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 172738us
I20260709 12:22:14.542029  3046 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:14.621124 32378 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:14.621558 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:14.584321 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:14.607570  3353 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.094s	user 0.001s	sys 0.001s
I20260709 12:22:14.556587  3165 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.068s
I20260709 12:22:14.554935  3305 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20260709 12:22:14.631217 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:14.684553 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53403.4us
W20260709 12:22:14.633960 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49742.8us
I20260709 12:22:14.639120 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:14.690060 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:14.620456  3221 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:14.689843 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:14.692893 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:14.635401  3237 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.632s	user 0.010s	sys 0.026s
I20260709 12:22:14.627331  3176 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:14.638931  3285 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:14.624199  3233 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:14.637053  3046 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.692155  3320 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:22:14.698514  3237 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.695s	user 0.010s	sys 0.039s
W20260709 12:22:14.711354 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 125112us
I20260709 12:22:14.698217  3342 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:14.711560 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:14.683251  3086 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.064s
W20260709 12:22:14.713291 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:14.727849 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.713621 32372 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:14.698341  3058 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:14.637346  3223 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.069s
I20260709 12:22:14.707568  3177 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:14.724594 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:14.742376 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:14.703812  3046 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:14.730594 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:14.707398  3358 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:14.743822 31747 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:22:14.722098 31820 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:14.748778 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:14.748948 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:14.750612 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.752046 31749 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:14.714461  3299 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.023s
W20260709 12:22:14.754743 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.731140  3086 consensus_queue.cc:1243] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 9.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:14.735472  3159 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:14.710729  3233 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:14.756034 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:14.731756  3058 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.770162 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:14.771976 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.757231 31820 raft_consensus.cc:2410] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 9.
I20260709 12:22:14.758581 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:14.772724 31748 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:22:14.741688  3320 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:14.759883 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.756179  3176 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:14.730109  3285 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.027s
I20260709 12:22:14.765642  3223 consensus_queue.cc:1243] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:22:14.766149  3342 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:14.766037  3305 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260709 12:22:14.751755  3308 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:14.750437  3358 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:14.777355 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:14.783157 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.784945 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.785847 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:14.785936 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:14.786193 31671 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:14.770704  3199 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:14.757727  3159 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:14.787146 31825 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:14.751293  3177 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.775662  3058 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:14.791793 31825 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:14.792058 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:14.788232 31749 raft_consensus.cc:2368] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
I20260709 12:22:14.781168  3313 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:14.795099 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.796216 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:14.807821  3362 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:22:14.786767  3358 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:14.854573 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:14.780400  3287 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.020s
I20260709 12:22:14.869382 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.870067 31827 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
W20260709 12:22:14.873051  3306 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.410s	user 0.007s	sys 0.009s
I20260709 12:22:14.873246 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:14.874111 31827 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
I20260709 12:22:14.875603 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:14.876564  3294 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:14.876789 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.877010  3294 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:14.877499  3294 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:14.790459  3188 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:14.894948  3188 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:14.895948  3188 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 8.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.788481  3176 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:14.859925  3359 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:22:14.860405  3313 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.861095  3324 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:14.866864  3177 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:14.856474 31752 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:14.914743 31752 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:14.885847 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:14.878545 31747 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260709 12:22:14.916855 31747 pending_rounds.cc:85] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 8
I20260709 12:22:14.917200 31747 pending_rounds.cc:107] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 12:22:14.917508 31747 raft_consensus.cc:2889] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:14.891809  3342 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.892961  3320 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.913185  3367 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:22:14.894389  3363 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:22:14.921278  3367 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:14.921730  3367 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:14.791091  3366 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:14.923282 31826 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: [9.7-9.7]
I20260709 12:22:14.924516 31826 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:14.924845 31826 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:14.925585 31826 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.861944  3233 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:14.904206 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49738.5us
I20260709 12:22:14.903594 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:14.918454 31749 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:14.933434 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:14.934360  3311 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:14.936064 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:14.937096  3311 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:14.932925  3237 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:14.939514  3237 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.941632  3237 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:14.941682  3203 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 6.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:14.912709 31825 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.913372  3324 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:14.944528  3324 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:14.945430  3367 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.927476  2975 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:14.898654  3176 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.797621  3056 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:14.913628  3313 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:14.942830  3223 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:14.928323 32372 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:22:14.954813 31749 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:14.946482  2975 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:14.912304  3305 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260709 12:22:14.965101 31749 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:22:14.956539  3340 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.558s	user 0.012s	sys 0.001s
I20260709 12:22:14.948194  3056 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:14.911665  3159 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:14.952802  3223 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:14.971369  3177 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:14.976590 31819 raft_consensus.cc:2410] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:22:14.932749  3294 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.977964  3367 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:14.979725 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:14.997819  3177 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.000113  3177 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:14.943109  3358 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:14.975660  3056 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:14.994392 31749 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:14.970031  3203 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:15.011085 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:15.011855 31749 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:15.012769 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.013355 31749 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:15.010381  3370 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.014533  3370 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.016368  3164 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.016789  3370 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:15.018388 32374 raft_consensus.cc:2368] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12.
I20260709 12:22:15.003949  3159 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:15.020414  3294 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:15.020733  3223 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.020941  3056 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.022202  3223 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:15.023353  3056 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:15.004522  3203 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:15.027678  3203 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.029124  3203 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:14.974017  3305 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.035s
I20260709 12:22:15.043336  3358 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:15.059793 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.070964 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:15.058393 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 136071us
I20260709 12:22:15.038139  3164 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:15.007017  3199 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:15.044806  3358 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 12:22:15.098980 31749 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:15.101814 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.103482  3058 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.106161  3058 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:15.106916 31750 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 12:22:15.107234  3199 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.088294  3056 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:15.111109  3056 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.115410 31749 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
W20260709 12:22:15.137586 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:15.180397 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58455.2us
I20260709 12:22:15.187439 32372 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:15.189453 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:15.190339 31678 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:15.117273  3162 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:15.197212 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:15.199955 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:15.200588 31669 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:15.208297  3233 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:15.208526  3056 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:15.199347  3305 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:15.208940  3233 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.209625  3305 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.212802 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:15.214359  3233 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:15.217043  3323 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:15.217553  3323 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:15.218524 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:15.218649  3323 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.205009  3358 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:15.221416  3358 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:15.222399  3358 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.223476 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1672) took 1294 ms. Trace:
I20260709 12:22:15.223891 31745 rpcz_store.cc:276] 0709 12:22:13.929413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.929846 (+   433us) service_pool.cc:224] Handling call
0709 12:22:13.930157 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:14.110462 (+180305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.223209 (+1112747us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b54000b86d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:15.223411 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":413100800}
I20260709 12:22:14.944629 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:15.228296 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 593) took 1198 ms. Trace:
I20260709 12:22:15.228659  2975 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.228701 32375 rpcz_store.cc:276] 0709 12:22:14.030286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.030893 (+   607us) service_pool.cc:224] Handling call
0709 12:22:14.031163 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:14.448888 (+417725us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.228020 (+779132us) spinlock_profiling.cc:243] Waited 417 ms on lock 0x7b54004c4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:15.228228 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":959190272}
I20260709 12:22:15.179368  3046 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:15.321229 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.233448 31670 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:15.238055 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:15.376570 31673 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:15.314599 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.358078 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:15.240864 31669 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
W20260709 12:22:15.366621 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.196904  3371 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:15.204777  3373 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.109s	user 0.000s	sys 0.002s
I20260709 12:22:15.232647  2975 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:15.238996 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:15.380146 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:15.392143  3164 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:15.392761  3164 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:15.393733  3164 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.382498  3324 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.385222 31678 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260709 12:22:15.396385 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:15.396847  3324 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:15.313382  3305 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:15.313651  3323 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:15.205996  3342 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:15.399189  3342 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:15.316960  3374 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.120s	user 0.000s	sys 0.001s
I20260709 12:22:15.387846 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:15.400854 31670 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:15.401769  3342 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:15.401849 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:15.402551 31670 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:15.405308 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:15.406078 31824 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
W20260709 12:22:15.325954  3378 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.215s	user 0.004s	sys 0.000s
W20260709 12:22:15.313611  3334 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.048s	user 0.002s	sys 0.003s
I20260709 12:22:15.396989 31678 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:15.408161 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:15.354759  3194 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.410548 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:15.411791 31824 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:15.411953 32371 raft_consensus.cc:2368] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11.
I20260709 12:22:15.319862  3172 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:15.412343 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:15.317602  3195 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:15.417016 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:15.202701  3058 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.429634  3324 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:15.437186 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.421147 31670 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:15.411953 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:15.427398 31678 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:15.434988 31824 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:15.439380 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:15.440886  3195 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:15.443239  3195 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.446516  3195 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:15.206918  3162 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.449250 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.449330 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.450351  3305 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:15.450860 31752 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:15.451035  3305 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:15.452219  3305 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.453107  3162 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:15.453722 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 519) took 1940 ms. Trace:
I20260709 12:22:15.414116  3297 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:15.455644  3297 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:15.456143  3297 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:15.456153  3237 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:15.456755  3237 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:15.457739  3192 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:15.458586 31748 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:15.459007  3237 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:22:15.459352 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2196) took 2104 ms. Trace:
I20260709 12:22:15.459751 31753 rpcz_store.cc:276] 0709 12:22:13.355077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.355226 (+   149us) service_pool.cc:224] Handling call
0709 12:22:13.355626 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:13.463606 (+107980us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:14.037999 (+574393us) log.cc:844] Serialized 14 byte log entry
0709 12:22:14.238885 (+200886us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:14.251054 (+ 12169us) log.cc:844] Serialized 26 byte log entry
0709 12:22:14.251239 (+   185us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:14.251277 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:14.384216 (+132939us) spinlock_profiling.cc:243] Waited 60.9 ms on lock 0x7b18003ab760. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:22:15.459025 (+1074809us) spinlock_profiling.cc:243] Waited 29.1 ms on lock 0x7b54000b8bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:15.459078 (+    53us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:15.459121 (+    43us) raft_consensus.cc:1685] finished
0709 12:22:15.459136 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:15.459270 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34900,"prepare.queue_time_us":130441,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":79,"raft.queue_time_us":180317,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":35852,"spinlock_wait_cycles":206832896,"thread_start_us":628790,"threads_started":3,"wal-append.queue_time_us":201716,"wal-append.run_cpu_time_us":2023,"wal-append.run_wall_time_us":1065787}
I20260709 12:22:15.463337  3324 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.464239 31826 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:15.454144 31674 rpcz_store.cc:276] 0709 12:22:13.513570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:13.515773 (+  2203us) service_pool.cc:224] Handling call
0709 12:22:13.593149 (+ 77376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:13.593404 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:13.593436 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:13.632328 (+ 38892us) log.cc:844] Serialized 26 byte log entry
0709 12:22:13.790710 (+158382us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:13.790774 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:13.790875 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:15.312742 (+1521867us) spinlock_profiling.cc:243] Waited 3.4 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:15.453379 (+140637us) raft_consensus.cc:1685] finished
0709 12:22:15.453405 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:15.453654 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7826304,"thread_start_us":40283,"threads_started":1,"wal-append.queue_time_us":433914}
I20260709 12:22:15.465204 31748 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:15.465570  3324 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:15.466823 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 504) took 3147 ms. Trace:
I20260709 12:22:15.467223 31677 rpcz_store.cc:276] 0709 12:22:12.319055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:12.321258 (+  2203us) service_pool.cc:224] Handling call
0709 12:22:12.321569 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:13.903220 (+1581651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.466568 (+1563348us) spinlock_profiling.cc:243] Waited 1.57 s on lock 0x7b540022b854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:15.466742 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3605039488}
I20260709 12:22:15.439500  3058 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:15.447389 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.447824 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:15.448485 31669 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 12 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 12:22:15.236934  3313 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.477205 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:15.477193 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:15.484465 31670 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:15.485481 31673 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:15.486217 31753 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:15.488533 31826 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:22:15.477747 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:15.414958  3194 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.499171 31677 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:15.496486 31824 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:15.508510 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:15.509531  3223 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:15.500519 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:15.503865 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.504809 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:15.388139  3371 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.511509 31753 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:15.511759  3046 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:15.513068  3371 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:15.515007 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:15.509934  3223 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:15.515223  3377 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:22:15.516886 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:15.521447 31752 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:15.517557 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:15.522126 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:15.517501 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:15.506899  3194 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:15.519544 31748 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:15.522008 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.522959 31826 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:22:15.523133 31745 raft_consensus.cc:2410] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 12:22:15.523274 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.522358 31674 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:22:15.527673 31753 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 11: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:15.514038  3297 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.549665 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:15.495290  3159 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:15.530234 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.552270 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.520514  3203 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.530582 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.531603  3058 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:15.559221 32375 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:22:15.531824 31674 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 5. Current term is 6. Ops: []
I20260709 12:22:15.560091 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 6 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.518270  3223 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:15.560894 31749 raft_consensus.cc:2410] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 10 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 6.
I20260709 12:22:15.535359  3313 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:15.550518  3246 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:15.556963 31718 leader_election.cc:400] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12.
I20260709 12:22:15.572484 31718 leader_election.cc:403] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:22:15.520239 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:15.573210 32374 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:15.562204 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:15.574246 31752 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:15.563735 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.573342 31824 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:15.561012 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.576292 31753 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:15.558370  3323 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.854856 31745 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
W20260709 12:22:15.824098 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 260106us
I20260709 12:22:15.817502 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2244) took 1427 ms. Trace:
W20260709 12:22:15.678833 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.686837 31750 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 8.
W20260709 12:22:15.725080 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 333653us
W20260709 12:22:15.649878 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.618938 31824 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:15.600669 31826 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:15.560185  3086 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:15.825268 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:15.867141 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:15.876830 31826 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
W20260709 12:22:15.876976 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 97064.8us
W20260709 12:22:15.878055 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:15.877959 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 225728us
I20260709 12:22:15.864552 31819 rpcz_store.cc:276] 0709 12:22:14.390376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.503643 (+113267us) service_pool.cc:224] Handling call
0709 12:22:15.817186 (+1313543us) spinlock_profiling.cc:243] Waited 471 ms on lock 0x7b54004475d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:15.817407 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1082999680}
I20260709 12:22:15.880420 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:15.885488 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:15.886310 31824 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:15.887027  3257 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:15.888890 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:15.886544 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:15.809270  3275 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:22:15.894995 31819 raft_consensus.cc:2410] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:22:15.562068  3159 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.875765  3086 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:15.883648  3367 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.699609  3336 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:15.602314  3246 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.295s
I20260709 12:22:15.699525  3324 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:15.928565  3317 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 2.459s	user 0.004s	sys 0.009s
I20260709 12:22:15.928251  3324 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.914053  3076 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.929873  3076 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.678583  3194 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.930778  3336 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:15.930994  3194 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.932991  3076 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:15.933421  3194 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:15.897454 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:15.935046 31826 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:15.923756  3367 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.897317  3237 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:15.939010  3237 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.940029  3159 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:15.942090  3237 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:15.901114 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:15.946359 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45323.5us
I20260709 12:22:15.903028 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:15.948709  3288 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:15.949301 31750 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:15.951191  3367 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:15.951834  3367 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.952316  3367 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.584024  3297 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:15.819605  3321 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:15.905997 31753 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:15.564503  3203 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.763695  3332 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.909909 31824 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:15.611230  3216 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:15.915071 31819 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 12:22:15.922751 31823 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
W20260709 12:22:15.926963 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.869321  3323 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:15.913759  3086 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:15.897899  3257 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:15.900401 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:15.933429  3336 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:15.904420  2870 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 12:22:15.961161  3288 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:15.979547 31819 pending_rounds.cc:85] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 5
I20260709 12:22:16.156864 31819 pending_rounds.cc:107] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:22:16.157451 31819 raft_consensus.cc:2889] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:15.980014 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2248) took 1191 ms. Trace:
I20260709 12:22:16.011730 31750 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
W20260709 12:22:16.024617 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:15.934465  3194 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.032263 31826 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:16.084837  3332 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.086042 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:16.158114 31823 rpcz_store.cc:276] 0709 12:22:14.788551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.910883 (+1122332us) service_pool.cc:224] Handling call
0709 12:22:15.922279 (+ 11396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:15.979674 (+ 57395us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:15.979938 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:16.164644 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2250) took 1375 ms. Trace:
I20260709 12:22:16.165264 31819 rpcz_store.cc:276] 0709 12:22:14.789309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.912185 (+1122876us) service_pool.cc:224] Handling call
0709 12:22:15.912536 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.164276 (+251740us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.164565 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:15.975663  3058 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12.
I20260709 12:22:15.975060  3321 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:16.168561 31823 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:16.200968 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2135) took 1301 ms. Trace:
I20260709 12:22:16.201539 31823 rpcz_store.cc:276] 0709 12:22:14.899057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.167860 (+1268803us) service_pool.cc:224] Handling call
0709 12:22:16.168160 (+   300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.200627 (+ 32467us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.200896 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:16.208923 31819 pending_rounds.cc:85] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 5
I20260709 12:22:16.208791 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 523) took 1109 ms. Trace:
I20260709 12:22:16.209427 31819 pending_rounds.cc:107] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:22:16.095611  3216 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.209818 31819 raft_consensus.cc:2889] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:16.210325 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.209542 31823 rpcz_store.cc:276] 0709 12:22:15.099133 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.202392 (+1103259us) service_pool.cc:224] Handling call
0709 12:22:16.208202 (+  5810us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.208405 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:16.208428 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:16.208474 (+    46us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:16.208509 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.208561 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.208709 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:16.215415 31823 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: [7.5-9.6]
I20260709 12:22:16.216116 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2138) took 1092 ms. Trace:
I20260709 12:22:16.216495 31823 rpcz_store.cc:276] 0709 12:22:15.123219 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.214813 (+1091594us) service_pool.cc:224] Handling call
0709 12:22:16.215144 (+   331us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:16.215856 (+   712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.216065 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:16.162775  3358 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.159950  3324 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:16.222189 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.167354  3159 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.211015  3367 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:16.163180  3194 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.280284 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.281253 31745 raft_consensus.cc:2376] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11. Re-sending same reply.
I20260709 12:22:16.280145 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.289085 31825 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:16.274058 32371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 597) took 1746 ms (client timeout 1976 ms). Trace:
I20260709 12:22:16.293910 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:16.299314  3393 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260709 12:22:16.301196  3394 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:22:16.309463 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2212) took 3727 ms. Trace:
I20260709 12:22:16.310009 31676 rpcz_store.cc:276] 0709 12:22:12.582531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:12.633545 (+ 51014us) service_pool.cc:224] Handling call
0709 12:22:12.653215 (+ 19670us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:13.859847 (+1206632us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.309104 (+2449257us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540036f1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:16.309362 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2755766144}
I20260709 12:22:16.318109 32372 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:22:16.323693 31746 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:16.325321  3207 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:16.325815  3207 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:16.327001 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2230) took 1017 ms. Trace:
I20260709 12:22:16.326921  3207 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:16.328387  3207 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:16.327577 31674 rpcz_store.cc:276] 0709 12:22:15.309651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.572364 (+262713us) service_pool.cc:224] Handling call
0709 12:22:15.603204 (+ 30840us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:15.603738 (+   534us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:15.603783 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:15.864095 (+260312us) log.cc:844] Serialized 52 byte log entry
0709 12:22:16.197217 (+333122us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:16.197281 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.197383 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:16.326713 (+129330us) raft_consensus.cc:1685] finished
0709 12:22:16.326742 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.326926 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3650,"raft.queue_time_us":488,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":31390,"thread_start_us":300609,"threads_started":1,"wal-append.queue_time_us":370470}
W20260709 12:22:16.331215 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.336063  3288 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:16.267086  3237 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.340129  3237 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.342453  3332 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:16.344192  3323 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:16.266784  3324 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.345396 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:16.345191  3324 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:16.345860 31671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1563) took 1713 ms (client timeout 2226 ms). Trace:
W20260709 12:22:16.346318 31671 rpcz_store.cc:269] 0709 12:22:14.632347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.632501 (+   154us) service_pool.cc:224] Handling call
0709 12:22:16.345667 (+1713166us) spinlock_profiling.cc:243] Waited 95.2 ms on lock 0x7b5400391cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:16.345797 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":219056384}
I20260709 12:22:16.267271  3321 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.348914  3321 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:16.349939  3073 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.266952  3246 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
I20260709 12:22:16.350569  3073 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.232766  3159 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.353701 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:16.353950  3215 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.232914  3358 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.318286  3292 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:22:16.356387 31672 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:16.357007  3076 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:16.354431  3215 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:16.357576  3076 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:16.358690  3076 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:16.360589  3397 consensus_queue.cc:1175] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 10 status { last_received { term: 6 index: 5 } last_committed_idx: 4 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: [7.5-9.6]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:16.362116 31674 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:16.285435  3367 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:16.363031  3367 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:16.349021 31818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2120) took 2246 ms (client timeout 1619 ms). Trace:
I20260709 12:22:16.365648 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:16.365736 31818 rpcz_store.cc:269] 0709 12:22:14.102446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.108485 (+  6039us) service_pool.cc:224] Handling call
0709 12:22:16.348782 (+2240297us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b54000fcb54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:16.348917 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":761521408}
I20260709 12:22:16.366781 32372 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:16.375509 31746 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260709 12:22:16.292275 32371 rpcz_store.cc:269] 0709 12:22:14.528073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.555663 (+ 27590us) service_pool.cc:224] Handling call
0709 12:22:16.273775 (+1718112us) spinlock_profiling.cc:243] Waited 824 ms on lock 0x7b5400427f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:16.273975 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1895446272}
I20260709 12:22:16.382669 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:41204 (request call id 2127) took 1980 ms. Trace:
I20260709 12:22:16.386586 32371 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260709 12:22:16.389339 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.035s	sys 0.058s
I20260709 12:22:16.389254 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:16.389746 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.504s	user 0.035s	sys 0.058s
I20260709 12:22:16.385262 31825 rpcz_store.cc:276] 0709 12:22:14.309038 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.338604 (+ 29566us) service_pool.cc:224] Handling call
0709 12:22:16.243657 (+1905053us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b54000fbc54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdcca3d 00007fe901d9ccd0 00007fe8fb321254 00007fe8fb3211e2 00007fe8fb321111 00007fe8fb32108c 00007fe8fb320342 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:16.289745 (+ 46088us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":672357760}
I20260709 12:22:16.388890 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:16.392382 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:16.392737 31671 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:22:16.393014 31820 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 6.
I20260709 12:22:16.393272 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:16.393900 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:16.394268 31820 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 12:22:16.394760 31672 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:16.394598 31820 pending_rounds.cc:85] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 6
I20260709 12:22:16.395721 31820 pending_rounds.cc:107] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 12:22:16.396771 31820 raft_consensus.cc:2889] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:16.397096 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:16.398188 31820 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:16.396102 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.399462 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:16.400213 32371 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:16.401121 31820 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:16.401432 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:16.402230 31818 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 12 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 12:22:16.401746 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:16.402585 31820 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:16.399900 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:16.404331 32371 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 11: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:16.405162 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:16.408982 31825 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:16.410035 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:16.387645  3205 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:16.387579  3164 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:16.421042 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:16.421106 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.424324 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.425576  2870 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.430371 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:16.419667  3372 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:16.433435  3205 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:16.436177  3367 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:16.436518 31671 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:16.436609  3203 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:16.438694 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:16.420658  3223 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.844s	user 0.010s	sys 0.005s
I20260709 12:22:16.440811 31674 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:16.441502 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.444067 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.455606 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:16.419502  3257 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:16.452181  3404 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:16.419968  3193 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.040s
I20260709 12:22:16.437489  3324 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:16.475168 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:16.475905  3404 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:16.477547 31672 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:16.481575 31642 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.028s	sys 0.042s
W20260709 12:22:16.481950 31642 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.029s	sys 0.042s
W20260709 12:22:16.482247 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 578867us
I20260709 12:22:16.483569 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:16.463389  3046 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:16.484786 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.490449 31745 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 12 because replica has last-logged OpId of term: 11 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 12:22:16.485769 31746 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:16.488942 31753 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:16.492313 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:16.493319 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:16.493898 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.494067 31746 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:16.495820 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.438375  3164 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:16.500393  3358 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:16.501278 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:16.505592 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.471117  3294 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:16.479007  3193 consensus_queue.cc:1243] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 6.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:22:16.504032  3205 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:16.495914  3324 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.506810 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:16.471309  3216 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:16.515501 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.516588 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.481348  3203 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:16.522805 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 215956us
I20260709 12:22:16.444919  2870 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:16.479548  3257 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:16.472050  3223 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:16.526420 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:16.516139 31746 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.528965 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:16.529212 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2137) took 1585 ms. Trace:
I20260709 12:22:16.480724  3215 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:16.511658  3367 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:16.535785 31746 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:16.537856 31718 leader_election.cc:400] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11.
I20260709 12:22:16.550930 31718 leader_election.cc:403] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
W20260709 12:22:16.539305 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 129340us
W20260709 12:22:16.544596 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.552304 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:16.553522  3216 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:16.548399 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:16.555501 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:16.557036 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:16.526922  3303 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:16.523027  3407 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:16.525607  3194 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:16.561681 32371 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.10-10.10]
I20260709 12:22:16.562951 31820 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:16.566066 31749 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:16.529466  3237 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:16.568253 31642 leader_election.cc:400] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
I20260709 12:22:16.595815 31642 leader_election.cc:403] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260709 12:22:16.596935 31642 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:16.571245  3058 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.572595 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2134) took 1673 ms. Trace:
I20260709 12:22:16.579305  3358 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.538319  3205 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:22:16.613320 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 116238us
I20260709 12:22:16.534260  3073 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:16.565267  3371 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:16.609570 31826 rpcz_store.cc:276] 0709 12:22:14.898823 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.167052 (+1268229us) service_pool.cc:224] Handling call
0709 12:22:16.167422 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.167673 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:16.402992 (+235319us) log.cc:844] Serialized 14 byte log entry
0709 12:22:16.567473 (+164481us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:16.567528 (+    55us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:16.567584 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.572340 (+  4756us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.572522 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":303456,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":399,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":120,"thread_start_us":357456,"threads_started":2}
I20260709 12:22:16.614398 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:16.527812  3324 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.608167 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.565426  3079 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11.
I20260709 12:22:16.531818  3203 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:16.633811 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:16.605489 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:16.623692 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:16.566782  3303 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:16.624298 31672 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:16.626950 31746 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:16.622321 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:16.548200  3246 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.069s
I20260709 12:22:16.628629 31750 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 11: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:16.629261 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:16.629993  3205 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:16.593513  3402 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:16.604081  3221 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.568050  3216 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:16.601106  3294 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:16.623327  3284 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.012s
I20260709 12:22:16.675470  3303 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:16.606993  3287 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11.
I20260709 12:22:16.617834  3404 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:16.610839  3370 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:16.553530  3367 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.613988  3298 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.026s
I20260709 12:22:16.628896  3073 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.011s
I20260709 12:22:16.632874  3237 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260709 12:22:16.594328  3372 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:16.645221 31821 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:22:16.646687 31671 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260709 12:22:16.668334 31820 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:16.669322 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:16.706830 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 528) took 1219 ms. Trace:
I20260709 12:22:16.707616 31678 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 12 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
W20260709 12:22:16.670907 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.707345 31823 rpcz_store.cc:276] 0709 12:22:15.487611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.222268 (+734657us) service_pool.cc:224] Handling call
0709 12:22:16.222609 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.222893 (+   284us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:16.465567 (+242674us) log.cc:844] Serialized 14 byte log entry
0709 12:22:16.700973 (+235406us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:16.701026 (+    53us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:16.701080 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.701147 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.706770 (+  5623us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15928,"prepare.queue_time_us":276896,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":36,"raft.queue_time_us":1028,"thread_start_us":424943,"threads_started":2}
I20260709 12:22:16.569056  3407 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:16.675917 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:16.635044  3371 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:16.646292  3291 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:16.718328 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96114.1us
I20260709 12:22:16.617425  3358 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.619248  3058 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:16.665237  3046 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:16.678128  3400 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:16.565536  3215 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:22:16.686866 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:16.742799 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:16.680982 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 199988us
I20260709 12:22:16.647385  3194 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:22:16.702543 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:16.745061 31674 raft_consensus.cc:2376] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12. Re-sending same reply.
I20260709 12:22:16.706830 31672 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:22:16.747094 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:16.747838 31672 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:16.642447  3257 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:16.715392 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:16.724189 31671 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:16.712585  3407 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:16.726758 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:16.684290  3221 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.681850  3402 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.746047 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:16.725333 31677 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:16.734139  3336 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.037s
I20260709 12:22:16.752384 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:16.695015  3417 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 12:22:16.722383  3164 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:16.754108 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.754173 31826 raft_consensus.cc:2410] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 12 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:22:16.693576  3416 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260709 12:22:16.755894 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:16.735958  3323 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 12:22:16.731223  3205 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:16.751142  3284 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:16.829708 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:16.997016 31678 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:16.997602  3284 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:16.998096  3284 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:16.816365 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:16.816890 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:16.801734 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:16.881333 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 160375us
I20260709 12:22:17.001436  3193 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:16.817368 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:17.001816  3193 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:16.742412  3215 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.002878  3215 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:17.002799  3193 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.003932  3193 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:17.005163 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:16.949415 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.006007  3233 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:17.006433  3233 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:17.006814  3233 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:17.007040 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.008275 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.009017 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:16.951757 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 520) took 1433 ms. Trace:
I20260709 12:22:17.009838 31678 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:17.011027 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.011689 31678 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:17.012632 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.013873 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.015295 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.015549  3233 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:16.830474  3405 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.222s	user 0.000s	sys 0.001s
I20260709 12:22:17.017616 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2133) took 2119 ms. Trace:
I20260709 12:22:17.018339  3233 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:17.016037 31678 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:17.017974 31824 rpcz_store.cc:276] 0709 12:22:14.898533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.976211 (+1077678us) service_pool.cc:224] Handling call
0709 12:22:15.976699 (+   488us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:16.083455 (+106756us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:16.083496 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:16.084079 (+   583us) log.cc:844] Serialized 26 byte log entry
0709 12:22:16.443931 (+359852us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:16.443971 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.444035 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:17.017421 (+573386us) raft_consensus.cc:1685] finished
0709 12:22:17.017442 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.017559 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1283,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":89802,"thread_start_us":359404,"threads_started":1,"wal-append.queue_time_us":447839}
I20260709 12:22:17.019867 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.022950 31678 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:17.022845 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:17.023828 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:17.021358 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 528) took 1137 ms. Trace:
I20260709 12:22:17.026263 31678 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:17.026259 31749 rpcz_store.cc:276] 0709 12:22:15.884279 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.884479 (+   200us) service_pool.cc:224] Handling call
0709 12:22:15.897055 (+ 12576us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.594027 (+696972us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.021120 (+427093us) spinlock_profiling.cc:243] Waited 669 ms on lock 0x7b54000f3cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:17.021297 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1538065792}
I20260709 12:22:17.021476  3188 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20260709 12:22:17.028398  3207 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.699s	user 0.015s	sys 0.000s
I20260709 12:22:16.878762  3297 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:17.028787  3324 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:17.029067  3297 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:17.030140  3297 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.031992  3297 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:16.877326 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:17.036159 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 280356us
I20260709 12:22:17.036527 32376 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:17.036971 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:16.882515 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.767294  3372 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:16.813277  3216 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.025651 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:16.801445 32371 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:17.039752  3303 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:16.869684 31643 connection.cc:256] RPC call timeout handler was delayed by -1.2551469620002536s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1736s for a response
I20260709 12:22:17.040395 31642 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:17.041077 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:17.041196  3015 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.042135  3303 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:17.041891  3015 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.040059 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:17.043059  3431 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:17.044085 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:17.044255  3015 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:17.044690 31750 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:16.974516  3432 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:16.769956  3203 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:17.046052  3323 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:16.781988 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:16.946264 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 526) took 1763 ms. Trace:
I20260709 12:22:17.046612  3323 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:17.048166 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:16.945441 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2131) took 2245 ms. Trace:
W20260709 12:22:16.784060  3409 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.154s	user 0.000s	sys 0.001s
I20260709 12:22:17.048858 31827 rpcz_store.cc:276] 0709 12:22:14.699882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:14.921046 (+221164us) service_pool.cc:224] Handling call
0709 12:22:14.921417 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:14.931012 (+  9595us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:14.931042 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:14.931081 (+    39us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:14.931112 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.945094 (+2013982us) spinlock_profiling.cc:243] Waited 9.4 ms on lock 0x7b54001c93d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:16.945159 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.945355 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21625088}
I20260709 12:22:16.743330  3400 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.050741 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2136) took 2137 ms. Trace:
I20260709 12:22:17.050870  3400 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:16.795852 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:16.863195  3058 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:17.047923  3216 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:16.879280  3367 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:17.052843 31669 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 11: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:17.051131 31819 rpcz_store.cc:276] 0709 12:22:14.913581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.168118 (+1254537us) service_pool.cc:224] Handling call
0709 12:22:16.168549 (+   431us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:16.212773 (+ 44224us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:16.212804 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:16.213388 (+   584us) log.cc:844] Serialized 52 byte log entry
0709 12:22:16.404358 (+190970us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:16.404400 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.007057 (+602657us) spinlock_profiling.cc:243] Waited 40.1 ms on lock 0x7b54004470d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:17.007106 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:17.050420 (+ 43314us) raft_consensus.cc:1685] finished
0709 12:22:17.050443 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.050682 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":115,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"spinlock_wait_cycles":92230912,"thread_start_us":190610,"threads_started":1,"wal-append.queue_time_us":354751}
I20260709 12:22:16.770740  3371 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:17.052851  3367 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:17.054355  3367 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:17.054776  3367 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:17.055833  3402 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:16.534936 31822 rpcz_store.cc:276] 0709 12:22:14.943439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.198781 (+1255342us) service_pool.cc:224] Handling call
0709 12:22:16.201329 (+  2548us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.201584 (+   255us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:16.395600 (+194016us) log.cc:844] Serialized 14 byte log entry
0709 12:22:16.528820 (+133220us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:16.528885 (+    65us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:16.528943 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.529015 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.529154 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8555,"prepare.queue_time_us":294679,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":47,"raft.queue_time_us":1745,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":6692,"thread_start_us":295751,"threads_started":2}
I20260709 12:22:16.885792  3076 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:17.059046  3216 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:17.009709 31752 rpcz_store.cc:276] 0709 12:22:15.519667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.584445 (+ 64778us) service_pool.cc:224] Handling call
0709 12:22:16.950735 (+1366290us) spinlock_profiling.cc:243] Waited 37.8 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:22:16.951032 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.951296 (+   264us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:16.951335 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:16.951396 (+    61us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:16.951449 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.951527 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.951679 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":86989184}
I20260709 12:22:16.801314  3046 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.061874 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:17.062537 31752 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:17.063782 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:16.788352 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.047492 31675 rpcz_store.cc:276] 0709 12:22:15.182872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.188306 (+  5434us) service_pool.cc:224] Handling call
0709 12:22:15.211720 (+ 23414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:15.511848 (+300128us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:15.512161 (+   313us) log.cc:844] Serialized 14 byte log entry
0709 12:22:15.860604 (+348443us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:15.860684 (+    80us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:15.860754 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:16.945983 (+1085229us) spinlock_profiling.cc:243] Waited 300 ms on lock 0x7b54002e7a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:16.946047 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:16.946194 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":596,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":126,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":117,"spinlock_wait_cycles":689759488,"thread_start_us":347968,"threads_started":1,"wal-append.queue_time_us":571374}
W20260709 12:22:17.065454 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.066398  3194 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.067116  3194 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:17.069705  3233 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:17.070181  3233 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.070524  3233 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:16.800900 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:16.818904 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:16.854175 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.076026 31670 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:16.829454  3285 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.224s
W20260709 12:22:17.052188 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 256390us
I20260709 12:22:17.095173 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:17.101895 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:16.717861  3398 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:17.051239  3323 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:17.169477 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.186854 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:16.748492  2870 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:16.976996  3086 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.819s	user 0.011s	sys 0.004s
I20260709 12:22:17.139115 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:17.182134  3267 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.540s	user 0.006s	sys 0.011s
W20260709 12:22:17.139412 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 66022.5us
I20260709 12:22:17.228464 31824 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:17.136646 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:17.231946 31821 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 11 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:22:17.187953 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:16.735630  3291 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.235991 31678 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
W20260709 12:22:17.236990 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 193842us
I20260709 12:22:17.267560 31643 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:16.803506  3159 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:16.771173  3401 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:17.309552 31673 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:16.771018  3426 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:22:17.243866 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 104870us
I20260709 12:22:17.239493 31822 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: [11.6-11.6]
W20260709 12:22:17.137280 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 100405us
I20260709 12:22:17.316305 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:17.253507 31671 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:22:17.317833 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:17.319082 31671 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:16.988217  3358 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:17.255244 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:16.753623  3223 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:17.182663  3321 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.833s	user 0.010s	sys 0.006s
W20260709 12:22:16.949103  3418 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:22:17.242719 31747 raft_consensus.cc:2376] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9. Re-sending same reply.
I20260709 12:22:17.113350  3303 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:17.269209 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:17.188347  3336 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:17.188881  3237 raft_consensus.cc:2764] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:22:17.182343  3324 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:17.342934  3324 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:17.342986 31669 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:17.343482  3324 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:17.341470  3336 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:17.345969  3336 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:17.133328  3216 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.347075  3216 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:17.349776  3216 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:17.280757 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.282063 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.283253 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.351145 31677 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:17.351753 31674 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:17.285522 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.268720  3203 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260709 12:22:17.287352  3207 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:17.287554  3292 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:22:17.290164 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.353510  3076 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:17.291450 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:17.357137 31826 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:22:17.357893  3221 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:17.241794  3432 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:17.264263  3194 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:17.259953  3086 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:17.361088  3432 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:17.355916  3215 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.297361  3291 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:17.301594  3233 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:17.367210  3215 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:17.367900  3415 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.668s	user 0.002s	sys 0.001s
I20260709 12:22:17.304306 31745 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:17.307132 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:17.372551 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:17.374125 31824 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 6 index: 3. (index mismatch)
I20260709 12:22:17.306856  3284 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:17.240530  3402 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.076s
I20260709 12:22:17.307852  3323 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:16.853341  3073 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.406s
I20260709 12:22:17.251432 31750 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:17.322171 31671 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:17.323881  3426 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:17.325011 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:17.326642 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:17.396260 31827 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:22:17.329797 31751 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 11 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260709 12:22:17.332744 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:17.332815 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.313714  3188 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:17.328033  2874 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.015s
I20260709 12:22:17.407135 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:17.407882 31747 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:17.409251  2874 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:17.376004  3058 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.284418 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.410441  3058 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:17.358327 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:17.328358  3407 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.544s	user 0.010s	sys 0.000s
I20260709 12:22:17.412693  3058 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:17.412545  3407 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.413314  3058 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:17.415792  3284 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:17.316876 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:17.418090 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 101288us
I20260709 12:22:17.378537  3424 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.419037  3424 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:17.419590 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:17.420667  3237 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:17.421255  3424 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:17.421376  3194 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:17.313834  3285 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.069s
I20260709 12:22:17.422008  3194 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:17.328264  3015 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.068s
I20260709 12:22:17.423648  3015 consensus_queue.cc:1243] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 10.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260709 12:22:17.425659  3086 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:17.327816  3199 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:17.426860  3199 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:17.368827  3233 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:17.401121 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:17.426400  3086 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:17.456561 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:17.428647 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 103753us
I20260709 12:22:17.325852  3267 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:17.436455 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.439625 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.479208 31670 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:17.440419 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:17.446444 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 177292us
I20260709 12:22:17.457386 31671 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:22:17.479347 31674 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:17.503645 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.392607  3321 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:17.472754 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:17.510766 31750 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:17.427300  3199 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:17.505501 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:17.505653 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:17.515668 31669 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
W20260709 12:22:17.517382 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.018s	sys 0.021s
I20260709 12:22:17.518146 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:17.518705 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.018s	sys 0.021s
W20260709 12:22:17.519025 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 711199us
I20260709 12:22:17.520639 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:17.455219  3015 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:17.473902  3076 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:17.475284  3267 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:17.506618 31676 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.834s	user 0.028s	sys 0.035s
I20260709 12:22:17.507961 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:17.539775 31676 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.867s	user 0.028s	sys 0.035s
I20260709 12:22:17.505981 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:17.473868 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.730s	user 0.024s	sys 0.032s
I20260709 12:22:17.455863  3086 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:17.540683 31671 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:17.422955  3194 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:17.512461  3429 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.001s	sys 0.028s
W20260709 12:22:17.563370 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.448226 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:17.588398 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 140263us
I20260709 12:22:17.514102  3412 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:17.590842 31669 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: [7.8-8.9]
I20260709 12:22:17.507009  3285 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 12:22:17.541188 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:17.615692 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 74587.5us
W20260709 12:22:17.550525 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.807s	user 0.024s	sys 0.032s
W20260709 12:22:17.616606 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 872939us
I20260709 12:22:17.551270 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 536) took 1201 ms. Trace:
W20260709 12:22:17.544430 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:17.550364 31675 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:22:17.517410  3284 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.070s
I20260709 12:22:17.549937 31750 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:17.619997 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:17.621663 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:17.623384 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:17.626859 31752 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.007s	sys 0.002s
I20260709 12:22:17.542920 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:17.627207 31752 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.007s	sys 0.002s
W20260709 12:22:17.627864 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 85593us
I20260709 12:22:17.554098  3297 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:17.592505  3429 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.002s	sys 0.029s
W20260709 12:22:17.606906  3320 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.586s	user 0.013s	sys 0.018s
I20260709 12:22:17.524662  3321 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:17.629207 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:17.622460 31750 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:17.661926 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:17.663376 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:17.664683 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:17.665685 31750 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 10 because replica has last-logged OpId of term: 8 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260709 12:22:17.615629 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:17.608568  2870 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:17.674019 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44936.7us
I20260709 12:22:17.675835 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:17.626420 31753 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.003s	sys 0.004s
W20260709 12:22:17.677994 31753 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.004s	sys 0.004s
I20260709 12:22:17.577553  3237 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.105s
I20260709 12:22:17.617697 31676 rpcz_store.cc:276] 0709 12:22:16.349423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.352350 (+  2927us) service_pool.cc:224] Handling call
0709 12:22:16.352665 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.352843 (+   178us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:16.535075 (+182232us) log.cc:844] Serialized 14 byte log entry
0709 12:22:17.550892 (+1015817us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:17.550950 (+    58us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:17.550995 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.551064 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.551204 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":335485,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":32291,"spinlock_wait_cycles":60326912,"thread_start_us":1197026,"threads_started":3,"wal-append.queue_time_us":182925}
W20260709 12:22:17.655548  3320 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.013s	sys 0.018s
I20260709 12:22:17.540722  3433 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:17.655439 32377 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.617s	user 0.000s	sys 0.010s
W20260709 12:22:17.735797 32377 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.000s	sys 0.011s
I20260709 12:22:17.602392  3015 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:17.590445  3407 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:17.541671  3267 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:17.671725 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 56174.4us
I20260709 12:22:17.673465  2874 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:17.666538 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:17.690379  3342 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260709 12:22:17.506675 31748 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.751s	user 0.035s	sys 0.041s
W20260709 12:22:17.767978 31748 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.013s	user 0.035s	sys 0.041s
I20260709 12:22:17.768868 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 519) took 2250 ms. Trace:
I20260709 12:22:17.674857 32369 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:17.651840 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:17.652066 31749 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.620s	user 0.009s	sys 0.001s
I20260709 12:22:17.650382  3303 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:17.587411  3233 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.073s
W20260709 12:22:17.651751  3431 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.008s	sys 0.000s
I20260709 12:22:17.650604  3358 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:17.652823  3216 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:17.684592  3400 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.626s	user 0.009s	sys 0.002s
I20260709 12:22:17.757215 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:17.612265 31747 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
W20260709 12:22:17.731473  3193 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.727s	user 0.008s	sys 0.004s
I20260709 12:22:17.764955 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:17.784225 31744 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.014s	sys 0.010s
W20260709 12:22:17.785023 31744 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.014s	sys 0.010s
I20260709 12:22:17.785089 31750 raft_consensus.cc:2376] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11. Re-sending same reply.
I20260709 12:22:17.785332 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:17.786103 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:17.657295  3046 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.001s	sys 0.010s
I20260709 12:22:17.657368  3215 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.095s
I20260709 12:22:17.603086  3412 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:17.776407 31749 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.010s	sys 0.001s
I20260709 12:22:17.657780  3297 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.769289 31748 rpcz_store.cc:276] 0709 12:22:15.518463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:15.571003 (+ 52540us) service_pool.cc:224] Handling call
0709 12:22:15.571463 (+   460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:15.571733 (+   270us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:15.655029 (+ 83296us) log.cc:844] Serialized 14 byte log entry
0709 12:22:16.637998 (+982969us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x7b180042f2e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:22:17.768396 (+1130398us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:17.768470 (+    74us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:17.768537 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.768637 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:17.768802 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16907,"prepare.queue_time_us":83093,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":59,"raft.queue_time_us":69792,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":1980,"spinlock_wait_cycles":30473984,"thread_start_us":1113043,"threads_started":2,"wal-append.queue_time_us":1333644}
W20260709 12:22:17.781364 31819 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.009s	sys 0.016s
I20260709 12:22:17.786755 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:17.826871 31819 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.641s	user 0.009s	sys 0.016s
I20260709 12:22:17.827093 31747 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:22:17.788800 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:17.828068 31750 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:17.738797  3434 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:17.830595  3434 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:17.831583  3367 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.776s	user 0.010s	sys 0.000s
I20260709 12:22:17.832062  3367 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.835376  3237 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:17.835987 31747 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:17.837568 31747 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 12:22:17.831598  3434 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 8.6, Last appended by leader: 6, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:17.780582  3431 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.010s	sys 0.000s
I20260709 12:22:17.847108 31747 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 9. Current term is 10. Ops: [9.5-9.5]
I20260709 12:22:17.790709 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:17.848089 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57490.7us
I20260709 12:22:17.792518  3073 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:17.849540  3073 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:17.850085  3073 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:17.737902  3076 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.048s
I20260709 12:22:17.852550  3076 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:17.688308  2870 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.855157  2870 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:17.809429 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:17.796869  3046 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.001s	sys 0.010s
I20260709 12:22:17.549839  3426 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.856963  3426 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:17.781550  3440 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.085s
I20260709 12:22:17.684154  3086 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.860612  3086 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:17.787966  2874 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:17.865566 31750 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:17.834358  3367 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:17.828969 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:17.800101  3358 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.836300  3237 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:17.790570 31818 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.556s	user 0.011s	sys 0.013s
W20260709 12:22:17.879902 31818 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.011s	sys 0.013s
I20260709 12:22:17.774272  3320 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:17.805433  3216 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:17.876984 31748 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:17.878058 31750 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:22:17.868463  3336 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.520s	user 0.012s	sys 0.000s
W20260709 12:22:17.877985  2975 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.004s	sys 0.023s
W20260709 12:22:17.891219  2975 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.004s	sys 0.023s
W20260709 12:22:17.891713 31825 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.003s	sys 0.024s
I20260709 12:22:17.891029  3336 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:17.892042 31825 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.003s	sys 0.024s
W20260709 12:22:17.877476  3401 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.001s	sys 0.026s
W20260709 12:22:17.893973  3401 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.585s	user 0.001s	sys 0.027s
I20260709 12:22:17.894145  3401 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:17.878027  3371 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.821s	user 0.003s	sys 0.025s
I20260709 12:22:17.797451  3400 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:17.878273  2874 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:17.913203 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 285944us
W20260709 12:22:17.900985  3402 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.028s	sys 0.038s
I20260709 12:22:17.925426 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:17.909278  3398 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.618s	user 0.012s	sys 0.016s
I20260709 12:22:17.904438  3216 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:17.901093  3336 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:17.944500  3193 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.006s
I20260709 12:22:17.977480  3193 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:17.978010  3193 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:17.979043  3193 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.8, Last appended by leader: 8, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:17.951412 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:17.980566 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1702) took 1552 ms. Trace:
I20260709 12:22:17.980995 31751 rpcz_store.cc:276] 0709 12:22:16.428102 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.428309 (+   207us) service_pool.cc:224] Handling call
0709 12:22:17.980332 (+1552023us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b54002959d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:17.980498 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1986228096}
W20260709 12:22:17.952476  3458 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.334s	user 0.002s	sys 0.000s
I20260709 12:22:17.957806 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:17.959718  3435 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.306s	user 0.001s	sys 0.000s
I20260709 12:22:17.961042 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:17.963493  3442 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
W20260709 12:22:18.044925 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 427689us
I20260709 12:22:17.907181  3437 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:17.907276  3367 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:17.913935  3371 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.853s	user 0.003s	sys 0.026s
I20260709 12:22:18.031315 32372 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
W20260709 12:22:18.006300 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.039103 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 78190us
I20260709 12:22:18.043946  3370 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:17.963871  3412 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:18.048522 32371 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:17.966284  3400 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:17.982154  3305 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.017s	sys 0.019s
I20260709 12:22:17.894930  3401 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:17.982857 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 759) took 1119 ms. Trace:
I20260709 12:22:18.106673 32369 rpcz_store.cc:276] 0709 12:22:16.863359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.863513 (+   154us) service_pool.cc:224] Handling call
0709 12:22:16.867649 (+  4136us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.770920 (+903271us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.982514 (+211594us) spinlock_profiling.cc:243] Waited 807 ms on lock 0x7b5400426dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:17.982779 (+   265us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1855869696}
I20260709 12:22:17.967730  3297 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:18.057984  3188 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.643s	user 0.012s	sys 0.001s
W20260709 12:22:18.105590  3305 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.018s	sys 0.020s
I20260709 12:22:18.059276 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:18.081701  3358 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:18.084559  3237 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:18.087998  3441 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.300s	user 0.002s	sys 0.000s
I20260709 12:22:18.089691  3424 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:17.983088  3193 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:18.090899  3292 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:18.091020  3216 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.098851  3370 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:18.099617 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:18.100346 31678 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
W20260709 12:22:18.078341  3323 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.686s	user 0.007s	sys 0.003s
I20260709 12:22:18.176522 31826 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
W20260709 12:22:18.176932 32341 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.017s	sys 0.006s
W20260709 12:22:18.177343 32341 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.017s	sys 0.006s
I20260709 12:22:18.178179 31826 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:18.004521  3321 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:22:18.105791  3324 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.762s	user 0.013s	sys 0.000s
W20260709 12:22:18.117898 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.033s	sys 0.018s
W20260709 12:22:18.119417 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60209.7us
I20260709 12:22:18.075629  3367 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:18.186616 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.033s	sys 0.018s
W20260709 12:22:18.187022 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 809566us
I20260709 12:22:18.187995 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:17.946352  3398 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.012s	sys 0.016s
I20260709 12:22:18.135648 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:18.138931 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:18.190551 31827 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:18.142680 31791 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.000s	sys 0.022s
W20260709 12:22:18.192610 31791 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.000s	sys 0.023s
W20260709 12:22:18.193251 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 730970us
I20260709 12:22:18.150952 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:17.968217  3073 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:18.195590 31821 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:18.199412 32371 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
W20260709 12:22:18.180616 31752 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.011s	sys 0.037s
W20260709 12:22:18.217531 31752 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.011s	sys 0.037s
I20260709 12:22:18.116991  3188 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:18.141696  3291 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:18.171579 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:18.215091 31822 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:22:18.189854 31823 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:17.969596  2874 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:18.179488 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.217541 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:18.219717 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 48286.9us
I20260709 12:22:18.153326  3424 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:18.228255 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 758) took 1373 ms. Trace:
W20260709 12:22:18.229343 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.173081  3292 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:18.236758 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:18.172183  3372 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.802s	user 0.010s	sys 0.019s
I20260709 12:22:18.174362  3432 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:22:18.289638 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 53001.2us
I20260709 12:22:18.219254  3058 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.047s
I20260709 12:22:18.219249  3291 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:18.225437  3324 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:18.264791 31827 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:18.267514  3367 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:18.267750 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:18.194196  3398 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:18.273743 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53893.3us
I20260709 12:22:18.273794 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2236) took 1236 ms. Trace:
W20260709 12:22:18.280053  3199 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.749s	user 0.008s	sys 0.004s
I20260709 12:22:18.283075  3370 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:18.284749  3450 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:18.223716  3426 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:18.310016 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:18.237730 32377 rpcz_store.cc:276] 0709 12:22:16.854983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:16.855144 (+   161us) service_pool.cc:224] Handling call
0709 12:22:16.855533 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:16.855819 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:17.037518 (+181699us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.227658 (+1190140us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:18.227737 (+    79us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:18.227815 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.227997 (+   182us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.228170 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":733991,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":64575,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":16887,"thread_start_us":1365383,"threads_started":3,"wal-append.queue_time_us":1049210}
I20260709 12:22:18.292309 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:18.310981 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:18.289933 31717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.012s	sys 0.015s
I20260709 12:22:18.229557  3412 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:18.195739  3267 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.242225  3440 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:18.242935  3371 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:18.293977  3372 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s	user 0.010s	sys 0.020s
I20260709 12:22:18.303192  3291 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:18.311635 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:18.303205  3454 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.312664 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:22:18.294451  3287 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.007s	sys 0.011s
I20260709 12:22:18.297111  3452 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.231117  3456 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:18.321718 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:18.359395 32370 raft_consensus.cc:2368] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13.
I20260709 12:22:18.329875 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:18.331296 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:18.325106  3412 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:18.333578 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:18.333767 31824 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:18.334111 31644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.010s	sys 0.015s
I20260709 12:22:18.297585  3323 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:18.224464  3283 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.083s
W20260709 12:22:18.347296  3015 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.023s	sys 0.004s
I20260709 12:22:18.304648 31752 rpcz_store.cc:276] 0709 12:22:17.036899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.064801 (+ 27902us) service_pool.cc:224] Handling call
0709 12:22:17.065148 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.065341 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:17.627588 (+562247us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.273399 (+645811us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:18.273460 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:18.273507 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.273576 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.273733 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55280,"prepare.queue_time_us":551497,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":55441,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"thread_start_us":1152028,"threads_started":2}
W20260709 12:22:18.340324 31819 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.028s	sys 0.043s
I20260709 12:22:18.294766  3432 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:18.342993 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.003s	sys 0.025s
W20260709 12:22:18.312420  3429 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.623s	user 0.009s	sys 0.017s
I20260709 12:22:18.316041  3237 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:18.340550  3465 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:18.348806 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:18.326090  3426 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:18.344400 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:18.360498 31826 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:18.333292 31821 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:18.321849  3367 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:18.312115 31717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.012s	sys 0.015s
I20260709 12:22:18.362571 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:18.363950 31644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.599s	user 0.010s	sys 0.015s
I20260709 12:22:18.366341 31823 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:18.351536  3257 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:18.339605 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:18.370270 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:18.356814  3287 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.009s	sys 0.013s
I20260709 12:22:18.316243  3205 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.047s
I20260709 12:22:18.317503  3288 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 12:22:18.306690  3199 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:18.346690 32369 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 8. Current term is 9. Ops: []
W20260709 12:22:18.372822 31819 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.028s	sys 0.043s
W20260709 12:22:18.374284  3429 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.009s	sys 0.018s
I20260709 12:22:18.459064 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:18.459817 31823 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:18.459921  3367 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:18.460443  3367 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:18.460681 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:18.461524 31823 raft_consensus.cc:2410] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:22:18.353399 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2256) took 1218 ms. Trace:
I20260709 12:22:18.463071 31818 rpcz_store.cc:276] 0709 12:22:17.135405 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.136295 (+   890us) service_pool.cc:224] Handling call
0709 12:22:17.234164 (+ 97869us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.234438 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:17.880471 (+646033us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.352947 (+472476us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:18.353023 (+    76us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:18.353084 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.353172 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.353324 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63079,"prepare.queue_time_us":940329,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":8512,"raft.queue_time_us":101,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":7613,"thread_start_us":1117567,"threads_started":2}
I20260709 12:22:18.464432  3199 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:18.464979 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:18.466847  3463 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
W20260709 12:22:18.467300 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 786243us
I20260709 12:22:18.468080 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:18.468140 32374 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:18.468266 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2241) took 1388 ms. Trace:
W20260709 12:22:18.468998 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 124694us
I20260709 12:22:18.469075 31746 rpcz_store.cc:276] 0709 12:22:17.080240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.372245 (+292005us) service_pool.cc:224] Handling call
0709 12:22:17.372596 (+   351us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:17.428008 (+ 55412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.428039 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:17.428623 (+   584us) log.cc:844] Serialized 52 byte log entry
0709 12:22:17.777213 (+348590us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:17.777270 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.777369 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:18.468045 (+690676us) raft_consensus.cc:1685] finished
0709 12:22:18.468065 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.468198 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54998,"raft.queue_time_us":72261,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":48957,"thread_start_us":246294,"threads_started":1,"wal-append.queue_time_us":630956}
I20260709 12:22:18.470871  3257 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:18.471297  3257 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:18.352777  3466 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.139s
I20260709 12:22:18.455648 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:18.474889  3432 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:18.475772  3398 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.476344  3058 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:18.466310 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:18.477862  3287 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.462249 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:18.478394  3287 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:18.479018 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:18.479497 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:18.477892  3159 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:18.480335 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2252) took 1346 ms. Trace:
I20260709 12:22:18.317284  3450 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.480782 31819 rpcz_store.cc:276] 0709 12:22:17.134265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.185398 (+ 51133us) service_pool.cc:224] Handling call
0709 12:22:17.185815 (+   417us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:17.827488 (+641673us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.827537 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:17.830229 (+  2692us) log.cc:844] Serialized 26 byte log entry
0709 12:22:18.474074 (+643845us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:18.474114 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.474181 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:18.480128 (+  5947us) raft_consensus.cc:1685] finished
0709 12:22:18.480151 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.480275 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":122898,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":13004,"thread_start_us":1284378,"threads_started":2,"wal-append.queue_time_us":648046}
I20260709 12:22:18.482928 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2255) took 1347 ms. Trace:
I20260709 12:22:18.438126  3323 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:18.484398 31827 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:18.485169  3323 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.483311 31825 rpcz_store.cc:276] 0709 12:22:17.135126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.136216 (+  1090us) service_pool.cc:224] Handling call
0709 12:22:17.234211 (+ 97995us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:17.892473 (+658262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.892505 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:17.893079 (+   574us) log.cc:844] Serialized 52 byte log entry
0709 12:22:18.358901 (+465822us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:18.361012 (+  2111us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.361129 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:18.482693 (+121564us) raft_consensus.cc:1685] finished
0709 12:22:18.482716 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.482870 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":90720,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":42035,"thread_start_us":1123196,"threads_started":2,"wal-append.queue_time_us":587198}
I20260709 12:22:18.362442  3358 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:18.456373 31718 leader_election.cc:400] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13.
W20260709 12:22:18.455776 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.476301  3283 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:18.342152  3371 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:18.477836 31818 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
W20260709 12:22:18.504123 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 47955.5us
I20260709 12:22:18.504485 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:18.357345  3086 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:18.505271 31744 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.720s	user 0.019s	sys 0.053s
W20260709 12:22:18.563268 31744 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.778s	user 0.019s	sys 0.054s
W20260709 12:22:18.506958 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 158221us
I20260709 12:22:18.508906 31823 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:22:18.564147 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:18.373246 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.003s	sys 0.025s
I20260709 12:22:18.374020 32370 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:18.342212  3292 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:18.518968  3257 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.513048  3471 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.520601  3267 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:18.589555 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2265) took 1332 ms. Trace:
I20260709 12:22:18.590101 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 538) took 1739 ms. Trace:
I20260709 12:22:18.590730 31677 rpcz_store.cc:276] 0709 12:22:16.929018 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.351743 (+422725us) service_pool.cc:224] Handling call
0709 12:22:17.749048 (+397305us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.749242 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.749266 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:17.749307 (+    41us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:17.749337 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.749389 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.589882 (+840493us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b54003921d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:18.590028 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":913021184}
I20260709 12:22:18.591789  3292 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:18.592322  3079 consensus_queue.cc:1175] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 11 status { last_received { term: 11 index: 7 } last_committed_idx: 7 last_received_current_leader { term: 11 index: 7 } }
I20260709 12:22:18.368404  3412 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:18.514787 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:18.376708  3015 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.607s	user 0.023s	sys 0.005s
I20260709 12:22:18.555735 31827 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:18.340019  3424 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:18.565246 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 913434us
I20260709 12:22:18.568730 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:18.370819  3464 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:18.572898 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 68481.9us
I20260709 12:22:18.574131 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2235) took 1538 ms. Trace:
W20260709 12:22:18.372689  3194 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.009s	sys 0.019s
W20260709 12:22:18.581313 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.583235  3287 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:18.584914  3367 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:18.585604 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1572) took 1878 ms. Trace:
I20260709 12:22:18.586560  3471 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.621932  3471 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:18.360549  3454 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.624269  3257 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:18.575644  2975 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:18.625695  3454 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:18.629251  3237 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:18.629503  3342 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:18.630267  3342 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:18.630604  3424 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:18.571187  3086 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.637085  3086 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:18.523152 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2233) took 1492 ms. Trace:
I20260709 12:22:18.637684 31749 rpcz_store.cc:276] 0709 12:22:17.031007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.031163 (+   156us) service_pool.cc:224] Handling call
0709 12:22:17.031513 (+   350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:17.031885 (+   372us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:17.801372 (+769487us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.314317 (+512945us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:18.315068 (+   751us) log.cc:844] Serialized 26 byte log entry
0709 12:22:18.315304 (+   236us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:18.315358 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.315441 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:18.522832 (+207391us) raft_consensus.cc:1685] finished
0709 12:22:18.522864 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.523064 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19466,"prepare.queue_time_us":584677,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":48,"raft.queue_time_us":21052,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":26129,"thread_start_us":1264793,"threads_started":2,"wal-append.queue_time_us":716516}
I20260709 12:22:18.614122 31678 rpcz_store.cc:276] 0709 12:22:16.706852 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.266358 (+559506us) service_pool.cc:224] Handling call
0709 12:22:17.266814 (+   456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:18.173809 (+906995us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.585267 (+411458us) spinlock_profiling.cc:243] Waited 833 ms on lock 0x7b5400229f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:18.585520 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1916284672}
I20260709 12:22:18.598515 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:18.513371  3283 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.606077 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:18.611483 31744 rpcz_store.cc:276] 0709 12:22:17.036175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.036325 (+   150us) service_pool.cc:224] Handling call
0709 12:22:17.185636 (+149311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:17.185944 (+   308us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:17.785457 (+599513us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.573489 (+788032us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.573692 (+   203us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:18.573747 (+    55us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:18.573810 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.573904 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:18.574057 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29767,"prepare.queue_time_us":432900,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":11895,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":131,"spinlock_wait_cycles":115839872,"thread_start_us":1309024,"threads_started":2,"wal-append.queue_time_us":817551}
W20260709 12:22:18.589591  3447 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.312s	user 0.001s	sys 0.001s
I20260709 12:22:18.572749  3159 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:18.649821  3194 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.779s	user 0.013s	sys 0.022s
I20260709 12:22:18.619616  3367 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:18.609009  3470 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:18.590046 31824 rpcz_store.cc:276] 0709 12:22:17.257333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.377093 (+119760us) service_pool.cc:224] Handling call
0709 12:22:17.377363 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:18.363891 (+986528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.589206 (+225315us) spinlock_profiling.cc:243] Waited 956 ms on lock 0x7b54001969d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:18.589476 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2199090432}
I20260709 12:22:18.572067  3466 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:18.601722 31676 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 11 because replica has last-logged OpId of term: 8 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 8.
I20260709 12:22:18.371239  3456 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:18.653679 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47725.5us
I20260709 12:22:18.652365  3285 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.730556 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:18.364030  3452 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:18.612143 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 43517.5us
I20260709 12:22:18.741202 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:18.652571  3203 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.248s	user 0.015s	sys 0.026s
I20260709 12:22:18.683629  3324 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:22:18.689847 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.602818  3015 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:18.652251  3257 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.731441 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:18.660429  3287 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:18.738832 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:18.745559  3193 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.569s	user 0.013s	sys 0.000s
I20260709 12:22:18.647969  3464 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.760703 31824 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:18.755340 31826 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:18.743093  3216 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:18.758055 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:18.748127 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:18.642443  3291 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:18.732445  3450 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:18.771847 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.672130  3400 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:18.744954  3203 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.341s	user 0.016s	sys 0.027s
I20260709 12:22:18.746271  3336 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:18.774976 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:18.759764  3287 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:18.775923  3426 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:18.756390  3257 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.778930  3440 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:18.779886  3456 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.752630  3323 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:18.787604  3267 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:18.787395  3237 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:18.783578 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:18.750720  3466 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:18.756613  3358 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:18.797087 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.771948  3370 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:18.749979  3285 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.806481 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:18.783420 31826 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:18.820966 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:18.812968 32370 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:18.773530  3233 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260709 12:22:18.745333  3470 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.831043  3433 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:18.831522  3433 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:18.831745 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:18.832423  3433 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:18.833330  3470 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:18.833423 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:18.799547  3073 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:18.833902  3454 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:18.800212  3193 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.834676  3450 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:18.834836  3193 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:18.837752  3193 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:18.818303  3015 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:18.825022  3398 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:18.803680  3407 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:18.826617  3257 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:18.826936  3473 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:18.828035  3194 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:18.798870  3400 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:18.801301  3287 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:18.838603 31676 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:18.833745  3433 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:18.823541 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:18.839680 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:19.435333 31678 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:18.919313 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:19.436039  3502 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.343s	user 0.001s	sys 0.000s
I20260709 12:22:18.853055  3464 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:18.814352  3267 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:18.839381 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:19.059950 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.845392 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:19.049854 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:18.851435  3432 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:18.857291  3407 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:19.436978  3267 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:19.447549  3407 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:18.857733  3473 consensus_queue.cc:1243] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 11.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260709 12:22:19.227293 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2263) took 1996 ms (client timeout 2151 ms). Trace:
W20260709 12:22:19.453363 31822 rpcz_store.cc:269] 0709 12:22:17.230896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.325216 (+ 94320us) service_pool.cc:224] Handling call
0709 12:22:19.227063 (+1901847us) spinlock_profiling.cc:243] Waited 797 ms on lock 0x7b54004466d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:19.227226 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1833270144}
I20260709 12:22:18.858748 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:18.858994 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 247731us
I20260709 12:22:18.840382 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:18.839041 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:19.006711 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:19.463786 31643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 2.151s (SENT) [suppressed 14 similar messages]
I20260709 12:22:19.465705 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 535) took 1329 ms. Trace:
W20260709 12:22:18.867123 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:19.476495 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:19.472538 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 874031us
I20260709 12:22:19.467960 32376 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:19.476181 31820 rpcz_store.cc:276] 0709 12:22:18.136754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.136928 (+   174us) service_pool.cc:224] Handling call
0709 12:22:18.137343 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:18.265800 (+128457us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:18.265837 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:18.265904 (+    67us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:18.265940 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.226880 (+960940us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b540043dad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:19.465403 (+238523us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.465619 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":294906240}
I20260709 12:22:19.468976 31675 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:18.897571 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:19.475374 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 651837us
I20260709 12:22:19.486603 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:19.487195 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:19.487612 32371 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 9 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:22:19.490481 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:19.491677 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:19.494386 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:19.495587 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:19.511816 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.482098 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:19.517001 31669 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:19.517427 31669 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:19.518859 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60942.5us
W20260709 12:22:19.196524  3292 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.589s	user 0.009s	sys 0.003s
W20260709 12:22:19.194796  3444 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.151s	user 0.007s	sys 0.000s
I20260709 12:22:18.819114  3358 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:19.223426  3457 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.945s	user 0.004s	sys 0.001s
I20260709 12:22:18.857990  3291 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:19.157656  3501 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.083s	user 0.003s	sys 0.000s
W20260709 12:22:19.065800  3461 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.790s	user 0.000s	sys 0.002s
W20260709 12:22:19.442533  3412 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.830s	user 0.006s	sys 0.002s
I20260709 12:22:19.528800  3370 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.513429  3076 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:18.859967  3342 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:19.416800  3324 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:19.532892 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:19.534267 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:19.535382 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:19.575053 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:18.856014  3205 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:18.969466  3426 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.548602  3221 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 12:22:19.541635 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:19.543889 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.580339 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:18.932260  3371 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:19.374192  3015 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.007s	sys 0.007s
I20260709 12:22:19.260896  3336 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:19.204658  3398 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.547511 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:19.562402  3076 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.247943  3285 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:19.056075  3452 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:19.473176  3438 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:19.515214  3194 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:18.859157  2975 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.687s
I20260709 12:22:19.532017  3321 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:19.537305  3492 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:19.523216 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:19.604444 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 81345.9us
I20260709 12:22:19.580238  3470 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:22:19.561662 32369 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.030s	sys 0.007s
W20260709 12:22:19.609328 32369 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.684s	user 0.030s	sys 0.007s
I20260709 12:22:19.528402 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:19.567327 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:19.575700 31825 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:19.575927 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:19.591264  3336 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:19.527741  3207 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.037s
I20260709 12:22:19.552209  2874 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:19.220333  3223 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:19.599543  3321 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.554549  3464 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:19.628075 31790 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.710s	user 0.038s	sys 0.044s
W20260709 12:22:19.561364  3323 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.747s	user 0.012s	sys 0.000s
I20260709 12:22:19.584213 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:19.584592 31824 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.008s	sys 0.030s
W20260709 12:22:19.629235 31824 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.008s	sys 0.030s
I20260709 12:22:19.586935  3426 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:19.586068 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:19.631111 32377 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:19.629328 32374 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:19.545436  3424 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.027s
I20260709 12:22:19.589993 32371 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:19.614075  3086 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:18.847667  3283 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:19.176728  3466 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:19.565196  3400 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:19.574733  3324 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:18.816186  3474 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.781s
W20260709 12:22:19.586433  3015 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.007s	sys 0.008s
I20260709 12:22:19.595532  3079 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.623073  3223 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:19.584055  3440 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.537914  3402 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:19.583354  3058 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.587051  3437 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.562723  3412 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:19.612341 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:19.576539 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:19.610764  3452 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:19.603811  3492 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:19.592980  3076 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:19.615633 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:19.563656  3370 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:19.593271  3194 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:19.594396  3205 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:19.582259 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 612) took 1246 ms. Trace:
I20260709 12:22:19.595650  3398 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:19.620563  2874 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:19.628350  3321 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:19.616997  3470 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:19.633078  3323 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:19.635910  3086 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:19.677078 32376 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:19.668705  3402 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.680668  3492 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:19.676023  3292 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:22:19.686326  3194 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:19.668473  3079 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:19.601429  3438 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:19.679948  3437 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:19.671849  3440 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:19.707576  3402 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:19.689256  3324 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:19.737002  3505 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:22:19.734681  3426 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:19.755924  3437 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:19.758349  3438 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:19.760234  3440 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:19.765079  3470 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:19.518163 31669 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:19.832481 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 364749us
I20260709 12:22:19.834313 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:19.840210 31669 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:19.580946 31744 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:19.586573 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:19.842670 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2238) took 2804 ms. Trace:
I20260709 12:22:19.844534 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:19.843286 31745 rpcz_store.cc:276] 0709 12:22:17.038302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.370810 (+332508us) service_pool.cc:224] Handling call
0709 12:22:17.371179 (+   369us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:17.421800 (+ 50621us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.421848 (+    48us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:17.422728 (+   880us) log.cc:844] Serialized 52 byte log entry
0709 12:22:17.881411 (+458683us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:17.905626 (+ 24215us) log.cc:844] Serialized 14 byte log entry
0709 12:22:17.905916 (+   290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.906026 (+   110us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:19.064981 (+1158955us) spinlock_profiling.cc:243] Waited 22.7 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:19.841650 (+776669us) raft_consensus.cc:1685] finished
0709 12:22:19.841865 (+   215us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.842363 (+   498us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56368,"prepare.queue_time_us":24572,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":115081,"raft.run_cpu_time_us":274,"raft.run_wall_time_us":48918,"spinlock_wait_cycles":52319488,"thread_start_us":458205,"threads_started":1,"wal-append.queue_time_us":853012}
I20260709 12:22:19.846050 31747 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:19.575671 31748 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:19.573441 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:19.587077 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:19.589313 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2234) took 2556 ms. Trace:
I20260709 12:22:19.851217 31749 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:19.851028 31753 rpcz_store.cc:276] 0709 12:22:17.032983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:17.033139 (+   156us) service_pool.cc:224] Handling call
0709 12:22:17.033467 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:17.033845 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:17.033869 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:17.034342 (+   473us) log.cc:844] Serialized 26 byte log entry
0709 12:22:17.678519 (+644177us) raft_consensus.cc:1645] Marking committed up to 2
0709 12:22:17.678586 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:17.678728 (+   142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:18.872108 (+1193380us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:19.584982 (+712874us) raft_consensus.cc:1685] finished
0709 12:22:19.585007 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.589240 (+  4233us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":540,"raft.run_cpu_time_us":44,"raft.run_wall_time_us":43,"spinlock_wait_cycles":3244288,"thread_start_us":643723,"threads_started":1,"wal-append.queue_time_us":1009310}
I20260709 12:22:19.574384 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:19.857796 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:19.838296 31717 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:19.867508  3523 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.868588  3523 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:19.875119 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 335705us
W20260709 12:22:19.632294 31790 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.039s	sys 0.044s
I20260709 12:22:19.607852 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:19.583006 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:19.960777  3324 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:19.961398 31819 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:19.964187 31819 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:19.964984 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 547) took 1260 ms. Trace:
I20260709 12:22:19.965353 31819 rpcz_store.cc:276] 0709 12:22:18.704853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.962937 (+1258084us) service_pool.cc:224] Handling call
0709 12:22:19.963795 (+   858us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:19.964680 (+   885us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.964901 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:19.678105 31821 raft_consensus.cc:2410] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:22:19.677784 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:19.675907 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 186413us
I20260709 12:22:19.685068 32372 rpcz_store.cc:276] 0709 12:22:18.335587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.335720 (+   133us) service_pool.cc:224] Handling call
0709 12:22:18.336037 (+   317us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:18.336448 (+   411us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:18.636181 (+299733us) spinlock_profiling.cc:243] Waited 8.35 ms on lock 0x7b18004db2a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:22:18.686864 (+ 50683us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.802385 (+115521us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.802504 (+   119us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:18.803161 (+   657us) log.cc:844] Serialized 52 byte log entry
0709 12:22:18.803321 (+   160us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:18.803957 (+   636us) log.cc:844] Serialized 14 byte log entry
0709 12:22:18.817053 (+ 13096us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.924137 (+107084us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:19.268846 (+344709us) spinlock_profiling.cc:243] Waited 2.72 ms on lock 0x7b18004db2a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:22:19.582015 (+313169us) raft_consensus.cc:1685] finished
0709 12:22:19.582037 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.582185 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96054,"prepare.queue_time_us":365519,"prepare.run_cpu_time_us":146,"prepare.run_wall_time_us":154,"raft.queue_time_us":16786,"raft.run_cpu_time_us":407,"raft.run_wall_time_us":55834,"spinlock_wait_cycles":25455360,"thread_start_us":162514,"threads_started":2,"wal-append.queue_time_us":281373}
I20260709 12:22:19.973506 31572 catalog_manager.cc:5697] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 6 to 7, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 7 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:19.975669 31572 catalog_manager.cc:5697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 3 to 7, term changed from 7 to 8, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to <none>, VOTER 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) evicted. New cstate: current_term: 8 committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:22:19.978775 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61325us
I20260709 12:22:19.981621 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:19.982579 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 543) took 1279 ms. Trace:
I20260709 12:22:19.984710  3371 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:19.987455  3291 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:22:19.987663 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.987968  3291 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:19.675768  3058 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.002931  3291 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:20.003103  3205 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.002344  3058 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:20.006739 31744 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
W20260709 12:22:19.931113 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 69022us
I20260709 12:22:19.692241  2874 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:19.935261  3437 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.692318  3223 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:20.018918 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.942014  3398 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:19.943027  3488 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:19.943547  3470 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:22:19.944181 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 86771.9us
I20260709 12:22:19.945021  3517 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.946121  3515 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:20.027132  3437 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.028337  3515 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.028899  3515 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:20.029680 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:20.030202  3437 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:19.955215 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 544) took 1251 ms. Trace:
I20260709 12:22:19.955828 31820 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:22:19.955462  3086 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:19.956214  2975 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.019s
I20260709 12:22:19.957121 31826 raft_consensus.cc:2410] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 6.
I20260709 12:22:20.048570 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 542) took 1345 ms. Trace:
I20260709 12:22:20.053486  3342 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:20.054950  3523 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:20.055562  3473 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:19.912258 31669 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:19.935339  3015 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:19.996289  3426 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:19.998725 31748 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:19.978008 31572 catalog_manager.cc:5697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 5 to 9, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 9 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:19.928565  3305 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:20.071806 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2274) took 1263 ms. Trace:
I20260709 12:22:20.023985 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 764) took 1199 ms. Trace:
I20260709 12:22:20.073490  3426 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.072342 31821 rpcz_store.cc:276] 0709 12:22:18.808727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.528112 (+719385us) service_pool.cc:224] Handling call
0709 12:22:20.071308 (+543196us) spinlock_profiling.cc:243] Waited 67.2 ms on lock 0x7b54003c9bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:20.071597 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":154582912}
I20260709 12:22:20.073362 32369 rpcz_store.cc:276] 0709 12:22:18.823970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.824160 (+   190us) service_pool.cc:224] Handling call
0709 12:22:18.824426 (+   266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:18.824727 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:19.609799 (+785072us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.023199 (+413400us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:20.023319 (+   120us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:20.023423 (+   104us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.023542 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.023756 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47087,"prepare.queue_time_us":106608,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":51759,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":175,"thread_start_us":1049224,"threads_started":2,"wal-append.queue_time_us":139401}
I20260709 12:22:20.024761  3079 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:20.071836 31572 catalog_manager.cc:5697] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 3 to 8. New cstate: current_term: 8 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:20.024504  3513 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:20.030576 31825 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:22:20.031149  3452 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.031857  3412 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:20.032651  3370 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:20.032910 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 425008us
I20260709 12:22:20.033082  3508 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:19.693058  3267 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.025807  3517 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:19.954165 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:20.036356 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:20.036903 32376 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 9 index: 9. (term mismatch)
I20260709 12:22:20.036974  3488 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.038563 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:20.038983 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:20.040357 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44064.9us
I20260709 12:22:20.077812 31572 catalog_manager.cc:5697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 2 to 8, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 8 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:20.043406  3223 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:20.044459 31822 rpcz_store.cc:276] 0709 12:22:18.703265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.472141 (+768876us) service_pool.cc:224] Handling call
0709 12:22:19.503181 (+ 31040us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:19.520991 (+ 17810us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:19.521042 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:19.527856 (+  6814us) log.cc:844] Serialized 26 byte log entry
0709 12:22:19.928421 (+400565us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:19.928544 (+   123us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.928653 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:19.928729 (+    76us) raft_consensus.cc:1685] finished
0709 12:22:19.928740 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.954912 (+ 26172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17211,"raft.queue_time_us":17926,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":13617,"thread_start_us":358202,"threads_started":1,"wal-append.queue_time_us":194862}
W20260709 12:22:20.060050 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:19.983660 31824 rpcz_store.cc:276] 0709 12:22:18.702969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.804538 (+101569us) service_pool.cc:224] Handling call
0709 12:22:18.824210 (+ 19672us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:19.629765 (+805555us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:19.629814 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:19.630436 (+   622us) log.cc:844] Serialized 26 byte log entry
0709 12:22:19.981517 (+351081us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:19.981655 (+   138us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.981870 (+   215us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:19.981943 (+    73us) raft_consensus.cc:1685] finished
0709 12:22:19.981975 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:19.982246 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2376,"raft.queue_time_us":100644,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":10330,"thread_start_us":1016103,"threads_started":2,"wal-append.queue_time_us":280504}
I20260709 12:22:20.067734  3015 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.056461  3473 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.069370  3398 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.049115 31818 rpcz_store.cc:276] 0709 12:22:18.702608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.778427 (+ 75819us) service_pool.cc:224] Handling call
0709 12:22:18.778804 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:18.779229 (+   425us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:18.802516 (+ 23287us) log.cc:844] Serialized 14 byte log entry
0709 12:22:19.226616 (+424100us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:19.227174 (+   558us) log.cc:844] Serialized 26 byte log entry
0709 12:22:19.227312 (+   138us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:19.227349 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:19.227433 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:19.259199 (+ 31766us) spinlock_profiling.cc:243] Waited 5.76 ms on lock 0x7b4800130158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:22:19.938145 (+678946us) raft_consensus.cc:1685] finished
0709 12:22:20.008188 (+ 70043us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.048331 (+ 40143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6696,"prepare.queue_time_us":23715,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":43,"raft.queue_time_us":444,"raft.run_cpu_time_us":1940,"raft.run_wall_time_us":479992,"spinlock_wait_cycles":13249408,"thread_start_us":423369,"threads_started":1,"wal-append.queue_time_us":778825}
W20260709 12:22:20.083149 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 101744us
I20260709 12:22:20.041290 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:20.086176 32376 pending_rounds.cc:85] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 8
I20260709 12:22:20.082067  3452 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:20.095665  2975 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election won for term 9
W20260709 12:22:20.095960 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59788.6us
I20260709 12:22:20.096295  2975 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:20.096565  3298 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260709 12:22:20.097648  2975 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.098872 32375 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:20.099177 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:20.100225 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:20.100745 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:20.102165 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:20.102689 32378 raft_consensus.cc:2368] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13.
I20260709 12:22:20.103284 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:20.104671  3517 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:20.100251 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:20.106745  3336 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.107225  3086 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:20.117472 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2273) took 1309 ms. Trace:
I20260709 12:22:20.107939  3424 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.117982 31826 rpcz_store.cc:276] 0709 12:22:18.808346 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.833105 (+ 24759us) service_pool.cc:224] Handling call
0709 12:22:20.117160 (+1284055us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540030e8d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:20.117375 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2435107200}
I20260709 12:22:20.118728  3424 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.119896 31826 pending_rounds.cc:85] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 5
I20260709 12:22:20.120244 31826 pending_rounds.cc:107] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:22:20.120507 31826 raft_consensus.cc:2889] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:20.121160  3473 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:20.093371 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:20.123983  3015 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:20.114001  3488 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.124893 31825 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 12:22:20.125391 31825 pending_rounds.cc:85] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 9
I20260709 12:22:20.128273 31825 pending_rounds.cc:107] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260709 12:22:20.128598 31825 raft_consensus.cc:2889] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:20.128690  3267 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:20.129558 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2278) took 1310 ms. Trace:
I20260709 12:22:20.129688  3454 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:20.091696 32376 pending_rounds.cc:107] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 8.9
I20260709 12:22:20.081544  3426 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:20.137177 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 550) took 1251 ms. Trace:
I20260709 12:22:20.114615 31822 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:20.083995  3508 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:20.127265 31576 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42070 (request call id 222) took 30549 ms (client timeout 15000 ms). Trace:
I20260709 12:22:20.127741 31748 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 10 index: 6. (term mismatch)
I20260709 12:22:20.140368 31748 pending_rounds.cc:85] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 5
I20260709 12:22:20.113147  3398 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.084609  3342 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.144150 31748 pending_rounds.cc:107] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 9.6
I20260709 12:22:20.144410 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2271) took 1332 ms. Trace:
I20260709 12:22:20.144663 31748 raft_consensus.cc:2889] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:20.136217 32376 raft_consensus.cc:2889] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:20.144881 31822 rpcz_store.cc:276] 0709 12:22:18.807570 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.113773 (+1306203us) service_pool.cc:224] Handling call
0709 12:22:20.114273 (+   500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.139841 (+ 25568us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.140039 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19712}
I20260709 12:22:20.129966 31825 rpcz_store.cc:276] 0709 12:22:18.819307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.123919 (+1304612us) service_pool.cc:224] Handling call
0709 12:22:20.124363 (+   444us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.129223 (+  4860us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.129460 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:20.137722 31669 rpcz_store.cc:276] 0709 12:22:18.834450 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.834594 (+   144us) service_pool.cc:224] Handling call
0709 12:22:18.834920 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:19.516902 (+681982us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x7b5400357620. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:22:20.080062 (+563160us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.086090 (+  6028us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1412903680}
I20260709 12:22:20.149510 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:20.150019  3398 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:20.151393 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:20.152412 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:20.152861 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:20.130231  3454 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:20.154919  3454 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:20.156049 32371 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.522s	user 0.014s	sys 0.002s
W20260709 12:22:20.156394  3506 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.543s	user 0.002s	sys 0.002s
I20260709 12:22:20.156528 32371 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:20.142692  3508 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:20.144146 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:20.159984 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:20.162701  3342 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:20.162819 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 70409.5us
I20260709 12:22:20.154165  3492 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.176429  3492 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:20.176712  3336 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:20.177356 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 602853us
I20260709 12:22:20.179884 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 551) took 1451 ms. Trace:
I20260709 12:22:20.180962 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 549) took 1475 ms. Trace:
I20260709 12:22:20.181425 31827 rpcz_store.cc:276] 0709 12:22:18.705638 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.968499 (+1262861us) service_pool.cc:224] Handling call
0709 12:22:20.179784 (+211285us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:22:20.180102 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.180409 (+   307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:20.180441 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:20.180499 (+    58us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:20.180542 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.180601 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.180741 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":46970112}
W20260709 12:22:20.183619  3358 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.629s	user 0.012s	sys 0.004s
I20260709 12:22:20.184389  3358 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.186877  3426 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.187650  3426 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.189412  3512 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:20.190436 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:20.180459 31821 rpcz_store.cc:276] 0709 12:22:18.729041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.075094 (+1346053us) service_pool.cc:224] Handling call
0709 12:22:20.085909 (+ 10815us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.179438 (+ 93529us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:20.179467 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:20.179532 (+    65us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:20.179563 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.179648 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.179785 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:20.189918  3512 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.260804 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:20.266036 31824 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
W20260709 12:22:20.263128 32342 connection.cc:256] RPC call timeout handler was delayed by -1.0214411130000371s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
W20260709 12:22:20.276086 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58125.8us
I20260709 12:22:20.218951  3526 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.282867 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:20.257035  3512 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:20.280579 31825 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:22:20.287881 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2279) took 1411 ms. Trace:
I20260709 12:22:20.288352 31825 rpcz_store.cc:276] 0709 12:22:18.876858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.165493 (+1288635us) service_pool.cc:224] Handling call
0709 12:22:20.217115 (+ 51622us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:20.287478 (+ 70363us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.287800 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:20.264546  3488 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:20.286455 31572 catalog_manager.cc:5697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 7 to 8, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 8 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:20.285763  3426 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:20.297376  3015 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.301261  3342 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:20.331966 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.332026 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.332078 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.331847 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 61000.5us
I20260709 12:22:20.338573  3323 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:20.340042  2874 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.341804 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:20.340479 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.308634  3267 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:20.379891 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:20.402230 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.379035 31572 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:20.381639 31745 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:20.378756 31718 leader_election.cc:400] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13.
I20260709 12:22:20.433585 31718 leader_election.cc:403] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 12:22:20.435402 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:20.444496 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.446462 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.394038  3529 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
W20260709 12:22:20.338594  3287 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.824s	user 0.017s	sys 0.001s
W20260709 12:22:20.391556  3433 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.904s	user 0.009s	sys 0.009s
W20260709 12:22:20.401202  3321 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.699s	user 0.006s	sys 0.005s
W20260709 12:22:20.394085  3194 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.670s	user 0.009s	sys 0.000s
W20260709 12:22:20.391074  3402 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.654s	user 0.013s	sys 0.002s
I20260709 12:22:20.396200  3358 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:20.342604  2874 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:20.461859 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 118790us
I20260709 12:22:20.380770  3342 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:20.343353  3437 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:20.465348 31572 catalog_manager.cc:5697] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 5 to 8, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 8 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:20.470294 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 147433us
W20260709 12:22:20.139895 31576 rpcz_store.cc:269] 0709 12:21:49.542719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:49.569243 (+ 26524us) service_pool.cc:224] Handling call
0709 12:22:20.087882 (+30518639us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:11.470088 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:11.570757 (+100669us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:11.570806 (+    49us) write_op.cc:435] Acquired schema lock
0709 12:22:11.570826 (+    20us) tablet.cc:662] Decoding operations
0709 12:22:11.572528 (+  1702us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:11.572608 (+    80us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:11.572637 (+    29us) tablet.cc:685] Acquiring locks for 17 operations
0709 12:22:11.744080 (+171443us) tablet.cc:701] Row locks acquired
0709 12:22:11.744145 (+    65us) write_op.cc:260] PREPARE: finished
0709 12:22:11.744389 (+   244us) write_op.cc:270] Start()
0709 12:22:11.744592 (+   203us) write_op.cc:276] Timestamp: P: 1783599731744300 usec, L: 0
0709 12:22:11.744622 (+    30us) op_driver.cc:348] REPLICATION: starting
0709 12:22:11.745692 (+  1070us) log.cc:844] Serialized 6845 byte log entry
0709 12:22:12.287264 (+541572us) op_driver.cc:464] REPLICATION: finished
0709 12:22:12.375284 (+ 88020us) write_op.cc:301] APPLY: starting
0709 12:22:12.375397 (+   113us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:12.375778 (+   381us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:12.375938 (+   160us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:12.442918 (+ 66980us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:12.443326 (+   408us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0709 12:22:12.443388 (+    62us) write_op.cc:312] APPLY: finished
0709 12:22:12.443907 (+   519us) log.cc:844] Serialized 119 byte log entry
0709 12:22:12.446915 (+  3008us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:12.447107 (+   192us) write_op.cc:454] Released schema lock
0709 12:22:12.447193 (+    86us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:12.451094 (+  3901us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:12.451128 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":13686,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":84220,"apply.run_cpu_time_us":8204,"apply.run_wall_time_us":75783,"num_ops":17,"prepare.queue_time_us":517841,"prepare.run_cpu_time_us":22937,"prepare.run_wall_time_us":449600,"raft.queue_time_us":184339,"raft.run_cpu_time_us":16237,"raft.run_wall_time_us":172972,"replication_time_us":503242,"spinlock_wait_cycles":3456,"thread_start_us":742317,"threads_started":4,"wal-append.queue_time_us":177932,"wal-append.run_cpu_time_us":71216,"wal-append.run_wall_time_us":1522174}]]}
I20260709 12:22:20.462559 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.343978 31822 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 12:22:20.338660  3440 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.203s
W20260709 12:22:20.501696  3076 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.772s	user 0.011s	sys 0.003s
W20260709 12:22:20.502193  3530 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260709 12:22:20.502296  3076 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.504994 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:20.505446  3076 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:20.506179 31746 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:20.472812  3194 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.507225 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:20.508324  3194 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:20.508425 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:20.475286  3287 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.510150 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:20.470239  3437 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.474391  3412 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 12:22:20.514428 31747 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:20.483035  3407 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:20.483695 32378 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:20.518952 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:20.470749  3342 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:20.484127 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:20.484391 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:20.485061 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.485486 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 548) took 1780 ms. Trace:
I20260709 12:22:20.387217  3015 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.328544  3526 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.457592  3336 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:20.460032  3492 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:20.487668 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:20.458348  3452 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.515601 31752 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:20.472118  3515 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.474246  3370 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.525991  3407 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.544880  3058 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.527659 32374 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 10 because replica has last-logged OpId of term: 8 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 7.
I20260709 12:22:20.527840 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.528735 31674 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:20.530539 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:20.520939 31819 rpcz_store.cc:276] 0709 12:22:18.705147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.966267 (+1261120us) service_pool.cc:224] Handling call
0709 12:22:20.061048 (+ 94781us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:20.139672 (+ 78624us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:20.139705 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:20.141905 (+  2200us) log.cc:844] Serialized 26 byte log entry
0709 12:22:20.271046 (+129141us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:20.271080 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.271145 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:20.485242 (+214097us) raft_consensus.cc:1685] finished
0709 12:22:20.485268 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.485416 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":128710,"threads_started":1,"wal-append.queue_time_us":327295}
I20260709 12:22:20.511445 32376 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:22:20.531416 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:20.501765 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:20.533033  3291 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.529s	user 0.014s	sys 0.000s
I20260709 12:22:20.526618 31751 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: [11.6-11.6]
I20260709 12:22:20.460186  3267 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:20.550175 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58051.5us
I20260709 12:22:20.511919  3287 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:20.524562  3437 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:20.556505 31672 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:20.510766  3460 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:20.525575  3342 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:20.566812 31746 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:20.553639 31572 catalog_manager.cc:5697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 1 to 7. New cstate: current_term: 7 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:20.542232  3371 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:20.488471  3402 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.477433  3321 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:20.572607 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 85077.5us
I20260709 12:22:20.585033 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:20.552506  3058 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.545764  3433 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.593684 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:20.591153 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:20.595436 31753 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:20.599675  3058 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:20.603322 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:20.567641  3492 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.610881 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2256) took 1037 ms. Trace:
I20260709 12:22:20.563421  3086 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:20.603261  3223 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.510s	user 0.005s	sys 0.008s
I20260709 12:22:20.613722 31751 rpcz_store.cc:276] 0709 12:22:19.573112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.573629 (+   517us) service_pool.cc:224] Handling call
0709 12:22:19.573927 (+   298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:20.445051 (+871124us) spinlock_profiling.cc:243] Waited 273 ms on lock 0x7b54004d15a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:22:20.587634 (+142583us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.610813 (+ 23179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":627280128}
I20260709 12:22:20.552793  3336 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.616627 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:20.615902 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:20.624079 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 92643.7us
I20260709 12:22:20.563831  3370 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.637845 31674 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:20.617405 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:20.639820  3233 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:20.640280 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:20.640441  3233 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:20.640954 31822 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 6 FOLLOWER]: Advancing to term 7
I20260709 12:22:20.641626  3233 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.609814  3371 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:20.647058 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 68439.1us
I20260709 12:22:20.651296 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.652334 31672 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:20.653299 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:20.653612 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60034.9us
I20260709 12:22:20.638679 31819 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:20.654578 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.575022  3291 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.671756 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.636811 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.674057 31672 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:20.654768 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:20.685214 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:20.682569 31677 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:20.691268 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:20.637406  3086 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.679649 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.706029 31670 raft_consensus.cc:2410] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 12 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:22:20.706332 31669 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
W20260709 12:22:20.699986 31791 connection.cc:256] RPC call timeout handler was delayed by -1.927030971000022s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.223s for a response
I20260709 12:22:20.685987  3076 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:20.716883  3545 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.104s	user 0.001s	sys 0.000s
W20260709 12:22:20.717555  3531 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.332s	user 0.001s	sys 0.000s
I20260709 12:22:20.719290  3267 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:20.655001  3433 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.697683  3526 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:20.728158 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.733992 31672 raft_consensus.cc:1813] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Handling vote request from an unknown peer 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:20.731676 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:20.728127 31572 catalog_manager.cc:5697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:20.740396 31673 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260709 12:22:20.712713  3194 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.743932 31672 raft_consensus.cc:2368] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12.
W20260709 12:22:20.725756 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 136747us
W20260709 12:22:20.742416  3454 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.587s	user 0.015s	sys 0.000s
W20260709 12:22:20.755846  3504 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.142s	user 0.000s	sys 0.003s
I20260709 12:22:20.756142  3454 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.758800 31818 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:20.759109 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2173) took 2058 ms. Trace:
I20260709 12:22:20.760392  3402 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:20.759529 31823 rpcz_store.cc:276] 0709 12:22:18.700897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:18.701082 (+   185us) service_pool.cc:224] Handling call
0709 12:22:18.701544 (+   462us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:18.743469 (+ 41925us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:18.743518 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:18.753153 (+  9635us) log.cc:844] Serialized 26 byte log entry
0709 12:22:18.974480 (+221327us) raft_consensus.cc:1645] Marking committed up to 3
0709 12:22:18.974522 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:18.974594 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:19.071623 (+ 97029us) spinlock_profiling.cc:243] Waited 2.98 ms on lock 0x7b4800130158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:22:20.670701 (+1599078us) spinlock_profiling.cc:243] Waited 2.85 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:20.758706 (+ 88005us) raft_consensus.cc:1685] finished
0709 12:22:20.758727 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.758920 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":34612,"raft.run_cpu_time_us":1253,"raft.run_wall_time_us":325944,"spinlock_wait_cycles":13388928,"thread_start_us":200193,"threads_started":1,"wal-append.queue_time_us":853977}
I20260709 12:22:20.761658 31747 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: []
I20260709 12:22:20.762247 31823 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:20.762560 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:20.721462  3287 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.033s
I20260709 12:22:20.767153 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2291) took 1124 ms. Trace:
I20260709 12:22:20.769491 31670 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
W20260709 12:22:20.770925 31791 connection.cc:256] RPC call timeout handler was delayed by -1.5186355610003375s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1539s for a response
I20260709 12:22:20.771988 31674 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:20.773439 31823 rpcz_store.cc:276] 0709 12:22:19.643375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.761451 (+1118076us) service_pool.cc:224] Handling call
0709 12:22:20.761795 (+   344us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.766908 (+  5113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.767091 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:20.776075 31791 connection.cc:256] RPC call timeout handler was delayed by -1.50783556100032s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15510000000000002s for a response
W20260709 12:22:20.773386 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.776520 31791 connection.cc:256] RPC call timeout handler was delayed by -1.5042355610003142s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1555s for a response
W20260709 12:22:20.776865 31791 connection.cc:256] RPC call timeout handler was delayed by -1.3161355610004648s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1764s for a response
W20260709 12:22:20.777163 31791 connection.cc:256] RPC call timeout handler was delayed by -1.2738355610003964s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1811s for a response
W20260709 12:22:20.777401 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0947355610001068s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20099999999999998s for a response
I20260709 12:22:20.780540 31671 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:20.780828 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.763504  3433 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:20.767056  3454 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:20.795794 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.779358  3515 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:20.779177  3358 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.766438  3194 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:20.808473  3194 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:20.803694 31572 catalog_manager.cc:5697] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 3 to 8. New cstate: current_term: 8 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:20.805459 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.812963 31672 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:20.814859 31671 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:20.817140 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.818890 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.819235 31671 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 12: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:20.819816 31672 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:20.820923 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.821256 31670 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:22:20.817209 31572 catalog_manager.cc:5697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:20.814751  3194 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:20.779968  3512 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.798220  3015 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:20.818015 31823 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
W20260709 12:22:20.835222  3548 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
W20260709 12:22:20.839713 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:20.834795  3552 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20260709 12:22:20.768747  3371 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:20.816913  3324 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.855s	user 0.012s	sys 0.006s
I20260709 12:22:20.811740  3321 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:20.873569  3194 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:20.885290 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.886068 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 51703.6us
I20260709 12:22:20.879215  3492 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:20.962307  3560 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 12:22:20.962791  3559 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.057s	user 0.000s	sys 0.002s
W20260709 12:22:20.890857  3508 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.731s	user 0.012s	sys 0.001s
I20260709 12:22:20.916806 31673 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 5. (term mismatch)
I20260709 12:22:20.988735 31673 pending_rounds.cc:85] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 4
I20260709 12:22:20.989041 31673 pending_rounds.cc:107] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:22:20.989290 31673 raft_consensus.cc:2889] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:20.974433 31671 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: [11.6-11.6]
W20260709 12:22:20.971783  3548 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.081s	user 0.004s	sys 0.000s
W20260709 12:22:21.002992 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.004995 31572 catalog_manager.cc:5697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:21.007915 31572 catalog_manager.cc:5697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 4 to 7. New cstate: current_term: 7 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:21.010805 31672 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:22:20.878170  3358 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:20.993310 31822 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 7.
W20260709 12:22:20.996440 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:20.997344  3553 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
W20260709 12:22:20.998243  3556 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.181s	user 0.000s	sys 0.000s
I20260709 12:22:20.971642  3291 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:20.998579  3438 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.028s
I20260709 12:22:20.999006 31670 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:20.999907 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2280) took 1512 ms. Trace:
I20260709 12:22:21.000437  3512 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:21.010313  3086 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:21.023845 31819 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 6. Current term is 7. Ops: [6.3-6.3]
I20260709 12:22:21.036368 31823 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:22:21.024876 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 236233us
I20260709 12:22:21.030706 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:20.967724  3336 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:21.036963 31823 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:21.041886 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.029412 31821 rpcz_store.cc:276] 0709 12:22:19.487657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.266322 (+778665us) service_pool.cc:224] Handling call
0709 12:22:20.323651 (+ 57329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:20.339591 (+ 15940us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:20.605120 (+265529us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.730586 (+125466us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:20.762875 (+ 32289us) log.cc:844] Serialized 26 byte log entry
0709 12:22:20.763036 (+   161us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:20.763068 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.763134 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:20.999589 (+236455us) raft_consensus.cc:1685] finished
0709 12:22:20.999613 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:20.999803 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":63267,"prepare.queue_time_us":287749,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":41731,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":131,"thread_start_us":214737,"threads_started":2,"wal-append.queue_time_us":152332}
I20260709 12:22:21.046198 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:21.047933 31671 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:21.013015  3508 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:21.061785 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.050755 31572 catalog_manager.cc:5697] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 7 to 8, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 8 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:21.067202  3223 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.068634 31747 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:21.070591 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:21.072252 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 417578us
I20260709 12:22:21.075667 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2288) took 1491 ms. Trace:
I20260709 12:22:21.076573  3488 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:21.064496 31753 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:21.041190  3454 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.091420 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:21.094775 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.090754 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:21.056483  3407 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:21.099900 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 7 candidate_status { last_received { term: 6 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:21.101231 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:21.102555  3424 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:21.045961  3437 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:21.105086  3437 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:21.107559 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:21.107858 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2260) took 1533 ms. Trace:
I20260709 12:22:21.108244 31752 rpcz_store.cc:276] 0709 12:22:19.574224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.848162 (+273938us) service_pool.cc:224] Handling call
0709 12:22:19.848772 (+   610us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.545128 (+696356us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.107526 (+562398us) spinlock_profiling.cc:243] Waited 666 ms on lock 0x7b54000b86d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:21.107747 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1532631424}
I20260709 12:22:21.109038 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:21.111505 31572 catalog_manager.cc:5697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 2 to 5. New cstate: current_term: 5 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:21.112782  3454 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:21.114543 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:21.113363  3452 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:21.123046  3223 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:21.119031  3342 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.528s	user 0.010s	sys 0.004s
I20260709 12:22:21.124853  3342 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:21.125878  3223 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:21.126672 31744 raft_consensus.cc:1813] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Handling vote request from an unknown peer 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:21.127171 31744 raft_consensus.cc:2368] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12.
I20260709 12:22:21.127326  3283 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:21.127876  3283 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:21.128980  3283 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:21.130183  3283 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:21.131347  3194 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:21.130995 31744 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:21.132978  3370 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:21.135388  3342 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:21.113556 31572 catalog_manager.cc:5697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:21.116158 31753 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:22:21.118943 31673 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:21.143911 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:21.144534 31572 catalog_manager.cc:5697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 3 to 6, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 6 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:21.145300 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:21.134516  3323 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:21.147137  3323 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:21.146451 31572 catalog_manager.cc:5697] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 1 to 7. New cstate: current_term: 7 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:21.140726  3521 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.092s	user 0.003s	sys 0.000s
W20260709 12:22:21.139151 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.149953 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2289) took 1553 ms. Trace:
W20260709 12:22:21.136610 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52463.4us
I20260709 12:22:21.150425 31819 rpcz_store.cc:276] 0709 12:22:19.596602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.700981 (+1104379us) service_pool.cc:224] Handling call
0709 12:22:21.019981 (+319000us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.042429 (+ 22448us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.149726 (+107297us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b54000fc150. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:21.149880 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":620628864}
W20260709 12:22:21.136340  3566 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:22:21.138129 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:21.159795 31827 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:21.120343 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:21.135516 31750 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: [8.9-8.9]
I20260709 12:22:21.137555 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:21.115478  3454 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:21.159210 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2295) took 1157 ms. Trace:
I20260709 12:22:21.137006  3517 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:21.165244  3407 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:21.153671 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:21.079996 31823 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:21.155824  3533 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.721s	user 0.000s	sys 0.004s
I20260709 12:22:21.168622 31822 rpcz_store.cc:276] 0709 12:22:20.001585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.031664 (+1030079us) service_pool.cc:224] Handling call
0709 12:22:21.049839 (+ 18175us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.158698 (+108859us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.158728 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:21.158789 (+    61us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:21.158848 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.158944 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.159125 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:21.168501 31673 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:21.175251 31678 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:21.177767 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:21.039711  3492 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260709 12:22:21.085294 31825 rpcz_store.cc:276] 0709 12:22:19.583849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.289396 (+705547us) service_pool.cc:224] Handling call
0709 12:22:20.325774 (+ 36378us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.382545 (+ 56771us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:20.624753 (+242208us) log.cc:844] Serialized 14 byte log entry
0709 12:22:21.074993 (+450240us) spinlock_profiling.cc:243] Waited 7.5 ms on lock 0x7b18003f7f60. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287
0709 12:22:21.075142 (+   149us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:21.075218 (+    76us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:21.075288 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.075388 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.075579 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78316,"prepare.queue_time_us":320726,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":41478,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":51745,"spinlock_wait_cycles":17252736,"thread_start_us":345104,"threads_started":3,"wal-append.queue_time_us":157384}
I20260709 12:22:21.134585  3358 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:21.203923 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:21.205271 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 156362us
I20260709 12:22:21.206988 31747 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:22:21.206997 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.208168 31671 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:21.208316 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:21.211378 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:21.218015 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2177) took 1133 ms. Trace:
I20260709 12:22:21.218114 31825 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:21.218748 31822 rpcz_store.cc:276] 0709 12:22:20.075689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.198929 (+1123240us) service_pool.cc:224] Handling call
0709 12:22:21.202330 (+  3401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.202543 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.202570 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:21.208954 (+  6384us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:21.209024 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.209104 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.209279 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:21.219602 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2178) took 1143 ms. Trace:
I20260709 12:22:21.193729  3517 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:21.220146 31825 rpcz_store.cc:276] 0709 12:22:20.075976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.205932 (+1129956us) service_pool.cc:224] Handling call
0709 12:22:21.217891 (+ 11959us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.219226 (+  1335us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.219527 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:21.221586 31673 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 6. (term mismatch)
I20260709 12:22:21.213915 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:21.221593 31823 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:21.202844  3342 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:21.223030 31673 pending_rounds.cc:85] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 5
I20260709 12:22:21.229410 31673 pending_rounds.cc:107] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260709 12:22:21.229724 31673 raft_consensus.cc:2889] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:21.217240 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:21.231858 31750 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:21.226886 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:21.234830 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2184) took 1157 ms. Trace:
I20260709 12:22:21.235412 31819 rpcz_store.cc:276] 0709 12:22:20.077765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.230709 (+1152944us) service_pool.cc:224] Handling call
0709 12:22:21.231102 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.234359 (+  3257us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.234399 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:21.234456 (+    57us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:21.234502 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.234583 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.234755 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:21.228041 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:21.240455 31819 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:21.226352  3407 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:21.244311  3407 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:21.245617 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 552) took 2510 ms. Trace:
I20260709 12:22:21.224823  3517 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:21.246079 31818 rpcz_store.cc:276] 0709 12:22:18.735155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.113704 (+1378549us) service_pool.cc:224] Handling call
0709 12:22:20.114273 (+   569us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.759294 (+645021us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.245355 (+486061us) spinlock_profiling.cc:243] Waited 644 ms on lock 0x7b5400197dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:21.245548 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1481800064}
I20260709 12:22:21.248914 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:21.251900 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:21.228718  3342 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.267421 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:21.270491 31819 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:21.272491 31750 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:21.283388 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:21.282858 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:21.288748 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:21.274843  3342 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:21.290549 32342 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:21.302693 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:21.303442 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:21.300549 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:21.304087 32371 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:21.304440 31818 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:21.305613 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:21.306468 31819 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:21.307920  3221 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Leader election won for term 7
I20260709 12:22:21.308519 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:21.309216 31819 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:21.266903  3437 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
I20260709 12:22:21.312275 32377 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:21.312835 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2277) took 2499 ms. Trace:
W20260709 12:22:21.318441 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 84430us
I20260709 12:22:21.318831 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:21.323625 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:21.324280  3407 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.324393  3512 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:21.324965  3407 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:21.316398 31826 rpcz_store.cc:276] 0709 12:22:18.813422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.119044 (+1305622us) service_pool.cc:224] Handling call
0709 12:22:20.119548 (+   504us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:20.123675 (+  4127us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:20.124061 (+   386us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:20.124778 (+   717us) log.cc:844] Serialized 52 byte log entry
0709 12:22:20.284877 (+160099us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:20.284922 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.284990 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:21.118033 (+833043us) spinlock_profiling.cc:243] Waited 34.5 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:21.304934 (+186901us) raft_consensus.cc:1685] finished
0709 12:22:21.304956 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.312755 (+  7799us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1791,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":220,"spinlock_wait_cycles":79441280,"thread_start_us":159351,"threads_started":1,"wal-append.queue_time_us":218253}
I20260709 12:22:21.320796 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:21.327198 31823 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 12:22:21.328980 31823 pending_rounds.cc:85] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 6
I20260709 12:22:21.325266 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:21.330853 31826 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:22:21.314415 31671 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:21.336651 31669 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: [9.5-11.6]
I20260709 12:22:21.329593 31823 pending_rounds.cc:107] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 12:22:21.343616 31823 raft_consensus.cc:2889] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:21.350891 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.342022 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2191) took 1011 ms. Trace:
I20260709 12:22:21.352398 31826 rpcz_store.cc:276] 0709 12:22:20.330917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.327044 (+996127us) service_pool.cc:224] Handling call
0709 12:22:21.328905 (+  1861us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.341722 (+ 12817us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.341954 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:21.354753 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2294) took 1353 ms. Trace:
I20260709 12:22:21.355227 31823 rpcz_store.cc:276] 0709 12:22:20.001189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.912967 (+911778us) service_pool.cc:224] Handling call
0709 12:22:20.990902 (+ 77935us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.354447 (+363545us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.354678 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:21.355986 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:21.346848 32371 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:21.358127 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:21.358927 32375 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:21.354895 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:21.340373  3221 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:21.330221  3492 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260709 12:22:21.362121  3221 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 6.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:21.366822 31672 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:21.368360 31672 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:21.369585 31672 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:21.369912 31672 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:21.370720  3508 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:21.372465 32377 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:22:21.372648  3466 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:22:21.373466  3460 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.797s	user 0.012s	sys 0.001s
I20260709 12:22:21.373962  3466 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:21.376566 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2272) took 1227 ms. Trace:
I20260709 12:22:21.376865 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 546) took 2672 ms. Trace:
I20260709 12:22:21.376998 31745 rpcz_store.cc:276] 0709 12:22:20.149225 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.158888 (+  9663us) service_pool.cc:224] Handling call
0709 12:22:20.159256 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.435441 (+276185us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.376322 (+940881us) spinlock_profiling.cc:243] Waited 184 ms on lock 0x7b54002945d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:21.376506 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":422927232}
I20260709 12:22:21.377388 31820 rpcz_store.cc:276] 0709 12:22:18.704563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.523795 (+819232us) service_pool.cc:224] Handling call
0709 12:22:19.525747 (+  1952us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.046844 (+521097us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.376598 (+1329754us) spinlock_profiling.cc:243] Waited 392 ms on lock 0x7b54004461d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:21.376807 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":902032256}
I20260709 12:22:21.374838  3547 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:21.388204  3547 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:21.387668 31820 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:21.389199  3046 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:21.389670  3046 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:21.390596  3046 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:21.392155  3046 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:21.394984  3542 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:21.395457  3542 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:21.397042 31820 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:21.399870 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:21.401600 31826 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:21.402736 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:21.403539 32376 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:21.403682 31818 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:21.406473 32375 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:21.349269  3437 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:21.415014  3437 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:21.416276  3437 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:21.417910  3437 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:21.418973 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:21.310828  3207 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.060s
I20260709 12:22:21.377647 31672 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:21.340476  3440 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:21.380560  3567 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:21.374023  3460 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13.
W20260709 12:22:21.421615 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 66801.9us
W20260709 12:22:21.378880  3544 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.765s	user 0.005s	sys 0.002s
I20260709 12:22:21.406252 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 767) took 1890 ms. Trace:
W20260709 12:22:21.362039  3550 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.686s	user 0.002s	sys 0.000s
W20260709 12:22:21.379760 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54504.9us
I20260709 12:22:21.351922  3513 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:21.438982 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:21.424238  3358 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:21.446796 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:21.453150  3525 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.058s	user 0.000s	sys 0.005s
W20260709 12:22:21.453677  3525 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.058s	user 0.000s	sys 0.005s
I20260709 12:22:21.428105  2874 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:21.447506 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:21.451169 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:21.435583 32370 rpcz_store.cc:276] 0709 12:22:19.515485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.521591 (+  6106us) service_pool.cc:224] Handling call
0709 12:22:19.522086 (+   495us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:19.983009 (+460923us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:19.983065 (+    56us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:19.983944 (+   879us) log.cc:844] Serialized 52 byte log entry
0709 12:22:20.062430 (+ 78486us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:20.092543 (+ 30113us) log.cc:844] Serialized 14 byte log entry
0709 12:22:20.135776 (+ 43233us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:20.146882 (+ 11106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:21.398923 (+1252041us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b18003ee540. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:22:21.405809 (+  6886us) raft_consensus.cc:1685] finished
0709 12:22:21.405926 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.406075 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4240,"prepare.queue_time_us":26855,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":68,"raft.queue_time_us":98912,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":13291,"spinlock_wait_cycles":33027456,"thread_start_us":538157,"threads_started":2,"wal-append.queue_time_us":110933}
I20260709 12:22:21.464308 31672 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:21.477771 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:21.438225  3440 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:21.426240  3492 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 12:22:21.454537  3460 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:22:21.492722 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.491457 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:21.503974 31823 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 13 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:22:21.462287  2874 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.509526 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:21.528052 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:21.389241  3547 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:21.465828  3207 consensus_queue.cc:1243] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 11.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:22:21.488931  3086 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:21.534085 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:21.536604 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2287) took 1974 ms. Trace:
W20260709 12:22:21.543421 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 93154.4us
I20260709 12:22:21.542131 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:21.524989  2874 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:21.549023 31748 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
W20260709 12:22:21.550009 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:21.536293  3456 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.290s	user 0.005s	sys 0.019s
I20260709 12:22:21.525264  3460 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:21.551612 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:21.560747 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:21.563313 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:21.564554  3407 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:21.564893 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:21.544195  3508 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:21.545940  3398 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:21.560989 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:21.576411 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:21.553436  3456 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.502s	user 0.006s	sys 0.019s
I20260709 12:22:21.569344  3207 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:21.580417  3207 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.580869  3207 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:21.548202  3086 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:21.555305  3221 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:21.535871  3488 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20260709 12:22:21.602242 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:21.566835  3543 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:21.606617 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:21.565318  3407 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.016s
I20260709 12:22:21.569525  3555 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:21.617298  3460 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:21.627576  3578 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:22:21.659087 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:21.570111  3508 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:21.630113 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:21.601876  3470 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:21.593660  3223 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
W20260709 12:22:21.668762 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 114274us
W20260709 12:22:21.649452 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.599789  3221 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:21.664445 31670 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 12: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:21.593770  3547 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:21.655210  3580 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
W20260709 12:22:21.681952 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 115327us
W20260709 12:22:21.659224  3323 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.512s	user 0.014s	sys 0.000s
I20260709 12:22:21.699580 31672 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 12:22:21.700212 31672 pending_rounds.cc:85] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 9
I20260709 12:22:21.700502 31672 pending_rounds.cc:107] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:22:21.700824 31672 raft_consensus.cc:2889] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:21.710788 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.711913 32376 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:21.569480  3440 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:21.743607 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:21.681378  3221 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:21.789574 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:21.792354 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 56635.5us
I20260709 12:22:21.737003 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:21.796659 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:21.832566 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:21.681282  3342 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:21.668910  3543 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.088s
I20260709 12:22:21.797411  3488 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:21.704010  3323 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:21.873224 31826 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:21.760890  3223 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:21.919652 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 557) took 1025 ms. Trace:
I20260709 12:22:21.923333 31821 rpcz_store.cc:276] 0709 12:22:20.894041 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.397987 (+503946us) service_pool.cc:224] Handling call
0709 12:22:21.398295 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.398499 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:21.537089 (+138590us) log.cc:844] Serialized 14 byte log entry
0709 12:22:21.918880 (+381791us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:21.918997 (+   117us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:21.919104 (+   107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.919301 (+   197us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.919555 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":212319,"prepare.run_cpu_time_us":491,"prepare.run_wall_time_us":4813,"thread_start_us":463781,"threads_started":2}
I20260709 12:22:21.929075 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:21.899823  3582 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.296s	user 0.001s	sys 0.000s
W20260709 12:22:21.808456  3585 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.180s	user 0.000s	sys 0.001s
I20260709 12:22:21.691890  3470 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:21.908762 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:21.966516 31670 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:21.703414  3555 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.106s
I20260709 12:22:21.958407 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:21.972684 31823 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:21.960623 31672 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
W20260709 12:22:21.984970 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:21.986011 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.992484 31672 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:21.978847  3336 raft_consensus.cc:2764] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:22:21.922802  3223 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:22.003450 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 558) took 1046 ms. Trace:
I20260709 12:22:21.800559  3086 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:21.999889  2874 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:22.071734 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2273) took 1262 ms. Trace:
W20260709 12:22:21.999567  3587 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:22:21.931092  3488 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:22.086155 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 502231us
I20260709 12:22:22.031275 31820 rpcz_store.cc:276] 0709 12:22:20.957460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.399185 (+441725us) service_pool.cc:224] Handling call
0709 12:22:21.399495 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.399722 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:21.537595 (+137873us) log.cc:844] Serialized 14 byte log entry
0709 12:22:22.003029 (+465434us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:22.003099 (+    70us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:22.003158 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.003235 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.003379 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":372976,"prepare.queue_time_us":246612,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":56,"raft.queue_time_us":350417,"thread_start_us":133102,"threads_started":2}
I20260709 12:22:22.001561  3223 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:22.098708 31745 raft_consensus.cc:2368] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11.
I20260709 12:22:22.101116  2874 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:22.101754 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2270) took 1222 ms. Trace:
W20260709 12:22:22.103184  3517 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.824s	user 0.015s	sys 0.000s
I20260709 12:22:22.103719  3517 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:22.082342  3407 consensus_queue.cc:1243] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 9.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:22:22.109129  3407 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:22:22.111166  3595 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.275s	user 0.001s	sys 0.000s
W20260709 12:22:22.112511  3586 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.304s	user 0.000s	sys 0.001s
I20260709 12:22:22.107468 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2190) took 1776 ms. Trace:
I20260709 12:22:22.114792 31819 rpcz_store.cc:276] 0709 12:22:20.330564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.310011 (+979447us) service_pool.cc:224] Handling call
0709 12:22:21.310378 (+   367us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.312235 (+  1857us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.312262 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:21.323612 (+ 11350us) log.cc:844] Serialized 26 byte log entry
0709 12:22:21.463785 (+140173us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:21.463821 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.463886 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:22.107153 (+643267us) raft_consensus.cc:1685] finished
0709 12:22:22.107181 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.107386 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":139798,"threads_started":1,"wal-append.queue_time_us":223145}
I20260709 12:22:21.909149  3508 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:22.125720 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:22.140246 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.994350  3551 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.474s
I20260709 12:22:22.102360 31675 rpcz_store.cc:276] 0709 12:22:20.710320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.793260 (+ 82940us) service_pool.cc:224] Handling call
0709 12:22:20.793799 (+   539us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:20.794401 (+   602us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:20.794435 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:21.165631 (+371196us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:22:21.166129 (+   498us) log.cc:844] Serialized 26 byte log entry
0709 12:22:21.339888 (+173759us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:21.339925 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.339995 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:21.921365 (+581370us) raft_consensus.cc:1685] finished
0709 12:22:21.921396 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.932315 (+ 10919us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":114,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"spinlock_wait_cycles":6502656,"thread_start_us":173338,"threads_started":1,"wal-append.queue_time_us":382806}
W20260709 12:22:22.181016 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 350285us
I20260709 12:22:22.186777 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1607) took 1561 ms. Trace:
W20260709 12:22:22.187026 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.117411  3517 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:22.187340 31674 rpcz_store.cc:276] 0709 12:22:20.625408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.776556 (+151148us) service_pool.cc:224] Handling call
0709 12:22:20.776899 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:20.785252 (+  8353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:20.785287 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:21.141963 (+356676us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b18003881e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:22:21.409146 (+267183us) spinlock_profiling.cc:243] Waited 2.32 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:22:21.409784 (+   638us) log.cc:844] Serialized 26 byte log entry
0709 12:22:21.535223 (+125439us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:21.535283 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.535370 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:22.186476 (+651106us) raft_consensus.cc:1685] finished
0709 12:22:22.186499 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.186686 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":93,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":10695,"spinlock_wait_cycles":10622592,"thread_start_us":123523,"threads_started":1,"wal-append.queue_time_us":217604}
I20260709 12:22:22.112521  3470 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:22.086257  3552 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.091s
W20260709 12:22:22.234472 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:21.543651 31824 rpcz_store.cc:276] 0709 12:22:19.562446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.277245 (+714799us) service_pool.cc:224] Handling call
0709 12:22:20.307684 (+ 30439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:20.323802 (+ 16118us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:20.573299 (+249497us) log.cc:844] Serialized 14 byte log entry
0709 12:22:21.381099 (+807800us) spinlock_profiling.cc:243] Waited 2.94 ms on lock 0x7b1800384280. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a
0709 12:22:21.488945 (+107846us) log.cc:844] Serialized 14 byte log entry
0709 12:22:21.536197 (+ 47252us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:21.536260 (+    63us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:21.536338 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.536419 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:21.536546 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48061,"prepare.queue_time_us":284269,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":18773,"raft.queue_time_us":53710,"raft.run_cpu_time_us":84,"raft.run_wall_time_us":89,"spinlock_wait_cycles":6822656,"thread_start_us":310575,"threads_started":3,"wal-append.queue_time_us":147009}
W20260709 12:22:22.274483 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:22.255764  3605 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:22:22.293545  3223 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:22.320292  3223 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:22.333784  3608 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:22:22.335165  3323 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:22.337173  3292 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:22.338042  3292 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:22.338948  3555 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:22.339754  3555 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:22.339315  3292 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.4, Last appended by leader: 4, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:22.335224  3223 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:22.343976  3046 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.951s	user 0.016s	sys 0.000s
I20260709 12:22:22.347465  3488 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:22.348610  3207 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.767s	user 0.012s	sys 0.000s
I20260709 12:22:22.349079  3207 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:22.353984 31670 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:22.355347 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:22.356877 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:22.358418  3223 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:22.359242  3223 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:22.359261  3488 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:22:22.363111  3584 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.439s	user 0.000s	sys 0.001s
I20260709 12:22:22.309733  3602 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:22.412572  3402 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:22.413141  3402 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:22.417322  3603 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.417s	user 0.002s	sys 0.015s
W20260709 12:22:22.398139  3437 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.980s	user 0.012s	sys 0.005s
I20260709 12:22:22.419045 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 564) took 1303 ms. Trace:
I20260709 12:22:22.419544 31823 rpcz_store.cc:276] 0709 12:22:21.115850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.606363 (+490513us) service_pool.cc:224] Handling call
0709 12:22:21.650716 (+ 44353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.973278 (+322562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.418767 (+445489us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b540037ad54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:22.418981 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":739845888}
I20260709 12:22:22.414177  3402 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:22.427675  3574 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.893s	user 0.000s	sys 0.016s
I20260709 12:22:22.421593  3402 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:22.429929 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 539) took 1365 ms. Trace:
I20260709 12:22:22.430346 31752 rpcz_store.cc:276] 0709 12:22:21.064373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.109260 (+ 44887us) service_pool.cc:224] Handling call
0709 12:22:21.109632 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.111545 (+  1913us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.111580 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:21.112152 (+   572us) log.cc:844] Serialized 26 byte log entry
0709 12:22:21.345483 (+233331us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:21.345518 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.345578 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:22.426809 (+1081231us) spinlock_profiling.cc:243] Waited 6.07 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:22.429588 (+  2779us) raft_consensus.cc:1685] finished
0709 12:22:22.429618 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.429861 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1215,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"spinlock_wait_cycles":13964160,"thread_start_us":232934,"threads_started":1,"wal-append.queue_time_us":326154}
I20260709 12:22:22.431365 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2282) took 2893 ms. Trace:
I20260709 12:22:22.432279 31827 rpcz_store.cc:276] 0709 12:22:19.538129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.269427 (+731298us) service_pool.cc:224] Handling call
0709 12:22:20.269873 (+   446us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.176056 (+906183us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.431020 (+1254964us) spinlock_profiling.cc:243] Waited 885 ms on lock 0x7b5400379454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:22.431250 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2034978048}
I20260709 12:22:22.072253 31677 rpcz_store.cc:276] 0709 12:22:20.809258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.146293 (+337035us) service_pool.cc:224] Handling call
0709 12:22:21.157056 (+ 10763us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:21.183709 (+ 26653us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.183768 (+    59us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:21.232188 (+ 48420us) log.cc:844] Serialized 52 byte log entry
0709 12:22:21.438467 (+206279us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:21.537141 (+ 98674us) log.cc:844] Serialized 14 byte log entry
0709 12:22:21.537350 (+   209us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.537428 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:22.039376 (+501948us) raft_consensus.cc:1685] finished
0709 12:22:22.039397 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.071657 (+ 32260us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":540734,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"raft.queue_time_us":16945,"raft.run_cpu_time_us":258,"raft.run_wall_time_us":4821,"thread_start_us":205545,"threads_started":2,"wal-append.queue_time_us":234886}
I20260709 12:22:22.435694 31750 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:22.436251 31750 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:22.441395 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:22.441923  3581 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.782s	user 0.005s	sys 0.009s
I20260709 12:22:22.452517 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 568) took 1030 ms. Trace:
W20260709 12:22:22.423274 31748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1720) took 2029 ms (client timeout 1666 ms). Trace:
W20260709 12:22:22.422015 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.332196  3342 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:22.455578 31748 rpcz_store.cc:269] 0709 12:22:20.393597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:20.473708 (+ 80111us) service_pool.cc:224] Handling call
0709 12:22:22.422991 (+1949283us) spinlock_profiling.cc:243] Waited 915 ms on lock 0x7b5400065254. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:22.423155 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2104682368}
I20260709 12:22:22.453135 31669 rpcz_store.cc:276] 0709 12:22:21.422381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.428873 (+  6492us) service_pool.cc:224] Handling call
0709 12:22:21.444111 (+ 15238us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.454707 (+ 10596us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.454751 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:21.468688 (+ 13937us) log.cc:844] Serialized 26 byte log entry
0709 12:22:21.677607 (+208919us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:22.023156 (+345549us) log.cc:844] Serialized 14 byte log entry
0709 12:22:22.023435 (+   279us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.023510 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:22.452202 (+428692us) raft_consensus.cc:1685] finished
0709 12:22:22.452232 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.452430 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8890,"prepare.queue_time_us":495812,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":4033,"raft.queue_time_us":18315,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":11265,"thread_start_us":420087,"threads_started":2,"wal-append.queue_time_us":924288}
I20260709 12:22:22.294601  3597 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.141s
I20260709 12:22:22.437279 31750 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:22.464039 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.770s	user 0.068s	sys 0.106s
W20260709 12:22:22.492136 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.069s	sys 0.106s
I20260709 12:22:22.421574  3207 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:22.466454 31671 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.914s	user 0.000s	sys 0.011s
W20260709 12:22:22.492450 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 862345us
I20260709 12:22:22.395289  3602 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:22.495261 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:22.495625 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:22.495812 31671 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.944s	user 0.007s	sys 0.013s
W20260709 12:22:22.502441 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.014s	sys 0.086s
W20260709 12:22:22.502847 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.014s	sys 0.086s
W20260709 12:22:22.505560 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 134159us
I20260709 12:22:22.512596 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2180) took 2436 ms. Trace:
I20260709 12:22:22.513553 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:22.492924  3513 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:22.513152 31825 rpcz_store.cc:276] 0709 12:22:20.076510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.223289 (+1146779us) service_pool.cc:224] Handling call
0709 12:22:21.223663 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.233281 (+  9618us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.233330 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:21.240525 (+  7195us) log.cc:844] Serialized 26 byte log entry
0709 12:22:21.463476 (+222951us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:21.463538 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.463723 (+   185us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:22.362516 (+898793us) spinlock_profiling.cc:243] Waited 23 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:22.512196 (+149680us) raft_consensus.cc:1685] finished
0709 12:22:22.512225 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.512518 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":52834048,"thread_start_us":183331,"threads_started":1,"wal-append.queue_time_us":355607}
I20260709 12:22:22.497422  3267 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:22.520861 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 562500us
I20260709 12:22:22.502131  3528 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:22.522779 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:22.523402 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:22.524187 32376 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:22.524293 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:22.525143 32374 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 11 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:22:22.526022 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:22.526230 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:22.526711 32377 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:22.526944 32370 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:22.529860 31750 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:22.531657 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:22.531657 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:22.538602 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:22.544653 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.523317  3513 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:22.528864  3528 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:22.562307 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:22.575507 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 43920.6us
I20260709 12:22:22.576717 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:22.522675  3611 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:22.528591  3466 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:22.538206  3223 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:22.583758 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.596027 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:22.553917  3207 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:22.602069 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:22.602978 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:22.603397 31751 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:22.602442 31752 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:22.579136  3470 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:22.609584 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:22.580214  3600 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:22.607769 31746 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 4. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:22:22.615595 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:22.616482 31744 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260709 12:22:22.616935 32376 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:22.618513 31745 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:22.586374  3517 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:22.586021  3555 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:22.531831  3267 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:22.626060 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 94503.6us
I20260709 12:22:22.627274 32377 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 9 index: 8. (term mismatch)
I20260709 12:22:22.627712 32377 pending_rounds.cc:85] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 7
I20260709 12:22:22.627993 32377 pending_rounds.cc:107] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 8.8
I20260709 12:22:22.628253 32377 raft_consensus.cc:2889] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:22.629784  3267 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:22.622581 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44586.4us
I20260709 12:22:22.630739  3600 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:22.631701 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:22.625483 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:22.633621  3600 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:22.599356  3551 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:22.626053 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:22.627722 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:22.634061 31746 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:22.630537  3267 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:22.638897  3267 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:22.637444  3615 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:22.622658  3470 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:22.643743  3470 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:22.645776 31748 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:22:22.646194  3604 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:22.647467 31745 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:22.647346 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:22.648942 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:22.649642 31744 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:22.651511 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:22.650833 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.652366 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:22.658084 32370 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:22.679286 31744 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
W20260709 12:22:22.683890 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56251.4us
I20260709 12:22:22.685788 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:22.622155  3611 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:22.650539  3267 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:22.695968 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.698060  3470 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:22.698309 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:22.698577  3470 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:22:22.684556  3221 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.871s	user 0.015s	sys 0.001s
I20260709 12:22:22.603422  3602 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:22.734159 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.622846  3207 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:22.745787 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:22.744863 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 7 candidate_status { last_received { term: 5 index: 2 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:22.764693 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1613) took 1769 ms. Trace:
W20260709 12:22:22.765942 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61844.1us
I20260709 12:22:22.767338 31717 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:22.766332 31671 rpcz_store.cc:276] 0709 12:22:20.995174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.338358 (+343184us) service_pool.cc:224] Handling call
0709 12:22:21.338776 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:21.355166 (+ 16390us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:21.459863 (+104697us) log.cc:844] Serialized 14 byte log entry
0709 12:22:22.764099 (+1304236us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:22.764204 (+   105us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:22.764287 (+    83us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.764401 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.764603 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":488721,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":56,"raft.queue_time_us":142228,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":61524,"thread_start_us":1241995,"threads_started":3,"wal-append.queue_time_us":1089085}
I20260709 12:22:22.775530 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:22.775748 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:22.700227  3342 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:22.658725  3606 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:22.784941 32372 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260709 12:22:22.683179  3551 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:22.699137  3470 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:22.805241 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:22.695865  3517 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:22.806694 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 230047us
I20260709 12:22:22.779135  3437 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:22.795140  3602 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:22.810252 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:22.772584  3460 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.028s
I20260709 12:22:22.793305  3606 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:22.800935  3292 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:22.815987 32376 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:22.696616  3611 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:22.819382 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:22.807817 31643 connection.cc:256] RPC call timeout handler was delayed by -2.195889963999889s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
W20260709 12:22:22.823205 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 49887.9us
W20260709 12:22:22.827157 31643 connection.cc:256] RPC call timeout handler was delayed by -1.9006899639998664s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18280000000000002s for a response
W20260709 12:22:22.827906 31643 connection.cc:256] RPC call timeout handler was delayed by -1.8718899639998199s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18600000000000003s for a response
W20260709 12:22:22.828532 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:22.828554 31643 connection.cc:256] RPC call timeout handler was delayed by -1.7446786999998949s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19340000000000002s for a response
W20260709 12:22:22.829072 31643 connection.cc:256] RPC call timeout handler was delayed by -1.7446786999998949s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19340000000000002s for a response
W20260709 12:22:22.829351 31643 connection.cc:256] RPC call timeout handler was delayed by -1.6603899639999327s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20950000000000002s for a response
W20260709 12:22:22.829658 31643 connection.cc:256] RPC call timeout handler was delayed by -1.6603899639999327s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20950000000000002s for a response
W20260709 12:22:22.830044 31643 connection.cc:256] RPC call timeout handler was delayed by -1.62168996399987s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2138s for a response
W20260709 12:22:22.832216 31643 connection.cc:256] RPC call timeout handler was delayed by -1.5532899639997595s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
W20260709 12:22:22.832582 31643 connection.cc:256] RPC call timeout handler was delayed by -1.5532899639997595s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2214s for a response
W20260709 12:22:22.832998 31643 connection.cc:256] RPC call timeout handler was delayed by -1.5217899639997086s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22490000000000002s for a response
W20260709 12:22:22.833374 31643 connection.cc:256] RPC call timeout handler was delayed by -1.492989963999662s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22810000000000002s for a response
W20260709 12:22:22.833767 31643 connection.cc:256] RPC call timeout handler was delayed by -1.3210899639998388s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2472s for a response
W20260709 12:22:22.834084 31643 connection.cc:256] RPC call timeout handler was delayed by -1.3021816059999765s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1763s for a response
I20260709 12:22:22.791559  3221 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:22.834482 31643 connection.cc:256] RPC call timeout handler was delayed by -1.2742899639997631s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2524s for a response
I20260709 12:22:22.821818  3606 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:22.843191 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2281) took 1967 ms. Trace:
I20260709 12:22:22.847368 32377 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:22.842804  3517 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:22.846235 31643 connection.cc:256] RPC call timeout handler was delayed by -1.2301899639996918s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.25730000000000003s for a response
I20260709 12:22:22.821753  3086 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:22.849728  3342 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:22.851223  3542 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:22.852382  3542 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:22.849776  3086 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:22.855906  3221 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:22.855840  3086 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 8.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:22.858122 31751 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:22:22.822319  3292 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:22.860991  3551 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.10, Last appended by leader: 10, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:22.884188 31750 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:22.885231 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 551) took 1573 ms. Trace:
I20260709 12:22:22.862200  3528 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:22.867403  3398 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:22.868541  3598 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:22.870003 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:22.913932  3598 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:22.914207 31824 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:22.915462 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:22.915539  3598 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:22.916740 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:22.873080 31819 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:22:22.847298 31673 rpcz_store.cc:276] 0709 12:22:20.875382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.278482 (+403100us) service_pool.cc:224] Handling call
0709 12:22:21.291647 (+ 13165us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:21.319330 (+ 27683us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:21.319365 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:22.454291 (+1134926us) spinlock_profiling.cc:243] Waited 6.93 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:22:22.460161 (+  5870us) log.cc:844] Serialized 52 byte log entry
0709 12:22:22.785665 (+325504us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:22.785710 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.785790 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:22.842747 (+ 56957us) raft_consensus.cc:1685] finished
0709 12:22:22.842782 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:22.843101 (+   319us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6230,"raft.queue_time_us":14391,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":14131,"spinlock_wait_cycles":16096384,"thread_start_us":266544,"threads_started":1,"wal-append.queue_time_us":334667}
W20260709 12:22:22.878895 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 761818us
I20260709 12:22:22.879942 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:22.918382 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:22.918967 31819 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:22:22.918817 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:22.885670 31750 rpcz_store.cc:276] 0709 12:22:21.311396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.319257 (+  7861us) service_pool.cc:224] Handling call
0709 12:22:21.319565 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:22.435548 (+1115983us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54003b9aa0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:22:22.884833 (+449285us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:22.885148 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":447620992}
I20260709 12:22:22.921052 31824 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:22.921954  3488 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:22.894646  3508 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:22.922827  3488 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:22.923228 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.899469 31825 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:22.923399  3488 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:22.924595 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:22.925772 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:22.926537 31825 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:22.900419 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:22.926932 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:22.901902  3464 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260709 12:22:22.883930  3454 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:22.884320  3292 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:22.844303  3624 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 12:22:22.912093  3460 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:22.910239 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:22.913293 31677 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 7 index: 8. (term mismatch)
I20260709 12:22:22.944517 31677 pending_rounds.cc:85] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 7
I20260709 12:22:22.944819 31677 pending_rounds.cc:107] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 6.8
I20260709 12:22:22.945066 31677 raft_consensus.cc:2889] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:22.881979 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:22.892693 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:22.927261 31823 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:22.922857 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:22.946175 31827 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:22.901729 31826 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:22.929796 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:22.931691 32374 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:22.935913 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:22.937291 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:22.943327 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:22.947600 31825 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:22.947698 31818 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:22.948676 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:22.917333  3598 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:22.887336  3567 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:22.955216 31818 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 8: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:22.965626 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:22.937839  3525 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:22.968272 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:22.964923  3567 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:22.988410 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78232us
I20260709 12:22:22.989367  3567 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:22.964885  3528 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:22.991748 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:22.993911 31827 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
W20260709 12:22:22.997066  3579 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.446s	user 0.015s	sys 0.023s
I20260709 12:22:23.003064 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2179) took 2926 ms. Trace:
I20260709 12:22:23.003500 31822 rpcz_store.cc:276] 0709 12:22:20.076247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.220105 (+1143858us) service_pool.cc:224] Handling call
0709 12:22:21.225389 (+  5284us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:21.225879 (+   490us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:21.225938 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:21.237086 (+ 11148us) log.cc:844] Serialized 26 byte log entry
0709 12:22:21.360999 (+123913us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:21.361050 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:21.361133 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:22.996330 (+1635197us) spinlock_profiling.cc:243] Waited 1.77 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:23.002699 (+  6369us) raft_consensus.cc:1685] finished
0709 12:22:23.002728 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:23.002983 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":122,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":14592,"spinlock_wait_cycles":4066688,"thread_start_us":83890,"threads_started":1,"wal-append.queue_time_us":232491}
W20260709 12:22:23.004990 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.009258 31818 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:23.011116 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.011210 31744 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:23.013670  3611 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:22.969113  3398 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260709 12:22:23.017469  3398 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:23.018105  3398 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:23.018733  3424 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:23.023859 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53679.7us
I20260709 12:22:23.019450 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:23.022058 31642 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.701s	user 0.007s	sys 0.024s
W20260709 12:22:23.056401 31642 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.747s	user 0.007s	sys 0.024s
W20260709 12:22:23.056751 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 756136us
I20260709 12:22:22.973148 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:23.058166 31826 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:23.026381 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:23.029963  3615 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:22.979369  3221 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:23.032439 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 631) took 1785 ms. Trace:
I20260709 12:22:23.061823 31821 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:23.035111  3547 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:23.061460 32372 rpcz_store.cc:276] 0709 12:22:21.247433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:21.267322 (+ 19889us) service_pool.cc:224] Handling call
0709 12:22:22.295229 (+1027907us) spinlock_profiling.cc:243] Waited 5.54 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:22:22.299445 (+  4216us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:22.797343 (+497898us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.032135 (+234792us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b540055da54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:23.032368 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1125716608}
I20260709 12:22:23.036408  3488 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:23.041386  3086 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:22.966259  3460 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:22.940546  3454 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.043012  3207 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:23.087865 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:23.089368 31744 raft_consensus.cc:2410] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 11 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
W20260709 12:22:23.091041 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 71822.4us
I20260709 12:22:23.092042  3460 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:23.094198  3454 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:23.096011 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:23.097306 31826 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 9: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:23.099514 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:23.101509  3513 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.518s	user 0.012s	sys 0.000s
I20260709 12:22:23.106802 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:23.107950  3606 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:23.108558 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:23.109849 31820 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:23.066505  3528 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:23.066883  3547 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:23.115104 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.113916  3547 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:23.119004  3424 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:23.119464  3513 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:23.119860  3424 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:23.120756 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:23.121567 31826 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:23.051872 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:23.060567 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.063191 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:23.126228 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.061461 31823 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:23.070423 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:23.070760 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:23.073979  3402 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.644s	user 0.013s	sys 0.000s
I20260709 12:22:23.081898  3086 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:23.110308  3606 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:23.137055 31752 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:23.094072 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:23.129065 31821 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:23.137804 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:23.129698  3620 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.138849  3620 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:23.140105  3398 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.141523  3398 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:23.130131  3492 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.130825 31749 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:23.131273  3424 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:23.134076 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:23.134562 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.135667 32378 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:23.137503  3086 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.143225 31820 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:23.137555 31825 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:23.111367 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:23.143785  3492 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:23.146436 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 94737.7us
I20260709 12:22:23.142203  3620 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:22.981312  3470 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:23.188925 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.153654  3323 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:23.154531  3552 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:23.187642  3086 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:23.172387  3621 raft_consensus.cc:2764] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:22:23.181329  3456 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:23.201324 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 107352us
I20260709 12:22:23.205951 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:23.186923  3267 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 6.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.017s
I20260709 12:22:23.205391  3513 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:23.225063 31824 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:23.228173  3470 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:23.240231 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.242377 32342 leader_election.cc:336] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.855s (SENT)
I20260709 12:22:23.243185 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:23.245183  3552 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:23.246902 31822 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 14 index: 11. (term mismatch)
I20260709 12:22:23.247422 31822 pending_rounds.cc:85] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 10
I20260709 12:22:23.247685 31822 pending_rounds.cc:107] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 12:22:23.247977 31822 raft_consensus.cc:2889] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:23.251191  3621 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:23.252519  3621 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:23.193886  3620 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:23.253937  3620 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:23.254483  3620 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:23.200865  3292 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.266100  3292 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:23.268335  3547 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:23.271117 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 65301.8us
I20260709 12:22:23.272688 31791 leader_election.cc:400] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12.
I20260709 12:22:23.273093 31791 leader_election.cc:403] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:23.274971  3567 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 12
W20260709 12:22:23.282866 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.298646 31819 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
W20260709 12:22:23.307363 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.308480 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.320848 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46551.4us
I20260709 12:22:23.307777  3398 raft_consensus.cc:2764] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
W20260709 12:22:23.340811 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.342574 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 412828us
I20260709 12:22:23.343031 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:23.340153 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 404251us
I20260709 12:22:23.328996  3492 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:23.354311  3470 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:23.364030 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:23.379267 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.378702 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:23.387774 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:23.390661 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:23.405467 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:23.414726 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.403918 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.422693 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.424206 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.425134 31673 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:23.427071 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.428208 31673 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:22:23.428285 31791 connection.cc:256] RPC call timeout handler was delayed by -1.6994623360001242s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
W20260709 12:22:23.428822 31791 connection.cc:256] RPC call timeout handler was delayed by -1.6391623360000267s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1583s for a response
W20260709 12:22:23.429136 31791 connection.cc:256] RPC call timeout handler was delayed by -1.5536623359998885s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1678s for a response
W20260709 12:22:23.429448 31791 connection.cc:256] RPC call timeout handler was delayed by -1.5014623360002588s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1736s for a response
I20260709 12:22:23.372685  3567 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12.
W20260709 12:22:23.430929 31791 connection.cc:256] RPC call timeout handler was delayed by -1.4924623360002442s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1746s for a response
I20260709 12:22:23.431540 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:23.435590 31669 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:23.436712 31791 connection.cc:256] RPC call timeout handler was delayed by -1.4123623360001147s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1835s for a response
W20260709 12:22:23.437287 31791 connection.cc:256] RPC call timeout handler was delayed by -1.4078623360001075s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18400000000000002s for a response
W20260709 12:22:23.437642 31791 connection.cc:256] RPC call timeout handler was delayed by -1.3466623360000085s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1908s for a response
W20260709 12:22:23.439428 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.440301 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:23.440585  3606 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:23.441082  3606 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:23.431783 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:23.432502 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44816.9us
W20260709 12:22:23.437992 31791 connection.cc:256] RPC call timeout handler was delayed by -1.3232623359999707s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19340000000000002s for a response
I20260709 12:22:23.373730  3598 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:23.445014 31791 connection.cc:256] RPC call timeout handler was delayed by -1.1810623360001955s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20920000000000002s for a response
W20260709 12:22:23.445441 31791 connection.cc:256] RPC call timeout handler was delayed by -1.1594623360001606s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
W20260709 12:22:23.446650 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0829623360000369s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22010000000000002s for a response
I20260709 12:22:23.447463 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:23.447767 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:23.449321 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:23.449483 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:23.450486  3456 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:23.450771 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:23.451324 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.451952 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:23.452461 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:23.452872 31753 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:23.453430  3454 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:23.368826  3398 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.066s
I20260709 12:22:23.436637  3470 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.020s
I20260709 12:22:23.456715 32378 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:22:23.435886  3563 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.854s	user 0.017s	sys 0.029s
I20260709 12:22:23.430166  3492 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:22:23.458727  3563 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.877s	user 0.018s	sys 0.029s
I20260709 12:22:23.459072  3492 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:23.459980  3492 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.456799 32372 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:22:23.462683  3598 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:23.464828  3598 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:23.465840 31749 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:23.429412  3323 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:23.475915  3323 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.483585 31752 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:22:23.485368 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:23.436812  3513 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:23.456404 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:23.490602 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:23.490914 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59186.2us
I20260709 12:22:23.491330 31748 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:23.491969 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:23.496258 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62722.5us
I20260709 12:22:23.497215 31669 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:23.499440 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.505532 31673 raft_consensus.cc:2410] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:22:23.505470 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.508831 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:23.432227  3464 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:23.512851 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:23.510686 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:23.513698 31672 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:23.514143 31749 raft_consensus.cc:2410] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:22:23.515751 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:23.516578 31753 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:23.516983 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.515753 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:23.517856 31744 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:23.501609  3398 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:23.518462  3323 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:23.530730 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:23.531708  3323 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:22:23.518827 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.532716 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:23.513016  3464 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:23.534075  3464 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:23.535636  3508 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:23.536164  3508 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:23.512449  3488 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:23.536713  3488 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:23.537147  3488 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:23.519907  3207 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:23.538959 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.539366  3207 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:23.539793 31674 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:23.539861  3207 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:23.519747 31746 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:22:23.526821 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:23.542657 31752 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:23.545234 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:23.548930 31751 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:23.517722 31676 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:23.553742 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.515144 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:23.552256 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:23.555600  3437 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:23.556859  3437 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:23.557281  3437 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:23.558054  3543 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:23.560817  3402 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:23.561411  3402 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:23.442024  3606 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.562414 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:23.556221 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.519636  3398 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:23.565407 31818 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.011s	sys 0.025s
W20260709 12:22:23.568991 31818 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.011s	sys 0.026s
W20260709 12:22:23.564271  3601 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.022s	sys 0.024s
W20260709 12:22:23.580828 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 384719us
W20260709 12:22:23.582294 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 471107us
I20260709 12:22:23.569891  3398 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:23.584903 31753 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:23.591305 31753 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:23.584973 31746 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:23.592206 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:23.589767 31752 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260709 12:22:23.592599 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:23.585498  3601 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.531s	user 0.022s	sys 0.025s
I20260709 12:22:23.594789  3402 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.599946 31753 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:23.601058 31753 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:23.537163  3508 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:23.602341  3508 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:23.577396 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:23.618860  3223 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:23.619647 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:23.620442  3528 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Stepping down as leader of term 6
I20260709 12:22:23.620991  3528 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:23.622509  3488 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.617666 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:23.589000  3600 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:23.600019  3567 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.621966  3528 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.3, Last appended by leader: 3, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.629277  3358 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:22:23.684938 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.715299 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 67757.7us
I20260709 12:22:23.636569  3086 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:23.725570 31753 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 9. Preceding OpId from leader: term: 10 index: 9. (term mismatch)
I20260709 12:22:23.726336 31753 pending_rounds.cc:85] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 8
I20260709 12:22:23.636916  3600 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:23.627444  3292 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:22:23.733397  3528 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 6 FOLLOWER]: Advancing to term 7
W20260709 12:22:23.735798 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.736755 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.753764 31670 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
W20260709 12:22:23.738793 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.744148  3456 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:23.747800 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.748728 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 124243us
I20260709 12:22:23.726913 31753 pending_rounds.cc:107] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 9.9
I20260709 12:22:23.761647 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:23.761665 31753 raft_consensus.cc:2889] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:23.755179 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.766286 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:23.767925 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.642954  3567 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.772145 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.774341 31675 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:23.772162 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.774900 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.775177 31674 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:23.775571 31672 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:23.777515 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:23.778445 31678 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:23.751366  3086 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:23.782086 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.782233 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:23.788270 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.789105 31672 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:23.791586 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:23.753086  3543 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:23.717410  3464 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.793118 31673 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:23.762873  3547 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:23.802735  3645 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:23.803274  3645 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:23.803009  3600 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.804250  3645 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:23.805323  3645 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:23.783537  3643 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:23.806990  3643 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:23.807909  3643 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:23.809234  3643 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:23.707453 31745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:22:23.795418 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:23.813304 31676 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:23.727159  3620 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:23.760070  3456 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:23.794309 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.812801 31745 raft_consensus.cc:606] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:23.820305 31745 raft_consensus.cc:614] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:23.797391 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.821281 31670 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
W20260709 12:22:23.815843 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.831296 31747 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 11: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:22:23.831425 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.837471 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 351112us
I20260709 12:22:23.802280  3547 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:23.845388 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.845645 31671 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: []
W20260709 12:22:23.846789 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.822785  3456 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:23.812752  3513 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
W20260709 12:22:23.903915 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.904353 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:23.964408 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 449345us
I20260709 12:22:23.965407 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:23.966327 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:23.823180  3543 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:23.977988 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 361388us
I20260709 12:22:23.984823 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:23.985829 31670 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:23.832005  3646 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.080s
I20260709 12:22:23.844328  3508 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.126s
I20260709 12:22:23.900434  3223 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:23.841408  3642 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.143s
W20260709 12:22:23.964682  3641 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
W20260709 12:22:24.000792  3640 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 12:22:24.007628 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:23.984012  3610 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:23.994845  3563 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:23.995823  3488 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:24.016633  3644 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
W20260709 12:22:24.031054 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.034837 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:24.038360 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:24.039084 31670 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:23.996218  3086 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:24.055158 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 293870us
W20260709 12:22:24.058176 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 440597us
W20260709 12:22:24.078549 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.086103 31671 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:24.036036  3650 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:24.087992 31670 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:24.088685 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:24.091957 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:24.093550 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:24.021116  3610 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:24.096036 31670 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:22:24.096714 32371 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
I20260709 12:22:24.096781 32375 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:24.097524 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:24.104460 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 569) took 1160 ms. Trace:
I20260709 12:22:24.089133 31671 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:24.025425  3563 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:24.106410 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 574) took 1141 ms. Trace:
I20260709 12:22:24.087692  3567 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:24.108285 31826 rpcz_store.cc:276] 0709 12:22:22.965326 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.123833 (+158507us) service_pool.cc:224] Handling call
0709 12:22:23.124179 (+   346us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:23.143718 (+ 19539us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:23.143754 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:23.144314 (+   560us) log.cc:844] Serialized 26 byte log entry
0709 12:22:23.624851 (+480537us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:23.624897 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.624975 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:24.106203 (+481228us) raft_consensus.cc:1685] finished
0709 12:22:24.106227 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:24.106352 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":28811,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":19875,"thread_start_us":480131,"threads_started":1,"wal-append.queue_time_us":652221}
I20260709 12:22:24.104872 31818 rpcz_store.cc:276] 0709 12:22:22.943991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.019994 (+ 76003us) service_pool.cc:224] Handling call
0709 12:22:23.020368 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:23.020771 (+   403us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:23.020803 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:23.021322 (+   519us) log.cc:844] Serialized 26 byte log entry
0709 12:22:23.571390 (+550068us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:23.571454 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.571561 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:24.104260 (+532699us) raft_consensus.cc:1685] finished
0709 12:22:24.104282 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:24.104403 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1157,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":8255,"thread_start_us":549607,"threads_started":1,"wal-append.queue_time_us":776887}
I20260709 12:22:24.090219  3464 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:24.117463 32372 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:24.106335 31671 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:24.095328  3662 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:24.100971  3454 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.101586  3655 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:22:24.105408 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 104185us
W20260709 12:22:24.117305  3651 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.103s	user 0.000s	sys 0.001s
I20260709 12:22:24.130102 31669 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:24.099990  3650 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:24.122484 31671 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:24.144739 31671 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:24.134410 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 169098us
I20260709 12:22:24.130012 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 580) took 1008 ms. Trace:
I20260709 12:22:24.123407 32377 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
W20260709 12:22:24.122499  3639 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.121s	user 0.001s	sys 0.002s
I20260709 12:22:24.123358  3620 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:24.126736  3662 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:24.095036  3223 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:24.187525 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.163369  3650 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:24.167182  3649 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
W20260709 12:22:24.168072  3652 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
I20260709 12:22:24.170955  3543 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:24.173077  3563 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:24.178198  3207 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.638s	user 0.012s	sys 0.000s
I20260709 12:22:24.178985  3606 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:24.179772  3610 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:24.144709  3615 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:24.188498 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 8 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:24.192428 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:24.204638  3667 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.064s
I20260709 12:22:25.327831 31819 raft_consensus.cc:2410] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 10 because replica has last-logged OpId of term: 9 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:22:25.328447 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2221) took 1136 ms. Trace:
I20260709 12:22:24.230801  3673 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
W20260709 12:22:24.728781 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.796985  3666 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:25.330006  3666 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:25.331140 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:24.806087 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.709s	user 0.006s	sys 0.037s
W20260709 12:22:24.812247 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.333446 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.236s	user 0.008s	sys 0.037s
W20260709 12:22:25.334134 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 1.24545e+06us
I20260709 12:22:25.334369  3697 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.334614 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 649) took 1236 ms. Trace:
I20260709 12:22:25.335067 32372 rpcz_store.cc:276] 0709 12:22:24.098708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.098836 (+   128us) service_pool.cc:224] Handling call
0709 12:22:24.107424 (+  8588us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.117857 (+ 10433us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.334347 (+1216490us) spinlock_profiling.cc:243] Waited 9.62 ms on lock 0x7b540074e854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:25.334548 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":22133504}
I20260709 12:22:25.336520 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:25.337312 31821 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:25.337098  3492 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:25.338181 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:25.337863  3492 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:24.852239 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.847028 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:25.344144 31826 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:25.344807 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2219) took 1155 ms. Trace:
I20260709 12:22:25.345165 31826 rpcz_store.cc:276] 0709 12:22:24.189748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.189902 (+   154us) service_pool.cc:224] Handling call
0709 12:22:25.344751 (+1154849us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:24.895411  3695 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:24.941915 31671 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.796s	user 0.013s	sys 0.000s
I20260709 12:22:25.346395 31671 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 9. (term mismatch)
I20260709 12:22:25.346854 31671 pending_rounds.cc:85] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 8
I20260709 12:22:25.347136 31671 pending_rounds.cc:107] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 10.9
I20260709 12:22:25.347473 31671 raft_consensus.cc:2889] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:25.348398 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2321) took 1278 ms. Trace:
W20260709 12:22:24.946326 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:24.979314 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.349856 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.349993 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.350387 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:24.968111 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 563) took 1392 ms. Trace:
I20260709 12:22:24.982815 31675 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 5. (term mismatch)
W20260709 12:22:24.986263 31676 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.167s	user 0.013s	sys 0.004s
I20260709 12:22:25.351886 31675 pending_rounds.cc:85] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 4
I20260709 12:22:25.352284 31675 pending_rounds.cc:107] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 12:22:25.352876 31675 raft_consensus.cc:2889] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:24.146906 31677 rpcz_store.cc:276] 0709 12:22:23.121669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.122034 (+   365us) service_pool.cc:224] Handling call
0709 12:22:23.122448 (+   414us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:23.123278 (+   830us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:23.123306 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:23.123937 (+   631us) log.cc:844] Serialized 52 byte log entry
0709 12:22:23.489508 (+365571us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:23.832828 (+343320us) log.cc:844] Serialized 14 byte log entry
0709 12:22:24.128042 (+295214us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.128176 (+   134us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:24.128243 (+    67us) raft_consensus.cc:1685] finished
0709 12:22:24.128262 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:24.129931 (+  1669us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":515332,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":323884,"raft.run_cpu_time_us":320,"raft.run_wall_time_us":37494,"thread_start_us":986757,"threads_started":3,"wal-append.queue_time_us":498753}
I20260709 12:22:24.998098 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2295) took 1244 ms. Trace:
I20260709 12:22:25.351258 31748 rpcz_store.cc:276] 0709 12:22:23.577954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.578130 (+   176us) service_pool.cc:224] Handling call
0709 12:22:23.590157 (+ 12027us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:23.590512 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:23.590556 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:23.593973 (+  3417us) log.cc:844] Serialized 26 byte log entry
0709 12:22:23.966787 (+372814us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:24.135618 (+168831us) log.cc:844] Serialized 14 byte log entry
0709 12:22:24.967705 (+832087us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.967796 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:24.967847 (+    51us) raft_consensus.cc:1685] finished
0709 12:22:24.967862 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:24.968025 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":831766,"prepare.queue_time_us":33968,"prepare.run_cpu_time_us":55,"prepare.run_wall_time_us":2500,"raft.queue_time_us":831968,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":106,"thread_start_us":465455,"threads_started":2,"wal-append.queue_time_us":493636}
I20260709 12:22:25.354171 31669 rpcz_store.cc:276] 0709 12:22:23.754208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.754839 (+   631us) service_pool.cc:224] Handling call
0709 12:22:24.129901 (+375062us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.143287 (+ 13386us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:24.997846 (+854559us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b54002544d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:24.998025 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":822879488}
W20260709 12:22:25.009784 31670 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.857s	user 0.041s	sys 0.028s
W20260709 12:22:25.355603 31670 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.203s	user 0.042s	sys 0.030s
I20260709 12:22:24.219985  3207 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:24.191900  3398 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.593s	user 0.008s	sys 0.004s
I20260709 12:22:24.978643  3402 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.491066  3086 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:25.491736  3086 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.494280  3086 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:24.509282  3456 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.544s	user 0.009s	sys 0.002s
I20260709 12:22:25.543891  3456 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.546526  3456 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:24.256255  3437 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.698s	user 0.012s	sys 0.000s
I20260709 12:22:25.550572  3437 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:25.553297  3437 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:24.498710  3552 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:24.302132  3528 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.542s	user 0.012s	sys 0.000s
I20260709 12:22:24.800343  3547 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:25.565148 31673 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260709 12:22:24.200075  3600 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:22:24.761500  3606 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.540s	user 0.005s	sys 0.007s
I20260709 12:22:25.569250  3600 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:25.570168 31750 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:24.194243  3642 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:24.189124  3655 raft_consensus.cc:476] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Not starting pre-election -- already a leader
I20260709 12:22:25.577219 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2313) took 1803 ms. Trace:
I20260709 12:22:25.577682 31673 rpcz_store.cc:276] 0709 12:22:23.773439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.819628 (+ 46189us) service_pool.cc:224] Handling call
0709 12:22:23.822552 (+  2924us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.565559 (+1743007us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.576930 (+ 11371us) spinlock_profiling.cc:243] Waited 1.74 s on lock 0x7b540017d9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:25.577131 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4006718720}
W20260709 12:22:25.087208  3647 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.245s	user 0.003s	sys 0.011s
I20260709 12:22:24.186779  3662 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
W20260709 12:22:25.171789 31674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.084s	user 0.010s	sys 0.014s
W20260709 12:22:25.180343 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.022s	user 0.010s	sys 0.010s
W20260709 12:22:25.213358 31678 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.420s	user 0.017s	sys 0.000s
I20260709 12:22:25.588384 31678 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
W20260709 12:22:25.588435 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.433s	user 0.011s	sys 0.010s
I20260709 12:22:25.589233  3620 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:25.589358 31678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2300) took 1832 ms (client timeout 1603 ms). Trace:
W20260709 12:22:25.589946 31678 rpcz_store.cc:269] 0709 12:22:23.756938 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.757065 (+   127us) service_pool.cc:224] Handling call
0709 12:22:25.589252 (+1832187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:25.218436 31818 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 13 because replica has last-logged OpId of term: 10 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 4.
I20260709 12:22:25.591152 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2218) took 1403 ms. Trace:
I20260709 12:22:25.591601 31818 rpcz_store.cc:276] 0709 12:22:24.187995 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.188168 (+   173us) service_pool.cc:224] Handling call
0709 12:22:25.591059 (+1402891us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:25.593068  3323 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.593639  3323 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:25.234443 31672 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.193s	user 0.012s	sys 0.024s
W20260709 12:22:25.594648 31672 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.553s	user 0.014s	sys 0.024s
I20260709 12:22:25.595422 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 572) took 2640 ms. Trace:
W20260709 12:22:25.588385 31674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.500s	user 0.010s	sys 0.014s
I20260709 12:22:25.280134  3693 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.285181  3691 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:25.306514 31564 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 1.818s	user 0.012s	sys 0.006s
I20260709 12:22:25.311996 31791 leader_election.cc:400] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12.
I20260709 12:22:25.597980  3691 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:25.599160 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 287379us
I20260709 12:22:25.599953 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:25.600818  3691 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:25.328778 31819 rpcz_store.cc:276] 0709 12:22:24.191991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.192118 (+   127us) service_pool.cc:224] Handling call
0709 12:22:25.328375 (+1136257us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:25.330900  3666 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:25.601189 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:25.602042  3666 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
W20260709 12:22:24.803944 31749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.009s	sys 0.009s
I20260709 12:22:25.597954  3693 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:24.812436  3689 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.602922  3456 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:25.603596  3456 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:25.603451  3689 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:25.604665  3693 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:25.604564  3456 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:25.605540  3689 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:25.339764  3621 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:25.352248 31676 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
W20260709 12:22:24.995134 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.610898 31819 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:24.839557 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:25.348794 31671 rpcz_store.cc:276] 0709 12:22:24.070142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.088631 (+ 18489us) service_pool.cc:224] Handling call
0709 12:22:24.088991 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.348028 (+1259037us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.348325 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:25.362998 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:25.363984 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:25.335037  3697 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:25.614578 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 775142us
I20260709 12:22:25.621002  3697 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:25.369115 31821 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:25.523591  3721 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.111s
I20260709 12:22:25.617316  3655 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
I20260709 12:22:25.619120 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2325) took 1484 ms. Trace:
I20260709 12:22:25.625626 31677 rpcz_store.cc:276] 0709 12:22:24.134952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.362620 (+1227668us) service_pool.cc:224] Handling call
0709 12:22:25.619051 (+256431us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:25.371104 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.629819 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:25.378293  3492 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:25.356712  3667 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.015s
I20260709 12:22:25.635819  3691 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:25.635905  3667 consensus_queue.cc:1243] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:22:25.403393 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:25.405843 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.407749 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66535.2us
I20260709 12:22:25.408816  3703 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:25.408959  3705 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.365433  3695 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.428719  3707 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260709 12:22:25.433509  3704 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:25.411147  3624 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.440315  3207 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:25.441177  3699 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:25.452569 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2323) took 1343 ms. Trace:
I20260709 12:22:25.453070  3714 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:25.453255  3715 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.455274  3698 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.451207  3267 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.457944 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:25.455375 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:25.488092  3398 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:24.939640  3358 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:25.792092  3358 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:25.490127  3402 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:25.507465  3086 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20260709 12:22:25.518931  3717 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:25.795317  3717 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.796768  3402 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:25.797971  3717 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:25.526161  3712 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.002s
I20260709 12:22:25.528554 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2315) took 1694 ms. Trace:
I20260709 12:22:25.531751 31826 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 9 index: 8. (index mismatch)
I20260709 12:22:25.798970 31675 rpcz_store.cc:276] 0709 12:22:23.834337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.841990 (+  7653us) service_pool.cc:224] Handling call
0709 12:22:23.874275 (+ 32285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.364653 (+1490378us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.528275 (+163622us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b5400253ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:25.528493 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2548216192}
I20260709 12:22:25.800343 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:25.800988 31675 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:24.141377  3454 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.554757  3718 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.558593  3437 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:25.559454 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 64378.1us
I20260709 12:22:25.825407 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:25.560904  3552 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:24.230309  3615 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.572273  3600 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:22:25.573035 31750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1741) took 2126 ms (client timeout 1797 ms). Trace:
W20260709 12:22:25.873514 31750 rpcz_store.cc:269] 0709 12:22:23.446384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.450479 (+  4095us) service_pool.cc:224] Handling call
0709 12:22:25.572787 (+2122308us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b5400064354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:25.572927 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4872277376}
I20260709 12:22:25.573472  3223 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.874886  3223 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:25.575261  3642 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:25.851114  3358 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.876574  3642 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:25.878095  3642 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:25.580367 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:25.878764 31820 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:25.582172  3547 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:25.588721  3614 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:25.879736  3716 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:25.879637  3614 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.879127 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 592) took 1042 ms. Trace:
I20260709 12:22:25.880378  3716 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:25.880793 31673 rpcz_store.cc:276] 0709 12:22:24.837070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.579961 (+742891us) service_pool.cc:224] Handling call
0709 12:22:25.879064 (+299103us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:25.881935  3614 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:25.882426 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 588) took 1046 ms. Trace:
I20260709 12:22:25.595858 31827 rpcz_store.cc:276] 0709 12:22:22.955371 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.117965 (+162594us) service_pool.cc:224] Handling call
0709 12:22:23.118466 (+   501us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:23.145572 (+ 27106us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:23.145623 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:23.191947 (+ 46324us) log.cc:844] Serialized 52 byte log entry
0709 12:22:23.565950 (+374003us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:23.911379 (+345429us) log.cc:844] Serialized 14 byte log entry
0709 12:22:23.965210 (+ 53831us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:23.965290 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:25.086446 (+1121156us) spinlock_profiling.cc:243] Waited 2.7 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:25.595067 (+508621us) raft_consensus.cc:1685] finished
0709 12:22:25.595173 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.595348 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26291,"prepare.queue_time_us":484441,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":45,"raft.queue_time_us":140596,"raft.run_cpu_time_us":310,"raft.run_wall_time_us":22237,"spinlock_wait_cycles":6207872,"thread_start_us":705749,"threads_started":2,"wal-append.queue_time_us":575865}
I20260709 12:22:25.325757  3525 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:25.602974 31749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.562s	user 0.010s	sys 0.010s
I20260709 12:22:25.607002  3620 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:25.589300 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 1.44544e+06us
I20260709 12:22:25.597268  3323 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:25.605541 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:25.606307  3689 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:25.372515  3702 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:25.610901 31676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2303) took 1852 ms (client timeout 1613 ms). Trace:
I20260709 12:22:25.631225 31677 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:25.617661 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:25.633433 31821 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:25.649488 31670 rpcz_store.cc:276] 0709 12:22:24.108843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:24.109441 (+   598us) service_pool.cc:224] Handling call
0709 12:22:24.135426 (+ 25985us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:24.135633 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:24.152630 (+ 16997us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.436357 (+1283727us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.452196 (+ 15839us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:25.452262 (+    66us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:25.452313 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.452381 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.452511 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":40337,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":94,"raft.queue_time_us":110691,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":230,"thread_start_us":1298312,"threads_started":4,"wal-append.queue_time_us":359757}
I20260709 12:22:25.659844 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2324) took 1535 ms. Trace:
I20260709 12:22:25.636389  3691 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:25.662925  3730 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.645565  3492 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:25.662914 31574 catalog_manager.cc:5697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:25.677732 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:25.679474 31824 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:25.676729  3697 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.684058 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:25.907231 31823 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
W20260709 12:22:25.908092 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 230428us
I20260709 12:22:25.908527 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:25.909685 31823 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 12: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:25.646047  3705 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.645788  3703 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.915467 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:25.916358 31818 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:25.916501  3703 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:25.660373  3624 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:25.665298  3699 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.920744  3699 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:25.677212  3207 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:25.921658  3207 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.922569 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:25.924419  3207 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:25.706086  3662 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:25.927004 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.714969  3709 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 12:22:25.715960 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:25.717742  3731 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:22:25.715837 31572 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42072 (request call id 240) took 32869 ms (client timeout 15000 ms). Trace:
I20260709 12:22:25.929131  3738 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:25.929591  3738 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:25.671778  3267 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.648990  3704 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:25.690222  3693 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:25.730945 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:25.932314  3704 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:25.932238  3693 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.933182  3600 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:25.666760  3714 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.736028  3398 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:25.736694 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:25.942502 31823 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 6. (term mismatch)
I20260709 12:22:25.738133 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2310) took 1974 ms. Trace:
I20260709 12:22:25.669796  3698 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.739912  3658 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:25.742869 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2307) took 1982 ms. Trace:
I20260709 12:22:25.903705  3691 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:25.779182  3732 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:25.785697 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.801904 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:25.788136 31564 maintenance_manager.cc:643] P ac4fa664fb214bda87cd96abb2116a4b: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 11.689s	user 0.194s	sys 0.107s Metrics: {"bytes_written":29564,"cfile_init":1,"dirs.queue_time_us":427364,"dirs.run_cpu_time_us":506,"dirs.run_wall_time_us":10089,"drs_written":1,"lbm_read_time_us":26910,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":2,"lbm_reads_lt_1ms":1,"lbm_write_time_us":261637,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":9,"lbm_writes_lt_1ms":19,"peak_mem_usage":0,"rows_written":63,"thread_start_us":250961,"threads_started":1}
W20260709 12:22:25.812306 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 103180us
W20260709 12:22:25.822822 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.960022 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:25.960774 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.841971 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:25.841922 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:25.961681 31826 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:25.838132  3437 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.853399 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:25.856750 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:25.884681  3620 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:25.967648 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 57859.1us
W20260709 12:22:25.852628 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50285.1us
I20260709 12:22:25.846325  3721 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.850872  3606 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.835985  3718 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.843050  3717 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 12:22:25.856694  3454 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:25.952345  3709 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:25.870267  3615 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:25.879535  3547 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.219739 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:25.284543  3692 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:25.934414  3492 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.888226 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:26.014235 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 126082us
I20260709 12:22:25.885246 31673 rpcz_store.cc:276] 0709 12:22:24.836023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.881609 (+1045586us) service_pool.cc:224] Handling call
0709 12:22:25.881910 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.882087 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:25.882112 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:25.882168 (+    56us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:25.882201 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.882253 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.882372 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:25.941861  3697 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.896412 31671 rpcz_store.cc:276] 0709 12:22:24.124616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.615060 (+1490444us) service_pool.cc:224] Handling call
0709 12:22:25.615338 (+   278us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.615543 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:25.615825 (+   282us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.659414 (+ 43589us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:25.659483 (+    69us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:25.659549 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.659629 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.659779 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":4728,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":39,"raft.queue_time_us":226,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":4562,"thread_start_us":43076,"threads_started":1,"wal-append.queue_time_us":42338}
I20260709 12:22:25.898522  3702 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.883754  3525 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:25.882606  3716 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:25.883361  3358 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:25.906138  3323 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:25.933459 31819 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:26.042423 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:25.668157  3715 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:25.935707 31820 raft_consensus.cc:2410] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 14 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:22:25.937662 31678 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:25.939055 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 578) took 2410 ms. Trace:
I20260709 12:22:26.046538 31824 rpcz_store.cc:276] 0709 12:22:23.528825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.540022 (+ 11197us) service_pool.cc:224] Handling call
0709 12:22:23.540361 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.902955 (+2362594us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.938742 (+ 35787us) spinlock_profiling.cc:243] Waited 2.14 s on lock 0x7b540043cbd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:25.938982 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4918957952}
I20260709 12:22:26.043236 31819 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:26.047848 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:25.914234  3705 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:25.930536  3738 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:25.904816 31574 catalog_manager.cc:5697] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 7 to 11, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 11 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.074447 31574 catalog_manager.cc:5697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 6 to 10, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 10 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.077208 31574 catalog_manager.cc:5697] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:25.944053 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 591) took 1107 ms. Trace:
I20260709 12:22:26.078847 31670 rpcz_store.cc:276] 0709 12:22:24.836818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.902160 (+1065342us) service_pool.cc:224] Handling call
0709 12:22:25.943713 (+ 41553us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.943844 (+   131us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:25.943860 (+    16us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:25.943887 (+    27us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:25.943904 (+    17us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.943935 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.944010 (+    75us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:25.944657 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 327069us
I20260709 12:22:26.080787 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:25.946537 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 209914us
I20260709 12:22:26.083204 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:26.084803 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:25.946693 31674 rpcz_store.cc:276] 0709 12:22:23.763841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.792603 (+ 28762us) service_pool.cc:224] Handling call
0709 12:22:23.819528 (+ 26925us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:24.021881 (+202353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:24.021932 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:24.046685 (+ 24753us) log.cc:844] Serialized 52 byte log entry
0709 12:22:25.596547 (+1549862us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:25.596935 (+   388us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.737681 (+140746us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.737799 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:25.737869 (+    70us) raft_consensus.cc:1685] finished
0709 12:22:25.737887 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.738057 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36464,"prepare.queue_time_us":1339,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":43,"raft.queue_time_us":340979,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":6313,"thread_start_us":1829984,"threads_started":3,"wal-append.queue_time_us":448124,"wal-append.run_cpu_time_us":1998,"wal-append.run_wall_time_us":1011305}
I20260709 12:22:25.949975 31818 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:25.757725  3695 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:25.946506 31823 pending_rounds.cc:85] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 5
W20260709 12:22:25.962656 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 109310us
I20260709 12:22:25.837826  3722 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:25.950951 31672 rpcz_store.cc:276] 0709 12:22:23.760485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.789882 (+ 29397us) service_pool.cc:224] Handling call
0709 12:22:23.790287 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:24.021065 (+230778us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:25.595189 (+1574124us) log.cc:844] Serialized 14 byte log entry
0709 12:22:25.738680 (+143491us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:25.739358 (+   678us) log.cc:844] Serialized 26 byte log entry
0709 12:22:25.739543 (+   185us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:25.739589 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:25.739679 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:25.742537 (+  2858us) raft_consensus.cc:1685] finished
0709 12:22:25.742567 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:25.742775 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37707,"prepare.queue_time_us":25329,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":355996,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":13657,"thread_start_us":1888776,"threads_started":3,"wal-append.queue_time_us":144781}
I20260709 12:22:25.967454 31675 raft_consensus.cc:2410] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 13 because replica has last-logged OpId of term: 10 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 9.
I20260709 12:22:25.855044 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:25.981834  3600 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:25.984968 31717 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:25.857404  3552 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:25.929219 31572 rpcz_store.cc:269] 0709 12:21:52.846571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:52.846823 (+   252us) service_pool.cc:224] Handling call
0709 12:22:25.715734 (+32868911us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:22.399146 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:22.400114 (+   968us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:22.400180 (+    66us) write_op.cc:435] Acquired schema lock
0709 12:22:22.400208 (+    28us) tablet.cc:662] Decoding operations
0709 12:22:22.578696 (+178488us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:22.603705 (+ 25009us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:22.607319 (+  3614us) tablet.cc:685] Acquiring locks for 19 operations
0709 12:22:22.865537 (+258218us) tablet.cc:701] Row locks acquired
0709 12:22:22.865605 (+    68us) write_op.cc:260] PREPARE: finished
0709 12:22:22.865911 (+   306us) write_op.cc:270] Start()
0709 12:22:22.866098 (+   187us) write_op.cc:276] Timestamp: P: 1783599742865759 usec, L: 0
0709 12:22:22.866161 (+    63us) op_driver.cc:348] REPLICATION: starting
0709 12:22:23.054523 (+188362us) log.cc:844] Serialized 7564 byte log entry
0709 12:22:24.508597 (+1454074us) op_driver.cc:464] REPLICATION: finished
0709 12:22:24.771006 (+262409us) write_op.cc:301] APPLY: starting
0709 12:22:24.771110 (+   104us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:24.772535 (+  1425us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:24.772691 (+   156us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:24.806848 (+ 34157us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:24.807400 (+   552us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=19,key_file_lookups=19,delta_file_lookups=0,mrs_lookups=38
0709 12:22:24.807472 (+    72us) write_op.cc:312] APPLY: finished
0709 12:22:24.808301 (+   829us) log.cc:844] Serialized 248 byte log entry
0709 12:22:24.809575 (+  1274us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:24.809838 (+   263us) write_op.cc:454] Released schema lock
0709 12:22:24.809954 (+   116us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:24.810592 (+   638us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:24.810651 (+    59us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1507056,"reported_tablets":49,"child_traces":[["op",{"apply.queue_time_us":262049,"apply.run_cpu_time_us":26457,"apply.run_wall_time_us":39502,"cfile_cache_hit":36,"cfile_cache_hit_bytes":37602,"cfile_cache_miss":4,"cfile_cache_miss_bytes":6216,"cfile_init":1,"lbm_read_time_us":3835,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":7,"num_ops":19,"prepare.queue_time_us":772513,"prepare.run_cpu_time_us":59235,"prepare.run_wall_time_us":1202958,"raft.queue_time_us":340694,"raft.run_cpu_time_us":17254,"raft.run_wall_time_us":258035,"replication_time_us":1642225,"thread_start_us":1019824,"threads_started":4,"wal-append.queue_time_us":780293,"wal-append.run_cpu_time_us":37414,"wal-append.run_wall_time_us":1969540}]]}
I20260709 12:22:25.968426  3704 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:26.006665 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 558) took 2435 ms. Trace:
I20260709 12:22:25.883487 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:26.002537 31822 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:25.968756  3267 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:26.016465 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 585) took 1855 ms. Trace:
W20260709 12:22:25.893772 31676 rpcz_store.cc:269] 0709 12:22:23.758208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.777014 (+ 18806us) service_pool.cc:224] Handling call
0709 12:22:25.610824 (+1833810us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:26.022238 31671 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:26.020944  3673 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:25.934564  3730 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:26.024559  3223 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:26.024740  3718 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:25.994158  3662 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:26.158656 31827 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:25.968987  3732 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:26.017966  3692 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:26.005265  3711 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:26.012681  3624 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:26.034554 31825 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:25.997680  3731 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 12:22:26.052162 31824 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 10 index: 5. (index mismatch)
I20260709 12:22:26.048662 31819 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:22:25.993409  3721 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:26.090155 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:26.096295 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:26.115991 31823 pending_rounds.cc:107] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260709 12:22:26.183187 31823 raft_consensus.cc:2889] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:26.093631 32370 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:26.103489 31673 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 12 index: 11. (index mismatch)
W20260709 12:22:26.117661 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 132796us
I20260709 12:22:26.184970 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:26.121860 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2220) took 1931 ms (client timeout 2094 ms). Trace:
W20260709 12:22:26.093911  3744 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
W20260709 12:22:26.190662 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 100567us
I20260709 12:22:26.086441 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:26.119905 31749 rpcz_store.cc:276] 0709 12:22:23.571689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.573784 (+  2095us) service_pool.cc:224] Handling call
0709 12:22:23.588093 (+ 14309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:23.607376 (+ 19283us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:23.607408 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:23.995519 (+388111us) spinlock_profiling.cc:243] Waited 7.8 ms on lock 0x7b2c00001158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:22:24.012329 (+ 16810us) log.cc:844] Serialized 26 byte log entry
0709 12:22:25.883889 (+1871560us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:25.906739 (+ 22850us) log.cc:844] Serialized 14 byte log entry
0709 12:22:26.005069 (+ 98330us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:26.006302 (+  1233us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:26.006346 (+    44us) raft_consensus.cc:1685] finished
0709 12:22:26.006428 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:26.006583 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4690,"prepare.queue_time_us":72982,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":31,"raft.queue_time_us":5010,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":21636,"spinlock_wait_cycles":17937280,"thread_start_us":1959476,"threads_started":3,"wal-append.queue_time_us":441061,"wal-append.run_cpu_time_us":2156,"wal-append.run_wall_time_us":1019058}
I20260709 12:22:26.103121  3707 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.084s
I20260709 12:22:26.102617  3528 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:26.127058 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 590) took 1290 ms. Trace:
I20260709 12:22:26.196996 31678 rpcz_store.cc:276] 0709 12:22:24.836539 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.599255 (+762716us) service_pool.cc:224] Handling call
0709 12:22:26.126886 (+527631us) spinlock_profiling.cc:243] Waited 50 ms on lock 0x7b54001d5954. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:26.126994 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":115058048}
I20260709 12:22:25.997332  3606 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:26.124742 31669 rpcz_store.cc:276] 0709 12:22:24.161608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.619871 (+1458263us) service_pool.cc:224] Handling call
0709 12:22:25.620202 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.915367 (+295165us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:25.915680 (+   313us) log.cc:844] Serialized 14 byte log entry
0709 12:22:26.000564 (+ 84884us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:26.000620 (+    56us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:26.000677 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:26.015723 (+ 15046us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b54001d5e54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:26.015793 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:26.016389 (+   596us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":603,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":89,"raft.run_cpu_time_us":534,"raft.run_wall_time_us":10149,"spinlock_wait_cycles":678263552,"thread_start_us":84503,"threads_started":1,"wal-append.queue_time_us":116614}
I20260709 12:22:26.091742  3667 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:26.160130  3673 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.122469  3705 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:26.132761 31565 maintenance_manager.cc:419] P ac4fa664fb214bda87cd96abb2116a4b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 8216 bytes on disk
I20260709 12:22:26.166208  3492 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:26.168640 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 194531us
W20260709 12:22:26.159552 31848 debug-util.cc:398] Leaking SignalData structure 0x7b0800314c40 after lost signal to thread 30506
W20260709 12:22:26.220924 31848 debug-util.cc:398] Leaking SignalData structure 0x7b08001ce5c0 after lost signal to thread 31610
W20260709 12:22:26.222141 31848 debug-util.cc:398] Leaking SignalData structure 0x7b08003540a0 after lost signal to thread 31702
W20260709 12:22:26.223278 31848 debug-util.cc:398] Leaking SignalData structure 0x7b080036dc00 after lost signal to thread 31777
W20260709 12:22:26.223551 31848 debug-util.cc:398] Leaking SignalData structure 0x7b0800343020 after lost signal to thread 31778
I20260709 12:22:26.194792  3693 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:26.125154  3722 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:26.119689  3704 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.159322  3738 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:26.230614  3747 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:26.231088  3747 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:26.232475  3743 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:22:26.185601 31822 rpcz_store.cc:269] 0709 12:22:24.191112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.219339 (+1028227us) service_pool.cc:224] Handling call
0709 12:22:26.121789 (+902450us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:26.233264 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:26.208357  3673 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:26.153326 31574 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 7 to 11, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 11 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:26.205101  3620 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:26.187055 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 209820us
I20260709 12:22:26.191828 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:26.238046 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 46325us
W20260709 12:22:26.209204 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 122871us
I20260709 12:22:26.196552  3528 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:26.187376  3437 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:26.249172 31826 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
W20260709 12:22:26.224516 31848 debug-util.cc:398] Leaking SignalData structure 0x7b0800355500 after lost signal to thread 31851
I20260709 12:22:26.092199  3716 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260709 12:22:26.204171  3718 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:26.197494  3456 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:26.197995  3624 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:26.236979  3704 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:26.229975 31564 maintenance_manager.cc:643] P ac4fa664fb214bda87cd96abb2116a4b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s	user 0.000s	sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":377,"lbm_reads_lt_1ms":4}
I20260709 12:22:26.232025  3747 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:26.185770 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 89603us
I20260709 12:22:26.235880 31574 catalog_manager.cc:5697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 6 to 9, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 9 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.046422  3703 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.133s
I20260709 12:22:26.286995  3525 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:26.241062  3731 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.006s
I20260709 12:22:26.244688  3207 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:26.289268  3697 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:26.290419  3718 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:26.291563  3718 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:26.292259 31574 catalog_manager.cc:5697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 8 to 11, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 11 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:26.244786  3600 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:26.245462  3714 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:26.295409 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:26.296765  3714 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:26.296921  3692 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:26.245712  3699 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:26.246441  3702 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:26.246683  3723 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260709 12:22:26.247521  3732 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:26.248273  3722 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:26.180971  3711 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:26.249220  3705 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:22:26.254806 31848 debug-util.cc:398] Leaking SignalData structure 0x7b080003eae0 after lost signal to thread 31852
I20260709 12:22:26.265429 31677 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:26.225690  3666 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:26.267252 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:26.228698  3547 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:26.283680  3086 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
W20260709 12:22:26.310851 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 43682us
W20260709 12:22:26.284474 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44463.6us
I20260709 12:22:26.284581  3402 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:22:26.311575 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 50489.4us
I20260709 12:22:26.293831 31574 catalog_manager.cc:5697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 8 to 11, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 11 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.313086 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:26.302755 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 211743us
W20260709 12:22:26.303892 31848 debug-util.cc:398] Leaking SignalData structure 0x7b08003e1920 after lost signal to thread 32402
W20260709 12:22:26.314739 31848 debug-util.cc:398] Leaking SignalData structure 0x7b08003b9ee0 after lost signal to thread 3640
W20260709 12:22:26.315059 31848 debug-util.cc:398] Leaking SignalData structure 0x7b080037b720 after lost signal to thread 3644
I20260709 12:22:26.253410  3693 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:26.251703  3528 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:26.308429 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:26.270866 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:26.313870 31750 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:26.268406  3698 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:26.326023 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:26.327647 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.316952 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.318500 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.319417 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.320792 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:26.375916 31642 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:26.321276 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:26.381824 31753 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
W20260709 12:22:26.381922 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 61234.7us
I20260709 12:22:26.322168 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:26.382706 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:26.323148 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:26.299365  3715 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:26.324036 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:26.377285 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 106515us
I20260709 12:22:26.384361 31674 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:26.272436  3614 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
W20260709 12:22:26.372902 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 64626.3us
I20260709 12:22:26.317804  3748 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:26.318975  3732 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:26.315917  3705 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:26.357098 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.325122 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:26.377403 31672 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:26.380373 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:26.395999 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:26.397022 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:26.381013 31678 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:26.318775 31574 catalog_manager.cc:5697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:26.383726 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:26.384516 31677 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:26.306561  3437 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.065s
I20260709 12:22:26.318692  3693 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.389104 31676 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:22:26.417654 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:26.419883 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:26.420280 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:26.388259 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:26.397655 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:26.423583 31753 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:26.392210 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:26.425879 32374 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:22:26.423629 32372 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:26.427049 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:26.394475  3756 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:26.427590 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:26.427886  3756 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:26.428350  3756 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:26.428776 31811 tablet_service.cc:1558] Processing DeleteTablet for tablet f6404cb9c2174326b06cbbb0d27d43f2 with delete_type TABLET_DATA_TOMBSTONED (TS 4d48c83c761f48079bc6ea176649dda0 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:56358
I20260709 12:22:26.429014 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:26.397404 31745 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:26.426199 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:26.431443 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.428282 32371 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:26.428225 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:26.432420 31670 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:26.426012 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:26.432888 31670 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:26.425019 31574 catalog_manager.cc:5697] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 5 to 12. New cstate: current_term: 12 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:26.433281 31753 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:26.399362 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:26.436921 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:26.437032 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.438223 31674 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:26.432655 31747 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:26.385788  3758 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:26.380916  3698 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:26.389881  3748 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:26.396968  3732 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:26.434896 31574 catalog_manager.cc:5697] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 8 to 11, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 11 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } }
I20260709 12:22:26.441730 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:26.445029 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.443390 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58335.2us
I20260709 12:22:26.414474 31669 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:22:26.385103  3757 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:26.398248 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:26.446481 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54336.9us
I20260709 12:22:26.447279 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:26.429793  3693 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:26.448011 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.448169  3606 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:26.449263 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.450752  3758 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:26.395581  3715 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:26.456769 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:26.441655 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.467065 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:26.535229 31678 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:26.467909 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.472229 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.468504 31574 catalog_manager.cc:5697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 7 to 10, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 10 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.538703 31574 catalog_manager.cc:5697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 3 to 7. New cstate: current_term: 7 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:22:26.525938 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 69545.2us
I20260709 12:22:26.529930 31676 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:26.418071  3714 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.059s
I20260709 12:22:26.385354  3620 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:26.540544 31574 catalog_manager.cc:5697] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:26.456866 31670 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:26.540339 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:26.475014  3402 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:26.548719 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:26.548441 31574 catalog_manager.cc:5697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 7 to 11. New cstate: current_term: 11 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:26.549988 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.551255 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.557371 31675 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:26.558678 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.455065  3703 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:26.559116 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:26.559919 31676 raft_consensus.cc:2410] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 11 because replica has last-logged OpId of term: 9 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260709 12:22:26.560145 31670 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:26.474421  3731 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:26.476150  3692 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:26.477324  3732 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:26.568955 31574 catalog_manager.cc:5697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 3 to 10, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 10 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:26.588191 31574 catalog_manager.cc:5697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:26.589246 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:26.577692 31678 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:26.568495 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:26.531387  3757 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.532603  3759 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:26.589903 31574 catalog_manager.cc:5697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 4 to 5, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 5 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:26.547341  3714 consensus_queue.cc:1243] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 6.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260709 12:22:26.582432  3658 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:26.594039 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:26.561091  3730 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:26.554692  3698 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:26.614648 31574 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 5 to 10, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 10 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:26.607870  3757 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:26.620167  3766 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:26.581398  3731 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:26.623927 31574 catalog_manager.cc:5697] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 3 to 11. New cstate: current_term: 11 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:26.643826 31574 catalog_manager.cc:5697] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 6 to 10, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 10 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:26.568164  3703 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:26.631851 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 184679us
I20260709 12:22:26.635869 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:26.585285  3402 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:26.656562 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62647.6us
I20260709 12:22:26.659376 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:26.611057  3715 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:26.738865 31574 catalog_manager.cc:5697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:26.734616 31670 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 11 index: 6. (term mismatch)
I20260709 12:22:26.749413 31670 pending_rounds.cc:85] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 5
I20260709 12:22:26.784066 31574 catalog_manager.cc:5697] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 7 to 10. New cstate: current_term: 10 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:26.619346  3692 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:26.795504 31670 pending_rounds.cc:107] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 10.6
I20260709 12:22:26.795928 31670 raft_consensus.cc:2889] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:26.796032 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 136883us
I20260709 12:22:26.797497 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:26.639299  3759 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:26.611649  3437 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.109s
I20260709 12:22:26.845713 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2329) took 1215 ms. Trace:
I20260709 12:22:26.846459 31670 rpcz_store.cc:276] 0709 12:22:25.629880 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.122664 (+492784us) service_pool.cc:224] Handling call
0709 12:22:26.156554 (+ 33890us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:26.432324 (+275770us) spinlock_profiling.cc:243] Waited 68.2 ms on lock 0x7b540035d020. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:22:26.845389 (+413065us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:26.845618 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":156963200}
I20260709 12:22:26.851719 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:26.655961  3703 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:26.648979  3731 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:26.658159  3620 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:22:26.853777 32375 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.603s	user 0.005s	sys 0.005s
I20260709 12:22:26.612358  3718 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.108s
I20260709 12:22:26.797068  3547 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:22:26.856060 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 288814us
I20260709 12:22:26.580128  3705 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:26.786501  3715 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.850893  3766 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:22:26.865746 32375 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.005s	sys 0.006s
W20260709 12:22:26.902887 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:26.916117 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:26.861963  3620 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.591202  3748 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:26.878329  3547 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:26.865597  3692 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:22:26.913380  3673 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.651s	user 0.011s	sys 0.000s
I20260709 12:22:26.900017  3715 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:26.941851  3692 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:26.948387  3692 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:26.950023  3760 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:22:26.950801  3673 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:26.951558  3758 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:26.953747  3673 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:26.963485  3711 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:26.966956 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:26.968546 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:26.972770  3715 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:26.916971  3666 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:26.907192  3766 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:26.930310  3620 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:26.976001  3692 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:26.939545  3547 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:27.015450  3552 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.715s	user 0.038s	sys 0.038s
I20260709 12:22:27.023272 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:27.033269  3552 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.038s	sys 0.038s
I20260709 12:22:26.992790  3666 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:26.923321  3698 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:27.066844  3692 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:26.922983  3402 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:27.032822  3691 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.064s	user 0.008s	sys 0.008s
W20260709 12:22:27.064145 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 212553us
I20260709 12:22:27.008049  3766 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:27.071043  3666 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:27.090067  3528 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.717s	user 0.011s	sys 0.000s
I20260709 12:22:27.096671 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:27.085001  3709 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:27.098891 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.006s	sys 0.004s
I20260709 12:22:27.094861  3770 tablet_replica.cc:333] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
W20260709 12:22:27.098491 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 242805us
W20260709 12:22:27.099241 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.006s	sys 0.004s
I20260709 12:22:27.107443 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:27.109822 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:27.101083 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:27.110986 31750 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:27.113819 32373 raft_consensus.cc:2410] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:22:27.106864  3667 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:27.101231  3738 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:27.127632 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.029543  3759 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:27.094427  3691 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:27.102651  3528 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:27.180272  3642 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.302s	user 0.011s	sys 0.005s
W20260709 12:22:27.180989  3779 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:22:27.181399  3778 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:22:27.193480 31750 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:27.196811 32377 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
W20260709 12:22:27.204499  3662 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260709 12:22:27.104504  3709 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:27.205852 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:27.208973 31671 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:27.166450  3715 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:27.213572  3773 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 12:22:27.219547  3547 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:27.221251  3747 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:27.221815  3547 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:27.225975 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:27.234603  3709 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:27.234989  3693 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.701s	user 0.004s	sys 0.008s
I20260709 12:22:27.329310 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:27.341521 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.344310  3666 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:27.347970  3772 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.247s	user 0.001s	sys 0.001s
W20260709 12:22:27.353281 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:27.333938  3757 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.690s	user 0.006s	sys 0.005s
W20260709 12:22:27.350324  3774 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.225s	user 0.001s	sys 0.000s
W20260709 12:22:27.331418  3775 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.263s	user 0.000s	sys 0.001s
W20260709 12:22:27.371903  3789 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
W20260709 12:22:27.358794  3704 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.045s	user 0.008s	sys 0.005s
W20260709 12:22:27.386979 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.000s	sys 0.011s
W20260709 12:22:27.387869 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.000s	sys 0.011s
I20260709 12:22:27.283946  3693 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:27.414384 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:27.415091 31675 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:27.417454 31745 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:27.422508  3787 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:22:27.424209  3666 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:27.425132 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:27.425978 31673 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:27.387140  3704 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:27.429086  3704 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:27.429260  3693 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:27.427031  3691 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:27.380223  3757 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:27.438840 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2304) took 1055 ms. Trace:
I20260709 12:22:27.439348 31748 rpcz_store.cc:276] 0709 12:22:26.383327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.385754 (+  2427us) service_pool.cc:224] Handling call
0709 12:22:26.420438 (+ 34684us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:26.437734 (+ 17296us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:26.437768 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:26.468116 (+ 30348us) log.cc:844] Serialized 52 byte log entry
0709 12:22:26.866783 (+398667us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:26.907660 (+ 40877us) log.cc:844] Serialized 14 byte log entry
0709 12:22:26.907902 (+   242us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:26.907977 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:27.235642 (+327665us) spinlock_profiling.cc:243] Waited 4.26 ms on lock 0x7b18000aac20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:22:27.438503 (+202861us) raft_consensus.cc:1685] finished
0709 12:22:27.438598 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:27.438775 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":89667,"prepare.queue_time_us":33201,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":18084,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":11447,"spinlock_wait_cycles":26629376,"thread_start_us":307378,"threads_started":1,"wal-append.queue_time_us":556250}
I20260709 12:22:27.431399 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2305) took 1047 ms. Trace:
I20260709 12:22:27.443400 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:27.447216  3788 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20260709 12:22:27.419087  3756 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.990s	user 0.009s	sys 0.004s
I20260709 12:22:27.449154 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:27.442744 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:27.452893 31643 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:27.442906 31746 rpcz_store.cc:276] 0709 12:22:26.383604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.383809 (+   205us) service_pool.cc:224] Handling call
0709 12:22:26.396035 (+ 12226us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:26.853287 (+457252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:26.853340 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:26.875707 (+ 22367us) log.cc:844] Serialized 26 byte log entry
0709 12:22:27.164496 (+288789us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:27.166056 (+  1560us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.166208 (+   152us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:27.431031 (+264823us) raft_consensus.cc:1685] finished
0709 12:22:27.431066 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:27.431301 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54040,"raft.queue_time_us":150945,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":16463,"thread_start_us":617329,"threads_started":2,"wal-append.queue_time_us":280459}
I20260709 12:22:27.430526  3528 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:27.452718  3267 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:27.456326  3703 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.582s	user 0.011s	sys 0.000s
I20260709 12:22:27.456764  3703 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:27.459372 31669 raft_consensus.cc:2368] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13.
I20260709 12:22:27.460156 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:27.450006  3731 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.572s	user 0.011s	sys 0.000s
I20260709 12:22:27.465842  3703 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:27.466197  3721 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:27.466879  3790 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:22:27.462352  3707 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:27.459196  3691 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:27.470445 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:27.467317  3732 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.848s	user 0.013s	sys 0.000s
I20260709 12:22:27.469659  3658 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:27.483163 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 633) took 1774 ms. Trace:
I20260709 12:22:27.463639  3731 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:27.486469 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:27.486675  3732 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:27.487610 31672 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:27.488219 32374 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:27.489291  3732 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:27.485603  3757 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:27.486070 32375 rpcz_store.cc:276] 0709 12:22:25.708844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.709110 (+   266us) service_pool.cc:224] Handling call
0709 12:22:25.709498 (+   388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:25.954926 (+245428us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:22:26.094719 (+139793us) log.cc:844] Serialized 14 byte log entry
0709 12:22:26.250604 (+155885us) log.cc:844] Serialized 14 byte log entry
0709 12:22:26.921842 (+671238us) log.cc:844] Serialized 14 byte log entry
0709 12:22:27.100313 (+178471us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:27.100378 (+    65us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:27.100442 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.482792 (+382350us) spinlock_profiling.cc:243] Waited 245 ms on lock 0x7b54006473d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:27.482845 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:27.483093 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31900,"prepare.queue_time_us":244579,"prepare.run_cpu_time_us":206,"prepare.run_wall_time_us":208,"raft.queue_time_us":128143,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":200,"spinlock_wait_cycles":569506432,"thread_start_us":1065207,"threads_started":4,"wal-append.queue_time_us":215523}
I20260709 12:22:27.492250  3223 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:27.495054  3223 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:27.496956  3748 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:27.496497  3223 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:27.493606  3752 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.108s	user 0.005s	sys 0.015s
I20260709 12:22:27.472200  3691 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.501261  3691 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:27.501888  3223 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:27.484582  3620 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:27.454051  3756 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:27.512605 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:27.511900  3658 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:27.473256  3666 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:22:27.517153 31677 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:27.518420 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2345) took 1125 ms. Trace:
W20260709 12:22:27.518770 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 1.29904e+06us
I20260709 12:22:27.518888 31677 rpcz_store.cc:276] 0709 12:22:26.392530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.441392 (+ 48862us) service_pool.cc:224] Handling call
0709 12:22:27.518234 (+1076842us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b54002b0a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:27.518349 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2258225920}
I20260709 12:22:27.519559 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.519934 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.520308 31827 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:27.520345 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:27.520808 31819 raft_consensus.cc:2410] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:22:27.521195 31677 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:27.521885 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.522511  3620 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:27.522936  3492 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.523448  3492 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:27.519754  3756 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:27.519593 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:27.526433 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.526948 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.527431 31672 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:27.528245 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.532647 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:27.533265 31824 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:27.533601 31818 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:22:27.535635 31818 pending_rounds.cc:85] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 4
I20260709 12:22:27.536069 31818 pending_rounds.cc:107] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 6.5
I20260709 12:22:27.536432 31818 raft_consensus.cc:2889] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:27.532160 31827 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:27.540733 31677 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:27.542268 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.543045 31821 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:27.546948 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.547482 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.548496 31825 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:27.552394 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:27.554090 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 1.24063e+06us
W20260709 12:22:27.554793  3795 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:22:27.543053  3658 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:27.512275  3707 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:27.556954  3707 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:27.558039 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:27.510941  3731 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:27.559243 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:27.560406 31792 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:27.567024 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.568156 31820 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:27.575541  3673 raft_consensus.cc:2764] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election decision vote started in defunct term 9: won
I20260709 12:22:27.575481 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.577510  3492 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:22:27.542459  3732 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.563627  3702 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:27.550096  3323 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.587093 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.619956 31822 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:27.587802 31825 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:27.578022  3454 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:27.590061 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.622879 31820 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:27.578451  3707 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:27.624325 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:27.623575 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 63177.2us
I20260709 12:22:27.630512 31824 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:27.639313  3702 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:27.639955  3702 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:27.640411  3702 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:27.626174 31792 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:27.652129 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:27.622479  3454 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:27.659787 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:27.619493  3732 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:27.624503  3323 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:27.665882 31825 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:27.668655 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:27.643375  3456 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:22:27.659842 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 100683us
I20260709 12:22:27.618291  3766 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:27.668654 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:27.683061 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.695791 31819 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:27.671448 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:27.681622 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.696986 31821 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:27.697362 31820 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 11: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:27.684911 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:27.690352 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:27.699649 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.666702  3454 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:27.706125 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:27.704066 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:27.706899 32371 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:27.696272 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:27.704550 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:27.707749 31827 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:27.710237 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.711004 31824 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:27.712116 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:27.712666 31820 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 11
I20260709 12:22:27.717763 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:27.718561 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:27.733335 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.733820 32376 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:27.724483 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.726732 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 576) took 1405 ms. Trace:
I20260709 12:22:27.729828 32375 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 12. Ops: []
I20260709 12:22:27.723145 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:27.738184 31825 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:27.737854 31751 rpcz_store.cc:276] 0709 12:22:26.321186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.321286 (+   100us) service_pool.cc:224] Handling call
0709 12:22:26.375862 (+ 54576us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:26.376079 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:26.391256 (+ 15177us) log.cc:844] Serialized 14 byte log entry
0709 12:22:27.721527 (+1330271us) spinlock_profiling.cc:243] Waited 3.06 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:27.725815 (+  4288us) spinlock_profiling.cc:243] Waited 8.14 ms on lock 0x7b180032fe60. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:22:27.726169 (+   354us) log.cc:844] Serialized 14 byte log entry
0709 12:22:27.726367 (+   198us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:27.726419 (+    52us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:27.726467 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.726530 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:27.726677 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":49277,"prepare.queue_time_us":856879,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":84,"raft.queue_time_us":45993,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":12792,"spinlock_wait_cycles":25757440,"thread_start_us":465584,"threads_started":1,"wal-append.queue_time_us":594942}
I20260709 12:22:27.736994 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:27.743533 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:27.744499 32371 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:27.729631  3766 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:27.749403  3721 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:27.749922  3721 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:27.749961 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:27.752283 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2336) took 1362 ms. Trace:
I20260709 12:22:27.752702 31669 rpcz_store.cc:276] 0709 12:22:26.389708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.471880 (+ 82172us) service_pool.cc:224] Handling call
0709 12:22:27.752063 (+1280183us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b54000465d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:27.752200 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2109392256}
I20260709 12:22:27.751896  3691 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:27.754083 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:27.672356  3801 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:27.750396  3721 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:27.770200 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:27.756801 31718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 2.094s (SENT) [suppressed 4 similar messages]
I20260709 12:22:27.776528 31669 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:22:27.702450  3323 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:27.722479  3776 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.654s	user 0.001s	sys 0.002s
I20260709 12:22:27.772806 32378 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:27.777258 31669 raft_consensus.cc:606] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Received request to transfer leadership to TS 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:27.777282 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:27.785799 31669 raft_consensus.cc:614] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:27.787781 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:27.788831 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.789460 31825 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
W20260709 12:22:27.775664 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74963.7us
I20260709 12:22:27.777364 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:27.780630 31821 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
W20260709 12:22:27.749403  3769 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.681s	user 0.000s	sys 0.004s
I20260709 12:22:27.782056 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:27.785068 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:27.790409 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:27.791019 32374 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:27.796190 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:27.728368  3732 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:27.793344 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:27.775388 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.798360 31826 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:27.799494 31819 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:27.754168  3707 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:27.797749 31824 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:27.803934 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:27.802044 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:27.806026 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:27.774946  3805 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:27.771857  3705 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:27.768097  3667 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:27.767729  3673 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:27.772600  3801 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:27.808476 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:27.785164  3547 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:27.808715 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:27.811918 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:27.809342 32376 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:27.881095 32371 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:27.874684 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.804158  3621 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:27.880335 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:27.884369 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:27.804347  3715 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:27.807737  3693 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:27.826426 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:27.826324 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:27.885046 31675 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
W20260709 12:22:27.894035 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:27.894853 31825 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:27.842530 31791 connection.cc:256] RPC call timeout handler was delayed by -2.9788295829998788s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
W20260709 12:22:27.895876 31791 connection.cc:256] RPC call timeout handler was delayed by -2.9527295829998366s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1632s for a response
W20260709 12:22:27.896752 31791 connection.cc:256] RPC call timeout handler was delayed by -2.8501295830001254s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1746s for a response
I20260709 12:22:27.896817 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:27.897147 31791 connection.cc:256] RPC call timeout handler was delayed by -2.8303295830000934s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1768s for a response
W20260709 12:22:27.898658 31791 connection.cc:256] RPC call timeout handler was delayed by -2.8042295830000512s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1797s for a response
W20260709 12:22:27.899398 31791 connection.cc:256] RPC call timeout handler was delayed by -2.5333295830000679s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2098s for a response
W20260709 12:22:27.901804 31791 connection.cc:256] RPC call timeout handler was delayed by -2.5261295830000563s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2106s for a response
W20260709 12:22:27.902436 31791 connection.cc:256] RPC call timeout handler was delayed by -2.5126295830000345s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2121s for a response
I20260709 12:22:27.897655 31819 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:27.900910 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:27.910531 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:27.826825 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:27.912590 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:27.902881 31791 connection.cc:256] RPC call timeout handler was delayed by -2.2102295830000003s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2457s for a response
I20260709 12:22:27.815712  3620 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election won for term 11
W20260709 12:22:27.921689 31791 connection.cc:256] RPC call timeout handler was delayed by -2.15442958299991s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2519s for a response
W20260709 12:22:27.922289 31791 connection.cc:256] RPC call timeout handler was delayed by -1.7404295830001502s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2979s for a response
W20260709 12:22:27.922824 31791 connection.cc:256] RPC call timeout handler was delayed by -1.6099295829999392s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3124s for a response
I20260709 12:22:27.816676  3805 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.807960  3806 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:27.831790  3456 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:27.976066 32373 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:27.991899 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:27.818984  3658 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:27.818852  3705 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.880357  3547 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:27.927048 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:27.816393  3323 raft_consensus.cc:2764] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:22:27.927876 31669 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
W20260709 12:22:27.887172 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 129604us
I20260709 12:22:27.880250  3673 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:27.816218  3707 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:28.025408 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:27.880978  3697 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:27.973565 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 653) took 2119 ms. Trace:
W20260709 12:22:27.887012 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78411.7us
I20260709 12:22:28.026794 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:27.972371  3799 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.302s	user 0.000s	sys 0.001s
I20260709 12:22:27.826709  3667 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:27.826951  3801 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:27.884891  3747 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:27.897102  3621 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:27.981241 31827 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:22:28.026379 32377 rpcz_store.cc:276] 0709 12:22:25.853706 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:25.854006 (+   300us) service_pool.cc:224] Handling call
0709 12:22:25.854394 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:27.208303 (+1353909us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:27.973232 (+764929us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54005947d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:27.973462 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3086588288}
I20260709 12:22:28.031214 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2350) took 1424 ms. Trace:
I20260709 12:22:28.031525 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:28.032058 31669 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:22:28.051632 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:27.923377 31791 connection.cc:256] RPC call timeout handler was delayed by -1.4650295830001596s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.3285s for a response
I20260709 12:22:27.918107  3693 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:28.033332 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:27.929446  3805 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:27.970117  3780 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:28.076000  3780 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:28.076473  3780 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:28.079473  3454 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:28.081807  3454 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:27.969863  3733 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.255s
I20260709 12:22:28.053031 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:28.082988  3693 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:28.084038 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:28.055024 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:28.038831 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:28.055564 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:28.079499  3621 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:28.087893  3621 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.051433 31678 rpcz_store.cc:276] 0709 12:22:26.607024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.607193 (+   169us) service_pool.cc:224] Handling call
0709 12:22:26.607550 (+   357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:26.619851 (+ 12301us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:26.619888 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:26.620484 (+   596us) log.cc:844] Serialized 26 byte log entry
0709 12:22:26.907597 (+287113us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:26.907639 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:26.907721 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:27.748540 (+840819us) spinlock_profiling.cc:243] Waited 3.15 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:28.021667 (+273127us) raft_consensus.cc:1685] finished
0709 12:22:28.021710 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:28.031146 (+  9436us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":22959,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":102,"spinlock_wait_cycles":7255936,"thread_start_us":286702,"threads_started":1,"wal-append.queue_time_us":443825}
I20260709 12:22:28.060634 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:28.019080  3705 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:28.088622  3704 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:28.089576  3704 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:28.023838  3547 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:28.095377  3800 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.292s	user 0.000s	sys 0.001s
I20260709 12:22:28.096310  3780 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:28.063988 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:28.098901 31669 raft_consensus.cc:1813] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Handling vote request from an unknown peer 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:28.099515 31669 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 12:22:28.100775 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:28.064788 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:28.102325 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:28.102926  3454 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:28.102907 31827 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:28.103871  3454 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:22:28.104586  3223 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.601s	user 0.010s	sys 0.002s
I20260709 12:22:28.104894 31827 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:28.105845  3805 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:28.107230 31825 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:28.107863 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47292.7us
I20260709 12:22:28.108601 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:28.108954  3805 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:28.109467  3715 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:28.109814  3780 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:28.110786  3658 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:28.070173 31819 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:28.088241  3621 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:28.104504  3454 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:28.045349  3624 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:22:28.040853  3620 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:28.040503  3806 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:28.048224  3756 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:28.356351 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:28.051151  3698 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:28.357560 31669 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:28.088512  3673 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:28.056269  3747 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.396606 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:28.125494  3788 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:22:28.448124 31678 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:28.452530  3707 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:28.453451  3691 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:28.456044  3456 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:28.458196  3811 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.435s	user 0.001s	sys 0.000s
I20260709 12:22:28.396876  3746 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:28.460062  3746 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:28.460958  3746 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:28.463757 31820 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.718s	user 0.010s	sys 0.000s
W20260709 12:22:28.388784 32370 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.068s	sys 0.121s
I20260709 12:22:28.464311 31820 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:28.059592  3717 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:28.441414 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:28.352517  3808 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.292s	user 0.001s	sys 0.000s
W20260709 12:22:28.464461 32370 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.648s	user 0.068s	sys 0.121s
I20260709 12:22:28.039947  3759 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.234s
I20260709 12:22:28.049108  3697 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:28.466359 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:28.482600 31820 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:28.389330  3698 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:28.265442 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.462363  3746 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:28.408242 31669 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
W20260709 12:22:28.475992  3803 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.708s	user 0.004s	sys 0.000s
I20260709 12:22:28.389684  3806 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:28.486344  3812 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:22:28.474860  3747 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:28.393256  3756 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.099s
I20260709 12:22:28.492172  3698 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:28.486006  3547 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:28.526285 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 439884us
W20260709 12:22:28.536250 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 448046us
I20260709 12:22:28.539940 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:28.537433 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:28.558337 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:28.585005 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45198.5us
I20260709 12:22:28.586598 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:28.606868 31820 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:28.608767 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:28.560240 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:28.622428 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62283.7us
I20260709 12:22:28.623718 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:28.653045 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 66738.3us
I20260709 12:22:28.654500 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:28.655953 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.546s	user 0.005s	sys 0.006s
W20260709 12:22:28.656673 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.005s	sys 0.006s
W20260709 12:22:28.657119 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 546850us
I20260709 12:22:28.560710  3759 raft_consensus.cc:2764] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:22:28.627017  3454 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:28.627904  3763 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:22:28.653744  3702 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.013s	user 0.010s	sys 0.005s
I20260709 12:22:28.626461  3717 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
W20260709 12:22:28.695104 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.689441 31820 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:28.630074  3624 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:28.703691 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49369.8us
W20260709 12:22:28.679834  3705 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.591s	user 0.009s	sys 0.001s
I20260709 12:22:28.784397 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:28.690053 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66457.8us
I20260709 12:22:28.678598  3809 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
W20260709 12:22:28.726722  3721 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.915s	user 0.003s	sys 0.009s
I20260709 12:22:28.790865 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 667) took 1034 ms. Trace:
I20260709 12:22:28.786300 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:28.743444  3807 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260709 12:22:28.791371 32370 rpcz_store.cc:276] 0709 12:22:27.756841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.757319 (+   478us) service_pool.cc:224] Handling call
0709 12:22:27.785862 (+ 28543us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:27.786107 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:22:28.475767 (+689660us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.786128 (+310361us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.790448 (+  4320us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:28.790503 (+    55us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:28.790555 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.790652 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:28.790798 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74021,"prepare.queue_time_us":671760,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"thread_start_us":924880,"threads_started":2}
W20260709 12:22:28.795153 32342 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.795s	user 0.244s	sys 0.487s
I20260709 12:22:28.793731 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:28.795549 32342 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.795s	user 0.244s	sys 0.487s
W20260709 12:22:28.795958 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 795709us
I20260709 12:22:28.796810 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:28.744673  3717 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.801080 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:28.801965 31751 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:28.804661 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:28.787425  3667 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:28.744345  3820 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:28.817559 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:28.800092  3717 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:28.783257  3624 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.806967 32372 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:28.806900 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 780) took 1161 ms. Trace:
W20260709 12:22:28.808081 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.794518  3454 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:28.829893 32375 rpcz_store.cc:276] 0709 12:22:27.645939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.646073 (+   134us) service_pool.cc:224] Handling call
0709 12:22:27.646534 (+   461us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:27.738144 (+ 91610us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.806545 (+1068401us) spinlock_profiling.cc:243] Waited 83.1 ms on lock 0x7b54004bb254. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:28.806837 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":191053696}
I20260709 12:22:28.830821 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:28.818409 32377 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:28.819978 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:28.792999  3658 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.006s	sys 0.007s
I20260709 12:22:28.798702  3763 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:28.800836  3673 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:28.696216  3620 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:28.779311  3702 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:28.790239  3323 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 12:22:28.841492  3748 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:28.835029  3824 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:22:28.838223  3658 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.727s	user 0.006s	sys 0.007s
I20260709 12:22:28.814536  3759 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:28.901588 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:28.840798  3454 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.790131  3705 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:28.862406 31820 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
W20260709 12:22:28.876871 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:28.903836 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:28.840013  3624 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:28.795871  3721 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:28.885825 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:28.891449 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 107149us
I20260709 12:22:28.855229  3763 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:22:28.892429 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:28.891997 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 95259.8us
I20260709 12:22:28.893644 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2332) took 2496 ms. Trace:
I20260709 12:22:28.895255 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:28.912691 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:28.912403 31671 rpcz_store.cc:276] 0709 12:22:26.387470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.387739 (+   269us) service_pool.cc:224] Handling call
0709 12:22:26.388010 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:27.209427 (+821417us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.883536 (+1674109us) spinlock_profiling.cc:243] Waited 819 ms on lock 0x7b5400229a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:28.883802 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1884420352}
I20260709 12:22:28.895527 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:28.806389  3822 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.059s
W20260709 12:22:28.899222  3819 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.076s	user 0.000s	sys 0.001s
I20260709 12:22:28.899541 31823 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 12: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:28.900312 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:28.900207 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:28.906472 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:28.909905 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:28.818296  3820 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:28.910727 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:28.881950  3737 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:28.921475 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:28.921370 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:28.902660  3658 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:28.938563 31750 raft_consensus.cc:2368] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13.
I20260709 12:22:28.959128 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:28.960070  3658 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:28.961194  3733 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:28.962297  3733 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:28.939718 31675 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:28.963019  3733 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:28.940922  3810 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:28.959203  3737 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:28.938941 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:28.971169  3810 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:28.971426 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:28.972074  3810 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:28.905267  3454 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:28.946419  3738 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:28.975237 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:28.940320 32375 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:28.940626 31752 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:28.916066  3832 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:28.982210  3832 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:28.985435  3832 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:28.974004 31747 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:28.973584 31820 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:28.992373 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:28.993238 31820 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
W20260709 12:22:28.938228  3621 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.605s	user 0.011s	sys 0.000s
I20260709 12:22:28.917335  3766 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:28.917672  3763 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:28.979334 31669 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:28.910885  3824 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:29.013348 31675 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:29.016654 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:28.937760  3806 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:29.017110 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58090.5us
I20260709 12:22:29.017584 31675 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:22:28.970489  3737 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:29.022480 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 109911us
I20260709 12:22:28.942435  3820 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:28.956380  3667 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:28.986452  3832 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:29.027629 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:29.031988 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:29.001102  3738 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:28.908978  3759 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:29.034833 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61790.2us
I20260709 12:22:28.975431  3547 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:29.017242  3715 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.901s	user 0.011s	sys 0.003s
I20260709 12:22:29.013887  3697 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:29.010592  3673 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:29.043131  3837 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:29.048213  3715 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:29.053289  3837 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.055543  3835 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:29.056252  3836 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.056147  3835 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:29.056658 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:29.056833  3836 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.057358 31673 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:29.059088  3836 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:29.065243  3759 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:29.041093  3667 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:28.964578  3733 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:29.068034  3667 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:29.068477 31824 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:29.069531 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2319) took 1382 ms. Trace:
I20260709 12:22:29.070290 31824 rpcz_store.cc:276] 0709 12:22:27.687069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.067734 (+1380665us) service_pool.cc:224] Handling call
0709 12:22:29.068185 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.069194 (+  1009us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.069466 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.011020  3621 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.062443  3705 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:29.060546  3837 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:29.058444  3835 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:29.041832  3806 raft_consensus.cc:2764] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:22:29.085433 31675 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:29.083307 31748 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:29.046470  3832 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:29.095542 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.087951 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.097285 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.074882 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.098428 31752 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:29.101414 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:29.089692 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:29.102536 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.106189 31675 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:22:29.117552 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:29.118072 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:29.119076 31676 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:29.119204 31673 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:29.121387 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.125485 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2316) took 1440 ms. Trace:
I20260709 12:22:29.131433 31747 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:29.126206 31827 rpcz_store.cc:276] 0709 12:22:27.685005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.105757 (+420752us) service_pool.cc:224] Handling call
0709 12:22:28.106111 (+   354us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:28.106478 (+   367us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:28.517983 (+411505us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.883118 (+365135us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.883260 (+   142us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:28.887564 (+  4304us) log.cc:844] Serialized 52 byte log entry
0709 12:22:28.887774 (+   210us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:28.888095 (+   321us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.888497 (+   402us) log.cc:844] Serialized 14 byte log entry
0709 12:22:28.888719 (+   222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.888856 (+   137us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.125145 (+236289us) raft_consensus.cc:1685] finished
0709 12:22:29.125181 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.125390 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":599842,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":18460,"raft.run_cpu_time_us":437,"raft.run_wall_time_us":183352,"spinlock_wait_cycles":49267072,"thread_start_us":580926,"threads_started":2,"wal-append.queue_time_us":312822}
I20260709 12:22:29.143967 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.145936 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:29.146684 31751 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:29.151481 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.152882 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.152834 31749 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 13 because replica has last-logged OpId of term: 12 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 5.
I20260709 12:22:29.154076 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.099538  3705 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:29.155464 31748 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:29.153337 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:29.163144 31747 raft_consensus.cc:2368] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12.
I20260709 12:22:29.117134  3845 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:29.149065  3552 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:29.171391 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.173625 31746 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:29.169947 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.175079 31748 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:29.177356 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.194725 31675 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:29.172271 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.165454 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:29.170535 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 126050us
I20260709 12:22:29.203408 31675 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:22:29.206401 31675 raft_consensus.cc:606] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:29.207978 31675 raft_consensus.cc:614] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:29.190865 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:29.181926 31751 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:29.216246 31748 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 13
W20260709 12:22:29.224699  3828 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.198s	user 0.000s	sys 0.001s
I20260709 12:22:29.224946  3717 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.164784  3705 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.226830  3832 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.229003  3705 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:29.229933 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:29.231257 31747 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:29.232822 31747 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:29.234258  3801 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.167s	user 0.006s	sys 0.005s
I20260709 12:22:29.235093 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:29.237874 31676 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:29.238221 31676 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:29.162825  3766 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:29.238880  3766 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:29.238932 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:29.239393  3766 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:29.082808  3810 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:29.247638 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 276288us
I20260709 12:22:29.260073 31748 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
W20260709 12:22:29.183960  3826 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 12:22:29.263482 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:29.264788 32377 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:29.265811 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.255947 31676 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.266791 31676 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:29.184880  3621 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:29.267143 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:29.186762  3721 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:29.179152  3843 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:29.275172 32378 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:29.417238 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:29.306990 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:29.307289 31717 connection.cc:256] RPC call timeout handler was delayed by -1.1602970909998476s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15000000000000002s for a response
W20260709 12:22:29.418524 31717 connection.cc:256] RPC call timeout handler was delayed by -1.0000970909995885s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1678s for a response
I20260709 12:22:29.420037 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:29.421124 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:29.222573  3552 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:29.393579 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.155618  3673 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:29.394083 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:29.103175  3824 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:29.432751 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.435312 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 666) took 1678 ms. Trace:
I20260709 12:22:29.435962 32372 rpcz_store.cc:276] 0709 12:22:27.756472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.757461 (+   989us) service_pool.cc:224] Handling call
0709 12:22:28.806771 (+1049310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:28.829299 (+ 22528us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.434992 (+605693us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b54004bb250. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:29.435218 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2412409600}
I20260709 12:22:29.437780 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:29.438997 31675 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:29.432219 32378 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:29.477125 31676 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:29.477978 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2314) took 2372 ms. Trace:
I20260709 12:22:29.478662 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:29.478554 31745 rpcz_store.cc:276] 0709 12:22:27.105858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.106735 (+   877us) service_pool.cc:224] Handling call
0709 12:22:27.111775 (+  5040us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:27.420227 (+308452us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.477751 (+2057524us) spinlock_profiling.cc:243] Waited 305 ms on lock 0x7b54001ed7d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:29.477909 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":701955328}
W20260709 12:22:29.482798 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:29.433346  3624 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.510s	user 0.007s	sys 0.005s
I20260709 12:22:29.396485  3758 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:29.496994 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:29.498179 31821 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
W20260709 12:22:29.488440 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 286529us
I20260709 12:22:29.411814  3759 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:29.504480 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.246554  3801 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:29.481870  3746 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.946s	user 0.006s	sys 0.007s
I20260709 12:22:29.441174  3402 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:29.488675  3826 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.087s	user 0.000s	sys 0.000s
I20260709 12:22:29.266824  3705 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.215s
I20260709 12:22:29.505239 31745 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
W20260709 12:22:29.511906 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2264) took 1747 ms (client timeout 1769 ms). Trace:
W20260709 12:22:29.512133  3856 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.150s	user 0.001s	sys 0.000s
I20260709 12:22:29.511962 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:29.513266  3829 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:22:29.513749  3846 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.577855  3846 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:29.579250 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2318) took 1892 ms. Trace:
I20260709 12:22:29.580986  3846 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:29.579811 31820 rpcz_store.cc:276] 0709 12:22:27.686793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.994837 (+1308044us) service_pool.cc:224] Handling call
0709 12:22:28.995300 (+   463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.110675 (+115375us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.110740 (+    65us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:29.138381 (+ 27641us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.249231 (+110850us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:29.249271 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.249339 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:29.578934 (+329595us) raft_consensus.cc:1685] finished
0709 12:22:29.578964 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.579162 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":59253,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":92,"thread_start_us":193421,"threads_started":2,"wal-append.queue_time_us":87779}
W20260709 12:22:29.588238 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 175076us
I20260709 12:22:29.514904  3848 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.251s
I20260709 12:22:29.520747  3849 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:29.514417  3860 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:29.191154  3845 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:29.507736  3454 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.532711  3717 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:29.532836 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.622972 31749 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:29.432586  3086 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.074s
W20260709 12:22:29.534134  3698 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.014s	user 0.013s	sys 0.001s
I20260709 12:22:29.535159 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.536181 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.537109 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:29.542618 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:29.627853 31747 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:29.628036 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 85574.7us
W20260709 12:22:29.551733 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:29.630369 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.630841  3358 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:29.631286  3358 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.631670  3358 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:29.631883  3454 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:29.616000  3860 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.632555  3454 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:22:29.632822  3705 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:29.633082  3860 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:29.633342  3705 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:29.634267  3705 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:29.635456  3837 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.635629  3705 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:29.636029  3837 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:29.554463 31822 rpcz_store.cc:269] 0709 12:22:27.764624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.783400 (+ 18776us) service_pool.cc:224] Handling call
0709 12:22:29.511631 (+1728231us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b540043d5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:29.511800 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2361550592}
I20260709 12:22:29.630252 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:29.638451 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2327) took 1947 ms. Trace:
I20260709 12:22:29.638856 31822 rpcz_store.cc:276] 0709 12:22:27.691107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.637474 (+1946367us) service_pool.cc:224] Handling call
0709 12:22:29.637892 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.638095 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.638123 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.638168 (+    45us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:29.638199 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.638265 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.638383 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.640471 31824 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:22:29.554358 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:29.641096 32372 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:29.503916  3747 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.985s	user 0.008s	sys 0.006s
I20260709 12:22:29.503547  3624 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.550954  3859 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:29.646482 31674 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:29.647219 31791 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:29.582559 31820 pending_rounds.cc:85] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 4
I20260709 12:22:29.648237 31820 pending_rounds.cc:107] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 9.5
I20260709 12:22:29.648679 31820 raft_consensus.cc:2889] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:29.521571  3833 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.031s
I20260709 12:22:29.416375  3843 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.618927 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:29.658425 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:29.660729 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:29.631773  3854 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:22:29.617508  3763 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.596s	user 0.012s	sys 0.000s
I20260709 12:22:29.505400  3758 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:29.602945 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:29.649659 31642 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:29.551321  3759 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.652933  3747 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:29.646162  3698 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:29.707298 31753 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:29.708514 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:29.708843  3698 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:29.663409 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:29.663654  3841 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:29.710659 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:29.711351 31749 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:29.712597  3758 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:29.665189  3624 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:29.663475 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:29.632556  3770 raft_consensus.cc:2243] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Raft consensus shutting down.
I20260709 12:22:29.716630 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.652088  3850 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:29.666483  3854 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:29.721978 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.622736  3552 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:29.724493 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2258) took 1962 ms. Trace:
I20260709 12:22:29.725112 31821 rpcz_store.cc:276] 0709 12:22:27.762358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.717934 (+1955576us) service_pool.cc:224] Handling call
0709 12:22:29.723931 (+  5997us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.724142 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.724168 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.724211 (+    43us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:29.724252 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.724306 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.724430 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.727245 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.617861 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:29.686237 31746 raft_consensus.cc:2368] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14.
I20260709 12:22:29.692906 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:29.695281 31675 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:29.619465  3849 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.685040  3689 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:29.697948  3801 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:29.657015  3454 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 12:22:29.691177  3763 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.699318  3822 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:29.702476  3860 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.703018  3843 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:29.659250  3859 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.703588  3358 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.710450  3841 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:29.671078 31747 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:29.710597 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:29.727797 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 238361us
I20260709 12:22:29.719322  3323 raft_consensus.cc:2706] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Received pre-election callback for term 13 while not running: Illegal state: RaftConsensus is not running: State = Stopping
I20260709 12:22:29.743500  3845 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:29.739635 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:29.746691 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:29.727169  3837 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:29.747655 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54851.7us
I20260709 12:22:29.749338  3667 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:29.767854 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2259) took 2005 ms. Trace:
W20260709 12:22:29.769073 31671 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.613s	user 0.003s	sys 0.007s
W20260709 12:22:29.804447 31671 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.003s	sys 0.008s
I20260709 12:22:29.804152 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:29.723004  3702 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:29.773178 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 105570us
I20260709 12:22:29.796917  3770 pending_rounds.cc:70] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Trying to abort 2 pending ops.
I20260709 12:22:29.776705 31673 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:22:29.773432 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 351298us
I20260709 12:22:29.814320 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:29.784976  3758 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.795295 31675 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:29.743006  3689 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.666201  3840 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 12:22:29.796917 31751 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:29.792716 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:29.800203 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60639.7us
I20260709 12:22:29.803882 31821 rpcz_store.cc:276] 0709 12:22:27.762721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.766606 (+2003885us) service_pool.cc:224] Handling call
0709 12:22:29.767097 (+   491us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.767383 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.767423 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.767481 (+    58us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:29.767529 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.767603 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.767773 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.810891 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:29.780246 31749 raft_consensus.cc:2410] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:22:29.807075  3822 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:29.806664  3746 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:29.810364  3715 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:29.831197 31745 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:29.816097 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:29.835803  3848 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:29.836463  3849 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:29.838145 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:29.837239  3873 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:29.830111 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:29.849198 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:29.822235  3689 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:29.849789 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:29.810930  3837 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:29.812860  3770 pending_rounds.cc:77] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Aborting op as it isn't in flight: id { term: 7 index: 6 } timestamp: 7305624365674995712 op_type: NO_OP noop_request { }
I20260709 12:22:29.851392  3770 raft_consensus.cc:2889] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:22:29.851897  3770 pending_rounds.cc:77] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Aborting op as it isn't in flight: id { term: 5 index: 5 } timestamp: 7305624304793550848 op_type: NO_OP noop_request { }
I20260709 12:22:29.852406  3770 raft_consensus.cc:2889] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:22:29.852867  3770 raft_consensus.cc:2272] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Raft consensus is shut down!
I20260709 12:22:29.853657  3454 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:29.820010  3853 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:29.831579  3667 consensus_queue.cc:1243] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:22:29.838353  3702 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:29.848419  3840 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:29.837988  3822 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:29.855702 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:29.856989 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:29.828086  3845 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:29.870664 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:29.871161 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 54564.2us
I20260709 12:22:29.867396 31749 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:22:29.872382 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:29.873507 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:29.878204 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:29.915654 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 611) took 2160 ms. Trace:
I20260709 12:22:29.855417  3746 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:29.848188  3835 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:29.892732 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:29.921321 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:29.921213  3746 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:29.885054  3702 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:29.882992  3840 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:29.860041  3801 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:29.828526  3758 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:29.924255  3835 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:29.927234  3758 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:29.859790  3402 raft_consensus.cc:2764] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election decision vote started in defunct term 10: won
I20260709 12:22:29.920503 31822 rpcz_store.cc:276] 0709 12:22:27.755177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.640064 (+1884887us) service_pool.cc:224] Handling call
0709 12:22:29.659260 (+ 19196us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.659487 (+   227us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:29.907843 (+248356us) log.cc:844] Serialized 14 byte log entry
0709 12:22:29.915086 (+  7243us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:29.915178 (+    92us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:29.915257 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.915359 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:29.915561 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":8071,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":99,"thread_start_us":254375,"threads_started":2}
I20260709 12:22:29.923624 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:29.924429  3801 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:29.929283  3801 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:29.901266 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 190732us
I20260709 12:22:29.930111 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:29.931449 31821 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:29.932498 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:29.935585 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:29.939858 32378 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:29.874025  3850 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:29.965086 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.871714  3841 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:29.882929  3525 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:29.871260  3845 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:29.871976  3822 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:29.855271  3846 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
W20260709 12:22:29.974846 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:29.882099  3086 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:29.990407 31821 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:29.859957  3873 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.009934 31821 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:30.013741 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:30.013707 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2263) took 2249 ms. Trace:
I20260709 12:22:30.015465  3848 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:29.973480  3853 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:29.975762  3525 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:29.981984  3845 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:30.015703 31821 rpcz_store.cc:276] 0709 12:22:27.764271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.994488 (+2230217us) service_pool.cc:224] Handling call
0709 12:22:30.009740 (+ 15252us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.013439 (+  3699us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.013648 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:29.797569  3854 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:29.923342  3840 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.7, Last appended by leader: 7, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:30.039539 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 54828.5us
I20260709 12:22:29.993880  3086 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:30.014360  3860 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:30.113715 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 241405us
I20260709 12:22:30.035074  3763 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:30.124655 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.118146 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:30.040601  3525 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:30.119045  3849 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 12:22:30.146586  3719 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:30.158087  3719 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:30.158951  3719 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.160132  3719 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:30.121685 31821 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 12:22:30.038089  3747 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:30.136173  3867 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 12:22:30.151839 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:30.137343  3876 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20260709 12:22:30.138213  3864 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
I20260709 12:22:30.141392  3801 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.113626  3837 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.109906  3869 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:30.040699  3848 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:30.273682  3834 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.330s	user 0.003s	sys 0.005s
W20260709 12:22:30.215541  3820 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.175s	user 0.007s	sys 0.007s
I20260709 12:22:30.216123  3879 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:30.284076  3820 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.284524 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2322) took 1111 ms. Trace:
I20260709 12:22:30.285107 31746 rpcz_store.cc:276] 0709 12:22:29.173593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.174246 (+   653us) service_pool.cc:224] Handling call
0709 12:22:30.284299 (+1110053us) spinlock_profiling.cc:243] Waited 435 ms on lock 0x7b54001a5ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:30.284452 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1000689920}
W20260709 12:22:30.216741 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 98687.2us
W20260709 12:22:30.217463  3766 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.978s	user 0.010s	sys 0.000s
I20260709 12:22:30.287068  3766 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.287235  3869 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.288868 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2309) took 3904 ms. Trace:
I20260709 12:22:30.136582  3871 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.289032 32375 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:30.286500 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:30.290299 31746 raft_consensus.cc:2368] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14.
I20260709 12:22:30.289314 31744 rpcz_store.cc:276] 0709 12:22:26.384759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.440767 (+ 56008us) service_pool.cc:224] Handling call
0709 12:22:26.473589 (+ 32822us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:26.547751 (+ 74162us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:26.547783 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:26.548284 (+   501us) log.cc:844] Serialized 26 byte log entry
0709 12:22:28.815426 (+2267142us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b1800475720. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:22:28.850277 (+ 34851us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:28.850319 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:28.850395 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.272979 (+1422584us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:30.288532 (+ 15553us) raft_consensus.cc:1685] finished
0709 12:22:30.288558 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.288792 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":73677,"raft.queue_time_us":85062,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":33083,"spinlock_wait_cycles":38869504,"thread_start_us":34629,"threads_started":1,"wal-append.queue_time_us":2364340}
W20260709 12:22:30.175629 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 206418us
W20260709 12:22:30.156158 32374 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 781) took 2494 ms (client timeout 2053 ms). Trace:
I20260709 12:22:30.152210 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:30.044816  3877 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:30.162143  3525 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.151119 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:30.219295  3738 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.171s	user 0.008s	sys 0.005s
W20260709 12:22:30.156670  3880 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
W20260709 12:22:30.213049  3872 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:22:30.267204  3873 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:30.273360  3848 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.115384  3086 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:30.041036  3870 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.186s
I20260709 12:22:30.163197  3859 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:30.287770 31791 leader_election.cc:400] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12.
I20260709 12:22:30.307142 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:30.164286 31821 pending_rounds.cc:85] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 6
I20260709 12:22:30.151428 32378 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:30.293810 32374 rpcz_store.cc:269] 0709 12:22:27.661499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.661658 (+   159us) service_pool.cc:224] Handling call
0709 12:22:30.155921 (+2494263us) spinlock_profiling.cc:243] Waited 25.2 ms on lock 0x7b540031d654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:30.156045 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":57951616}
I20260709 12:22:30.307650 31821 pending_rounds.cc:107] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 11.7
W20260709 12:22:30.295138  3705 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.659s	user 0.007s	sys 0.007s
I20260709 12:22:30.151401  3843 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.144s
I20260709 12:22:30.295717  3820 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:30.302573  3801 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:30.156405  3689 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.307957 31821 raft_consensus.cc:2889] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:30.280290  3837 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.311725  3859 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:30.302768  3738 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:30.320600  3707 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 12 current term: 12 is higher.
I20260709 12:22:30.323866  3525 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:30.325191  3766 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:30.325985  3759 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:30.299371  3871 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.328423  3844 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:30.328889  3738 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:30.315856  3870 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:30.330675  3837 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:30.298295  3879 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20260709 12:22:30.331624  3715 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:30.333441  3844 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 9 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:30.334828  3870 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:30.338119  3859 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:30.343390  3715 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:30.297143 31717 leader_election.cc:400] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13.
I20260709 12:22:30.432188 31717 leader_election.cc:403] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:30.309388 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:30.404629 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2351) took 3769 ms. Trace:
I20260709 12:22:30.435835  3689 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:30.433542 31670 rpcz_store.cc:276] 0709 12:22:26.635047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.848054 (+213007us) service_pool.cc:224] Handling call
0709 12:22:26.848543 (+   489us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:27.163121 (+314578us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:27.195010 (+ 31889us) log.cc:844] Serialized 14 byte log entry
0709 12:22:27.335787 (+140777us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:28.742407 (+1406620us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b18003b64a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a
0709 12:22:29.583804 (+841397us) spinlock_profiling.cc:243] Waited 2.61 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:22:29.584236 (+   432us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.847898 (+263662us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:29.847954 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:29.848062 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.403629 (+555567us) raft_consensus.cc:1685] finished
0709 12:22:30.403721 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.404300 (+   579us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":33214,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":67,"raft.queue_time_us":13713,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":12303,"spinlock_wait_cycles":46750848,"thread_start_us":688177,"threads_started":3,"wal-append.queue_time_us":666758,"wal-append.run_cpu_time_us":1492,"wal-append.run_wall_time_us":1071334}
I20260709 12:22:30.275182 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 603) took 2864 ms. Trace:
W20260709 12:22:30.407411 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 256205us
I20260709 12:22:30.438937 31672 rpcz_store.cc:276] 0709 12:22:27.411144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.543588 (+132444us) service_pool.cc:224] Handling call
0709 12:22:27.543939 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:27.562018 (+ 18079us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:27.562039 (+    21us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:27.562069 (+    30us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:27.562092 (+    23us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.274880 (+2712788us) spinlock_profiling.cc:243] Waited 17.9 ms on lock 0x7b54001b2bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:30.274946 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.275096 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":41186048}
I20260709 12:22:30.300817 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:30.300480 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:30.444624 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2321) took 2755 ms. Trace:
I20260709 12:22:30.445995 31746 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:30.411932 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2325) took 2721 ms. Trace:
I20260709 12:22:30.415136 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2335) took 2615 ms. Trace:
W20260709 12:22:30.416079 31825 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2326) took 2725 ms (client timeout 2236 ms). Trace:
I20260709 12:22:30.416587  3759 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
W20260709 12:22:30.416982 31823 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 610) took 2716 ms (client timeout 2921 ms). Trace:
I20260709 12:22:30.351197  3859 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:30.457988  3873 raft_consensus.cc:2764] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:22:30.458526 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 585) took 1655 ms. Trace:
I20260709 12:22:30.459072  3859 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.361994  3870 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.461844  3859 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:30.361665  3869 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:30.463737  3870 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:30.448470 31825 rpcz_store.cc:269] 0709 12:22:27.690806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.929327 (+238521us) service_pool.cc:224] Handling call
0709 12:22:30.415384 (+2486057us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b5400447fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:30.415709 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":235057024}
I20260709 12:22:30.364192  3871 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:30.353816  3620 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.406s	user 0.018s	sys 0.012s
W20260709 12:22:30.463757 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:30.466786  3620 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.521s	user 0.018s	sys 0.013s
I20260709 12:22:30.359198  3844 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.448378 31821 rpcz_store.cc:276] 0709 12:22:27.799573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.041640 (+2242067us) service_pool.cc:224] Handling call
0709 12:22:30.121270 (+ 79630us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.414180 (+292910us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.414866 (+   686us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.445566 31824 rpcz_store.cc:276] 0709 12:22:27.689147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.071857 (+1382710us) service_pool.cc:224] Handling call
0709 12:22:29.098604 (+ 26747us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.645491 (+546887us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.414461 (+768970us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b54001978d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:30.444256 (+ 29795us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1245302528}
W20260709 12:22:30.449803 31823 rpcz_store.cc:269] 0709 12:22:27.700455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.100498 (+400043us) service_pool.cc:224] Handling call
0709 12:22:30.416353 (+2315855us) spinlock_profiling.cc:243] Waited 798 ms on lock 0x7b54001987d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:30.416728 (+   375us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1834926848}
I20260709 12:22:30.469347  3820 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.469693 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:30.456410  3759 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:30.459127 31753 rpcz_store.cc:276] 0709 12:22:28.802604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.802760 (+   156us) service_pool.cc:224] Handling call
0709 12:22:28.803078 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.707772 (+904694us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.457688 (+749916us) spinlock_profiling.cc:243] Waited 904 ms on lock 0x7b54001ee1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:30.458155 (+   467us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2079075584}
I20260709 12:22:30.466454 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:30.428741 31827 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:30.365651  3845 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:30.446075 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 136646us
I20260709 12:22:30.472615  3525 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13.
W20260709 12:22:30.472630 31677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1636) took 2658 ms (client timeout 1926 ms). Trace:
W20260709 12:22:30.567191 31677 rpcz_store.cc:269] 0709 12:22:27.812342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.910157 (+ 97815us) service_pool.cc:224] Handling call
0709 12:22:30.470164 (+2560007us) spinlock_profiling.cc:243] Waited 2.55 s on lock 0x7b54003926d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:30.470422 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5861822336}
W20260709 12:22:30.568356 32373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 663) took 3518 ms (client timeout 1999 ms). Trace:
I20260709 12:22:30.568854 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:30.568929 32373 rpcz_store.cc:269] 0709 12:22:27.050122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.173555 (+123433us) service_pool.cc:224] Handling call
0709 12:22:30.567785 (+3394230us) spinlock_profiling.cc:243] Waited 767 ms on lock 0x7b54006c0554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:30.568037 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1764188032}
I20260709 12:22:30.579804 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 588) took 1762 ms. Trace:
I20260709 12:22:30.580369 31750 rpcz_store.cc:276] 0709 12:22:28.816902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.817103 (+   201us) service_pool.cc:224] Handling call
0709 12:22:30.579351 (+1762248us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54002945d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:30.579571 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":275531648}
I20260709 12:22:30.474323  3844 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:30.471378  3820 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:30.478930 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 181907us
I20260709 12:22:30.481436 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:30.482862  3689 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.483417  3759 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.490000 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2323) took 2792 ms. Trace:
I20260709 12:22:30.493731  3358 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:30.494716 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2262) took 2654 ms. Trace:
I20260709 12:22:30.499281 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:30.500633 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:30.501721  3770 ts_tablet_manager.cc:1916] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260709 12:22:30.509676 32378 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:30.518710 32370 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:22:30.518468 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:30.588949 31827 rpcz_store.cc:276] 0709 12:22:27.689892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.149281 (+1459389us) service_pool.cc:224] Handling call
0709 12:22:30.426580 (+1277299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.480995 (+ 54415us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.481771 (+   776us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b54001987d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:30.481955 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2921159552}
I20260709 12:22:30.523137 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:30.531553  3715 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.535526  3456 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:30.537227 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2320) took 1366 ms. Trace:
I20260709 12:22:30.587692 31821 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 7.
I20260709 12:22:30.560647  3835 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:30.560837 31825 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:30.519171  3822 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.510s	user 0.010s	sys 0.005s
I20260709 12:22:30.473587 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:30.570367  3454 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:30.573901  3845 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:30.576196 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1650) took 1319 ms. Trace:
I20260709 12:22:30.576974  3620 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:30.577454 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.578918  3731 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:30.574007  3868 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.564s	user 0.002s	sys 0.000s
I20260709 12:22:30.447083 31820 rpcz_store.cc:276] 0709 12:22:27.690479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.581879 (+1891400us) service_pool.cc:224] Handling call
0709 12:22:29.582324 (+   445us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.661166 (+ 78842us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.661209 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:29.661635 (+   426us) log.cc:844] Serialized 26 byte log entry
0709 12:22:29.873460 (+211825us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:30.044605 (+171145us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.044903 (+   298us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.044978 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.411084 (+366106us) raft_consensus.cc:1685] finished
0709 12:22:30.411143 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.411607 (+   464us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4692,"prepare.queue_time_us":290017,"prepare.run_cpu_time_us":78,"prepare.run_wall_time_us":36990,"raft.queue_time_us":78,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":7349,"thread_start_us":288422,"threads_started":2,"wal-append.queue_time_us":257920}
I20260709 12:22:30.523523 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:30.518116  3086 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.593050 31751 rpcz_store.cc:276] 0709 12:22:29.170381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.216496 (+ 46115us) service_pool.cc:224] Handling call
0709 12:22:29.216849 (+   353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:29.822276 (+605427us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.536785 (+714509us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b54002134d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:30.537088 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1333497216}
I20260709 12:22:30.594949  3689 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:30.604549 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.603804 31823 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:30.598526 31674 rpcz_store.cc:276] 0709 12:22:29.256200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.511459 (+255259us) service_pool.cc:224] Handling call
0709 12:22:30.575761 (+1064302us) spinlock_profiling.cc:243] Waited 69.4 ms on lock 0x7b54001824d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:30.575969 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":159683328}
I20260709 12:22:30.595875  3456 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:30.610497  3454 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:30.615514  3454 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.617342  3832 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:30.617918  3832 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:30.611398 32372 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:30.618783  3738 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.015s
I20260709 12:22:30.619493  3832 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.612275  3731 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.620932  3832 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:30.605515  3716 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:30.623392  3716 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.611049 32377 raft_consensus.cc:2410] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:22:30.629251  3456 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.634326  3820 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:30.636158 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 135681us
I20260709 12:22:30.639218  3883 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:30.640281  3715 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:30.640470  3693 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:30.641379  3693 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.641419  3620 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.644379 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1770) took 1053 ms. Trace:
I20260709 12:22:30.645586  3620 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:30.613368 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:30.647487 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 670) took 1395 ms. Trace:
I20260709 12:22:30.647954 32375 rpcz_store.cc:276] 0709 12:22:29.252461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.252631 (+   170us) service_pool.cc:224] Handling call
0709 12:22:29.252953 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.289543 (+1036590us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.646899 (+357356us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400544f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:30.647298 (+   399us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2382299520}
I20260709 12:22:30.648699 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:30.606091  3822 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.613839  3223 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 12:22:30.652019 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2261) took 2888 ms. Trace:
I20260709 12:22:30.652513  3822 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:30.653421 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:30.654383 31827 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:30.652446 31822 rpcz_store.cc:276] 0709 12:22:27.763467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.940139 (+2176672us) service_pool.cc:224] Handling call
0709 12:22:29.942761 (+  2622us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:29.979964 (+ 37203us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.979995 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:30.033079 (+ 53084us) log.cc:844] Serialized 52 byte log entry
0709 12:22:30.449508 (+416429us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:30.478427 (+ 28919us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.485689 (+  7262us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.651376 (+165687us) spinlock_profiling.cc:243] Waited 36.4 ms on lock 0x7b5400379e54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:30.651436 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.651496 (+    60us) raft_consensus.cc:1685] finished
0709 12:22:30.651634 (+   138us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.651829 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8395,"prepare.queue_time_us":365,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":108,"raft.queue_time_us":7670,"raft.run_cpu_time_us":392,"raft.run_wall_time_us":10710,"spinlock_wait_cycles":83669760,"thread_start_us":327316,"threads_started":1,"wal-append.queue_time_us":300057}
I20260709 12:22:30.637959 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:30.655553 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:30.640333  3883 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.016s
I20260709 12:22:30.656621 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:30.657162  3883 consensus_queue.cc:1243] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 12.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:22:30.657579  3717 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.657819  3883 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:30.659128 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:30.650519 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:30.659217 31748 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
I20260709 12:22:30.661245 31753 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:22:30.660140 31752 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:22:30.662220 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:30.656069 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:30.661886 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.662276 31746 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:30.622319  3731 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:30.669034 31825 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:30.661048 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:30.590030 31826 rpcz_store.cc:276] 0709 12:22:27.763916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.991568 (+2227652us) service_pool.cc:224] Handling call
0709 12:22:29.991944 (+   376us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.992345 (+   401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:29.992377 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.217678 (+225301us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b2c0007dbc8. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:22:30.218210 (+   532us) log.cc:844] Serialized 26 byte log entry
0709 12:22:30.417474 (+199264us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:30.417636 (+   162us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.417821 (+   185us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.417944 (+   123us) raft_consensus.cc:1685] finished
0709 12:22:30.417973 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.418464 (+   491us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":16821,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":1361,"spinlock_wait_cycles":45094272,"thread_start_us":198415,"threads_started":1,"wal-append.queue_time_us":137969}
I20260709 12:22:30.661732 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:30.663133  3689 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:30.663717 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:30.686017 31827 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:30.626513  3716 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:30.660633  3845 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:30.670588  3552 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:30.692162 31749 raft_consensus.cc:2368] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12.
I20260709 12:22:30.686604 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:30.696591  3456 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:30.658385  3717 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.644801 31747 rpcz_store.cc:276] 0709 12:22:29.590772 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:29.799475 (+208703us) service_pool.cc:224] Handling call
0709 12:22:29.812135 (+ 12660us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:29.821908 (+  9773us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:30.018733 (+196825us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.404574 (+385841us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.405069 (+   495us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:30.419235 (+ 14166us) log.cc:844] Serialized 26 byte log entry
0709 12:22:30.419982 (+   747us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:30.423234 (+  3252us) log.cc:844] Serialized 14 byte log entry
0709 12:22:30.643607 (+220373us) spinlock_profiling.cc:243] Waited 44.8 ms on lock 0x7b54004d0ba0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd50733 00007fe8fcdd5dbf 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c
0709 12:22:30.643667 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.643899 (+   232us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:30.643947 (+    48us) raft_consensus.cc:1685] finished
0709 12:22:30.643960 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.644187 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":120723,"prepare.queue_time_us":340423,"prepare.run_cpu_time_us":169,"prepare.run_wall_time_us":163,"raft.queue_time_us":149879,"raft.run_cpu_time_us":959,"raft.run_wall_time_us":22982,"spinlock_wait_cycles":102961536,"thread_start_us":345064,"threads_started":2,"wal-append.queue_time_us":237850}
I20260709 12:22:30.752723 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:30.754323 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:30.691165  3731 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.692307 31745 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:30.611277  3766 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.095s
I20260709 12:22:30.699213 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:30.658603  3883 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.745350 31824 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:30.747244  3693 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:30.748870  3620 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.751225 31822 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:30.659060  3698 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:30.770056  3698 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:30.771391  3698 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.772971  3698 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:30.768983 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:30.775452 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 92012.3us
W20260709 12:22:30.769356 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:30.769714 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.742609 31820 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:30.768739 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:30.777532 31674 raft_consensus.cc:2376] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12. Re-sending same reply.
I20260709 12:22:30.779475 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:30.779536  3854 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:30.782847 31826 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:30.761320  3552 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:30.787168  3620 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.783947 31820 raft_consensus.cc:2410] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 12 because replica has last-logged OpId of term: 9 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:22:30.787889  3552 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.789976  3620 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:30.790887  3763 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:30.795730 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:30.802713 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:30.804641  3738 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:30.775452 31824 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:30.809628 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 616) took 2354 ms. Trace:
I20260709 12:22:30.810299 31824 rpcz_store.cc:276] 0709 12:22:28.455252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.808207 (+2352955us) service_pool.cc:224] Handling call
0709 12:22:30.808740 (+   533us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.809148 (+   408us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.809182 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:30.809257 (+    75us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:30.809304 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.809379 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:30.809524 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:30.693389  3822 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.158s
I20260709 12:22:30.765971  3731 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.853963  3731 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:30.813777  3883 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:30.815797  3699 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:22:30.855046 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 175961us
I20260709 12:22:30.855443  3699 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:30.857088 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:30.815840 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 129425us
I20260709 12:22:30.817719  3716 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.858177  3716 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.859351  3854 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:30.860769  3716 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:30.818787  3689 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.861750  3689 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.862977 31823 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:30.864216 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:30.833454  3738 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.865041 31823 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:30.865182  3738 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:30.866139 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 103815us
I20260709 12:22:30.868029  3738 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:30.869316  3716 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:30.870085 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.870266  3716 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.873389  3716 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:30.844295  3717 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:30.856945  3699 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:30.842921 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:30.869164  3877 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:30.875512 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:30.864271  3689 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:30.878880  3715 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:30.879782  3693 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:30.880440 31674 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 13. Ops: []
I20260709 12:22:30.881577 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:30.883888 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:30.834817  3086 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:30.891183 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:30.899225 31825 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:30.892308 31822 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:30.901140 31826 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:30.902040 31820 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:30.892599  3693 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:30.902933 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:30.906424 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:30.904681 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47710.7us
I20260709 12:22:30.925001 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:30.923533 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:30.933643 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:30.936722 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:30.938987 31669 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:30.911944  3693 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:30.941957 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:30.942525 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44990.2us
I20260709 12:22:30.939088 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:30.944171 31745 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:30.901944  3715 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:30.949361 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:30.944352 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:30.967316 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:30.932754  3845 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:30.940420  3822 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:30.999450 31672 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:31.000393 31826 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:30.976898 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:31.009570 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:31.014132 31678 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:31.016404 31676 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:31.017501 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:31.017831 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:31.014317 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:31.014958 31677 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 7.
I20260709 12:22:31.014174 31670 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: [11.8-11.8]
W20260709 12:22:31.021207 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 145899us
I20260709 12:22:31.000636 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:31.016031  3898 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
W20260709 12:22:31.026984 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78625.1us
I20260709 12:22:31.027118 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:30.975490  3763 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:31.033169 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:31.033516 31672 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:31.033670 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:31.035322 31819 raft_consensus.cc:1217] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Deduplicated request from leader. Original: 4.4->[7.5-10.6]   Dedup: 7.5->[10.6-10.6]
I20260709 12:22:31.036069  3849 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:31.038897  3845 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:31.041002  3859 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.572s	user 0.008s	sys 0.004s
I20260709 12:22:31.041141  3840 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.041749 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2257) took 3510 ms. Trace:
I20260709 12:22:31.061146 31674 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:31.041858  3731 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:22:31.042407 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 73988.5us
I20260709 12:22:31.043111 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:31.044046 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:31.046818 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:31.043129  3717 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:31.053511 31678 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:31.055757 31819 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: [7.5-10.6]
I20260709 12:22:30.949868  3883 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election won for term 9
I20260709 12:22:30.953332  3738 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:31.059764 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:30.975514 32375 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:31.020453  3822 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:31.080354  3822 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:31.082839 31670 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
W20260709 12:22:31.083356 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49834.2us
I20260709 12:22:31.084070 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:31.084565 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:31.085395 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:31.085791  3621 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:31.089783  3822 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:31.091063  3871 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:31.091739  3871 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.092218  3871 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:31.092357 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2317) took 3406 ms. Trace:
I20260709 12:22:31.092823 31819 rpcz_store.cc:276] 0709 12:22:27.686474 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:28.113927 (+427453us) service_pool.cc:224] Handling call
0709 12:22:31.034888 (+2920961us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:31.091285 (+ 56397us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.092148 (+   863us) spinlock_profiling.cc:243] Waited 2.68 s on lock 0x7b54001c89d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:31.092290 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6161907968}
I20260709 12:22:31.063548 31821 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:22:31.063207 31676 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:31.059955 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 613) took 2936 ms. Trace:
I20260709 12:22:31.094532 31819 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:31.070012 32377 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:31.079007 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:31.078452 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:31.073333  3738 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:31.089960 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.090819 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:31.079023  3883 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:31.095710 31820 rpcz_store.cc:276] 0709 12:22:28.117783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.803235 (+2685452us) service_pool.cc:224] Handling call
0709 12:22:30.900348 (+ 97113us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:30.912304 (+ 11956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.053478 (+141174us) spinlock_profiling.cc:243] Waited 96.6 ms on lock 0x7b5400447ad0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:31.053635 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":222091008}
W20260709 12:22:31.102228 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44685.6us
I20260709 12:22:31.061074 31818 rpcz_store.cc:276] 0709 12:22:27.531696 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:27.534847 (+  3151us) service_pool.cc:224] Handling call
0709 12:22:27.535340 (+   493us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:30.439530 (+2904190us) spinlock_profiling.cc:243] Waited 4.53 ms on lock 0x7b1800371aa0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcdcdbfb 00007fe8fcdd864b
0709 12:22:30.440123 (+   593us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:30.440167 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:30.501688 (+ 61521us) log.cc:844] Serialized 52 byte log entry
0709 12:22:30.594449 (+ 92761us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:30.594475 (+    26us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:30.594568 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:31.019542 (+424974us) raft_consensus.cc:1685] finished
0709 12:22:31.019563 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.041547 (+ 21984us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12732,"raft.queue_time_us":42463,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":1533,"spinlock_wait_cycles":10419072,"thread_start_us":150043,"threads_started":2,"wal-append.queue_time_us":172084}
I20260709 12:22:31.056363  3763 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:31.102671 31671 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:31.055307  3849 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:30.969949  3492 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:31.111924 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2271) took 1836 ms. Trace:
I20260709 12:22:31.136821 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:31.095043 31745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:22:31.138475 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.138705 31745 raft_consensus.cc:606] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:31.138835 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:31.139148 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:31.140324 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:31.140606 31670 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:31.141513 31820 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:31.143622 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:31.144845  3846 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.145591  3846 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.146858 31678 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:31.147119 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2266) took 2349 ms. Trace:
I20260709 12:22:31.147563 31826 rpcz_store.cc:276] 0709 12:22:28.797306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.011361 (+2214055us) service_pool.cc:224] Handling call
0709 12:22:31.099058 (+ 87697us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.146745 (+ 47687us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.146783 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:31.146847 (+    64us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:31.146881 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.146942 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.147061 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.148202  3846 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:31.149730  3763 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:31.150838 31751 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:22:31.142295 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:31.153376 31670 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:31.156896  3086 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.157605  3086 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:31.158135 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 73732.2us
I20260709 12:22:31.159339  3698 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:31.160146 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:31.162034 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:31.162199  3756 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:31.165867 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.072475  3717 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:31.166337 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:31.166810  3859 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:31.167451  3717 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.111889 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:31.162686 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:31.168046  3883 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:31.164191  3854 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:31.171854  3854 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:31.173063  3854 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.167899 32370 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:31.174527  3854 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:31.177493  3766 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:31.177948  3766 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:31.178282  3766 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:31.179400 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2272) took 1764 ms. Trace:
I20260709 12:22:31.180011  3845 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:31.179834 31821 rpcz_store.cc:276] 0709 12:22:29.415409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.095215 (+1679806us) service_pool.cc:224] Handling call
0709 12:22:31.136671 (+ 41456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.179006 (+ 42335us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.179035 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:31.179077 (+    42us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:31.179122 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.179189 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.179316 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.182427 31745 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:31.182807 31670 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:31.157959  3822 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:31.107873  3738 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:31.162297 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:31.137588 31819 rpcz_store.cc:276] 0709 12:22:29.266005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.093807 (+1827802us) service_pool.cc:224] Handling call
0709 12:22:31.094226 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.102253 (+  8027us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.102430 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.132830 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 617) took 2652 ms. Trace:
I20260709 12:22:31.196046 31822 rpcz_store.cc:276] 0709 12:22:28.480847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:30.957745 (+2476898us) service_pool.cc:224] Handling call
0709 12:22:31.067235 (+109490us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.132320 (+ 65085us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.132364 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:31.132433 (+    69us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:31.132481 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.132576 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:31.132746 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.197741 31676 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:31.199144 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:31.201422 31678 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:31.201535 32377 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:31.173961  3693 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.192410 31823 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:31.212941 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:31.213652 31825 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:22:31.214419 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2269) took 2409 ms. Trace:
I20260709 12:22:31.214972 31825 rpcz_store.cc:276] 0709 12:22:28.805356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.134072 (+2328716us) service_pool.cc:224] Handling call
0709 12:22:31.193612 (+ 59540us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.214122 (+ 20510us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.214348 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.220330 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:31.223857 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.224188 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:31.136238  3492 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:31.227916 31821 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:31.231784 31822 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:31.201252  3844 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:31.234486 31669 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
W20260709 12:22:31.231648 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 302114us
I20260709 12:22:31.236634 31749 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 12 index: 10. (term mismatch)
I20260709 12:22:31.068782  3840 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:31.237936 31672 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: [11.6-13.7]
W20260709 12:22:31.236387 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.240826 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:31.167480  3859 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.238268 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2276) took 1441 ms. Trace:
I20260709 12:22:31.243634 31821 rpcz_store.cc:276] 0709 12:22:29.790324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.181117 (+1390793us) service_pool.cc:224] Handling call
0709 12:22:31.226046 (+ 44929us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:22:31.226256 (+   210us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.231487 (+  5231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.231704 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28150144}
I20260709 12:22:31.237108 31749 pending_rounds.cc:85] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 9
I20260709 12:22:31.243150 31820 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 8.
I20260709 12:22:31.248050 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:31.251497 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 89620.2us
I20260709 12:22:31.253103 31818 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:31.253734 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2275) took 1463 ms. Trace:
I20260709 12:22:31.254082 31818 rpcz_store.cc:276] 0709 12:22:29.789971 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.162083 (+1372112us) service_pool.cc:224] Handling call
0709 12:22:31.225459 (+ 63376us) spinlock_profiling.cc:243] Waited 31.4 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:22:31.231195 (+  5736us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.253506 (+ 22311us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.253685 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":72170240}
I20260709 12:22:31.256569 31642 leader_election.cc:400] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14.
I20260709 12:22:31.256975 31642 leader_election.cc:403] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:22:31.244877 31749 pending_rounds.cc:107] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 11.10
I20260709 12:22:31.257408 31749 raft_consensus.cc:2889] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:31.259383 31824 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:22:31.260130 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2267) took 2460 ms. Trace:
I20260709 12:22:31.261394 31826 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:22:31.262090 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2274) took 1472 ms. Trace:
I20260709 12:22:31.262410 31826 rpcz_store.cc:276] 0709 12:22:29.789561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.148644 (+1359083us) service_pool.cc:224] Handling call
0709 12:22:31.235761 (+ 87117us) spinlock_profiling.cc:243] Waited 44.7 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:22:31.242472 (+  6711us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.261804 (+ 19332us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.262032 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":102783872}
I20260709 12:22:31.263502 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:31.209508  3877 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.039s
I20260709 12:22:31.260502 31824 rpcz_store.cc:276] 0709 12:22:28.799781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.068977 (+2269196us) service_pool.cc:224] Handling call
0709 12:22:31.225016 (+156039us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:22:31.231027 (+  6011us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.259825 (+ 28798us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.260069 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":69477760}
W20260709 12:22:31.250859  3832 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.629s	user 0.010s	sys 0.006s
I20260709 12:22:31.276041 31820 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:31.278054 31819 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:31.282004 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2277) took 1491 ms. Trace:
I20260709 12:22:31.282474 31819 rpcz_store.cc:276] 0709 12:22:29.790679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.195827 (+1405148us) service_pool.cc:224] Handling call
0709 12:22:31.247956 (+ 52129us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.278518 (+ 30562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.281937 (+  3419us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:31.280962 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:31.291393 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:31.258776  3699 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.296221 31792 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:31.294327 31748 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:31.294962 31746 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 11: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:31.288760 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:31.288403 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:31.301518 31669 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:31.308175  3840 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:31.315263  3492 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.316725  3492 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:31.280608  3849 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:31.321022  3849 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:31.261102  3763 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:31.325151 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:31.299737  3715 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:31.327247 31826 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:31.296180  3689 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:31.330953  3689 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:31.331400  3086 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:31.336755  3832 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14.
I20260709 12:22:31.338486  3454 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:31.288730  3620 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:31.339351  3454 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:22:31.339479  3620 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.340373  3454 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:31.340766  3454 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:31.341295  3759 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:31.341708  3454 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.342655  3624 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:31.343067  3624 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:31.344281  3624 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:31.346052  3624 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:31.346861  3716 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:31.347489  3716 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:31.316592 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:31.338305 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:31.352911 31672 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
W20260709 12:22:31.371573 31827 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.635s	user 0.013s	sys 0.000s
W20260709 12:22:31.372128 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.372040 31827 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:31.373554 31745 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:31.377063 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:31.377844 31672 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
W20260709 12:22:31.379169 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.382160 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 157281us
I20260709 12:22:31.326756  3699 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.392127 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:31.396219 31642 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:31.390735  3456 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.621s	user 0.010s	sys 0.007s
I20260709 12:22:31.406946 31671 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
W20260709 12:22:31.418951 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.436213 31825 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 8. (term mismatch)
I20260709 12:22:31.436667 31825 pending_rounds.cc:85] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 7
I20260709 12:22:31.436916 31825 pending_rounds.cc:107] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260709 12:22:31.437167 31825 raft_consensus.cc:2889] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:31.413090  3859 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:31.441578 31671 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:31.425794  3845 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:31.423349  3693 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:31.423831  3454 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:31.423403  3763 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:31.511842 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:31.427871  3456 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:31.522622 31671 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:31.526244 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:31.528363 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 132239us
I20260709 12:22:31.483606  3845 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:31.554664 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 118835us
W20260709 12:22:31.560134 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 448308us
I20260709 12:22:31.554286  3456 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.563053  3693 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:31.537707  3845 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:31.574416 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:31.591740 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:31.592445 31822 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:31.592592 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:31.605731 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:31.606302 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 178974us
I20260709 12:22:31.608321 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:31.606112  3844 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:31.613518 31671 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:31.615883  3770 ts_tablet_manager.cc:1929] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 7.6
I20260709 12:22:31.616358  3624 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:31.616861  3689 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:31.618925  3716 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:31.623375  3699 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:31.626075 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:31.626760  3723 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:31.594121  3871 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:31.629287  3621 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.543s	user 0.008s	sys 0.004s
I20260709 12:22:31.629719  3621 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:31.591584  3759 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
W20260709 12:22:31.630505 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.630587  3871 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:31.632828 31671 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: [10.8-10.8]
I20260709 12:22:31.633255 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:31.634949 31671 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:22:31.635658 31671 raft_consensus.cc:606] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:31.635938 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:31.636037 31671 raft_consensus.cc:614] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:31.636739 31820 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:31.638330  3871 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:31.641467 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:31.642256 31717 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:31.603493 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:31.642812 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:31.643741 31826 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:31.616580  3770 log.cc:1199] T f6404cb9c2174326b06cbbb0d27d43f2 P 4d48c83c761f48079bc6ea176649dda0: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-2-root/wals/f6404cb9c2174326b06cbbb0d27d43f2
I20260709 12:22:31.647012 32375 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:31.542331  3620 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:31.606134  3904 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.304s	user 0.000s	sys 0.003s
I20260709 12:22:31.658205 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:31.571511  3840 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.077s
I20260709 12:22:31.659333 31827 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:31.661826 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:31.570303  3766 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.658643 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:31.667500  3770 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56358 (request call id 0) took 5281 ms. Trace:
I20260709 12:22:31.680038 31822 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 13 because replica has last-logged OpId of term: 11 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:22:31.681600 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:31.681876  3693 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.682375 31827 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:31.682667  3693 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.683418 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:31.684706 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:31.685308 31827 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:31.686347 31827 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []
W20260709 12:22:31.687810 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 78155.2us
I20260709 12:22:31.646407  3871 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.015s
W20260709 12:22:31.662353  3903 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.360s	user 0.003s	sys 0.000s
I20260709 12:22:31.695235 31751 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:31.686744  3705 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:31.662739  3759 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:31.711992 32373 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:22:31.670290  3731 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.012s	sys 0.016s
I20260709 12:22:31.717139 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:31.713076  3705 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:31.724879 31827 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:31.728332  3705 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:31.686499  3693 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:31.730927 31825 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:31.735307 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:31.726223 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:31.658082  3620 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 12:22:31.729518 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:31.743422 31822 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260709 12:22:31.747437  3620 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:31.747997  3620 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:31.749431  3620 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:31.750749 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:31.753247 31717 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:31.742702 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:31.755030 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:31.762051 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:31.765288 31675 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
W20260709 12:22:31.767100 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.722143  3731 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:31.709499  3844 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.030s
I20260709 12:22:31.763814 31827 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:31.754528 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:22:31.781580 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 362837us
W20260709 12:22:31.784114 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55109.7us
W20260709 12:22:31.787961 31674 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.684s	user 0.006s	sys 0.013s
I20260709 12:22:31.788520 31674 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:31.779013  3822 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:31.726536  3756 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:22:31.795100 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.797101 31678 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:31.771946  3763 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:31.802606 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47683.7us
I20260709 12:22:31.805006 31745 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:22:31.804919 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.813644 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.814383 31820 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:22:31.814656 31751 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 11: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:31.813936 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.817801 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:31.819777 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:31.792050  3086 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:22:31.811128  3738 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.582s	user 0.007s	sys 0.006s
I20260709 12:22:31.827906 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:31.823010 32342 leader_election.cc:336] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.640s (SENT)
I20260709 12:22:31.789448  3620 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:31.824321 31827 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:31.829945 31827 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:31.831348 31746 raft_consensus.cc:2410] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:22:31.831224 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.831667 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:31.824666 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:31.832124 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.825219 31749 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:31.820706 31820 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:31.827313 31750 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:31.792112  3840 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:31.834805  3840 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:31.834586  3738 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.835290  3840 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:31.837179  3738 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:31.838001 32378 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:31.838613  3913 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:22:31.838963 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.839344 31820 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:22:31.840246  3766 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:31.842536 32342 leader_election.cc:336] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.677s (SENT)
I20260709 12:22:31.843077 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:31.828862 31752 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:31.785753  3759 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.057s
I20260709 12:22:31.853457  3908 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:31.853499 31827 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:31.854319  3908 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:31.854908 31827 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:31.855551  3908 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:31.857836 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.858311 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.859548  3716 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:31.860183  3716 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.863067 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.863430  3717 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:31.863924 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:31.864329  3763 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:22:31.865372  3492 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.548s	user 0.010s	sys 0.002s
I20260709 12:22:31.866146  3871 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:31.866003  3763 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:31.863451 31820 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:22:31.865403 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:31.881711 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:31.670714  3770 rpcz_store.cc:276] 0709 12:22:26.419007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:26.428506 (+  9499us) service_pool.cc:224] Handling call
0709 12:22:27.032213 (+603707us) ts_tablet_manager.cc:1151] Deleting tablet f6404cb9c2174326b06cbbb0d27d43f2
0709 12:22:27.065135 (+ 32922us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0709 12:22:30.158169 (+3093034us) spinlock_profiling.cc:243] Waited 53.8 ms on lock 0x7b54001676d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcddd4cd 00007fe9007c834c 00007fe901deb60a 00007fe901e1c2d0 00007fe901e0283d 00007fe901e027e9 00007fe901e02779 00007fe901e02741 00007fe901e0196d 0000000000388634
0709 12:22:31.063188 (+905019us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:31.615709 (+552521us) tablet_metadata.cc:665] Metadata flushed
0709 12:22:31.667372 (+ 51663us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2500149,"spinlock_wait_cycles":123677056,"tablet-delete.queue_time_us":496954,"thread_start_us":434580,"threads_started":1}
I20260709 12:22:31.883379  3909 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:31.883508 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:31.884033  3731 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:31.807204  3883 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.043s
I20260709 12:22:31.802311  3698 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.070s
I20260709 12:22:31.846498  3723 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:31.893499 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:31.816825  3846 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.051s
I20260709 12:22:31.896298 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:31.903977 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:31.832511  3822 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.067s
W20260709 12:22:31.922648 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:31.925760 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:31.895304  3699 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:31.923924 31744 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:31.844547  3693 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:22:31.929765 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:31.908492  3766 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:31.906985  3849 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:31.887537  3717 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.040s
W20260709 12:22:31.937750 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:31.932336 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 357998us
I20260709 12:22:31.902539  3716 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:31.945780 31827 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
I20260709 12:22:31.950026 31827 pending_rounds.cc:85] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 6
I20260709 12:22:31.992875 31791 leader_election.cc:400] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12.
I20260709 12:22:31.923122  3844 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:32.002122 31827 pending_rounds.cc:107] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 12:22:32.003443 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2283) took 1519 ms. Trace:
I20260709 12:22:32.003962 31823 rpcz_store.cc:276] 0709 12:22:30.484167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.226538 (+742371us) service_pool.cc:224] Handling call
0709 12:22:31.241018 (+ 14480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.249287 (+  8269us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:31.553798 (+304511us) log.cc:844] Serialized 14 byte log entry
0709 12:22:31.946425 (+392627us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:32.002875 (+ 56450us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:32.002969 (+    94us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.003107 (+   138us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.003358 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24515,"prepare.queue_time_us":568168,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"thread_start_us":556479,"threads_started":2}
W20260709 12:22:31.994063 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 98292.1us
I20260709 12:22:31.995776 31791 leader_election.cc:403] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Cancelling election due to peer responding with higher term
I20260709 12:22:32.009420 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:31.928299  3699 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:32.002578 31827 raft_consensus.cc:2889] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:31.942767  3723 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:32.024497 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:31.917488  3492 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.095s
I20260709 12:22:32.001652  3844 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.001116  3766 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:32.011159  3877 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:32.030705 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 218012us
I20260709 12:22:32.040246 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:32.028218  3913 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:22:32.039798 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:32.048574 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:32.047896 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 118246us
I20260709 12:22:32.049631 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:32.094719 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 54532.5us
I20260709 12:22:32.011521  3845 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:32.037950  3844 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:32.120826 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:32.122918 31671 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:32.051822 31642 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:32.027770  3738 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 12:22:32.040038  3849 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260709 12:22:32.111238 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:32.115348 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 181964us
I20260709 12:22:32.016250  3909 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 12:22:32.125159  3854 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:32.124833  3716 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12.
I20260709 12:22:32.042950  3705 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:32.195732 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.186779 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:32.197710 31673 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:32.206578 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:32.208047 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:32.208756 31673 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:32.209795 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:32.197706 31674 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:32.122486  3845 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:32.220785  3454 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.705s	user 0.010s	sys 0.001s
W20260709 12:22:32.186193  3911 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.257s	user 0.000s	sys 0.000s
W20260709 12:22:32.183490  3917 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.253s	user 0.000s	sys 0.000s
W20260709 12:22:32.184454  3924 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.171s	user 0.000s	sys 0.000s
I20260709 12:22:32.235890 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:32.205204  3916 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.186s	user 0.000s	sys 0.001s
I20260709 12:22:32.237313 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:32.186380  3915 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.238466  3915 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:32.238899  3854 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:32.239527  3919 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:32.239750  3844 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:32.241056  3854 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:32.241086  3919 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:32.178944  3699 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:32.242523  3919 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:22:32.242785  3897 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.106s	user 0.000s	sys 0.002s
I20260709 12:22:32.243136  3699 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:32.243505  3849 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:32.241882  3844 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:32.245487  3456 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:32.241088  3915 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:32.255009  3716 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.255606  3716 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:32.258607 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:32.273100 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:32.273998 31822 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 12
W20260709 12:22:32.291052 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 485109us
I20260709 12:22:32.124521 31562 catalog_manager.cc:5028] TS 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): tablet f6404cb9c2174326b06cbbb0d27d43f2 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:22:32.242416  3705 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:32.297999 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:32.298878 31821 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
W20260709 12:22:32.299024 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 247299us
W20260709 12:22:32.301311 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 62761.9us
W20260709 12:22:32.304138 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 254482us
I20260709 12:22:32.276515  3715 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:32.308243 31827 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260709 12:22:32.308043 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.314661 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:32.316308 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.318166 31823 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:32.320462 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.315869 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2374) took 1269 ms. Trace:
I20260709 12:22:32.316049 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:32.329103 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:32.378147 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62576.4us
W20260709 12:22:32.328756 31562 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 387206us
I20260709 12:22:32.379582 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:32.306932  3454 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:22:32.380546 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 258754us
I20260709 12:22:32.314488  3715 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.306928  3945 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:32.327021 31670 rpcz_store.cc:276] 0709 12:22:31.042090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.200514 (+158424us) service_pool.cc:224] Handling call
0709 12:22:31.213281 (+ 12767us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.242975 (+ 29694us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:31.243010 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:31.278824 (+ 35814us) log.cc:844] Serialized 26 byte log entry
0709 12:22:31.665922 (+387098us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:31.665974 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:31.666060 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:32.310973 (+644913us) raft_consensus.cc:1685] finished
0709 12:22:32.311038 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.311349 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":91702,"raft.queue_time_us":9572,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":6512,"thread_start_us":362337,"threads_started":1,"wal-append.queue_time_us":603940}
I20260709 12:22:32.301499  3846 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:32.186455  3766 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:32.388411 31821 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:32.332291 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:32.390411 31823 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:32.401825 31822 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:32.197642  3723 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:32.380580  3849 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:32.313077  3716 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:32.332758  3845 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.402518  3945 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:32.396974  3715 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:32.412497  3620 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.568s	user 0.012s	sys 0.000s
I20260709 12:22:32.436383 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:32.438943  3952 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
W20260709 12:22:32.439400  3930 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 12:22:32.440651  3943 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:32.441449  3943 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.442023  3943 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:32.409416  3693 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 12:22:32.443717  3693 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:32.444137  3693 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:32.428870  3945 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:32.445163  3693 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:32.338338  3871 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.086s
I20260709 12:22:32.446970  3717 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:32.447443  3871 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:32.448029  3717 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:32.449645  3717 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:32.451273  3849 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:32.440725  3952 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:32.464258  3954 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.465303 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2376) took 1398 ms. Trace:
I20260709 12:22:32.465021  3954 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:32.465713 31678 rpcz_store.cc:276] 0709 12:22:31.066793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.260235 (+193442us) service_pool.cc:224] Handling call
0709 12:22:31.789082 (+528847us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:31.797535 (+  8453us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.465068 (+667533us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b54001810d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:32.465240 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1147428224}
W20260709 12:22:32.466763  3931 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.279s	user 0.000s	sys 0.001s
I20260709 12:22:32.467501 31678 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 10 index: 9. (index mismatch)
I20260709 12:22:32.468154 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2388) took 1058 ms. Trace:
I20260709 12:22:32.468451 31678 rpcz_store.cc:276] 0709 12:22:31.409449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.466779 (+1057330us) service_pool.cc:224] Handling call
0709 12:22:32.467151 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.467872 (+   721us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.468082 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:22:32.469089  3928 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.250s	user 0.001s	sys 0.002s
I20260709 12:22:32.469816 31678 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
W20260709 12:22:32.471237 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 86126.8us
I20260709 12:22:32.416041  3942 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:32.511129 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.540925 31678 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:32.464262  3952 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.543001 31670 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
W20260709 12:22:32.519775 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 109386us
I20260709 12:22:32.419023  3854 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.089s
I20260709 12:22:32.546696 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:32.434067  3716 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:32.567961  3954 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:32.570070 32372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 646) took 1511 ms (client timeout 1508 ms). Trace:
I20260709 12:22:32.571131  3832 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:32.433050  3845 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:32.512529  3693 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:32.450582  3717 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.100s
I20260709 12:22:32.554338  3952 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:32.576360  3705 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:32.582162  3854 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
W20260709 12:22:32.589564 31675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1662) took 1488 ms (client timeout 1794 ms). Trace:
I20260709 12:22:32.590588 31672 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:22:32.590804  3951 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:22:32.593954 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 99205.6us
I20260709 12:22:32.597231 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2387) took 1197 ms. Trace:
W20260709 12:22:32.591676 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 147095us
I20260709 12:22:32.624497 31670 rpcz_store.cc:276] 0709 12:22:31.400166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.456423 (+1056257us) service_pool.cc:224] Handling call
0709 12:22:32.537026 (+ 80603us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.596915 (+ 59889us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.597154 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:32.624853 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:32.605168 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58571.2us
I20260709 12:22:32.625288 31747 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 11 index: 8. (index mismatch)
I20260709 12:22:32.626609 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:32.605504 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2385) took 1230 ms. Trace:
W20260709 12:22:32.600864 32372 rpcz_store.cc:269] 0709 12:22:31.058482 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.064626 (+  6144us) service_pool.cc:224] Handling call
0709 12:22:32.569807 (+1505181us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54004c4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:32.569964 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2986442496}
I20260709 12:22:32.590363  3955 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.628564  3944 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.629007  3955 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:32.629304  3944 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:22:32.631076  3840 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.795s	user 0.006s	sys 0.011s
I20260709 12:22:32.631610  3840 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:32.632090  3955 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:32.634019  3840 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:32.610335 31827 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:32.615567 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:32.615897 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:32.615221 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 300692us
I20260709 12:22:32.628291  3951 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:32.634194  3705 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:32.634765  3948 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:32.635104  3943 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:32.636188  3621 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:22:32.623425 31675 rpcz_store.cc:269] 0709 12:22:31.106014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.136501 (+ 30487us) service_pool.cc:224] Handling call
0709 12:22:32.589332 (+1452831us) spinlock_profiling.cc:243] Waited 499 ms on lock 0x7b5400229a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:32.589489 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1147906816}
I20260709 12:22:32.636837 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:32.637710  3552 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:32.606890  3832 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:32.639624  3934 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.435s	user 0.000s	sys 0.002s
I20260709 12:22:32.639720  3715 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:32.641423  3854 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:32.641964  3845 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:32.601528 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:32.665839  3621 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:32.627671 31674 rpcz_store.cc:276] 0709 12:22:31.375244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.212514 (+837270us) service_pool.cc:224] Handling call
0709 12:22:32.212827 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.213129 (+   302us) raft_consensus.cc:1534] Early marking committed up to index 5
0709 12:22:32.214094 (+   965us) log.cc:844] Serialized 14 byte log entry
0709 12:22:32.605063 (+390969us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:32.605126 (+    63us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:32.605194 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.605285 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.605434 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23936,"prepare.queue_time_us":312,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":59,"raft.queue_time_us":88106,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":12087,"thread_start_us":303892,"threads_started":1}
I20260709 12:22:32.642911 31751 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:32.644212 31745 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:32.654564 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:32.614826  3620 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:32.665565 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:32.676678 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:32.691953 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 67596.4us
W20260709 12:22:32.693732 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 92312.9us
I20260709 12:22:32.693850  3738 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:32.672322  3849 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.024s
I20260709 12:22:32.651186  3944 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:22:32.716451 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 61980.7us
W20260709 12:22:32.719870 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.723474 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:32.729424 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:32.730406 31821 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:32.641990  3960 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 8.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.096s
W20260709 12:22:32.700723  3936 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.463s	user 0.003s	sys 0.000s
I20260709 12:22:32.700534  3621 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:32.697279  3883 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:32.673273  3717 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 12:22:32.706045  3832 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:32.666946  3942 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:32.699733 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:32.762647 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62990.9us
W20260709 12:22:32.709281 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 43841.8us
I20260709 12:22:32.766692 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:32.669652  3552 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:32.697350  3620 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:32.700588  3766 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:32.658790 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2386) took 1280 ms. Trace:
I20260709 12:22:32.754786  3938 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.080s
I20260709 12:22:32.758417 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:32.732007  3738 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:32.753305 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.756588 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:32.768868 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:32.856412 31819 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:32.856904  3723 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:32.777618 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:32.809613 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:32.761564  3689 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:32.732057  3954 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:32.777561  3948 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.073s
I20260709 12:22:32.768182  3883 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:32.903475  3883 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:32.861660  3832 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:32.672317  3951 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:32.769397  3942 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:32.923300 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:32.793964 31673 rpcz_store.cc:276] 0709 12:22:31.378247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.323125 (+944878us) service_pool.cc:224] Handling call
0709 12:22:32.325561 (+  2436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.327577 (+  2016us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:22:32.399294 (+ 71717us) log.cc:844] Serialized 14 byte log entry
0709 12:22:32.658036 (+258742us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:32.658118 (+    82us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:32.658231 (+   113us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.658369 (+   138us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:32.658658 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18029,"prepare.queue_time_us":66179,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":45865,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":10092,"thread_start_us":186036,"threads_started":1,"wal-append.queue_time_us":336696}
I20260709 12:22:32.780735  3620 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:32.807766 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 69197.6us
I20260709 12:22:32.928860 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:32.806887  3838 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.987s	user 0.009s	sys 0.008s
I20260709 12:22:32.765230 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:32.811712  3944 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.810940  3822 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:32.824515  3699 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:22:32.813271 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46739us
I20260709 12:22:32.740032  3952 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:32.878773 31821 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
W20260709 12:22:32.867798 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:32.866916  3731 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.033s
I20260709 12:22:32.892292  3689 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:32.893025  3840 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.005s
W20260709 12:22:32.844759  3756 consensus_queue.cc:1175] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "039d66a6c85e43c3936bf5017501b344" responder_term: 15 status { last_received { term: 14 index: 11 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:32.895306  3705 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:32.896385  3763 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:32.896870  3621 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:32.844442  3766 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:32.901242  3454 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:32.910995  3920 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Stepping down as leader of term 10
W20260709 12:22:32.936033 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47733.6us
I20260709 12:22:32.858347  3854 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:32.887850  3723 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:32.952111  3715 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:32.951607  3944 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:32.951776  3822 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.062455 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 683) took 1109 ms. Trace:
I20260709 12:22:33.062898 32374 rpcz_store.cc:276] 0709 12:22:31.952885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.953052 (+   167us) service_pool.cc:224] Handling call
0709 12:22:32.018334 (+ 65282us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:32.021838 (+  3504us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.021881 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:32.034233 (+ 12352us) log.cc:844] Serialized 26 byte log entry
0709 12:22:32.302916 (+268683us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:32.302987 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.303079 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.062192 (+759113us) raft_consensus.cc:1685] finished
0709 12:22:33.062247 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.062391 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":30460,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":2529,"thread_start_us":177138,"threads_started":1,"wal-append.queue_time_us":749185}
W20260709 12:22:33.055421  3967 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.121s	user 0.000s	sys 0.003s
I20260709 12:22:32.986133  3946 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:33.066946  3946 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.066784  3738 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.069383  3946 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:33.070181 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2393) took 1195 ms. Trace:
I20260709 12:22:32.997949  3962 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:32.986718  3941 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.079500  3941 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.081679  3954 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:33.082137  3941 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.009822 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2379) took 1921 ms. Trace:
I20260709 12:22:33.090077 31672 rpcz_store.cc:276] 0709 12:22:31.087927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.390441 (+302514us) service_pool.cc:224] Handling call
0709 12:22:32.590236 (+1199795us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:32.640599 (+ 50363us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.009587 (+368988us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b5400229a50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:33.009745 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2750021376}
I20260709 12:22:32.924736  3942 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:32.933832  3838 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.153s	user 0.012s	sys 0.008s
W20260709 12:22:33.018761 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.023254 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:33.009452 31792 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.560s (SENT) [suppressed 15 similar messages]
I20260709 12:22:32.912792  3945 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:32.941558  3943 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:32.951731  3844 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:32.952236  3950 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:33.001540  3877 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.006773  3955 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.051568  3689 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.052397  3766 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.053006  3940 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:22:33.053460 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.055424  3854 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:33.056629  3953 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election won for term 12
W20260709 12:22:33.057708  3961 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.278s	user 0.001s	sys 0.000s
W20260709 12:22:33.058830  3965 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.232s	user 0.000s	sys 0.000s
I20260709 12:22:33.045066  3763 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.006538  3716 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:33.061134  3944 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.165239  3763 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.075753 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 634) took 1785 ms. Trace:
I20260709 12:22:33.166220 31671 rpcz_store.cc:276] 0709 12:22:31.290856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.637456 (+346600us) service_pool.cc:224] Handling call
0709 12:22:31.637745 (+   289us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.127704 (+489959us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.075404 (+947700us) spinlock_profiling.cc:243] Waited 485 ms on lock 0x7b54001824d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:33.075666 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1115198208}
I20260709 12:22:33.078820  3962 consensus_queue.cc:1243] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 11.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
W20260709 12:22:33.169541 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.007332 31752 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 8. Current term is 9. Ops: [8.7-8.7]
I20260709 12:22:33.082547  3954 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.078843  3952 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:33.070577 31678 rpcz_store.cc:276] 0709 12:22:31.874348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.550526 (+676178us) service_pool.cc:224] Handling call
0709 12:22:32.567352 (+ 16826us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:32.611853 (+ 44501us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.611888 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:32.612658 (+   770us) log.cc:844] Serialized 52 byte log entry
0709 12:22:32.863622 (+250964us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:32.863664 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.863736 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.069944 (+206208us) raft_consensus.cc:1685] finished
0709 12:22:33.069966 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.070119 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":65945,"raft.queue_time_us":46413,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":7141,"thread_start_us":171912,"threads_started":1,"wal-append.queue_time_us":393863}
I20260709 12:22:33.008556 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 627) took 1736 ms. Trace:
W20260709 12:22:33.043759 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 120557us
I20260709 12:22:33.032665  3920 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:33.046885  3454 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.093957 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:33.154994 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.184747 31669 rpcz_store.cc:276] 0709 12:22:31.272315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.555366 (+283051us) service_pool.cc:224] Handling call
0709 12:22:31.587064 (+ 31698us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.652949 (+ 65885us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:31.652985 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:31.669966 (+ 16981us) log.cc:844] Serialized 26 byte log entry
0709 12:22:32.002574 (+332608us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:32.002623 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.002765 (+   142us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.008207 (+1005442us) raft_consensus.cc:1685] finished
0709 12:22:33.008237 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.008471 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4948,"raft.queue_time_us":6596,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":3749,"thread_start_us":261454,"threads_started":1,"wal-append.queue_time_us":1126778}
W20260709 12:22:33.191527 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.006826  3960 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.078s
I20260709 12:22:33.047161  3951 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:33.088423  3731 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:33.162575  3949 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:33.163060  3716 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:33.097324 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.160883  3950 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:33.155270  3946 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.214385 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:33.207520 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 277064us
I20260709 12:22:33.223057 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:33.232190 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:33.232081 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:33.099943  3908 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.115s
I20260709 12:22:33.261211 31678 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:22:33.248657 31669 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:22:33.203608  3731 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:33.170575  3877 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.186939  3955 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.223862  3715 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.333433 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:33.367568 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 144733us
I20260709 12:22:33.196870  3844 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.356798  3982 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:33.362102 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:33.323061 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 108813us
I20260709 12:22:33.375649 32372 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:33.229406  3763 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:33.366799 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 272943us
I20260709 12:22:33.207947  3962 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.228704  3832 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.386612 31821 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:33.368441 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:22:33.387028 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 377577us
W20260709 12:22:33.324349  3846 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.893s	user 0.007s	sys 0.007s
I20260709 12:22:33.247691  3946 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:33.231801  3689 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:33.376180 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:33.366832 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 135255us
W20260709 12:22:33.376856 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 364627us
I20260709 12:22:33.233584  3943 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:33.393502 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:22:33.386755 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.229902  3950 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:33.388932 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:33.234469  3953 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:33.389405 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:33.396870 31678 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:33.397095 31822 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:33.398012 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:33.397003 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:33.331717  3859 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:33.401461 31672 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:33.233265  3716 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.374922  3941 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.229192  3949 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:33.420112 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51787.2us
I20260709 12:22:33.417079 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.393913  3846 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:33.378716  3982 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:33.410708 31669 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:33.382820  3952 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.403795  3859 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:33.414551 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:33.415999 32344 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.011s	sys 0.012s
W20260709 12:22:33.433764 32344 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.837s	user 0.012s	sys 0.012s
I20260709 12:22:33.378935  3715 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.423026 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:22:33.411401  3963 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:22:33.428937 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:33.445024 31826 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:33.436080  3877 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:33.436753  3698 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:33.439339  3859 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:33.439916  3988 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260709 12:22:33.394984  3944 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:33.426380  3716 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.435953  3945 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:33.417589  3926 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:33.385881  3763 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.447923 31747 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:33.393549  3832 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:33.407765 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 377788us
I20260709 12:22:33.417795  3086 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Stepping down as leader of term 7
I20260709 12:22:33.418671 31821 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:33.450064 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 809) took 1623 ms. Trace:
I20260709 12:22:33.455106 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:33.399581  3962 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.457616 31822 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:33.450577  3980 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:33.477129  3454 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:33.477458  3975 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:22:33.358714  3920 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.430267  3982 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:33.432368  3981 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:33.525938 32370 rpcz_store.cc:276] 0709 12:22:31.826673 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.826966 (+   293us) service_pool.cc:224] Handling call
0709 12:22:31.856124 (+ 29158us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:31.856619 (+   495us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:32.034479 (+177860us) log.cc:844] Serialized 14 byte log entry
0709 12:22:33.419966 (+1385487us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x7b18002b6180. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a
0709 12:22:33.420151 (+   185us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:33.426847 (+  6696us) log.cc:844] Serialized 26 byte log entry
0709 12:22:33.427005 (+   158us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:33.427040 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.427100 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.443498 (+ 16398us) raft_consensus.cc:1685] finished
0709 12:22:33.449846 (+  6348us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.450002 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":475114,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":59,"raft.queue_time_us":9950,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":4717,"spinlock_wait_cycles":10647936,"thread_start_us":372893,"threads_started":2,"wal-append.queue_time_us":396232}
I20260709 12:22:33.495916  3871 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.496316  3992 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.531692 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:33.427991  3624 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:22:33.451521  3723 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.442096  3941 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:33.525712 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 70678us
I20260709 12:22:33.452461  3698 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:33.459527  3988 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:33.498208  3991 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:33.516011  3926 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:33.566363 31792 leader_election.cc:336] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.508s (SENT)
I20260709 12:22:33.525131  3086 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:33.577636 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.556670 31677 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:33.565377 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 635) took 1260 ms. Trace:
I20260709 12:22:33.567675 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:33.540285  3980 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:33.597573  3991 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:33.600348  3960 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:33.601544  3926 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.595153  3988 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:33.553081  3822 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:33.564512  3763 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.556131  3954 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:33.607748  3715 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.557245  3946 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:22:33.608781  3698 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.530690  3984 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.555712  3844 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.609601  3949 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:33.611479 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2394) took 1702 ms. Trace:
W20260709 12:22:33.537298  3621 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.507s	user 0.008s	sys 0.004s
W20260709 12:22:33.556134  3977 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:22:33.623919 31820 rpcz_store.cc:276] 0709 12:22:32.305088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.305193 (+   105us) service_pool.cc:224] Handling call
0709 12:22:32.305547 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:32.305852 (+   305us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.305905 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:32.312485 (+  6580us) log.cc:844] Serialized 26 byte log entry
0709 12:22:32.520271 (+207786us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:32.520342 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:32.520468 (+   126us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.565135 (+1044667us) raft_consensus.cc:1685] finished
0709 12:22:33.565157 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.565308 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":131905,"threads_started":1,"wal-append.queue_time_us":533431}
I20260709 12:22:33.630082 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:33.449654  3952 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.443837  3731 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.633617 31673 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:33.530763  3908 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:33.625982 31749 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:33.639690 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:33.555338  3955 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.683774 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.684072 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.684494 32377 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:33.686981 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.687285 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.689424 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2360) took 1083 ms. Trace:
I20260709 12:22:33.689905 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2365) took 1040 ms. Trace:
I20260709 12:22:33.690526 31748 rpcz_store.cc:276] 0709 12:22:32.649939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.655885 (+  5946us) service_pool.cc:224] Handling call
0709 12:22:32.672359 (+ 16474us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:32.699173 (+ 26814us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.699207 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:32.721288 (+ 22081us) log.cc:844] Serialized 26 byte log entry
0709 12:22:33.199738 (+478450us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:33.199779 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.199846 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.689685 (+489839us) raft_consensus.cc:1685] finished
0709 12:22:33.689713 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.689851 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2752,"raft.queue_time_us":4328,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":7389,"thread_start_us":375883,"threads_started":1,"wal-append.queue_time_us":651926}
I20260709 12:22:33.694969 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.695807  3763 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.696377  3763 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.681680  3723 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.698019  3723 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.698906  3086 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.699982 31673 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:22:33.683106  3908 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:33.701035  3980 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.703405 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:33.704504 31674 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:33.695039  3952 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:33.641848 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 243354us
I20260709 12:22:33.708079 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:33.642274 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:33.708745 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:33.591730  3552 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.651100 31826 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:33.651787 31824 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:33.641618  3454 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.640793  3920 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:33.637624  3624 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.678574  3973 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.755585  3973 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.516193  3944 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.759654  3944 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.568521  3871 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.669473 31676 rpcz_store.cc:276] 0709 12:22:31.908601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.586478 (+677877us) service_pool.cc:224] Handling call
0709 12:22:32.596316 (+  9838us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:32.662158 (+ 65842us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:32.662198 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:32.672130 (+  9932us) log.cc:844] Serialized 26 byte log entry
0709 12:22:33.028906 (+356776us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:33.028974 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.415188 (+386214us) spinlock_profiling.cc:243] Waited 44.1 ms on lock 0x7b540017c0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:33.415232 (+    44us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:33.611057 (+195825us) raft_consensus.cc:1685] finished
0709 12:22:33.611086 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.611393 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36712,"raft.queue_time_us":30458,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":23199,"spinlock_wait_cycles":101402496,"thread_start_us":317882,"threads_started":1,"wal-append.queue_time_us":535845}
W20260709 12:22:33.676884 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 280400us
I20260709 12:22:33.765424 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.766248  3621 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.680158 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:33.766845  3621 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.767644 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.768968  3621 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:33.769479 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.500041  3953 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.681849 31669 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:33.681708 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.682243 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:33.701690  3908 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:33.771925 31675 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:33.707547 31673 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
W20260709 12:22:33.606124  3620 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.664s	user 0.011s	sys 0.000s
I20260709 12:22:33.709337 31672 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:33.689844 31744 rpcz_store.cc:276] 0709 12:22:32.605800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.605967 (+   167us) service_pool.cc:224] Handling call
0709 12:22:32.616742 (+ 10775us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:32.692651 (+ 75909us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:33.026848 (+334197us) log.cc:844] Serialized 14 byte log entry
0709 12:22:33.234031 (+207183us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:33.234107 (+    76us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:33.234171 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.687910 (+453739us) spinlock_profiling.cc:243] Waited 11.7 ms on lock 0x7b1800316a20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419
0709 12:22:33.689205 (+  1295us) spinlock_profiling.cc:243] Waited 57.7 ms on lock 0x7b54002f62d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:33.689242 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:33.689364 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5668,"prepare.queue_time_us":467718,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":41,"raft.queue_time_us":5832,"raft.run_cpu_time_us":215,"raft.run_wall_time_us":110586,"spinlock_wait_cycles":159748352,"thread_start_us":449058,"threads_started":2,"wal-append.queue_time_us":357165}
I20260709 12:22:33.773801  3620 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.774938 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:33.776240  3620 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.776777 31744 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:33.776906  3620 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:33.777510  3620 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.732721 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:33.555914  3992 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.778945 31678 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:33.780290  3992 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:33.737326 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:33.737423 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:33.737854 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:33.738046 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 315084us
I20260709 12:22:33.781839 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:33.750846 31824 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
I20260709 12:22:33.783492 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:33.784129 31824 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:33.784289 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:33.784952 31825 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:33.785048 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:33.785851 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:33.786217 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:33.700042  3086 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 7 FOLLOWER]: Advancing to term 8
I20260709 12:22:33.788225 31824 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260709 12:22:33.789925 31669 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:33.773242  3715 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.790907  3715 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.791110  3908 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.793815  3715 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:33.793799  3908 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:33.799829 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.786465 31825 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:33.698616  3763 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.810802 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:33.656451 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.802274 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:33.812899 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:33.780350 31675 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:33.646872  3960 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.810753 32378 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:33.721894  3698 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:33.723284  3952 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:33.834467 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:33.814060  3951 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.599s	user 0.008s	sys 0.003s
I20260709 12:22:33.670622  3984 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:33.852344 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 70599.4us
I20260709 12:22:33.853245 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:33.854924 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:33.843571 32375 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 14: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:22:33.858496 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.843741 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:33.841571 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.852171 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.817332  3871 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:33.736166  3854 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.112s
I20260709 12:22:33.864217 31750 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:33.837255  3952 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:33.816656  3926 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:33.827783  3715 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:33.868214  3945 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:33.879357  3945 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:33.879817  3945 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:33.881047  3945 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.882200 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:33.868654 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:33.882730  3846 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:33.883859  3854 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:33.884040 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:33.884336  3854 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:33.868964  3948 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:33.885365  3854 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.887430  3945 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:33.887728 32376 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:22:33.800130  3624 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.859596 32369 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:33.750947  3454 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.874205 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.877817 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.890456 31746 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:33.861033 32377 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:33.873090 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:33.852655 31822 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:22:33.853169  3996 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:33.898543 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:33.852627  3994 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:33.897039 31822 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: [12.8-12.8]
I20260709 12:22:33.894302 32371 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:33.900893 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:33.902163 31672 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 14 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
W20260709 12:22:33.903149 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 50003.7us
I20260709 12:22:33.905937 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.908847 31669 pending_rounds.cc:85] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 7
I20260709 12:22:33.909230 31669 pending_rounds.cc:107] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 11.8
I20260709 12:22:33.909643 31669 raft_consensus.cc:2889] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:33.864905  3832 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:33.891005  3952 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:22:33.923576 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 88141.8us
I20260709 12:22:33.911283 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:33.925395 31751 raft_consensus.cc:2376] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11. Re-sending same reply.
I20260709 12:22:33.890789  3715 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:33.914381  3992 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:33.908383  3996 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.919811 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:33.920768  3620 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:33.899211  3738 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:33.922008  3953 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:33.921725  3859 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.937148 31822 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:33.954052 31822 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:33.899281  3846 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:33.902951  3854 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:33.943272  3951 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.002s
W20260709 12:22:33.946234  3950 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.535s	user 0.010s	sys 0.000s
I20260709 12:22:33.941891  3738 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:33.946851  3973 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:33.933434  3715 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:33.922720  3962 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:33.923343  3983 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:33.950062  3940 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:22:33.951007  4003 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.239s	user 0.001s	sys 0.000s
I20260709 12:22:33.951216  3960 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:33.953177 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:33.916079  3908 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:33.975445  3992 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:33.799124  3991 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.957564 32377 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:33.982725  3940 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:33.979331  3738 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:33.998976 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:34.000315 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:33.941090  3832 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:34.001457  3832 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.001847  3832 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:33.979904  3950 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:33.950085  3942 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.854s	user 0.003s	sys 0.007s
I20260709 12:22:33.953622  3944 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.030s
I20260709 12:22:34.004295  3942 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.938143  3941 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.967307 31822 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:33.943950  3953 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:33.977646 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:33.956244 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 9 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:33.986480 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 66729.2us
I20260709 12:22:33.958261 31826 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:22:34.001178 31819 raft_consensus.cc:2410] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 13 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 7.
I20260709 12:22:34.006657  3942 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:33.981178  3983 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:34.013552  3996 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:34.014232 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 641) took 1074 ms. Trace:
I20260709 12:22:34.004320  3950 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:33.980150  3883 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:22:34.015375 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:34.017549  3624 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:34.004768  3846 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:33.991762 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:34.055795 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.056838 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:34.101533 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:34.058723 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 89293.7us
W20260709 12:22:34.071777 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.092111 32373 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:34.095027 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:34.102882 31826 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:34.092412 31818 rpcz_store.cc:276] 0709 12:22:32.939559 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.939912 (+   353us) service_pool.cc:224] Handling call
0709 12:22:32.987239 (+ 47327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:33.014015 (+ 26776us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:22:33.074857 (+ 60842us) log.cc:844] Serialized 14 byte log entry
0709 12:22:33.560542 (+485685us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b1800481f60. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:22:33.701824 (+141282us) spinlock_profiling.cc:243] Waited 5.38 ms on lock 0x7b1800481f60. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:22:33.859810 (+157986us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:33.875831 (+ 16021us) log.cc:844] Serialized 26 byte log entry
0709 12:22:33.875982 (+   151us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:33.876025 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:33.876105 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:34.014011 (+137906us) raft_consensus.cc:1685] finished
0709 12:22:34.014032 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:34.014168 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11832,"prepare.queue_time_us":60772,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":14676,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":43983,"spinlock_wait_cycles":64544640,"thread_start_us":135759,"threads_started":1,"wal-append.queue_time_us":801339}
I20260709 12:22:33.989638  3763 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:34.101411 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44623.1us
I20260709 12:22:34.105214 31818 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:34.105171  3763 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.105676 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:34.107165 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:34.072635 31822 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:34.114985 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:34.115482 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.117049 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:34.119241 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.119120 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 812) took 1391 ms. Trace:
I20260709 12:22:34.121734 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:34.041057  3908 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:34.015372  3984 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:34.120896 32373 rpcz_store.cc:276] 0709 12:22:32.728120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.728310 (+   190us) service_pool.cc:224] Handling call
0709 12:22:34.015210 (+1286900us) spinlock_profiling.cc:243] Waited 15.9 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0e9c 00007fe8f9cb0a5b 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:22:34.118962 (+103752us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":36540160}
W20260709 12:22:34.125160 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 133493us
I20260709 12:22:34.138144 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:34.039790  3844 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:34.141184  3996 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:34.141925  3844 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.064004  3859 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:34.098387  3983 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:34.142354  3996 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:34.106611  3949 raft_consensus.cc:2764] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:22:34.122052  3992 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:34.156352 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.183596 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:34.154343 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.151381  3996 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:34.183187 31673 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:34.134579  3705 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:34.176592  3953 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.185462 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.126804  3994 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:34.190310 31753 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:34.061609  3940 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.197445 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:34.209653 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.147517  3946 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:34.210564 31676 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:34.143294  3984 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.211560 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.141438  4013 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 2, Time since last communication: 0.060s
I20260709 12:22:34.221576 31822 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 8. (term mismatch)
I20260709 12:22:34.194905  3705 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.194756  3996 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:34.292531 31822 pending_rounds.cc:85] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 7
I20260709 12:22:34.292725  3705 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:34.206820  3964 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:34.218037  3454 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:34.227770  3953 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:34.195266  4023 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:34.225407  3946 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:34.283653  3952 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:34.285224  3982 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.693s	user 0.000s	sys 0.011s
I20260709 12:22:34.287937  3738 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:34.288949 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.309581 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:34.290184  3988 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.634s	user 0.011s	sys 0.000s
I20260709 12:22:34.291765  3698 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:34.190675  3992 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:34.292045 31745 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:22:34.292923 31822 pending_rounds.cc:107] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 12:22:34.333037 31822 raft_consensus.cc:2889] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:34.225996  3984 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.334517 31748 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:34.200991  3994 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:34.301515 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.303406 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:34.336277 32369 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:34.337724  3454 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.338166  3984 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:34.337976  4014 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.035s
I20260709 12:22:34.338814  3994 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:34.300256  4023 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:34.338325  3454 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.343369  4023 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.344699  4023 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:34.344975  3454 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:34.345593  3454 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.313360  3988 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.347283  3946 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.348917  3844 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:34.348910 31820 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 7. Current term is 8. Ops: []
I20260709 12:22:34.350068  3988 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:34.350394 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:34.351897  3738 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:34.352511  4025 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:22:34.353574  3715 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.354125 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:34.346074  3454 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.354763 31753 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:34.355295  3994 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.355754 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:34.356943 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:34.357393  3994 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:34.357510  3454 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:34.357671 31753 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 13: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:34.362856 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.363548 31674 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:34.364394 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.365513 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.346280  3832 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.311836  3982 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.320453  3698 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:34.333218  3992 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:34.368656 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:34.435876 31822 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
W20260709 12:22:34.369367 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.387669 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:34.437866 31827 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:34.438515 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2383) took 1028 ms. Trace:
I20260709 12:22:34.438983 31827 rpcz_store.cc:276] 0709 12:22:33.410412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.414433 (+  4021us) service_pool.cc:224] Handling call
0709 12:22:34.387467 (+973034us) spinlock_profiling.cc:243] Waited 6.81 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:22:34.438443 (+ 50976us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15656192}
I20260709 12:22:34.388235 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2369) took 1672 ms. Trace:
I20260709 12:22:34.440016 31747 rpcz_store.cc:276] 0709 12:22:32.716214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.718546 (+  2332us) service_pool.cc:224] Handling call
0709 12:22:34.388069 (+1669523us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b5400064d54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:34.388179 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":481483136}
I20260709 12:22:34.366578 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.371793 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:34.372678 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.442487 31674 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:34.368927  3832 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:34.388679 31672 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:34.369709 31826 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:34.365860  3705 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.391628  3698 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.423772 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:34.451766 31753 raft_consensus.cc:2410] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 14 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:22:34.452680 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:34.453702 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:34.366305 31676 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 14: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:34.426745 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:34.337229 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:34.430154  3986 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.991s	user 0.001s	sys 0.001s
I20260709 12:22:34.379896  3952 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:34.433230 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2371) took 1036 ms. Trace:
I20260709 12:22:34.444679 31673 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 11: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:34.448670 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:34.460708 31750 rpcz_store.cc:276] 0709 12:22:33.396637 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.396757 (+   120us) service_pool.cc:224] Handling call
0709 12:22:34.432899 (+1036142us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b5400295ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:34.433125 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1070112000}
I20260709 12:22:34.461575 31822 raft_consensus.cc:2410] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
W20260709 12:22:34.385906  4012 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.464s	user 0.000s	sys 0.003s
I20260709 12:22:34.376449  3715 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:34.455829 31818 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
W20260709 12:22:34.458786 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 121660us
I20260709 12:22:34.390228  3982 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:34.451092  3620 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.332749  3908 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.467557 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:34.463208  3926 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.570s	user 0.011s	sys 0.000s
I20260709 12:22:34.469475 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:34.440795  3996 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.477557 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:34.478282 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:34.470160 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.474828 31753 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:22:34.482082 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:34.486344 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:34.486583 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 303687us
I20260709 12:22:34.488708 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:34.489835 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:34.478600  3991 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:34.483839  3454 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:34.491541  3454 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:34.491930  3454 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:34.494917 32375 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:34.483757  3994 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:34.475809  3620 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.498100  3620 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:34.499019  4035 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:34.467847  3698 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:34.502521 31677 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:22:34.503621 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:34.504190 31677 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:34.504906  3996 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:34.507376 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:34.508184  3991 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:34.508841  3991 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:34.509303  3983 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.509829  3991 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:34.510257  3991 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.510599  3991 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:34.511893  3983 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:34.486327 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:34.513882 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.515149 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:34.516991 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:34.519033 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.519827 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:34.522440 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:34.523738 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:34.523742 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:34.525317 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.525306 31818 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:34.525583 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:34.527352 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:34.519620  3983 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:34.528475  3983 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.527446 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:34.530767 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:34.532877 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:34.538856  3966 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:34.540573 31822 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:34.541788  4027 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.542484  4027 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:34.543896  4034 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:22:34.545233  4027 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:34.555830 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.556448 31675 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:22:34.556610 31673 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:34.557633 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:34.510834  3988 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Stepping down as leader of term 11
W20260709 12:22:34.550779 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.556638 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.559808 31678 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:34.561812 31677 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:34.532692 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:34.562976 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:34.564175 31747 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:34.564520 31673 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 8: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:34.564488 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:34.566570 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.567513 31677 raft_consensus.cc:2410] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 14 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:22:34.568359 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:34.568914 31677 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:34.569299 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:34.572999 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45614.8us
I20260709 12:22:34.574687 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:34.576395 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:34.578374 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.578853 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:34.579147 31676 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:34.580675 31750 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 8.
I20260709 12:22:34.529474  3953 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:34.584010 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:34.584630 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 51831.8us
I20260709 12:22:34.586438 31717 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:34.588241 32377 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:34.588660 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:34.596995  3949 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:34.589680  3951 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:34.599076  3951 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:34.595563 31750 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:22:34.600075  3951 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.597483 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:34.602945 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:34.595644  3454 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.604701 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:34.607439  3454 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:34.607197 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.610260 31676 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:34.610451  3983 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:34.611363 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:34.608152  4027 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:34.564175  3988 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:34.608366  3763 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:34.563180  3992 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:34.587864  3946 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:34.580848  4035 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:34.530407  3948 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:22:34.623375 31818 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:22:34.647771 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.584697  3994 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:34.651726 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60378.6us
I20260709 12:22:34.557731  3945 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.042s
I20260709 12:22:34.656189 31677 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
W20260709 12:22:34.656181 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 173664us
I20260709 12:22:34.658246 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.663761 31676 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:34.611868  3738 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:34.626778  3955 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:34.669608 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:34.669723 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.671841 31677 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:34.673089 31677 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:22:34.674850 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:34.654934  3946 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:34.681385 31676 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:34.681273 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:34.688169 31818 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:34.655474  4035 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.667088  3945 consensus_queue.cc:1243] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 9.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:22:34.675670  3955 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.693287  3994 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.038s
I20260709 12:22:34.731119  3955 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:34.731660  3994 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:34.703052  3705 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:34.706208  4010 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.549s	user 0.000s	sys 0.001s
I20260709 12:22:34.691967  3946 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:34.708941  3992 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:34.709358 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:34.737383 31670 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:34.711170 31671 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
W20260709 12:22:34.711802 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.712692 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2358) took 2107 ms. Trace:
I20260709 12:22:34.714253  3988 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.681794  3454 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:34.717737  4031 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:34.754070 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:34.754859 31818 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:34.718400  3723 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:34.696117  4035 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:34.720391 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:34.724246  3731 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:34.766304  3945 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:34.771593  3948 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:34.735513 32373 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:34.738449 31752 rpcz_store.cc:276] 0709 12:22:32.604925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:32.605060 (+   135us) service_pool.cc:224] Handling call
0709 12:22:32.607266 (+  2206us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:33.186047 (+578781us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.712321 (+1526274us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b54002963d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:34.712578 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":891405568}
I20260709 12:22:34.736028  3949 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:34.789227 31671 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:34.792387 31746 pending_rounds.cc:85] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 6
I20260709 12:22:34.792809 31746 pending_rounds.cc:107] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 9.7
I20260709 12:22:34.793131 31746 raft_consensus.cc:2889] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:34.793849 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 112918us
I20260709 12:22:34.796376 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:34.802229 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 204830us
I20260709 12:22:34.803313 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:34.786042  3992 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:34.805845 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:34.807276 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:34.810940 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:34.814558 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:34.815325 31671 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:34.817961 31669 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:34.817188 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:34.819780 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:34.820362 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:34.798202  3988 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:34.825572 31717 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:34.828667 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:34.830716 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:34.832522 31778 heartbeater.cc:646] Failed to heartbeat to 127.29.201.254:37495 (2 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.29.201.254:37495 timed out after 15.000s (SENT)
I20260709 12:22:34.812739  3992 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:34.833788  4047 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:34.834040  3992 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 7, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.834806  4047 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:34.835814  3992 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:34.835853  4047 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:34.813769  3454 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:34.836476  4047 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.836786  3454 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:34.836943  4047 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:34.837791 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:34.837920  3454 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.838835 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:34.822114  3705 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:34.839854  3962 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:34.840186  3705 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.840432  3962 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:34.840608  3705 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:34.841432  3962 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.844358 32377 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:34.846676  3996 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:22:34.847913  3996 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:34.848313  3996 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:34.849200  3996 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.825459  3698 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:34.851433  3698 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.851859  3698 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:34.805114  3949 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.776698  4031 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:34.858948 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:34.863679 32374 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 653) took 1184 ms (client timeout 1508 ms). Trace:
W20260709 12:22:34.864087 32374 rpcz_store.cc:269] 0709 12:22:33.678832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.678979 (+   147us) service_pool.cc:224] Handling call
0709 12:22:34.858820 (+1179841us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:22:34.863610 (+  4790us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2629632}
W20260709 12:22:34.873952 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.827310  3948 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:34.901408 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:34.972152 31675 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:22:34.907328 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.967036 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:34.967422 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:34.855589  4023 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:34.829514  4004 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:34.896643  4021 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
W20260709 12:22:34.908937 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.850487  3996 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:34.864965  3949 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:35.051970 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.094055 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 606) took 1387 ms. Trace:
I20260709 12:22:34.864523  3943 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.116s
I20260709 12:22:35.096148 31748 rpcz_store.cc:276] 0709 12:22:33.706475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.723031 (+ 16556us) service_pool.cc:224] Handling call
0709 12:22:33.752437 (+ 29406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.335413 (+582976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.093720 (+758307us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b54000b90d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:35.093976 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1297572224}
W20260709 12:22:35.071586 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.096123 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.898874  3759 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:22:35.102232 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.103044  3759 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.104585  3759 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:34.919818  4026 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.193s
I20260709 12:22:35.112340 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2304) took 1387 ms. Trace:
I20260709 12:22:35.112970 31820 rpcz_store.cc:276] 0709 12:22:33.724888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.736137 (+ 11249us) service_pool.cc:224] Handling call
0709 12:22:33.818429 (+ 82292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.349869 (+531440us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.111967 (+762098us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b54000fc154. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:35.112249 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1219368192}
W20260709 12:22:35.115382 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:34.976845  3926 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:34.896039  3990 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.125s
I20260709 12:22:34.978189  4023 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:34.934811  3951 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.162s
I20260709 12:22:34.975628  4027 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:35.125309  4046 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.981849  3948 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:35.118391  4032 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:34.901333  3944 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.212s
W20260709 12:22:35.134023 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.122174 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:35.153899 31792 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:35.091138  3952 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.147629 31671 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:34.974557  3991 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:35.160683 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.133826  4023 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:35.171205 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 350875us
I20260709 12:22:35.173362 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:35.179087  4051 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.219s	user 0.001s	sys 0.000s
I20260709 12:22:35.163749  3994 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:35.162153  3945 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:35.099483  4004 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.154686  4046 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:35.201256 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.163852  3952 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.216145 31678 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:35.181267  3948 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:35.161203  4032 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:35.224390  4048 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.307s	user 0.001s	sys 0.000s
I20260709 12:22:35.214709  3994 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:35.227808  3859 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:22:35.253859 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 100032us
I20260709 12:22:35.254685 32370 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:35.259865  3908 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.260525  3908 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:35.264201 32373 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:35.244848  3689 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:35.277077  4047 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:35.269968  3955 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.538s	user 0.012s	sys 0.000s
I20260709 12:22:35.288308  4032 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:35.289422  4054 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
I20260709 12:22:35.268802  4055 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:22:35.278358  4022 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.932s	user 0.000s	sys 0.003s
W20260709 12:22:35.350227 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.254737  4035 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.290175  3698 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:35.282539  3949 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:35.447506  3948 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.467005  3991 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:35.467741 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2321) took 1484 ms. Trace:
I20260709 12:22:35.469105 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2308) took 1742 ms. Trace:
I20260709 12:22:35.471035 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:35.477850 32377 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:35.485946 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.486649  3948 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:35.487756 31674 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:35.487648 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1677) took 1329 ms. Trace:
I20260709 12:22:35.488554 31672 rpcz_store.cc:276] 0709 12:22:34.157983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.185216 (+ 27233us) service_pool.cc:224] Handling call
0709 12:22:35.487434 (+1302218us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b540017c0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:35.487588 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":442419968}
I20260709 12:22:35.490900  3991 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:35.491528  3991 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.492828  4047 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:35.494015  3991 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:35.490288  3763 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:35.495065  4031 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:35.495736  4031 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.495471  3763 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.496263  4031 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:35.498334  3946 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.721s	user 0.006s	sys 0.002s
I20260709 12:22:35.498953  4067 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.502665  4031 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:35.503376  3822 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.507490  3952 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:35.510771  3624 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.511574  4046 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:35.513285  3940 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:35.515480  3698 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:35.515978 31752 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:35.521032  3945 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.522521 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:35.475431  4004 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:35.490921  3926 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:35.506322 31671 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:35.525628 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:35.580478 31750 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:35.525997 31672 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:22:35.526299  3992 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.690s	user 0.011s	sys 0.000s
I20260709 12:22:35.527362 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:35.564105 31746 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:35.531869 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:35.475672 31824 rpcz_store.cc:276] 0709 12:22:33.726246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.789059 (+ 62813us) service_pool.cc:224] Handling call
0709 12:22:33.789538 (+   479us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:33.846620 (+ 57082us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:33.846779 (+   159us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:33.863091 (+ 16312us) log.cc:844] Serialized 26 byte log entry
0709 12:22:34.063932 (+200841us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:34.063979 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:34.064052 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:35.277827 (+1213775us) spinlock_profiling.cc:243] Waited 20 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:35.468810 (+190983us) raft_consensus.cc:1685] finished
0709 12:22:35.468831 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:35.469041 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60700,"raft.queue_time_us":59662,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":10271,"spinlock_wait_cycles":45975296,"thread_start_us":155189,"threads_started":1,"wal-append.queue_time_us":384080}
W20260709 12:22:35.480753  4053 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.307s	user 0.001s	sys 0.000s
I20260709 12:22:35.452347  4055 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:35.599562  4055 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.305368  3955 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.565234  3624 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:35.604559  3955 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:35.475159 31819 rpcz_store.cc:276] 0709 12:22:33.983107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.075389 (+ 92282us) service_pool.cc:224] Handling call
0709 12:22:34.123903 (+ 48514us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:34.427407 (+303504us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:35.111448 (+684041us) log.cc:844] Serialized 14 byte log entry
0709 12:22:35.219140 (+107692us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:35.221102 (+  1962us) log.cc:844] Serialized 26 byte log entry
0709 12:22:35.221341 (+   239us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:35.221393 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.221501 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:35.467496 (+245995us) raft_consensus.cc:1685] finished
0709 12:22:35.467520 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:35.467662 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":453406,"prepare.queue_time_us":505896,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":7088,"raft.queue_time_us":120444,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":16961,"thread_start_us":560008,"threads_started":3,"wal-append.queue_time_us":135230}
I20260709 12:22:35.605247  3624 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:35.605499  4035 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:35.563326  3763 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:35.532542 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:35.482854  3988 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.651s	user 0.009s	sys 0.003s
I20260709 12:22:35.523181  3738 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:22:35.608340  3738 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:22:35.608834  3738 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:35.611090 31820 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:22:35.569298 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:35.612103 31818 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:35.570204 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:35.538656  4062 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:22:35.613349 31827 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:35.575291 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:35.609869  3738 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:35.614414 31826 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:35.614960  3738 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:35.615443 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2312) took 1816 ms. Trace:
I20260709 12:22:35.617036  3962 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:35.615834 31823 rpcz_store.cc:276] 0709 12:22:33.799251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.884556 (+ 85305us) service_pool.cc:224] Handling call
0709 12:22:33.884872 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:33.885338 (+   466us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:22:34.866788 (+981450us) log.cc:844] Serialized 14 byte log entry
0709 12:22:35.281199 (+414411us) log.cc:844] Serialized 14 byte log entry
0709 12:22:35.281427 (+   228us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:35.281903 (+   476us) log.cc:844] Serialized 26 byte log entry
0709 12:22:35.308395 (+ 26492us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:35.308464 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.308556 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:35.491946 (+183390us) spinlock_profiling.cc:243] Waited 5.93 ms on lock 0x7b4800130158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:22:35.615236 (+123290us) raft_consensus.cc:1685] finished
0709 12:22:35.615258 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:35.615388 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1026840,"prepare.queue_time_us":985469,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":62,"raft.queue_time_us":68191,"raft.run_cpu_time_us":3651,"raft.run_wall_time_us":1610668,"spinlock_wait_cycles":13632000,"thread_start_us":412680,"threads_started":2,"wal-append.queue_time_us":384573}
I20260709 12:22:35.553666  4067 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:35.618974 31677 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:35.620296 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:35.620558  4067 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:35.621030 31677 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:35.617460  3962 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:35.621711 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:35.622398  3962 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.575750 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:35.623492  3962 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:35.582370 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:35.529189 31673 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:35.625483 31673 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:35.582907 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:35.545379  4064 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.627004 31822 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:35.583954 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:35.627142  4064 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:35.627987 31821 raft_consensus.cc:2368] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13.
I20260709 12:22:35.585147 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:35.592447 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:35.628819 32376 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:35.629541  4064 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:35.582289  3926 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:35.632143 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:35.553442  3822 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:35.593364 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:35.634994 31672 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:35.635502  3822 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:35.583633  3945 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:35.578194  3705 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.737s	user 0.009s	sys 0.004s
I20260709 12:22:35.636545  3705 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.637713  3945 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:35.638827 32376 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:35.639014  3705 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:35.640542 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.642577 32371 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 12:22:35.643169 32371 pending_rounds.cc:85] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 9
I20260709 12:22:35.643473 32371 pending_rounds.cc:107] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 12:22:35.643748 32371 raft_consensus.cc:2889] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:35.541249  3946 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.648918  3946 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:35.650436 31675 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:35.510270  3908 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:35.583220 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:35.659925 31745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1796) took 1504 ms (client timeout 1658 ms). Trace:
I20260709 12:22:35.664637 32377 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:35.667061 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:35.660385 31745 rpcz_store.cc:269] 0709 12:22:34.155366 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.156875 (+  1509us) service_pool.cc:224] Handling call
0709 12:22:35.659721 (+1502846us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b5400404cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:35.659868 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":248369664}
I20260709 12:22:35.670059 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:35.672075 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:35.672761 31672 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:35.673497 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 90367us
I20260709 12:22:35.675000 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:35.678517 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 138067us
I20260709 12:22:35.685676 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:35.685948 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.688010 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:35.696975 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:35.709168 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.011s	sys 0.020s
W20260709 12:22:35.709685 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.011s	sys 0.021s
W20260709 12:22:35.710090 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 536789us
W20260709 12:22:35.712248 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.719595 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:35.720394 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:35.722693 31671 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:22:35.727597 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:35.728468 32377 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:35.730330 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:35.731146 31821 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:35.734858 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47441.4us
I20260709 12:22:35.734970 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:35.736570 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:35.736889 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:35.737780 31673 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
W20260709 12:22:35.738727 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.716172  3822 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:35.740077  3822 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:35.740404  4026 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:35.740875  4026 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.741225  4026 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:35.741120  3822 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:35.742152 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.743098 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:35.744064 32371 raft_consensus.cc:2410] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 12 because replica has last-logged OpId of term: 10 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 7.
I20260709 12:22:35.654505  3759 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.745765 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:35.754673 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:35.755162 31750 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:22:35.746738 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:35.758881 31747 raft_consensus.cc:2410] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:22:35.759951 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:35.760776 31747 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:35.761698 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:35.678959  3948 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:35.678186  4032 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:35.763906 31746 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
I20260709 12:22:35.764451 31746 pending_rounds.cc:85] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 6
I20260709 12:22:35.764707 31746 pending_rounds.cc:107] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 12:22:35.764956 31746 raft_consensus.cc:2889] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:35.767671 31672 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:35.769220 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:35.769220 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:35.770387 31824 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:35.771301 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:35.772043 31671 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:35.762529 31747 raft_consensus.cc:2410] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 14 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:22:35.773460 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:35.777724 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:35.778693 31673 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:35.782328 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:35.783284 31745 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:22:35.788465 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 49991.1us
I20260709 12:22:35.790131 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:35.794011 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:35.794715 31673 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:35.795698 31677 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
I20260709 12:22:35.721531  4015 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.065s
I20260709 12:22:35.797462 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:35.798203 31822 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:35.787880 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:35.745968  3908 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.057s
I20260709 12:22:35.721269  4064 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.066s
I20260709 12:22:35.745388  3945 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 12:22:35.803110 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:35.782857  4032 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:35.793663  4023 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.621s	user 0.008s	sys 0.007s
I20260709 12:22:35.787812 31821 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:35.815474 31750 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:35.829150 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:35.829679 31675 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
W20260709 12:22:35.830663 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 119491us
I20260709 12:22:35.835916 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:35.836676 31821 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:22:35.839222 31822 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:35.843003 31673 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:35.843153 31671 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:35.846040 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:35.847409 31677 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:35.855729 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:35.828900  3952 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:35.832594 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:35.864372 31671 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:35.868402 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:35.828164  4023 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.861397  3952 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.860421  3763 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:35.836556  4031 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:22:35.870167  3964 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:35.893850 31823 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:35.892550 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 694) took 1832 ms. Trace:
I20260709 12:22:35.877987 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:35.896737 32369 rpcz_store.cc:276] 0709 12:22:34.060404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.096025 (+ 35621us) service_pool.cc:224] Handling call
0709 12:22:34.134198 (+ 38173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:34.336679 (+202481us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.892272 (+1555593us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b54006dd654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:35.892489 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":464261376}
I20260709 12:22:35.906665 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:35.889438  3952 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:35.916512 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:35.918679  3689 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:35.919261 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.919533  3689 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Leader election won for term 8
I20260709 12:22:35.920130  3689 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:35.921317  3689 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.912382 31821 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:22:35.949218 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:35.912789 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:35.950670 31827 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:35.889434  4076 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.896770  3759 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:35.895359  3964 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:35.901906  3988 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:35.956418  3988 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.957803  3552 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:35.958710  3988 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:35.906579  3955 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.948725 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:35.908538  4046 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:35.914089  3946 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:35.914327  4035 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:35.952786 31823 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:35.948796 31826 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: []
W20260709 12:22:35.954989 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.955909  3964 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:35.916738 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.976572 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:35.977363 31827 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:35.979527 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:35.965739 31824 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:35.959375  3955 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:35.897971  3763 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:35.993446 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 189942us
I20260709 12:22:35.994432 31824 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
W20260709 12:22:35.974597  4083 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.078s	user 0.002s	sys 0.000s
I20260709 12:22:35.978217  3946 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:35.998067 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:36.007002 31671 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:36.007364 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:35.974772  4082 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.072s	user 0.000s	sys 0.002s
I20260709 12:22:36.008186 31824 raft_consensus.cc:2376] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14. Re-sending same reply.
I20260709 12:22:36.009366 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:36.002148 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:36.010301 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:36.009898 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:35.974385  4067 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:36.011816 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:36.014810 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:36.017155 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:36.018618 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:36.017066 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:36.024711 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:36.027697 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:35.981659  4035 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.035328 31671 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:36.036823 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:36.037691 31671 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:35.976728  3988 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 12:22:35.911968  4032 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.018016  3992 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:35.970657  4076 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.015741  4055 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.018297  3908 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:36.015470  4000 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:36.016281  4067 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:36.009352  3759 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.049s
I20260709 12:22:35.983088  4046 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:35.999303  3766 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:22:36.059942 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:36.079021 31676 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:36.074272 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 163492us
I20260709 12:22:36.040536  3945 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:36.026515  3991 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:36.080768 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:36.020619  3731 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 12:22:36.057749  4073 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:36.091929 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:36.095109 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78606.1us
I20260709 12:22:36.100399 31675 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.9-10.9]
I20260709 12:22:36.027022  3948 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.056s
I20260709 12:22:36.049526  4071 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:36.082353  4080 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:36.098791  3840 raft_consensus.cc:2764] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election decision vote started in defunct term 13: won
I20260709 12:22:36.081972  4067 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:36.135070  3955 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:36.071481  3693 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.042s
W20260709 12:22:36.120873  4084 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.160s	user 0.000s	sys 0.000s
I20260709 12:22:36.103401 32343 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:36.099064  4074 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:36.123921 31676 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:36.111629 31671 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:36.083827  3945 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.116547  4023 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:36.134145  4098 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
W20260709 12:22:36.115072  4081 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 12:22:36.067565  3992 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.079779  4055 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.106575  4096 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:36.130405 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:36.191346  4086 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
I20260709 12:22:36.191843 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:36.108417  4088 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:36.122908  3991 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:36.121882  4071 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.132814  4080 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:36.150748 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47494.8us
I20260709 12:22:36.152386  3763 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:36.156563  4100 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:36.157574  4002 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:36.157696  3992 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:36.165673  4076 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:36.142876  4074 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.122546  4026 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:36.184240 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 165666us
I20260709 12:22:36.196269 32371 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:36.197278  4096 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:36.147562  3945 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:36.248682 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:36.204141  4046 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:36.207116 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:36.252249 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 211345us
I20260709 12:22:36.252786 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:36.253037 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45989us
I20260709 12:22:36.105319  4101 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.143s
W20260709 12:22:36.194732  3926 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.564s	user 0.008s	sys 0.014s
I20260709 12:22:36.206266  3991 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:36.209507  3946 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:36.204773  4088 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.208401  4100 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.208485  4080 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:36.206378  4071 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:36.218683  4074 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:36.210891  4002 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:36.245786 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:36.200942 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 189184us
I20260709 12:22:36.250113 31751 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:36.252072 31827 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:36.253319 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:36.265956 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:36.260852 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:36.267627 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:36.268287 31823 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:36.254076 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:36.262622 31827 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:36.272442 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:36.273239 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:36.273415 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:36.273938 31818 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:36.273247 32371 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:36.268422 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:36.276034 31675 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:36.276374 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:36.276646 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:36.276449 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:36.278759 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:36.279183 31822 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 12: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:36.279421 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.249459  4096 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:36.296952  4059 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.017s	user 0.007s	sys 0.000s
I20260709 12:22:36.299350 31673 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:36.277494  4102 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:36.299896 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:36.300439  4035 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:36.259989  4002 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.302241  4026 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:36.253279  4108 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:36.331379  4102 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:36.331529  4108 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 8 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:36.276206  4046 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:36.211203  3964 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.279693 32369 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:36.255174  3731 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:36.306105 32372 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 10 index: 11. (index mismatch)
W20260709 12:22:36.312763 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.340744 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:36.344049  3962 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:36.344630 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 684) took 2618 ms. Trace:
I20260709 12:22:36.344798  3962 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
W20260709 12:22:36.302577 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.346808 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:36.347612  3962 raft_consensus.cc:2764] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:22:36.348556  3964 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:36.349290  3920 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.349411 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:36.350344  3962 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:36.305965  4055 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:36.353585 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:36.315025 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:36.256978  4100 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:36.354373 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2309) took 2627 ms. Trace:
I20260709 12:22:36.354749  3945 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.356390  4088 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:36.354882 31825 rpcz_store.cc:276] 0709 12:22:33.726547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.801299 (+ 74752us) service_pool.cc:224] Handling call
0709 12:22:33.835817 (+ 34518us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:33.863514 (+ 27697us) raft_consensus.cc:1534] Early marking committed up to index 6
0709 12:22:34.860662 (+997148us) spinlock_profiling.cc:243] Waited 2.5 ms on lock 0x7b18003a1ec0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:22:34.860985 (+   323us) log.cc:844] Serialized 14 byte log entry
0709 12:22:35.541397 (+680412us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:35.541941 (+   544us) log.cc:844] Serialized 26 byte log entry
0709 12:22:35.542456 (+   515us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:35.542502 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:35.548679 (+  6177us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:36.250566 (+701887us) spinlock_profiling.cc:243] Waited 9.32 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:36.354136 (+103570us) raft_consensus.cc:1685] finished
0709 12:22:36.354162 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.354305 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13157,"prepare.queue_time_us":1003081,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":106198,"raft.run_cpu_time_us":286,"raft.run_wall_time_us":20865,"spinlock_wait_cycles":27173376,"thread_start_us":678986,"threads_started":2,"wal-append.queue_time_us":385462}
I20260709 12:22:36.254998  3926 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.315754 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:36.255972  3991 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.360440  3926 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:36.319857 31744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 620) took 1751 ms (client timeout 2082 ms). Trace:
W20260709 12:22:36.361703 31744 rpcz_store.cc:269] 0709 12:22:34.568307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.576583 (+  8276us) service_pool.cc:224] Handling call
0709 12:22:36.319639 (+1743056us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b54002f71d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:36.319791 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":755463040}
I20260709 12:22:36.320241 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:36.277578  3621 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:36.363355  3621 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:36.362948  3952 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.363831  3621 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:36.363857 31748 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:36.366801 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.368078  3945 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:36.368306 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:36.369824 32370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 651) took 2692 ms (client timeout 1560 ms). Trace:
I20260709 12:22:36.323426 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:36.370294 32370 rpcz_store.cc:269] 0709 12:22:33.677540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.677763 (+   223us) service_pool.cc:224] Handling call
0709 12:22:36.369624 (+2691861us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b5400646c54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:36.369736 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3585209088}
I20260709 12:22:36.320503 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:36.372287 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2424) took 1703 ms. Trace:
I20260709 12:22:36.325634 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:36.373078 31823 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:36.379904 31750 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:36.380383 31750 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:36.313583  3822 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:36.329736  3763 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.381310  3763 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.383672  3763 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:36.267378  4067 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.386061  4067 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:36.336503  4026 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.386936  4026 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.389319  4026 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:36.337177  4023 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.390060  4023 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.391327  4080 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.392168  4046 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:36.392534  4023 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:36.393467  4046 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.393949 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:36.354403 31822 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:36.395219  4080 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:36.363044 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 626) took 1436 ms. Trace:
I20260709 12:22:36.395848 31752 rpcz_store.cc:276] 0709 12:22:34.926670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.926866 (+   196us) service_pool.cc:224] Handling call
0709 12:22:34.927285 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.563831 (+636546us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.362793 (+798962us) spinlock_profiling.cc:243] Waited 588 ms on lock 0x7b5400213ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:36.362974 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1353412864}
I20260709 12:22:36.396592  4055 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:36.397271 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.397328  4055 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:36.398011 31752 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:36.363044 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.336694 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:36.399355 31749 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:36.372747 31674 rpcz_store.cc:276] 0709 12:22:34.668888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.783644 (+114756us) service_pool.cc:224] Handling call
0709 12:22:35.487557 (+703913us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.488290 (+   733us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.372097 (+883807us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b540017c0d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:36.372226 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1588640512}
I20260709 12:22:36.338389  3759 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.402541  3759 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.396044 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:36.349969  3920 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.405646  4067 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:36.406111  4067 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.312474 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:36.406296  3759 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:36.406479  4067 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:36.337702  4102 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.326887  3552 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.067s
I20260709 12:22:36.399427 31744 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 9: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:36.338981  3723 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:36.337450  4108 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.372786 31675 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:36.401574 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.423007  4108 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:36.429829  4074 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:36.401952 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.433596 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1800) took 2238 ms. Trace:
I20260709 12:22:36.434120 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:36.435195 32371 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:22:36.436174 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:36.357851 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:36.372313 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.437222 31821 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:36.410487 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:36.411432 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 108259us
I20260709 12:22:36.410713 31823 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:36.443874 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:36.413522 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60032.5us
I20260709 12:22:36.399421  3952 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:36.392668 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:36.418975 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 95634.9us
I20260709 12:22:36.280277 31824 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:36.447105 31751 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:36.434186 31753 rpcz_store.cc:276] 0709 12:22:34.195575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.454489 (+258914us) service_pool.cc:224] Handling call
0709 12:22:34.474546 (+ 20057us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:34.483257 (+  8711us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.433383 (+1950126us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x7b54001a59d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:36.433531 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":45067648}
I20260709 12:22:36.345115 32372 rpcz_store.cc:276] 0709 12:22:33.726568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:33.726897 (+   329us) service_pool.cc:224] Handling call
0709 12:22:33.752516 (+ 25619us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:36.342498 (+2589982us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.344333 (+  1835us) spinlock_profiling.cc:243] Waited 2.55 s on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:36.344543 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5871493504}
I20260709 12:22:36.381186 31750 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.447966 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:36.450138 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.458343 31750 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:36.458678 31752 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:36.448019 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:36.462002 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:36.463668 31675 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:22:36.463985 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.465404 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.467274 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:36.467770 31822 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:36.463387 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:36.468237 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:36.470822 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:36.472291 31675 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 12: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:36.474341 31824 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:36.476580 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.431244  3822 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260709 12:22:36.477665 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:36.421479  4032 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:36.467094 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:36.477142 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.499119 31752 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 8.
I20260709 12:22:36.500150 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:36.484925 31821 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:36.486538 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:36.503337 32371 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:36.448283  3552 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:36.491513 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.507933 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.509253 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.509995 31748 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:36.491959 31750 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 11 index: 10. (index mismatch)
I20260709 12:22:36.511478 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.512704 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.513808 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.492187 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:36.515218 31745 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:36.515411 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:36.516222 31821 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:36.519553  3552 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.520085  3946 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:36.520578 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:36.485822 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:36.520754  3946 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.520412  3552 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.460906  4074 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:36.523222 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:36.523816 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.524646 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:36.524732 31748 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:36.521673 32371 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:36.514540 31750 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:36.525619  3552 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:36.528788 31824 raft_consensus.cc:2376] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13. Re-sending same reply.
I20260709 12:22:36.530005 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:36.467324  4080 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.027s
I20260709 12:22:36.464377  3723 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:36.532476 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:36.468786  3938 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:36.422166  4071 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.534423 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:36.546896 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:36.547900 31824 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:22:36.500226  4004 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:36.535183 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:36.587270 31752 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:36.587975 31744 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:36.588901 31827 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:36.589432 31744 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 12:22:36.589794 31744 pending_rounds.cc:85] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 7
I20260709 12:22:36.590010 31744 pending_rounds.cc:107] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:22:36.590237 31744 raft_consensus.cc:2889] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:36.536001 31748 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:36.466956  3964 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.073s
W20260709 12:22:36.534825 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57255.2us
I20260709 12:22:36.592624 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:36.476500  3759 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.068s
I20260709 12:22:36.540555 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:36.594897  3759 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:36.595450  3759 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:36.595104  4032 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:36.595978  3759 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:36.598018 31676 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:22:36.598253 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:36.597908 31678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2429) took 1812 ms (client timeout 1708 ms). Trace:
W20260709 12:22:36.600492 31678 rpcz_store.cc:269] 0709 12:22:34.785511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.830344 (+ 44833us) service_pool.cc:224] Handling call
0709 12:22:36.597532 (+1767188us) spinlock_profiling.cc:243] Waited 385 ms on lock 0x7b5400229a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:36.597815 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":884494080}
I20260709 12:22:36.603454 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:36.594929 32371 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:36.605186 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:36.605917 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:36.541857 31745 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:36.543421 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.544539 31751 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:36.461436  3909 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.070s
I20260709 12:22:36.534790 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:36.620251 31825 raft_consensus.cc:2376] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13. Re-sending same reply.
I20260709 12:22:36.548743 32372 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:36.622922 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:36.623880 32371 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:36.491204  4064 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:36.495973  3454 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:36.530337  3552 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.021s
I20260709 12:22:36.551899 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:36.639767 31824 raft_consensus.cc:2376] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13. Re-sending same reply.
W20260709 12:22:36.538250  3705 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 11 current term: 11 is higher.
I20260709 12:22:36.552958  3859 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:36.555971 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:36.557104 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2337) took 1333 ms. Trace:
I20260709 12:22:36.650024 31820 rpcz_store.cc:276] 0709 12:22:35.224269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.224477 (+   208us) service_pool.cc:224] Handling call
0709 12:22:35.245817 (+ 21340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:35.611558 (+365741us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.556830 (+945272us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b540043e9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:36.557043 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":839059712}
I20260709 12:22:36.558933 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:36.651657 31677 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:36.562665 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:36.652655 31823 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 9.
I20260709 12:22:36.563901 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 650) took 2061 ms. Trace:
I20260709 12:22:36.653075 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:36.653498 31669 rpcz_store.cc:276] 0709 12:22:34.502259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:34.502747 (+   488us) service_pool.cc:224] Handling call
0709 12:22:34.503153 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:34.818404 (+315251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.563628 (+1745224us) spinlock_profiling.cc:243] Waited 314 ms on lock 0x7b54003921d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:36.563836 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":723330304}
I20260709 12:22:36.567629  3920 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:36.571981 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:36.572419  3966 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:36.550058  4098 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:36.576045 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:36.629643  3844 raft_consensus.cc:2764] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:22:36.638938  3909 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:36.481689  3738 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.067s
I20260709 12:22:36.580717  4071 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:36.581058 31821 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:36.581429 31819 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:36.583432  3938 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:36.499130  3822 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:36.585186 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:36.612035 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:36.613366 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:36.611686 31753 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:36.618896  3689 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:36.628175 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:36.545411  3952 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:36.633157 31642 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:36.642164 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:36.575482  3945 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:36.658576 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:36.659997 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 88097.2us
I20260709 12:22:36.587332  3086 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:36.664829 31747 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:36.576850 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:36.655874  3859 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:36.629040  3849 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:36.577706 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:36.670905 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.629266  4032 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:36.707280 31751 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:36.656608  3966 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:36.708457 31749 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:36.663527  4098 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:36.673009  3909 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:36.741525 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 83048.7us
I20260709 12:22:36.711463 31744 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 13: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:36.647383  3454 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:36.762230 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 150285us
I20260709 12:22:36.765285  3621 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.742276 31747 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:22:36.821236 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:36.709249 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 76205us
W20260709 12:22:36.746012 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 132751us
I20260709 12:22:36.742012  3086 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:36.640516  4064 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.771498  4071 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:36.659961  4016 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:22:36.671056  3953 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:36.709020  3844 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:36.775978 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.790609 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 819) took 1252 ms. Trace:
I20260709 12:22:36.781697 31749 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:36.792840 31751 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:36.673467  3908 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.108s
I20260709 12:22:36.860075 32375 rpcz_store.cc:276] 0709 12:22:35.538520 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:35.538772 (+   252us) service_pool.cc:224] Handling call
0709 12:22:35.539132 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:36.115956 (+576824us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:36.127837 (+ 11881us) log.cc:844] Serialized 14 byte log entry
0709 12:22:36.200595 (+ 72758us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:36.200669 (+    74us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:36.200719 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.790345 (+589626us) spinlock_profiling.cc:243] Waited 576 ms on lock 0x7b54004272d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:36.790385 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:36.790540 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2688,"prepare.queue_time_us":243,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":59,"raft.queue_time_us":2985,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":2351,"spinlock_wait_cycles":1325668736,"thread_start_us":69472,"threads_started":1,"wal-append.queue_time_us":69987}
I20260709 12:22:36.763583  4023 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:22:36.770546 31752 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:36.707126  3822 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 8 FOLLOWER]: Advancing to term 9
W20260709 12:22:36.838868 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 319217us
I20260709 12:22:36.775120 31826 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 10.
I20260709 12:22:36.839335 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:36.841265 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:36.918159 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 77010.6us
I20260709 12:22:36.920977 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:36.925278 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:36.842695 31744 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260709 12:22:36.773124  3763 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:22:36.852111 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.745252 31672 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:36.854872 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:36.716842  3689 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.788021  4101 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:36.836161  3723 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:36.862368 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:36.943821 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:36.944607  4116 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
W20260709 12:22:36.946872 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 84636us
I20260709 12:22:36.917122  3844 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:36.849923  3938 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.799762  4100 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.916239 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:36.847930  3991 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:22:36.929204 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 89971.7us
I20260709 12:22:36.881850  3953 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:36.922158  4071 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:22:36.927975 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 73239.7us
W20260709 12:22:36.946094  4108 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.523s	user 0.012s	sys 0.000s
I20260709 12:22:36.935052  4102 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:36.877862  3909 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:36.962725 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46630.1us
I20260709 12:22:36.945515  3763 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:36.976941 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:36.966547 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:36.958389  4096 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.625s	user 0.007s	sys 0.003s
W20260709 12:22:36.971094 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45887.6us
I20260709 12:22:36.931627  3705 raft_consensus.cc:2764] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:22:36.946879  3849 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:36.980407 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:36.915999  3086 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.982673  3689 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:36.996541 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:36.987425  4118 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
W20260709 12:22:36.988266  4115 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:22:36.961244  3844 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:36.947021  4101 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:36.996882 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:37.015856 31717 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:37.000982 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:37.010851 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:36.978722  4046 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:37.096601  4046 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:37.097133  4046 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:36.953224  4023 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:36.972373  4121 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:22:37.103219  4120 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
I20260709 12:22:37.103279  4023 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:37.107929  4023 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.112361  4100 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:37.071882  3953 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.012332  3689 raft_consensus.cc:2764] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election decision vote started in defunct term 10: won
I20260709 12:22:37.012185  4004 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:37.016498  3086 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:37.024374  4088 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:37.025151  4025 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:37.001317  4096 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:37.067803  4119 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.147s	user 0.001s	sys 0.001s
W20260709 12:22:37.042963 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:36.999115  4108 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:37.002981  4091 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:37.084923  4114 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.238s	user 0.000s	sys 0.001s
I20260709 12:22:37.043895  3621 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:37.197100 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 196194us
I20260709 12:22:37.198305 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:37.094673  4123 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:22:37.016314  3964 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20260709 12:22:37.015167  4102 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.071s
W20260709 12:22:37.123306 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.025455  3705 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.068s
W20260709 12:22:37.216792 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 205972us
W20260709 12:22:37.129359 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 127189us
I20260709 12:22:37.063560  4026 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 12:22:37.178174  3766 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.174645  3908 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.233259  3908 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.235662  3908 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:37.068342  3759 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.244917  3759 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:37.085527  4000 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.085s
I20260709 12:22:37.246565  4000 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:37.247135  4000 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.249197  4000 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:37.119982  3454 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:37.199810 31745 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.088s	sys 0.148s
W20260709 12:22:37.250399 31745 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.088s	sys 0.149s
I20260709 12:22:37.250723 31746 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:37.121272  4064 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:37.210873 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.212196 31670 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:37.253259  3966 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.249740  4000 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.005s
I20260709 12:22:37.171643  4121 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:37.175650  3849 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:37.216166  4102 consensus_queue.cc:1243] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 13.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 12:22:37.127930 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 117137us
I20260709 12:22:37.194077  4088 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:37.273135 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:37.231559 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:37.262177 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:37.263406 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.175107  3990 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.050s
I20260709 12:22:37.207746  3981 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:37.281749 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:37.267864  4122 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.243s	user 0.000s	sys 0.001s
I20260709 12:22:37.120245  3763 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.296646 31820 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:22:37.285544 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:37.303943 31821 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:22:37.222730  3621 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:37.291817 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:22:37.293278 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61842.1us
I20260709 12:22:37.301059 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:22:37.283541  4123 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:22:37.306582 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:37.283833  3849 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:37.300190  3845 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:37.263751  4091 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.044s
I20260709 12:22:37.295774  4064 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:37.294905  3946 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:37.271696  4108 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 12:22:37.283015  4102 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:37.308261  3621 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.265763  3766 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.351402  4102 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:37.323654  3981 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:37.353469  3981 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:37.354039  4102 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:22:37.354818  3621 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:37.354486  3981 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.358213  3723 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:37.358989  3723 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:37.309947  4132 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:37.363070  4132 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.363480  4132 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:37.328989  4067 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.910s	user 0.005s	sys 0.008s
I20260709 12:22:37.313447  3840 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:37.323482  3908 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:37.369683 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2338) took 1122 ms. Trace:
I20260709 12:22:37.370499 31821 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:37.370250 31819 rpcz_store.cc:276] 0709 12:22:36.243090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.243358 (+   268us) service_pool.cc:224] Handling call
0709 12:22:37.364975 (+1121617us) spinlock_profiling.cc:243] Waited 322 ms on lock 0x7b5400447fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:37.365179 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":740298624}
I20260709 12:22:37.314112  3964 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.053s
I20260709 12:22:37.371863  3844 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:37.368235  4064 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:37.369112  4101 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:37.372330  3908 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.420881 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2376) took 1292 ms. Trace:
I20260709 12:22:37.317130  4121 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.421468 31750 rpcz_store.cc:276] 0709 12:22:36.128886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.526176 (+397290us) service_pool.cc:224] Handling call
0709 12:22:36.526615 (+   439us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:36.527211 (+   596us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:36.527241 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:36.528011 (+   770us) log.cc:844] Serialized 52 byte log entry
0709 12:22:36.822877 (+294866us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:36.822921 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:36.822994 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:37.420534 (+597540us) raft_consensus.cc:1685] finished
0709 12:22:37.420568 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.420813 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":94,"raft.run_cpu_time_us":83,"raft.run_wall_time_us":84,"thread_start_us":294410,"threads_started":1,"wal-append.queue_time_us":446939}
I20260709 12:22:37.424273 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2397) took 1121 ms. Trace:
I20260709 12:22:37.371747  4067 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.411828  3908 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:37.404448  3849 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.409894  3621 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.371965  4088 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:37.452275 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 234514us
I20260709 12:22:37.457384 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:37.454869 31745 rpcz_store.cc:276] 0709 12:22:36.302395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.612422 (+310027us) service_pool.cc:224] Handling call
0709 12:22:36.612738 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:36.612976 (+   238us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:37.250845 (+637869us) log.cc:844] Serialized 14 byte log entry
0709 12:22:37.423890 (+173045us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:37.423954 (+    64us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:37.424006 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.424076 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.424210 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":62607,"prepare.queue_time_us":225887,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":7577,"thread_start_us":747568,"threads_started":2}
I20260709 12:22:37.426592  4025 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:37.434072  4096 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:37.468832 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:37.462739  3859 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.659s	user 0.010s	sys 0.003s
I20260709 12:22:37.464032  4046 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.470517  4064 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.479521  3859 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:37.509977 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 218285us
I20260709 12:22:37.508399  3766 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:37.510761  3940 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
W20260709 12:22:37.532403 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 220516us
I20260709 12:22:37.537595 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:37.538389 31826 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:37.543898 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:37.545125 32370 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:37.548235 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:37.549212 31821 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:37.550676 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:37.551826 31819 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:37.553475 31822 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:37.561882 31826 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:37.523010  3845 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:37.571364 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:37.575013 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 357002us
I20260709 12:22:37.536856  4147 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:37.580524  3938 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:37.585928 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:37.573781  4067 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:37.587018 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:37.589125 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:37.581729  4046 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:37.583127  4147 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.562810  4025 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:37.593758  4136 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:22:37.593305  3845 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:37.594375  4135 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 12:22:37.610754  4067 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:37.595067  4143 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:22:37.595665  4145 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:22:37.598312  3859 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:37.591128  3946 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:37.598012  3723 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.586701  4102 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.585140  3621 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:37.636538 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:37.644608 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:37.645310 31677 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:37.608179  4147 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:37.647217 31671 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:22:37.650858  4096 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.651152 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:37.651618  4096 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.652518 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:37.653141 31678 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:37.654183  3698 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:37.654682  3698 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:37.655606  4132 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.620769  3723 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:37.590615  3938 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.624365  4002 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:37.626912 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 635) took 1146 ms. Trace:
W20260709 12:22:37.628515  3822 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.707s	user 0.013s	sys 0.001s
I20260709 12:22:37.630533  3086 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.631491 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:37.633198 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:37.634574 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:37.656136  4151 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:37.643666  4139 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:37.638556  4102 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.650851 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:37.643905  4046 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.014s
W20260709 12:22:37.658215  4126 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.482s	user 0.001s	sys 0.001s
I20260709 12:22:37.658959 31671 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:37.659008  3689 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:37.659500 31822 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:37.646811  3859 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.662103 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:37.704777 31823 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:37.663008 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:37.629855  3908 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.663434  4096 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:37.665115 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:37.666311  3946 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260709 12:22:37.609577 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 140824us
I20260709 12:22:37.679912 31678 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
W20260709 12:22:37.684056 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:37.686337 31752 rpcz_store.cc:276] 0709 12:22:36.480671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.499888 (+ 19217us) service_pool.cc:224] Handling call
0709 12:22:37.626682 (+1126794us) spinlock_profiling.cc:243] Waited 86.2 ms on lock 0x7b54001ee1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:37.626841 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":198369664}
W20260709 12:22:37.689323  3952 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.505s	user 0.004s	sys 0.004s
I20260709 12:22:37.690541  3086 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:37.694823 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 63471.2us
I20260709 12:22:37.692035 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:37.692461 31669 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:37.689947  4162 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.710085 32370 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:37.709580 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:37.714722 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:37.715000 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:37.716444 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:37.717406 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 446040us
I20260709 12:22:37.702978  3689 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:37.703531  4071 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:37.721593 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:37.724691  3086 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:37.712746  3952 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.004s	sys 0.005s
I20260709 12:22:37.780205 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:37.727672  3086 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:37.731120  4158 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.004s
I20260709 12:22:37.702122  4139 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.696388  3822 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.719982  4162 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.724107  4025 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:37.716584  3723 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:37.655650  3698 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.809657 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:37.803514 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:37.737191  4067 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:37.738905  3938 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:37.697475  4151 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.741044  4102 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:37.821573 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:37.822060 31677 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:37.728109  4046 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:37.718806  4168 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:37.748685  4141 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:37.748909  3990 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:22:37.751611  3766 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:37.746790  3689 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.767726  4098 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.716131  4101 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.818876  3991 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:37.772773  3908 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:37.774540  4132 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:37.786856  4138 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:22:37.788407 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2464) took 1256 ms. Trace:
I20260709 12:22:37.836091 31670 rpcz_store.cc:276] 0709 12:22:36.532394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.624933 (+ 92539us) service_pool.cc:224] Handling call
0709 12:22:37.788226 (+1163293us) spinlock_profiling.cc:243] Waited 558 ms on lock 0x7b540022b354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:37.788336 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1283682048}
I20260709 12:22:37.834050 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:37.798604 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:37.837656 31752 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:37.800180 31746 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:37.800091 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:37.799434 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:37.845374 31750 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:37.808029 32375 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:22:37.808595 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:37.847054 31745 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:22:37.736608 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:37.858800 31752 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:37.809963 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:37.819751 31823 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:37.820290  3086 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:22:37.821085 31675 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 9: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:37.822193 32372 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:37.711110  3859 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:37.824057 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 109142us
I20260709 12:22:37.824816  4141 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:37.808682  3723 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:37.808447  4025 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:37.825392 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:37.914500 32377 raft_consensus.cc:2368] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11.
I20260709 12:22:37.835143 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:37.795584 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:37.828254  3698 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:37.901635 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:37.903769 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:37.901639 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:37.916746 32374 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:37.824883  3952 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.076s
I20260709 12:22:37.925263 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:37.836071  4138 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:37.925367 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 129884us
I20260709 12:22:37.931891  4138 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:37.933367 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:37.934849  4102 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:37.935314  4102 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:37.935672  4102 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:37.935935 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:37.938926  4055 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:37.939662  4055 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:37.941211  4169 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:37.934878  4000 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.007s
I20260709 12:22:37.840299  4091 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:37.945530 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:37.898921  4088 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:37.913403  3849 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:38.009366  3849 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.848730  4098 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:38.012115  3849 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:38.013293  4139 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:37.904836  4046 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.014585  4046 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.938926 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:37.947712  4096 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:37.954012  4121 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:37.956305  3621 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:37.958315  3966 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:37.956326  4004 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:38.041122 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:37.963635 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2457) took 1497 ms. Trace:
I20260709 12:22:37.963311  4138 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:37.804831 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:37.970184 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:38.076968 31822 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:37.978276 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:38.078161 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 99992.3us
I20260709 12:22:38.025938  3840 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:37.984973  3738 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:37.911549  3723 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:37.922602  4025 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:37.911844  4067 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
I20260709 12:22:37.988469  3859 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:37.993919  4147 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:37.996555  3822 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:37.841600  4168 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.156s
I20260709 12:22:37.906481  3689 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:37.999305  4162 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:37.999295 31575 catalog_manager.cc:5697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 5 to 9, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 9 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:38.003597 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:38.017849 31670 raft_consensus.cc:2410] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 15 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:22:37.830152  3766 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.025921 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:38.128448 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:38.027906 32372 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:38.028914 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:38.129850 31827 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:37.835623  4101 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:38.134894 31824 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:22:37.960106  3845 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:38.067974 31674 rpcz_store.cc:276] 0709 12:22:36.466307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.466528 (+   221us) service_pool.cc:224] Handling call
0709 12:22:36.497284 (+ 30756us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:36.880859 (+383575us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:36.880890 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:36.880938 (+    48us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:36.880977 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.963370 (+1082393us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b54003917d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:37.963423 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:37.963568 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":876223872}
I20260709 12:22:38.069880 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:38.075281 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:38.008764  4088 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:38.080256 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:38.152901 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 72743.5us
I20260709 12:22:37.982081 31823 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:38.078228  4169 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.034772  4096 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:38.125394  4178 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:38.086540 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:38.167284 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:38.167842 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 81403.6us
I20260709 12:22:38.168820 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:38.043638  4046 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:38.095623 31678 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:38.168053 31674 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:38.100687 31669 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:22:38.086575  3946 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:38.103453 31677 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:38.104197 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:38.066193  4121 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:38.065706  4004 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:38.106372 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.188320 31674 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:38.189750  4138 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:38.192173 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:38.171633  4169 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:38.168401  4096 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:38.194880 32376 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.11-10.11]
I20260709 12:22:38.196807  4121 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:38.107191 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 201034us
W20260709 12:22:38.089632  4145 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
I20260709 12:22:38.088330  3763 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.199196 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:38.107837 31822 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:38.117131 31820 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:38.090543  3738 consensus_queue.cc:1243] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:22:38.017807  4091 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:38.133693 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:38.135957  4101 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:37.836733  3991 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:38.138213 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:38.144610 31824 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:22:38.156975 31825 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:38.088220  3840 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:38.211607 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 127328us
I20260709 12:22:38.084565 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:38.173157 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:38.220796 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50151.2us
I20260709 12:22:38.176681 31678 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:22:38.160976  4088 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:38.202030 32374 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:22:38.204731 31827 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.11-10.11]
W20260709 12:22:38.205277 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.200963 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:38.165187  4000 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:38.209378 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:38.183259  3940 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:38.186286  3698 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:38.180508  4178 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.248082 31825 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:38.225558 31678 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:22:38.236709 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:38.246703 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:38.255003 31669 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:38.255535 31678 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:38.199363  3763 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:38.210107  3946 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:38.250767  3985 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.008s
I20260709 12:22:38.250798  3981 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:38.252135  4002 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.559s	user 0.007s	sys 0.004s
W20260709 12:22:38.259014 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.258888  3738 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.259406  3086 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:38.259474  4184 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:38.209789  4173 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:38.228184  4168 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:38.264936  3981 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:38.266196 32369 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:38.266562  4175 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:38.330963  4175 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:38.332698  4178 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:38.271474  4147 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:38.274659  4091 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:38.277604  3845 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:38.278541  3766 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:38.235185  4000 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.247465  3698 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.304207 31825 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:22:38.188383  4004 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:38.257423  4074 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:38.304790  4173 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.326836  4130 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.306659  4168 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:38.340369 31820 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
W20260709 12:22:38.372020 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 188061us
I20260709 12:22:38.489059 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:38.332491  4175 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.330472  4025 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.283846  3985 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:22:38.649488 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.681085 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:38.661156 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:38.747056 31676 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
W20260709 12:22:38.747593 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 254647us
W20260709 12:22:38.756140 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 141872us
I20260709 12:22:38.609599  4185 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:38.619251  3698 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:38.305558  3738 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.613314  4000 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:38.611433  3990 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.756832 31825 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:22:38.342110  4102 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:38.618556  4173 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:38.829106  3990 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.640416  4130 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.801151  3991 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:38.804512 31676 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:38.808199  3940 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:38.810493  3738 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:38.816051  4025 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:38.816556  4091 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:38.640786  4074 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.631503  4147 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.630771  4168 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:38.818776 32375 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:38.818948  3689 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:38.821193  4133 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.437s	user 0.003s	sys 0.011s
I20260709 12:22:38.822793 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:38.873788 31827 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:38.823004  4004 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:38.874758  3689 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.877307  3689 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:38.823478 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:38.879269 31820 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:22:38.823843  3983 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.880347  3983 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:38.881189  3723 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.776s	user 0.000s	sys 0.008s
I20260709 12:22:38.881640  3723 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.637988  3985 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:38.883630  3985 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:38.756412  3946 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:38.830868 31824 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:38.818910  3763 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:38.799288  4191 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 9.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.006s
I20260709 12:22:38.841125 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 706) took 1008 ms. Trace:
I20260709 12:22:38.816604 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:38.890204  4191 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:38.306442  4184 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.890738  4191 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:38.891737  4191 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 10.11, Last appended by leader: 11, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:38.892421  4188 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.642s	user 0.002s	sys 0.000s
I20260709 12:22:38.892719  4184 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:38.830338  4102 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:38.639103  3981 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:38.840914  3990 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:38.783210  4088 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.535s	user 0.009s	sys 0.000s
I20260709 12:22:38.870918 32375 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:38.894145  3990 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:38.893931  4147 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.894899  3990 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:38.895879  3990 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 9.10, Last appended by leader: 10, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.896533  4147 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:38.892479  4096 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.697s	user 0.005s	sys 0.006s
I20260709 12:22:38.890017 32369 rpcz_store.cc:276] 0709 12:22:37.833176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.833316 (+   140us) service_pool.cc:224] Handling call
0709 12:22:38.840933 (+1007617us) spinlock_profiling.cc:243] Waited 342 ms on lock 0x7b54004cfed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:38.841066 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":787725568}
I20260709 12:22:38.897727  4096 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:38.868867 31670 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:38.899597 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:38.899941  4096 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:38.901820 31670 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:38.869921  4091 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:38.889957  3966 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.813s	user 0.004s	sys 0.004s
I20260709 12:22:38.904421  4091 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:38.882884 31744 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 11: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:38.905269 32373 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 12:22:38.905651 32373 pending_rounds.cc:85] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 9
I20260709 12:22:38.905923 32373 pending_rounds.cc:107] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 12:22:38.906157 32373 raft_consensus.cc:2889] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:38.906724 31675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2462) took 2406 ms (client timeout 1801 ms). Trace:
W20260709 12:22:38.907187 31675 rpcz_store.cc:269] 0709 12:22:36.499970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.524319 (+ 24349us) service_pool.cc:224] Handling call
0709 12:22:38.906443 (+2382124us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b540017d9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:38.906617 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2979008256}
I20260709 12:22:38.907697 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 681) took 1070 ms. Trace:
I20260709 12:22:38.907526  4173 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.908126 32373 rpcz_store.cc:276] 0709 12:22:37.837024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.837231 (+   207us) service_pool.cc:224] Handling call
0709 12:22:37.908063 (+ 70832us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:38.906838 (+998775us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.907398 (+   560us) spinlock_profiling.cc:243] Waited 997 ms on lock 0x7b540031d654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:38.907576 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2292490112}
I20260709 12:22:38.909520 31672 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 8. Current term is 9. Ops: []
I20260709 12:22:38.911028 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1710) took 1277 ms. Trace:
I20260709 12:22:38.911644  4074 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:38.912549  3723 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:38.894316  4088 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:38.916646  4088 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:38.916007  4141 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:38.890749 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 74233.7us
I20260709 12:22:38.917922  4141 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:38.918324 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:38.920535  4174 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:38.921140  4174 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:38.922950  4140 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:38.922812  3698 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.871795  4032 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:38.924950  4032 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:38.925947  3698 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:38.926244  4032 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.927112  4000 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:38.928239 31826 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:38.911407 31672 rpcz_store.cc:276] 0709 12:22:37.633930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.634065 (+   135us) service_pool.cc:224] Handling call
0709 12:22:38.909320 (+1275255us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:38.909974 (+   654us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:38.910845 (+   871us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b540017d9d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:38.910972 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2932090624}
I20260709 12:22:38.893224  4191 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:38.932991 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:38.946808 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:38.958272 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:38.964273 31678 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.034s	sys 0.061s
I20260709 12:22:38.964135 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:38.878381  4151 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:38.968158 31825 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:22:38.922013  4174 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:38.972766 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:38.940105  3983 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:38.960474  4102 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.959903  3859 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.960232  3991 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:38.983448 31827 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:22:38.844480  4185 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:39.015802  4190 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.459s	user 0.003s	sys 0.000s
W20260709 12:22:39.020061 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:39.022459 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.022533  3698 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.031694 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:39.031711 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:38.964886 31678 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.661s	user 0.034s	sys 0.061s
I20260709 12:22:39.039801 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:39.041832 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:39.042699 31752 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:39.045819 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:39.049573 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:39.052078 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:39.053784 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1824) took 2706 ms. Trace:
I20260709 12:22:39.054355 31753 rpcz_store.cc:276] 0709 12:22:36.347590 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:36.707420 (+359830us) service_pool.cc:224] Handling call
0709 12:22:36.707829 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:36.708185 (+   356us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:36.708210 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:36.764370 (+ 56160us) log.cc:844] Serialized 26 byte log entry
0709 12:22:37.128791 (+364421us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:37.128847 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:37.128947 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:38.820439 (+1691492us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:39.036111 (+215672us) raft_consensus.cc:1685] finished
0709 12:22:39.036147 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:39.053690 (+ 17543us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":88,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":51082,"spinlock_wait_cycles":38282112,"thread_start_us":281207,"threads_started":1,"wal-append.queue_time_us":498129}
I20260709 12:22:39.073680 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:39.078159 31745 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 12:22:39.082912 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 459379us
I20260709 12:22:39.087242 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:39.090003 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:39.094115 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:39.096518 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 64950.4us
I20260709 12:22:38.998224  4169 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:39.105451 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:38.968896 31644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.029s	sys 0.063s
I20260709 12:22:39.062956  4195 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:39.099835 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:39.116452 31744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 645) took 1313 ms (client timeout 1568 ms). Trace:
W20260709 12:22:39.117062 31744 rpcz_store.cc:269] 0709 12:22:37.803152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.803278 (+   126us) service_pool.cc:224] Handling call
0709 12:22:39.116174 (+1312896us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54000f3cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:39.116366 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2445446400}
I20260709 12:22:39.100801 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:39.113766 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:39.103724 31751 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:22:39.120029 31671 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 12:22:39.108815 31644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.805s	user 0.030s	sys 0.063s
I20260709 12:22:39.110831 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:39.121308 31749 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
W20260709 12:22:39.123445 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:39.124024 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.125790 31745 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:22:39.121523 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:39.031075  4102 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:39.131688 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:39.033514  3859 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:39.080161  3689 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:39.132393  4023 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:39.035445  3991 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:39.140812 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:39.137122 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.157202 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:39.140640 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:39.141960 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:39.144865 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:39.158901 32374 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:39.149431 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:39.065670  3698 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:39.115542  3693 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:22:39.158980 32370 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:39.095491  4121 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.164389 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:39.180995 31670 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:22:39.181314 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:39.127655  4169 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.185910 31824 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:39.143056  4025 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:39.138918  3920 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.188822 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:39.151899  4131 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:39.194021 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.148938  3966 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.148295  4023 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:39.197149 31820 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:39.139118  4147 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:22:39.199026 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:39.198917  4137 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.213405 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:39.207504 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:39.130461  4195 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:22:39.233870 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 476793us
I20260709 12:22:39.222553 31745 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:39.235543 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:39.240540 31670 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:39.162017  3689 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:39.221148  4023 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:39.231444  4074 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:39.249894 31642 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.584s (SENT) [suppressed 10 similar messages]
I20260709 12:22:39.251304 31750 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:39.197098  3723 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.052s
I20260709 12:22:39.189378  4169 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:39.188896  4121 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:39.204248  3763 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.050s
I20260709 12:22:39.275838  3859 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:39.221390  3966 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:39.307192 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:39.276249  3909 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:39.209048  4131 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:39.255539  4160 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:39.205384  3920 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:39.264544  4147 raft_consensus.cc:2764] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:22:39.250893  3981 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:39.255009  4074 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:39.283159 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:39.247004  4000 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:39.231247  4159 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:22:39.328648 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:39.359066 31819 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:39.360845 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:39.361212 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:39.362149 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:39.367151 32377 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
W20260709 12:22:39.369513 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 86493.5us
W20260709 12:22:39.369983 31676 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.028s	sys 0.027s
I20260709 12:22:39.244282  4137 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:39.302341  4088 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:39.366964  3981 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:39.359375  4151 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:39.380435 31676 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.028s	sys 0.027s
I20260709 12:22:39.379772 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:39.607288 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.608137 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:39.378803  3946 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.625169 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:39.619096 31678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.011s	sys 0.021s
W20260709 12:22:39.628229 31678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.012s	sys 0.021s
W20260709 12:22:39.619776 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 240111us
I20260709 12:22:39.385811  4108 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:39.621369 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:39.365836  4187 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.035s
I20260709 12:22:39.361766  4160 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:39.635737 31825 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:22:39.381121  4185 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:39.643613  3689 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:39.644677  4131 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:39.645927  4169 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:39.703262 31717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.559s	user 0.025s	sys 0.063s
I20260709 12:22:39.647506  4180 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:39.648110  3698 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:39.379864  4200 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.713371 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:39.656544 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:39.657387 31818 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.013s	sys 0.024s
I20260709 12:22:39.642248  3705 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:39.379760  4198 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:22:39.695694 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 445812us
W20260709 12:22:39.696581 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.016s	sys 0.032s
W20260709 12:22:39.719324 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.524s	user 0.016s	sys 0.032s
W20260709 12:22:39.719712 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 525069us
I20260709 12:22:39.608952  3844 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:39.606921  4102 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:39.686050  3693 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:39.701800 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 373233us
I20260709 12:22:39.391556  4147 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.256s
W20260709 12:22:39.696943  4175 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.679s	user 0.019s	sys 0.030s
W20260709 12:22:39.703861 31717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.025s	sys 0.063s
I20260709 12:22:39.628741  3981 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:39.632359  3723 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:39.709327 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:39.716550 31818 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.707s	user 0.013s	sys 0.024s
I20260709 12:22:39.754980 31745 raft_consensus.cc:2368] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14.
I20260709 12:22:39.723192 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:39.755689 32371 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:39.710471  3991 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:39.733624 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:39.718060  3920 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:39.741547 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 597566us
I20260709 12:22:39.742566 32369 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:39.389209  4000 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.304s
I20260709 12:22:39.720700  4169 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:39.711921  4131 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:39.764969 32372 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:39.637235  4088 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:39.651574  3946 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:39.760382  4004 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.885s	user 0.010s	sys 0.003s
W20260709 12:22:39.741254  4175 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.723s	user 0.019s	sys 0.030s
I20260709 12:22:39.784947 31751 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:39.788599 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:39.790742 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:39.733345  3844 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:39.767477 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:39.720024  4180 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:39.752221  4129 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:39.730486  4200 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:39.776089 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:39.777269 31717 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:39.765174  4206 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
W20260709 12:22:39.782145  4168 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.885s	user 0.011s	sys 0.000s
I20260709 12:22:39.783047  4151 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.791500 31751 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:39.793025 31747 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:39.796437 31825 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:22:39.853063  3844 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:39.854561  4151 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:39.775054  3705 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.073s
I20260709 12:22:39.859896  4088 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:39.763419  4197 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.107s
I20260709 12:22:39.860445  4168 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:39.862206  4160 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:39.863343  4137 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:39.863672 31753 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 15: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:39.859997  3705 consensus_queue.cc:1243] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 15.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:22:39.870756  4211 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260709 12:22:39.872592  4088 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:39.873136  4088 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:39.874049  4091 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.874959 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:39.874756  4091 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:39.874581  4088 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:39.876452  4088 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:39.790617  4131 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:39.880241  3693 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:39.885108 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1723) took 1939 ms. Trace:
I20260709 12:22:39.885747 31673 rpcz_store.cc:276] 0709 12:22:37.946103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.960354 (+ 14251us) service_pool.cc:224] Handling call
0709 12:22:37.960678 (+   324us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:37.961005 (+   327us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:37.961032 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:38.882075 (+921043us) spinlock_profiling.cc:243] Waited 45.5 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:22:38.931002 (+ 48927us) log.cc:844] Serialized 26 byte log entry
0709 12:22:39.360008 (+429006us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:39.360069 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.360169 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:39.884634 (+524465us) raft_consensus.cc:1685] finished
0709 12:22:39.884666 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:39.885014 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1165,"raft.run_cpu_time_us":48,"raft.run_wall_time_us":48,"spinlock_wait_cycles":104597760,"thread_start_us":382396,"threads_started":1,"wal-append.queue_time_us":910921}
I20260709 12:22:39.887387  4168 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:39.888903  4220 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
W20260709 12:22:39.889598  4023 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.635s	user 0.010s	sys 0.003s
I20260709 12:22:39.890594 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:39.890460  4023 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:39.891623 32369 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:39.893584  4091 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:39.814342 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 675) took 1709 ms. Trace:
I20260709 12:22:39.894898 31678 rpcz_store.cc:276] 0709 12:22:38.104893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.256233 (+151340us) service_pool.cc:224] Handling call
0709 12:22:38.256465 (+   232us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:39.039325 (+782860us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:39.039378 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:39.040218 (+   840us) log.cc:844] Serialized 52 byte log entry
0709 12:22:39.629802 (+589584us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:39.629876 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.629974 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:39.813963 (+183989us) raft_consensus.cc:1685] finished
0709 12:22:39.813996 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:39.814226 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":186430,"raft.queue_time_us":85831,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":10108,"thread_start_us":1263134,"threads_started":2,"wal-append.queue_time_us":706242}
I20260709 12:22:39.732685 31792 leader_election.cc:400] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11.
I20260709 12:22:39.898273 31792 leader_election.cc:403] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
W20260709 12:22:39.898970 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 166526us
I20260709 12:22:39.816634 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 679) took 1684 ms. Trace:
W20260709 12:22:39.828616 31790 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.006s	sys 0.005s
I20260709 12:22:39.905084 31751 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260709 12:22:39.905284 31790 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.006s	sys 0.005s
W20260709 12:22:39.845825 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 68654.9us
I20260709 12:22:39.910777 31670 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:22:39.912284 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:39.913298 31670 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:39.918426 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:39.893105  3981 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:39.926283 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:39.901547 31676 rpcz_store.cc:276] 0709 12:22:38.132213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.840103 (+707890us) service_pool.cc:224] Handling call
0709 12:22:38.840544 (+   441us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:38.840772 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:39.617786 (+777014us) log.cc:844] Serialized 14 byte log entry
0709 12:22:39.815576 (+197790us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:39.815643 (+    67us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:39.815701 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:39.816248 (+   547us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:39.816516 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":808344,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"raft.queue_time_us":109,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":29647,"thread_start_us":973841,"threads_started":2,"wal-append.queue_time_us":246534}
I20260709 12:22:39.927521 31678 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:39.930179 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:39.816340  3966 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:39.933624 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.816411  4129 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:39.952440 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:39.953142 31818 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:39.953188 31747 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:22:39.936725 31744 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:22:39.982681 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:39.939380 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:39.929276  3859 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.013s
I20260709 12:22:39.937398  3723 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:39.937770  4168 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.937947  3844 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:39.929755  4160 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:39.946954 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:39.948069  4180 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:39.791352  4147 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.119s
W20260709 12:22:39.828097 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:39.986894  4004 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:22:39.939404  4141 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.021s	user 0.011s	sys 0.011s
I20260709 12:22:39.976934 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:39.910857  4000 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 10. Current term is 11.
I20260709 12:22:39.921293  3705 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:40.025801 31744 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:40.026145 31747 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:40.012253 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:40.011749 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 276048us
I20260709 12:22:40.013981 31676 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:39.922044  4158 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.019490 32378 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:40.038697 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.038854 31678 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:39.953311  4091 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.047542  4091 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.032341 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:40.055280  4174 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:40.056145  4174 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:40.056459  3981 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.060015  4091 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.021133 31818 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:40.018662  3723 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.025190  4168 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.064529  3723 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.039649  4158 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.065482  4168 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.036530  3766 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.066699  3766 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.040112  4131 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.041204  3946 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:40.023346  3844 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.042503  4023 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.036940  4160 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:40.036176  3705 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.044741  4129 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.028930  4200 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:40.044806  4191 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.100s	user 0.014s	sys 0.000s
I20260709 12:22:40.013846  4187 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.074229 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.074014  4187 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.075456 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.076273  4187 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.076684 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:40.057439  4174 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 12.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.016187  4225 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:40.079715 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:40.080971 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.033579 31751 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:40.083223 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.084331 31751 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 13 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:22:40.081595 31676 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:40.087033 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.087854 31676 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:40.088958 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.095844 31749 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:40.098748 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.099658 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.106720 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:40.099754 31747 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:22:40.101783 31673 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:40.102070 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.100688  3909 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.002s
I20260709 12:22:40.103487 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.080523  4129 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:40.109894 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:40.119722 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.126160 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.029426  4004 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.081574  4158 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.124433 31676 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:40.126206 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:40.130600 31747 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:40.102439  4191 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 12:22:40.138671 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:40.099823 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:40.111781  4175 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.029s
I20260709 12:22:40.141712 31678 raft_consensus.cc:2410] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 12:22:40.144531 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:40.124800  4169 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:40.145817  4169 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.148886  4169 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.149080 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.149900 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.151398 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:40.152479 31744 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:22:40.152585 31676 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:40.157382 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.158080 31747 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:40.158815 31744 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
W20260709 12:22:40.159953 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60181.7us
I20260709 12:22:40.160856 31818 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:40.172973 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:40.173182  4191 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:40.172910 31744 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:40.174695 31744 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:40.175583 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:40.175758 31744 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.176579  4191 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:40.177048  4191 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.177426  4191 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:40.098002  4225 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.150964  4216 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:40.185295  4216 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.179064 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:40.178790 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:40.187706 31751 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:22:40.187783 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:40.189610 31747 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:22:40.189779 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:40.191659 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:40.153013  4213 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:40.185770  4216 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:40.200268 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.172082  4141 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:40.185770  3738 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:40.139921  4160 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.220002 31747 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 11. Preceding OpId from leader: term: 11 index: 11. (term mismatch)
I20260709 12:22:40.195609  4175 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:40.222126 31747 pending_rounds.cc:85] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 10
I20260709 12:22:40.222500 31747 pending_rounds.cc:107] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 10.11
I20260709 12:22:40.222847 31747 raft_consensus.cc:2889] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:40.224298 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.225091 31747 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:40.226766 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.227463  4004 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.207381  3990 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:40.210034  4158 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:40.194655  4213 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:40.210223  4023 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:40.232342 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:40.218185  4141 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:40.238162  4239 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:40.244479  4168 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.244864  4213 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:40.229925  3990 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:40.253218  4239 consensus_queue.cc:1243] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:22:40.256400  3966 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:40.257002  3698 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:40.249568  4158 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:40.268061 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:40.269304 31818 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:40.271588  4151 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.262476  3966 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:40.263413  3698 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:40.273165  3698 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.274348  3698 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:40.275470  3966 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.276695  3966 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:40.279066  4191 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.281687  3766 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:40.288981  3705 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.289350  4191 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.260973  3990 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:22:40.306258 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 109792us
I20260709 12:22:40.309319 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.310420 31747 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:40.254528  4168 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.323521 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:40.325448 31818 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:40.327240 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.329828 31676 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:40.343308 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:40.348601 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.341509  4004 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.362260 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:40.363034 31818 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:40.365356 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.319031  4141 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.368211 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:40.368899 31818 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:40.343009  4187 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:40.370929 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.371963 31747 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
W20260709 12:22:40.371074 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 162136us
W20260709 12:22:40.372648 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58136us
I20260709 12:22:40.322774  4158 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.343168  3705 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.026s
I20260709 12:22:40.375697 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:40.361493  4004 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.380985 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:40.381723 31818 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:40.381832 32369 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:40.382397  4004 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.384773 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.387439 31747 raft_consensus.cc:2410] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 14 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:22:40.374939 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:40.391340  4233 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.130s	user 0.000s	sys 0.001s
I20260709 12:22:40.391834  4168 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.394083 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.395565 31670 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:40.396189 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:40.396836 31670 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
W20260709 12:22:40.397977  4234 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
I20260709 12:22:40.397395 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.399724 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:40.400141  4191 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:40.400861 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:40.401480 32371 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:40.402823  4211 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:40.403585  3985 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:40.403703  4211 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:40.404107  4211 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:40.404978  4211 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.373435  4187 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.408679 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.423240  4211 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:40.425776 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:40.426403 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:40.429239  4236 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.118s	user 0.001s	sys 0.000s
W20260709 12:22:40.426990 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.433076 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.433627 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.428490 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:40.440419  4147 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:40.413197 31818 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:40.441242 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:40.390839  4129 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:40.441095 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:40.446498 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:40.447706 31821 raft_consensus.cc:2410] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 10 because replica has last-logged OpId of term: 8 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 6.
I20260709 12:22:40.450871 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:40.444921 31791 connection.cc:256] RPC call timeout handler was delayed by -1.4329665159998513s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1574s for a response
I20260709 12:22:40.452303 32370 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 12 because replica has last-logged OpId of term: 11 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:22:40.453933 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:40.456357 31791 connection.cc:256] RPC call timeout handler was delayed by -1.3618665159997363s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1653s for a response
I20260709 12:22:40.456590 31818 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
W20260709 12:22:40.457043 31791 connection.cc:256] RPC call timeout handler was delayed by -1.2286665159999757s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1801s for a response
W20260709 12:22:40.457451 31791 connection.cc:256] RPC call timeout handler was delayed by -1.1539665159998549s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1884s for a response
W20260709 12:22:40.457834 31791 connection.cc:256] RPC call timeout handler was delayed by -1.1431665159998374s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1896s for a response
W20260709 12:22:40.458189 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0891665159997501s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1956s for a response
I20260709 12:22:40.456504 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:40.459409 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:40.462417 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.471773 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:40.475710 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:40.476280  3840 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.476658  3723 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:40.477167  3723 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.477564  3723 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:40.407637  4242 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.081s
W20260709 12:22:40.479350  4241 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
W20260709 12:22:40.480376 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.482910 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:40.448346  4129 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.487363 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:40.490085 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.446238  4200 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:40.492638 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:40.454073  3698 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:40.454108  4169 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.501502 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:40.456357  4168 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:40.491613  3705 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:40.506541  3705 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:40.507015  3705 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:40.507436  4243 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:22:40.495143  4023 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:40.508484  4023 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:40.508879  4023 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:40.509279  4023 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:40.507511  4200 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.513301  4200 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.513890  4200 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:40.514375  4169 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.517158  4169 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:40.518153 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:40.518899 32373 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:40.518523  4023 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.519927 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:40.521045 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:40.522708  4023 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.528389  4200 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:40.529333  4200 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.471601  4000 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.455878  3946 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:40.472582  4147 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:40.485846  4160 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.487066  4158 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:40.495733  3859 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:40.491751  4088 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.488234  4129 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:40.481240  4137 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:40.582774 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 99953.9us
I20260709 12:22:40.556143 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.558367  3844 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:40.559731 31820 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2444) took 1611 ms (client timeout 1514 ms). Trace:
I20260709 12:22:40.562137 31744 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:40.587482 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.588673 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.537034  3698 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:40.564450 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.542027  4158 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.568995  4025 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:40.578419  4130 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:40.579501  4159 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:40.535518  4198 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:40.544479 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.563032 32375 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.619733 32375 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:40.619894 31670 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:22:40.621081 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.621925 31670 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:40.594218 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.622972 31677 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:40.610944  4169 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.030s
I20260709 12:22:40.623826  4169 consensus_queue.cc:1243] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 12.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:22:40.623958 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.595219 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:40.625072  4121 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.586712  4088 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.543946  4160 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.589954  4137 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.599542  4200 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.600399  3698 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:40.587157 31820 rpcz_store.cc:269] 0709 12:22:38.947973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.959166 (+ 11193us) service_pool.cc:224] Handling call
0709 12:22:40.559539 (+1600373us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b54003c91d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:40.559657 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":507700992}
W20260709 12:22:40.635458 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 142900us
I20260709 12:22:40.602059  3723 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.637552 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:40.638185 32370 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:40.608763 31744 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:22:40.609184  4129 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.545764  3859 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:40.613174  4025 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:40.613432  4158 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:40.566037  4173 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:40.626842 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.596205  4187 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.541865  3946 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.540462  4000 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.610261  4231 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.636235 32375 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 10. (term mismatch)
I20260709 12:22:40.618952  4159 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:40.657183 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.618460  4213 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.655344  4195 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:22:40.677606  4160 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:40.678678  4129 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:40.669716  4137 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:40.643774  4147 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.683240  4147 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.687842  4147 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.630496  4216 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.691789  4074 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:40.652103  3763 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 15 current term: 15 is higher.
I20260709 12:22:40.692224  4213 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.694525  4216 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.674268  4231 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.699978  3723 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.700280  4231 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.674925  4159 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.675758  4225 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:40.676827  4088 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.674352  4025 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.669891  4000 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:40.694280  3990 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.654330  3859 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:40.656800  4200 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 12:22:40.711464  4025 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:40.710071  4198 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260709 12:22:40.717381  4159 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:40.727835  4158 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:40.765255 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.625361 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:40.653909 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:40.769579 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 115598us
W20260709 12:22:40.640390 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:40.772909 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.772783 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:40.628005 31674 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:22:40.611860 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.776105 31747 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:22:40.642789 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1846) took 1548 ms. Trace:
I20260709 12:22:40.655141 31820 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:22:40.786116 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:40.787120 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.786870  3698 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:22:40.787803 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.658018  3705 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.624083 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.790498 31678 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:40.792462 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:40.670773 32375 pending_rounds.cc:85] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 9
I20260709 12:22:40.795205 32375 pending_rounds.cc:107] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 12:22:40.795683 32375 raft_consensus.cc:2889] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:40.797075 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 711) took 2606 ms. Trace:
I20260709 12:22:40.797327  3705 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:40.797585 32375 rpcz_store.cc:276] 0709 12:22:38.190113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.191271 (+  1158us) service_pool.cc:224] Handling call
0709 12:22:38.191575 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:38.818677 (+627102us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b54004bb520. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:22:40.562680 (+1744003us) spinlock_profiling.cc:243] Waited 45.5 ms on lock 0x7b4c0014fe40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb0a1a20 00007fe8fb0b2a29 00007fe8fb078644 00007fe8fb09ff2d 00007fe8fb09fb93 00007fe8fcdd018b 00007fe8fcdd0cb5 00007fe8fcdcbf30 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416
0709 12:22:40.796605 (+233925us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.796928 (+   323us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":137247872}
I20260709 12:22:40.653956 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.788160  3698 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.852679 31744 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:40.794260 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:40.853730 31670 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:22:40.702119 32370 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
W20260709 12:22:40.857549 31750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1838) took 1810 ms (client timeout 1884 ms). Trace:
I20260709 12:22:40.777245 31748 rpcz_store.cc:276] 0709 12:22:39.093958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.109211 (+ 15253us) service_pool.cc:224] Handling call
0709 12:22:39.109696 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:39.136982 (+ 27286us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:39.137012 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:39.137493 (+   481us) log.cc:844] Serialized 26 byte log entry
0709 12:22:40.405657 (+1268164us) spinlock_profiling.cc:243] Waited 2.44 ms on lock 0x7b180031a6e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:22:40.531089 (+125432us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:40.531137 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.531303 (+   166us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:40.642477 (+111174us) raft_consensus.cc:1685] finished
0709 12:22:40.642493 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.642725 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31940,"raft.queue_time_us":58521,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":8500,"spinlock_wait_cycles":5608320,"thread_start_us":125212,"threads_started":1,"wal-append.queue_time_us":1414777}
I20260709 12:22:40.861636 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.862231 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:40.863945 31818 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:40.866925 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2360) took 2834 ms. Trace:
I20260709 12:22:40.867556 31819 rpcz_store.cc:276] 0709 12:22:38.031930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.210288 (+178358us) service_pool.cc:224] Handling call
0709 12:22:38.225554 (+ 15266us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:39.375441 (+1149887us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.866232 (+1490791us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b540043d5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:40.866665 (+   433us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2606217472}
I20260709 12:22:40.868937  4200 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.869745  4200 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:40.870978  4160 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.872172  3454 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.872079  4160 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.872560 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:40.872947  3454 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.876477 32369 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: []
W20260709 12:22:40.876638 31824 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2440) took 1939 ms (client timeout 1845 ms). Trace:
W20260709 12:22:40.877257 31824 rpcz_store.cc:269] 0709 12:22:38.936702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.963764 (+ 27062us) service_pool.cc:224] Handling call
0709 12:22:40.876119 (+1912355us) spinlock_profiling.cc:243] Waited 218 ms on lock 0x7b540030ded4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:40.876405 (+   286us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":502169984}
W20260709 12:22:40.858136 31750 rpcz_store.cc:269] 0709 12:22:39.047527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.048559 (+  1032us) service_pool.cc:224] Handling call
0709 12:22:40.857152 (+1808593us) spinlock_profiling.cc:243] Waited 177 ms on lock 0x7b5400212fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:40.857341 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":406191488}
I20260709 12:22:40.879756 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2365) took 2846 ms. Trace:
I20260709 12:22:40.880363 31826 rpcz_store.cc:276] 0709 12:22:38.033515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.280644 (+247129us) service_pool.cc:224] Handling call
0709 12:22:38.280898 (+   254us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:38.928610 (+647712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.879144 (+1950534us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b54004470d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:40.879516 (+   372us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1488312064}
I20260709 12:22:40.881286 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.797950 31671 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:22:40.816200  4137 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:40.882407  4137 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.882969  4137 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:40.820036  4139 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:22:40.837206  4032 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:40.838320  4000 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:40.886782 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.886927  4000 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:40.887557  4000 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:40.888445 31748 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:40.889449 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:40.889807 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.840881 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 674) took 2736 ms. Trace:
I20260709 12:22:40.890820 31669 rpcz_store.cc:276] 0709 12:22:38.104578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:38.255738 (+151160us) service_pool.cc:224] Handling call
0709 12:22:38.264311 (+  8573us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:38.910236 (+645925us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:22:39.385463 (+475227us) log.cc:844] Serialized 14 byte log entry
0709 12:22:39.715548 (+330085us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:39.715686 (+   138us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:39.715820 (+   134us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.840199 (+1124379us) spinlock_profiling.cc:243] Waited 646 ms on lock 0x7b54001810d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:40.840283 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.840635 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14229,"prepare.queue_time_us":739119,"prepare.run_cpu_time_us":26,"prepare.run_wall_time_us":26,"raft.queue_time_us":67644,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"spinlock_wait_cycles":1484960640,"thread_start_us":789493,"threads_started":2,"wal-append.queue_time_us":381511,"wal-append.run_cpu_time_us":1273,"wal-append.run_wall_time_us":1067100}
I20260709 12:22:40.896798 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:40.839154  4015 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 12:22:40.900738  4174 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:40.901120 31827 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 16 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 9 index: 5.
I20260709 12:22:40.901706 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2450) took 1574 ms. Trace:
I20260709 12:22:40.901685  4174 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:40.902472 31825 rpcz_store.cc:276] 0709 12:22:39.327840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.703525 (+375685us) service_pool.cc:224] Handling call
0709 12:22:39.703902 (+   377us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:39.849390 (+145488us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.901088 (+1051698us) spinlock_profiling.cc:243] Waited 92.1 ms on lock 0x7b540037a854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:40.901486 (+   398us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":211758464}
I20260709 12:22:40.904542 31825 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:22:40.906379 31827 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:40.898819 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.851496 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:40.894230 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:40.858103 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.893239 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:40.913610 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.915908 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2482) took 1007 ms. Trace:
W20260709 12:22:40.916297 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.917038 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:40.919363 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2407) took 1142 ms. Trace:
I20260709 12:22:40.904115 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:40.873468 31823 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:40.920992 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:40.927641 32378 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 707) took 3093 ms (client timeout 1964 ms). Trace:
I20260709 12:22:40.913589 31671 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:22:40.929498 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:40.931454 32377 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 682) took 1791 ms (client timeout 1896 ms). Trace:
I20260709 12:22:40.932930 32374 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:22:40.935434 31819 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:40.937987 31672 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
W20260709 12:22:40.938897 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:40.938972 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2368) took 1006 ms. Trace:
W20260709 12:22:40.992604 32378 rpcz_store.cc:269] 0709 12:22:37.833927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.834376 (+   449us) service_pool.cc:224] Handling call
0709 12:22:40.927113 (+3092737us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b5400544f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:40.927376 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4426694528}
W20260709 12:22:40.942058 31753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1845) took 1848 ms (client timeout 2390 ms). Trace:
I20260709 12:22:40.882557 31750 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:40.946384 31748 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:40.942234 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:40.847409 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:40.953657 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:40.932977 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:40.964130 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:40.965844 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:40.968039 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:40.968858 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 191334us
I20260709 12:22:40.801882 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:40.990988 31752 rpcz_store.cc:276] 0709 12:22:39.776865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.777172 (+   307us) service_pool.cc:224] Handling call
0709 12:22:40.858945 (+1081773us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.859439 (+   494us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.859468 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:40.859538 (+    70us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:40.859576 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.859660 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.918530 (+ 58870us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400212fd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:40.919087 (+   557us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2486796544}
I20260709 12:22:40.995587 31821 rpcz_store.cc:276] 0709 12:22:39.932922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.472563 (+539641us) service_pool.cc:224] Handling call
0709 12:22:40.475365 (+  2802us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.783637 (+308272us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.783710 (+    73us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:40.783823 (+   113us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:40.783908 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:40.938449 (+154541us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b54000fc654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:40.938510 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:40.938773 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":706795264}
I20260709 12:22:41.000232 32369 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:41.009624 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.010591 31821 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:22:41.010797 31752 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:41.011104 32375 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:41.012411 31824 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:41.012701 31823 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
W20260709 12:22:40.994324 32377 rpcz_store.cc:269] 0709 12:22:39.140131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.140291 (+   160us) service_pool.cc:224] Handling call
0709 12:22:40.931015 (+1790724us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b54004984d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:40.931204 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":479577984}
I20260709 12:22:41.014959 31678 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:41.018064 31822 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:22:40.990193 31673 rpcz_store.cc:276] 0709 12:22:39.907944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.914077 (+  6133us) service_pool.cc:224] Handling call
0709 12:22:40.915424 (+1001347us) spinlock_profiling.cc:243] Waited 88.4 ms on lock 0x7b540022a954. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:40.915673 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":203374464}
I20260709 12:22:41.018826  3990 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:40.997958 31753 rpcz_store.cc:269] 0709 12:22:39.093573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.093743 (+   170us) service_pool.cc:224] Handling call
0709 12:22:40.941618 (+1847875us) spinlock_profiling.cc:243] Waited 768 ms on lock 0x7b5400295ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:40.941872 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1767063424}
I20260709 12:22:40.927330 31575 catalog_manager.cc:5697] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 9 to 10, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 10 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:41.010229 31744 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:41.024376 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2375) took 1064 ms. Trace:
I20260709 12:22:41.025030 31827 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 10. Preceding OpId from leader: term: 11 index: 11. (index mismatch)
I20260709 12:22:41.024961 31819 rpcz_store.cc:276] 0709 12:22:39.960025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.869179 (+909154us) service_pool.cc:224] Handling call
0709 12:22:40.935032 (+ 65853us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.997433 (+ 62401us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.024069 (+ 26636us) spinlock_profiling.cc:243] Waited 65.3 ms on lock 0x7b540030ded0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:41.024274 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":150148864}
I20260709 12:22:41.028330 31818 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:41.028647 32372 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:41.029374 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 836) took 1868 ms. Trace:
I20260709 12:22:41.029773 32374 rpcz_store.cc:276] 0709 12:22:39.160736 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.160884 (+   148us) service_pool.cc:224] Handling call
0709 12:22:40.932002 (+1771118us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:40.994722 (+ 62720us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.029110 (+ 34388us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b54004984d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:41.029289 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4072190720}
W20260709 12:22:41.031180 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 216239us
I20260709 12:22:41.032765 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:41.033298 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.034319 31750 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:41.035409 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:41.036439 31750 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:41.036484  4015 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:41.037556 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.038508  3946 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:41.039896 31751 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 9.
I20260709 12:22:41.041117 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:41.044493 31745 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:41.035475 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 102828us
I20260709 12:22:41.047055 31753 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: [11.10-11.10]
I20260709 12:22:41.047281  4158 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:41.047842  3698 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:41.066823 31745 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:41.068840 31823 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:22:41.073359 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.074483 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:41.078253 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:41.079941 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:41.080948 31822 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:41.081935 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:41.084023 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:41.085268 31750 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 15: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:41.085356 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:41.090543 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 57979.6us
I20260709 12:22:41.091329  4023 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:41.092062  4023 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.092370 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:41.092631  4023 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:41.094331 31822 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:41.070472 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:41.102806 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:41.108258 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:41.116963 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:41.125118 31747 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:41.123085 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:41.139663 31678 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:41.139539 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:41.144109 31670 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:41.148260  4015 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:41.148875  4015 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.149281  4015 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:41.143219 31748 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:41.155435 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 63239.7us
I20260709 12:22:41.157320 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:41.155385  4160 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:41.159463  3454 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:41.160874 31749 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 9 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:41.163861 31827 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:41.165616 31749 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 12:22:41.170859 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47340.8us
I20260709 12:22:41.175745 32372 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
I20260709 12:22:41.177533 31746 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 13 because replica has last-logged OpId of term: 12 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 8.
I20260709 12:22:41.179517  3859 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.182219  3859 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:41.182339 31752 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
W20260709 12:22:41.179603 31746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1840) took 2128 ms (client timeout 1574 ms). Trace:
I20260709 12:22:41.184350  4196 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:41.185266 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:41.186303  4191 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:22:41.187345 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.192526  4174 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:41.145237  4139 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
W20260709 12:22:41.198257 31746 rpcz_store.cc:269] 0709 12:22:39.051567 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:39.051766 (+   199us) service_pool.cc:224] Handling call
0709 12:22:41.179284 (+2127518us) spinlock_profiling.cc:243] Waited 2.12 s on lock 0x7b54002f71d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:41.179491 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4885086592}
I20260709 12:22:41.210264 31826 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:22:41.211238 31753 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:41.204084 31747 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:22:41.197839  3705 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.216070 31750 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
W20260709 12:22:41.218003 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.231980 31752 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:41.233003 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:41.224570  4200 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:41.247481  4032 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.248211  4032 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.263743 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:41.267019  4159 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.542s	user 0.016s	sys 0.006s
I20260709 12:22:41.232163  3705 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:41.241688  4196 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:22:41.249593 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 961315us
I20260709 12:22:41.275362 31670 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:41.288100 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 722) took 1068 ms. Trace:
I20260709 12:22:41.288712 32371 rpcz_store.cc:276] 0709 12:22:40.219459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.402087 (+182628us) service_pool.cc:224] Handling call
0709 12:22:40.402425 (+   338us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:40.406425 (+  4000us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:40.406452 (+    27us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:40.407105 (+   653us) log.cc:844] Serialized 52 byte log entry
0709 12:22:40.691521 (+284416us) spinlock_profiling.cc:243] Waited 6.25 ms on lock 0x7b18000d9a20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:22:41.009672 (+318151us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:41.009762 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.009874 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:41.287415 (+277541us) raft_consensus.cc:1685] finished
0709 12:22:41.287450 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.287841 (+   391us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1993,"raft.queue_time_us":909,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":11198,"spinlock_wait_cycles":14369664,"thread_start_us":317686,"threads_started":1,"wal-append.queue_time_us":861130}
W20260709 12:22:41.311014 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 125954us
I20260709 12:22:41.315980 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:41.316903 31678 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
W20260709 12:22:41.320333 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 276721us
I20260709 12:22:41.322394  3946 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 LEADER]: Stepping down as leader of term 8
I20260709 12:22:41.326783 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:41.323431  4196 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:41.329258 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:41.326406  3946 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:41.331413  3990 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.008s
I20260709 12:22:41.339067 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:41.339794 31823 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:41.340087 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 172035us
I20260709 12:22:41.331981 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:41.343817 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.306807  4174 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.028s
I20260709 12:22:41.346400 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:41.352545 31818 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:41.322521  4191 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:41.354984 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:41.359256 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:41.339216  4196 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:41.361002 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:41.362712 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:41.357613 31826 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:41.365491 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:41.369745 31822 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:41.375886 31821 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
W20260709 12:22:41.379786 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:41.381196 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.379388 31677 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:41.387940 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:41.392338 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:41.396484 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:41.398209  3840 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:41.398871  3840 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.380273  4032 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:41.405910  4147 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.406719  4147 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.409266  4147 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:41.410071 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.369730  4191 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.411015 31745 raft_consensus.cc:2368] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14.
I20260709 12:22:41.412910  4191 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:41.410375 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.417014 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:41.419137  3620 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:41.419428 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 82888us
I20260709 12:22:41.421496 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:41.406082 31676 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:22:41.424845  4160 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:41.375159  3990 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:22:41.373769  4000 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.428774 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:41.427497 31818 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:22:41.426900 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:41.438364 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.438886 31670 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:41.403785  4195 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.437420 31827 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:22:41.445022  3086 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.445263  4147 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:41.445834  4147 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:41.445675  3086 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:41.446359  4147 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:41.448117  3086 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:41.452565 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.453436 31752 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:41.454663  3946 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.455081 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:41.456233 31750 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:41.457393 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.458727  4195 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.455164 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.459878 31746 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:41.460598  4147 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.461212  4195 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:41.459005 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.462927  4147 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:41.454550 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.443442 31748 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:41.456184  3946 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 8 FOLLOWER]: Advancing to term 9
I20260709 12:22:41.465193 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.466518 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:41.468695 31750 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 10. (index mismatch)
W20260709 12:22:41.463739 31642 connection.cc:256] RPC call timeout handler was delayed by -1.1103577570002017s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1519s for a response
I20260709 12:22:41.480201 31745 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:41.481027  3620 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:41.482403  4025 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:41.482978  4147 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:41.483660  4147 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.485131  3705 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:41.485755  4147 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:41.491750 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.478849  4195 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:41.496965 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:41.499879 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:41.502388 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:41.512372 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.513036 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:41.509397  4147 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:22:41.515121  4147 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.515013  3705 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:41.515604  4147 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:41.523504 31575 catalog_manager.cc:5697] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 1 to 6, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 6 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:41.525965  3840 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:41.525578 31575 catalog_manager.cc:5697] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 4 to 8. New cstate: current_term: 8 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:41.527510 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.531059  3086 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.532238 31823 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:41.531581 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:41.534276  3086 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.534590 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:41.535419 31675 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:41.531785 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:41.538249 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:41.489753  4023 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:41.539433  4137 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.653s	user 0.022s	sys 0.006s
W20260709 12:22:41.590170 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 80343.3us
W20260709 12:22:41.598057 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 196101us
I20260709 12:22:41.600950  4269 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:41.601608  4269 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.602818  4269 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:41.602916  4263 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.356s	user 0.002s	sys 0.004s
I20260709 12:22:41.598990 31678 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
W20260709 12:22:41.606583 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 187926us
I20260709 12:22:41.607831 31746 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:41.608248 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:41.615418 31575 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:41.617092  4158 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.617923  4158 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:41.610958  3086 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:41.627077 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:41.628233 31671 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:41.620635  4158 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:41.633353 31575 catalog_manager.cc:5697] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 4 to 7, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 7 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:41.644208 31744 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
W20260709 12:22:41.653671 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53310.3us
I20260709 12:22:41.651587  3086 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:41.598047  4137 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.660797 31575 catalog_manager.cc:5697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:41.661376 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:41.663967 31822 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:41.609313  4015 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:41.660532  4000 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:41.676610  3086 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.013s
I20260709 12:22:41.685148 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:41.687409 31819 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:22:41.652347  4158 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 12:22:41.693487 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:41.676478  4032 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:41.731155  4015 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:41.737151  4147 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.745030 31675 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:41.765595 31673 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:41.742291  4032 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.769150 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:41.719091  4139 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.060s
I20260709 12:22:41.681342  3620 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.092s
I20260709 12:22:41.773228  4032 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:41.776683 31574 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42072 (request call id 241) took 21729 ms (client timeout 15000 ms). Trace:
I20260709 12:22:41.750813 31575 catalog_manager.cc:5697] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 5 to 9. New cstate: current_term: 9 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:41.777491  4191 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:41.790493 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:41.723110  4158 consensus_queue.cc:1243] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 14.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260709 12:22:41.800346 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 122303us
I20260709 12:22:41.774562  4198 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:22:41.803993 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63074us
I20260709 12:22:41.808837 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:41.806248 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:41.770499  4195 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:22:41.814807 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 487560us
I20260709 12:22:41.824690  3086 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:41.825186  3086 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:41.825529 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.825594  3086 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:41.804646  4159 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:41.834201 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:41.837524 31751 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.028s	sys 0.043s
W20260709 12:22:41.839531 31751 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.656s	user 0.029s	sys 0.043s
I20260709 12:22:41.811441  4015 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.814939  4198 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:41.851539 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:41.826361  4023 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:41.835649  3705 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.833487  3946 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:41.840359  4159 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:41.861443  4194 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:41.826807  4191 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.860586 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:41.855088  3698 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:41.875980 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49012.9us
I20260709 12:22:41.865407  4194 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.865161  4137 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:41.855654  4015 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:41.886464 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:41.875396  3698 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:41.860189  3705 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:41.864840  3946 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:41.903192 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 370709us
I20260709 12:22:41.893509  4194 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:41.903167  3698 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:41.902562  4193 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:41.910349 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:41.917121 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:41.871685  3620 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.035s
W20260709 12:22:41.913631 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 254577us
W20260709 12:22:41.920406 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.920552  4193 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.924209  4193 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:41.925419  3946 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:41.926458  4279 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:41.927215  4279 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:41.929191  4198 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.930236  4279 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:41.931818  3620 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election won for term 10
W20260709 12:22:41.934276 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 71395us
I20260709 12:22:41.939651  3705 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:41.940167  4272 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:22:41.941771 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 699) took 1301 ms. Trace:
I20260709 12:22:41.942246 31825 rpcz_store.cc:276] 0709 12:22:40.640352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:40.958423 (+318071us) service_pool.cc:224] Handling call
0709 12:22:41.008151 (+ 49728us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.101151 (+ 93000us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.101203 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:41.194565 (+ 93362us) log.cc:844] Serialized 26 byte log entry
0709 12:22:41.535568 (+341003us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:41.535613 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.535678 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:41.941473 (+405795us) raft_consensus.cc:1685] finished
0709 12:22:41.941504 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:41.941690 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":64773,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":263,"thread_start_us":340220,"threads_started":1,"wal-append.queue_time_us":644927}
I20260709 12:22:41.951656  4015 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:41.954962  4000 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:41.955498  4000 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:41.956013  4191 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:41.956167  4015 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:41.956665  4015 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:41.956509  4000 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.958035  4000 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:41.792045 31574 rpcz_store.cc:269] 0709 12:22:19.800662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:19.803191 (+  2529us) service_pool.cc:224] Handling call
0709 12:22:41.522601 (+21719410us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:27.353538 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:27.446974 (+ 93436us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:27.447025 (+    51us) write_op.cc:435] Acquired schema lock
0709 12:22:27.447054 (+    29us) tablet.cc:662] Decoding operations
0709 12:22:27.469110 (+ 22056us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:27.469173 (+    63us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:27.469203 (+    30us) tablet.cc:685] Acquiring locks for 23 operations
0709 12:22:27.477224 (+  8021us) tablet.cc:701] Row locks acquired
0709 12:22:27.477245 (+    21us) write_op.cc:260] PREPARE: finished
0709 12:22:27.477403 (+   158us) write_op.cc:270] Start()
0709 12:22:27.477523 (+   120us) write_op.cc:276] Timestamp: P: 1783599747477349 usec, L: 0
0709 12:22:27.477550 (+    27us) op_driver.cc:348] REPLICATION: starting
0709 12:22:27.582019 (+104469us) log.cc:844] Serialized 9162 byte log entry
0709 12:22:28.879859 (+1297840us) op_driver.cc:464] REPLICATION: finished
0709 12:22:29.178979 (+299120us) write_op.cc:301] APPLY: starting
0709 12:22:29.191570 (+ 12591us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:30.479235 (+1287665us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:30.479526 (+   291us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:31.748729 (+1269203us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:31.749527 (+   798us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=46,key_file_lookups=46,delta_file_lookups=0,mrs_lookups=0
0709 12:22:31.749605 (+    78us) write_op.cc:312] APPLY: finished
0709 12:22:31.750441 (+   836us) log.cc:844] Serialized 204 byte log entry
0709 12:22:32.641345 (+890904us) spinlock_profiling.cc:243] Waited 6.77 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8f9ccb0ac 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b
0709 12:22:33.226152 (+584807us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:33.436748 (+210596us) write_op.cc:454] Released schema lock
0709 12:22:33.436841 (+    93us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:33.437425 (+   584us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:33.437465 (+    40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":163724,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":263933,"apply.run_cpu_time_us":76254,"apply.run_wall_time_us":4271090,"cfile_cache_hit":94,"cfile_cache_hit_bytes":100221,"num_ops":23,"prepare.queue_time_us":127306,"prepare.run_cpu_time_us":14208,"prepare.run_wall_time_us":323054,"raft.queue_time_us":319345,"raft.run_cpu_time_us":22501,"raft.run_wall_time_us":243119,"replication_time_us":1366779,"spinlock_wait_cycles":15570304,"thread_start_us":1704417,"threads_started":5,"wal-append.queue_time_us":1424539,"wal-append.run_cpu_time_us":28184,"wal-append.run_wall_time_us":3022745}]]}
I20260709 12:22:41.915671  4147 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:41.960659  3990 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:41.966864  4196 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:41.967123 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:41.967772  3705 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:41.968374 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:41.951792  4271 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:22:41.952617  4159 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:41.973361 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:41.957118  4015 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:41.978770  3086 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:41.983740 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 63613.1us
I20260709 12:22:42.011541 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:42.012284  3086 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:42.013092  3840 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260709 12:22:42.013334 31642 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:41.986954 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:41.992540 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:41.999358 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:42.000474 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:42.005972  4279 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:41.938186  3620 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:42.022600  3620 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:42.019760 31744 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:42.026415 31673 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:42.002321 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:42.016606 31746 raft_consensus.cc:2410] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:22:42.015054 31750 pending_rounds.cc:85] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 7
I20260709 12:22:42.030216 31750 pending_rounds.cc:107] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:22:42.030552 31750 raft_consensus.cc:2889] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:42.019712  3990 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.020432  4274 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:22:41.946333  4032 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.036829 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:42.033885  3840 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:42.046976  3990 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:42.016294  3946 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:42.035034 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:42.048264 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:41.937945  4257 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.178s
I20260709 12:22:42.068650 31792 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:42.055975  4023 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:42.074231  3086 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:42.080862  3086 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:42.077979 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 665) took 1546 ms. Trace:
W20260709 12:22:42.092475 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 123418us
I20260709 12:22:42.113654  3086 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:42.112485 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.037s	sys 0.053s
W20260709 12:22:42.124022 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.637s	user 0.037s	sys 0.053s
W20260709 12:22:42.124413 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 637613us
I20260709 12:22:42.125242 31752 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:22:42.081537  3946 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.129171 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:42.130438 31744 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:42.117251 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:42.084875 31749 rpcz_store.cc:276] 0709 12:22:40.507203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.184809 (+677606us) service_pool.cc:224] Handling call
0709 12:22:41.185350 (+   541us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.185957 (+   607us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.185986 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:41.186598 (+   612us) log.cc:844] Serialized 26 byte log entry
0709 12:22:41.442893 (+256295us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:41.442934 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:41.442999 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.053226 (+610227us) raft_consensus.cc:1685] finished
0709 12:22:42.053248 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.053402 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2900,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":159,"thread_start_us":255792,"threads_started":1,"wal-append.queue_time_us":643031}
I20260709 12:22:42.132377 31746 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:22:42.079030  4023 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:42.138139 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:42.140165 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 246751us
I20260709 12:22:42.112164  4196 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:42.147377 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:42.149411 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:42.150609 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.150988 31752 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:42.135919  3946 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:42.177618 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:42.179594 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:42.150190  4137 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:42.146735  4191 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:42.189823 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63969.5us
I20260709 12:22:42.197863 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.133657  4158 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.054s
I20260709 12:22:42.153745  4196 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:42.200129 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:42.200285 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:42.201505 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:42.204159 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.203081 32372 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:42.134020  4147 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.048s
I20260709 12:22:42.187770 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:42.211028 31674 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:42.209275  4191 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.211472 31674 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:42.212479 31674 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.213442  4191 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:42.213814 31674 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:42.214099  4191 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:42.208659  4196 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:42.191434  4137 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:42.210662 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:42.215780 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:42.240434 31748 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:42.219700 31676 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:42.220383 31669 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:42.221419 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:42.243069 31675 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:22:42.221714 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:42.244400 31678 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:22:42.224937 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 211776us
I20260709 12:22:42.230605 31749 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:42.248742 31674 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:42.216574 31747 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:42.221971 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:42.235924 31752 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
W20260709 12:22:42.271760 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62499.7us
W20260709 12:22:42.272200 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 84556.7us
I20260709 12:22:42.277683  4279 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:42.274946 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:42.283308 31672 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:42.170058  4015 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:42.287976 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 90248.5us
I20260709 12:22:42.220947  4137 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:42.215407  4032 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:42.279623  3840 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:42.220520  3990 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:22:42.270982  4273 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:42.214612  4191 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:42.336773 31745 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
W20260709 12:22:42.338114 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 282139us
W20260709 12:22:42.343483 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.347219 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.316772 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:42.353873 31748 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:42.355957 31792 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.360232 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.360507 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.386107 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 658) took 2224 ms. Trace:
I20260709 12:22:42.393083 31672 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:22:42.394316 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.317903  3840 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:42.402151 31826 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:42.403985 31675 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:22:42.419375 31818 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:42.420324 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:42.355213  4278 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.425853  3086 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:42.425935 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:42.392419 31751 rpcz_store.cc:276] 0709 12:22:40.161480 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.042136 (+880656us) service_pool.cc:224] Handling call
0709 12:22:41.069019 (+ 26883us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.182978 (+113959us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:41.844035 (+661057us) log.cc:844] Serialized 14 byte log entry
0709 12:22:42.280965 (+436930us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:42.294236 (+ 13271us) log.cc:844] Serialized 26 byte log entry
0709 12:22:42.294416 (+   180us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:42.294449 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.294513 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.385674 (+ 91161us) raft_consensus.cc:1685] finished
0709 12:22:42.385718 (+    44us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.386010 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24543,"prepare.queue_time_us":899425,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":3611,"raft.queue_time_us":42388,"raft.run_cpu_time_us":375,"raft.run_wall_time_us":3369,"thread_start_us":1043227,"threads_started":2,"wal-append.queue_time_us":518389}
I20260709 12:22:42.435808 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.348387  3698 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:42.438252 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.401538  3840 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:42.427006  4269 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.776s	user 0.013s	sys 0.000s
I20260709 12:22:42.387665  3620 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.298135  4023 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:42.440176 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 161465us
I20260709 12:22:42.439764 31826 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:42.452837 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.453603 31818 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
W20260709 12:22:42.457388 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 166413us
I20260709 12:22:42.464779 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.392807  4196 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.471397 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:42.472535 31818 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:42.454613  4195 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.483036 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.491894 31678 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
W20260709 12:22:42.496976 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 292877us
I20260709 12:22:42.497897 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:42.500022 32372 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:22:42.516657 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:42.521070  4288 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.088s	user 0.001s	sys 0.000s
I20260709 12:22:42.522352 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.526089 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:42.527472 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.528431  4025 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:42.529157 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.529474  4025 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:42.531895  4307 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:42.533854  4307 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:42.536808 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.540202  3698 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:22:42.538326 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.542315 31821 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:42.542654 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.543378 31821 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:42.471397  4279 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.048s
I20260709 12:22:42.546306 31821 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:22:42.546551  4279 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:42.546725  4286 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.147s
I20260709 12:22:42.547174  4279 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.549500  4279 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:42.550064  4279 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:42.476527  3990 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:42.520809 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:42.522680 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.543397 31822 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:22:42.541568  4277 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.357s	user 0.000s	sys 0.003s
I20260709 12:22:42.528834 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:42.560868 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 51056.5us
I20260709 12:22:42.459425  4269 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.566082  4299 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:42.566742 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:42.566859 31791 connection.cc:256] RPC call timeout handler was delayed by -1.1497583699997449s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.28140000000000004s for a response
I20260709 12:22:42.568107 31818 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:22:42.569044 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.585168 32369 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
W20260709 12:22:42.587282 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0651583699996081s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2908s for a response
I20260709 12:22:42.459848  4278 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:42.570523 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 72734.6us
W20260709 12:22:42.586313 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67264.4us
I20260709 12:22:42.550506  4279 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:42.554503  4032 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.604971 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.605542  3086 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.605770 31822 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:42.602599 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:42.606112  3086 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.589504 32372 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:42.600567 31818 pending_rounds.cc:85] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 6
I20260709 12:22:42.497604  4195 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.608045 31749 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:42.609163 31818 pending_rounds.cc:107] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 14.7
I20260709 12:22:42.609536 31818 raft_consensus.cc:2889] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:42.610702 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:42.611816 32369 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:42.610316  4299 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:42.617082  4299 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 10.9, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.621124  4137 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.621125  3840 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.554908  4191 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.623402  3620 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:42.593189  4139 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:42.628191 32371 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:42.632061 32369 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:42.639724 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:42.635031  3086 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:42.631987 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.640411 31822 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:42.534519  4307 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.609086  4015 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:42.568589  3990 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.604403  4032 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.655416 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:42.647990 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:42.654279 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:42.656095 32369 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:22:42.652732 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:42.658229  4292 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:42.661007  4302 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:42.662099  3946 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.663843 31827 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:42.664855 32372 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:42.666181  3840 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:42.668887  4279 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.670928  4279 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:42.671357  4196 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:42.672637  4195 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:42.674742  4293 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:42.675640  4293 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.676581  4278 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:42.676991  3990 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:42.677527 31824 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:42.678145  4293 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:42.680665 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:42.683811 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.674856  4032 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:42.688573 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:42.689536 32374 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:42.687592 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:42.694569 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:42.700193 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 226506us
W20260709 12:22:42.702365 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 86379us
W20260709 12:22:42.687618  4194 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.765s	user 0.011s	sys 0.002s
I20260709 12:22:42.716710 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:42.717171 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:42.720933 31820 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:42.720911 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:42.723781 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:42.742992 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:42.735837 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:42.744797 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 118703us
I20260709 12:22:42.702337  4032 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:42.704748  4147 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:42.759464 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:42.716446  3840 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:42.757107 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.762130  4147 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:42.777199 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.764943 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.699779  4303 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:42.757635  4000 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.799s	user 0.005s	sys 0.009s
I20260709 12:22:42.775132 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:42.714924  3705 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 12:22:42.793251  3086 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:42.780624 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:42.793916  3086 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:42.793995 31822 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:42.751014  3990 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:22:42.780066 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44342.2us
I20260709 12:22:42.770221 32369 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:22:42.761673  4032 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.794255 31827 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:42.794515  3086 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:42.783174 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.756572  4198 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:42.745996  4196 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.037s
I20260709 12:22:42.781559  4303 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.796988 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:42.746585  4194 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.808096 32374 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:22:42.756526  3620 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.040s
I20260709 12:22:42.790337  4159 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:42.784178  3698 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:42.810621 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:42.810851 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.817050 32369 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:42.812404  4311 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:22:42.818534 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53704.6us
I20260709 12:22:42.804471  4032 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:42.811842  4193 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.882s	user 0.010s	sys 0.005s
I20260709 12:22:42.808465  4257 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:42.812613  4158 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:42.807571  3990 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:42.895105 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.897719  3086 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.818847  4025 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:22:42.898921  4191 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:22:42.815743  4159 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.900749 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.813603  4303 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:42.809420  4198 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:42.907013 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 157987us
I20260709 12:22:42.909135 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:42.912890  3086 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:42.910344 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 103559us
W20260709 12:22:42.914387 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53221.4us
I20260709 12:22:42.911649 31819 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
W20260709 12:22:42.906664 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 100897us
I20260709 12:22:42.915988 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.916347 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.916622 31819 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:42.917300  4324 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:42.917508 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:42.917663 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.917819 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:42.918463  4324 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:42.903963  4158 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.928620  4158 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:42.929112  4318 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:42.919672  4191 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:42.912788  4159 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:42.912093  4025 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:42.906762  4147 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.908864  4311 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
W20260709 12:22:42.919694  4280 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.310s	user 0.001s	sys 0.000s
I20260709 12:22:42.938062  4158 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.940251 31821 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:42.940722 31821 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:42.920969 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.921963 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:42.944258 31827 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:42.923657  4194 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:42.949891  3990 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.950572  4285 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:42.951083  4285 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:42.951444  4285 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:42.951372 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2530) took 1343 ms. Trace:
I20260709 12:22:42.952054 31670 rpcz_store.cc:276] 0709 12:22:41.610484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:41.636414 (+ 25930us) service_pool.cc:224] Handling call
0709 12:22:41.636841 (+   427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:41.673630 (+ 36789us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:41.673693 (+    63us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:41.694255 (+ 20562us) log.cc:844] Serialized 26 byte log entry
0709 12:22:42.151593 (+457338us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:42.151657 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:42.151761 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:42.951082 (+799321us) raft_consensus.cc:1685] finished
0709 12:22:42.951111 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:42.951273 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44613,"raft.queue_time_us":14157,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":5170,"thread_start_us":376282,"threads_started":1,"wal-append.queue_time_us":582779}
I20260709 12:22:42.893561  4193 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.954830  4198 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:42.955624 31677 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:42.925436  4312 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:42.925657  4023 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:42.925763 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:42.927516  4307 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:42.928131  4291 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.318s	user 0.001s	sys 0.001s
I20260709 12:22:42.939308  4025 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.920269 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:42.946076 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:42.940086  4191 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:42.955744  4193 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:42.911339  4137 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:42.959889 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:42.965806 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:42.956537  4312 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:42.972496 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47859.1us
I20260709 12:22:42.970005 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:42.971091 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:42.961284 31821 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:42.981583 31821 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:42.978696  3086 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:42.965333  4324 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:42.993772  4242 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:42.991189 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:42.987577  3705 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:43.013183  3940 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:43.022092 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 246570us
I20260709 12:22:42.979789  4307 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:43.014010 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:43.014495  3840 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:43.018409 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96005.4us
I20260709 12:22:43.013340  4242 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:43.017473  3705 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:43.040902 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49846.7us
I20260709 12:22:43.015745  4198 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:43.035917 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:43.061328 31820 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:43.071345 31821 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:43.015609  4318 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:43.031381  3086 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:22:43.083547 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 272976us
I20260709 12:22:43.073230  4196 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:43.087592 32369 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
W20260709 12:22:43.098771 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.097332  3086 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:43.097224 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:43.099886  3086 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:43.104709  4320 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:22:43.087823  3705 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:43.102403 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:43.108250  3086 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:43.118604 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:43.119308 31820 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:43.102432  4196 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:43.094534  4302 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:43.104056  4198 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:43.128602 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:43.119302  4147 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:43.086230  3940 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.185475  4032 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:43.134976  4307 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:22:43.124500  4317 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.182s	user 0.000s	sys 0.001s
W20260709 12:22:43.136890 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 123027us
W20260709 12:22:43.138441 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.130285 32377 raft_consensus.cc:2410] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 14 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 9.
I20260709 12:22:43.178761  4325 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:43.180761  4322 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.063s
I20260709 12:22:43.128929  3990 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.183159  4158 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:43.184919  4191 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.183576  4198 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:43.184885  4321 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:22:43.185263  4147 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:43.127943  4196 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:43.236311 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:43.236493  4159 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:43.237129  4322 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:43.138114  4195 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.083s
I20260709 12:22:43.238811  4307 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:43.239598  4322 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:22:43.190421 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63517.9us
I20260709 12:22:43.238739  4191 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:43.182528  4331 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:22:43.225739  3990 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:43.243306  4147 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:43.244751  3990 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:43.244478  4285 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:43.250703  4332 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.115s	user 0.002s	sys 0.000s
I20260709 12:22:43.250569  4196 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.216454 31820 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
W20260709 12:22:43.188024 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59496.4us
I20260709 12:22:43.217953 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:43.253141 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:43.135049  4242 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:43.187433 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 59742.4us
I20260709 12:22:43.178797  4302 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:43.263552 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 178137us
I20260709 12:22:43.221571  4193 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:43.266938  4193 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:43.267661 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:43.267918 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:43.268905 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1751) took 1264 ms. Trace:
I20260709 12:22:43.269083 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:43.258834  4257 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.261404 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:43.278461 32369 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
W20260709 12:22:43.279230 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.261904  4286 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:43.269366 31677 rpcz_store.cc:276] 0709 12:22:42.004297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.004502 (+   205us) service_pool.cc:224] Handling call
0709 12:22:43.268679 (+1264177us) spinlock_profiling.cc:243] Waited 902 ms on lock 0x7b5400181fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:43.268825 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2075036928}
I20260709 12:22:43.290372 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:43.284088 31678 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:43.291924 31820 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:43.294435 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:43.301748 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:43.298209 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:43.305089 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:43.305050  3086 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:43.306272  3086 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:43.306377 31826 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:22:43.307451 31669 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 9. Current term is 10. Ops: []
I20260709 12:22:43.308110 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:43.308768  3086 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:43.308761 31670 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:43.259590  4285 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:43.310621 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.314884 32377 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:43.315583 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:43.312198 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:43.290831  4023 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.323688 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:43.290300  4158 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
W20260709 12:22:43.328764 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.318466 31676 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:43.334614 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:43.334993 31670 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:43.335011 31824 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:22:43.337647 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:43.338733 31820 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:43.340875 31675 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:43.339869 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:43.269188  4303 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.316123  4269 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:43.267936  4193 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.348948  4193 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:43.328006  3840 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:43.351619  4137 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:43.277962  4200 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.050s
I20260709 12:22:43.340934 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:43.342655  3086 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:43.347276  4334 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
I20260709 12:22:43.350787  4032 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:43.316419  4191 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:43.373039  3946 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:43.357196 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 92923us
I20260709 12:22:43.305681  4322 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:22:43.374543 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:43.375044  4311 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:43.375545  4311 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:43.305305  4307 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.033s
W20260709 12:22:43.370741 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 132989us
I20260709 12:22:43.343284  4312 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:43.371042 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:43.381606 32369 raft_consensus.cc:2368] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
W20260709 12:22:43.373401 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 43761.8us
I20260709 12:22:43.330487  3940 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:43.297744  4257 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.380204 31671 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:43.380666 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:43.387743 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:43.388934  3086 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:43.390017 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:43.343628  4281 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:43.389914 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:43.375929  4311 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:43.394531 31675 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:43.395606 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:43.395895 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:43.397429 31820 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:43.397686 31821 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:43.399260 31821 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:43.377068  4198 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.403352  4198 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.405077 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:43.405779  4198 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:43.406564  4325 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:22:43.382824  4158 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.409152  4158 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:43.409586 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:43.390803  3840 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:43.415354  3840 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:43.416497  3840 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.418212  3840 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:43.419061 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:43.425099  4257 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:43.426931  4307 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:43.428148  4304 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:43.428613  4278 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:22:43.428995  4015 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:43.334086  4023 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.431455  4023 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:43.431519  4325 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:43.400424 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:43.432134  4303 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:43.399300 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:43.401265 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:43.402124 32369 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: [12.10-12.10]
I20260709 12:22:43.432580 31818 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:43.428313 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:43.400861 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:43.437268 31678 raft_consensus.cc:2410] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
W20260709 12:22:43.447105 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 145487us
I20260709 12:22:43.449224 31826 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:43.452280 31820 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:43.455681 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:43.459030 31821 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: [12.7-14.8]
I20260709 12:22:43.460985 31821 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 13. Ops: []
W20260709 12:22:43.464876 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.479020 31824 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:22:43.480597 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:43.512914 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 103404us
I20260709 12:22:43.513274  3086 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.513834  3086 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.433524  4302 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:43.475594  4299 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:22:43.433372  4307 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.087s
I20260709 12:22:43.514529  4312 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:43.526666  3086 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:43.534289 31824 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:43.432195  4325 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:43.519297  3940 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.519479  3698 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:43.539439 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:43.516266  4015 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 12:22:43.519984  4278 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 12:22:43.547629 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:43.552997 31824 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:43.541870  4299 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:43.558527 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:43.524474  4303 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20260709 12:22:43.556299 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:43.572914  4326 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.573766  4326 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.576515  4326 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:43.581552  4307 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:43.584765  4322 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:43.585404  4322 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:43.538002  4312 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:43.570096 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:43.542527  3940 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:43.546989  3698 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:43.629449 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:43.631615 31675 raft_consensus.cc:2410] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 13 because replica has last-logged OpId of term: 12 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:22:43.633002 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:43.594259  4279 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
I20260709 12:22:43.646734 31670 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 11.
I20260709 12:22:43.670436 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:43.671557 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:43.672124 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:43.672456 31821 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:43.651525 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:43.653250  4307 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:43.675585 31678 raft_consensus.cc:2368] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16.
I20260709 12:22:43.654083  3940 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:43.659634 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 285140us
I20260709 12:22:43.653491  4325 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:43.678546 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:43.679524 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:43.683693 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:43.687412 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 65011us
I20260709 12:22:43.687705 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:43.690327 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:43.697311 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:43.647658  4158 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:43.699553 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:43.659830  3946 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:43.701630 31824 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:43.702056 31672 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:22:43.708485 31823 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:43.700476 31826 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:43.660553  4159 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:43.707154 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:43.591886  4322 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.718245 31819 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:43.710404 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:43.691195  4194 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:43.749321 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:43.767509 31823 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
W20260709 12:22:43.767637 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 80009.2us
I20260709 12:22:43.749679 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:43.768687 31792 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:43.695966  4000 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.064s
I20260709 12:22:43.769588  4137 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:43.755115  4356 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:43.755327  4350 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 12:22:43.695338  4139 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 12:22:43.658615  3840 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.059s
I20260709 12:22:43.758608  4353 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:43.758911 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 67160.6us
W20260709 12:22:43.706827  3705 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.586s	user 0.009s	sys 0.004s
I20260709 12:22:43.759107  3620 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20260709 12:22:43.760138  4347 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 12:22:43.746132  4312 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.761960 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:43.764585  4323 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:22:43.708194  3946 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:43.775004 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:43.781174 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:43.706537  4318 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:43.701220  4269 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.075s
I20260709 12:22:43.776369 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:43.772733  4356 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:43.770408  4137 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.008s
I20260709 12:22:43.770314  4000 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:43.800588 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:43.798602  3946 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:43.800980  4257 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:43.776937  3705 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.805722  4174 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:43.805944  4000 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:43.804533  4307 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:43.846091 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 224665us
I20260709 12:22:43.849071  3705 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:43.798518  4353 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.851845  4000 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:43.873083 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:43.875756 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 105702us
I20260709 12:22:43.815384  4325 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:43.804932  4303 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.041s
W20260709 12:22:43.880569 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 97956us
I20260709 12:22:43.882553 31792 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:43.882550 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:43.844062  4318 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:43.846894  4174 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:43.889487 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 200698us
I20260709 12:22:43.878748  4325 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:43.902769 31642 leader_election.cc:400] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14.
I20260709 12:22:43.907891 31642 leader_election.cc:403] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:43.891908  4174 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:43.896209  4139 raft_consensus.cc:2764] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260709 12:22:43.921753  3698 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:43.933408  4365 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.005s
I20260709 12:22:43.930172 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:43.898113  4000 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:43.924612 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:43.938409  4365 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:43.911885  4356 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:43.947798  4358 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 13. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14.
I20260709 12:22:43.949373 32342 leader_election.cc:400] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16.
I20260709 12:22:43.949833 32342 leader_election.cc:403] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:43.950836 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:43.917060  4357 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 12:22:43.918445  4137 consensus_queue.cc:1243] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 14.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:22:43.959851  4278 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16.
I20260709 12:22:43.966910  4353 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:43.971680  4193 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.622s	user 0.011s	sys 0.007s
I20260709 12:22:43.940339  4307 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 8.7, Last appended by leader: 7, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.948428  4356 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:43.991097 31672 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 13 index: 12. (index mismatch)
I20260709 12:22:43.991961 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:43.948959  4318 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:43.933815  4350 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20260709 12:22:43.979573  3990 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:22:43.998467 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96919.4us
W20260709 12:22:44.000861 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 118371us
I20260709 12:22:44.017498 31792 leader_election.cc:400] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:44.017410 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:44.017891 31792 leader_election.cc:403] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Cancelling election due to peer responding with higher term
I20260709 12:22:44.007071  4325 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:43.980126  3840 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:44.015626 32376 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:22:44.034485 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:44.035440 31818 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:44.036928 31819 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:22:44.033499 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:44.042475 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:44.043321 31670 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:44.044431 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:44.045040 31670 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:44.047525 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.048138 31750 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.048512 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.049688 31748 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 13: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:44.050413 31744 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:44.051026 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.051491 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:44.052145 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.051069 31748 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:22:44.054267 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.054929  3940 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.055269 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.056018 31747 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 13: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:44.019793  3698 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.057636  3698 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:44.059099  4361 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:44.059566  4361 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:44.060055  4361 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:44.063536  4325 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:44.073895  4366 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election lost for term 12. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:43.993237  4312 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.077195 31749 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:44.080137 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:44.081847 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:44.079008 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47635.1us
I20260709 12:22:44.084673 31819 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 13. Ops: []
I20260709 12:22:44.077987 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2415) took 1369 ms. Trace:
I20260709 12:22:44.086189 31825 rpcz_store.cc:276] 0709 12:22:42.715359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:42.735202 (+ 19843us) service_pool.cc:224] Handling call
0709 12:22:43.273129 (+537927us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:43.273523 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:43.273565 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:43.273610 (+    45us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:43.273645 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:43.273704 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.077753 (+804049us) spinlock_profiling.cc:243] Waited 517 ms on lock 0x7b54001c89d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:44.077913 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1189224832}
W20260709 12:22:44.024818  4311 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.578s	user 0.010s	sys 0.003s
W20260709 12:22:44.091095 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.055490  3940 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.087854  4193 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:43.986308  4285 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.091s
I20260709 12:22:44.108989 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:44.083011  3698 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:44.127511 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 108765us
I20260709 12:22:44.133363 31750 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:44.141095 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:44.082803  4318 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:44.134653  4279 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:44.152133 31670 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:44.131669  4350 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:44.124132  4196 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:44.156068 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:44.038496  4174 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:44.180174  3940 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:44.126813  3840 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 12:22:44.123646  4309 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 12:22:44.181532  4174 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:44.190830 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.191238  4318 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.193756 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:44.095676  4312 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.197381 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:44.199517  4194 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:44.200068 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:44.201383 31674 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:44.204973  4312 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:44.145479  4193 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.209558  4193 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:44.097469  4311 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.162791  4354 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 12:22:44.162822  4198 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:44.159523  4025 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
I20260709 12:22:44.163944  4355 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:44.164609 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:44.164726  4281 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:44.098027  3946 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.165491 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:44.177762  4304 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:44.200094  4194 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.250658  4304 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:44.250918 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:44.251727 31677 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:44.251855  4304 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:44.252748 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:44.253412  4304 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.242772 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:44.259087 31674 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:44.260435 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:44.248860  3705 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:44.250078  4322 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:44.254987  4309 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Stepping down as leader of term 9
I20260709 12:22:44.255539  3940 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:44.244360 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:44.241804 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:44.267719 31672 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.262365 31677 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:44.263371 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 380883us
I20260709 12:22:44.269389 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:44.269702 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 104396us
I20260709 12:22:44.272491 31676 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:44.275347 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.276408 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:44.250936  4194 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:44.273063 31676 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:44.289321 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.279843 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:44.279368 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 132661us
I20260709 12:22:44.263798  3705 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:44.316210 31671 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 8. (term mismatch)
I20260709 12:22:44.316553 31678 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:44.266598  4309 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:44.319357 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:44.278673 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:44.333740 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 55167.5us
I20260709 12:22:44.335354 31717 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:44.322705 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 166729us
I20260709 12:22:44.322986 31671 pending_rounds.cc:85] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 7
I20260709 12:22:44.323837  4354 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:44.336422 31671 pending_rounds.cc:107] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 14.8
I20260709 12:22:44.271065  3940 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:44.327764 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 734) took 1069 ms. Trace:
I20260709 12:22:44.355777 31671 raft_consensus.cc:2889] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:44.310490  4269 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:44.281201  4335 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:44.344053 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:44.363659 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:44.367317 31818 raft_consensus.cc:2410] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260709 12:22:44.332067  4198 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:44.331131  3705 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.352027  3946 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:44.263278  4299 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.055s
I20260709 12:22:44.352847 31676 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:44.375643 31676 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.376744 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.377454 31748 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:44.377683 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.356135 31827 rpcz_store.cc:276] 0709 12:22:43.258212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:43.290775 (+ 32563us) service_pool.cc:224] Handling call
0709 12:22:43.311810 (+ 21035us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:43.328369 (+ 16559us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:43.328405 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:43.328446 (+    41us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:43.328479 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:44.327549 (+999070us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b540043dad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:44.327592 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:44.327705 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37620608}
I20260709 12:22:44.361212 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 9 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.378903 31747 raft_consensus.cc:2368] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:44.378811 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.322186  4318 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:44.336510 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:44.346446  4354 consensus_queue.cc:1243] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 14.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:22:44.342846  4200 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:44.370757 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.380541  4354 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:44.377604 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.381904 31745 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
W20260709 12:22:44.381912 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45471.4us
I20260709 12:22:44.383576 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.383742 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:44.386160 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.386461  4354 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.386912 31745 raft_consensus.cc:2376] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11. Re-sending same reply.
I20260709 12:22:44.389654 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:44.374306 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.390333 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.343381  4196 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:44.391113 31744 raft_consensus.cc:2368] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14.
I20260709 12:22:44.391062 31745 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:44.354254  4286 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Stepping down as leader of term 13
W20260709 12:22:44.396697  4375 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.078s	user 0.001s	sys 0.000s
I20260709 12:22:44.362857  4269 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:44.368777  4195 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:44.415072 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.412263 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.412515 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.413928 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.416646 31750 raft_consensus.cc:2368] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:44.374567  4198 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.415977 31748 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.417357  4198 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:44.422739 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.417028 31676 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:22:44.427450 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.398610  3698 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:44.433601 31748 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:44.434954 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.436184 31744 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:22:44.399084  4286 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:44.415560  4269 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.452600 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:44.454207 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.410936  4200 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:44.411762  4196 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:44.415333  4195 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.413841  4318 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:44.456189 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.457250 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:44.420785  4353 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:44.458695  4195 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:44.461557 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:44.463426 32374 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:22:44.462512 32377 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:44.466868  4000 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:44.467293  4311 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:44.436568  3698 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.448205  4025 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.468967  4196 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.473248 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.476570 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:44.481344 31747 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.481648 31676 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 9.
I20260709 12:22:44.474748 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:44.480268 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0806868039999245s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1549s for a response
W20260709 12:22:44.482465 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0239868039998328s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1612s for a response
I20260709 12:22:44.483752 31672 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:44.484901 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:44.485579 31672 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:22:44.486590 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.492244 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:44.475822  3698 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:44.495193 31669 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:44.467582  4000 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:44.437053  3940 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:44.504055 31672 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:44.483659  4139 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.428845  4309 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.519004  4139 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.512144 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.530927 31669 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
W20260709 12:22:44.536000  4388 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.128s	user 0.001s	sys 0.000s
I20260709 12:22:44.541419  4318 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.526614  3946 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:44.459929 31750 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:22:44.545112 31750 raft_consensus.cc:606] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:44.545585 31750 raft_consensus.cc:614] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:44.516317  4353 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.516635  4137 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:44.547652 31747 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:44.484331  4025 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.552039 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.563684 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:44.549310  4384 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:22:44.557389  4354 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:44.565977 31677 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:44.551807 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.569095 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:44.571655 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:44.572881  4194 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.552685  4353 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:44.575160  4362 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:44.575598  4362 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:44.576287  4353 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:44.576589 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:44.574359 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.577960 32372 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:22:44.578298 31670 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:44.579849 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:44.580147  4361 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.520s	user 0.012s	sys 0.001s
I20260709 12:22:44.580637  4361 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.567222 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:44.580662 31744 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:44.583709 31676 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:44.586652 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.588654 32376 raft_consensus.cc:2410] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:22:44.519958  4286 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:44.524497  4200 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.564826  4309 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:44.601788 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:44.615456 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:44.616402 31827 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:44.616531 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:44.617216 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.615545 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:44.622462 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:44.624022 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:44.624534 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.623091 31821 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:44.631539 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.629088  4286 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:44.588783  4032 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:44.633093 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 81366us
I20260709 12:22:44.638175 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.625803  4324 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.646180  4324 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:44.626016  4137 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:44.625205  3946 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:44.650074  3946 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:44.653301  3946 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:44.649320  4137 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:44.655059 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.655776  4159 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:22:44.656489  4361 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:44.646934  4194 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:44.649981  4158 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.008s
I20260709 12:22:44.655769 31744 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:44.648579  4324 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:44.643206  4032 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.663542 31674 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:22:44.672581 31744 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:44.675028  4032 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:44.676136 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.681931 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.682574  4200 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:44.674397  4195 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.680337 31827 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:44.621495  4281 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:44.686712 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:44.702544 32372 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:44.707155 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:44.709165 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:44.710086 31670 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:44.711794 31669 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:22:44.712714 31676 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:44.714999 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.637728  4312 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.064s
I20260709 12:22:44.717617 31671 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:22:44.718118 31753 raft_consensus.cc:2368] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13.
I20260709 12:22:44.718935 31752 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:44.720391 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.721674 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.719194 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:44.726339 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:44.727290 31677 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:44.642174  4385 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.082s
I20260709 12:22:44.722360 31745 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:44.734817  4195 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:44.740253  4357 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 13 current term: 13 is higher.
I20260709 12:22:44.744594  4025 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:44.708946  4281 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:44.749114  4281 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:44.752137  4269 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.753315  4374 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:44.759456  4374 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:44.762085  4374 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:44.725402 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:44.754119 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:44.757716 31678 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 10. (term mismatch)
I20260709 12:22:44.763899 31678 pending_rounds.cc:85] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 9
I20260709 12:22:44.764274 31678 pending_rounds.cc:107] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260709 12:22:44.764602 31678 raft_consensus.cc:2889] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:44.765491 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.729970  4200 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260709 12:22:44.766832 31818 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:44.766482  3705 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:44.767228  4200 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:44.767741  4200 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:44.768954  3705 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:44.718215  3698 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:44.771747  3698 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:44.717803  4000 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:44.775389 31678 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:44.776106 31747 raft_consensus.cc:2368] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12.
I20260709 12:22:44.776502  4269 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:44.709444  4137 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.022s
I20260709 12:22:44.724131  4198 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:44.778505  4384 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:22:44.780889 31818 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:44.785176 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:44.787688 31821 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 7. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 12:22:44.787352 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 156580us
I20260709 12:22:44.788930 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:44.788981 31745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:22:44.794025 31745 raft_consensus.cc:606] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Received request to transfer leadership to TS b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.795097 31824 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 13. Ops: []
I20260709 12:22:44.795455 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:44.797379 31827 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: []
W20260709 12:22:44.798345 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 164935us
I20260709 12:22:44.798686 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:44.801172 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.802170 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.807224 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.807992 31747 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:44.813635 31791 leader_election.cc:400] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:44.814165 31791 leader_election.cc:403] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:44.817052 31750 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:44.816558 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:44.819475 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:44.788612  4383 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:44.825397 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.803256 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:44.789770  4374 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:44.828804 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.810223  4000 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:44.836452 31747 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
W20260709 12:22:44.837252 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.837410 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.837656 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:44.829844 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.838498 31747 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 13: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:44.832967 31745 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:44.839150  4335 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:44.841758  4335 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:44.834059 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.811908  3698 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:22:44.826179  4137 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:44.823931  4023 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:22:44.812348  4309 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 12:22:44.847842 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:44.849619 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.843078  4312 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:44.825635  4025 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.021s
I20260709 12:22:44.824260  4357 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 12:22:44.836897  4383 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:44.852870  4368 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:44.820899  3705 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.023s
I20260709 12:22:44.859105  4281 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.006s
I20260709 12:22:44.869097 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:44.869969 32369 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:44.849561  3698 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:44.826077  4356 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 12:22:44.832060  4374 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:44.856259 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.827488  4361 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 12:22:44.847369  4269 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:44.860256 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.862798 31753 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: []
W20260709 12:22:44.872378 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:44.854995  4401 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:44.875656 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:44.850530  4195 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:44.856436  4309 consensus_queue.cc:1243] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 12.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:22:44.878022 31746 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: [12.7-13.8]
I20260709 12:22:44.891556 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.862088  4368 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:44.963315 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 71864.1us
I20260709 12:22:44.890974  4374 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:44.965174 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:44.967862 31746 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:22:44.969995 31753 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:22:44.971657 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:44.971753 31753 raft_consensus.cc:606] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:44.972445 31753 raft_consensus.cc:614] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:22:44.896117  4394 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:44.953608  4302 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:44.955698  4307 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:44.991334 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 132595us
I20260709 12:22:44.962853  4318 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:22:44.973708  4385 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:22:44.987635  4257 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:22:45.010846  4257 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:45.012108  4257 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:45.013537  4257 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:44.983373  4400 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:22:45.024940 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.028908  3940 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:44.963135  4376 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:44.861851  4335 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:45.098863 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 103199us
I20260709 12:22:45.003306 31746 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 10 index: 8. (index mismatch)
W20260709 12:22:45.006745 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.006660 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:22:45.103230 31752 raft_consensus.cc:606] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Received request to transfer leadership to TS b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:44.986215  4394 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.089399 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:45.103612 31752 raft_consensus.cc:614] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:22:45.105304 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 107440us
I20260709 12:22:45.111840 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:45.113377 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:45.100982 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:45.143695  4023 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:45.143899  3705 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:45.143790 31752 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:44.992177  4302 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:44.999231  4307 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.149241 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:45.095600  4000 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.166144 31752 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:22:45.080726  4198 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:45.173895 31746 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:45.127704  4394 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:45.182403 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 81510.9us
I20260709 12:22:44.862313  4312 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:45.145538  4242 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.144363  4353 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.118858  4376 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:45.190976 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:45.191283  4353 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.190976  4242 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:45.193090  4376 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:45.193748  4376 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.193857  4353 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:45.167944  4368 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:45.195169  4242 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:45.195736  4368 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:45.196076 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:45.198700 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:45.199445 31746 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:45.201797  4312 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.144394  4023 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.203006  4023 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:45.209532 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:45.210328 31675 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:45.187711 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:45.153855  4032 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:45.154282  4322 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.153730  4324 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.233083  4324 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.162102  4307 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:45.108197  4025 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.074s
I20260709 12:22:45.185495 31791 leader_election.cc:400] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14.
I20260709 12:22:45.187189 31752 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:45.110919  4281 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.074s
W20260709 12:22:45.237545 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53962.9us
I20260709 12:22:45.213140 31678 raft_consensus.cc:2410] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 8.
I20260709 12:22:45.144492  4385 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.075s
I20260709 12:22:45.239373 31791 leader_election.cc:400] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:45.184820  4366 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:45.240351 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:45.241997  4366 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:45.231841  3705 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:45.242791  4269 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:45.244053 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:45.244709 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:45.244732 31678 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 12
I20260709 12:22:45.248181 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 675) took 1200 ms. Trace:
I20260709 12:22:45.249081 31749 rpcz_store.cc:276] 0709 12:22:44.047276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.054013 (+  6737us) service_pool.cc:224] Handling call
0709 12:22:45.247971 (+1193958us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b54004042d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:45.248111 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50984192}
I20260709 12:22:45.194295  4376 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:45.255448 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:45.106887  4383 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.227466  4195 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:45.185871  4000 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:45.272078 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:45.274324 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.276895 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:45.279903 31822 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:45.259398  4269 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:45.257000  3946 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:45.289198 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:45.244757  4198 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:22:45.245499  4322 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.286581  4303 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.007s
I20260709 12:22:45.298353 31748 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:45.303900 31675 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
W20260709 12:22:45.305459 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 156355us
I20260709 12:22:45.308411 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:45.317584 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.243120  4366 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.338296 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:45.339771 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:45.348398 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:45.349103 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:45.349308 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 200167us
I20260709 12:22:45.314723  4242 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:45.312547  4400 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:45.351431 31748 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:45.275089  4302 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.360967 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:45.353286 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:45.361699 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:45.362016 31677 pending_rounds.cc:85] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 9
I20260709 12:22:45.361873 32369 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:45.362564 31677 pending_rounds.cc:107] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 12.10
I20260709 12:22:45.353822 31822 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:45.300312  3946 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:45.363732 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 10 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:45.354892 31824 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:45.364135 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:45.365506 31677 raft_consensus.cc:2889] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:45.366312 31642 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:45.367821 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:45.368852 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:45.371665 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:45.375653 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:45.383890  4322 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:45.388159 31744 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
W20260709 12:22:45.390496 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.361250  4318 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:22:45.357894  4358 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:45.390455 31678 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:45.361878  4269 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:45.376899 32376 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:45.363481  4303 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:45.396699 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:45.363283  3940 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:45.402165 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:45.392161  4322 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.423347 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:45.425774 31821 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:45.403152  3940 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:45.388549  3990 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 12:22:45.444736  4269 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:45.409925  4361 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:45.456846  4357 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:22:45.446967  4200 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.679s	user 0.013s	sys 0.001s
I20260709 12:22:45.490075 31823 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 12:22:45.492758 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 178485us
I20260709 12:22:45.442474  4376 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:45.499508 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:45.501008 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:45.497022 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 187749us
I20260709 12:22:45.434149  4322 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:45.506819 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:45.473033  4000 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:45.499167  4335 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.494333  4361 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:45.523988 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 157744us
I20260709 12:22:45.525355 31642 leader_election.cc:400] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13.
I20260709 12:22:45.554026 31642 leader_election.cc:403] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:22:45.521121 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 164300us
W20260709 12:22:45.553298 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:45.536317  4423 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.079s	user 0.000s	sys 0.004s
I20260709 12:22:45.499183  4200 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.507217  4394 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.578570  4394 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:45.576368  4200 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:45.588707 31826 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 10: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:45.597160 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.602118  4362 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.603974  4362 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:45.606813  4429 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.013s
W20260709 12:22:45.607510  4407 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.182s	user 0.001s	sys 0.000s
W20260709 12:22:45.608467  4397 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.876s	user 0.010s	sys 0.006s
I20260709 12:22:45.608204  4417 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:22:45.556075  4242 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:45.615283 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 108547us
I20260709 12:22:45.616748 31717 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:45.618316 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:45.626066 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62312.9us
I20260709 12:22:45.557514  4421 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:45.630367 31820 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 10. (term mismatch)
I20260709 12:22:45.631007 31820 pending_rounds.cc:85] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 9
I20260709 12:22:45.631260 31820 pending_rounds.cc:107] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260709 12:22:45.631706 31820 raft_consensus.cc:2889] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:45.534252  4361 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.581583  4419 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:45.609047  4406 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
I20260709 12:22:45.528457  4335 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:45.609737  4374 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.624s	user 0.006s	sys 0.005s
I20260709 12:22:45.526456  3940 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:45.648066 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 122855us
W20260709 12:22:45.650192 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 206194us
W20260709 12:22:45.653452 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.596992  4422 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13.
I20260709 12:22:45.637718  3946 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:45.637872  4421 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.646481  4419 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:45.648692  4412 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:45.560873  4269 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.649129  4416 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:45.641487  4200 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20260709 12:22:45.653585 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:45.704531  4413 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
I20260709 12:22:45.671496  4419 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:45.675014 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47625.2us
I20260709 12:22:45.652719 31642 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:45.661070  4409 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.250s	user 0.001s	sys 0.000s
W20260709 12:22:45.707849 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54315.5us
I20260709 12:22:45.668679 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:45.716068 31823 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:45.685346  4303 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.717393  4303 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:45.704223  3940 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:45.720822 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2456) took 1195 ms. Trace:
I20260709 12:22:45.667382  4421 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:45.674410  4426 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.686343  4412 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:45.712720  4200 consensus_queue.cc:1243] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 12.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:22:45.707228  4418 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:45.734937 31745 rpcz_store.cc:276] 0709 12:22:44.525351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.840763 (+315412us) service_pool.cc:224] Handling call
0709 12:22:44.843126 (+  2363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:44.847336 (+  4210us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:44.847375 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:44.859150 (+ 11775us) log.cc:844] Serialized 26 byte log entry
0709 12:22:45.212455 (+353305us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:45.212501 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.212585 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:45.720544 (+507959us) raft_consensus.cc:1685] finished
0709 12:22:45.720574 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:45.720757 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1323,"raft.queue_time_us":2515,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":6524,"thread_start_us":346352,"threads_started":1,"wal-append.queue_time_us":463460}
I20260709 12:22:45.656219  4374 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:45.753453 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.755271  4412 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.759166  4139 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:45.759338 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2440) took 1197 ms. Trace:
I20260709 12:22:45.759711  4322 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:45.713248  4420 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:22:45.760759  4361 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:45.668396  4307 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:45.763101 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 110469us
W20260709 12:22:45.748641  4257 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.735s	user 0.019s	sys 0.002s
I20260709 12:22:45.765038 31818 rpcz_store.cc:276] 0709 12:22:44.561548 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.561717 (+   169us) service_pool.cc:224] Handling call
0709 12:22:45.759155 (+1197438us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54000fc654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:45.759274 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":435965696}
I20260709 12:22:45.766773 31748 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:45.769958 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:45.719952  4242 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:45.738389  4383 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 12:22:45.751413  3990 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:45.782366 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:45.774017  4335 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:45.787133 31642 leader_election.cc:400] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12.
W20260709 12:22:45.790776 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 172547us
I20260709 12:22:45.797886 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:45.798988  4139 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:45.800140 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:45.800283  4439 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:45.801581 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:45.833168 31675 raft_consensus.cc:2368] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14.
I20260709 12:22:45.776942  4418 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:45.803135 31678 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:22:45.766245  4358 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:45.828262 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2458) took 1298 ms. Trace:
I20260709 12:22:45.838970 31750 rpcz_store.cc:276] 0709 12:22:44.530020 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.612647 (+ 82627us) service_pool.cc:224] Handling call
0709 12:22:45.827974 (+1215327us) spinlock_profiling.cc:243] Waited 143 ms on lock 0x7b54000b81d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:45.828153 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":329532288}
I20260709 12:22:45.807020 31673 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:22:45.829532 31642 leader_election.cc:403] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:22:45.840926 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54044.4us
I20260709 12:22:45.841729 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:45.829909 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:45.826412 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:45.830649 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 335216us
I20260709 12:22:45.836642 31675 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 9. (term mismatch)
I20260709 12:22:45.844726 31675 pending_rounds.cc:85] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 8
I20260709 12:22:45.845089 31675 pending_rounds.cc:107] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:22:45.845435 31675 raft_consensus.cc:2889] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:45.802412  4440 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:45.830123 31745 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:22:45.851320 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1760) took 1486 ms. Trace:
I20260709 12:22:45.815207 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:45.848616 31747 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:22:45.851604 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:45.852974 31672 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:45.851738 31674 rpcz_store.cc:276] 0709 12:22:44.365148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.511542 (+146394us) service_pool.cc:224] Handling call
0709 12:22:44.663268 (+151726us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:44.664069 (+   801us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.851118 (+1187049us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b5400046fd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:45.851257 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":319639936}
I20260709 12:22:45.805053  4303 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:45.855685 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:45.853031 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2454) took 1335 ms. Trace:
I20260709 12:22:45.751084  4426 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:45.860390 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:45.861469 31823 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:45.859052 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:45.862702 31818 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:22:45.861620 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:45.855711 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:45.847589  4322 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12.
I20260709 12:22:45.857009 31747 rpcz_store.cc:276] 0709 12:22:44.517910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.839673 (+321763us) service_pool.cc:224] Handling call
0709 12:22:44.850216 (+ 10543us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:45.848384 (+998168us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b54003b5ea0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:22:45.852773 (+  4389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.852975 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23929216}
I20260709 12:22:45.843158  4358 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:45.841624  3990 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:45.874890 31818 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:22:45.875048 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:45.853086  4394 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:45.879875  4429 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:45.880715  4318 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:22:45.850528  4439 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.864595  4174 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 12:22:45.859341  4440 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:45.890407  3946 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.874212  4358 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:45.891258 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:45.921291 31818 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
W20260709 12:22:45.892010  4023 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.646s	user 0.009s	sys 0.005s
I20260709 12:22:45.842216  4139 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.894460  3990 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:45.865706  4356 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260709 12:22:45.895203  4430 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
I20260709 12:22:45.934417 31825 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:22:45.896032 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:45.834838  4200 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 12:22:45.870056  4303 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:45.901991 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:45.805228  4361 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:45.923791 31822 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:22:45.927800 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:45.928879 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 59195.9us
I20260709 12:22:45.904639  4312 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:45.906569  4318 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:45.935235 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2442) took 1369 ms. Trace:
I20260709 12:22:45.943789 31825 rpcz_store.cc:276] 0709 12:22:44.566084 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.566276 (+   192us) service_pool.cc:224] Handling call
0709 12:22:45.933863 (+1367587us) spinlock_profiling.cc:243] Waited 6.44 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:22:45.934077 (+   214us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:45.934946 (+   869us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.935165 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":14817664}
I20260709 12:22:45.941342 31669 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:45.942324 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:45.927676  4322 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:22:45.950093 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 88564us
I20260709 12:22:45.952165 31675 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:45.929512  4023 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.975916 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:45.939810  4394 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:45.926862  3946 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.947927  4318 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:22:45.950819  4174 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:45.988821 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:45.989396 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:45.949136  4312 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:45.993523  4414 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:45.951150  4418 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 11.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:45.995890  4414 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:45.999727 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:46.000746  4414 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:46.007819  4456 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:46.008636  4456 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:46.026233 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:45.971762  4322 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:45.990928  4374 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:45.991839  4356 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:46.041800 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:46.041365 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:46.050276 32372 raft_consensus.cc:1217] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Deduplicated request from leader. Original: 10.9->[13.10-13.10]   Dedup: 13.10->[]
W20260709 12:22:46.051127  4447 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.122s	user 0.001s	sys 0.001s
I20260709 12:22:46.051312 32372 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:22:46.052773 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:46.053579 32373 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:46.055810  4023 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:46.056150  4439 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:46.042433 32378 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:46.049474 32374 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
W20260709 12:22:46.042234 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.009161  4456 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:46.042302 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:46.057014  4426 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:45.947085  4361 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:46.068017 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 112643us
I20260709 12:22:45.970697  4411 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 12.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.128s
I20260709 12:22:46.073583  4452 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:45.983789  4196 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.074s
I20260709 12:22:46.045326  4174 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:46.110103  4361 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:46.111047  4174 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.112291  4446 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:46.112526  4358 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:46.033177  4422 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.067s
I20260709 12:22:46.082904 31669 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:46.048024  4307 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:46.091902 31676 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:46.000708  4312 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:46.098609  4322 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:46.099267  4139 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:46.068889  4356 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.118119  4394 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:46.155674  4356 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:46.157061 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:46.107372 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 98333.4us
I20260709 12:22:46.160156 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:46.116199 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:46.161273 31669 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
W20260709 12:22:46.117357  4401 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.149s	user 0.015s	sys 0.000s
I20260709 12:22:46.114562  4433 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:46.177302 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:46.179286  4418 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:46.116783  3946 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:46.115227  4358 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:46.118829  4440 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:46.183600 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 154583us
I20260709 12:22:46.192953 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:46.193444 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:46.177202  4307 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader election won for term 10
I20260709 12:22:46.173763  4401 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:46.176654  4356 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:46.198349  4457 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.174099  4414 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:46.198967  4433 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:46.195459  4451 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.213s
W20260709 12:22:46.201087 31744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2459) took 1648 ms (client timeout 2153 ms). Trace:
I20260709 12:22:46.219309 31751 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
W20260709 12:22:46.218447 31744 rpcz_store.cc:269] 0709 12:22:44.553028 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.681649 (+128621us) service_pool.cc:224] Handling call
0709 12:22:46.200952 (+1519303us) spinlock_profiling.cc:243] Waited 676 ms on lock 0x7b5400213ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:46.201031 (+    79us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1554780928}
I20260709 12:22:46.226256 31674 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:22:46.228207  4461 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
W20260709 12:22:46.239282 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.215368  4433 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.240384  4433 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:46.215483  4457 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.243152 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:46.243867 31672 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:46.245689  4457 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:46.208786  4307 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:46.247694  4307 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:46.190523  4418 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.208098  4242 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:46.205780  4394 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.055s
I20260709 12:22:46.281656  4366 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:46.284711  4464 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:46.285326  4464 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:46.277469  4441 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:46.287005  4441 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:46.289697  4441 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:46.282289  4366 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:46.291716  4366 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.293131  4366 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:46.286506  4464 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.296277  4464 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:46.298903  4418 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:46.299613  4421 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.548s	user 0.012s	sys 0.000s
I20260709 12:22:46.274981  4358 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:46.300072  4421 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.300994  4358 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:46.219638  4356 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.262910  4242 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.218072  4312 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 12:22:46.417062 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:46.400998 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:46.496731 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:46.497073 31751 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
W20260709 12:22:46.497522 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96611.2us
I20260709 12:22:46.411470 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:46.362596 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.499897 31752 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
W20260709 12:22:46.415179  4462 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.314s	user 0.004s	sys 0.000s
I20260709 12:22:46.355273 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:46.361496  4419 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.617s	user 0.002s	sys 0.008s
W20260709 12:22:46.397778  4477 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.051s	user 0.002s	sys 0.000s
I20260709 12:22:46.507828  4353 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.425066 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:46.508487  4353 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.509104 31747 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:46.343552  4242 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:46.511447  4015 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:46.511886  4015 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:46.424672 31791 leader_election.cc:400] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14.
I20260709 12:22:46.512475 31791 leader_election.cc:403] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
W20260709 12:22:46.513252 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 88758.6us
I20260709 12:22:46.513998 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:46.514987  4427 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14.
I20260709 12:22:46.515663  4427 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:46.517211  4383 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:46.386473  4356 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:46.355435  4472 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:22:46.425942 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:46.520980 31745 raft_consensus.cc:2368] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14.
I20260709 12:22:46.302587  4421 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:46.418341 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:46.468947  3698 raft_consensus.cc:993] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: : Instructing follower b95aeeaf4c51442b8092cff1de674f2b to start an election
I20260709 12:22:46.528007 31744 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:46.528990 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:46.420955 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:46.417240 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 743) took 1455 ms. Trace:
I20260709 12:22:46.407246  4318 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:46.492060 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.492846  4433 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.535951 31820 rpcz_store.cc:276] 0709 12:22:44.961596 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.961741 (+   145us) service_pool.cc:224] Handling call
0709 12:22:44.961994 (+   253us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:45.640215 (+678221us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.416913 (+776698us) spinlock_profiling.cc:243] Waited 668 ms on lock 0x7b54001e9bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:46.417163 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1536073600}
I20260709 12:22:46.539307  3698 raft_consensus.cc:1081] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Signalling peer b95aeeaf4c51442b8092cff1de674f2b to start an election
I20260709 12:22:46.478219  4281 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:46.308213  4457 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:46.387743  4428 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:46.346428  4467 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.541795  4281 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:46.542059  4467 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.543040  4318 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:46.543680  4137 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:22:46.544262  4353 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:46.355167  4471 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260709 12:22:46.424607 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:46.546258  4433 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:46.362879  4324 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:46.546917  4433 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:46.546993  4324 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:46.547650  4324 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.466446  4429 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.344583  4414 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:46.494755  4358 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:46.571499 31677 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:22:46.540608 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:46.544392  4467 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:46.547559  4433 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.503082 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:46.575276 32373 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:22:46.503073  4357 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:46.346000  4456 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:46.589466 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:46.586519  3698 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.551476 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:46.895087 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 343724us
I20260709 12:22:46.894766 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:46.895886 31751 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:46.896139 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:46.897796  4474 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.898419  4474 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.899438  4307 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:46.900496  4307 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.900995  4307 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:46.902926  4358 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:46.903436  4358 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.903766  4358 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:46.904318 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:46.905823 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:46.906395 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:46.907668 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:46.908277 31753 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:46.907662 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:46.549175  4427 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:46.571344  4457 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:46.862885  4456 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:46.544322  4281 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:46.828887 31674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.001s	sys 0.001s
W20260709 12:22:46.871078 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:46.870836 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 195745us
I20260709 12:22:46.872605 32373 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:22:46.544598  4137 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:46.874109 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:46.876334  4415 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.350s	user 0.003s	sys 0.000s
I20260709 12:22:46.534289  4383 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.066s
W20260709 12:22:46.941701 31674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.005s	sys 0.009s
I20260709 12:22:46.586956  4318 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260709 12:22:46.879617 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:46.880513 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:46.880685 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:46.881291 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:46.881565 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:46.672247  4425 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:46.882388 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:46.528481  4419 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:46.885527 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 325349us
I20260709 12:22:46.887778  4357 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:46.541805  4428 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:46.900002  4309 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.910275 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2583) took 2213 ms. Trace:
I20260709 12:22:46.912338  4427 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:46.892647 31744 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:46.601835 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:46.548573  4429 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:46.979418 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 737) took 1793 ms. Trace:
I20260709 12:22:46.934790 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:46.979841 31676 rpcz_store.cc:276] 0709 12:22:44.696426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.751326 (+ 54900us) service_pool.cc:224] Handling call
0709 12:22:44.751681 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.144065 (+1392384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.902452 (+758387us) spinlock_profiling.cc:243] Waited 1.34 s on lock 0x7b54001b35d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:46.910209 (+  7757us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3081167616}
I20260709 12:22:46.986131 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2602) took 1113 ms. Trace:
W20260709 12:22:46.963442  4483 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:22:46.967351 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:46.914386 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 327004us
I20260709 12:22:46.959570  4383 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:46.942265  4137 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:46.909921  3698 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:46.914098 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:46.915627 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:47.006673 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:46.962195  4474 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:46.959055  4419 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:46.954744  4425 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:46.985639 31671 rpcz_store.cc:276] 0709 12:22:45.186240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.186420 (+   180us) service_pool.cc:224] Handling call
0709 12:22:45.199327 (+ 12907us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:45.199772 (+   445us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:45.199803 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:45.200348 (+   545us) log.cc:844] Serialized 26 byte log entry
0709 12:22:45.395269 (+194921us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:45.395319 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:45.395396 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:46.875592 (+1480196us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:46.965436 (+ 89844us) raft_consensus.cc:1685] finished
0709 12:22:46.965469 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.979326 (+ 13857us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":415,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"spinlock_wait_cycles":25620736,"thread_start_us":194538,"threads_started":1,"wal-append.queue_time_us":252879}
I20260709 12:22:47.022173 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:47.025699 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:47.027935 31676 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:47.031356 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:47.031008  4357 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.032622 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:47.033371 31750 raft_consensus.cc:2368] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16.
I20260709 12:22:47.036062  4457 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:47.038798  4366 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.745s	user 0.011s	sys 0.009s
I20260709 12:22:47.040125  4324 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:47.041119  4456 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.043727  4137 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.044349  4429 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:47.044916  4429 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:47.045396  4318 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:47.045463  4429 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.046118  4318 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.048238  4318 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:46.992331  4309 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.048734  4318 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:47.034173 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 10 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:47.049206  4318 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.050325  4309 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:47.051096 31750 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:22:46.986403  4427 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:47.053249 31748 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:22:47.071543 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:47.072830 31744 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:47.074405 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:46.973189  4312 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:22:47.084183 31676 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:47.085426 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:47.086416 31750 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:22:46.958503  4242 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.041942  4456 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.102224  3698 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:47.106243  4433 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:46.986732 31669 rpcz_store.cc:276] 0709 12:22:45.872223 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.179180 (+306957us) service_pool.cc:224] Handling call
0709 12:22:46.179589 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.179821 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:46.333833 (+154012us) log.cc:844] Serialized 14 byte log entry
0709 12:22:46.985551 (+651718us) log.cc:844] Serialized 14 byte log entry
0709 12:22:46.985752 (+   201us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:46.985802 (+    50us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:46.985850 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.985950 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:46.986068 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":83330,"prepare.queue_time_us":323147,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":44,"raft.queue_time_us":78160,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":11261,"spinlock_wait_cycles":28607232,"thread_start_us":712017,"threads_started":3,"wal-append.queue_time_us":607520}
I20260709 12:22:47.044937  4137 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:47.108114 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.111717  4429 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:47.148043 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:47.034477  4358 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:47.135612 31821 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2447) took 1783 ms (client timeout 1940 ms). Trace:
W20260709 12:22:47.152630 31821 rpcz_store.cc:269] 0709 12:22:45.352641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.354187 (+  1546us) service_pool.cc:224] Handling call
0709 12:22:47.135322 (+1781135us) spinlock_profiling.cc:243] Waited 57 ms on lock 0x7b54004461d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:47.135526 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":131180288}
I20260709 12:22:47.153319 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:47.154217  4358 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:47.154554 31745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:22:47.155179 31745 raft_consensus.cc:606] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:47.156608 31745 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:47.157873 31745 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:47.160259 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:22:47.115911 31748 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:22:47.070802  4353 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:47.051323  4322 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.123036 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 742) took 1069 ms. Trace:
I20260709 12:22:47.135568  4307 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.137916  4419 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.174285 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:47.174659 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:47.175299 31669 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:47.138707  4433 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:47.145864  4482 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.173s	user 0.000s	sys 0.001s
I20260709 12:22:47.169485 31753 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:47.191795 31751 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:22:47.172380 31745 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:22:47.202795 31753 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: [14.12-14.12]
I20260709 12:22:47.164292 31672 rpcz_store.cc:276] 0709 12:22:46.053795 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.244566 (+190771us) service_pool.cc:224] Handling call
0709 12:22:46.244935 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.245303 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:46.245329 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:46.245837 (+   508us) log.cc:844] Serialized 26 byte log entry
0709 12:22:46.496361 (+250524us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:46.496410 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.496483 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:47.122809 (+626326us) raft_consensus.cc:1685] finished
0709 12:22:47.122831 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.122969 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":327,"raft.queue_time_us":1288,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":250072,"threads_started":1,"wal-append.queue_time_us":396740}
W20260709 12:22:47.120106  4464 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.823s	user 0.015s	sys 0.006s
I20260709 12:22:47.177663 32370 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:47.259168 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:47.016991  4383 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.325819 31670 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 15: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:47.312974 32377 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
W20260709 12:22:47.326344 31718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.940s (SENT) [suppressed 26 similar messages]
I20260709 12:22:47.175122  4451 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.050957  4428 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:47.351939 31793 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.512s	user 0.019s	sys 0.019s
W20260709 12:22:47.352631 31793 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.019s	sys 0.019s
W20260709 12:22:47.353310 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:47.354622 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.357511 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:47.358565 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:47.359357 31669 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:22:47.359753  3698 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:47.313668  4436 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.306212  4361 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.173870  4353 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.378902 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:47.174870  4322 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:47.362962  4482 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:22:47.181064  4419 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.357430  4366 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:47.306660  4433 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.386658 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:22:47.369876  4198 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:22:47.447717 31792 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.005s	sys 0.006s
W20260709 12:22:47.448324 31792 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.005s	sys 0.006s
I20260709 12:22:47.378587 31675 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
 from {username='slave'} at 127.0.0.1:43200
I20260709 12:22:47.313179  4425 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.390575  4400 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:47.407918  4318 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:47.382777  4361 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.377462  4436 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.476104  4356 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:47.480577 31675 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:22:47.519443  4307 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:47.520670  4456 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:47.363454  4451 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.519150 31675 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:47.522944  4451 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:47.430437  4366 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.525329  4366 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:47.515518  4322 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:47.476641  4428 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:47.526115  4198 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:47.526760  4400 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:47.484809  4242 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:47.495536  4425 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:47.492038  4353 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:47.495280  4419 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:47.493932  4361 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:47.507944  4383 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:47.566347 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:47.553069  4309 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.497882  4436 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:47.489470  4433 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:47.569438 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:47.572055 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:47.572715 31790 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.631s	user 0.015s	sys 0.024s
W20260709 12:22:47.607517 31790 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.015s	sys 0.024s
W20260709 12:22:47.608338 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:47.573534 32341 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.000s	sys 0.039s
I20260709 12:22:47.565441  4400 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:47.609187 32341 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.623s	user 0.000s	sys 0.039s
I20260709 12:22:47.550521  4427 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.617964  4427 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:47.575366 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:47.619567 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:47.500010  4015 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.621192  4015 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:47.601982  4383 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:47.569763  4309 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:47.609715 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.615309  4366 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:47.592713  4198 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.625308  4366 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.604621  4318 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:47.604714  4429 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.507222  4356 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.603674  4493 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:22:47.624684 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.001s	sys 0.013s
I20260709 12:22:47.619482  4427 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.639477 31744 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
W20260709 12:22:47.631084 31749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.670s	user 0.006s	sys 0.009s
W20260709 12:22:47.641471 31749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.006s	sys 0.009s
W20260709 12:22:47.623538 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.708s	user 0.007s	sys 0.009s
W20260709 12:22:47.637181 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.014s	sys 0.026s
W20260709 12:22:47.644523 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.729s	user 0.007s	sys 0.009s
W20260709 12:22:47.644644 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.014s	sys 0.026s
W20260709 12:22:47.637643 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.647s	user 0.001s	sys 0.013s
W20260709 12:22:47.636260 31671 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.560s	user 0.005s	sys 0.009s
W20260709 12:22:47.652773 31671 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.576s	user 0.006s	sys 0.009s
W20260709 12:22:47.641048 31674 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.020s	sys 0.048s
W20260709 12:22:47.653450 31674 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.021s	sys 0.049s
W20260709 12:22:47.647116 31641 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.731s	user 0.016s	sys 0.025s
W20260709 12:22:47.654208 31641 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.016s	sys 0.025s
W20260709 12:22:47.646857 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:47.606544 31792 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:47.658666 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 741) took 1605 ms. Trace:
I20260709 12:22:47.672209 31674 rpcz_store.cc:276] 0709 12:22:46.052725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.227675 (+174950us) service_pool.cc:224] Handling call
0709 12:22:46.227893 (+   218us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:46.228095 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:46.312741 (+ 84646us) log.cc:844] Serialized 14 byte log entry
0709 12:22:47.160478 (+847737us) spinlock_profiling.cc:243] Waited 27.5 ms on lock 0x7b4800080058. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:22:47.635862 (+475384us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:47.658183 (+ 22321us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:47.658248 (+    65us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:47.658307 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.658409 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:47.658559 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":275041,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":74679,"raft.run_cpu_time_us":1159,"raft.run_wall_time_us":130430,"spinlock_wait_cycles":93961984,"thread_start_us":1395613,"threads_started":3,"wal-append.queue_time_us":586291}
I20260709 12:22:47.676709 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:47.682016  4309 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:47.647042  4025 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:47.632688  4383 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.693908 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:47.695048 31674 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:47.696067 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:47.694188 31750 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.009s	sys 0.003s
W20260709 12:22:47.698088 31750 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.546s	user 0.009s	sys 0.003s
I20260709 12:22:47.668548  4137 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:47.686120 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 79643.8us
I20260709 12:22:47.639031  4429 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.670729  4464 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:47.637470  4318 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.639941  4493 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.711784 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:47.733428 32369 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:47.643853  4427 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:47.697265  4357 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.001s	sys 0.010s
I20260709 12:22:47.696687 31674 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:47.729259 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:47.671368  4467 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:22:47.703572  4501 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:47.740705 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 415380us
I20260709 12:22:47.737618 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:47.744385 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:47.746320 32344 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.770s	user 0.014s	sys 0.008s
W20260709 12:22:47.746729 32344 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.820s	user 0.015s	sys 0.008s
W20260709 12:22:47.739555  4357 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.001s	sys 0.011s
I20260709 12:22:47.893234 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:47.714115  4137 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:47.898180 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2607) took 1003 ms. Trace:
I20260709 12:22:47.899304 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1912) took 1001 ms. Trace:
I20260709 12:22:47.902833 31748 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 13. Ops: []
I20260709 12:22:47.930414 31670 rpcz_store.cc:276] 0709 12:22:46.894335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.934328 (+ 39993us) service_pool.cc:224] Handling call
0709 12:22:47.897915 (+963587us) spinlock_profiling.cc:243] Waited 339 ms on lock 0x7b54002b0554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:47.898099 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":780054272}
I20260709 12:22:47.850782 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:47.932996 31751 rpcz_store.cc:276] 0709 12:22:46.897514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.914208 (+ 16694us) service_pool.cc:224] Handling call
0709 12:22:47.191414 (+277206us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:47.192382 (+   968us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.899081 (+706699us) spinlock_profiling.cc:243] Waited 224 ms on lock 0x7b54001ed7d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:47.899234 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":516047616}
I20260709 12:22:47.901059 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:47.848081 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:47.942607 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 91938us
I20260709 12:22:47.730063  4464 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:47.944571 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 43600.3us
I20260709 12:22:47.937054 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:47.944682 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:47.487705 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:47.788121  4500 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.983495 31670 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:47.871234  4499 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:47.930684  4503 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.061s	user 0.001s	sys 0.001s
W20260709 12:22:47.883739  4492 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.984s	user 0.010s	sys 0.014s
I20260709 12:22:47.852546  4394 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.989493 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:47.860088  4494 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:47.898413  4429 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:47.917137  4507 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:47.912518  4383 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:47.913658  4356 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:47.960618  4486 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.068s	user 0.001s	sys 0.006s
W20260709 12:22:47.975422  4506 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.099s	user 0.003s	sys 0.000s
I20260709 12:22:47.958510  4436 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:47.892488  4493 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:47.959220  4309 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:22:47.948272  4421 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:22:48.019804 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 171826us
I20260709 12:22:47.728410 31675 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:47.988426  4366 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:48.021931 31676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.017s	sys 0.028s
I20260709 12:22:48.046329 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:48.077134 31676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.019s	sys 0.030s
I20260709 12:22:47.901829  4433 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.113s
W20260709 12:22:48.079293 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.019826  4429 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:47.822252  4501 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:48.086604 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:48.092882 32374 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:48.088944 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.015175  4383 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.092090 32370 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:48.013365  4509 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:48.041525  4484 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.076s	user 0.005s	sys 0.042s
I20260709 12:22:48.077270  4436 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:48.091815  4486 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:22:48.014303  4494 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.157789  4494 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:48.091485  4429 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:48.158879  4429 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:48.161775  4436 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:48.163429  4302 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:22:48.166179  4427 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:48.106572  4505 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.166924  4281 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:48.167224  4505 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:48.167518  4281 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:48.169668  4281 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:48.169826  4505 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:48.014969  4507 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:48.102032  4309 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:48.171489  4400 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.561s	user 0.009s	sys 0.000s
I20260709 12:22:48.082033  4493 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:48.172204  4507 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:22:48.172049  4400 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:48.172997 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.172228  4309 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:48.173897 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.174249  4400 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:48.175717  4485 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.284s	user 0.005s	sys 0.015s
I20260709 12:22:48.103621  4137 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:47.991568  4419 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.091s
I20260709 12:22:48.177145  4318 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:48.045272  4421 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:48.178050  4421 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:48.149469  4451 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.178766  4451 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:48.180915  4451 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:48.145860  4484 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.181s	user 0.005s	sys 0.043s
I20260709 12:22:48.013767  4394 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.182842 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2597) took 2312 ms. Trace:
I20260709 12:22:48.041690  4414 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.183701 32374 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:48.184155  4394 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:48.183990  4414 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.186210  4414 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:48.186319  4025 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:48.187896 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1911) took 1603 ms. Trace:
I20260709 12:22:48.188460 31749 rpcz_store.cc:276] 0709 12:22:46.584484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.863304 (+278820us) service_pool.cc:224] Handling call
0709 12:22:46.863949 (+   645us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:46.908333 (+ 44384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:46.908366 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:46.913843 (+  5477us) log.cc:844] Serialized 26 byte log entry
0709 12:22:47.641944 (+728101us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:47.641986 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:47.642057 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:48.187595 (+545538us) raft_consensus.cc:1685] finished
0709 12:22:48.187627 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:48.187815 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46717,"raft.queue_time_us":44320,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":28448,"thread_start_us":681019,"threads_started":1,"wal-append.queue_time_us":901070}
I20260709 12:22:48.007541  4499 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:48.153206 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 12 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.194346 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 865) took 1314 ms. Trace:
I20260709 12:22:48.197547 32377 rpcz_store.cc:276] 0709 12:22:46.880259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:46.880400 (+   141us) service_pool.cc:224] Handling call
0709 12:22:48.194175 (+1313775us) spinlock_profiling.cc:243] Waited 361 ms on lock 0x7b54004312d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:48.194287 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":829537536}
I20260709 12:22:48.196290 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: true dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:48.198958 31675 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:48.183365 31673 rpcz_store.cc:276] 0709 12:22:45.870855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.873011 (+  2156us) service_pool.cc:224] Handling call
0709 12:22:45.873246 (+   235us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:45.948053 (+ 74807us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:45.948086 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:46.249377 (+301291us) spinlock_profiling.cc:243] Waited 1.27 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:22:46.249840 (+   463us) log.cc:844] Serialized 26 byte log entry
0709 12:22:46.570392 (+320552us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:22:46.570448 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:46.570552 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:48.175085 (+1604533us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:48.182591 (+  7506us) raft_consensus.cc:1685] finished
0709 12:22:48.182614 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:48.182779 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3381,"raft.queue_time_us":28651,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":4089,"spinlock_wait_cycles":70708864,"thread_start_us":394535,"threads_started":2,"wal-append.queue_time_us":641538}
I20260709 12:22:48.199342 32374 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:48.200301 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:47.992574  4500 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.203970 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.207638 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.186954  4025 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:48.206485 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:48.213001  4456 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:48.213533  4456 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:48.214010  4456 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:48.218178 32374 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:48.229758  4361 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:48.230553 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:48.230690  4361 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.232188  4353 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:48.232921  4353 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:48.234025  4361 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:48.224794  4015 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.603s	user 0.010s	sys 0.001s
I20260709 12:22:48.225848  4464 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:48.227178  4499 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:48.234136  4353 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.238517 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.284560 32372 raft_consensus.cc:2410] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 16 because replica has last-logged OpId of term: 14 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 11.
W20260709 12:22:48.240541  4425 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.674s	user 0.010s	sys 0.000s
I20260709 12:22:48.287412 31745 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:22:48.289264 31749 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 13. Current term is 14. Ops: []
W20260709 12:22:48.240686 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.223974  4429 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.249589 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RunLeaderElection from 127.0.0.1:43200 (request call id 2616) took 1080 ms. Trace:
W20260709 12:22:48.252431 31678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.029s	sys 0.070s
W20260709 12:22:48.297549 31678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.770s	user 0.029s	sys 0.070s
I20260709 12:22:48.260155 31744 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 10 index: 9. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:22:48.276081 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:48.299229 31748 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 14: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:48.277715 31751 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:22:48.254664  4304 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:48.297446 31675 rpcz_store.cc:276] 0709 12:22:47.119512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.349055 (+229543us) service_pool.cc:224] Handling call
0709 12:22:48.199673 (+850618us) inbound_call.cc:177] Queueing success response
Metrics: {"dns_us":261}
W20260709 12:22:48.276091 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 538511us
I20260709 12:22:48.278204 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.278431 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:48.302371 31673 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:48.112210  4383 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:48.280606  4428 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.723s	user 0.010s	sys 0.004s
I20260709 12:22:48.293924 32371 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:48.310375 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.312467 31790 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.703s	user 0.057s	sys 0.091s
I20260709 12:22:48.187919  4509 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.320998 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.334939 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:48.337335 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.007s	sys 0.017s
W20260709 12:22:48.337826 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.007s	sys 0.017s
W20260709 12:22:48.338758 31642 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.006s	sys 0.018s
W20260709 12:22:48.339109 31642 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.006s	sys 0.018s
W20260709 12:22:48.339444 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 548137us
I20260709 12:22:48.340595 32342 leader_election.cc:400] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16.
I20260709 12:22:48.340965 32342 leader_election.cc:403] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:22:48.312887 31790 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.704s	user 0.057s	sys 0.091s
I20260709 12:22:48.341922 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:48.301497  4015 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 12:22:48.346992 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.347831 31675 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:48.350718 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.338393 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:48.353085 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.353859 31827 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:48.355916 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.356783 31826 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:48.358752 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.359504 31822 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:22:48.361238  4318 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:48.361629 31827 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:48.362864 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:48.363519 31753 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:48.362937 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:48.364198 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:48.363021 31825 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:22:48.364849 31752 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:48.364799 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:48.372133 31750 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:48.383229 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.372658 31750 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:48.392864 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:48.393879 31824 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:48.396095 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.398736  4356 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:48.399624  4356 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.400758  4429 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:48.311820  4304 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.402129  4356 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:48.415453 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:48.403223  4304 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:48.403890 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.404645 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.404948 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:48.405822 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.405982 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.407897  4318 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.392928 31822 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:22:48.408836 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:48.424930 31827 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:48.424850 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:48.344177  4499 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.342571  4505 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:48.411435 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 69580.9us
I20260709 12:22:48.430459 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:48.387184 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:48.416211 31753 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:22:48.422510 32377 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
I20260709 12:22:48.348980  4361 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:48.425606 31822 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:48.433632 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.426209 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.434877 31669 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:48.385811 31750 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.427443 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:48.431748 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44690.6us
I20260709 12:22:48.432935 31752 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:22:48.446089 31672 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:22:48.434268 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:48.447211 31747 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:48.448541 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:48.449371 31747 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:48.451041 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:48.434386 31819 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:48.451779 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:48.453177 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:48.433099  4318 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.455850 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.464342 31675 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:48.465327 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:48.465965 31675 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:48.457504 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:48.454432 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:48.459860 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:48.460777 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.479187 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:48.467734 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.434762  4322 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:48.429618  4356 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:48.471608 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:48.485476 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:48.486832 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:48.459003  3698 consensus_queue.cc:1175] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5206fd0087404d2da964a5d985a04ff4" responder_term: 14 status { last_received { term: 14 index: 11 } last_committed_idx: 11 last_received_current_leader { term: 14 index: 11 } }
I20260709 12:22:48.445818  4324 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:48.451607  4358 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.499651 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:48.500618 31747 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 10: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:22:48.503274 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 355079us
I20260709 12:22:48.467013  4429 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.502743 31822 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:48.507838 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.433970  4499 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:48.468084  4451 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:48.490962  4419 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:48.510437 31827 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:48.547097 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.547818 31827 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:48.487138 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:48.510933 31747 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:48.482811  4414 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:22:48.494706  4417 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.584s	user 0.021s	sys 0.025s
I20260709 12:22:48.554515 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.527249  4502 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16.
I20260709 12:22:48.528275  4442 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:48.561556 31745 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:48.528364 31675 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:48.528851  4513 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:48.529453  4521 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:48.625219 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54440.5us
I20260709 12:22:48.446272  4505 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.509255 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.633769 31822 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 9.
I20260709 12:22:48.455567  4361 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.641516 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.527909  4516 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:22:48.564814  4487 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.669s	user 0.012s	sys 0.011s
I20260709 12:22:48.556701 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:48.491369  4441 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.038s	sys 0.045s
I20260709 12:22:48.651430 31819 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:48.498908  4324 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:48.528079  4519 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:22:48.494407  4304 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:22:48.652712  4304 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:48.583889 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.654253  4324 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:48.654423 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.502734  4358 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:48.661598  4421 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:48.623144 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 136051us
W20260709 12:22:48.697619  4514 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.357s	user 0.001s	sys 0.000s
I20260709 12:22:48.626531 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.525252  4464 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:48.494590  4015 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.068s
I20260709 12:22:48.631388 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.735778 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.737135 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.647876 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.739632 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.493702  4356 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.503130  4025 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:48.569648 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.653146  4304 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:48.670285 31819 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:22:48.673525 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2598) took 2802 ms. Trace:
I20260709 12:22:48.641326  4521 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:48.679826  4361 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:48.683404 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:48.606246 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:48.626729  4500 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:48.530189  4429 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:48.564327  4501 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:48.689052 32371 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
W20260709 12:22:48.657397  4441 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.758s	user 0.038s	sys 0.050s
I20260709 12:22:48.554827  4456 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.656558  4383 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:48.712332 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:48.820739 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.686928  4516 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:48.823210  4025 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:48.823561  4516 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.714572 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:48.826872 31746 raft_consensus.cc:2368] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:48.822222 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.691488  4505 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:48.723883 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:48.829317 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 114896us
I20260709 12:22:48.691321  4318 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:48.731745  4499 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:48.740011 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2625) took 1585 ms. Trace:
I20260709 12:22:48.740406 31750 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:48.745349 31673 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:48.748628 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.836750 31677 rpcz_store.cc:276] 0709 12:22:47.154297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.078763 (+924466us) service_pool.cc:224] Handling call
0709 12:22:48.108596 (+ 29833us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:48.108893 (+   297us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:48.148298 (+ 39405us) log.cc:844] Serialized 14 byte log entry
0709 12:22:48.739600 (+591302us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:48.739665 (+    65us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:48.739743 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.739822 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:48.739950 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":40444,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":220205,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":80,"thread_start_us":552954,"threads_started":2,"wal-append.queue_time_us":509012}
I20260709 12:22:48.837996 31819 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:48.753716 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.738461 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.749303 31678 rpcz_store.cc:276] 0709 12:22:45.871138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:45.873757 (+  2619us) service_pool.cc:224] Handling call
0709 12:22:46.862212 (+988455us) spinlock_profiling.cc:243] Waited 29 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:22:46.862391 (+   179us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:47.366831 (+504440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:47.366893 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:47.377281 (+ 10388us) log.cc:844] Serialized 26 byte log entry
0709 12:22:48.298010 (+920729us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:48.298054 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.298130 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:48.673092 (+374962us) raft_consensus.cc:1685] finished
0709 12:22:48.673134 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:48.673440 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52752,"raft.queue_time_us":157222,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":22732,"spinlock_wait_cycles":66644096,"thread_start_us":1221834,"threads_started":2,"wal-append.queue_time_us":1028374}
I20260709 12:22:48.719192  4509 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:48.810427 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:48.554929  4417 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.647s	user 0.021s	sys 0.025s
I20260709 12:22:48.846515 31674 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:48.847002 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:48.847930  4421 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:48.836184  4383 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:48.829612 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.849525 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.849941  4356 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:48.851503 31678 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 12 index: 6. (index mismatch)
I20260709 12:22:48.852162 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2626) took 1464 ms. Trace:
I20260709 12:22:48.852463 31678 rpcz_store.cc:276] 0709 12:22:47.387574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.850754 (+1463180us) service_pool.cc:224] Handling call
0709 12:22:48.851138 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:48.851921 (+   783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.852108 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25728}
I20260709 12:22:48.853302  4429 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:48.853790 31678 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:48.854501 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1780) took 1172 ms. Trace:
I20260709 12:22:48.854848 31678 rpcz_store.cc:276] 0709 12:22:47.681958 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.853310 (+1171352us) service_pool.cc:224] Handling call
0709 12:22:48.853624 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:48.854226 (+   602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.854436 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:48.850371 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.856781  4456 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:48.857014 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.857704 31820 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:48.858736 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:48.860994 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.861660 31820 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:48.744462  4464 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.858664 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.848095 31675 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 15: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:48.853755  4304 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:48.863801 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:48.866263 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2624) took 1712 ms. Trace:
I20260709 12:22:48.866600 31827 raft_consensus.cc:2376] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16. Re-sending same reply.
I20260709 12:22:48.869931 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.868753 31670 rpcz_store.cc:276] 0709 12:22:47.153849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.985230 (+831381us) service_pool.cc:224] Handling call
0709 12:22:47.985632 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:48.045102 (+ 59470us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:48.045138 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:48.082695 (+ 37557us) log.cc:844] Serialized 26 byte log entry
0709 12:22:48.502978 (+420283us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:48.503029 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.503101 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:48.865993 (+362892us) raft_consensus.cc:1685] finished
0709 12:22:48.866023 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:48.866190 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":58914,"raft.queue_time_us":59140,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":7565,"thread_start_us":309814,"threads_started":1,"wal-append.queue_time_us":719030}
I20260709 12:22:48.866678 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.871907 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:48.851503 31673 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:22:48.872732 31670 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:48.873041 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2630) took 1311 ms. Trace:
I20260709 12:22:48.862654 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.873559 31673 rpcz_store.cc:276] 0709 12:22:47.561652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.850721 (+1289069us) service_pool.cc:224] Handling call
0709 12:22:48.851142 (+   421us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:48.872726 (+ 21584us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.872975 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28160}
I20260709 12:22:48.874598 31820 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:48.876070 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.876899 31819 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:48.878199 31678 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:22:48.878001 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.878955 31677 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:48.870859 31827 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 12: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:48.880385 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.881187 31819 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 10.
I20260709 12:22:48.889353 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:48.891243 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.882314 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.893497 31819 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:48.894901 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.896430 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.899703 31819 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 9.
W20260709 12:22:48.862578  4525 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.225s	user 0.000s	sys 0.001s
I20260709 12:22:48.898066 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2632) took 1044 ms. Trace:
W20260709 12:22:48.902860 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 398785us
I20260709 12:22:48.896831 31750 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:48.903896 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.913641 31673 rpcz_store.cc:276] 0709 12:22:47.853313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.897018 (+1043705us) service_pool.cc:224] Handling call
0709 12:22:48.897412 (+   394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:48.897642 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:48.897676 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:48.897736 (+    60us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:48.897784 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.897858 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:48.897993 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:48.905997 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:48.920462 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.925521 31827 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:48.925261 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.926553 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.928718 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.929138 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.930153 32370 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:48.863991  4521 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:48.926568 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:48.933898 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.934367 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.936479 31819 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:48.865864  4464 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:48.937960 31827 raft_consensus.cc:2368] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14.
I20260709 12:22:48.938179 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:48.940213 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.526s	user 0.008s	sys 0.036s
W20260709 12:22:48.940652 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.008s	sys 0.036s
W20260709 12:22:48.940968 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 526924us
I20260709 12:22:48.910969  4509 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:48.954852 31750 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 12:22:48.959635 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:48.964005 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.945505 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:48.965097 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.950600 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.966985 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.967905 32370 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 17: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:22:48.968029 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 135584us
W20260709 12:22:48.969697 31671 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.017s	sys 0.018s
W20260709 12:22:48.970296 31671 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.018s	sys 0.018s
I20260709 12:22:48.960763 31750 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:48.966434 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:48.972710 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.929658  4376 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.035s
I20260709 12:22:48.972733 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.973877 31670 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:48.935339  4502 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 12:22:48.939138  4015 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.032s
I20260709 12:22:48.979061 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.979442 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:48.984369 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:48.991717 32370 raft_consensus.cc:2368] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14.
I20260709 12:22:48.993013 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:48.935269  4456 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:22:48.995513 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:48.944875  4437 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 12:22:48.996201 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:48.995605 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:48.997296 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.997352 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:49.000617 32374 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:49.000782 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:49.001884 31823 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 16 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:22:49.002421 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:49.003285 32372 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 13: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:49.005038 31792 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:49.008105 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:48.995016  4528 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.281s	user 0.000s	sys 0.001s
W20260709 12:22:49.063635 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58170.2us
W20260709 12:22:49.064584 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 98221.1us
I20260709 12:22:49.075249 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:49.084390 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:49.087311 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:49.088189 31818 raft_consensus.cc:2376] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16. Re-sending same reply.
I20260709 12:22:49.089016 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:48.967794  4509 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:49.094904 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:49.092160 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:49.061885  4539 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:22:49.101189 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:49.102078 31823 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:49.089051 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:49.092494 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 116872us
I20260709 12:22:49.105376 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2623) took 1951 ms. Trace:
I20260709 12:22:49.096999 31819 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:49.105693 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:49.105165 31818 raft_consensus.cc:2376] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14. Re-sending same reply.
W20260709 12:22:49.106760 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 101807us
W20260709 12:22:49.102890 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 110004us
I20260709 12:22:49.107586 31792 leader_election.cc:400] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14.
I20260709 12:22:49.107868 31792 leader_election.cc:403] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:22:49.103392 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2610) took 2152 ms. Trace:
I20260709 12:22:49.004263  4429 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:49.105811 31671 rpcz_store.cc:276] 0709 12:22:47.153519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.987466 (+833947us) service_pool.cc:224] Handling call
0709 12:22:47.987881 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.988257 (+   376us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:22:48.016974 (+ 28717us) log.cc:844] Serialized 14 byte log entry
0709 12:22:48.975831 (+958857us) log.cc:844] Serialized 14 byte log entry
0709 12:22:49.104847 (+129016us) log.cc:844] Serialized 14 byte log entry
0709 12:22:49.105008 (+   161us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:49.105074 (+    66us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:22:49.105125 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.105192 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.105311 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":128477,"prepare.queue_time_us":30070,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":20406,"raft.queue_time_us":302806,"raft.run_cpu_time_us":49,"raft.run_wall_time_us":49,"thread_start_us":953955,"threads_started":2,"wal-append.queue_time_us":490446}
I20260709 12:22:49.111404 31818 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:22:49.111603 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:49.112291 31671 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:49.108071 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:49.108716 31676 rpcz_store.cc:276] 0709 12:22:46.951163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.089305 (+138142us) service_pool.cc:224] Handling call
0709 12:22:47.089585 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:47.477017 (+387432us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:22:47.508139 (+ 31122us) log.cc:844] Serialized 14 byte log entry
0709 12:22:48.426289 (+918150us) log.cc:844] Serialized 14 byte log entry
0709 12:22:48.426466 (+   177us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:48.426512 (+    46us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:48.426563 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.103129 (+676566us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b54002544d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:49.103177 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.103330 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1236,"prepare.queue_time_us":37345,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":3048,"raft.queue_time_us":309609,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":7624,"spinlock_wait_cycles":890126080,"thread_start_us":917470,"threads_started":2,"wal-append.queue_time_us":791341}
I20260709 12:22:49.109249 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:49.132016 31823 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:22:49.132375 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2618) took 1980 ms. Trace:
I20260709 12:22:49.132920 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:49.133145 31672 rpcz_store.cc:276] 0709 12:22:47.152082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.380655 (+228573us) service_pool.cc:224] Handling call
0709 12:22:47.429660 (+ 49005us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:48.446571 (+1016911us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.132108 (+685537us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002e6b54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:49.132305 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2336604416}
I20260709 12:22:49.133538 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:49.097451  4525 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:22:49.086925  4410 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:22:49.101361  4534 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:22:49.091020  4517 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:49.138166 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:49.100687  4361 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:49.149863 31823 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:22:49.090260  4541 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:49.160812 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.110802  4499 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:49.131685  4437 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:49.135521  4533 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:49.112375  4357 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:22:49.184902  4322 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:49.185534  4322 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:49.141027  4500 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:49.135342  4318 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:49.179844 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1781) took 1497 ms. Trace:
I20260709 12:22:49.219123 31678 rpcz_store.cc:276] 0709 12:22:47.682238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.855574 (+1173336us) service_pool.cc:224] Handling call
0709 12:22:48.877959 (+ 22385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:48.878770 (+   811us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.179596 (+300826us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b54001ff9d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:49.179776 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50589952}
I20260709 12:22:49.104175  4464 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.070s
I20260709 12:22:49.181684  4437 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.196568 31820 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:49.148203  4540 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:22:49.172518  4324 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.514s	user 0.010s	sys 0.003s
I20260709 12:22:49.206499 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:49.208185 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:49.211042 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:49.213265 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:49.215373 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.216500 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:49.277593 31752 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:49.261309 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:49.279695 31747 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:49.220791 31678 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:22:49.282928 31678 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:22:49.145030  4501 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:49.163729  4541 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.135742  4521 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14.
I20260709 12:22:49.231590 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:49.185098  4544 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.218s	user 0.001s	sys 0.000s
W20260709 12:22:49.185065  4547 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
W20260709 12:22:49.233744 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 93074.2us
W20260709 12:22:49.263351  4485 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
W20260709 12:22:49.235848 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 141055us
I20260709 12:22:49.174964  4546 raft_consensus.cc:2764] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election decision vote started in defunct term 12: won
I20260709 12:22:49.174714  4499 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:49.259752 31820 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:49.277572 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:49.287564  4525 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:22:49.198293  4304 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:49.300464 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:49.234763 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 99765.2us
I20260709 12:22:49.258870  4437 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:49.285447  4541 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:49.293211  4540 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:49.356426  4540 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:49.286728  4501 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:49.340602 31824 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:22:49.348417 31820 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:49.215047  4394 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:49.362092 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61749.3us
W20260709 12:22:49.365067 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 227959us
I20260709 12:22:49.367555 31818 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 9 FOLLOWER]: Advancing to term 10
I20260709 12:22:49.367868 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2644) took 1110 ms. Trace:
I20260709 12:22:49.370672 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:49.371583 31747 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:49.370282 31670 rpcz_store.cc:276] 0709 12:22:48.257302 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.000516 (+743214us) service_pool.cc:224] Handling call
0709 12:22:49.000824 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.001050 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:22:49.149069 (+148019us) log.cc:844] Serialized 14 byte log entry
0709 12:22:49.367488 (+218419us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:49.367547 (+    59us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:22:49.367605 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.367672 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.367807 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":61462,"prepare.run_cpu_time_us":54,"prepare.run_wall_time_us":4578,"raft.queue_time_us":30647,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":16366,"thread_start_us":365718,"threads_started":2}
I20260709 12:22:49.363540 31820 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:49.374406 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 241553us
I20260709 12:22:49.375356 31820 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:49.376071 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:49.385653 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:49.374433 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:49.392364 31670 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:49.393579 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:49.393653 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:49.395392 31678 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
W20260709 12:22:49.408958 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.384433  4546 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:22:49.409934 31824 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:22:49.244560  4318 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:49.416023 31670 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:49.362262  4501 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:49.294863  4324 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:49.482522 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.493153 31752 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
W20260709 12:22:49.496613 32376 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.573s	user 0.009s	sys 0.001s
W20260709 12:22:49.417176  4547 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:22:49.402585  4533 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:49.511108  4557 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:49.417429  4425 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.099s
W20260709 12:22:49.668810 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 283173us
I20260709 12:22:49.669864 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:49.546474  4564 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 12:22:49.405529  4517 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.111s
I20260709 12:22:49.549355 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2633) took 1451 ms. Trace:
I20260709 12:22:49.672835 31675 rpcz_store.cc:276] 0709 12:22:48.098335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.900974 (+802639us) service_pool.cc:224] Handling call
0709 12:22:48.912055 (+ 11081us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:48.912644 (+   589us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:48.912691 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:48.927022 (+ 14331us) log.cc:844] Serialized 52 byte log entry
0709 12:22:48.998648 (+ 71626us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:48.998708 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.005972 (+  7264us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:49.549026 (+543054us) raft_consensus.cc:1685] finished
0709 12:22:49.549055 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.549271 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10603,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":9489,"thread_start_us":37410,"threads_started":1,"wal-append.queue_time_us":425980}
I20260709 12:22:49.392064  4361 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.120s
I20260709 12:22:49.392271  4456 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.120s
I20260709 12:22:49.436414  4421 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:49.500401  4356 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:49.384675  4394 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:49.384790  4499 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:49.538122  4558 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:22:49.631783 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 238260us
I20260709 12:22:49.615985  4546 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:22:49.615626  4509 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:49.653280 32376 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:49.654223 31818 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260709 12:22:49.655082 31820 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 12 index: 6. (term mismatch)
I20260709 12:22:49.701184 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2529) took 1037 ms. Trace:
I20260709 12:22:49.701527 31820 pending_rounds.cc:85] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 5
I20260709 12:22:49.701922 31820 pending_rounds.cc:107] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 11.6
I20260709 12:22:49.701582 31818 rpcz_store.cc:276] 0709 12:22:48.663871 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.351794 (+687923us) service_pool.cc:224] Handling call
0709 12:22:49.367255 (+ 15461us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.700981 (+333726us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.701135 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:49.702401 31820 raft_consensus.cc:2889] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:49.703044 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:49.703189 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2489) took 1201 ms. Trace:
W20260709 12:22:49.674786  4513 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.035s	user 0.006s	sys 0.015s
I20260709 12:22:49.703581 31820 rpcz_store.cc:276] 0709 12:22:48.501727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.259104 (+757377us) service_pool.cc:224] Handling call
0709 12:22:49.259560 (+   456us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.702966 (+443406us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.703140 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:22:49.657562 31747 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:22:49.616140  4560 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:22:49.675117 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 162968us
I20260709 12:22:49.671507 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:49.677150 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 283624us
I20260709 12:22:49.706187 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:49.675384 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 286851us
I20260709 12:22:49.658326  4533 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:49.696511 31643 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:49.711562 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:49.711982 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:49.712934 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:49.713250 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:49.704568 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:49.714007 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:49.714926 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:49.716318 31820 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:49.717538 31820 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:22:49.716455 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:49.722548 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2641) took 1516 ms. Trace:
I20260709 12:22:49.723047 31677 rpcz_store.cc:276] 0709 12:22:48.206265 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.941913 (+735648us) service_pool.cc:224] Handling call
0709 12:22:48.942390 (+   477us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:48.942990 (+   600us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:48.943039 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:48.943828 (+   789us) log.cc:844] Serialized 52 byte log entry
0709 12:22:49.006889 (+ 63061us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:49.006932 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.007000 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:49.722319 (+715319us) raft_consensus.cc:1685] finished
0709 12:22:49.722346 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.722486 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":318,"raft.queue_time_us":1781,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":13137,"thread_start_us":19298,"threads_started":1,"wal-append.queue_time_us":612677}
I20260709 12:22:49.679023  4410 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
I20260709 12:22:49.707533  4560 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:49.715770  4513 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:49.719940  4499 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:49.691664  4421 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:49.718989  4533 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:49.761703 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:49.741379  4570 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:22:49.768167 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96770.7us
I20260709 12:22:49.762347 32377 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:49.770967 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2537) took 1049 ms. Trace:
I20260709 12:22:49.751451  4546 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:49.776893 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:49.747372  4566 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:22:49.782435 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 76331.9us
I20260709 12:22:49.783699 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:49.784240 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 71067.4us
I20260709 12:22:49.785125 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 768) took 1796 ms. Trace:
I20260709 12:22:49.785653 32371 rpcz_store.cc:276] 0709 12:22:47.988845 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:47.989153 (+   308us) service_pool.cc:224] Handling call
0709 12:22:49.784897 (+1795744us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:49.785047 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":657211136}
I20260709 12:22:49.756906  4509 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:22:49.792641 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 76254.7us
I20260709 12:22:49.776772 31824 rpcz_store.cc:276] 0709 12:22:48.725649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.544906 (+819257us) service_pool.cc:224] Handling call
0709 12:22:49.613703 (+ 68797us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.613972 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:49.632532 (+ 18560us) log.cc:844] Serialized 14 byte log entry
0709 12:22:49.769945 (+137413us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:49.770032 (+    87us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:49.770098 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.770194 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.770884 (+   690us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":19761,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":45,"thread_start_us":136821,"threads_started":2}
I20260709 12:22:49.749315  4560 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:49.751214  4324 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:49.748019  4571 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:49.793465 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:49.793980 31643 leader_election.cc:400] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14.
I20260709 12:22:49.798269 31643 leader_election.cc:403] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:22:49.801837 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:49.804629 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:49.776767  4570 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:49.789875  4566 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.778800  4318 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:49.809019 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:49.811087 31672 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:49.797142  4513 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:49.797127  4361 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:49.797695  4546 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:49.821749  4394 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:49.821931  4581 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:49.824620  4580 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:49.828136 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 756) took 1532 ms. Trace:
I20260709 12:22:49.831252 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:49.831853  4358 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:49.839901 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:49.844733 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:49.846879 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:49.856572 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54828.9us
I20260709 12:22:49.868820 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:49.801543  4558 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:49.808287  4571 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14.
I20260709 12:22:49.860699 31676 rpcz_store.cc:276] 0709 12:22:48.295965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.117430 (+821465us) service_pool.cc:224] Handling call
0709 12:22:49.117686 (+   256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.190180 (+ 72494us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.190214 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:49.190259 (+    45us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:49.190293 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.827896 (+637603us) spinlock_profiling.cc:243] Waited 72.3 ms on lock 0x7b54001b2bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:49.827942 (+    46us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.828073 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":166218624}
I20260709 12:22:49.862107 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:49.817152  4540 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:49.871693 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:49.812649  4456 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:49.813930  4570 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:49.874151 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:49.812289  4566 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:49.815436  4509 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.070s
I20260709 12:22:49.877976 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:49.876026 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:49.930088 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52232us
I20260709 12:22:49.931618 31753 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:49.875385 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:49.934499 31749 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:49.936288 31751 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 15: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:49.937824 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:49.855413  4517 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:49.874059 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 92795.5us
I20260709 12:22:49.860455  4581 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.892745 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:49.873286  4410 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:22:49.886600  4574 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
W20260709 12:22:49.886803  4565 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.149s	user 0.000s	sys 0.001s
I20260709 12:22:49.884507  4540 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:49.885663  4562 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:49.944058 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:49.913396  4513 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:22:49.941315  4456 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:49.944032  4517 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:49.975412  4540 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:49.995198 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2636) took 1886 ms. Trace:
I20260709 12:22:49.995960 31673 rpcz_store.cc:276] 0709 12:22:48.108522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.916765 (+808243us) service_pool.cc:224] Handling call
0709 12:22:48.917137 (+   372us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.139098 (+221961us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.139145 (+    47us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:49.139192 (+    47us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:49.139252 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.994908 (+855656us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b5400253fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:49.994964 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.995125 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":509750528}
I20260709 12:22:49.981556  4545 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:49.983093 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2482) took 1580 ms. Trace:
I20260709 12:22:49.886899  4358 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:49.960503  4581 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:49.999811  4545 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:50.000083 31821 rpcz_store.cc:276] 0709 12:22:48.402369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.190678 (+788309us) service_pool.cc:224] Handling call
0709 12:22:49.191067 (+   389us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:49.191469 (+   402us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.191497 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:49.218470 (+ 26973us) log.cc:844] Serialized 52 byte log entry
0709 12:22:49.650509 (+432039us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:22:49.704398 (+ 53889us) log.cc:844] Serialized 14 byte log entry
0709 12:22:49.770204 (+ 65806us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.770292 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:49.982817 (+212525us) raft_consensus.cc:1685] finished
0709 12:22:49.982844 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:49.983021 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9379,"prepare.queue_time_us":26749,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":92809,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":25184,"thread_start_us":407391,"threads_started":2,"wal-append.queue_time_us":410663}
I20260709 12:22:50.005824 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 761) took 1667 ms. Trace:
W20260709 12:22:50.004134  4575 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260709 12:22:50.006316 31672 rpcz_store.cc:276] 0709 12:22:48.338170 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.163562 (+825392us) service_pool.cc:224] Handling call
0709 12:22:49.203414 (+ 39852us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.811628 (+608214us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.005558 (+193930us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b54002b0f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:50.005752 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1397025536}
I20260709 12:22:49.985312  4517 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:49.972681  4578 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:49.986296  4518 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:22:50.009274  4573 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
I20260709 12:22:50.019570  4576 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:50.020061  4545 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.021997  4324 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.024119  4383 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:50.033915  4383 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.038170  4383 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:50.030759 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:50.039831 31821 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:50.040701  4589 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:49.971482  4558 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.041316  4589 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:50.042289  4551 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.878s	user 0.010s	sys 0.001s
I20260709 12:22:50.031370  4576 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:50.043736 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2627) took 2655 ms. Trace:
I20260709 12:22:50.043416  4576 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.044978  4576 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:50.044193 31674 rpcz_store.cc:276] 0709 12:22:47.388036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.847320 (+1459284us) service_pool.cc:224] Handling call
0709 12:22:48.847686 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:48.866998 (+ 19312us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:48.867031 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:48.867546 (+   515us) log.cc:844] Serialized 26 byte log entry
0709 12:22:48.998265 (+130719us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:48.998313 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.998381 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:50.041677 (+1043296us) spinlock_profiling.cc:243] Waited 2.55 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:50.043400 (+  1723us) raft_consensus.cc:1685] finished
0709 12:22:50.043426 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.043664 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18840,"raft.queue_time_us":19694,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":18966,"spinlock_wait_cycles":5857024,"thread_start_us":75561,"threads_started":1,"wal-append.queue_time_us":214730}
I20260709 12:22:50.034962  4545 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:50.047276  4583 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:22:49.978086  4546 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.049141 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 758) took 1737 ms. Trace:
I20260709 12:22:50.049547 31671 rpcz_store.cc:276] 0709 12:22:48.311412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.133767 (+822355us) service_pool.cc:224] Handling call
0709 12:22:49.134051 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.207366 (+ 73315us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.207414 (+    48us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:49.207473 (+    59us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:49.207518 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.048892 (+841374us) spinlock_profiling.cc:243] Waited 73.1 ms on lock 0x7b54003926d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:50.048940 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.049081 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":168016128}
I20260709 12:22:50.035416  4324 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.024695  4580 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:50.036219  4318 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:50.016068  4437 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.662s	user 0.005s	sys 0.009s
I20260709 12:22:50.052986 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:50.053637 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 90178.6us
I20260709 12:22:50.064375 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:50.065853 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:50.054143 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:50.069356 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:50.072099 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53194.6us
I20260709 12:22:50.055100 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:50.014760  4421 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:50.032006  4357 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.076265 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:50.080106 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:50.146559 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:50.080823 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:50.147760 31750 raft_consensus.cc:2376] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18. Re-sending same reply.
W20260709 12:22:50.058879  4322 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.873s	user 0.012s	sys 0.005s
I20260709 12:22:50.149307 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 724) took 1170 ms. Trace:
I20260709 12:22:50.149811 32377 rpcz_store.cc:276] 0709 12:22:48.979262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.981473 (+  2211us) service_pool.cc:224] Handling call
0709 12:22:50.149091 (+1167618us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x7b5400546354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:50.149226 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1546306944}
I20260709 12:22:50.152835 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:50.153113 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:50.153792 31747 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:50.154174  4577 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:50.154860  4577 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.156795  4500 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:50.157469  4500 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.158988 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:50.160818 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:50.163327 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:50.146463 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 73564.3us
I20260709 12:22:50.172669  4429 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:50.173206  4429 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:50.173413  4500 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:50.173651  4429 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:50.175374  4577 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:50.185467 31751 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:50.181896 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:50.192044 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:50.073513  4521 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:50.199190 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:50.201232 31747 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:50.269542 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:50.210685 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.073381  4383 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:22:50.272971 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 73921.2us
I20260709 12:22:50.146814  4557 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.286468 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:50.146528  4318 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:50.314539 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:50.072405  4437 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.178735  4322 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.202211  4578 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.177s
W20260709 12:22:50.325873  4585 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.312s	user 0.003s	sys 0.000s
I20260709 12:22:50.201555  4456 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:50.060061  4580 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.324630  4358 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:50.367240 31811 tablet_service.cc:1558] Processing DeleteTablet for tablet f6404cb9c2174326b06cbbb0d27d43f2 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:56358
I20260709 12:22:50.273614  4521 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.331089  4579 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:50.408639 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:50.415961 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:50.371500 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2476) took 1970 ms. Trace:
I20260709 12:22:50.423599 31819 rpcz_store.cc:276] 0709 12:22:48.400660 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.109528 (+708868us) service_pool.cc:224] Handling call
0709 12:22:49.149525 (+ 39997us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:49.173805 (+ 24280us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:49.173839 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:49.173890 (+    51us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:49.173932 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.371236 (+1197304us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x7b54000fcb54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:50.371290 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.371431 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":55307776}
I20260709 12:22:50.373791 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:50.426080 31675 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:50.374527 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:50.374241 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:50.382977 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:50.429919 31823 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:22:50.383299 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:50.382675 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:50.283638  4410 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.383590 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:50.361519 31746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1924) took 1945 ms (client timeout 2499 ms). Trace:
I20260709 12:22:50.459089 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:50.465495 32376 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
W20260709 12:22:50.465189 31746 rpcz_store.cc:269] 0709 12:22:48.415600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.459574 (+ 43974us) service_pool.cc:224] Handling call
0709 12:22:50.361292 (+1901718us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b54002959d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:50.361453 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":761391872}
I20260709 12:22:50.450871  4577 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:50.365183  4501 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.860s	user 0.006s	sys 0.012s
W20260709 12:22:50.382298  4536 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.656s	user 0.019s	sys 0.011s
W20260709 12:22:50.473435 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 326959us
I20260709 12:22:50.459432 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:50.177129  4357 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:50.463201  4582 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.289s	user 0.000s	sys 0.000s
I20260709 12:22:50.289880  4500 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.480509 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:50.505148 31822 raft_consensus.cc:2376] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14. Re-sending same reply.
I20260709 12:22:50.402410  4376 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:50.193625  4562 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.259s
I20260709 12:22:50.289527  4558 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:50.287664  4383 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:22:50.498449  4560 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.690s	user 0.010s	sys 0.000s
I20260709 12:22:50.450100 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:50.579936 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 129980us
I20260709 12:22:50.390700  4521 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:50.492076 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 73483.2us
I20260709 12:22:50.395709  4564 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:50.383425  4322 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:50.379690  4324 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:50.179291  4584 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:50.498908 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:50.385044  4561 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:22:50.398659  4567 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.403409  4579 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.404078  4421 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.349829  4437 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:50.381142  4589 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:50.486240 32371 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:22:50.490993  4577 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.325042  4557 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.512784 31571 catalog_manager.cc:5697] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:50.391718  4540 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.501911  4456 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:50.383394  4581 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.520645  4357 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:50.521162  4590 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 12:22:50.521684  4518 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:50.603991  4579 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:50.522352 31674 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:50.526298  4353 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:50.532815 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:50.652904 31818 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:50.654074 31749 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:50.654559 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2464) took 2284 ms. Trace:
I20260709 12:22:50.650689  4518 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:50.655267 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:50.659304  4353 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.655004 31826 rpcz_store.cc:276] 0709 12:22:48.369861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.370314 (+   453us) service_pool.cc:224] Handling call
0709 12:22:48.370604 (+   290us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:48.382941 (+ 12337us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:48.382972 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:48.403549 (+ 20577us) log.cc:844] Serialized 26 byte log entry
0709 12:22:49.857702 (+1454153us) spinlock_profiling.cc:243] Waited 2.47 ms on lock 0x7b18002fa220. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:22:49.969364 (+111662us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:49.969429 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:49.969554 (+   125us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:50.654227 (+684673us) raft_consensus.cc:1685] finished
0709 12:22:50.654249 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.654495 (+   246us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11904,"raft.queue_time_us":12424,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":70,"spinlock_wait_cycles":5686016,"thread_start_us":110699,"threads_started":1,"wal-append.queue_time_us":1670060}
I20260709 12:22:50.534329 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:50.667088 31821 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:50.668375  4577 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:50.361133  4429 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.536723  4580 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:50.537217  4500 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.667238  4353 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:50.673846  4500 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:50.374195  4566 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.369694  4517 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.545512  4576 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:50.546311 31820 raft_consensus.cc:2410] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
W20260709 12:22:50.549345  4533 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.779s	user 0.008s	sys 0.000s
I20260709 12:22:50.464781  4499 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:50.552491 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:50.569660 31575 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 236) took 54911 ms (client timeout 15000 ms). Trace:
I20260709 12:22:50.572816 31822 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:50.573611 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:50.386205  4541 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.032s	user 0.018s	sys 0.001s
I20260709 12:22:50.384644  4570 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:50.596360 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 95209.5us
I20260709 12:22:50.550407  4376 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:50.598941 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 100121us
I20260709 12:22:50.579391  4383 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:50.498140  4501 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.479450  4410 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.588899  4400 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:50.588311  4564 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:50.512144 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2470) took 2113 ms. Trace:
W20260709 12:22:50.610338 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.737511 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:50.613934 32377 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:22:50.739037 31672 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:50.616696 31673 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:50.600026  4561 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:50.604324  4567 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.332739  4318 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.628979 32372 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:50.749639 31821 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:50.796275  4410 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:50.646715 31744 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:22:50.649895 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:50.587718  4560 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.798612  4570 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:50.799036  4560 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:50.615116  4589 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:50.799760  4589 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:50.669561 31827 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:22:50.800343  4517 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:50.801543 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 771) took 1001 ms. Trace:
I20260709 12:22:50.801970 31827 rpcz_store.cc:276] 0709 12:22:49.800316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.800485 (+   169us) service_pool.cc:224] Handling call
0709 12:22:50.011327 (+210842us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.800680 (+789353us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:50.801263 (+   583us) spinlock_profiling.cc:243] Waited 658 ms on lock 0x7b54000fa854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:50.801470 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1513143680}
I20260709 12:22:50.722860  4509 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.080s
I20260709 12:22:50.804572 31827 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:22:50.690321  4576 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.607075  4421 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.807127  4576 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:50.808557  4421 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:50.808967  4567 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:50.677455 31826 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:50.810024 32376 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:22:50.679114 31824 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:50.683722 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:50.812572 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 769) took 1016 ms. Trace:
W20260709 12:22:50.812650 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 129053us
I20260709 12:22:50.812983 31824 rpcz_store.cc:276] 0709 12:22:49.796035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.796782 (+   747us) service_pool.cc:224] Handling call
0709 12:22:50.812407 (+1015625us) spinlock_profiling.cc:243] Waited 864 ms on lock 0x7b540043eed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:50.812519 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1987604736}
I20260709 12:22:50.689435 31819 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:50.815217 31819 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:50.673853  4429 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:50.737394 31825 rpcz_store.cc:276] 0709 12:22:48.398950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.406396 (+  7446us) service_pool.cc:224] Handling call
0709 12:22:48.406778 (+   382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:48.407149 (+   371us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:48.407179 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:48.407673 (+   494us) log.cc:844] Serialized 26 byte log entry
0709 12:22:48.575880 (+168207us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:22:48.575951 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:48.576063 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:50.381662 (+1805599us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:50.511751 (+130089us) raft_consensus.cc:1685] finished
0709 12:22:50.511778 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:50.512071 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":563,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":82,"spinlock_wait_cycles":40379648,"thread_start_us":166533,"threads_started":1,"wal-append.queue_time_us":315578}
I20260709 12:22:50.820084 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:50.820859 32370 raft_consensus.cc:2368] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16.
I20260709 12:22:50.800220  4589 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:50.827630 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:50.828482 31827 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:50.723148 31823 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:50.729772 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:50.830775 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:50.831451 31820 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:50.732774 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:50.832612 31822 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:22:50.732774 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:50.727701 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:50.625746 31571 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:22:50.608247  4546 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.013s	sys 0.026s
I20260709 12:22:50.757300  4318 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:50.687016 31575 rpcz_store.cc:269] 0709 12:21:55.529818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:21:55.530832 (+  1014us) service_pool.cc:224] Handling call
0709 12:22:50.441488 (+54910656us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:42.485914 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:42.762269 (+276355us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:42.762332 (+    63us) write_op.cc:435] Acquired schema lock
0709 12:22:42.762347 (+    15us) tablet.cc:662] Decoding operations
0709 12:22:42.763383 (+  1036us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:42.763487 (+   104us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:42.763514 (+    27us) tablet.cc:685] Acquiring locks for 8 operations
0709 12:22:42.764408 (+   894us) tablet.cc:701] Row locks acquired
0709 12:22:42.764462 (+    54us) write_op.cc:260] PREPARE: finished
0709 12:22:42.774817 (+ 10355us) write_op.cc:270] Start()
0709 12:22:42.774997 (+   180us) write_op.cc:276] Timestamp: P: 1783599762764581 usec, L: 0
0709 12:22:42.779295 (+  4298us) op_driver.cc:348] REPLICATION: starting
0709 12:22:42.899178 (+119883us) log.cc:844] Serialized 3258 byte log entry
0709 12:22:43.303231 (+404053us) op_driver.cc:464] REPLICATION: finished
0709 12:22:43.368504 (+ 65273us) write_op.cc:301] APPLY: starting
0709 12:22:43.368701 (+   197us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:43.425793 (+ 57092us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:43.426019 (+   226us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:43.929726 (+503707us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:43.930094 (+   368us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=16,key_file_lookups=16,delta_file_lookups=0,mrs_lookups=0
0709 12:22:43.930169 (+    75us) write_op.cc:312] APPLY: finished
0709 12:22:43.930590 (+   421us) log.cc:844] Serialized 81 byte log entry
0709 12:22:43.930894 (+   304us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:43.931076 (+   182us) write_op.cc:454] Released schema lock
0709 12:22:43.931158 (+    82us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:43.931692 (+   534us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:43.931738 (+    46us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6241540,"reported_tablets":48,"child_traces":[["op",{"apply.queue_time_us":64945,"apply.run_cpu_time_us":19405,"apply.run_wall_time_us":563152,"cfile_cache_hit":34,"cfile_cache_hit_bytes":37551,"num_ops":8,"prepare.queue_time_us":494910,"prepare.run_cpu_time_us":30009,"prepare.run_wall_time_us":693063,"raft.queue_time_us":50727,"raft.run_cpu_time_us":38059,"raft.run_wall_time_us":86564,"replication_time_us":523731,"thread_start_us":579133,"threads_started":4,"wal-append.queue_time_us":235969,"wal-append.run_cpu_time_us":31899,"wal-append.run_wall_time_us":1817288}]]}
I20260709 12:22:50.763461  4540 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:50.764369 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:50.843675 32371 raft_consensus.cc:2368] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
W20260709 12:22:50.843884 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 263269us
I20260709 12:22:50.765290 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:50.772146 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:50.772279 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:50.777228 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:50.789110  4557 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:50.790271  4566 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:50.792399  4501 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:50.855777 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.794103  4376 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:50.724522 31675 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:50.856572  4376 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:50.856189 31571 catalog_manager.cc:5697] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 7 to 10. New cstate: current_term: 10 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:50.834836 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 107292us
I20260709 12:22:50.838003 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:50.859141 31821 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:50.859350 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:50.859594  4410 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:22:50.846740 31672 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
W20260709 12:22:50.853585 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:50.860827  4410 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:50.854022  4353 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:50.861157 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:50.862679  4318 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:50.861485  4566 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:50.864953 31571 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
W20260709 12:22:50.865823 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 93777.9us
I20260709 12:22:50.866226 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:50.880697 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:50.883163 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:50.867182 31643 leader_election.cc:400] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16.
I20260709 12:22:50.884971 31643 leader_election.cc:403] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Cancelling election due to peer responding with higher term
I20260709 12:22:50.751861  4581 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:50.901186 31827 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:22:50.863405  4318 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:50.889613 31792 leader_election.cc:400] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:50.919696 31792 leader_election.cc:403] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:50.868448  4353 consensus_queue.cc:1243] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 13.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:22:50.861224  4410 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:50.898214 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 194409us
W20260709 12:22:50.840550  4546 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.789s	user 0.013s	sys 0.026s
I20260709 12:22:50.843660  4560 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:22:50.897837 31811 thread.cc:641] tablet-delete [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.053s	sys 0.068s
I20260709 12:22:50.899909  4500 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:50.900622  4557 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:22:50.900115  4580 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:50.900422  4568 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.946506 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:50.917387  4576 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:50.926800  4570 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election lost for term 15. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16.
I20260709 12:22:50.923309  4571 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:50.890722 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:51.029979 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 139354us
I20260709 12:22:51.032071 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:50.959824 31571 catalog_manager.cc:5697] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 5 to 10, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 10 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:50.931468  4566 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 13
W20260709 12:22:50.975257 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 84935us
W20260709 12:22:50.980540 31718 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.001s	sys 0.009s
I20260709 12:22:50.731628  4541 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:50.992758 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46362.8us
I20260709 12:22:50.715121  4533 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.938874  4578 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:51.001230 31820 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
W20260709 12:22:51.004040 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 69239.3us
I20260709 12:22:51.047161 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:51.005409 32369 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.006s	sys 0.010s
I20260709 12:22:50.917265  4421 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260709 12:22:50.886288  4540 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.073s
W20260709 12:22:51.016054 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:50.961738 31811 thread.cc:608] tablet-delete [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.053s	sys 0.069s
I20260709 12:22:50.957412  4560 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:50.787356  4564 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:50.961771  4517 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.059551  4567 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:51.060117  4517 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.062579  4517 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:50.957626  4581 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:51.063341  4581 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.065478  4581 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:50.963195  4505 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:51.050388 32369 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.006s	sys 0.011s
I20260709 12:22:51.026645  4425 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:50.945611  4383 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.052896  4429 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 12:22:51.082551  4429 consensus_queue.cc:1243] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 13.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:22:51.078799  4425 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.059092  4322 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:22:51.083096  4581 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:51.083261  4425 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:51.083384  4322 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:51.083709  4581 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:51.083963  4322 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:22:51.084633  4581 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.085990  4581 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:51.040606  4500 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:51.057111  4560 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:51.036159 31718 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.002s	sys 0.009s
I20260709 12:22:50.793339  4561 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.039070  4571 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.066217  4505 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.129101 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:50.948804  4242 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.076s
I20260709 12:22:51.021971  4501 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:51.021584  4546 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260709 12:22:51.103610 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:51.088757  4598 consensus_queue.cc:1175] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b95aeeaf4c51442b8092cff1de674f2b" responder_term: 15 status { last_received { term: 12 index: 8 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: [14.9-14.9]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:51.107621  4383 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:51.055554  4578 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.025784  4518 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.157490  4421 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.158293  4421 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.158993 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 709) took 1769 ms. Trace:
I20260709 12:22:51.023376  4568 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.159476 31748 rpcz_store.cc:276] 0709 12:22:49.390048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.358388 (+968340us) service_pool.cc:224] Handling call
0709 12:22:50.438609 (+ 80221us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.468533 (+ 29924us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.468587 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:50.486005 (+ 17418us) log.cc:844] Serialized 26 byte log entry
0709 12:22:51.004656 (+518651us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:51.004740 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.004864 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.158744 (+153880us) raft_consensus.cc:1685] finished
0709 12:22:51.158766 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.158915 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":189,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":20825,"thread_start_us":423653,"threads_started":1,"wal-append.queue_time_us":640574}
I20260709 12:22:51.110934 31824 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:51.069633  4577 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 12:22:51.117396 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:51.163573 31751 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 15: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:51.118260 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:51.087618  4564 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:51.110973 31562 catalog_manager.cc:5028] TS 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): tablet f6404cb9c2174326b06cbbb0d27d43f2 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:22:51.122758 31753 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:22:51.124061 31571 catalog_manager.cc:5697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 9 to 10, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 10 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:22:51.167963 31562 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57130.8us
I20260709 12:22:51.049573 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:51.129186 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 658374us
W20260709 12:22:51.170080 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 120629us
I20260709 12:22:51.068542 31678 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:22:51.059618 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:51.171176 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:51.181253 31718 connection.cc:256] RPC call timeout handler was delayed by -1.4351381319997927s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20179999999999998s for a response
W20260709 12:22:51.182145 31718 connection.cc:256] RPC call timeout handler was delayed by -1.0522952129999794s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1755s for a response
I20260709 12:22:51.183027 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:51.162384  4577 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:51.066186  4521 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.044552  4318 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.156103  4562 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.087323  4410 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.159057  4518 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:51.148000 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:51.179590 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 120404us
I20260709 12:22:51.146870  4561 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:22:51.141741  4499 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:22:51.146502  4570 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:51.198417 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:51.187479  4540 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:51.250267 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 762) took 2310 ms. Trace:
I20260709 12:22:51.184742  4509 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:51.253438 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 395270us
I20260709 12:22:51.252892 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:51.183991  4576 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:22:51.215214 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.144907  4501 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:51.146678  4464 consensus_queue.cc:1175] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 18 status { last_received { term: 17 index: 13 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:22:51.180486  4400 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:51.189921  4577 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:51.201778  4242 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.204648  4533 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:51.278411  4509 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.296128  4505 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:51.285096  4540 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:51.297420  4540 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.277815  4500 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:51.204432  4572 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:51.264617  4562 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.277302  4581 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.028s
I20260709 12:22:51.287240  4607 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:51.305835  4500 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.334053  4456 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:51.331543  4578 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:51.337033  4568 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:51.349449 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 769) took 3332 ms. Trace:
W20260709 12:22:51.345328  4610 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:22:51.321615  4577 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.300858  4242 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.280741  4357 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:51.348531  4400 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.385605  4357 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.324890  4501 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:51.350196  4467 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:51.354220  4566 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:51.183931  4546 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:22:51.352502  4421 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:51.362440  4376 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.505s	user 0.014s	sys 0.002s
I20260709 12:22:51.388295  4357 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:51.370796  4456 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:51.388613  4546 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:51.389761  4546 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.389631  4456 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.391671  4456 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
W20260709 12:22:51.362727  4579 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.692s	user 0.014s	sys 0.000s
I20260709 12:22:51.384938  4577 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:51.371349  4500 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:51.330435  4571 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:51.371989  4572 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:51.404456  4579 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:51.428411  4589 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.542s	user 0.012s	sys 0.000s
I20260709 12:22:51.429301  4589 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.440132  4589 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:51.193995 31571 catalog_manager.cc:5697] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 5 to 7, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 7 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:51.484081 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:51.484962 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:51.486060 31675 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:51.259620 31678 rpcz_store.cc:276] 0709 12:22:48.939504 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:49.396608 (+457104us) service_pool.cc:224] Handling call
0709 12:22:49.397166 (+   558us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:51.180036 (+1782870us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.249851 (+ 69815us) spinlock_profiling.cc:243] Waited 1.67 s on lock 0x7b54002e8454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:51.250164 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3843164544}
I20260709 12:22:51.484103 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:51.494114 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 228836us
I20260709 12:22:51.496838 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:51.486465 31673 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 15 because replica has last-logged OpId of term: 12 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
W20260709 12:22:51.498688 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 300182us
I20260709 12:22:51.499619 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:51.482069 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 334018us
I20260709 12:22:51.478297 31571 catalog_manager.cc:5697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 8 to 10, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 10 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:51.502691 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:51.508342 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.516214 31747 raft_consensus.cc:2368] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:51.517939 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.508428 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.520599 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.522051 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:51.510462 31751 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:22:51.523578 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:51.524688 31678 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:51.510553 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:51.526034 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:51.526389 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:51.511953 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.527832 31671 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 15: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:51.528294 31672 raft_consensus.cc:2410] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260709 12:22:51.527940 31752 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:51.529975 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:51.513000 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.534056 31674 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:51.512300 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:51.535320 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.511109 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.524708 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:51.530251 31745 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 17: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:22:51.535348 31819 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.011s	sys 0.039s
W20260709 12:22:51.543399 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:51.546567 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.009s	sys 0.009s
I20260709 12:22:51.373688 32369 rpcz_store.cc:276] 0709 12:22:48.017532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.017677 (+   145us) service_pool.cc:224] Handling call
0709 12:22:48.017995 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:48.018214 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:22:48.479616 (+461402us) log.cc:844] Serialized 14 byte log entry
0709 12:22:50.355314 (+1875698us) spinlock_profiling.cc:243] Waited 18.1 ms on lock 0x7b180049d500. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a
0709 12:22:51.067329 (+712015us) log.cc:844] Serialized 14 byte log entry
0709 12:22:51.348963 (+281634us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:51.349041 (+    78us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:51.349112 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.349226 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.349382 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":97703,"prepare.queue_time_us":754675,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":104,"raft.queue_time_us":1370,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":4055,"spinlock_wait_cycles":41626368,"thread_start_us":1544947,"threads_started":4,"wal-append.queue_time_us":440078,"wal-append.run_cpu_time_us":1569,"wal-append.run_wall_time_us":1048918}
I20260709 12:22:51.549909  4242 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:51.551164 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.551937 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:51.553957  4357 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:51.554114  4376 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.555760 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:51.384433  4425 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:51.572759 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54356us
I20260709 12:22:51.576121 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:51.583022 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.583601  4429 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:51.540571 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:51.664337  4429 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.604694  4501 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:51.605695  4567 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:51.666414  4567 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:51.606563  4425 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:51.607580 31818 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:51.610985 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.667114  4357 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.668079 31748 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:51.506579 31571 catalog_manager.cc:5697] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 6 to 10, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 10 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:51.657126  4579 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:51.403533  4521 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:51.658185  4546 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20260709 12:22:51.622397  4562 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:51.624776  4598 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:51.672456  4376 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:51.625622 31819 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.012s	sys 0.039s
I20260709 12:22:51.671037 31571 catalog_manager.cc:5697] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 8 to 10, leader changed from <none> to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 10 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:51.675691 31571 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 3 to 10. New cstate: current_term: 10 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:51.677973  4580 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260709 12:22:51.679064 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:51.678437 31571 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:51.680133 31672 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:51.680994 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2563) took 1130 ms. Trace:
I20260709 12:22:51.681504 31819 rpcz_store.cc:276] 0709 12:22:50.550412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:50.815941 (+265529us) service_pool.cc:224] Handling call
0709 12:22:50.816264 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:50.816744 (+   480us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:50.816776 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:50.817353 (+   577us) log.cc:844] Serialized 26 byte log entry
0709 12:22:51.674066 (+856713us) raft_consensus.cc:1645] Marking committed up to 5
0709 12:22:51.674145 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:51.674305 (+   160us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:51.674358 (+    53us) raft_consensus.cc:1685] finished
0709 12:22:51.674371 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:51.680922 (+  6551us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1492,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":116,"thread_start_us":856040,"threads_started":1,"wal-append.queue_time_us":567848}
I20260709 12:22:51.682994  4577 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:51.683269 31571 catalog_manager.cc:5697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 6 to 10. New cstate: current_term: 10 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:51.635589 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:51.694654  4376 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:51.694880 31824 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:51.696743  4429 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:51.383150  4541 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:51.639979 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.698999 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:51.640937 31825 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:22:51.641835  4609 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:51.701951  4609 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:51.636214  4509 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:51.645780 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 90558.3us
I20260709 12:22:51.703498  4609 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.704587 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:51.646461  4566 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:22:51.643675 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.010s	sys 0.009s
W20260709 12:22:51.707391 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 672106us
I20260709 12:22:51.707435  4566 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:51.374794  4557 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:51.650952 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:51.709857  4557 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.655803  4324 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:51.711637  4324 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:51.712213  4324 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:51.650951  4421 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:51.715147  4421 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.663105 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:51.673126  4598 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:51.719197  4598 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.720397 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:51.624475  4560 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.632953 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.684077  4577 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.723522  4560 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:51.727311  4598 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:51.654268  4410 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:51.719180  4357 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:51.733307  4421 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:51.667107  4567 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:51.736011 31674 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: [14.8-14.8]
I20260709 12:22:51.736374 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:51.737834  4318 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:22:51.657753 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49202us
I20260709 12:22:51.657274 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:51.656558  4572 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.740618 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:51.692664 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.696593  4618 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:51.738432  4318 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:51.742515  4618 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.701377  4242 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:51.656057 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 144122us
I20260709 12:22:51.744944 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:51.745704  4618 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:51.746755  4242 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:51.747300  4242 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:51.748961  4572 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:51.750974 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:51.752995 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:51.754192  4614 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:51.745858  4318 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.755330  4614 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:51.726500  4577 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:51.765293  4577 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:51.769997  4429 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:51.774753 31824 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:51.746553  4618 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:51.771324 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51202.1us
I20260709 12:22:51.786893 31672 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
W20260709 12:22:51.784174 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 43828.9us
W20260709 12:22:51.787170 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 276972us
I20260709 12:22:51.770980  4425 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:22:51.809562 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.812618 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.776983  4429 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.794836  4560 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:51.828974 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 608849us
I20260709 12:22:51.805253  4425 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:51.829604 31751 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
W20260709 12:22:51.830783 31644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.539s	user 0.011s	sys 0.006s
W20260709 12:22:51.826748 31744 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.015s	sys 0.029s
I20260709 12:22:51.826802 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:51.839039 31644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.587s	user 0.012s	sys 0.006s
I20260709 12:22:51.844342 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:51.846343 31744 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.015s	sys 0.029s
I20260709 12:22:51.856149 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.850205 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:51.863893 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:51.864763 31747 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:51.866797  4560 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.867323 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:51.868675 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:51.869262  4560 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:51.873605  4561 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.610s	user 0.010s	sys 0.005s
W20260709 12:22:51.875644 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.879297 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.891883  4429 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:51.892138 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:51.892668 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.893458 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.895081 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:51.912931 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.896466 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:51.915197 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 111461us
W20260709 12:22:51.870234 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52271.2us
I20260709 12:22:51.870124 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:51.904129 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 48061.2us
I20260709 12:22:51.905330  4634 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:51.907069 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:51.898290 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:51.925107 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:51.926944  4376 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:51.927596  4376 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:51.930341 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:51.886121  4322 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.801s	user 0.012s	sys 0.013s
I20260709 12:22:51.931236 31824 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:51.928979  4376 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.945415  4322 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.946791 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:51.947424 32374 pending_rounds.cc:85] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 9
I20260709 12:22:51.947770 32374 pending_rounds.cc:107] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 13.10
I20260709 12:22:51.947827 31819 raft_consensus.cc:2410] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:22:51.948189 32374 raft_consensus.cc:2889] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:51.899330  4425 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:51.869697  4324 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:51.968098 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:51.948083  4322 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:51.927004  4561 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:51.970680 31744 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:22:51.975917 31752 raft_consensus.cc:2410] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:22:51.982929 31744 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:51.977566 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:51.974009  4577 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:22:51.997740 32377 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:52.002693 31745 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 11: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:51.979254  4557 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:22:51.987735 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:51.993860 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:52.010025 32369 raft_consensus.cc:2410] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:22:52.009841 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:52.010152 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:52.019009  4577 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:52.027037 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:52.006012  4357 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.026700 31744 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.027860  4561 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:52.002774  4421 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
W20260709 12:22:52.031728 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 68796.9us
I20260709 12:22:52.032081  4570 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:52.034204  4500 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.035607  4579 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:52.038868 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:52.039593 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:52.040266  4629 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:52.040791  4607 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.598s	user 0.007s	sys 0.012s
I20260709 12:22:52.042984 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:52.043583  4456 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.603s	user 0.011s	sys 0.015s
I20260709 12:22:52.031734 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:52.026688 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:52.049160 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.054294 31824 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
W20260709 12:22:52.056226 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 57763.7us
I20260709 12:22:52.061374 31818 raft_consensus.cc:2410] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:22:52.049467  4557 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.059072 31819 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:52.071774 32378 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:22:52.057197  4570 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.078234 31827 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 10.
I20260709 12:22:52.083767 31826 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:52.083988 31821 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:52.085301  4607 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:52.088152 31822 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 15: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:52.090696 31821 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260709 12:22:52.092226 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.090734 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:52.094516 31820 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:52.104660 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:52.105737 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.110174 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:52.075119  4557 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:52.126731  4614 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:52.131273  4614 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:52.131819  4614 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:52.133289 31744 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:52.117381  4629 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.136433 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:52.136931  4629 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:52.127925  4564 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.961s	user 0.015s	sys 0.008s
I20260709 12:22:52.128226  4569 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 11.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:52.137842  4322 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:22:52.127950 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 45767.5us
I20260709 12:22:52.141857 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:52.142742 31818 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:22:52.098161  4570 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:52.143267 31819 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260709 12:22:52.143705 31819 pending_rounds.cc:85] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 8
I20260709 12:22:52.143977 31819 pending_rounds.cc:107] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 12:22:52.144212 31819 raft_consensus.cc:2889] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:52.149657 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.139314  4577 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.057641  4242 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:52.150902 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:52.173883 32377 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:52.158056 31673 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:22:52.159051  4508 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:52.161062 32370 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
W20260709 12:22:52.162358 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 45364.8us
I20260709 12:22:52.183723 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:52.184623 31823 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:52.149317  4456 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:52.186764 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 212221us
W20260709 12:22:52.194772 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.185010  4579 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:52.181624  4508 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:52.166005  4357 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.215507 31744 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260709 12:22:52.223311 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 123821us
I20260709 12:22:52.229720 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:52.211762  4500 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:52.236778  4567 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.502s	user 0.016s	sys 0.000s
I20260709 12:22:52.291415 31823 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:52.213475  4456 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.236411  4579 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:52.311901 31818 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:22:52.312498 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:52.227284  4564 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.247745  4324 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:52.249512  4561 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:52.305742 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:52.305780 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 63666us
I20260709 12:22:52.303087  4501 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:52.252051  4508 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:52.320932  4611 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:52.321169  4614 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:52.324817 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:52.327365 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 97777.6us
I20260709 12:22:52.340793 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:52.285108  4567 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.295701  4560 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.306483  4628 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:52.347083 31747 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:22:52.322077  4618 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.024s
I20260709 12:22:52.357405 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:52.384588 31745 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:52.328070  4607 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.029s
I20260709 12:22:52.356031  4242 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:52.410851 31823 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:22:52.338461  4501 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:52.409602 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:52.417779 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 225172us
I20260709 12:22:52.350744  4564 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.442327 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:52.446390 31818 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:52.446610 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:52.412312 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 92127us
I20260709 12:22:52.413339  4500 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:52.428146  4508 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:52.417317  4357 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:52.464862 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61073.9us
W20260709 12:22:52.476316 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59378.6us
I20260709 12:22:52.413700  4560 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:52.483323 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 126023us
I20260709 12:22:52.484413 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:52.440153  4598 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:22:52.410032  4628 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:52.468943  4540 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:52.510274  4664 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:22:52.510697  4353 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:52.511463  4667 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:52.511986  4667 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:52.511003  4664 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:52.490763  4357 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:52.512388  4667 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:52.570384 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 86105.1us
I20260709 12:22:52.571406 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:52.524942  4664 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:52.506510  4634 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:52.512408  4353 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.575286 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:52.577346  4655 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.089s	user 0.001s	sys 0.000s
I20260709 12:22:52.577780 31746 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 13. Current term is 14. Ops: [13.12-13.12]
I20260709 12:22:52.578727  4653 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.579457 31751 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
W20260709 12:22:52.580299  4644 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
I20260709 12:22:52.466400  4579 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:52.581598  4649 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 12:22:52.582209  4567 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:52.582777  4577 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.661011  4577 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.664237  4577 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:52.583611 31747 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
W20260709 12:22:52.587189  4647 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.194s	user 0.001s	sys 0.000s
I20260709 12:22:52.587399  4614 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:52.666554 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:52.588124  4541 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:52.667876  4541 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:52.668282 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:52.669322 31747 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:52.669116  4541 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.589483 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:52.615451 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44191.2us
I20260709 12:22:52.629674  4560 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:52.590776  4357 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:52.631055  4546 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:52.631496  4564 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:52.632512  4501 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:52.633396  4353 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.595647  4456 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:52.617149  4425 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.693522  4425 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:52.635757 31744 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:52.670902  4653 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:52.578768  4628 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:52.639042  4634 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:52.684883 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.619810  4540 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:52.649446  4400 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:52.635290 31751 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:52.650839  4324 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:52.651858  4618 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:52.715690 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:52.717068 31747 raft_consensus.cc:2410] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 15 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:22:52.680971 31818 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:52.686877 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:52.692605  4541 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:52.650002 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:52.702617 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:52.700367  4540 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:52.708570 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:52.719664 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:52.728238 31752 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:52.720396 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:52.638605  4557 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.507s	user 0.009s	sys 0.001s
I20260709 12:22:52.732739 31744 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
W20260709 12:22:52.740985 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.686210  4546 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:52.696888  4665 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:52.696630  4425 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:52.733731  4655 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:22:52.694552  4353 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:52.766883 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 80134.7us
W20260709 12:22:52.774930 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.733927  4540 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:52.704427  4560 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.063s
I20260709 12:22:52.776232 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:52.781978 31675 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
W20260709 12:22:52.783665 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.696733  4607 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.079s
I20260709 12:22:52.765195  4546 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:22:52.788614 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.777144  4400 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:22:52.780838  4628 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:52.778015  4653 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:52.793351 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:52.746759  4557 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:52.700137  4634 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.805459 31818 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:22:52.806915 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:22:52.808030 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 158189us
I20260709 12:22:52.809054 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:52.811105  4324 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:52.790035  4500 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:52.830789 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:52.831293  4674 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:52.843040 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:52.843952 32375 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:52.845618 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:52.848309 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:52.850015  4569 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.850876 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:52.850679  4569 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:52.854259  4569 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:52.854988  4569 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.855593  4569 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:52.857823  4646 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260709 12:22:52.858836  4569 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:52.859342  4357 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:52.805181  4564 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:52.885779  4564 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:22:52.860982  4570 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.717s	user 0.014s	sys 0.000s
I20260709 12:22:52.887152  4570 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.888208  4564 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:22:52.889364  4570 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:52.862162 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:52.792090  4665 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:52.863240 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48133.4us
I20260709 12:22:52.819836  4322 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:22:52.864641 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 731) took 1012 ms. Trace:
I20260709 12:22:52.865355  4557 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:52.866669 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:52.866899  4500 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:52.890606  4666 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:52.917946  4666 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:52.920583  4666 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:52.920637 31818 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:22:52.921165 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:52.921424  4672 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.922081 31827 raft_consensus.cc:2410] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:22:52.896341 31824 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:52.923228 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:52.872958 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1811) took 1344 ms. Trace:
I20260709 12:22:52.878261  4674 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:52.901764 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:52.924158 31673 rpcz_store.cc:276] 0709 12:22:51.522382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.525517 (+  3135us) service_pool.cc:224] Handling call
0709 12:22:52.867002 (+1341485us) spinlock_profiling.cc:243] Waited 551 ms on lock 0x7b54001ff4d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:52.867105 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1267487104}
I20260709 12:22:52.884159 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:52.921913 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:52.906840 32369 rpcz_store.cc:276] 0709 12:22:51.890851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.894541 (+  3690us) service_pool.cc:224] Handling call
0709 12:22:52.864434 (+969893us) spinlock_profiling.cc:243] Waited 99.3 ms on lock 0x7b5400427f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:52.864558 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":228377472}
I20260709 12:22:52.909551 31819 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:22:52.868379 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 1.02762e+06us
I20260709 12:22:52.922724 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:52.922019 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:52.931128 32376 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
W20260709 12:22:52.827004 32342 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.668s (SENT) [suppressed 22 similar messages]
I20260709 12:22:52.942395 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:52.903227  4322 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:52.946955 31674 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:22:52.945338 31827 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:52.947949 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:52.923286  4577 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.947410 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:52.948544 31821 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:52.955184 31826 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:52.960632 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:52.922068  4672 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:52.992455 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 90781.2us
I20260709 12:22:52.953146  4564 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:52.993567 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2547) took 1150 ms. Trace:
I20260709 12:22:52.953043  4607 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:52.986841  4665 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:52.995751  4680 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.129s	user 0.000s	sys 0.002s
I20260709 12:22:53.020690  4664 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.979285  4674 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:52.978590  4322 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.001602 31752 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:52.907370  4618 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.059s
I20260709 12:22:53.015564 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.018225  4672 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:53.008955 31750 rpcz_store.cc:276] 0709 12:22:51.842877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:51.914716 (+ 71839us) service_pool.cc:224] Handling call
0709 12:22:51.957999 (+ 43283us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:51.999550 (+ 41551us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:51.999585 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:52.020964 (+ 21379us) log.cc:844] Serialized 26 byte log entry
0709 12:22:52.343111 (+322147us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:52.343163 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:52.343252 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:52.857070 (+513818us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:52.993212 (+136142us) raft_consensus.cc:1685] finished
0709 12:22:52.993233 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:52.993472 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1501,"raft.queue_time_us":9138,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":51744,"spinlock_wait_cycles":27247488,"thread_start_us":321668,"threads_started":1,"wal-append.queue_time_us":671564}
I20260709 12:22:53.018971  4667 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.019471  4607 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:22:53.011639 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.052506 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:53.041841 31673 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 15 because replica has last-logged OpId of term: 13 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 11.
I20260709 12:22:53.049852  4682 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:53.052109 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 98923.3us
I20260709 12:22:53.059630 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:53.026021  4572 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.068259 31819 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:22:53.072105 31751 raft_consensus.cc:2410] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 16 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:22:53.075412 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:53.033169  4598 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:53.088335 31750 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:53.088794 31745 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:53.093648 31747 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:53.061987  4607 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:53.026820  4577 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.097515 31745 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:53.102051 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:53.105584 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:53.107172 31669 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:53.041055  4540 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.051561  4665 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:22:53.111379 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:53.118242 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 291340us
I20260709 12:22:53.119717 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:53.124063 31747 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:53.066576  4564 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.101398  4683 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:53.149278 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:53.081580  4572 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.153113  4572 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:53.116943  4664 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:53.155300 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.156176 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:53.161000 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:53.163527 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.162513 31791 leader_election.cc:400] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:53.164995 31791 leader_election.cc:403] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:53.165125 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:53.162232 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:53.162086 31750 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:22:53.110992  4629 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.145768 31745 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.167239 31673 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:53.096240  4682 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.179140 31678 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:53.183992  4577 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:53.185498 31819 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:53.191232 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:53.172791  4683 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.192873 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:53.193958 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:53.195235 31678 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:53.196517 31750 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:53.197875 31678 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: []
I20260709 12:22:53.199117 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:53.199319 32369 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:22:53.200532 31678 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:53.202162  4681 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:22:53.202606  4681 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:53.203630  4681 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.204645  4664 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:53.205185  4681 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:53.206707  4564 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:53.206845 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 84855.3us
I20260709 12:22:53.207342 31674 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:53.208927 31674 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
W20260709 12:22:53.169751  4679 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:22:53.222234  4629 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:53.204054 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 113879us
I20260709 12:22:53.200636 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:53.231092 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.239064 31745 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:53.239840 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 70742.6us
W20260709 12:22:53.240757 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.242048 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:53.258611 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.254865 31751 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:53.254066 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:53.266819 31750 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 12 index: 11. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:22:53.251176  4500 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.283864 31675 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:53.286697 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:53.291042 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.288839 31678 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:22:53.286731 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:53.300691 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.291965 31674 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:53.283995 31748 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:53.251716  4618 raft_consensus.cc:2764] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election decision vote started in defunct term 13: won
W20260709 12:22:53.302927 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 57046.5us
I20260709 12:22:53.314280 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.304384 31748 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:53.306090 31753 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:22:53.320160 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.251402  4577 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:53.315367 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:53.323704 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.303261  4628 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:53.287956  4564 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:53.324762 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:53.317569  4546 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.524s	user 0.010s	sys 0.001s
I20260709 12:22:53.337241 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.336404 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:53.343621 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:53.323777  4322 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:22:53.346590 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 242970us
I20260709 12:22:53.348299 31669 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:22:53.350772 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:53.290021  4567 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.045s
I20260709 12:22:53.360946 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:53.353225 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:53.362743 31819 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:53.359035 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:53.367192 31671 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:53.329718  4541 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:22:53.369005 31747 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:53.372643 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:53.373849 31819 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:53.374719  4570 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.375204 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:53.375523  4570 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.360669  4400 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.016s
I20260709 12:22:53.379540  4669 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:53.380223  4669 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:53.381264  4324 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:22:53.381789  4324 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:53.382902  4324 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:53.385426  4615 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.052s	user 0.002s	sys 0.001s
I20260709 12:22:53.338953  4357 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:53.386379  4357 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.323529  4611 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.037s
I20260709 12:22:53.386826  4357 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:53.387598  4611 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:53.388029  4611 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:53.389019  4611 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.311578  4609 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.032s
I20260709 12:22:53.374953  4559 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:22:53.375468  4685 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.091s	user 0.000s	sys 0.004s
I20260709 12:22:53.399811 31745 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:22:53.390393  4611 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.303956  4500 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:53.404045  4638 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.091s	user 0.002s	sys 0.001s
W20260709 12:22:53.404287  4658 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.590s	user 0.006s	sys 0.011s
I20260709 12:22:53.404677  4662 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:53.404858  4437 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20260709 12:22:53.405683 31677 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:53.405390  4652 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.138s
W20260709 12:22:53.406162  4684 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.132s	user 0.004s	sys 0.000s
I20260709 12:22:53.366302  4564 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.409888  4607 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.359840  4322 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:53.437517 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:53.370486  4541 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:22:53.410952 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.397511  4376 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:53.400839 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:53.415007 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.378322  4570 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:53.373167  4667 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:53.404170  4690 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.064s	user 0.002s	sys 0.001s
I20260709 12:22:53.354568  4634 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260709 12:22:53.469400  4634 consensus_queue.cc:1243] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 15.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 7
I20260709 12:22:53.470060  4634 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:53.471168  4624 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:22:53.401470  4559 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.471869  4624 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:53.472267  4559 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:53.472385  4692 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.155s	user 0.002s	sys 0.000s
I20260709 12:22:53.472999  4624 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.342123  4546 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.476656  4546 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:53.470400  4634 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:53.478892  4634 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.480188  4607 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:53.464212  4500 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:53.381287  4669 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 11.11, Last appended by leader: 11, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.467679 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:53.489341  4683 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:53.489950  4652 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:53.492202 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:53.494089 31673 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:53.495602 31745 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:53.496124 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:53.501255 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:53.501830 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:53.497558  4652 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:53.501300 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:53.505756 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55837.2us
I20260709 12:22:53.505250 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:53.442147  4322 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:53.444046  4541 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:53.442368  4376 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:53.512142 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 185611us
I20260709 12:22:53.514297 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:53.515404 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:53.515507 31819 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:53.497694 31791 leader_election.cc:400] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:22:53.480261  4634 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.516129 31746 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:22:53.498890  4456 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:22:53.498749  4628 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:53.540434 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:53.516985  4376 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:53.543195 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 210907us
I20260709 12:22:53.523713  4652 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:53.512727  4353 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.027s
I20260709 12:22:53.548501 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.556319 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:53.539999  4562 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:53.558830 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.543279  4500 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.549793  4665 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:53.561560  4456 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:53.545073  4667 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
I20260709 12:22:53.571424 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:53.557369  4668 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.018s
I20260709 12:22:53.555455  4628 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.567507  4683 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:53.573879  4674 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:22:53.585933 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 142463us
I20260709 12:22:53.560933  4652 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.587769 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.563279  4357 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.593385 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:53.575016  4456 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:53.589922 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.564924  4500 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.596315 31671 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:22:53.596519  4410 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:22:53.599792  4559 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.618568  4410 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:53.619115  4410 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:53.620297  4559 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:53.606755 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:53.608995  4500 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:53.606810  4357 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:53.612550  4628 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:53.584868  4598 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 12:22:53.614032  4682 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:53.617566  4687 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.216s	user 0.000s	sys 0.003s
W20260709 12:22:53.624290 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62989us
W20260709 12:22:53.637084 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78358.4us
I20260709 12:22:53.637215 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:53.623380 31674 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:22:53.565405  4562 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.585343  4669 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:53.589020  4674 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:53.637806 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.621793  4456 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.643273 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:53.647239  4682 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:53.647643  4682 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:53.648074  4682 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.628299  4425 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:53.648010 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.648726  4425 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:53.649106  4546 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:53.649813  4546 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:53.650229  4546 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.649201  4425 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:53.657306  4682 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.657557  4456 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.660143 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:53.636417  4572 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:22:53.662278 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:53.638862  4500 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:53.668133 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:53.684898 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.668365  4598 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:53.686925  4598 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:53.687377  4598 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:53.686345  4322 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.641559  4564 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:53.688336 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:53.689045 31675 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:53.689040 31823 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:53.643185  4509 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:53.675911  4652 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:53.669237 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:53.703660 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:53.704742 31819 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:53.645592  4674 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.688186  4562 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:53.633024  4541 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.703305 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:53.715662 31821 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:22:53.715983 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46810.1us
I20260709 12:22:53.719376 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:53.643153  4669 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:53.693969 31670 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:53.709092 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:53.712100 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.714556 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:53.727010 31819 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:53.645045  4357 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.062s
I20260709 12:22:53.728322  4541 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:53.730384 31753 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:53.730903 31753 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:53.731920 31753 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.733132 31753 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:22:53.734078 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:53.736099  4456 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:53.737306 31821 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:22:53.739252 31746 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:22:53.740533 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:53.741398 32371 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:53.743188  4711 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:53.744115  4711 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:53.709729  4705 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.134s	user 0.000s	sys 0.001s
I20260709 12:22:53.724655 31825 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:22:53.709599  4509 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:53.717185 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:53.696995  4564 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:22:53.689846  4628 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:53.667889  4710 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 12:22:53.751945 31826 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:22:53.709252  4666 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:53.756588 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:53.756798 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 94614.9us
I20260709 12:22:53.753491  4541 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:53.749485  4669 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.765561 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:53.763801 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:53.771142 31825 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:22:53.776124  4667 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:53.754390  4425 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.785054  4324 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:53.786273  4425 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:53.770403  4666 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:53.791218 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:53.791659  4540 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:53.796689 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:53.803493 31670 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 14: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:53.806212 31823 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:53.809140 31750 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.810272  4662 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:53.810806  4662 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:53.818495  4674 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.821254  4242 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:53.821988  4242 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.822407  4674 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:53.826004 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:53.827665  4322 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:53.831547  4682 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:53.832643 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 123756us
I20260709 12:22:53.838200 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:53.841157 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:53.844324 31750 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:53.849452 31669 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:53.764905  4546 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:53.850119 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:53.850622 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.853837 31669 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:53.854852 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.855899 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.808175  4711 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:53.859112 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.860337 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:53.861013 31672 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 15: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:53.858374 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.865928 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.869370  4500 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:53.870000  4500 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.873564 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.874220 31670 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:53.871119 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:53.875551  4500 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:53.875735 31821 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.871802 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:53.879719 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:53.880539 31746 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:53.880287  4376 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.882007 32372 raft_consensus.cc:2368] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15.
I20260709 12:22:53.855218  4560 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:22:53.873301 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:53.883876  4546 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:53.884636 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:53.885317 31826 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.886933 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:53.888123 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:53.888759 31825 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.891151 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.888149 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:53.893257 31824 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.898047 31826 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:53.898917 31821 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:53.899016 31678 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:22:53.899945 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:53.900898 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.901072 31821 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.901073 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.901711 31669 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:53.903123 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:53.903074 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:53.904925 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:53.898700  4322 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:53.908187 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.908239  4674 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260709 12:22:53.909502  4669 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.907544 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.915331 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:53.914322 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:53.919354 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:53.922422 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.924279 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:53.925028 31827 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:53.923000 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.927809 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:53.928010 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:53.928660 32375 raft_consensus.cc:2368] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15.
I20260709 12:22:53.930614 31672 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:53.921849  4662 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.937434  4662 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:53.908213  4322 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:53.938117 31746 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:53.938493  4322 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:53.938902 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.939967 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:53.940102 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:53.940543 31826 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:53.943399  4410 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:53.938149  4669 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:53.946036  4410 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:53.908070  4540 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 12:22:53.925886  4577 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:53.948582  4577 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.905819  4711 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:22:53.955972 31825 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:53.953454 31824 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:53.956079 32378 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:53.952780 31669 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:53.959568 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.951205 31826 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:53.925318  4682 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:53.925060  4651 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:53.963655  4711 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:53.969312  4711 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.924911  4666 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.857378  4509 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.966409 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:53.916641  4570 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 12:22:53.957929  4546 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:54.000747 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:54.006824 31821 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
W20260709 12:22:54.008081 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.004388  4665 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:54.006013  4683 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:53.992934  4682 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:54.012418  4682 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:53.999133  4651 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:54.017228  4651 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:54.020294 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:53.993997  4540 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:54.023377 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:54.025439 31753 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 14 index: 9. (term mismatch)
I20260709 12:22:54.025840 31753 pending_rounds.cc:85] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 8
I20260709 12:22:54.026085 31753 pending_rounds.cc:107] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:22:54.026326 31753 raft_consensus.cc:2889] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:53.962687  4598 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:54.028860  4376 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:54.029328  4665 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:54.029845  4683 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:54.022519  4546 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:54.050920 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:54.037602 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:54.050945 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:54.051507 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:54.052341 31821 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.052937 31675 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:54.053910 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47864.8us
I20260709 12:22:54.054214 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:54.052242 31669 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:54.054860 31820 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:54.034533 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:54.054873 31678 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.031330  4577 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:54.061102 31674 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 17: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:54.035647  4540 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:54.068110 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:54.079914 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:54.081225 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:54.090077 32376 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:22:54.078312 31748 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:54.092033 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:54.095438 31753 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:54.068517  4353 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:22:54.097070 32375 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:54.103374 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:54.083411  4666 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:54.104411 31748 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.104904 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:54.107941 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:54.111475 31821 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:54.115286 31819 raft_consensus.cc:2368] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15.
I20260709 12:22:54.116735 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:54.096393  4711 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.118479 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:54.119166 31753 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:54.113144 31678 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:54.078213  4570 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 12:22:54.121196 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:54.101809  4353 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:54.125267 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:54.133025 31821 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:54.133175 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:54.134474  4353 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.136255 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:54.159627 31823 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:54.159927 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:54.138136 31819 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:54.167501 31753 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:54.173028 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:54.175930 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:54.177238 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:54.178200  4682 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:54.178668  4682 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.179018  4682 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.179772 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:54.180593  4664 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:54.180974 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:54.181376 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:54.181929 31746 raft_consensus.cc:2410] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 15 because replica has last-logged OpId of term: 14 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20260709 12:22:54.182192  4664 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:54.182773 31643 leader_election.cc:400] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15.
I20260709 12:22:54.183112 31643 leader_election.cc:403] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:22:54.183589 31750 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:22:54.183974 31750 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:54.184697 31750 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.185684 31750 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:22:54.138559 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0711303080001926s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16000000000000003s for a response
W20260709 12:22:54.187651 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0684303080001882s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1603s for a response
W20260709 12:22:54.187999 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0360303080001358s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16390000000000002s for a response
W20260709 12:22:54.188280 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0099303080000936s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1668s for a response
I20260709 12:22:54.190704 31746 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:22:54.192370 31819 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:22:54.192971 31746 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:54.120831  4666 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:54.113945  4577 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 12:22:54.193907 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:54.199280 31819 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:22:54.200714 31819 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:22:54.165517  4353 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:54.186267  4664 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:22:54.207898 31819 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:54.132766  4546 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.204356 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:54.259310  4666 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:54.262791  4717 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.223s	user 0.001s	sys 0.001s
I20260709 12:22:54.268237 31823 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260709 12:22:54.137145  4683 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:54.133443  4711 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.114355  4665 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.250670 31750 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:22:54.201858  4628 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:22:54.275127  4598 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:54.165701  4540 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.305434 31675 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
W20260709 12:22:54.322219 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 128419us
I20260709 12:22:54.240289  4728 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.105s
I20260709 12:22:54.305546  4682 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:54.328135  4629 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:54.333988 31827 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: [12.8-14.9]
W20260709 12:22:54.334746 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.335059 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:54.337802 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:54.339046  4652 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:22:54.340189  4725 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.324s	user 0.000s	sys 0.003s
I20260709 12:22:54.341029  4652 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:54.341507  4652 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:54.341858  4652 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.343609 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2559) took 1308 ms. Trace:
W20260709 12:22:54.345319 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 800680us
I20260709 12:22:54.312072  4674 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15.
W20260709 12:22:54.357945 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 157314us
I20260709 12:22:54.348210 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:54.375660 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:54.376701 31821 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 16 because replica has last-logged OpId of term: 13 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 7.
I20260709 12:22:54.344648 32342 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:54.320897  4721 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:54.377727  4421 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.014s
I20260709 12:22:54.392071 31827 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: [14.7-14.7]
I20260709 12:22:54.393263 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:54.393245 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:54.395100 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:54.344007 31749 rpcz_store.cc:276] 0709 12:22:53.034836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.035028 (+   192us) service_pool.cc:224] Handling call
0709 12:22:53.050823 (+ 15795us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.051164 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:22:53.338660 (+287496us) log.cc:844] Serialized 14 byte log entry
0709 12:22:53.530372 (+191712us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.343100 (+812728us) spinlock_profiling.cc:243] Waited 1.52 ms on lock 0x7b1800498160. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:22:54.343250 (+   150us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:54.343316 (+    66us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:54.343363 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.343430 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.343555 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":829862,"prepare.queue_time_us":270446,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":4083,"raft.queue_time_us":832200,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":7129,"spinlock_wait_cycles":3490176,"thread_start_us":458148,"threads_started":2,"wal-append.queue_time_us":263807}
I20260709 12:22:54.388661  4541 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:54.410725  4618 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:54.411521  4242 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:54.411794  4618 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:54.414276  4702 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:54.414757  4702 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:54.414794  4425 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:54.415488  4425 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:54.415647  4702 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.417106  4702 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.416599  4425 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 13.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:54.421118  4567 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:54.421703  4567 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:54.388446  4376 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:22:54.416276 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.357147  4628 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 12:22:54.442755 31824 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
W20260709 12:22:54.427448 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.438398 31827 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: [14.11-16.12]
W20260709 12:22:54.449436 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 48043.8us
I20260709 12:22:54.445572 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:54.454835  4703 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:54.426756  4710 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:54.465270 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:54.424255  4400 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:54.427773  4376 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:54.459795 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:54.458153  4721 raft_consensus.cc:2764] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election decision vote started in defunct term 14: won
I20260709 12:22:54.459199  4541 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.468214 31827 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:54.463930 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:54.462394 31821 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:54.461158  4634 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:22:54.492478 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 97490.4us
I20260709 12:22:54.437005  4322 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.493197 31823 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20260709 12:22:54.493628 31823 pending_rounds.cc:85] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 6
I20260709 12:22:54.493932 31823 pending_rounds.cc:107] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260709 12:22:54.493705 31748 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 9. (term mismatch)
I20260709 12:22:54.494197 31823 raft_consensus.cc:2889] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:54.494402 31748 pending_rounds.cc:85] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 8
I20260709 12:22:54.494688 31748 pending_rounds.cc:107] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 14.9
I20260709 12:22:54.495038 31748 raft_consensus.cc:2889] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:54.498076 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 729) took 1339 ms. Trace:
I20260709 12:22:54.498920 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:54.498524 31748 rpcz_store.cc:276] 0709 12:22:53.158797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.167097 (+  8300us) service_pool.cc:224] Handling call
0709 12:22:53.227988 (+ 60891us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.035873 (+807885us) spinlock_profiling.cc:243] Waited 74.9 ms on lock 0x7b54003b68a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd5306f 00007fe8fcdcbf60 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62
0709 12:22:54.497786 (+461913us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.498014 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":172169600}
W20260709 12:22:54.500943 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 153935us
W20260709 12:22:54.503943 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.422590  4567 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.511497 31827 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:54.492982  4634 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:54.496729  4614 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260709 12:22:54.516773 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52965.3us
I20260709 12:22:54.499877  4721 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.465042  4703 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:54.515066 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 207447us
I20260709 12:22:54.488397  4541 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.522862 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:54.524847 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:54.529546 31748 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:54.509263  4577 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:54.507911  4629 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:54.488298  4400 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:54.517917  4634 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:54.566758 31823 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
W20260709 12:22:54.570878 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53260.6us
I20260709 12:22:54.572432 31718 leader_election.cc:400] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15.
I20260709 12:22:54.572891 31718 leader_election.cc:403] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:22:54.524456  4611 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:54.564077 31717 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:54.531136  4567 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.585359 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:54.577947 31823 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:54.516083  4607 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 12:22:54.596866  4322 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:54.598037  4400 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:54.600857  4703 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:54.601313  4560 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:54.567875  4728 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:22:54.524457  4376 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:54.604620 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 88890.2us
I20260709 12:22:54.526808  4721 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:54.566445  4421 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:22:54.592504  4704 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.632877 31823 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:54.641393 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:54.641839 32372 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:54.616386  4560 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:54.615633  4739 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:54.592502  4629 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.639282  4400 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 6, Time since last communication: 0.015s
I20260709 12:22:54.656759  4728 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:54.657270  4728 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:54.657693  4728 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:54.665441 31821 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:22:54.617522  4652 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:54.671934 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47300.7us
I20260709 12:22:54.677205 31821 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
W20260709 12:22:54.674913 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 45526.4us
I20260709 12:22:54.676100  4541 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:54.672240  4421 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:54.687397 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:54.680656 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:54.690105 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:54.691823 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:54.692665 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.681249 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:54.682787 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:54.683092 31821 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:22:54.684525 31674 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: []
I20260709 12:22:54.676754  4400 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:22:54.694880 31823 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:54.698848 31821 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:22:54.702119 31678 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:22:54.696473 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:54.694078  4421 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:54.710949 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:54.712453 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:54.713845 31821 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:22:54.714614 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:54.715557 31750 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:54.717666 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:54.719322 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:54.719638 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:54.720098 31745 raft_consensus.cc:2376] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16. Re-sending same reply.
I20260709 12:22:54.697005  4400 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:54.729822 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:54.730763 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:54.732939 31749 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
W20260709 12:22:54.743464  4744 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.149s	user 0.001s	sys 0.000s
I20260709 12:22:54.745863 31750 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:54.746330 31822 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260709 12:22:54.747829 31745 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:22:54.749889 31745 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:54.751631 31670 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:22:54.754606  4666 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.755239  4666 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:54.756716 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.757206  4400 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.758260  4704 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15.
I20260709 12:22:54.711541  4421 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:54.721444  4733 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:22:54.759522  4666 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:54.760098  4651 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:22:54.712173  4624 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 12:22:54.777971 31745 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:54.758760  4400 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:54.779368 31745 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:22:54.779733 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 69696.2us
I20260709 12:22:54.709687  4702 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 13.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.071s
I20260709 12:22:54.788450 31823 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:54.796988 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:54.800439 31748 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:54.802558 31822 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:54.803115 31822 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:54.777990  4651 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:54.792366  4662 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:22:54.801959  4607 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:54.800606  4652 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:54.779299  4624 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
W20260709 12:22:54.853793 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 142464us
I20260709 12:22:54.820468 31643 leader_election.cc:400] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15.
I20260709 12:22:54.881929 31643 leader_election.cc:403] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:22:54.825244 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.837433 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:54.901459 31745 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:22:54.901950 31745 pending_rounds.cc:85] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 11
I20260709 12:22:54.902251 31745 pending_rounds.cc:107] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:22:54.902540 31745 raft_consensus.cc:2889] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:54.904481 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 84191.4us
I20260709 12:22:54.808418 31822 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.907634 31822 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:22:54.901149 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 227051us
W20260709 12:22:54.836000  4741 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.236s	user 0.002s	sys 0.000s
W20260709 12:22:54.836150  4737 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.365s	user 0.003s	sys 0.001s
W20260709 12:22:54.828949  4735 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.359s	user 0.005s	sys 0.004s
W20260709 12:22:54.909224  4727 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:22:54.815711  4357 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:54.810236  4651 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:22:54.811812  4662 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:54.818507  4560 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:54.931125 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:54.948702  4758 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:22:54.950920  4757 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:54.957073 31826 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:22:54.957279  4560 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:54.959554  4421 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.846329  4634 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.960795  4728 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.950532  4662 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.962649  4634 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:54.963102 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2588) took 1123 ms. Trace:
I20260709 12:22:54.963477  4662 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:54.964982  4421 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:54.963696 31751 rpcz_store.cc:276] 0709 12:22:53.840096 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.840292 (+   196us) service_pool.cc:224] Handling call
0709 12:22:53.852724 (+ 12432us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:54.418802 (+566078us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:54.418845 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:54.419326 (+   481us) log.cc:844] Serialized 26 byte log entry
0709 12:22:54.509326 (+ 90000us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:22:54.509372 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.509445 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:54.834063 (+324618us) spinlock_profiling.cc:243] Waited 3.56 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:54.962711 (+128648us) raft_consensus.cc:1685] finished
0709 12:22:54.962742 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.963015 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":565699,"raft.queue_time_us":566121,"raft.run_cpu_time_us":74,"raft.run_wall_time_us":76,"spinlock_wait_cycles":8180480,"thread_start_us":89427,"threads_started":1,"wal-append.queue_time_us":161497}
W20260709 12:22:54.967018 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:54.814359  4607 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:54.973418  4607 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:54.974184  4607 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.909317  4712 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:54.912173  4683 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15.
I20260709 12:22:54.927538  4704 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:54.989753 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 913) took 1099 ms. Trace:
I20260709 12:22:54.931483  4357 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:22:55.011317 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 425992us
W20260709 12:22:55.011827 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.012742 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:54.990538 32369 rpcz_store.cc:276] 0709 12:22:53.890617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.891577 (+   960us) service_pool.cc:224] Handling call
0709 12:22:53.916084 (+ 24507us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:53.916376 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:22:53.957317 (+ 40941us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.810700 (+853383us) spinlock_profiling.cc:243] Waited 3.09 ms on lock 0x7b1800420fa0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:22:54.989008 (+178308us) log.cc:844] Serialized 14 byte log entry
0709 12:22:54.989235 (+   227us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:54.989284 (+    49us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:54.989335 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:54.989424 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:54.989648 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":39491,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":94,"raft.queue_time_us":179,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":155,"spinlock_wait_cycles":7099264,"thread_start_us":177501,"threads_started":1,"wal-append.queue_time_us":1028302}
W20260709 12:22:55.037773  4763 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 12:22:55.050673 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:55.058424 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:55.060871 31822 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 13 index: 8. (term mismatch)
I20260709 12:22:55.062186 31819 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:22:54.991925  4704 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.062675 31822 pending_rounds.cc:85] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 7
I20260709 12:22:55.069813 31822 pending_rounds.cc:107] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 12.8
I20260709 12:22:55.043040  4421 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:22:55.070747 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:55.068241 31823 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:22:55.080029 32372 raft_consensus.cc:2368] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15.
I20260709 12:22:55.070706 31822 raft_consensus.cc:2889] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:55.071197 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:55.081564 31825 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
W20260709 12:22:55.073647 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 236304us
I20260709 12:22:55.076442  4766 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:54.983695  4712 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.085707 32377 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:22:55.086931 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:55.089051 31822 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:55.036855  4607 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.097949 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:55.101006  4770 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:55.099912 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:55.103157  4760 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.031s
I20260709 12:22:55.103811 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:55.107897 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:55.045548  4634 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 12:22:55.112797 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:55.082485  4577 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:55.076898  4772 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:55.085342  4766 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:55.086699  4421 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:55.125672 32369 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: [14.9-14.9]
I20260709 12:22:55.130951 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:55.135151 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:55.135898 31819 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:22:55.109875  4770 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:55.136413  4772 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.152974 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:55.123356  4766 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:55.123868  4768 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260709 12:22:55.129558  4400 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:55.163890 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:55.164368 32369 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:22:55.163139 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:55.146464  4728 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:55.143651  4651 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.107591  4357 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.126863  4754 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.122s
I20260709 12:22:55.175887  4564 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.010s
I20260709 12:22:55.186822  4768 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:22:55.187402  4768 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:55.188380  4768 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.162925  4762 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:22:55.191614  4772 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:55.192389  4772 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:55.192800  4772 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:55.193620  4772 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.218742 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:55.220722 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:55.221123  4762 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:55.221990  4762 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.222538  4762 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:55.223946  4753 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.484s	user 0.000s	sys 0.010s
I20260709 12:22:55.225167  4762 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:55.199334  4712 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:55.163170  4634 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:55.195088  4704 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:55.199865  4756 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.215s	user 0.000s	sys 0.001s
I20260709 12:22:55.198978  4421 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.220386 31718 leader_election.cc:400] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15.
I20260709 12:22:55.231033 31718 leader_election.cc:403] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:55.205960  4607 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:55.207567 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:55.207918 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:55.240140 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 102616us
I20260709 12:22:55.209921 32369 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: []
I20260709 12:22:55.211594  4357 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:55.211920 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2607) took 1505 ms. Trace:
I20260709 12:22:55.211594 32375 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
W20260709 12:22:55.214742  4750 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.474s	user 0.011s	sys 0.000s
I20260709 12:22:55.218577 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:55.252429 32342 leader_election.cc:336] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.668s (SENT)
I20260709 12:22:55.173506  4400 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.202181  4651 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:55.234784 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:55.245541 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:55.257936 31822 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:55.180243  4770 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:55.243649 31818 rpcz_store.cc:276] 0709 12:22:53.706500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.706757 (+   257us) service_pool.cc:224] Handling call
0709 12:22:54.666512 (+959755us) spinlock_profiling.cc:243] Waited 26.2 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:22:54.673389 (+  6877us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:54.673758 (+   369us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:54.841476 (+167718us) spinlock_profiling.cc:243] Waited 9.17 ms on lock 0x7b180020f2c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:22:54.841949 (+   473us) log.cc:844] Serialized 14 byte log entry
0709 12:22:55.115259 (+273310us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:55.115955 (+   696us) log.cc:844] Serialized 26 byte log entry
0709 12:22:55.116190 (+   235us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:55.116597 (+   407us) log.cc:844] Serialized 14 byte log entry
0709 12:22:55.118864 (+  2267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.118975 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:55.211645 (+ 92670us) raft_consensus.cc:1685] finished
0709 12:22:55.211667 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:55.211825 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":116707,"prepare.queue_time_us":168793,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":88,"raft.queue_time_us":82670,"raft.run_cpu_time_us":539,"raft.run_wall_time_us":27267,"spinlock_wait_cycles":81416832,"thread_start_us":260469,"threads_started":2,"wal-append.queue_time_us":235325}
I20260709 12:22:55.260388 31822 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:22:55.269460 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:22:55.255400 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 110792us
I20260709 12:22:55.290735 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:55.250910  4704 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:55.302567 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:55.302863 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:55.305662 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:55.312625 31822 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
W20260709 12:22:55.318899 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.300858  4634 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.016s
I20260709 12:22:55.324184 31818 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:22:55.317320  4509 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.327212  4509 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.327286  4770 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.329609  4509 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:55.320540  4540 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.006s
W20260709 12:22:55.325279  4598 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.007s	sys 0.016s
I20260709 12:22:55.323081  4560 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.016s
I20260709 12:22:55.365149  4400 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:55.367412 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:55.369158  4760 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:55.369726  4760 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:55.351421 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 206078us
I20260709 12:22:55.371810 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 830) took 1432 ms. Trace:
I20260709 12:22:55.372470 31820 rpcz_store.cc:276] 0709 12:22:53.939478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.971817 (+ 32339us) service_pool.cc:224] Handling call
0709 12:22:53.972594 (+   777us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.058424 (+ 85830us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.371497 (+1313073us) spinlock_profiling.cc:243] Waited 82.1 ms on lock 0x7b54000fb754. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:55.371746 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":188721536}
I20260709 12:22:55.380487 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2532) took 1121 ms. Trace:
I20260709 12:22:55.381038 31826 rpcz_store.cc:276] 0709 12:22:54.273465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.423057 (+149592us) service_pool.cc:224] Handling call
0709 12:22:54.956880 (+533823us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.957636 (+   756us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.380216 (+422580us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b54004466d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:55.380400 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1226797440}
I20260709 12:22:55.382766 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2527) took 1243 ms. Trace:
I20260709 12:22:55.383260 31824 rpcz_store.cc:276] 0709 12:22:54.139258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.323680 (+184422us) service_pool.cc:224] Handling call
0709 12:22:54.442544 (+118864us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:54.443249 (+   705us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:55.382483 (+939234us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54000fad50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:55.382684 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":272298368}
I20260709 12:22:55.397521 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:55.406217 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:55.406936 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.426045 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:55.403654  4598 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.008s	sys 0.016s
W20260709 12:22:55.431149  4774 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260709 12:22:55.432829  4500 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:55.438289  4771 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:55.470047 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:55.476516 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:55.419909  4760 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:55.445245  4779 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.037s
I20260709 12:22:55.467077  4771 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.484431 31824 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:55.488233 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:55.489594 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:55.482606  4782 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:22:55.494357 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:55.465059  4766 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.499675  4577 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:55.500238  4577 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.500720  4577 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:55.504058  4779 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:55.504971  4779 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:55.505340  4779 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.505671  4779 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:55.508426  4766 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:55.466959  4500 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.509702  4760 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:55.512086  4500 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:55.513549 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:55.486799  4728 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:22:55.515100  4711 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:55.491647  4778 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:55.487488  4771 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:55.528659 31824 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:55.527333  4778 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.531365  4728 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:55.571236  4787 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:22:55.583410  4728 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.586807  4324 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.587481  4324 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.589562  4704 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15.
I20260709 12:22:55.589892  4324 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:55.555153  4711 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:55.591255  4711 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.595757  4711 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:55.560286  4778 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:55.603906 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:55.604735 31824 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:55.627918  4790 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:55.642976  4790 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.645768  4778 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.646060  4790 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:55.658787 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.679214 31717 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:55.643795  4771 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.690168  4784 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.690949  4784 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.692335  4771 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:55.639889  4739 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.930s	user 0.002s	sys 0.007s
I20260709 12:22:55.696377 31674 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:22:55.699764  4779 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.700347  4778 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:55.700555 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:55.700888  4797 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:55.702481  4760 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:22:55.731604 31673 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:55.703102  4567 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:55.732761  4567 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:55.733808  4567 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.735164  4567 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:55.735247  4721 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:55.703948 31670 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 16. Current term is 17. Ops: []
I20260709 12:22:55.711041  4784 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:55.711187  4357 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:55.739552 31673 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:55.740116  4357 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.743237  4357 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:55.658440  4577 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:55.715063 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:55.715531  4746 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.992s	user 0.002s	sys 0.000s
I20260709 12:22:55.716266 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:55.760648 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44480.7us
I20260709 12:22:55.717043 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:55.764142 31753 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
W20260709 12:22:55.718029 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54546.7us
I20260709 12:22:55.766266 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:55.768218 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:55.768546 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:55.769945 31674 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:55.762286 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:55.699028  4739 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.990s	user 0.002s	sys 0.007s
I20260709 12:22:55.720947 31671 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 13 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:22:55.722481  4749 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.981s	user 0.000s	sys 0.003s
I20260709 12:22:55.725314  4772 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:55.713127 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:55.747138 31673 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:55.755270  4577 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:55.756578  4751 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:55.777518 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:55.781750 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:55.782810 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1974) took 1739 ms. Trace:
I20260709 12:22:55.790367  4798 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:22:55.791427  4779 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:55.792461  4795 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:22:55.825703 31753 rpcz_store.cc:276] 0709 12:22:54.025272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.192409 (+167137us) service_pool.cc:224] Handling call
0709 12:22:55.716900 (+1524491us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:22:55.764695 (+ 47795us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2873600}
I20260709 12:22:55.823752 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:55.795857 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:55.797102 31678 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:55.798023 31677 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
W20260709 12:22:55.800333 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:55.839633 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.824697 31672 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:22:55.795111 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 484068us
I20260709 12:22:55.824764 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:55.797133  4624 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:55.846899  4624 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.843564 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:55.843770 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:55.845192 31676 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:22:55.844247 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.778074  4760 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:22:55.811576  4664 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.336s	user 0.011s	sys 0.010s
W20260709 12:22:55.840917 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 347248us
I20260709 12:22:55.828739  4704 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:22:55.800832  4739 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:55.855608 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:55.798810  4797 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.056s
I20260709 12:22:55.858994  4357 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:22:55.859517  4357 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.859876  4357 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:55.827397  4751 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:55.864148  4611 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:55.864292  4751 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.872680  4682 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:55.873461 31820 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: [12.8-14.9]
W20260709 12:22:55.873921  4800 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.080s	user 0.000s	sys 0.001s
W20260709 12:22:55.849535  4746 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:22:55.879025 31826 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:22:55.879890  4357 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:55.883994 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.864960  4611 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:55.887919  4775 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:22:55.854841  4664 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.379s	user 0.011s	sys 0.010s
W20260709 12:22:55.891944 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 105072us
I20260709 12:22:55.853776  4570 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:22:55.898792 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:55.853278  4624 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:55.910817 32372 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: []
I20260709 12:22:55.859014  4799 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:55.889533  4704 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:55.891913  4683 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:55.919307 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:55.914418 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:55.924177 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:55.925628 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:55.927595 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:55.929016 31678 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:55.929677 31669 raft_consensus.cc:2368] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17.
I20260709 12:22:55.938926 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:55.941372 31669 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:55.930193  4674 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.963384  4674 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.922811  4797 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:55.970068 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:55.922508  4793 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:55.975194 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:55.975690  4801 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:55.978758  4776 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:55.981173 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:55.982141  4786 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:55.982839  4324 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:22:55.983284  4806 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:55.983886  4806 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:55.984774  4785 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:55.984853 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:55.985627  4785 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:55.986191  4785 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:55.986882 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:55.987833 31673 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:55.988894 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:55.965013  4739 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.990777  4806 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:55.990914 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:55.991669 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 736) took 1293 ms. Trace:
I20260709 12:22:55.992085 31748 rpcz_store.cc:276] 0709 12:22:54.699179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:54.699348 (+   169us) service_pool.cc:224] Handling call
0709 12:22:55.991505 (+1292157us) spinlock_profiling.cc:243] Waited 44.2 ms on lock 0x7b54004033d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:55.991608 (+   103us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":101563264}
I20260709 12:22:55.993736  4704 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:55.932955  4570 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:55.940236  4760 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:55.923017  4784 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.050s
W20260709 12:22:55.997180  4703 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.262s	user 0.008s	sys 0.011s
I20260709 12:22:55.925000  4682 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.052s
I20260709 12:22:56.000052 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:55.988873 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:56.142903 31673 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:55.929405  4683 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:56.059301 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:56.072512 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:56.160538 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 160549us
I20260709 12:22:55.929217  4611 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.161944 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:56.206290 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44395.7us
I20260709 12:22:56.207831 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:56.209244 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:56.231344  4577 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:56.232442  4577 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:56.233302  4808 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.234050  4808 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:56.236723  4808 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:56.240317  4577 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:22:55.976608  4797 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.240610 31791 leader_election.cc:400] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17.
I20260709 12:22:56.255474 31791 leader_election.cc:403] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:56.241140  4779 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:22:56.095510  4703 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.365s	user 0.009s	sys 0.012s
I20260709 12:22:56.243018  4674 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:55.973708  4793 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:56.247300 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 79085.2us
I20260709 12:22:56.269829 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:56.290156 31669 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
W20260709 12:22:56.288367 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 388746us
W20260709 12:22:56.299057 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.203660  4801 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:56.333788 31745 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:22:56.205309  4664 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.338460 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:56.002208  4760 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:56.328109 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:56.332610 31673 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:22:56.339167 31745 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:56.339898 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:56.340548 31673 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:56.324635 31750 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
I20260709 12:22:56.343561 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:56.344350 31747 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 11. Preceding OpId from leader: term: 14 index: 12. (index mismatch)
I20260709 12:22:56.344493 31751 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:22:56.254770  4357 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:56.355418 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:56.324149  4776 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:56.258781  4652 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.092s
I20260709 12:22:56.349536  4400 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:56.353825  4798 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:56.369570  4400 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:56.372004  4400 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:56.370319  4798 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:56.377583  4798 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:56.373044  4805 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:56.343873  4324 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.379024  4324 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.379555  4798 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:22:56.380187  4798 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:56.339118  4376 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:22:56.382005  4798 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:56.388764 31751 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:22:56.343453  4806 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.396521 31753 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:56.396770 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:22:56.398250 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 157807us
I20260709 12:22:56.381826  4324 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:56.372706  4400 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.021s
I20260709 12:22:56.364319  4802 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 12:22:56.408340  4793 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:56.409245  4793 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.409746  4683 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:56.397678  4376 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:56.409858  4793 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:56.411347  4376 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:56.413530  4793 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:56.413753  4376 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:56.349462  4664 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:56.416857 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 459055us
W20260709 12:22:56.418794 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.433699  4400 consensus_queue.cc:1243] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 15.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:22:56.363850  4567 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 12:22:56.398972  4801 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:56.424341 31673 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:22:56.344210  4570 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.448593 31669 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:56.451306 31673 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: []
I20260709 12:22:56.428514  4376 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.455557  4376 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:56.458244  4376 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:56.459369 31670 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:56.460500  4567 consensus_queue.cc:1243] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 15.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:22:56.384728  4799 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:22:56.400863  4806 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.468961 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:56.475684 31669 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: [13.8-15.9]
I20260709 12:22:56.481747 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:56.487072 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:56.495194 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:56.495368 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:56.498276 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:56.498414 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:56.507416 31670 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
W20260709 12:22:56.485116 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 133182us
I20260709 12:22:56.516579 31827 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:22:56.537873 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:56.540099 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:56.536391  4353 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.544878  4353 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:56.547171  4812 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:56.550971 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:56.489419  4739 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.561055 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:56.566448 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 201624us
I20260709 12:22:56.451177  4801 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:56.551371  4772 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.714s	user 0.009s	sys 0.003s
I20260709 12:22:56.565794 31827 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:56.584657 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:56.596222 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:56.567909  4812 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:56.586042 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:56.535023  4803 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.087s
I20260709 12:22:56.604820 31824 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:22:56.572436  4757 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:22:56.608163 31820 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:22:56.612156  4629 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:56.605741  4624 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.008s
I20260709 12:22:56.614670 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:56.567090  4739 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.582196  4760 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:56.618009 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 246849us
I20260709 12:22:56.622996 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:56.624279  4739 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:56.596455  4748 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.636739 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:56.601722  4797 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:56.616672  4703 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:56.639722  4703 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.641922  4703 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:56.602443  4812 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:56.643180  4812 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.645030  4812 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:56.646927  4353 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:56.630651  4567 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:56.648916 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:56.649791 31675 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:56.603646  4802 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:56.655885 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:56.624807  4776 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17.
I20260709 12:22:56.659917  4802 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:56.660310  4748 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.661015  4802 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:56.599772  4376 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.056s
I20260709 12:22:56.679795  4748 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:56.668228  4664 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:56.683920 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 98069us
I20260709 12:22:56.690062  4353 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:22:56.682476 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 106787us
I20260709 12:22:56.668038  4821 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 12:22:56.693632  4806 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:56.709199  4669 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:22:56.707928  4421 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:22:56.726532 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.736395 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:56.737782 32369 raft_consensus.cc:2410] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:22:56.725065  4669 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:56.726825  4421 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:56.729172  4772 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.748867 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:56.735447  4798 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:56.756287 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:56.759235  4664 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:56.760988 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:56.761978 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:56.761790  4664 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:56.763294 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:56.764286  4421 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:56.765646  4421 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:22:56.767740  4828 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:22:56.756644  4772 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.769006 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:56.771600  4772 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:56.771589  4664 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:56.772455  4664 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:56.772977  4664 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:56.775971 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:56.776950 31821 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 15: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:56.780601 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:56.781818  4785 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.795s	user 0.012s	sys 0.000s
I20260709 12:22:56.761994 31822 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:56.806953  4785 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.807339 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:56.807577 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:56.808157 31823 raft_consensus.cc:2368] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15.
I20260709 12:22:56.808463 31822 raft_consensus.cc:2410] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
W20260709 12:22:56.808633 31823 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2547) took 1695 ms (client timeout 1727 ms). Trace:
I20260709 12:22:56.809319 31673 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:56.763146  4801 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:56.811084 31674 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:22:56.778209 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:56.806001 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 748) took 1721 ms. Trace:
I20260709 12:22:56.812036 32375 rpcz_store.cc:276] 0709 12:22:55.084374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.084517 (+   143us) service_pool.cc:224] Handling call
0709 12:22:55.093363 (+  8846us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:55.243807 (+150444us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:56.805715 (+1561908us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b54000127d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:56.805932 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":271425280}
I20260709 12:22:56.814105 31819 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:22:56.821379  4324 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:56.821897  4324 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:22:56.824815 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 619759us
I20260709 12:22:56.830919  4785 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:56.834738  4751 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:56.840799 31826 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:56.842193 31825 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
W20260709 12:22:56.843825 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:56.845016 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.850471  4819 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:22:56.809052 31823 rpcz_store.cc:269] 0709 12:22:55.112775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:55.113305 (+   530us) service_pool.cc:224] Handling call
0709 12:22:56.807084 (+1693779us) spinlock_profiling.cc:243] Waited 8.68 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:22:56.808535 (+  1451us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":19974016}
I20260709 12:22:56.854744 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:56.856105  4546 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:56.856351 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:56.857573 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:56.856935  4546 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.858332  4801 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:56.846868  4775 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:56.847641  4324 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.848758  4752 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:22:56.858462 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:56.870152  4577 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:22:56.884872  4357 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:56.861639  4564 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:22:56.893540  4564 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:56.893262  4357 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:56.894091  4564 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:56.896274  4357 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:56.876446  4611 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:56.897250  4400 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:56.878543  4546 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:56.898802  4775 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:56.860276  4669 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:56.903216  4669 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:56.912423  4752 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:56.916952  4752 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:56.918097  4752 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.919957  4752 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:56.920527 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67252.1us
W20260709 12:22:56.920799 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 200890us
I20260709 12:22:56.921494  4703 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:56.916432  4324 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:56.933763 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 75369.1us
I20260709 12:22:56.928751 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:56.935314 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:56.947873  4776 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:22:56.951653 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:56.955953 31670 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:56.957278 31676 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:56.957139 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:56.958562 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:56.951877 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 182954us
W20260709 12:22:56.959415 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:56.959802 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:56.962880  4757 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:22:56.963490 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:56.964182 31678 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:56.948307  4801 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:22:56.965309  4801 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:56.967525 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:56.971405  4776 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:56.971211  4733 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:56.972400  4703 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:22:56.943328  4577 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:56.966298  4801 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:56.969116 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:56.974231 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45598us
I20260709 12:22:56.975189 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:56.974799 31676 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:56.972250 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:56.979678 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:56.990474 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:56.943439  4772 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:56.987197 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:56.972298  4733 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:22:57.010085 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:56.943445  4400 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.048s
W20260709 12:22:57.009262 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49580.1us
W20260709 12:22:57.010910 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 373580us
I20260709 12:22:56.985414  4703 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:57.028669 31753 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:22:57.028946 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:57.029565 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:56.998056  4806 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:57.036625  4806 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:57.038996  4806 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:56.998931  4634 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:57.030933 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:57.041036 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:57.042070 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:57.042744 31745 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:57.043325 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:57.043505 31753 raft_consensus.cc:2376] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15. Re-sending same reply.
I20260709 12:22:57.044754 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:57.007563  4817 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:57.007524  4546 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:57.007404  4357 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.052310 31750 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:57.014881  4733 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:57.020517  4611 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:57.055776 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:57.073348 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:57.056702 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:57.016394  4818 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.021696  4567 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:57.057957 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:57.083034 31746 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:57.019588  4779 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.064180 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:57.054353 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:57.088213 31753 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:57.089171 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:56.999451  4757 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.047s
I20260709 12:22:57.055578 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:57.076468 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:57.080400 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:56.972463  4776 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:57.049769  4834 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.008565  4772 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.083968 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:57.060710 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50733.7us
W20260709 12:22:57.091179 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 452648us
I20260709 12:22:57.110459 31745 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:57.114027 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58555.9us
I20260709 12:22:57.048846  4509 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.049s
I20260709 12:22:57.067056  4400 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:57.071240  4806 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:57.067622  4802 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:57.067823  4799 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:57.101392 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:57.148131 31747 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:57.118175 31750 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:57.101468 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:57.023283  4577 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:57.149758 31753 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:22:57.158996 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:57.069140  4353 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.052s
I20260709 12:22:57.163952 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:57.072273  4357 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.176450 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:57.178357 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:57.179056 31753 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:57.179502 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:57.179947 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:57.145097  4712 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:57.138449  4400 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:57.138707  4806 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:22:57.192976 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 71826.6us
I20260709 12:22:57.193768 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:57.194662 31745 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:57.097991  4567 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:57.099860  4779 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:57.194708 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:57.089576  4818 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.114990  4834 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:57.169341  4421 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.066s
I20260709 12:22:57.218394 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:57.198930 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:57.253659 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:57.310247 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 91929.8us
I20260709 12:22:57.196835 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:57.334951 31745 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
W20260709 12:22:57.206815 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 84302.9us
I20260709 12:22:57.339283 31744 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:22:57.340483 31747 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:22:57.347460 31819 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
W20260709 12:22:57.343071 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 148459us
I20260709 12:22:57.364012 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:57.168010  4799 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.364147 31745 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:22:57.190935  4703 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:22:57.185509  4611 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:57.308923  4400 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:57.385830 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:57.198210  4634 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.308568  4806 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:57.310160  4623 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:57.217837  4718 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:22:57.407042 31745 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
W20260709 12:22:57.393674 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.254406  4803 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:22:57.390358 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 412815us
I20260709 12:22:57.160077  4802 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.338678  4761 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:57.390447 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55981.4us
I20260709 12:22:57.364042  4509 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.403590 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:57.368520  4353 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:57.265650  4832 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.184s
I20260709 12:22:57.172101  4733 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:57.216202  4748 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.184s
I20260709 12:22:57.379487  4817 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:57.456840 31746 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:22:57.429131 31745 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260709 12:22:57.503170 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50399.9us
I20260709 12:22:57.422250 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:57.510317 31819 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:57.510991 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:57.400877  4623 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:57.415144  4772 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:57.418275  4818 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:57.416841  4834 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:57.466874  4718 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:57.539108  4623 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:57.499290  4849 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:22:57.501248  4739 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.391505  4611 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:57.391508  4736 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:22:57.423321  4751 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.588s	user 0.011s	sys 0.001s
W20260709 12:22:57.415753  4775 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.516s	user 0.010s	sys 0.002s
W20260709 12:22:57.431380  4842 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.150s	user 0.000s	sys 0.000s
W20260709 12:22:57.533634 31571 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42070 (request call id 223) took 47777 ms (client timeout 15000 ms). Trace:
W20260709 12:22:57.534058 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 43704us
W20260709 12:22:57.572721  4843 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.166s	user 0.000s	sys 0.004s
I20260709 12:22:57.433022  4357 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:57.490137  4353 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:57.403940  4634 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:57.427284 31574 catalog_manager.cc:5697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 10 to 16, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 16 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:22:57.512173  4812 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.867s	user 0.012s	sys 0.000s
I20260709 12:22:57.534222  4845 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:57.565856 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:57.543192  4733 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:57.555856  4665 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:57.508181  4761 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.507128  4509 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:57.541421  4772 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:57.561254  4818 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:57.560503  4834 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:57.535177 31746 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 14 index: 7. (term mismatch)
I20260709 12:22:57.656430 31746 pending_rounds.cc:85] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 6
I20260709 12:22:57.656773 31746 pending_rounds.cc:107] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 13.7
I20260709 12:22:57.657084 31746 raft_consensus.cc:2889] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:22:57.584185 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:57.659539 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 75454.5us
I20260709 12:22:57.414390  4567 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:57.414433  4799 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:57.669087 31746 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
W20260709 12:22:57.599071 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 400211us
I20260709 12:22:57.669675 32378 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:22:57.670178 31718 leader_election.cc:400] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15.
I20260709 12:22:57.670615 31718 leader_election.cc:403] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:22:57.599336 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:57.602012 31677 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.000s	sys 0.006s
W20260709 12:22:57.672179 31677 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.001s	sys 0.006s
W20260709 12:22:57.604259 31821 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.005s	sys 0.001s
W20260709 12:22:57.673189 31821 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.005s	sys 0.002s
W20260709 12:22:57.604642 31820 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.004s	sys 0.002s
W20260709 12:22:57.674849 31820 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.004s	sys 0.002s
I20260709 12:22:57.574162  4611 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:57.588284  4849 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:22:57.640800 31753 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:57.644475 31747 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:22:57.600157  4353 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:57.641956 31574 catalog_manager.cc:5697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 10 to 15. New cstate: current_term: 15 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:57.576578  4739 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.595114 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:57.678861  4849 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:22:57.654966  4669 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.751s	user 0.014s	sys 0.000s
I20260709 12:22:57.582782  4775 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:57.719326 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:57.721614 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:57.724697 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:57.574064  4817 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:57.663379  4748 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.647045  4757 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:57.645370  4852 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:57.783586 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:57.793377 31673 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:57.668507  4798 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:57.696604  4865 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.718900  4760 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.627497  4665 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.805523 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2654) took 1152 ms. Trace:
I20260709 12:22:57.819707 31824 rpcz_store.cc:276] 0709 12:22:56.652669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.652816 (+   147us) service_pool.cc:224] Handling call
0709 12:22:56.653252 (+   436us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:56.653764 (+   512us) raft_consensus.cc:1534] Early marking committed up to index 4
0709 12:22:56.653806 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:56.654546 (+   740us) log.cc:844] Serialized 52 byte log entry
0709 12:22:56.829901 (+175355us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b1800313900. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:22:57.109348 (+279447us) raft_consensus.cc:1645] Marking committed up to 4
0709 12:22:57.109405 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.109513 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.434423 (+324910us) spinlock_profiling.cc:243] Waited 24.4 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:57.805199 (+370776us) raft_consensus.cc:1685] finished
0709 12:22:57.805227 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.805449 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":101,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":8319,"spinlock_wait_cycles":59859072,"thread_start_us":279175,"threads_started":1,"wal-append.queue_time_us":668322}
I20260709 12:22:57.718370  4779 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:57.820680  4871 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:57.806362 31574 catalog_manager.cc:5697] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 10 to 13, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 13 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:57.788261 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2653) took 1151 ms. Trace:
I20260709 12:22:57.718091  4567 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15.
I20260709 12:22:57.866292  4871 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.867007 31574 catalog_manager.cc:5697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 8 to 13. New cstate: current_term: 13 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:22:57.866461 31820 rpcz_store.cc:276] 0709 12:22:56.636657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.638152 (+  1495us) service_pool.cc:224] Handling call
0709 12:22:56.651055 (+ 12903us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:56.651300 (+   245us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:57.021574 (+370274us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.782385 (+760811us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:57.782606 (+   221us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:57.782784 (+   178us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.782970 (+   186us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.788173 (+  5203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53898,"prepare.queue_time_us":131072,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":106877,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":27441,"thread_start_us":1022722,"threads_started":2}
I20260709 12:22:57.869321  4871 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:57.869206 31574 catalog_manager.cc:5697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:57.620262  4845 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:57.827636 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:57.663753  4772 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:57.829895  4848 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.250s	user 0.000s	sys 0.000s
I20260709 12:22:57.885635 31574 catalog_manager.cc:5697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 10 to 14, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 14 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:57.797900  4798 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:57.696856  4759 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.142s
I20260709 12:22:57.809885  4669 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:57.840098  4752 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.904s	user 0.004s	sys 0.008s
W20260709 12:22:57.844995  4564 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.950s	user 0.009s	sys 0.002s
W20260709 12:22:57.845525  4851 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.303s	user 0.001s	sys 0.001s
W20260709 12:22:57.581112 31571 rpcz_store.cc:269] 0709 12:22:09.684997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:09.685315 (+   318us) service_pool.cc:224] Handling call
0709 12:22:09.868307 (+182992us) spinlock_profiling.cc:243] Waited 7.33 ms on lock 0x7b5400002210. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb1640 00007fe902b88846 00007fe902c9290e 00007fe9011532a4 00007fe901153232 00007fe901153161 00007fe9011530dc 00007fe901152392 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:57.461810 (+47593503us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:52.503006 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:22:52.590322 (+ 87316us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:22:52.590388 (+    66us) write_op.cc:435] Acquired schema lock
0709 12:22:52.590422 (+    34us) tablet.cc:662] Decoding operations
0709 12:22:52.591942 (+  1520us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:22:52.592051 (+   109us) write_op.cc:641] Partition lock acquired for write op
0709 12:22:52.592087 (+    36us) tablet.cc:685] Acquiring locks for 13 operations
0709 12:22:52.593562 (+  1475us) tablet.cc:701] Row locks acquired
0709 12:22:52.593621 (+    59us) write_op.cc:260] PREPARE: finished
0709 12:22:52.593859 (+   238us) write_op.cc:270] Start()
0709 12:22:52.594052 (+   193us) write_op.cc:276] Timestamp: P: 1783599772593776 usec, L: 0
0709 12:22:52.594094 (+    42us) op_driver.cc:348] REPLICATION: starting
0709 12:22:52.777727 (+183633us) log.cc:844] Serialized 5197 byte log entry
0709 12:22:53.262852 (+485125us) op_driver.cc:464] REPLICATION: finished
0709 12:22:53.609160 (+346308us) write_op.cc:301] APPLY: starting
0709 12:22:53.609290 (+   130us) tablet.cc:1367] starting BulkCheckPresence
0709 12:22:53.649822 (+ 40532us) tablet.cc:1370] finished BulkCheckPresence
0709 12:22:53.649982 (+   160us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:22:53.657595 (+  7613us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:22:53.657973 (+   378us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=26,key_file_lookups=26,delta_file_lookups=0,mrs_lookups=0
0709 12:22:53.658038 (+    65us) write_op.cc:312] APPLY: finished
0709 12:22:53.658491 (+   453us) log.cc:844] Serialized 121 byte log entry
0709 12:22:53.707252 (+ 48761us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:53.718887 (+ 11635us) write_op.cc:454] Released schema lock
0709 12:22:53.753377 (+ 34490us) write_op.cc:341] FINISH: Updating metrics
0709 12:22:53.789108 (+ 35731us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:22:53.789143 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":50,"spinlock_wait_cycles":16885888,"child_traces":[["op",{"apply.queue_time_us":269449,"apply.run_cpu_time_us":18886,"apply.run_wall_time_us":178695,"cfile_cache_hit":54,"cfile_cache_hit_bytes":58441,"num_ops":13,"prepare.queue_time_us":451436,"prepare.run_cpu_time_us":17640,"prepare.run_wall_time_us":353587,"raft.queue_time_us":128635,"raft.run_cpu_time_us":23197,"raft.run_wall_time_us":370842,"replication_time_us":577707,"thread_start_us":395491,"threads_started":4,"wal-append.queue_time_us":78409,"wal-append.run_cpu_time_us":19515,"wal-append.run_wall_time_us":1792950}]]}
I20260709 12:22:57.792425  4748 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.621510  4718 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:57.852408  4760 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:57.855573 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:57.809674  4819 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:22:57.818313 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:57.870381 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2655) took 1213 ms. Trace:
I20260709 12:22:57.871174  4761 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:57.695433  4803 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.105s
W20260709 12:22:57.872150 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49988us
I20260709 12:22:57.663074  4818 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.874511 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:57.876165 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:57.828385 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:57.920114  4324 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.976s	user 0.011s	sys 0.001s
I20260709 12:22:57.910584  4802 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:57.910665  4849 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.910398  4819 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:57.891055  4665 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:57.800261  4757 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.897151  4798 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:57.910596 31818 rpcz_store.cc:276] 0709 12:22:56.656613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.656759 (+   146us) service_pool.cc:224] Handling call
0709 12:22:56.660630 (+  3871us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:56.661200 (+   570us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:57.007626 (+346426us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.595100 (+587474us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:57.596089 (+   989us) log.cc:844] Serialized 26 byte log entry
0709 12:22:57.596270 (+   181us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:57.596319 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.596386 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:57.870189 (+273803us) raft_consensus.cc:1685] finished
0709 12:22:57.870208 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:57.870329 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":121452,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":28,"raft.queue_time_us":334855,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":1476,"spinlock_wait_cycles":71283584,"thread_start_us":872351,"threads_started":3,"wal-append.queue_time_us":232834}
I20260709 12:22:57.933780 31745 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:22:57.934923 31574 catalog_manager.cc:5697] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 10 to 14, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 14 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:57.943372 31574 catalog_manager.cc:5697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:22:57.938493 31823 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.000s	sys 0.014s
W20260709 12:22:57.938823 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 277862us
W20260709 12:22:57.946928 31823 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.000s	sys 0.015s
I20260709 12:22:57.947387 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:57.946259 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 177848us
I20260709 12:22:57.950464 31677 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
W20260709 12:22:57.954993 31791 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.624s (SENT) [suppressed 10 similar messages]
I20260709 12:22:57.806689  4852 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.966567 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:57.983507 31574 catalog_manager.cc:5697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 7 to 14. New cstate: current_term: 14 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:57.992771 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:57.992973 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:57.993515 31673 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:22:57.993880 31677 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:57.907719  4669 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.907307  4564 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:57.823460  4865 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.014806 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.023937 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:57.961557  4802 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:22:58.040552 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 104011us
I20260709 12:22:58.069016 32375 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: []
I20260709 12:22:58.070375 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:58.071051 31677 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:58.049712 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.040478 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:22:57.998466  4799 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:22:58.074090 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.033823  4736 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.028067  4634 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:58.077000  4703 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.674s	user 0.004s	sys 0.008s
I20260709 12:22:58.048782  4859 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:58.046432  4797 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:22:58.093804 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:58.080767  4664 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.307s	user 0.012s	sys 0.004s
I20260709 12:22:58.110052 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:58.078665  4799 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:58.078550  4802 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:58.120805 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:57.997392  4611 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.117075 31825 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:22:58.121593 32378 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:22:58.086796  4795 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:22:58.104686  4703 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:58.103730  4859 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.107385  4797 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:58.081043  4779 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.047s
W20260709 12:22:58.130792  4881 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.086s	user 0.001s	sys 0.001s
I20260709 12:22:58.104940  4509 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:58.172415 31677 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
W20260709 12:22:58.176585  4862 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.068s	user 0.000s	sys 0.001s
W20260709 12:22:58.177302  4875 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.186s	user 0.001s	sys 0.000s
I20260709 12:22:58.196245  4760 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:58.197418  4748 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:58.176585  4878 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
W20260709 12:22:58.180660  4806 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.771s	user 0.008s	sys 0.004s
I20260709 12:22:58.199127  4806 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.199864  4669 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:58.201602  4806 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:58.201364  4797 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.202435  4801 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:58.203154  4797 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:58.203087  4801 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:58.177608  4623 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.604s	user 0.009s	sys 0.000s
I20260709 12:22:58.049108  4819 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.205583  4623 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.206007  4819 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:58.207041 32372 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 15: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:58.208099  4623 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:58.209203  4894 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:58.210049  4894 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:58.212800 31748 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:58.086931  4736 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:58.178965  4879 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
W20260709 12:22:58.179361  4877 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:22:58.179487 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:58.126096  4799 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:58.120805  4634 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:58.182508 31752 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 10 index: 5. Preceding OpId from leader: term: 11 index: 6. (index mismatch)
I20260709 12:22:58.122236  4775 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:58.126652  4324 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:58.181667  4859 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.190285  4852 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:58.194056  4733 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.570s	user 0.012s	sys 0.012s
I20260709 12:22:58.194442  4611 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:22:58.178020  4866 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.202s	user 0.001s	sys 0.000s
I20260709 12:22:58.120626  4664 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.223933  4736 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:58.185205  4772 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:58.185115 31574 catalog_manager.cc:5697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:58.256467 31822 raft_consensus.cc:2410] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
W20260709 12:22:58.262153 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 138592us
I20260709 12:22:58.265010 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:22:58.239373  4400 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.836s	user 0.012s	sys 0.000s
I20260709 12:22:58.275849 31574 catalog_manager.cc:5697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 10 to 12, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 12 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:58.178018  4802 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:58.282620 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.290035 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.289059 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:58.238996  4776 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:22:58.270256  4353 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.573s	user 0.012s	sys 0.001s
I20260709 12:22:58.297914 31678 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: []
I20260709 12:22:58.173552  4795 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.309760  4795 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:58.300711 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 852) took 1271 ms. Trace:
W20260709 12:22:58.310798 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.313225  4894 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:22:58.315320  4845 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:58.316443  4739 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:58.275965  4818 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:58.231163  4634 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.320689 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.305188 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:58.349685 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.310546 31821 rpcz_store.cc:276] 0709 12:22:57.029738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.029923 (+   185us) service_pool.cc:224] Handling call
0709 12:22:57.030322 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.673670 (+643348us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.673715 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:57.674397 (+   682us) log.cc:844] Serialized 26 byte log entry
0709 12:22:57.803432 (+129035us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:22:57.803493 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.803583 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:58.300501 (+496918us) raft_consensus.cc:1685] finished
0709 12:22:58.300522 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.300651 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74961,"raft.queue_time_us":378891,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":129,"thread_start_us":766455,"threads_started":2,"wal-append.queue_time_us":283044}
I20260709 12:22:58.351274 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.352609  4806 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.350813  4760 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.353224  4806 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.353650  4760 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:58.355746  4806 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:58.355988  4760 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:22:58.356359  4806 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.356909  4806 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.351007  4623 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:22:58.357820  4623 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.358215  4623 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:22:58.359177  4887 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.178s	user 0.000s	sys 0.001s
W20260709 12:22:58.360564  4838 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.108s	user 0.002s	sys 0.001s
I20260709 12:22:58.364296 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2604) took 1019 ms. Trace:
I20260709 12:22:58.364732 31748 rpcz_store.cc:276] 0709 12:22:57.344874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.500113 (+155239us) service_pool.cc:224] Handling call
0709 12:22:57.500449 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:58.213291 (+712842us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.364005 (+150714us) spinlock_profiling.cc:243] Waited 712 ms on lock 0x7b54002ac6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:58.364218 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1637694848}
I20260709 12:22:58.366786 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:58.366967 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 827) took 1233 ms. Trace:
I20260709 12:22:58.367486 32376 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
W20260709 12:22:58.362385  4867 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.314s	user 0.001s	sys 0.001s
I20260709 12:22:58.373104 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:58.373759 32378 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:22:58.374126 31574 catalog_manager.cc:5697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 6 to 11, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 11 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:22:58.377172 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.370256 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.371537 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.306615  4776 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:58.379057 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 73975.6us
W20260709 12:22:58.362387  4902 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:22:58.380987 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:58.298421  4400 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.348297  4739 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.383556 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.384145 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:58.367538 31676 rpcz_store.cc:276] 0709 12:22:57.116557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.116712 (+   155us) service_pool.cc:224] Handling call
0709 12:22:57.117260 (+   548us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.146567 (+ 29307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.146647 (+    80us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:57.591451 (+444804us) spinlock_profiling.cc:243] Waited 5.97 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:22:57.598576 (+  7125us) log.cc:844] Serialized 26 byte log entry
0709 12:22:57.803874 (+205298us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:22:57.803909 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.803975 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:58.319789 (+515814us) raft_consensus.cc:1685] finished
0709 12:22:58.319812 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.349598 (+ 29786us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4944,"raft.queue_time_us":25200,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":18683,"spinlock_wait_cycles":13731584,"thread_start_us":140768,"threads_started":1,"wal-append.queue_time_us":383303}
I20260709 12:22:58.386420 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:58.389529 31750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1981) took 1359 ms (client timeout 1717 ms). Trace:
I20260709 12:22:58.389642 31751 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 13. Current term is 14. Ops: []
W20260709 12:22:58.390008 31750 rpcz_store.cc:269] 0709 12:22:57.030484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.030663 (+   179us) service_pool.cc:224] Handling call
0709 12:22:58.389336 (+1358673us) spinlock_profiling.cc:243] Waited 12.1 ms on lock 0x7b54001ee1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:58.389472 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":27792128}
I20260709 12:22:58.387763  4324 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:22:58.391362  4873 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:58.391906  4873 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.342406  4353 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.393216 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.393994 31818 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:22:58.394053  4353 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:58.372678  4798 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.396132 32371 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:58.390744 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:58.392238  4873 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.404474 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:58.389201 31821 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:58.401759 31749 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:58.402252 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 937) took 1009 ms. Trace:
I20260709 12:22:58.405622 32369 rpcz_store.cc:276] 0709 12:22:57.392805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.393160 (+   355us) service_pool.cc:224] Handling call
0709 12:22:57.406501 (+ 13341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.512632 (+106131us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.512684 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:57.512746 (+    62us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:57.512804 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.401989 (+889185us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b540053a554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:58.402038 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.402193 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":243353472}
I20260709 12:22:58.385435  4892 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:58.412109 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 853) took 1380 ms. Trace:
I20260709 12:22:58.382836  4865 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:58.412968 31826 rpcz_store.cc:276] 0709 12:22:57.031563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.112983 (+ 81420us) service_pool.cc:224] Handling call
0709 12:22:57.168972 (+ 55989us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.378549 (+209577us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.378588 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:57.379230 (+   642us) log.cc:844] Serialized 26 byte log entry
0709 12:22:57.646303 (+267073us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:57.646348 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.646435 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:58.411900 (+765465us) raft_consensus.cc:1685] finished
0709 12:22:58.411922 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.412049 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":331532,"spinlock_wait_cycles":13414272,"thread_start_us":418460,"threads_started":2,"wal-append.queue_time_us":488246}
I20260709 12:22:58.427187 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:58.428239  4611 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:22:58.428678  4611 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:58.429358  4611 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:58.429497 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1991) took 1359 ms. Trace:
I20260709 12:22:58.429997 31747 rpcz_store.cc:276] 0709 12:22:57.070121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.348510 (+278389us) service_pool.cc:224] Handling call
0709 12:22:57.348943 (+   433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.679109 (+330166us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.429207 (+750098us) spinlock_profiling.cc:243] Waited 295 ms on lock 0x7b5400295ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:58.429429 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":678604032}
I20260709 12:22:58.431365 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.432606 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:58.433177 31747 raft_consensus.cc:2376] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16. Re-sending same reply.
I20260709 12:22:58.434063 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.440358 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.447698 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:58.457259 31574 catalog_manager.cc:5697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 7 to 10, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 10 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:22:58.407001  4834 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:58.407559  4324 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:22:58.428568  4830 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:22:58.507301  4830 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:58.508477  4830 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.509670  4830 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:58.510201  4651 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.510866  4651 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.513423  4651 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:58.465364 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:58.520815 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:58.521576 31826 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:58.525687 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2665) took 1173 ms. Trace:
I20260709 12:22:58.527204 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 1985) took 1470 ms. Trace:
I20260709 12:22:58.527729 31744 rpcz_store.cc:276] 0709 12:22:57.057134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.057257 (+   123us) service_pool.cc:224] Handling call
0709 12:22:58.526977 (+1469720us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b54002134d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:58.527129 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":553177984}
I20260709 12:22:58.529263 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.530396 31744 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 16 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 10.
I20260709 12:22:58.531510 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 1997) took 1394 ms. Trace:
I20260709 12:22:58.532008 31752 rpcz_store.cc:276] 0709 12:22:57.136973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.397288 (+260315us) service_pool.cc:224] Handling call
0709 12:22:57.411857 (+ 14569us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:58.220545 (+808688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.531211 (+310666us) spinlock_profiling.cc:243] Waited 770 ms on lock 0x7b54001a4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:58.531438 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1771265408}
I20260709 12:22:58.533414 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 5 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.534106 31752 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 11.
I20260709 12:22:58.539799 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 758) took 1503 ms. Trace:
I20260709 12:22:58.540482 31751 rpcz_store.cc:276] 0709 12:22:57.036301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.040334 (+  4033us) service_pool.cc:224] Handling call
0709 12:22:58.389465 (+1349131us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:58.390138 (+   673us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.539497 (+149359us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54001ee1d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:58.539704 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3101926272}
I20260709 12:22:58.542151 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.545917 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2666) took 1193 ms. Trace:
I20260709 12:22:58.546519 31822 rpcz_store.cc:276] 0709 12:22:57.352694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.464660 (+111966us) service_pool.cc:224] Handling call
0709 12:22:58.545682 (+1081022us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b54000fbc54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:58.545836 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1711344384}
I20260709 12:22:58.526309 31827 rpcz_store.cc:276] 0709 12:22:57.352379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.352524 (+   145us) service_pool.cc:224] Handling call
0709 12:22:57.352873 (+   349us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:57.577427 (+224554us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.577465 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:57.594010 (+ 16545us) log.cc:844] Serialized 26 byte log entry
0709 12:22:57.809232 (+215222us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:57.809281 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.525131 (+715850us) spinlock_profiling.cc:243] Waited 41.2 ms on lock 0x7b54003ca5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:58.525198 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:58.525268 (+    70us) raft_consensus.cc:1685] finished
0709 12:22:58.525290 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.525565 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":147981,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":12687,"spinlock_wait_cycles":94766848,"thread_start_us":334494,"threads_started":2,"wal-append.queue_time_us":451459}
I20260709 12:22:58.547891 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.474926  4748 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:58.548808 31822 raft_consensus.cc:2410] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:22:58.549103 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:58.481673 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2710) took 1512 ms. Trace:
I20260709 12:22:58.416608  4892 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:58.549823 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.485046  4834 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:58.550555 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:58.550747  4834 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.550927 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:58.551759 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.483845 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.552565 31752 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:22:58.553422 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.485661  4400 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:58.549647 31678 rpcz_store.cc:276] 0709 12:22:56.969121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.969240 (+   119us) service_pool.cc:224] Handling call
0709 12:22:56.973235 (+  3995us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:58.307838 (+1334603us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.481376 (+173538us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b5400181fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:58.481585 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3045968768}
I20260709 12:22:58.488355  4797 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.554986  4400 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:58.555361  4797 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:58.557075 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.557616  4797 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:58.558166 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.559044 31751 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:58.553843 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.561808 31744 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:22:58.402135  4739 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.563683 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.564277  4798 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:58.564702  4739 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:58.495064  4664 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:58.495517  4760 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:58.565084 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.565820  4760 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.566356 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.565851 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 13 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.567028 31825 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:22:58.567401 31752 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:22:58.567960 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.568280 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.568423  4760 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:58.495515 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:58.547909 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.569118 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.569314 32369 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:22:58.569608 31827 raft_consensus.cc:2410] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:22:58.456168  4865 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.570375  4865 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:58.571691  4883 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.526s	user 0.000s	sys 0.001s
I20260709 12:22:58.573712 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 849) took 1549 ms. Trace:
I20260709 12:22:58.575060  4865 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:58.550153  4892 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:58.576424 31827 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:22:58.578141 31827 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:58.566488 31744 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:22:58.494132  4634 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:58.582597  4751 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.582840 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.583212  4751 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.574296 31823 rpcz_store.cc:276] 0709 12:22:57.024484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.024660 (+   176us) service_pool.cc:224] Handling call
0709 12:22:57.025126 (+   466us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:57.148462 (+123336us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:22:57.216439 (+ 67977us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.437208 (+220769us) spinlock_profiling.cc:243] Waited 2.79 ms on lock 0x7b180041d940. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:22:57.947596 (+510388us) log.cc:844] Serialized 14 byte log entry
0709 12:22:57.947952 (+   356us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:57.986225 (+ 38273us) log.cc:844] Serialized 52 byte log entry
0709 12:22:57.986444 (+   219us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:22:57.986495 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:57.986592 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:58.165351 (+178759us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b180041d940. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:22:58.507961 (+342610us) spinlock_profiling.cc:243] Waited 6.75 ms on lock 0x7b180041d940. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f
0709 12:22:58.508458 (+   497us) spinlock_profiling.cc:243] Waited 4.17 ms on lock 0x7b4800130158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:22:58.571047 (+ 62589us) spinlock_profiling.cc:243] Waited 7.85 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:22:58.573410 (+  2363us) raft_consensus.cc:1685] finished
0709 12:22:58.573449 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.573627 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26486,"prepare.queue_time_us":95318,"prepare.run_cpu_time_us":110,"prepare.run_wall_time_us":12732,"raft.queue_time_us":291972,"raft.run_cpu_time_us":8478,"raft.run_wall_time_us":585169,"spinlock_wait_cycles":100533888,"thread_start_us":632614,"threads_started":2,"wal-append.queue_time_us":772700}
I20260709 12:22:58.584008 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.584720 31744 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 15: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:58.585347  4751 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:58.585536 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:58.586824 31827 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:22:58.568893 31752 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:58.589622 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:58.591380 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:58.591754 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:58.590941 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.592526 31752 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:22:58.593225  4564 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:58.593348 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:58.594578  4739 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader pre-election won for term 12
W20260709 12:22:58.595024 31753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2599) took 1434 ms (client timeout 1751 ms). Trace:
I20260709 12:22:58.595160  4739 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:58.595747  4739 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
W20260709 12:22:58.595512 31753 rpcz_store.cc:269] 0709 12:22:57.160932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.179688 (+ 18756us) service_pool.cc:224] Handling call
0709 12:22:58.594851 (+1415163us) spinlock_profiling.cc:243] Waited 434 ms on lock 0x7b54000b8bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:58.594961 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":997321472}
I20260709 12:22:58.588459 31827 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:58.596941 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:58.597808 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.598549 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:58.600279 31821 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:58.600658 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.601508 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.602295 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.553648  4834 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:58.602620 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.609973  4801 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:58.533663  4623 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.612831 32371 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:58.612120 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.614848  4623 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:58.615962 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:58.617293 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.641219  4849 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:22:58.605955 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.648177  4400 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:58.606788 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:58.651546  4834 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:22:58.652550 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45856.7us
I20260709 12:22:58.654198  4834 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 12.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260709 12:22:58.655267  4751 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:58.655656  4751 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:58.656489  4751 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.6, Last appended by leader: 6, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.657549 31751 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:22:58.661749 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:58.663544  4577 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:58.664633  4577 consensus_queue.cc:1243] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 12.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:22:58.641571 31744 raft_consensus.cc:2410] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 17 because replica has last-logged OpId of term: 14 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:22:58.667640  4834 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:22:58.671792  4834 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:22:58.672101  4776 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:58.672454  4834 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:58.626796 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 756) took 1043 ms. Trace:
I20260709 12:22:58.667649 31744 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:22:58.643100 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:58.647744  4739 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.673388 32372 rpcz_store.cc:276] 0709 12:22:57.583842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.589889 (+  6047us) service_pool.cc:224] Handling call
0709 12:22:58.626529 (+1036640us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b540071d4d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:58.626722 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1163308288}
I20260709 12:22:58.673797  4834 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.674835 31749 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:58.502331  4611 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.661307  4757 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:58.676075  4760 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:58.659811  4846 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.676401 31749 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:22:58.676760  4760 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.660686 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:58.670753  4896 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:58.719277 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2678) took 1182 ms. Trace:
I20260709 12:22:58.719151  4896 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.719935 31669 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15. Ops: []
I20260709 12:22:58.721019 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:58.721827 31669 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:58.722206  4896 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:58.663568  4751 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:58.624706 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2668) took 1271 ms. Trace:
I20260709 12:22:58.634940 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.584712 31823 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:58.719063 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.675349  4834 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.621344  4806 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:58.605763 31827 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 11 index: 9. (index mismatch)
I20260709 12:22:58.676934  4846 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.676342  4739 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:58.622349 31821 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: []
W20260709 12:22:58.678259 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 209320us
I20260709 12:22:58.691669 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.744800 31676 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 15: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:58.692911 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2608) took 1112 ms. Trace:
I20260709 12:22:58.722867  4665 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:58.693552 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57112us
I20260709 12:22:58.687647 31744 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:22:58.752036 31826 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:22:58.695824 31677 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:22:58.696866 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:58.755540 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:58.755856 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.755982 31673 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:58.756912 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.757616 31673 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:58.697110 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.697445 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:58.701447 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:58.670257 31751 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:22:58.758401 31672 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:58.755622 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.759238 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.760005 31751 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:22:58.760164 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.466370 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.761406 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:58.761790 31748 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:22:58.469314 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:58.762599 31747 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:58.762781 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.763432 31673 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:22:58.723855  4896 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:22:58.677512  4611 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:58.713142 31678 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: [13.8-15.9]
I20260709 12:22:58.714211 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:58.713142  4634 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.715929 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:58.722611 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.725806 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:58.723150 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:58.724792 31675 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:22:58.724371 31750 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
W20260709 12:22:58.724507 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 63887.7us
I20260709 12:22:58.724077 31824 rpcz_store.cc:276] 0709 12:22:57.353245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.854731 (+501486us) service_pool.cc:224] Handling call
0709 12:22:57.855082 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.902554 (+ 47472us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:57.902590 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:57.902666 (+    76us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:57.902707 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.624421 (+721714us) spinlock_profiling.cc:243] Waited 47.2 ms on lock 0x7b54000fcb54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:58.624475 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.624634 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":108657408}
I20260709 12:22:58.723620 31574 catalog_manager.cc:5697] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 10 to 13, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 13 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:22:58.779347 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:58.734356 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 352148us
I20260709 12:22:58.781260 31823 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:22:58.735661 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:22:58.782315 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 46789.1us
I20260709 12:22:58.785068 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:58.719966 31819 rpcz_store.cc:276] 0709 12:22:57.537260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.209138 (+671878us) service_pool.cc:224] Handling call
0709 12:22:58.209510 (+   372us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:22:58.209895 (+   385us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:58.209924 (+    29us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:22:58.210690 (+   766us) log.cc:844] Serialized 78 byte log entry
0709 12:22:58.269715 (+ 59025us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:58.301181 (+ 31466us) log.cc:844] Serialized 14 byte log entry
0709 12:22:58.301486 (+   305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.301569 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:58.575915 (+274346us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b18002c22a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:22:58.718778 (+142863us) raft_consensus.cc:1685] finished
0709 12:22:58.719012 (+   234us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:58.719199 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":55171,"prepare.queue_time_us":31178,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":12696,"raft.run_cpu_time_us":378,"raft.run_wall_time_us":46928,"spinlock_wait_cycles":3777280,"thread_start_us":58600,"threads_started":1,"wal-append.queue_time_us":174426}
I20260709 12:22:58.688092  4760 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:58.788018 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.745438 31745 rpcz_store.cc:276] 0709 12:22:57.580628 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.598903 (+ 18275us) service_pool.cc:224] Handling call
0709 12:22:57.631781 (+ 32878us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:57.941888 (+310107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.692618 (+750730us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b54002f67d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:22:58.692832 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":693023488}
I20260709 12:22:58.747552 31753 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:22:58.747746 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:58.752192 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.789251 31744 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:58.790197 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.695605 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:58.742834  4799 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.511s	user 0.007s	sys 0.003s
I20260709 12:22:58.751966 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.764222 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.800100 31751 raft_consensus.cc:2368] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17.
I20260709 12:22:58.800968 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.801784 31751 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:58.764312 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:58.767232 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.771975 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:58.727617 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 346123us
I20260709 12:22:58.803788 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.808149 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:58.809810 31674 raft_consensus.cc:2368] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15.
I20260709 12:22:58.810441 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:58.812929 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:58.815160 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:58.818598 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:58.819523 31744 raft_consensus.cc:2376] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15. Re-sending same reply.
I20260709 12:22:58.820719 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.780619 31574 catalog_manager.cc:5697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 4 to 9, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 9 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:22:58.617209  4892 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.787786 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:58.798316 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:58.835965 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 48945.1us
I20260709 12:22:58.836982 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:58.838976 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:58.839705 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.840591 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:58.841573 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:58.840601 31669 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:22:58.824349 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:22:58.826297  4760 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:58.826841  4806 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:22:58.826987 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:58.852339 31672 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:58.852816 31678 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.853065 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.854264 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:58.827731  4888 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:22:58.828383 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:58.845659 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:58.857625 31748 raft_consensus.cc:2376] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14. Re-sending same reply.
W20260709 12:22:58.845826 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58122us
I20260709 12:22:58.804901  4400 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:58.874701 31675 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 12:22:58.846519 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.750909  4570 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.064s
I20260709 12:22:58.847421 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.847177 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.847620 31747 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:22:58.816452  4757 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:22:58.760975  4353 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.075s
I20260709 12:22:58.761793  4664 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.082s
I20260709 12:22:58.761296  4736 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.075s
I20260709 12:22:58.859431 31673 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:22:58.867080 31678 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:22:58.806548  4799 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.852506  4577 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:22:58.882007 31750 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 17: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:58.831365  4896 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:58.768097  4859 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.885501 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:58.885268  4785 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.011s	sys 0.012s
W20260709 12:22:58.890002 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:58.767763  4665 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:58.847119  4324 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:22:58.852353  4797 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:58.801838  4634 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.899920 31678 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 10. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:22:58.874251 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:58.903896 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 79619.5us
I20260709 12:22:58.880378 31672 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:22:58.858716  4704 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:58.879108  4400 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:58.957425 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:58.921604 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:58.889261 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:58.962268 31669 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:22:58.879853  4611 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:22:58.935613 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:58.886157  4739 raft_consensus.cc:2764] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:22:58.937461 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.884574  4757 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:58.903761  4760 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:22:58.973371 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 206202us
W20260709 12:22:58.938342  4785 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.011s	sys 0.013s
I20260709 12:22:58.965442 31820 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 15: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:22:58.946283 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:58.965849  4400 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:58.969954 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:22:58.972076 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50535.3us
I20260709 12:22:58.982995 31748 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:22:58.970046 31745 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:22:58.914031  4776 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:58.922659  4855 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:58.975265 31676 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:22:58.965303  4704 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:22:58.980063 31747 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 16 because replica has last-logged OpId of term: 14 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 7.
I20260709 12:22:58.978585 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:58.965541 31750 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:59.013378 31750 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 8. (index mismatch)
I20260709 12:22:58.896384  4779 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.071s
W20260709 12:22:58.836016  4718 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.015s	sys 0.037s
I20260709 12:22:58.939672  4907 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:22:59.015055  4907 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.015476  4907 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:59.016988  4665 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:58.971725  4819 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.753s	user 0.012s	sys 0.000s
I20260709 12:22:59.019033  4819 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.019395  4846 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:59.000252  4890 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 12.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:22:59.001331  4858 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:58.979118  4577 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:22:58.979238  4873 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.568s	user 0.011s	sys 0.000s
I20260709 12:22:58.966620  4859 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:59.005918 32342 leader_election.cc:400] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15.
I20260709 12:22:59.008071  4634 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:59.008507  4776 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:58.991382 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:59.043484 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52191.1us
I20260709 12:22:59.021083 31753 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:22:59.045362 31791 leader_election.cc:400] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17.
I20260709 12:22:59.046579 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:59.048336 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:22:59.025309 31669 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:22:59.027666 31750 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:22:58.978488  4757 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:22:58.936827  4801 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.054s
I20260709 12:22:59.036880 32342 leader_election.cc:403] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:22:58.958431  4896 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:22:59.030435  4718 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.016s	sys 0.038s
I20260709 12:22:59.066449 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:59.072990 31747 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:22:59.019575  4819 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.033408  4577 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:22:59.065588  4704 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.079773 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:59.009294  4760 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.069469  4779 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:59.075875  4855 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:59.067229  4846 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:22:59.089359 31670 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:22:59.091203 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:22:59.086118 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 80381us
I20260709 12:22:59.100373  4760 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:59.036747  4858 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.094502  4564 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:22:59.085963  4577 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:59.101226  4892 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:59.102584  4830 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.592s	user 0.013s	sys 0.000s
I20260709 12:22:59.103178  4795 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:22:59.084008  4865 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:22:59.081547 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:59.066023  4873 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.093631  4821 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:22:59.133916  4846 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:22:59.135623 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44502.9us
I20260709 12:22:59.134661 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:22:59.137295 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:22:59.137374 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 389731us
I20260709 12:22:59.166702 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:59.139911  4779 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:22:59.095233  4704 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:59.147881  4795 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:59.160621 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 79158.1us
I20260709 12:22:59.131839  4855 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:59.177187  4623 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:22:59.176478  4779 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:59.176577  4865 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:22:59.181377  4779 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:59.182235  4704 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:59.182075  4865 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.170229  4846 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:22:59.187979  4873 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:59.191368  4795 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.191932  4907 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.194289  4907 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:59.180634  4855 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:59.100431  4776 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:59.215814 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:59.176220  4836 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:22:59.221858  4836 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.222357  4836 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:22:59.199143 31752 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 15: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:59.148662  4888 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:22:59.165340  4739 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:22:59.208371  4733 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:22:59.209656  4859 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:22:59.210974  4757 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:59.138216  4712 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.028s	sys 0.038s
I20260709 12:22:59.214936  4896 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:59.217554  4779 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:59.220512 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:59.159512  4564 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:59.225550 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:59.238109 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:59.236694  4913 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:22:59.244333 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:59.244273 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:59.279095 32378 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 15: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:22:59.218240  4855 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:22:59.234440 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:59.260349 31823 raft_consensus.cc:2410] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
W20260709 12:22:59.260474 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 83847us
I20260709 12:22:59.261721 31677 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 12. Current term is 13. Ops: []
I20260709 12:22:59.264977 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2610) took 1618 ms. Trace:
W20260709 12:22:59.283136 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48782.8us
I20260709 12:22:59.281939 31746 rpcz_store.cc:276] 0709 12:22:57.646843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:57.687425 (+ 40582us) service_pool.cc:224] Handling call
0709 12:22:58.535481 (+848056us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:58.535929 (+   448us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:58.536061 (+   132us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:58.536695 (+   634us) log.cc:844] Serialized 26 byte log entry
0709 12:22:58.876384 (+339689us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:22:58.876432 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:58.876496 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:59.019805 (+143309us) raft_consensus.cc:1685] finished
0709 12:22:59.019826 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.243498 (+223672us) spinlock_profiling.cc:243] Waited 848 ms on lock 0x7b54002134d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:22:59.264887 (+ 21389us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2498,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":138,"spinlock_wait_cycles":1949488512,"thread_start_us":339155,"threads_started":1,"wal-append.queue_time_us":480591}
I20260709 12:22:59.286026 31746 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:22:59.286891 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:59.271356 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 429835us
I20260709 12:22:59.291586 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:59.244664 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.293996 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:59.289070 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:22:59.275079 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 279945us
W20260709 12:22:59.288496 31749 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.017s	sys 0.028s
I20260709 12:22:59.294183 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:22:59.254683  4712 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.635s	user 0.029s	sys 0.038s
I20260709 12:22:59.273188 31751 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:22:59.215004  4830 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.049s
I20260709 12:22:59.297645 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:59.299602 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:59.257382  4739 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:59.258050  4623 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:59.302073 31749 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.017s	sys 0.028s
I20260709 12:22:59.306522 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:59.311146 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:22:59.280481  4849 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.633s	user 0.008s	sys 0.005s
W20260709 12:22:59.342443 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44931.3us
I20260709 12:22:59.263080  4892 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:22:59.352424 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45326.6us
I20260709 12:22:59.357535 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:59.360601 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:59.363075 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:59.371621  4776 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:59.372519  4776 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.413899  4776 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:59.414741 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:22:59.423767 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.433207  4776 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.301496  4873 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.455488 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:59.458860 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 164958us
I20260709 12:22:59.460268 31717 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 12 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:22:59.460570 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 154250us
I20260709 12:22:59.445461  4876 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:22:59.302490  4751 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:22:59.463056 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:59.303068  4664 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:22:59.463891 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:59.464435 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:59.308460  4704 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:59.476248 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:22:59.280617  4564 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:59.445843  4888 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:22:59.540697 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 76918.2us
W20260709 12:22:59.555238 31748 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.013s	sys 0.032s
W20260709 12:22:59.559876 31748 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.013s	sys 0.032s
W20260709 12:22:59.541879 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 77627.3us
I20260709 12:22:59.457257  4801 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.304805  4798 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.174s
I20260709 12:22:59.599913 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:59.462563  4806 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:59.477631  4751 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:22:59.475275  4760 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election lost for term 12. Reason: could not achieve majority
I20260709 12:22:59.446533  4634 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.195s
I20260709 12:22:59.477835  4664 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:22:59.614832 31574 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.012s	sys 0.038s
I20260709 12:22:59.495600  4821 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15.
W20260709 12:22:59.627419 31574 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.013s	sys 0.038s
I20260709 12:22:59.435770  4776 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:59.460165  4892 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.591840  4834 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:22:59.639864 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 283271us
I20260709 12:22:59.475199  4916 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:22:59.602135  4819 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:59.548647  4733 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:59.610327  4704 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:22:59.652534 31676 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:22:59.479008  4873 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:59.614493  4877 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.110s	user 0.000s	sys 0.000s
I20260709 12:22:59.477167  4846 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.498095  4876 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:22:59.599042  4849 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:22:59.660033 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 489889us
I20260709 12:22:59.613030  4751 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:22:59.650907 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 339850us
I20260709 12:22:59.631122  4664 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:59.680141 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:59.680935 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:59.682303 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:59.683813 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:59.684854 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:59.669843 32372 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:59.681195 31671 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 15: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:22:59.692189  4919 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:22:59.693308  4855 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:22:59.694052  4855 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:59.694725  4855 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:59.696058  4846 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:22:59.696596  4918 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:22:59.696719  4846 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:59.697342  4846 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:59.698035  4797 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:59.698616  4797 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:22:59.698865 31822 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:22:59.698941  4776 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:59.699728  4776 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:22:59.699484  4797 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:22:59.700063  4400 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.674s	user 0.004s	sys 0.008s
I20260709 12:22:59.700310  4776 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:59.701006  4830 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.701520  4830 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:59.703702  4830 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:59.704618 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:59.705358 31675 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
W20260709 12:22:59.707645  4909 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.691s	user 0.004s	sys 0.000s
I20260709 12:22:59.700855  4400 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.711186  4849 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:59.711731 31744 raft_consensus.cc:2368] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15.
I20260709 12:22:59.714155  4400 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:22:59.686825  4760 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:22:59.697007  4855 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:59.718031  4915 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.622s	user 0.002s	sys 0.000s
I20260709 12:22:59.720839 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:59.722211 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:59.724406 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:59.725641 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:59.726233 31746 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:22:59.727043 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:59.727603 31746 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:22:59.728381 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:59.706403 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:59.729045 31746 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:59.729800 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:59.730340 31746 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:59.732154  4798 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:22:59.694108  4876 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.733572  4895 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:22:59.733973  4895 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:22:59.734105  4912 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.438s	user 0.000s	sys 0.001s
W20260709 12:22:59.734967 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 86057.5us
I20260709 12:22:59.734828  4895 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.638065  4718 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:22:59.748704 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:59.749831 31827 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:22:59.749614 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:22:59.665174  4736 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.645s	user 0.030s	sys 0.000s
I20260709 12:22:59.736886 32378 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:22:59.637328  4858 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:59.740083 31819 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:59.739981 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:59.742035 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:22:59.736639 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:59.762311 31821 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:59.614147  4801 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.764045 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:59.747686 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 139457us
I20260709 12:22:59.766244 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:59.766270 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:59.769734 31822 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:22:59.768977 31746 raft_consensus.cc:2376] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16. Re-sending same reply.
I20260709 12:22:59.770422 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:22:59.770483 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:59.676595  4712 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:22:59.760392  4736 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.769s	user 0.032s	sys 0.000s
W20260709 12:22:59.781030 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96438.3us
I20260709 12:22:59.831586 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:22:59.772297 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 11 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:22:59.771471 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:22:59.638511  4739 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:59.842919 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:59.844122 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:59.772679 32369 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:22:59.847465 31826 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:22:59.849900 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:59.850998 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:59.852216 31671 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 15: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:22:59.761462  4776 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:59.846354  4324 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:22:59.853572  4324 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:59.855072  4785 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:22:59.855549  4785 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:22:59.855928  4785 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:22:59.856729  4945 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:22:59.857185  4945 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:22:59.858246  4945 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:59.859706  4945 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:22:59.860271 31746 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:22:59.861904 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 11 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:59.862738 31821 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:22:59.862587 31746 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:22:59.867164 31746 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:22:59.869634 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:22:59.866065 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:22:59.669855  4733 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:22:59.771543  4760 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:22:59.873535 31673 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:22:59.738468  4819 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.103s
I20260709 12:22:59.874058 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:59.875344 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2622) took 1680 ms. Trace:
I20260709 12:22:59.876644 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:59.888684 31669 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
W20260709 12:22:59.890069 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.877364 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:59.787325  4895 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:22:59.648830  4623 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.882237 31746 pending_rounds.cc:85] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 8
I20260709 12:22:59.902341 31746 pending_rounds.cc:107] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 12:22:59.785732  4757 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:59.888433 31749 rpcz_store.cc:276] 0709 12:22:58.195269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.688727 (+493458us) service_pool.cc:224] Handling call
0709 12:22:58.689105 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:58.689773 (+   668us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:59.355192 (+665419us) log.cc:844] Serialized 14 byte log entry
0709 12:22:59.670964 (+315772us) log.cc:844] Serialized 14 byte log entry
0709 12:22:59.701096 (+ 30132us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:59.701601 (+   505us) log.cc:844] Serialized 26 byte log entry
0709 12:22:59.701759 (+   158us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:59.701800 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.701887 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:22:59.875046 (+173159us) raft_consensus.cc:1685] finished
0709 12:22:59.875079 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.875260 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34807,"prepare.queue_time_us":855352,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":67,"raft.queue_time_us":76937,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":232,"thread_start_us":822459,"threads_started":2,"wal-append.queue_time_us":400662}
I20260709 12:22:59.903716 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:59.904953 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:22:59.856017  4324 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:22:59.893735 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2632) took 1695 ms. Trace:
I20260709 12:22:59.911715 31745 rpcz_store.cc:276] 0709 12:22:58.214659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.983696 (+769037us) service_pool.cc:224] Handling call
0709 12:22:58.983988 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:58.984173 (+   185us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:59.484702 (+500529us) log.cc:844] Serialized 14 byte log entry
0709 12:22:59.893232 (+408530us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:22:59.893319 (+    87us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:59.893384 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.893481 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:22:59.893648 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":672641,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":675,"thread_start_us":864622,"threads_started":2}
I20260709 12:22:59.865424  4779 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:22:59.877476  4883 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:22:59.906507 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:22:59.867309 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:22:59.914431 31746 raft_consensus.cc:2889] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:22:59.939459 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.941283 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:22:59.970206 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:22:59.877077 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.007572 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 66395.8us
I20260709 12:23:00.008818 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:00.012861 31673 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:00.013235  4751 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:22:59.939185  4865 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:00.015251  4855 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:00.015424  4931 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:23:00.159397  4865 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:00.018712  4733 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:00.019060  4938 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:00.160481  4733 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:00.161062  4733 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.160871  4865 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.020334 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:00.163378 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 143121us
I20260709 12:23:00.018126 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:22:59.923084  4353 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
W20260709 12:23:00.014490  4913 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.733s	user 0.009s	sys 0.002s
I20260709 12:23:00.042898  4324 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.043345 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2631) took 1845 ms. Trace:
I20260709 12:23:00.018759  4776 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.025s
I20260709 12:23:00.044863  4834 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:00.047107  4564 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:00.048779  4846 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:22:59.918278  4628 raft_consensus.cc:2764] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:23:00.050520  4873 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:00.050830 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:00.051419 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:00.054479 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2015) took 1353 ms. Trace:
I20260709 12:23:00.246011 32371 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.246269 31753 rpcz_store.cc:276] 0709 12:22:58.701227 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.810144 (+108917us) service_pool.cc:224] Handling call
0709 12:23:00.054148 (+1244004us) spinlock_profiling.cc:243] Waited 210 ms on lock 0x7b54003e3ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:00.054377 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":482120960}
I20260709 12:23:00.248075 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:00.248895 31753 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:00.249995 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:00.055613 31750 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 12:23:00.250800 31750 pending_rounds.cc:85] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 8
I20260709 12:23:00.251134 31750 pending_rounds.cc:107] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:23:00.251432 31750 raft_consensus.cc:2889] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:00.056770 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2717) took 1365 ms. Trace:
I20260709 12:23:00.253423 31670 rpcz_store.cc:276] 0709 12:22:58.690874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.691046 (+   172us) service_pool.cc:224] Handling call
0709 12:22:58.691698 (+   652us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:59.137141 (+445443us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.056409 (+919268us) spinlock_profiling.cc:243] Waited 397 ms on lock 0x7b5400391cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:00.056681 (+   272us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":913759616}
I20260709 12:23:00.057894 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2722) took 1341 ms. Trace:
I20260709 12:23:00.255059 31677 rpcz_store.cc:276] 0709 12:22:58.716580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.848538 (+131958us) service_pool.cc:224] Handling call
0709 12:22:58.848890 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:59.281842 (+432952us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.057564 (+775722us) spinlock_profiling.cc:243] Waited 413 ms on lock 0x7b5400253ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:00.057816 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":948923776}
W20260709 12:23:00.092604 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:22:59.919348  4757 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:22:59.934934  4577 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.788s	user 0.012s	sys 0.000s
I20260709 12:23:00.154392 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:00.159487  4570 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:00.267083 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:00.267970 31753 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:00.165135 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:00.022361 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:00.166815 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 148780us
I20260709 12:23:00.178471 31748 rpcz_store.cc:276] 0709 12:22:58.197808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.983635 (+785827us) service_pool.cc:224] Handling call
0709 12:22:59.027557 (+ 43922us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:22:59.027776 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:22:59.599381 (+571605us) log.cc:844] Serialized 14 byte log entry
0709 12:23:00.042677 (+443296us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:00.042735 (+    58us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:00.042784 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.043147 (+   363us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.043279 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":132391,"prepare.queue_time_us":638000,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":64,"raft.queue_time_us":132756,"raft.run_cpu_time_us":88,"raft.run_wall_time_us":89,"thread_start_us":834484,"threads_started":2}
W20260709 12:23:00.156621  4946 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.270s	user 0.000s	sys 0.001s
W20260709 12:23:00.157011  4936 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
W20260709 12:23:00.156359  4940 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.381s	user 0.000s	sys 0.001s
W20260709 12:23:00.293422 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 128379us
I20260709 12:23:00.294695 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:00.156136  4921 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.497s	user 0.000s	sys 0.001s
I20260709 12:23:00.168656  4948 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:00.202067  4733 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:00.196736  4324 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.158527  4855 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:00.216044 31749 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:23:00.303615 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2623) took 2108 ms. Trace:
I20260709 12:23:00.270280 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:00.274060 31819 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.506s	user 0.022s	sys 0.034s
I20260709 12:23:00.275228 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:00.306785 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:00.307699  4846 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:00.189852  4776 consensus_queue.cc:1243] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 13.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:23:00.182001  4913 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:00.312799  4855 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.207098  4779 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:23:00.295795 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.304738 31752 rpcz_store.cc:276] 0709 12:22:58.195584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.593125 (+397541us) service_pool.cc:224] Handling call
0709 12:23:00.303332 (+1710207us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b54002954d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:00.303529 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1391107328}
I20260709 12:23:00.313848 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 761) took 1603 ms. Trace:
I20260709 12:23:00.316871 31747 rpcz_store.cc:276] 0709 12:22:58.710502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.087678 (+377176us) service_pool.cc:224] Handling call
0709 12:22:59.133665 (+ 45987us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:22:59.140126 (+  6461us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:22:59.140176 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:22:59.145665 (+  5489us) log.cc:844] Serialized 26 byte log entry
0709 12:22:59.592179 (+446514us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:22:59.592242 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.592350 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:00.313577 (+721227us) raft_consensus.cc:1685] finished
0709 12:23:00.313602 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.313777 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":150,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":5771,"thread_start_us":268753,"threads_started":1,"wal-append.queue_time_us":760745}
W20260709 12:23:00.315073 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 449063us
I20260709 12:23:00.321225 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:00.321588 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:00.266374  4570 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:00.324756 31753 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:23:00.329388 31748 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:00.330823 31748 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 12. Ops: []
W20260709 12:23:00.325402 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55293us
I20260709 12:23:00.336669 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:00.338552 31825 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.013s	sys 0.004s
I20260709 12:23:00.338361 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:00.338914 31825 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.013s	sys 0.004s
W20260709 12:23:00.344435 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.298545 31745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:00.358304 31745 raft_consensus.cc:606] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Received request to transfer leadership to TS b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:00.322677  4757 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:23:00.358799 31745 raft_consensus.cc:614] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:00.362033 32371 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:00.363377  4849 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:00.345245  4357 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.005s	sys 0.012s
W20260709 12:23:00.364920  4357 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.764s	user 0.006s	sys 0.012s
W20260709 12:23:00.366111 31744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 765) took 1626 ms (client timeout 2098 ms). Trace:
I20260709 12:23:00.366479 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:00.297405  4953 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
W20260709 12:23:00.328963  4836 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.106s	user 0.014s	sys 0.000s
W20260709 12:23:00.366530 31744 rpcz_store.cc:269] 0709 12:22:58.740207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.820387 (+ 80180us) service_pool.cc:224] Handling call
0709 12:23:00.365893 (+1545506us) spinlock_profiling.cc:243] Waited 890 ms on lock 0x7b5400064854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:00.366049 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2046972800}
I20260709 12:23:00.369603 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:00.322283  4892 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:00.380028 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 548489us
I20260709 12:23:00.335491  4834 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:00.391686 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.278518  4577 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:00.345729  4945 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:00.398939 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:00.396473 31821 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.012s	sys 0.017s
W20260709 12:23:00.401429 31821 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.012s	sys 0.017s
I20260709 12:23:00.371420  4776 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:00.414065 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:00.393833  4958 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:00.399281  4819 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:00.399406  4570 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.395715  4873 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:00.395594 31822 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.006s	sys 0.021s
I20260709 12:23:00.356285  4733 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.044s
I20260709 12:23:00.395363  4953 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:00.409422  4849 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:00.424456 31818 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:00.426609 31822 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.572s	user 0.006s	sys 0.021s
I20260709 12:23:00.412209  4324 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:00.430263 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:00.371709  4920 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.051s
I20260709 12:23:00.383459  4836 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:00.438695 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53139.3us
I20260709 12:23:00.408497  4927 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:00.415547  4779 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:00.423933 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:00.427145  4846 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:00.446203 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:00.467069 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:00.431775  4873 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.457759  4751 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:00.472098  4785 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.616s	user 0.010s	sys 0.002s
I20260709 12:23:00.472558  4785 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.473067  4873 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:00.475370  4776 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:00.475785  4836 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:00.477480  4895 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.580s	user 0.002s	sys 0.009s
I20260709 12:23:00.441598  4849 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.483134  4849 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:00.488536  4964 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:00.489068  4785 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:00.490258 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1863) took 1770 ms. Trace:
I20260709 12:23:00.490675 31674 rpcz_store.cc:276] 0709 12:22:58.719465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.852220 (+132755us) service_pool.cc:224] Handling call
0709 12:22:58.852601 (+   381us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:58.857721 (+  5120us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:58.857768 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:58.874839 (+ 17071us) log.cc:844] Serialized 52 byte log entry
0709 12:22:59.260167 (+385328us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:22:59.302272 (+ 42105us) log.cc:844] Serialized 14 byte log entry
0709 12:22:59.303496 (+  1224us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:22:59.303586 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:00.490020 (+1186434us) raft_consensus.cc:1685] finished
0709 12:23:00.490042 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:00.490185 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17654,"prepare.queue_time_us":41864,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":56,"raft.queue_time_us":122793,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":58354,"thread_start_us":336726,"threads_started":1,"wal-append.queue_time_us":613810}
I20260709 12:23:00.493278 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:00.495296 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:00.488402  4819 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:00.496353  4819 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.496654 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:00.497560 31827 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:00.497639  4888 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:00.498705  4888 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:00.489125  4964 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:00.499364 31751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2014) took 1798 ms (client timeout 1866 ms). Trace:
I20260709 12:23:00.499601  4964 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:00.499927 31751 rpcz_store.cc:269] 0709 12:22:58.700849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:58.814899 (+114050us) service_pool.cc:224] Handling call
0709 12:23:00.499155 (+1684256us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b5400403dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:00.499291 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":807966464}
I20260709 12:23:00.500922  4964 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.501410 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:00.502857  4873 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:00.503449  4873 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:00.504547  4873 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:00.510372 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 1.04832e+06us
I20260709 12:23:00.508493  4888 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:00.511082  4819 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:00.511873 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:00.513536 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:00.514307  4888 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:00.514840  4733 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
W20260709 12:23:00.536500 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 106312us
I20260709 12:23:00.527963 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:00.516916  4846 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
W20260709 12:23:00.517541 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50594.6us
I20260709 12:23:00.517894 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2003) took 2168 ms. Trace:
I20260709 12:23:00.544580 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:00.544497 31750 rpcz_store.cc:276] 0709 12:22:58.349349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.051905 (+702556us) service_pool.cc:224] Handling call
0709 12:23:00.055078 (+1003173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.251987 (+196909us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.517696 (+265709us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54003e3ad0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:00.517831 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2306139520}
W20260709 12:23:00.521044  4952 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.120s	user 0.001s	sys 0.000s
I20260709 12:23:00.522809  4949 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:00.524132 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.442587  4945 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.102s
I20260709 12:23:00.531266 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:00.502274 31751 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
W20260709 12:23:00.534806 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.537451 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:00.515334  4859 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:00.574210 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:00.575135  4859 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:00.575445 31751 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:00.575664  4859 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:00.576354 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:00.577013 31751 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:00.577836 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:00.538581 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 92443us
I20260709 12:23:00.545866 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:00.580531 31750 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:00.581964 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:00.548635 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.527551 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:00.550927 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:00.545444  4927 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:00.579926 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:00.551730  4949 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.552292  4564 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:00.553610  4965 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:00.543527  4846 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:00.545857  4819 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:00.573118 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:00.574333 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:00.431785  4958 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:00.586337 31671 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:00.580857 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:00.587199 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:00.554486  4963 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:23:00.619561 31677 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:00.546913  4665 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:00.618238 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:00.614487 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:00.615682 31674 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: [11.9-11.9]
I20260709 12:23:00.592133 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:00.596807 31827 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:23:00.585189  4751 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:00.645620  4751 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.646163  4751 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:00.648350  4945 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:00.585726  4927 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.630015 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:00.630867 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:00.657828 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 65796.1us
I20260709 12:23:00.660158 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:00.620397  4819 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.623883  4834 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:00.585805  4888 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:00.633239  4938 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:00.671551  4959 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.105s	user 0.000s	sys 0.003s
I20260709 12:23:00.671926  4938 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:00.672585  4938 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:00.625138  4748 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 12:23:00.675038 31819 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:00.675879  4951 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.011s
I20260709 12:23:00.640705  4958 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:00.619761  4965 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:00.677608  4958 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:00.641474  4954 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:00.618021  4846 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:00.678294  4958 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:00.679339  4965 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:00.680022  4970 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.082s	user 0.001s	sys 0.000s
I20260709 12:23:00.679363  4954 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:00.680583  4954 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:00.681427  4954 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:00.682080  4664 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.919s	user 0.010s	sys 0.002s
I20260709 12:23:00.682703  4954 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:00.682513  4664 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.684727  4664 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:00.658995  4927 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:00.687484  4846 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.690176  4846 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:00.642846  4948 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:00.612594  4865 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:00.612793  4785 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.031s
I20260709 12:23:00.643724 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:00.709658 31676 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
W20260709 12:23:00.617908  4739 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.010s	sys 0.020s
I20260709 12:23:00.644547 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:00.610111  4913 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:00.627324  4953 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:00.624464  4357 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.007s
I20260709 12:23:00.663862 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:00.656947  4967 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.069s	user 0.002s	sys 0.000s
I20260709 12:23:00.676213  4938 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:00.713331  4948 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:00.716593  4969 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
I20260709 12:23:00.714919  4865 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:00.705928 31671 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:00.644474  4712 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:00.710561 31674 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:00.659511  4665 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:00.660375  4776 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:00.638748 31750 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:00.668145  4819 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:00.712035  4896 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.652s	user 0.012s	sys 0.016s
W20260709 12:23:00.724611 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.662053  4564 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:00.724057 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60170.5us
I20260709 12:23:00.719826  4949 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:00.657635  4628 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 12:23:00.700639  4611 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:00.731657 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:00.727591  4938 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:00.736279 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:00.719913  4865 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 12.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.738195  4948 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:00.739519  4799 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:00.742875 31678 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:00.744979  4751 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.734138  4324 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.746367  4795 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:00.712445  4739 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.666s	user 0.011s	sys 0.021s
I20260709 12:23:00.760208 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:00.771374 31750 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:00.772863 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:00.729231  4712 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:00.709393  4855 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:00.735777  4896 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.012s	sys 0.016s
I20260709 12:23:00.760927  4564 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:00.765779  4665 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:00.791069 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:00.792025 31750 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:00.767510  4776 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:00.769269  4733 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.033s
I20260709 12:23:00.832508 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 882) took 1093 ms. Trace:
I20260709 12:23:00.833078 31818 rpcz_store.cc:276] 0709 12:22:59.739549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.753940 (+ 14391us) service_pool.cc:224] Handling call
0709 12:22:59.754285 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.438475 (+684190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.832162 (+393687us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x7b54003ca0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:00.832421 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1540660480}
I20260709 12:23:00.836272 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:00.842814 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.846017 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:00.850873 31824 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:00.852340 31750 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:00.853957 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:00.855371 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:00.855711 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:00.854135 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:00.857441 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:00.859270 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:00.860541 31750 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:00.862066 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:00.864821 32378 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 814) took 1556 ms (client timeout 1979 ms). Trace:
W20260709 12:23:00.866138 32378 rpcz_store.cc:269] 0709 12:22:59.288598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.288746 (+   148us) service_pool.cc:224] Handling call
0709 12:23:00.845166 (+1556420us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b540055da54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:00.845343 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":828675456}
I20260709 12:23:00.872084 31673 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:00.863514 31750 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:00.775300  4611 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.824802  4564 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:00.905015 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:00.906002 31674 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:23:00.907687 31677 raft_consensus.cc:1217] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Deduplicated request from leader. Original: 11.6->[12.7-16.9]   Dedup: 12.7->[14.8-16.9]
I20260709 12:23:00.908423 31677 pending_rounds.cc:85] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 7
I20260709 12:23:00.908691 31677 pending_rounds.cc:107] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 13.8
I20260709 12:23:00.908946 31677 raft_consensus.cc:2889] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:00.830567  4896 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:00.912592 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:00.915380 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.785409  4324 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:00.910266 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:00.905009 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:00.929957 31750 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:00.829221  4776 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:00.916441  4896 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:00.928675  4748 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:00.925581  4846 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:00.990280  4611 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:00.991791  4748 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:00.993734  4976 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:23:00.995554  4855 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:00.987738  4846 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:01.003542  4846 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:01.008245 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:00.978582  4939 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.919s	user 0.000s	sys 0.003s
I20260709 12:23:01.024338 31750 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:00.985342  4896 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:01.037056 31745 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.004s	sys 0.012s
W20260709 12:23:01.037513 31745 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.606s	user 0.004s	sys 0.012s
W20260709 12:23:01.064414 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 405988us
W20260709 12:23:01.074143 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:00.932271  4665 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 8, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:01.018764  4797 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:01.087457 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:01.088552 31822 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:01.089775 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:01.092140 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2594) took 1630 ms. Trace:
I20260709 12:23:01.092403 31823 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:23:01.092600 31826 rpcz_store.cc:276] 0709 12:22:59.462138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.462799 (+   661us) service_pool.cc:224] Handling call
0709 12:23:01.091876 (+1629077us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b5400446bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:01.092053 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":881660288}
W20260709 12:23:01.097443  4978 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:23:01.097785 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:01.100008 31826 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
W20260709 12:23:01.098130  4971 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.120s	user 0.000s	sys 0.000s
W20260709 12:23:01.100163 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 485761us
W20260709 12:23:01.102034 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 521262us
I20260709 12:23:01.078959  4611 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:01.085407 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:01.138000 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:01.141155 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:01.142583 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 57280.7us
W20260709 12:23:01.147492 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.007s	sys 0.016s
W20260709 12:23:01.147910 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.007s	sys 0.016s
W20260709 12:23:01.148221 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 523381us
I20260709 12:23:01.149307  4949 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election won for term 15
W20260709 12:23:01.133548  4873 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.032s	sys 0.055s
W20260709 12:23:01.136708  4973 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 12:23:01.084340  4895 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 12:23:01.152395 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:01.158794 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:01.084614  4927 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:01.153817  4849 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.670s	user 0.010s	sys 0.005s
I20260709 12:23:01.171144  4849 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:01.174170  4849 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:01.138993  4748 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:01.168501  4974 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.085s	user 0.000s	sys 0.001s
I20260709 12:23:01.170360  4776 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:01.166889 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:01.187470  4975 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.209s	user 0.001s	sys 0.000s
I20260709 12:23:01.174464 31827 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:01.153753  4834 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:01.150707  4948 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:01.176817 31672 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:23:01.162753  4949 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:01.160068  4751 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:01.158821  4873 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.032s	sys 0.056s
W20260709 12:23:01.198549 31746 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.010s	sys 0.014s
W20260709 12:23:01.203042 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 430239us
I20260709 12:23:01.203924 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2031) took 1520 ms. Trace:
W20260709 12:23:01.202188 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53339.6us
I20260709 12:23:01.139582  4611 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:01.208189 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:01.209139 31753 rpcz_store.cc:276] 0709 12:22:59.683778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.329116 (+645338us) service_pool.cc:224] Handling call
0709 12:23:00.329522 (+   406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:00.330408 (+   886us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:00.379399 (+ 48991us) log.cc:844] Serialized 14 byte log entry
0709 12:23:00.480026 (+100627us) spinlock_profiling.cc:243] Waited 7.01 ms on lock 0x7b18003cade0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:00.831806 (+351780us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:00.865215 (+ 33409us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.865430 (+   215us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:00.865481 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.865570 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.193655 (+328085us) raft_consensus.cc:1685] finished
0709 12:23:01.203589 (+  9934us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.203838 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2728,"prepare.queue_time_us":48340,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":40,"raft.queue_time_us":5195,"raft.run_cpu_time_us":302,"raft.run_wall_time_us":57935,"spinlock_wait_cycles":16120704,"thread_start_us":420956,"threads_started":1,"wal-append.queue_time_us":795227}
I20260709 12:23:01.212265 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:01.216879 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:01.209318 31746 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.556s	user 0.010s	sys 0.014s
W20260709 12:23:01.205716  4977 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
I20260709 12:23:01.209834 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:01.222252 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:01.201172  4948 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.270816 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:01.272176 31753 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:01.224802 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:01.278409 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53684.5us
I20260709 12:23:01.279227 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:01.281874 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:01.275509 31753 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:01.292548 31753 raft_consensus.cc:606] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:01.283077 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:01.294027 31753 raft_consensus.cc:614] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:01.294392 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:01.305083 32342 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.771s	user 0.013s	sys 0.037s
W20260709 12:23:01.305768 32342 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.773s	user 0.014s	sys 0.038s
W20260709 12:23:01.306820 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 773799us
I20260709 12:23:01.275082  4948 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:01.268811  4873 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:01.273599  4776 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:01.296391  4751 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:01.269925  4938 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:01.273923  4748 raft_consensus.cc:2764] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election decision vote started in defunct term 17: won
I20260709 12:23:01.274399  4849 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:01.297500  4991 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:01.308679 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:01.291078  4954 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.035s
W20260709 12:23:01.311146 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.307935 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:01.306758  4855 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:01.315829 31753 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:23:01.317217 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2026) took 1637 ms. Trace:
I20260709 12:23:01.330242 31752 rpcz_store.cc:276] 0709 12:22:59.682067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.317026 (+634959us) service_pool.cc:224] Handling call
0709 12:23:00.317373 (+   347us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:00.317720 (+   347us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:00.317750 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:00.318314 (+   564us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.772473 (+454159us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:00.772542 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:00.772645 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.316990 (+544345us) raft_consensus.cc:1685] finished
0709 12:23:01.317011 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.317151 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":800,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":34919,"thread_start_us":392529,"threads_started":1,"wal-append.queue_time_us":618723}
I20260709 12:23:01.312085  4733 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:01.314467 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:01.278174  4785 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.036s
I20260709 12:23:01.315764  4751 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.315578  4776 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:01.336969 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:23:01.335011  4972 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.203s	user 0.000s	sys 0.001s
I20260709 12:23:01.332937  4849 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:01.345077  4981 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.211s	user 0.000s	sys 0.001s
I20260709 12:23:01.336418  4855 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:01.339475  4733 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:01.350455  4776 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:01.374735 31749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 767) took 1488 ms (client timeout 1584 ms). Trace:
W20260709 12:23:01.383221 31749 rpcz_store.cc:269] 0709 12:22:59.886244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.904679 (+ 18435us) service_pool.cc:224] Handling call
0709 12:23:01.374411 (+1469732us) spinlock_profiling.cc:243] Waited 310 ms on lock 0x7b54001ee1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:01.374619 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":713800960}
I20260709 12:23:01.346395  4927 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:01.344712  4751 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:01.379015 31752 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:01.314760  4949 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:01.395738  4994 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
W20260709 12:23:01.397301  4986 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:23:01.397735  4954 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:23:01.398047  4712 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.601s	user 0.010s	sys 0.000s
W20260709 12:23:01.403564  4983 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:23:01.404788 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2030) took 1721 ms. Trace:
I20260709 12:23:01.405539  4846 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:01.408252  4995 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:23:01.406069  4564 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:01.412092  4577 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:01.418787 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 881) took 1679 ms. Trace:
I20260709 12:23:01.464941 31823 rpcz_store.cc:276] 0709 12:22:59.739268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.740336 (+  1068us) service_pool.cc:224] Handling call
0709 12:23:01.092028 (+1351692us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:01.092870 (+   842us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.418478 (+325608us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b5400446bd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:01.418692 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3107811712}
I20260709 12:23:01.381366  4733 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:01.436014 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 269196us
W20260709 12:23:01.439324 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 360267us
I20260709 12:23:01.468801 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:01.440600 31747 rpcz_store.cc:276] 0709 12:22:59.683478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.318277 (+634799us) service_pool.cc:224] Handling call
0709 12:23:00.318596 (+   319us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:00.318962 (+   366us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:00.318996 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:00.319478 (+   482us) log.cc:844] Serialized 26 byte log entry
0709 12:23:00.855477 (+535999us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b1800090400. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:23:01.207299 (+351822us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:01.220671 (+ 13372us) log.cc:844] Serialized 14 byte log entry
0709 12:23:01.221490 (+   819us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.221587 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:01.404323 (+182736us) raft_consensus.cc:1685] finished
0709 12:23:01.404477 (+   154us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.404695 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29888,"prepare.queue_time_us":14077,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":2301,"raft.queue_time_us":2053,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":28217,"spinlock_wait_cycles":26387200,"thread_start_us":351596,"threads_started":1,"wal-append.queue_time_us":988317}
W20260709 12:23:01.477596 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.478367 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 202897us
I20260709 12:23:01.479605 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:01.435743  4954 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:01.485445 31674 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:01.462689  5000 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:01.493503 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:01.496759 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.478644  4611 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:01.458842  4712 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:01.460006  4997 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.058s
W20260709 12:23:01.494887  4859 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.919s	user 0.013s	sys 0.002s
I20260709 12:23:01.436475  4938 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:01.488878  5000 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:01.505983  4795 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.720s	user 0.005s	sys 0.009s
I20260709 12:23:01.508459  4785 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:01.522768  4859 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:01.528857 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:01.530181  4992 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:01.530946  4992 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:01.532163  4992 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:01.533947  4992 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:01.535272  4785 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.535914  4785 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:01.538290  4785 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:01.539884  4989 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:01.545830 31669 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:01.550908  4785 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:01.547868 31674 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:23:01.551370  4785 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:01.528859  4859 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:01.530364 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:01.552328  4785 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:01.472697  4849 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:01.568595  4896 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.525s	user 0.012s	sys 0.000s
I20260709 12:23:01.569298  4896 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:01.572795  4819 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.833s	user 0.012s	sys 0.000s
I20260709 12:23:01.573458  4819 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:01.575915  4564 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:01.578217  4888 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:01.579192  4888 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:01.581040 32371 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:23:01.582564  4888 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:01.520618  5000 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:01.587852  4876 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.588735  4876 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:01.488005  4855 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:01.604924  4858 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.605551  4858 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:01.608253  4858 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:01.616111  4846 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:01.618832  4712 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:01.520174  4954 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:01.628911  4954 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:01.646242 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:01.647845 31675 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:01.661639 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 324738us
I20260709 12:23:01.672626 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:01.679142 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:01.693392  4819 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:01.695168 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2039) took 1290 ms. Trace:
I20260709 12:23:01.695756 31752 rpcz_store.cc:276] 0709 12:23:00.404294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:01.331779 (+927485us) service_pool.cc:224] Handling call
0709 12:23:01.374658 (+ 42879us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:01.390887 (+ 16229us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.694882 (+303995us) spinlock_profiling.cc:243] Waited 42.5 ms on lock 0x7b54001ee1d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:01.695084 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":97664896}
I20260709 12:23:01.698448 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 860) took 1125 ms. Trace:
I20260709 12:23:01.698951 31678 rpcz_store.cc:276] 0709 12:23:00.573506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.573659 (+   153us) service_pool.cc:224] Handling call
0709 12:23:00.667024 (+ 93365us) spinlock_profiling.cc:243] Waited 2.6 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:23:00.734131 (+ 67107us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.778717 (+ 44586us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.698192 (+919475us) spinlock_profiling.cc:243] Waited 66.8 ms on lock 0x7b54001d6350. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:01.698376 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":159609344}
I20260709 12:23:01.640790  4751 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:01.712507 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:01.680729  4564 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 11.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260709 12:23:01.687435  4846 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.042s
I20260709 12:23:01.722486  4951 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:01.748023  4797 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.663s	user 0.009s	sys 0.005s
I20260709 12:23:01.723187  4997 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:23:01.763388  4951 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:01.771281  5004 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.017s
I20260709 12:23:01.780207  4951 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:01.781358  5014 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:01.788187  5014 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:01.789124  5014 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:01.790585  5014 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:01.796615  4948 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:01.794384  5000 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:01.800752  5000 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:01.850345 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 339830us
I20260709 12:23:01.857188 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:01.865378 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 114639us
I20260709 12:23:01.876567  5000 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.002s
I20260709 12:23:01.769459  4739 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:01.889071 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:01.890653  4751 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:01.895963 31826 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:01.897749 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 857) took 1343 ms. Trace:
I20260709 12:23:01.899147 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:01.907325 31824 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:01.781703  4896 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:01.911391  5019 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:01.912889  4896 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:01.914170 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:01.916599  4948 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:01.918962 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:01.927268 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.929414  4739 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:01.918869 31675 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
W20260709 12:23:01.933495 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.934057 31751 raft_consensus.cc:2410] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
W20260709 12:23:01.934208 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:01.935660 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:01.887461  5010 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:01.938056  5010 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:01.938499  5010 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:01.919242  5019 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:01.898505 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:01.948007 31825 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:01.949106 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2702) took 1561 ms. Trace:
I20260709 12:23:01.914036  4896 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:01.953809  4896 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:01.905251 31670 rpcz_store.cc:276] 0709 12:23:00.554367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.555424 (+  1057us) service_pool.cc:224] Handling call
0709 12:23:01.895922 (+1340498us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:23:01.896492 (+   570us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:01.897057 (+   565us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:01.897088 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:01.897149 (+    61us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:01.897191 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.897278 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:01.897425 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2810752}
I20260709 12:23:01.949702 31819 rpcz_store.cc:276] 0709 12:23:00.387534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.547673 (+160139us) service_pool.cc:224] Handling call
0709 12:23:00.547943 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.675511 (+127568us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:01.948421 (+1272910us) spinlock_profiling.cc:243] Waited 127 ms on lock 0x7b54003c8cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:01.948884 (+   463us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":291543936}
I20260709 12:23:01.968291  4718 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:01.969185  4718 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:01.975272  4857 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:01.975828  4857 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:01.976586 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:01.976922 31823 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:01.977322  4857 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:01.978128 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:01.974188  4757 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:01.985636  4757 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:01.991690 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:01.992897  4785 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:01.999744  4857 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:01.999985  4718 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:02.001729  4718 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:02.003218  4757 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.007746  4849 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:02.015693 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.016569  5004 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:02.018076 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2023) took 2339 ms. Trace:
I20260709 12:23:02.018198  4903 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:02.019841  4925 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:02.030707 31825 raft_consensus.cc:2410] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 17 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:23:02.033982  4611 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.035034  4611 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.036047  5004 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:02.036607  5004 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:02.034063 31819 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:23:02.001612  4876 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:02.070899 31818 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:02.038477  4611 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:02.078179 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 75073.1us
I20260709 12:23:02.080435  4849 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.081395  4953 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.082079  4953 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.082923 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.085794  4873 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.087839 31826 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:02.089610  4797 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:23:02.091598 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 177822us
I20260709 12:23:02.091574  4757 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:02.100486 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:02.038126  5004 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.105598 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.090546  4913 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:02.112999  4888 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:02.115447  4925 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.116763 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:02.121423  4733 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.643s	user 0.021s	sys 0.005s
I20260709 12:23:02.123627 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:02.127852  4665 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:23:02.129149 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.178803 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.129030  5021 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.028951 31746 rpcz_store.cc:276] 0709 12:22:59.678566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:59.879397 (+200831us) service_pool.cc:224] Handling call
0709 12:22:59.881348 (+  1951us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:22:59.921811 (+ 40463us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:22:59.921864 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:22:59.925407 (+  3543us) log.cc:844] Serialized 52 byte log entry
0709 12:23:00.555728 (+630321us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:00.626050 (+ 70322us) log.cc:844] Serialized 14 byte log entry
0709 12:23:01.267220 (+641170us) log.cc:844] Serialized 14 byte log entry
0709 12:23:02.016796 (+749576us) spinlock_profiling.cc:243] Waited 7.41 ms on lock 0x7b1800439f60. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:23:02.017234 (+   438us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:02.017366 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:02.017522 (+   156us) raft_consensus.cc:1685] finished
0709 12:23:02.017538 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:02.017769 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":120604,"prepare.queue_time_us":119398,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":84,"raft.queue_time_us":53068,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":95632,"spinlock_wait_cycles":17054080,"thread_start_us":853721,"threads_started":2,"wal-append.queue_time_us":664361}
I20260709 12:23:02.136485  4801 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.146986  4954 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.151326 31818 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []
W20260709 12:23:02.159286  5025 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 12:23:02.091068  4988 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:02.183653  4988 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.179644 31819 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:02.181120 31825 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:02.199002 31823 raft_consensus.cc:2410] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:23:02.093175  4873 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.209429  4925 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:02.209933 31751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2035) took 1807 ms (client timeout 1676 ms). Trace:
I20260709 12:23:02.210289 31827 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:02.211344  4913 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.213874  4913 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.193813  4733 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.214552  4665 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.215384  5004 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:02.216821  4733 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:02.217403  5021 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.219982  5021 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.220623 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:02.221539 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 117640us
I20260709 12:23:02.222999  4948 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:23:02.223801 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 93061.2us
I20260709 12:23:02.225667  4751 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.226225 31678 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:02.226390  4751 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.227641  4949 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.228495 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 778) took 2212 ms. Trace:
I20260709 12:23:02.228385  4949 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.229786  4895 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:02.228986 31744 rpcz_store.cc:276] 0709 12:23:00.015943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.385562 (+369619us) service_pool.cc:224] Handling call
0709 12:23:00.387834 (+  2272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.388086 (+   252us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:00.865934 (+477848us) log.cc:844] Serialized 14 byte log entry
0709 12:23:02.059300 (+1193366us) log.cc:844] Serialized 14 byte log entry
0709 12:23:02.103222 (+ 43922us) log.cc:844] Serialized 14 byte log entry
0709 12:23:02.227685 (+124463us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:02.227773 (+    88us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:02.227897 (+   124us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:02.228033 (+   136us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:02.228240 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":911345,"prepare.queue_time_us":164342,"prepare.run_cpu_time_us":125,"prepare.run_wall_time_us":132,"raft.queue_time_us":936233,"raft.run_cpu_time_us":361,"raft.run_wall_time_us":359,"thread_start_us":867236,"threads_started":3,"wal-append.queue_time_us":612616}
I20260709 12:23:02.231184 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:02.232653  4611 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.233407  4611 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.233898 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 775) took 2294 ms. Trace:
I20260709 12:23:02.234287 31748 rpcz_store.cc:276] 0709 12:22:59.939636 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.364036 (+424400us) service_pool.cc:224] Handling call
0709 12:23:00.364344 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:00.364524 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:00.830925 (+466401us) log.cc:844] Serialized 14 byte log entry
0709 12:23:02.199583 (+1368658us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:02.233362 (+ 33779us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:02.233450 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:02.233553 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:02.233731 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":772782,"prepare.queue_time_us":115719,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":28,"raft.queue_time_us":822301,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":147,"thread_start_us":1004224,"threads_started":3,"wal-append.queue_time_us":573502}
I20260709 12:23:02.236455  4797 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:02.236445  5007 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:02.238837 31747 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:02.242148  5007 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.243436 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:02.244105  4849 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:02.244408 31744 raft_consensus.cc:2410] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 12 because replica has last-logged OpId of term: 11 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:23:02.223860  4948 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.237560  4876 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.013s
I20260709 12:23:02.247915  4895 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:02.248420  4895 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:02.249531  4895 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:02.251266 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 127826us
I20260709 12:23:02.197932  4357 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:23:02.247457  4954 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.239748 31822 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:02.216909  4665 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.254408 32342 leader_election.cc:400] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15.
I20260709 12:23:02.265357 31677 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
W20260709 12:23:02.210489 31751 rpcz_store.cc:269] 0709 12:23:00.402874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.577597 (+174723us) service_pool.cc:224] Handling call
0709 12:23:02.209603 (+1632006us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54002963d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:02.209782 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3114414848}
I20260709 12:23:02.277513  4570 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:02.278445  5012 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:02.278723  4951 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:02.285007  4776 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.892s	user 0.020s	sys 0.004s
I20260709 12:23:02.286480  4873 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:02.287091  4797 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.271214  4876 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.017s
I20260709 12:23:02.235867  4611 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:02.228930  4751 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.247231  4948 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.244789  4849 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:02.256541  5007 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.260030  4949 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:02.307132 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.329231 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:02.257009  4988 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.334039  4988 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:02.325006 31827 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:02.262156  4954 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:02.334934 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 80826.2us
W20260709 12:23:02.337795  5031 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.220s	user 0.002s	sys 0.000s
I20260709 12:23:02.338855 31821 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:02.339540 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.339967  4951 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:02.340391 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.341161 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.342005  4849 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:02.299086  5012 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:02.237210  4888 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.371672  4324 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:02.375073 31676 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:02.375941  4895 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:02.314484  4776 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.379166 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:02.385651 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:02.399780 31751 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:02.369284  4953 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:02.403386 31824 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:02.416714 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.419278 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 50797us
I20260709 12:23:02.423764 31818 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 15: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:02.425963  5029 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
W20260709 12:23:02.427711 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 944878us
I20260709 12:23:02.430290 31748 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:23:02.431612 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:02.431803 31753 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:02.431982  4888 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.432283 31748 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:02.433341  5040 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
W20260709 12:23:02.435809  5005 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:23:02.437146  4353 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:02.437367 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 772) took 2519 ms. Trace:
I20260709 12:23:02.437904  4353 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.437911 31745 rpcz_store.cc:276] 0709 12:22:59.917608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.359697 (+442089us) service_pool.cc:224] Handling call
0709 12:23:00.360061 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:00.373598 (+ 13537us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:01.071143 (+697545us) log.cc:844] Serialized 14 byte log entry
0709 12:23:02.192453 (+1121310us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:02.291433 (+ 98980us) log.cc:844] Serialized 26 byte log entry
0709 12:23:02.291686 (+   253us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:02.291748 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:02.291891 (+   143us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:02.436943 (+145052us) raft_consensus.cc:1685] finished
0709 12:23:02.436973 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:02.437143 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":605948,"prepare.queue_time_us":106798,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":25,"raft.queue_time_us":645007,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":11252,"thread_start_us":1004424,"threads_started":3,"wal-append.queue_time_us":437318}
I20260709 12:23:02.441015 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:02.441701 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:02.444646 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:02.445256 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:02.446401 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:02.447283 31751 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:23:02.442174 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:02.425853  4938 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.088s
I20260709 12:23:02.454245  4938 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.443055 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:02.425920 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.442813 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:02.445407  4846 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:02.452122  4776 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:02.415915 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.392791  5010 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.444190  4849 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.464883 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:02.465121 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.465950 31822 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:02.466962 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:02.466569  5012 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.441061 32376 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:02.425781 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:02.425796 31750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2040) took 1976 ms (client timeout 2200 ms). Trace:
W20260709 12:23:02.469728 31750 rpcz_store.cc:269] 0709 12:23:00.404591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.904614 (+500023us) service_pool.cc:224] Handling call
0709 12:23:02.353038 (+1448424us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b54002968d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:02.380794 (+ 27756us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3384576}
I20260709 12:23:02.461400 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:02.463124 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.471442 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.464522 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:02.471837 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:02.472254  4739 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:02.472659  5039 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:23:02.473140 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.473469 31827 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 16 because replica has last-logged OpId of term: 15 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:23:02.474061 31818 raft_consensus.cc:2410] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:23:02.474831 31827 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:02.476764  5010 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:02.477155 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.477452  5010 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:23:02.477787 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.478940 31672 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:02.491376 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:02.492156 31746 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:02.478348 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.494587 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:02.495000 31676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1881) took 1640 ms (client timeout 1759 ms). Trace:
W20260709 12:23:02.495422 31676 rpcz_store.cc:269] 0709 12:23:00.854068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:00.854211 (+   143us) service_pool.cc:224] Handling call
0709 12:23:02.494780 (+1640569us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b540017c0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:02.494917 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3351513472}
I20260709 12:23:02.495785 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:02.496868 32369 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:02.497857 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:02.470844  4846 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:02.500204  4846 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.501034 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:02.501770 31751 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:02.503239 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:02.503551  4903 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:02.504112  4903 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:02.505257  4903 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.506875  4903 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:02.481283 31744 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 11. Current term is 12. Ops: [11.7-11.7]
I20260709 12:23:02.512480  4797 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.480955 31745 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 13. Current term is 14. Ops: [13.8-13.8]
I20260709 12:23:02.490782 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:02.486141 32375 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:02.517048 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:02.519795 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 125923us
I20260709 12:23:02.457470  4938 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.521134 32371 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:02.533365 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:02.537703 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.539860 32375 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:02.539994 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:02.539992 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:02.545032 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.545918 31671 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:02.546763 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.547395 32369 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 13 index: 11. (index mismatch)
I20260709 12:23:02.548322 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:02.536262 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.551420 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.552491 31671 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 14 index: 7. (index mismatch)
I20260709 12:23:02.553586 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.554550 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.554979 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:02.555219 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:02.555895 31673 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:02.555984 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.556537 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:02.556712 31676 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:02.557090 31670 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:02.559015 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:02.559494 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:02.559720  5016 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:02.560312  5016 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:02.561331  5016 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.559763 31747 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:23:02.563230  5037 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.564023  5037 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:02.564963  5043 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.086s	user 0.001s	sys 0.001s
I20260709 12:23:02.566120 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:02.566990 31674 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:02.562459  5016 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:02.571077 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.572161 31671 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:02.573769  4353 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:02.576210  4938 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:02.478202  5010 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:02.587785 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.589555 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:02.597317 31746 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:02.599331 32371 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:02.601922 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:02.602396 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:02.611966 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.611956 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.590822  4949 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:02.615236 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:02.605638 32369 raft_consensus.cc:2376] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18. Re-sending same reply.
I20260709 12:23:02.581233  5037 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:02.611843 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.626951 32374 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:02.632022 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.636021 31676 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:23:02.639086  4797 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:02.639724  4570 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.640391  4570 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.641062  4797 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:02.641570  4797 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:02.642813  5008 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.643601  5008 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.645781  4819 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.646514  4819 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.646907  4739 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:02.647006  5008 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:02.647656  4739 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.648567  4873 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:02.648855 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.649014  4819 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.649236  4873 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.649720  4819 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.650149  4739 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.650938  5036 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.651827  4873 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.651592  5036 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.652400  4873 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.652434 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:02.653076  4873 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.654235  5036 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.648042  4948 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.679584  4948 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.682209  4948 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:02.654937 31675 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:02.654500  4865 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:02.683911  4865 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:02.684394  4865 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:02.654829  5007 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:02.655772  4570 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:02.639122  4895 raft_consensus.cc:2764] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election decision vote started in defunct term 14: won
I20260709 12:23:02.687250  5007 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:02.658458  4873 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:02.659228 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.688546 31678 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
W20260709 12:23:02.660135  5052 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:23:02.689852 32377 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:02.618275  4353 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:02.650368  4819 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.690920  4353 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.692224 31753 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:02.693075  4819 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:02.693805  4353 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:02.661548 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 46441.5us
I20260709 12:23:02.663913  5010 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:02.665956 31752 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: [15.10-15.10]
W20260709 12:23:02.655177 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.714157  4797 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.665257 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:02.669916 31750 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 12 index: 9. (term mismatch)
I20260709 12:23:02.717499 31750 pending_rounds.cc:85] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 8
I20260709 12:23:02.717677 31670 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:23:02.672672  4776 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:02.673146 32373 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:02.638185  4949 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:02.729828 31674 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:02.676082 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:02.678875 31746 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:02.687635  5007 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:02.702735 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56238.7us
I20260709 12:23:02.705710 31751 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:23:02.706683 31748 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
W20260709 12:23:02.707541 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 72216.3us
I20260709 12:23:02.707739 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:02.711024 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:02.713910 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:02.715067 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78127.7us
I20260709 12:23:02.717778 31750 pending_rounds.cc:107] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 11.9
I20260709 12:23:02.739565 31753 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:02.742420 31750 raft_consensus.cc:2889] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:02.744912 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 68936.4us
I20260709 12:23:02.739681 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:02.740537 31749 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:02.757071 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:02.757758 31753 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:23:02.750850  4849 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:02.767254 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:02.765859 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:02.763150 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:02.768600 31749 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:02.755326  5010 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:02.759797  5046 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:23:02.779181 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.779774  5046 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:02.781105  5046 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:02.783921  4797 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:02.773947  4903 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:02.775347  4938 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:02.759905  4324 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:02.803742 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:02.806850 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:02.804585 31678 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:02.769896  4849 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:02.812978 31673 raft_consensus.cc:2410] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 17 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
W20260709 12:23:02.814586 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.817977 31670 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:02.762180  4739 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.040s
I20260709 12:23:02.784859  4353 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:23:02.825623 31753 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:02.826423 31676 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:02.831670 31676 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:02.832715 31676 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.833812 31676 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:02.835554 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.836350 31825 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:23:02.839593 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.797029  4938 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:02.849870 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:02.851106 31749 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:23:02.812500  4819 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:02.813614  4849 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:02.856868 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:02.859881 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.804908  4324 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.862823 32374 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:02.867738 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:02.865280 32373 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:02.865126 31822 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:02.804565  4859 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260709 12:23:02.875278 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.890552 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:02.891980 31749 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:02.904450  4846 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:02.904837 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.905217  4846 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.911217 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:02.914774 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 105708us
I20260709 12:23:02.917277 31821 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:02.918687 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 154800us
I20260709 12:23:02.912012 31753 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:23:02.934064 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:02.937992 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:02.907941  4846 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:02.948765 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:02.949847 31751 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:02.938436 31749 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:02.943323 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:02.957938 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 208692us
I20260709 12:23:02.951413 31753 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:02.925424  4570 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.044s
I20260709 12:23:02.845433  4949 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.971920 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.975243 31825 raft_consensus.cc:2410] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:23:02.967087 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:02.979766 31750 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:02.974968 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:02.982542  5009 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:02.983125  5009 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:02.984176  5057 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 12:23:02.984356  5009 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.7, Last appended by leader: 7, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.985870  5009 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:02.987318 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:02.988060 31745 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:02.988811 31749 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:23:02.990003 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:02.924189  4353 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:23:02.991016 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:02.991417  4353 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:02.991681 31747 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:02.991955  4353 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:02.993012  4353 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:02.994321  4353 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:02.994899 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:02.997197 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:02.998119  4819 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:02.999222 31674 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: [13.11-13.11]
I20260709 12:23:02.999603 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:02.999877 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.001811  5008 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:03.003116 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:03.002928  4570 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:03.003495 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:03.021327 31749 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:03.003787 31745 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:02.978363  4938 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.11, Last appended by leader: 11, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:03.004415 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:03.005296  4785 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:03.005638  4865 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.007611 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:02.936910  4888 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.071s
I20260709 12:23:03.007862  4324 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:02.979027 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:03.009740 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:03.014663 31672 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:03.013370  4846 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.006s
I20260709 12:23:03.013182  4757 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.007s
W20260709 12:23:03.032639 31791 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.625s (SENT) [suppressed 15 similar messages]
I20260709 12:23:03.037832  5016 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:03.046001  5016 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:03.046097 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 46330.5us
I20260709 12:23:03.046509  5016 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:03.046892 31753 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:23:03.047039 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.048029 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:03.049052 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.049206 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:03.049901 31753 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:23:03.051398 31749 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:03.051805  4865 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:03.052740 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:03.054119 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:03.054414 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.055609 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:03.056773 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:03.057503 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:03.059926  5055 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 12:23:03.062407 31670 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:23:03.063721 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:03.064514 31670 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 14: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:03.066807 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:03.067562 31671 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:23:03.028980 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:03.020545  4819 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.070590 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:03.071374 31749 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:03.030273 31745 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:03.072589  4819 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:03.031643 31676 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:23:03.073273 31745 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:03.033308 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:03.074585 31745 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:23:03.082289 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:03.035157 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:03.040795 31748 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 12: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:03.042251  4846 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:03.057224  5016 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.028007 31750 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
W20260709 12:23:03.096086 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47102.3us
I20260709 12:23:03.097173 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.079089 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.100615 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:03.101872  4757 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:03.102841  4757 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:03.103396  4757 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:03.104519  4757 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:03.032765  4948 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:03.107183  4324 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:03.114457  5011 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.035s
I20260709 12:23:03.117331  5010 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:03.041433  4797 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:23:03.108687 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 73620.1us
I20260709 12:23:03.094609  4846 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:03.096097  4873 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260709 12:23:03.077317 31751 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:03.139602 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:03.013574  4739 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.105s
I20260709 12:23:03.137120 31744 raft_consensus.cc:2376] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16. Re-sending same reply.
W20260709 12:23:03.150143  5056 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 12:23:03.141036  4324 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.010s
W20260709 12:23:03.151755 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.159307 31747 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
W20260709 12:23:03.159686 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.160266 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.118961  4819 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:03.108100  5008 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:03.132313  5037 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:03.173105 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:03.132526  5010 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:03.176578 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:03.074082 31749 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
W20260709 12:23:03.183748 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 86664us
I20260709 12:23:03.185417 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.188392  5007 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.189929  5007 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:03.193017 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:03.193341  4846 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.142494  4938 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:03.184785 31819 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
W20260709 12:23:03.213624 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.217185 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44188us
W20260709 12:23:03.220664 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50809.4us
W20260709 12:23:03.224185 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 221208us
W20260709 12:23:03.225508 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 124992us
I20260709 12:23:03.229194 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:03.173925  5008 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:03.174135  4948 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:03.172784  4865 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.194665  4846 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:03.235970 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:03.245038 32374 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:23:03.236263 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:03.236464 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:03.234529 31792 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:03.239395 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:03.183651  4849 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:03.346297 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.365046 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 286595us
W20260709 12:23:03.368197 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.245405 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:03.450760 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 205445us
W20260709 12:23:03.301785 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60938.8us
W20260709 12:23:03.238813  5058 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 12:23:03.332801  5065 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:23:03.472139 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:03.487078 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 243943us
I20260709 12:23:03.485608 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:23:03.378852  5044 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.108s	user 0.000s	sys 0.000s
I20260709 12:23:03.488551 32374 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:03.236299  4948 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:03.492352 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 266075us
W20260709 12:23:03.327651  5061 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.213s	user 0.001s	sys 0.001s
I20260709 12:23:03.482952 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:03.235759  4873 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:03.331861  4797 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:03.523551 32373 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 15: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:03.522949 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:03.321550  4819 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:03.495288 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:03.494048 31792 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:03.454795  4570 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:03.450320  5016 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:03.472689  4801 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:03.546989 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 66900.5us
W20260709 12:23:03.551206 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 365874us
I20260709 12:23:03.551641 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:03.544665 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 49474.1us
I20260709 12:23:03.289006  4865 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.552402 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.553283 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:03.473470  4739 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:03.519328  4948 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:03.559567 31677 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:23:03.563246  5069 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.567031  5069 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:03.567524  5016 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:03.568149  5016 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:03.569303  5016 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:03.569813  5014 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:03.570397  5014 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:03.560992  4801 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.572614 31676 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:03.419348  5008 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.574215  5008 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:03.577585  5064 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.148s	user 0.000s	sys 0.000s
I20260709 12:23:03.579758  4801 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:03.580960 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:03.571368  5014 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:03.539286  4873 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.391449  5010 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:03.543131  4570 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:03.543103  4776 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:23:03.599567  5061 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.052s	user 0.000s	sys 0.001s
W20260709 12:23:03.613821 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 429579us
I20260709 12:23:03.593009  4797 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:03.560732  4739 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:03.563282  4819 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:23:03.632558 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.600073  4873 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:03.641036 32374 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:23:03.606009  5036 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:03.651719 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.667351 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:03.668346 31677 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:03.622745  4324 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:03.635403  4739 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:03.661204  5036 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.691432  5036 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:03.693941 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 113069us
I20260709 12:23:03.694985 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:03.659382  5074 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:03.707872 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.650985  5077 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.660135  4776 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:03.717362 31677 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:03.683064  4819 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:03.672338  4903 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.046s
I20260709 12:23:03.697201  4888 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:03.776867  5069 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:03.777431  4776 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:03.718127  5074 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:03.674898  5085 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:03.759711  5009 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:03.779124  5009 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 13 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:03.779881  4819 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:03.780074  5074 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:03.780213  5009 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.730922  4570 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.694993  5012 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:03.781906  5009 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:03.756336  4785 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:03.782562 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:03.782971  4785 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:03.783383 31749 raft_consensus.cc:2376] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17. Re-sending same reply.
I20260709 12:23:03.783752 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:03.784531 31752 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:03.783958  4785 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.784760 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:03.785399  4785 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:03.726697  5081 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 12:23:03.786948  5087 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.787114  5081 consensus_queue.cc:1243] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 16.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:03.781786  5078 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:03.787592  5087 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:03.718995 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 225024us
I20260709 12:23:03.719213 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:03.788902 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 236621us
I20260709 12:23:03.798171 31675 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:03.698487  5010 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:03.789973 31744 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:23:03.730926  4849 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:03.748032  5079 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:23:03.791194 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:03.798790 31717 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:03.800339 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.789467  4353 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.776s	user 0.008s	sys 0.004s
W20260709 12:23:03.806259 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 87116.4us
I20260709 12:23:03.717924  5077 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.812475 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.813717 31747 raft_consensus.cc:2376] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16. Re-sending same reply.
I20260709 12:23:03.815069 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.816462 31752 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
W20260709 12:23:03.818490 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 265271us
I20260709 12:23:03.728789  4865 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:03.763772  5037 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:03.824088 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:03.843967  5007 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.654s	user 0.005s	sys 0.010s
I20260709 12:23:03.843850  5037 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:03.848366 31753 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:03.835932  5072 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.849442  5072 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.849107  5037 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:03.851487  5037 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:03.852524  4865 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:03.823010 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:03.815984  4819 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:03.750172  5090 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:03.840176 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:03.852613 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 54972.4us
I20260709 12:23:03.857048 31821 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:03.836073 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:03.856842 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:03.864630  5091 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:03.865290  5091 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.867030  5007 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:03.867319 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 52342us
I20260709 12:23:03.867650  5007 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.868588 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.870101  5007 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:03.870287 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:03.858129 31744 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:03.874107 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:03.846652  4938 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.626s	user 0.004s	sys 0.008s
I20260709 12:23:03.875211 31819 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 15: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:03.874498 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:03.859148 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 164215us
I20260709 12:23:03.844041  5074 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:03.887566 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:03.887176 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 335601us
I20260709 12:23:03.851984  5072 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:03.819574  4776 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:03.893363  4776 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.893805  4776 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:03.891242 31642 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:03.853610  4865 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:03.896749 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 212821us
I20260709 12:23:03.896353 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:03.902419 31819 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:23:03.870929  4324 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:03.907773 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:03.843145  4846 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:23:03.905709 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:03.909375 31819 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
W20260709 12:23:03.911774 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 229835us
I20260709 12:23:03.888382  4949 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:03.915645 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:03.914548 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:03.890673  5074 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:03.889389  5016 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:03.879675  5092 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:03.898403  4896 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:03.918277 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:03.918967 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:03.921551 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:03.916778 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:03.927687 31824 raft_consensus.cc:2410] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 14 because replica has last-logged OpId of term: 13 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 10.
I20260709 12:23:03.908231  5007 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:03.924546 31823 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:23:03.911419  5072 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:23:03.934108 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 46687.7us
I20260709 12:23:03.934974 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.937769  4779 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:03.938372  4779 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.940620  4779 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:03.943692  4873 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.945051 31750 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:03.958779 31820 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 12 index: 9. (index mismatch)
I20260709 12:23:03.960426 31820 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:03.960938 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.962044  5091 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:03.962931 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.963771  4757 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:03.964447  4757 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:03.932956  5092 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:03.967283  5092 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.920109  4819 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:03.990808 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:03.946328  4873 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:04.025568 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 113002us
I20260709 12:23:03.981922 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.975174 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:04.026533 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 91632.3us
I20260709 12:23:03.932155  5007 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:03.958359  4888 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:04.035140 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53340.2us
I20260709 12:23:04.036047 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:04.036186 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 104010us
W20260709 12:23:04.038456 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.924703  5094 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:04.038832 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:03.935858  4896 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:03.982857  4324 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
W20260709 12:23:04.048228 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:03.970386  4739 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.927000  5016 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:03.979966  5100 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:04.043205  5072 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:03.980942  4757 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.060s
I20260709 12:23:04.058014 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.063175 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.069595  5036 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:04.071240 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.097692  5072 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:04.098286 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.099004 31675 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:04.098894  5072 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 13.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:04.100004 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.101135 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.071638 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.102818  4779 raft_consensus.cc:2764] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election decision vote started in defunct term 11: won
I20260709 12:23:04.072299 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.073025 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.104523 31671 raft_consensus.cc:2410] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 13 because replica has last-logged OpId of term: 11 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 6.
I20260709 12:23:04.104631 31676 raft_consensus.cc:2410] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 17 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:23:04.073067 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.040045  5007 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:04.055800 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:04.040853  5094 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.039916  5014 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:04.093607 31678 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:04.123974 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:04.127734 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:04.073393 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:04.105707 31675 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:04.114697 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:04.129243 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:04.095508  4873 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.130880  4873 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:04.130967  5099 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 12:23:04.122587 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:04.135399  5083 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:04.135620 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:04.135955  5083 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.136439  5083 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:04.136435 31747 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:23:04.137770 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:04.095839  5012 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.043s
I20260709 12:23:04.139362 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.140007 31670 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:04.121878  5077 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:04.123912 31671 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:23:04.138325 31747 raft_consensus.cc:2376] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18. Re-sending same reply.
I20260709 12:23:04.112996  4819 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.127108 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2068) took 1078 ms. Trace:
I20260709 12:23:04.128021 31678 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:04.127816 31672 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:04.122308 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:04.141963 31753 rpcz_store.cc:276] 0709 12:23:03.049071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:03.050472 (+  1401us) service_pool.cc:224] Handling call
0709 12:23:04.126926 (+1076454us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b54001a5ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:04.127036 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1759469824}
I20260709 12:23:04.134275  5099 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:04.147264 31749 raft_consensus.cc:2368] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:23:04.148944 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:04.149261 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:04.150038 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:04.140829  5008 raft_consensus.cc:2764] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:23:04.155328 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.155966  4819 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:04.157028  4949 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.012s
I20260709 12:23:04.158260 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:04.179207 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:04.180207 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:03.975803  4865 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:04.111567 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55846.9us
I20260709 12:23:04.134176  4873 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:04.076077  4948 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:04.175560 31676 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:04.192698  5085 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:04.204698 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 82481.8us
I20260709 12:23:04.205260  5077 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:04.205853  5077 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:04.205662  5004 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:04.208509  5077 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:04.211817  4739 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:04.198081  4857 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.213965  4857 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:04.216431  5083 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:03.979220  5074 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:04.195535 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:04.218822 31717 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:04.220130 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:04.162532  4570 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:04.221752 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.165234 31820 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:04.222728  4888 consensus_queue.cc:1175] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 18 status { last_received { term: 15 index: 10 } last_committed_idx: 10 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: [17.11-17.11]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:23:04.225659  5085 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:04.138350  4846 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:04.226408  4846 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:04.227524  4739 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:04.227556  4846 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:04.226449  5085 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:04.228853 31745 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:04.176389 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 140392us
I20260709 12:23:04.182042 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:04.142323  5094 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:04.230777 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48807.3us
I20260709 12:23:04.231173 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:04.216902  4857 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:04.237519 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:04.239069 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:04.192525  4324 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:04.241238 31746 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:23:04.240240 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:04.162993  5099 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:04.246109 31753 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:23:04.254933 31674 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:04.249550 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:04.258240 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:04.240957  4888 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
W20260709 12:23:04.261718 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:04.266187 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.271250 31752 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 11. Current term is 13. Ops: []
I20260709 12:23:04.247869  4739 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:04.281306 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:04.281998 31819 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 13
I20260709 12:23:04.277995 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:04.283104 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:04.272454 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:04.253571  4896 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:04.285079 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:04.285691 31826 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:04.280396  5096 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:04.286820 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:04.248646  4324 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:04.287760  4324 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:04.287904  5088 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.008s
I20260709 12:23:04.289112 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:04.290578  5019 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:04.250818 31747 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8725620974f7465eb0371f48761a1625"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:04.291671 31747 raft_consensus.cc:606] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:04.280972  4739 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:04.292431 32374 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:04.292991  4819 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:04.276880  5085 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:04.297986  5085 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.298386  5085 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:04.292397  4739 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:04.299301 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:04.257153  4873 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:04.261138  4859 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:04.306577  5096 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:04.309092 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:04.272189  4888 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:04.313937  5016 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:04.242700  5092 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.038s
I20260709 12:23:04.261395  5010 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260709 12:23:04.276445  5036 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:04.317359 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:04.321834 31819 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:23:04.301717  5088 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.268646  5009 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.054s
I20260709 12:23:04.337222 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:04.338012 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:04.305799  4570 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:04.341456  5008 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:04.347004  5008 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:04.347980  5008 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:04.330142  5083 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:04.349364  5008 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:04.342061  5081 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:04.350852  5081 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:04.351846  5081 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 12.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:04.323108  4888 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:04.266536  5084 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.068s
I20260709 12:23:04.344184 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:04.323637  4785 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:04.329196  4873 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:04.358484 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:04.369726 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.360466 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.396374 31671 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:04.363742 31749 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:04.373183 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:04.398005 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:04.399237 31676 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:04.400579 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:04.403040 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:04.400583 31751 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:04.404299 31751 raft_consensus.cc:606] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:04.404907 31751 raft_consensus.cc:614] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:04.406572 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:04.407755 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62094.9us
W20260709 12:23:04.412292 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.412901 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:04.416725 32374 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:04.420059 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 95463.2us
I20260709 12:23:04.394363  5087 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:04.338794  4776 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:04.388159  4785 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:23:04.426162 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 102043us
I20260709 12:23:04.406963  4938 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:04.430152 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:04.407140  5077 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:04.410476  5036 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:04.407362  5093 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:04.412411  4570 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:04.412169  5091 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:04.445410 31825 raft_consensus.cc:2368] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18.
I20260709 12:23:04.436507 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:04.358126  5088 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:04.414851  5083 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:04.439504 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:04.336223  5099 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:04.408052  4873 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:04.450114 32374 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:23:04.433099  4785 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:04.448457  5083 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:04.589905  5083 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:04.475047 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44983.2us
W20260709 12:23:04.508647 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 72221.8us
I20260709 12:23:04.596238 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:04.479924 31751 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:04.597862 31751 raft_consensus.cc:606] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:04.556879 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.599645 31751 raft_consensus.cc:614] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:04.440119  4903 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:04.451176 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 107046us
I20260709 12:23:04.609094 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:04.442289  4570 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.448627  5075 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:04.617878 31675 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:04.624012 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.623831 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:04.620599 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:04.621486 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:04.450680  5016 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:04.442075  4859 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.170s
I20260709 12:23:04.557912  4846 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:04.652993 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.456228  5077 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:04.658569 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:04.573226  4895 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.743717 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:04.684927 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.464800  4948 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:04.660602 31670 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:04.624835 31669 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:23:04.622714  5075 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:04.582620  4324 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:04.654034  5084 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.612728  4903 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:04.758244 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 162078us
I20260709 12:23:04.624387  4776 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:04.599009  4857 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.156s
I20260709 12:23:04.555755 31570 catalog_manager.cc:5697] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 8 to 11, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 11 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:04.763733  5075 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:04.500916  5094 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:04.787081 32376 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:23:04.788631  5094 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:04.791205 31677 raft_consensus.cc:2368] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16.
I20260709 12:23:04.793927 31821 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:04.794786  5077 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:04.797302  4776 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:04.800385  4873 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:04.803350 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:04.806257  5007 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.661s	user 0.012s	sys 0.000s
I20260709 12:23:04.653513  5004 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:04.752233  5116 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:23:04.554780  4888 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:04.559823  5088 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:04.782711  5084 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:04.782959  5016 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:04.783772  5019 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.785192  4903 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:04.794507  5094 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:04.624570  5092 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:04.808286 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.612396  5100 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.144s
I20260709 12:23:04.809775 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:04.618956  5090 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:04.830865 31671 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:04.813071  5014 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:04.817137  4324 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:04.818270  5117 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.300s	user 0.002s	sys 0.000s
W20260709 12:23:04.822655 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 419638us
I20260709 12:23:04.755769  4895 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:04.622973  4570 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:04.813234 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.824954  5004 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:04.835636 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 50485.1us
W20260709 12:23:04.838320 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:04.812194  5007 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:04.613637  4896 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.164s
I20260709 12:23:04.517967  5085 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:04.849700 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 229187us
I20260709 12:23:04.830469  5037 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:04.860971 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 57730.5us
I20260709 12:23:04.832942  5019 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:04.872973 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:04.873183 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:04.873775 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:04.891865 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:04.849021  5036 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:04.875129 31718 leader_election.cc:400] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18.
I20260709 12:23:04.898919 31718 leader_election.cc:403] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:23:04.830186  5094 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:04.847523 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:04.900976 31825 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:04.879698 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 70181.7us
I20260709 12:23:04.903976 31791 leader_election.cc:400] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:23:04.904491 31791 leader_election.cc:403] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:04.837226  5090 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:04.878803  5088 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:04.883971  4888 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:04.908144 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:04.873950  4776 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:04.874792  5091 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.858672  5092 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:04.882800  5008 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:04.890226 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:04.915238  5004 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:04.915285 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:04.916410  5075 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:04.918160 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:04.920843 32373 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:04.953491 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 63380us
I20260709 12:23:04.952135 31669 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:04.944715 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:04.954681 31676 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:04.955037 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:04.889456  5085 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:04.958151 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:04.929860 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:04.930917 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:04.935714 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:04.889632  4876 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.043s
I20260709 12:23:04.940047 31826 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:04.943310 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.925148  4324 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:04.883080  5010 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:04.896106  4949 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.951797 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:04.952675 31819 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:04.972648 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:04.892283  5007 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:04.910924  5088 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:04.911181  5094 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:04.911345  5090 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:04.960244 31750 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:04.964844 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:04.965655 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:04.894351  5019 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:04.942754  4865 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 12. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13.
I20260709 12:23:04.925343  5093 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:04.967687 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.999614 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:05.000268 31675 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:04.855093  4873 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 12:23:04.968108 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:04.950814 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:04.932310  4776 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:04.895582 31823 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:04.974957 31672 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260709 12:23:04.976152 31748 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:04.976766 31744 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 10.
I20260709 12:23:04.892043  4903 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.068s
I20260709 12:23:04.966477 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:04.982309 32373 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:04.966274  4888 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:05.010442 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.011932 31747 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:04.958467  5125 raft_consensus.cc:2764] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:23:05.014920 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:05.021189 31675 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:05.012306 31752 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:05.022982 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:05.023617 31750 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 16
W20260709 12:23:05.000785  4739 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.670s	user 0.011s	sys 0.003s
I20260709 12:23:04.981936  5088 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:04.998870  5123 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:04.952867  5091 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.018587 31825 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:23:05.010887  5100 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.034297 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:05.000049  5093 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:05.036511  4739 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.046875  5093 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:05.048681  4739 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.050417  4895 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:05.037096  4896 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.051227  4896 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:04.984319  4949 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.053649  4949 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.021960  4776 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:05.056537 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.057546 31748 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 19: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:05.046356  5014 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:05.059144  5121 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.179s	user 0.000s	sys 0.001s
I20260709 12:23:05.059286  5014 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:05.059911 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.061827  5014 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:05.019994  4888 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:05.070469  4888 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:05.070802 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.039124 31821 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:23:05.071938 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:05.072728 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.074622 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:05.075935 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.076400 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:05.076637 31818 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:23:05.077246 31824 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 15: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:05.037326  4865 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.077795  4865 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.078363  5036 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 11.9, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.080292  5037 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:05.058566 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.081807 31748 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:05.082959 32369 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 12 index: 10. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:05.081887 32373 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:05.069835 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.084313  5081 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:05.084929  5081 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.087306  5092 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.053717  4896 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:05.089025 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:05.092115  5081 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.046356  5100 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.093556  5081 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.094200  5081 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.095218  4865 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:05.095638  5100 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.097131  5081 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:05.082916 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.098321 31748 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:05.099466 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:04.970144  5008 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.019613  5102 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.106793 31676 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:05.101653 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:05.104470 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 204147us
I20260709 12:23:05.124109 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:05.120275 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 82111.5us
I20260709 12:23:05.126348 31675 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:05.131618 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.133860 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:05.101310  5087 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:05.020418  5010 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.132364 31752 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:05.294179 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.144186 31750 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
W20260709 12:23:05.159688 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.137503  5084 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.122502  5102 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.306085 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.308367 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.309211 31751 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:23:05.309438 31824 raft_consensus.cc:2410] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 10.
I20260709 12:23:05.121803  5014 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.121454  4949 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:05.131456  5037 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:05.271759 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:05.273463 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.321792 31744 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:23:05.274947 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:05.277616 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.132349  5016 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.115s
I20260709 12:23:05.286444 31671 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:05.323803 31753 raft_consensus.cc:2410] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:23:05.129158  5081 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:05.292642 31747 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:05.329722 31747 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:05.297190 31752 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:23:05.301752  5007 raft_consensus.cc:993] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:23:05.301977  5072 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.331673 31752 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 12. (term mismatch)
I20260709 12:23:05.332146 31752 pending_rounds.cc:85] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 11
I20260709 12:23:05.332465 31752 pending_rounds.cc:107] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 15.12
I20260709 12:23:05.332863 31752 raft_consensus.cc:2889] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:05.299407 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:05.300289 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:05.102281  5123 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:05.305297 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.145668  5087 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:05.309487 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 175699us
I20260709 12:23:05.265625  5091 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:05.303834 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.324611 31751 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:05.325675 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:05.296936 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:05.310487  4859 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:05.340546 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.347173 31750 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:05.341272  5125 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:05.343153 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:05.348482 31676 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:05.344341 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:05.334429 32369 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:05.354555 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:05.298025 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 150418us
I20260709 12:23:05.359164  5075 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.324508  5037 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.369438 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.355891 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:05.324770  4948 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:05.304234  5019 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:23:05.364989 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:05.343756  5087 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:05.392944 31751 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:05.322647  5014 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:05.361800  4785 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.773s	user 0.013s	sys 0.000s
W20260709 12:23:05.362947  5083 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.766s	user 0.004s	sys 0.008s
W20260709 12:23:05.401710 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 51458.2us
I20260709 12:23:05.334537  5036 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:05.378737  4846 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.621s	user 0.006s	sys 0.006s
I20260709 12:23:05.377707  5037 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:05.411854  5139 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:05.378307  4895 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
I20260709 12:23:05.411643 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.412405  4859 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:05.385927  5091 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.332178  5081 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.387729  4948 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.421554 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:05.366422 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:05.356931  5100 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.050s
I20260709 12:23:05.363528  5009 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
I20260709 12:23:05.345515  5072 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.437577 31676 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:05.603248  4739 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.603950  4739 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.605423 31676 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:05.606235 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.591676  4949 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:05.592526  5093 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.545s	user 0.007s	sys 0.005s
I20260709 12:23:05.607700  5093 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.545982 32376 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:05.581094 31677 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:05.547361 32373 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 15 index: 13. (index mismatch)
W20260709 12:23:05.495652  5119 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.979s	user 0.005s	sys 0.000s
I20260709 12:23:05.606462  4739 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.612870 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2784) took 1253 ms. Trace:
W20260709 12:23:05.606962 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.613402 31678 rpcz_store.cc:276] 0709 12:23:04.359630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.359791 (+   161us) service_pool.cc:224] Handling call
0709 12:23:04.401187 (+ 41396us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:04.434070 (+ 32883us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:04.434104 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:04.438951 (+  4847us) log.cc:844] Serialized 26 byte log entry
0709 12:23:04.516416 (+ 77465us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:04.516464 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:04.516541 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.470842 (+954301us) spinlock_profiling.cc:243] Waited 34.3 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:05.612582 (+141740us) raft_consensus.cc:1685] finished
0709 12:23:05.612604 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.612806 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4666,"raft.queue_time_us":12836,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":8502,"spinlock_wait_cycles":78798592,"thread_start_us":76985,"threads_started":1,"wal-append.queue_time_us":77382}
I20260709 12:23:05.613157  4739 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:05.615454  4739 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:05.497103  5135 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
W20260709 12:23:05.616787  4776 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.562s	user 0.009s	sys 0.001s
I20260709 12:23:05.617283  4776 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.618018  4739 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.611305  5093 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.619333 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 877) took 1003 ms. Trace:
I20260709 12:23:05.619755 31674 rpcz_store.cc:276] 0709 12:23:04.615467 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.615614 (+   147us) service_pool.cc:224] Handling call
0709 12:23:04.660013 (+ 44399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:04.972914 (+312901us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:04.972943 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:04.973424 (+   481us) log.cc:844] Serialized 26 byte log entry
0709 12:23:05.147861 (+174437us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:05.147903 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.147971 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.619054 (+471083us) raft_consensus.cc:1685] finished
0709 12:23:05.619079 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.619267 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":130147,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":3464,"thread_start_us":415248,"threads_started":2,"wal-append.queue_time_us":296570}
I20260709 12:23:05.621464  4776 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:05.618409 31825 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:05.625185  4888 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.554s	user 0.006s	sys 0.006s
W20260709 12:23:05.533238  5132 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.237s	user 0.002s	sys 0.000s
I20260709 12:23:05.524479 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:05.497881  5137 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
W20260709 12:23:05.523427  5128 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:23:05.481204  5019 consensus_queue.cc:1243] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 16.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:23:05.631137 31750 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:05.417696  5085 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:05.594748 31717 leader_election.cc:400] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16.
I20260709 12:23:05.607023 31747 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
W20260709 12:23:05.637522 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 271189us
I20260709 12:23:05.686460 31717 leader_election.cc:403] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:05.687343 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:05.643682 31751 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:05.643271  5091 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.403978  4785 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.471220  5142 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:05.522612  5008 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.521929  5141 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:05.647912  4948 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.651417  4949 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 10.8, Last appended by leader: 9, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.651760  5139 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:05.652881 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 231463us
I20260709 12:23:05.721328 31792 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:05.653402  5081 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:05.654125 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 300087us
W20260709 12:23:05.574702 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 220198us
I20260709 12:23:05.727852 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.674671 32369 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:05.728317 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:05.680909 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.728996 31824 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 15: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:05.611344  5138 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:23:05.694897 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.738992 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:05.696389 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 101847us
I20260709 12:23:05.740394 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:05.740586 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:05.697526  5019 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:05.686242  5085 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:05.482327  5140 raft_consensus.cc:1081] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Signalling peer 4d48c83c761f48079bc6ea176649dda0 to start an election
W20260709 12:23:05.681211  5090 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.665s	user 0.013s	sys 0.000s
I20260709 12:23:05.678618  5075 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:05.757959  5148 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:05.708314  5037 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.673763  4895 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:05.645329 31746 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:23:05.760373 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.761407 31746 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
W20260709 12:23:05.698911  4570 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.822s	user 0.012s	sys 0.000s
W20260709 12:23:05.699199  5088 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.580s	user 0.009s	sys 0.001s
I20260709 12:23:05.769768 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:05.702965  5144 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16.
I20260709 12:23:05.747098  4948 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:05.406522  5083 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.710009 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:05.791460 31826 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:05.683399  5010 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:05.629593  4739 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.727341 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:05.744232 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:05.745358 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:05.799069 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53802.7us
I20260709 12:23:05.745935 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:05.698830 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.801896 31819 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:05.736981 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:05.747536 31818 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:23:05.698272  5077 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:05.698599  5142 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:05.792562  5094 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.777s	user 0.013s	sys 0.000s
I20260709 12:23:05.755865 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:05.753414  5090 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.738421  4776 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:05.707187  5014 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:05.744899  5016 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.022s
I20260709 12:23:05.678337  4888 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.771857 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:05.750981  5092 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:05.611325  5036 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.776587 31744 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 13: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:05.773856  4570 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.778308  5147 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 12:23:05.779069  5124 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:05.779592  5152 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:05.824156  5144 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:05.817785  5142 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:05.763031  4895 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.784078  5084 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:05.785460  5102 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.785614 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:05.748085  4859 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:23:05.811765 31753 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: [12.9-12.9]
I20260709 12:23:05.798964 32378 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:05.863204 31676 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:05.752425 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:05.868775 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 116426us
W20260709 12:23:05.848289  4324 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.869s	user 0.013s	sys 0.000s
I20260709 12:23:05.816138 32374 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []
W20260709 12:23:05.819906 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 74076.2us
I20260709 12:23:05.869551 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:05.875197 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74437.7us
I20260709 12:23:05.820509  5100 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:05.760272  5081 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
W20260709 12:23:05.824954 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 69163.6us
I20260709 12:23:05.821396 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 798) took 1201 ms. Trace:
I20260709 12:23:05.855057  5144 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.730702  5141 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.814210  5096 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:05.780037  5145 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.024s
W20260709 12:23:05.847692 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 44749.5us
I20260709 12:23:05.849404 31675 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:05.814384  5085 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.787933  5088 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.816730  5072 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.863430  5152 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:05.827109  4776 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.764362  5125 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:05.799136  4948 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.813686  5148 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:23:05.877820 31672 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:05.886519  4785 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:05.808228  4739 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.879374 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:05.826874  5094 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.817167  5077 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.800931  5037 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.870855  5084 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:05.881978  5102 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
I20260709 12:23:05.862416  4895 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:05.888676 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:05.895002 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.961125 31818 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:05.896786 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.898524 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:05.962074 31823 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:23:05.900197 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.900497 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:05.902072 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 165183us
I20260709 12:23:05.902886 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.877241  4324 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18.
I20260709 12:23:05.934796  5161 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.970448 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.971191 31823 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:23:05.972112 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.883559 31749 rpcz_store.cc:276] 0709 12:23:04.619506 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.620655 (+  1149us) service_pool.cc:224] Handling call
0709 12:23:04.621078 (+   423us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:04.837199 (+216121us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:04.837235 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:04.879918 (+ 42683us) log.cc:844] Serialized 52 byte log entry
0709 12:23:05.040830 (+160912us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:05.134166 (+ 93336us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.279852 (+145686us) spinlock_profiling.cc:243] Waited 2.9 ms on lock 0x7b18004ae600. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:23:05.280153 (+   301us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.395741 (+115588us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.395830 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:05.416666 (+ 20836us) spinlock_profiling.cc:243] Waited 3.99 ms on lock 0x7b18004ae600. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:05.709815 (+293149us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b18004ae600. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:05.821084 (+111269us) raft_consensus.cc:1685] finished
0709 12:23:05.821108 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:05.821305 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":211766,"prepare.queue_time_us":378213,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":92,"raft.queue_time_us":256467,"raft.run_cpu_time_us":410,"raft.run_wall_time_us":21397,"spinlock_wait_cycles":24279424,"thread_start_us":197821,"threads_started":2,"wal-append.queue_time_us":222102}
I20260709 12:23:05.973655 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:05.880491  5100 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:05.883432  5010 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:23:05.878247 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.939266 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:05.938318 31746 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:23:05.939982 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:05.890347  5144 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:05.945376 31745 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:23:05.945740 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 66536.6us
I20260709 12:23:05.947283 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:05.981632 32377 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:23:05.950763  5092 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:05.951725  5096 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:05.884685 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:05.937434  4776 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:05.780697  5146 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:05.999570  5146 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.001259  5008 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.001848  5008 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.004062  5090 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:06.008044  5074 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:06.009609  5088 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:05.967567 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 123148us
I20260709 12:23:05.970156 31670 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:05.969734 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:05.884694  5014 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:05.958818  5081 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:23:06.017108 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47446.3us
W20260709 12:23:06.018071 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:05.983793 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:06.020231 31749 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:05.985872 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 101312us
I20260709 12:23:05.977083  5161 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:05.988193  5159 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.084s	user 0.000s	sys 0.003s
I20260709 12:23:05.980738  5100 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:05.991748  5151 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:05.993993  5141 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:05.995955  5152 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.997057  5037 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:05.997560  5077 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:05.998188  5125 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.123068  5075 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.142584  5075 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:05.998910  5083 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:05.966938  5091 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.022567 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:05.991031  5155 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:06.149928  5155 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.150390  5155 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:06.036826 31748 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:05.981070  5139 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.011955  5010 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:06.068715  5087 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.598s	user 0.011s	sys 0.001s
I20260709 12:23:06.080638  5092 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:06.095296  5158 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.122s	user 0.003s	sys 0.000s
I20260709 12:23:06.097817 31674 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:23:06.099892 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2791) took 1193 ms. Trace:
I20260709 12:23:06.103065 31669 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:06.104068 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 966) took 1508 ms. Trace:
I20260709 12:23:06.153187 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:06.105054 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1940) took 1014 ms. Trace:
I20260709 12:23:06.162814 31676 rpcz_store.cc:276] 0709 12:23:05.090908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.606432 (+515524us) service_pool.cc:224] Handling call
0709 12:23:05.673787 (+ 67355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:05.868484 (+194697us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.104821 (+236337us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b540017c5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:06.104990 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":434959744}
I20260709 12:23:06.164273 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.164953 31676 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:06.166134 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:06.167188 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 144702us
I20260709 12:23:06.167495 31673 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:06.168094 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:06.169052  4876 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:06.169495  4876 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.169550 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:06.170008  4876 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:06.171006 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:06.162173 32374 rpcz_store.cc:276] 0709 12:23:04.595704 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.598721 (+  3017us) service_pool.cc:224] Handling call
0709 12:23:04.623410 (+ 24689us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:05.872543 (+1249133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.103791 (+231248us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54004bb254. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:06.103978 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2742591872}
I20260709 12:23:06.173090  5134 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:06.173635  5134 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.174006  5134 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:06.174968  5083 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:06.175467  5083 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:06.175871  5083 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:06.176754  4739 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:06.177603  5091 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.178200  5091 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.154325  5010 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.181022  5091 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.181597 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:06.154814  5087 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.185458  5087 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:06.186491 31671 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:23:06.186941 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2683) took 1229 ms. Trace:
I20260709 12:23:06.187569 31745 rpcz_store.cc:276] 0709 12:23:04.957308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.957482 (+   174us) service_pool.cc:224] Handling call
0709 12:23:06.186684 (+1229202us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x7b54002945d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:06.186868 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2226115840}
I20260709 12:23:06.189019 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:06.189863 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.107403  5099 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:06.107048  5161 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:06.107918  5084 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.107030  4895 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.108569 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2693) took 1129 ms. Trace:
I20260709 12:23:06.109205 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2092) took 1127 ms. Trace:
I20260709 12:23:06.110301  4888 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.034950  5014 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.111007  4570 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.016517  5081 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:06.033915  5093 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:06.119980 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:06.121454  5072 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.129839  5168 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.007s
I20260709 12:23:06.107329  5096 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.190335 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:06.219679 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 99807.2us
I20260709 12:23:06.138984  5151 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:06.079217  5036 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:06.121658  5090 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.258590  5090 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.261062  5090 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.076701  5146 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:06.070106  5162 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.111s
I20260709 12:23:06.145288  5037 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:06.148568  5142 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.263376  5037 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 14.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.264665  5142 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:06.267109  5084 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.267866  5084 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.270972  5084 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.271133  5091 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:06.271811  5090 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:06.271888  5091 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.191035 31751 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:23:06.161230 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2094) took 1147 ms. Trace:
I20260709 12:23:06.220809 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:06.190399 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:06.220819 31753 rpcz_store.cc:276] 0709 12:23:04.979357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.333224 (+353867us) service_pool.cc:224] Handling call
0709 12:23:05.357696 (+ 24472us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:05.867625 (+509929us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.108294 (+240669us) spinlock_profiling.cc:243] Waited 454 ms on lock 0x7b54002963d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:06.108507 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1043531008}
I20260709 12:23:06.161906 31670 rpcz_store.cc:276] 0709 12:23:04.906925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.907319 (+   394us) service_pool.cc:224] Handling call
0709 12:23:06.099611 (+1192292us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b5400253ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:06.099813 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2322125184}
I20260709 12:23:06.174396  5008 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.286100 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1942) took 1194 ms. Trace:
I20260709 12:23:06.137949  5100 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.218803 31750 rpcz_store.cc:276] 0709 12:23:04.982271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.683242 (+700971us) service_pool.cc:224] Handling call
0709 12:23:05.702178 (+ 18936us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:05.739217 (+ 37039us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.739253 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:05.739859 (+   606us) log.cc:844] Serialized 52 byte log entry
0709 12:23:05.842142 (+102283us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:05.842187 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.842259 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.108962 (+266703us) raft_consensus.cc:1685] finished
0709 12:23:06.108985 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.109144 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7339,"raft.queue_time_us":235,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":9920,"thread_start_us":42202,"threads_started":1,"wal-append.queue_time_us":138530}
I20260709 12:23:06.191337 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:06.222191 31749 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:06.222913 32373 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:06.221096 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:06.209944 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1944) took 1082 ms. Trace:
I20260709 12:23:06.229153  5014 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:06.274986 31747 rpcz_store.cc:276] 0709 12:23:05.013778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.686992 (+673214us) service_pool.cc:224] Handling call
0709 12:23:05.687371 (+   379us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:05.687798 (+   427us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:05.687829 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:05.722045 (+ 34216us) log.cc:844] Serialized 52 byte log entry
0709 12:23:05.821395 (+ 99350us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:05.848467 (+ 27072us) log.cc:844] Serialized 14 byte log entry
0709 12:23:05.848820 (+   353us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:05.848897 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:06.094568 (+245671us) spinlock_profiling.cc:243] Waited 8.8 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:06.160838 (+ 66270us) raft_consensus.cc:1685] finished
0709 12:23:06.160969 (+   131us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.161143 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":26789,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":3598,"raft.queue_time_us":200,"raft.run_cpu_time_us":363,"raft.run_wall_time_us":17204,"spinlock_wait_cycles":20244224,"thread_start_us":31656,"threads_started":1,"wal-append.queue_time_us":123821}
I20260709 12:23:06.306731 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:06.307801 31747 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:06.271176  5094 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:06.291129 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 70405.8us
I20260709 12:23:06.310381 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:06.222582  5166 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Leader pre-election won for term 11
W20260709 12:23:06.298679 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 108385us
I20260709 12:23:06.314514 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:06.298938 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 107712us
I20260709 12:23:06.315845 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:06.297487 31674 rpcz_store.cc:276] 0709 12:23:05.127433 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.621431 (+493998us) service_pool.cc:224] Handling call
0709 12:23:06.097105 (+475674us) spinlock_profiling.cc:243] Waited 5.56 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:23:06.097402 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.161520 (+ 64118us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.209835 (+ 48315us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12798848}
I20260709 12:23:06.192788  5085 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:06.209213  4873 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.193284  5102 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.193480  5139 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.272825  5090 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.157687  4948 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:06.343057 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:06.222946  5081 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:06.162019  5141 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.312502 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:06.353144 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:06.224752  4895 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.366945  5141 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.368674 31745 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:06.366974 32375 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:23:06.325580  5036 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:06.242079  5075 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.223868  5093 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:06.287901 31677 rpcz_store.cc:276] 0709 12:23:05.091633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.610085 (+518452us) service_pool.cc:224] Handling call
0709 12:23:06.236520 (+626435us) spinlock_profiling.cc:243] Waited 5.62 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:23:06.236828 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.237242 (+   414us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:06.237269 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:06.285727 (+ 48458us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:06.285785 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.285861 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:06.286034 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":815,"spinlock_wait_cycles":12918528}
I20260709 12:23:06.298043  5074 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:06.324152 31676 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:06.327719 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:06.385699 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58046us
I20260709 12:23:06.386444 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:06.305889  5161 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:06.331283 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:06.387490 31753 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:06.332023 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.335165  5139 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.301146  4776 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.342362  5100 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.310070  4570 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:06.306303  4876 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.343039 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.325448  5014 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.056s
I20260709 12:23:06.400151 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:06.316740  5166 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:06.346446  5091 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:06.335631  5127 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:06.245878  5072 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.356305 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 807) took 1082 ms. Trace:
I20260709 12:23:06.356742  4739 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:06.358060 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:06.358474 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1934) took 1380 ms. Trace:
I20260709 12:23:06.412168 31751 rpcz_store.cc:276] 0709 12:23:05.274490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.709182 (+434692us) service_pool.cc:224] Handling call
0709 12:23:06.190740 (+481558us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:06.283121 (+ 92381us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.356004 (+ 72883us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b54002945d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:06.356218 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1106013952}
I20260709 12:23:06.412884 31672 rpcz_store.cc:276] 0709 12:23:04.978573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.031399 (+ 52826us) service_pool.cc:224] Handling call
0709 12:23:06.358263 (+1326864us) spinlock_profiling.cc:243] Waited 830 ms on lock 0x7b54002b0554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:06.358409 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1909227776}
I20260709 12:23:06.358709  5151 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.359629  5147 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:06.360416  5146 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:06.380030 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:06.392182  4776 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:06.383678 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:06.317036  4846 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:06.388495 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:06.392390  5139 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.407426  5166 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:06.346052  5090 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:06.354081  5009 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:06.432031 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.370606  4785 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:06.402585 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:06.403447 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.331140  4873 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.385128  5093 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:06.385581  5168 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.332304  5094 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.055s
I20260709 12:23:06.331786  5102 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.353868  5084 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
I20260709 12:23:06.417766 32374 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:23:06.397292  5075 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:06.383634 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:06.564853 31677 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 17: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:06.562412 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:06.565783 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.566969 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.567560 31677 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:06.568312 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:06.569293 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.397535  5010 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:06.425541 31745 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:06.425611 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:06.425439 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:06.576611 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 151081us
I20260709 12:23:06.427610 31671 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:06.409552  5127 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.387084  5074 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.390079  5092 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:23:06.432687 31670 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:23:06.434623 31747 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:06.421680  5146 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.436931  5167 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:06.438464 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 886) took 1170 ms. Trace:
I20260709 12:23:06.438767  5169 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:06.439497 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:06.440515 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2782) took 2290 ms. Trace:
I20260709 12:23:06.440749  5072 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.442221 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:06.426461  4846 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.430691  5009 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:06.421651  4739 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.454072  5123 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.462757 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:06.477169  5149 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.850s	user 0.003s	sys 0.003s
I20260709 12:23:06.427486  5160 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 12:23:06.542367 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:06.575006  5010 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.545621 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:06.551088 31678 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:06.649355 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1943) took 1557 ms. Trace:
I20260709 12:23:06.649720 31678 rpcz_store.cc:276] 0709 12:23:05.091926 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.615510 (+523584us) service_pool.cc:224] Handling call
0709 12:23:06.465356 (+849846us) spinlock_profiling.cc:243] Waited 5.59 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:23:06.550911 (+ 85555us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.649100 (+ 98189us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.649297 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":12858112}
I20260709 12:23:06.551753 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2093) took 1539 ms. Trace:
I20260709 12:23:06.650761 31748 rpcz_store.cc:276] 0709 12:23:05.012867 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.099092 (+ 86225us) service_pool.cc:224] Handling call
0709 12:23:06.465065 (+1365973us) spinlock_profiling.cc:243] Waited 876 ms on lock 0x7b5400295ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:06.551689 (+ 86624us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2014575488}
I20260709 12:23:06.431274  5139 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:06.434408  4785 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:06.560575 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1933) took 1591 ms. Trace:
I20260709 12:23:06.452672  5126 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:05.841836 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:06.571568 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:06.576663 31751 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:23:06.394148 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.737962 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:06.442896  5096 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:06.579617 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:06.559754  5148 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.011s	sys 0.008s
I20260709 12:23:06.738831 31818 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:23:06.586031 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:06.639737  4846 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:06.588670 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:06.548906  5094 consensus_queue.cc:1243] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 13.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:23:06.434157  4895 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.125s
I20260709 12:23:06.591444 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:06.800266 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:06.596832 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:06.593852 31675 rpcz_store.cc:276] 0709 12:23:05.268455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.676887 (+408432us) service_pool.cc:224] Handling call
0709 12:23:05.677241 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:05.895635 (+218394us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.438199 (+542564us) spinlock_profiling.cc:243] Waited 172 ms on lock 0x7b54001d6354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:06.438397 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":395446016}
I20260709 12:23:06.616320 31752 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:06.600104 31673 rpcz_store.cc:276] 0709 12:23:04.150477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:04.153162 (+  2685us) service_pool.cc:224] Handling call
0709 12:23:04.153672 (+   510us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.167903 (+2014231us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.440252 (+272349us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b5400391cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:06.440424 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4630478720}
I20260709 12:23:06.617727 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:06.470511  5168 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.627437 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.561789  5100 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:06.584327  5091 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:06.584066  5127 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:06.635432  5174 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.072s
I20260709 12:23:06.637599  5161 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:23:06.635473 31574 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42072 (request call id 242) took 22452 ms (client timeout 15000 ms). Trace:
I20260709 12:23:06.643041 31746 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:06.575263  5014 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.408740  5144 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.646322 31749 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:06.882123 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:06.882808 31749 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:23:06.883839 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:06.884418 31749 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:06.544711 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2100) took 1205 ms. Trace:
I20260709 12:23:06.593608  5167 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:06.899886 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:06.584475  5075 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:06.593322  5146 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:06.654383 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:06.732807 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:06.602620  5141 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.738112 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:06.739846 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:06.907188 31823 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:06.734023 31669 rpcz_store.cc:276] 0709 12:23:04.969497 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.039558 (+ 70061us) service_pool.cc:224] Handling call
0709 12:23:06.102850 (+1063292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:06.170389 (+ 67539us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.464618 (+294229us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400253ad0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:06.560511 (+ 95893us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2444546304}
I20260709 12:23:06.784657 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.908277 31747 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:06.592679 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:06.642323  4865 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.738144 31674 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:23:06.810971 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:06.812088 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.813902 31671 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
W20260709 12:23:06.816833 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 271252us
I20260709 12:23:06.919716 31749 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:06.817816 31753 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:06.654392  5010 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:06.884294  5083 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.708s	user 0.010s	sys 0.001s
I20260709 12:23:06.861397  4776 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:06.929327 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.854983 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:06.791674  5148 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.825s	user 0.012s	sys 0.008s
I20260709 12:23:06.731838  5133 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:06.938802 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:06.939594 31747 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:06.590683 31570 catalog_manager.cc:5697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:06.865125 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:06.861207  5100 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:23:06.867468  5182 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.289s	user 0.000s	sys 0.001s
W20260709 12:23:06.867612  5181 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:23:06.805857  5096 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:06.799410 31818 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8725620974f7465eb0371f48761a1625"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:41204
I20260709 12:23:06.959842 31818 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:06.960297 31818 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:06.868165  4873 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.868654  5077 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:06.869045  4948 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:06.870569  5168 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:06.873505 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:06.938803  5133 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:06.855633 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 284182us
I20260709 12:23:06.979050 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:06.936257  5126 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 12:23:06.886906 31744 rpcz_store.cc:276] 0709 12:23:05.338862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.344113 (+  5251us) service_pool.cc:224] Handling call
0709 12:23:06.458325 (+1114212us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54001a4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:06.544658 (+ 86333us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":933960576}
I20260709 12:23:06.984880 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:06.818514  5099 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.596s	user 0.008s	sys 0.009s
I20260709 12:23:06.898288  5014 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:06.906245 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 46833.3us
I20260709 12:23:06.909811 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:06.993444 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:06.995503 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:06.996166 31744 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:23:06.997133 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 808) took 1722 ms. Trace:
I20260709 12:23:06.997565 31744 rpcz_store.cc:276] 0709 12:23:05.274966 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.995354 (+1720388us) service_pool.cc:224] Handling call
0709 12:23:06.995832 (+   478us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:06.996772 (+   940us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:06.997057 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.965631  5160 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:06.865331  5179 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.450s	user 0.000s	sys 0.001s
I20260709 12:23:06.908560 31672 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:23:06.922559 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:06.923992 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.009754 31669 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
W20260709 12:23:06.821908 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 242372us
I20260709 12:23:07.007393  5008 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:07.013525  5008 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:07.012981 31570 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 8 to 11, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 11 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:07.014248  5036 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:07.014927  5036 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.015811 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:07.016408 31752 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:07.017237  5036 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:06.918098 31670 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:07.019627 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.020831 31670 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 19: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:06.928035 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:06.821461  5169 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:06.820693 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:06.931032 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:07.026167 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 95221.8us
I20260709 12:23:07.027084 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:06.944602 31674 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:06.821000  5139 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:07.028683 31827 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 14: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:07.029400 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:06.946201 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:06.881644  5162 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:06.903046  5167 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:06.867859  5155 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.717s	user 0.008s	sys 0.002s
I20260709 12:23:06.929790  5150 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:06.966292 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:06.936627  5123 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.932260  5183 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.028s
I20260709 12:23:06.854960 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:07.067193 31751 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:07.028939  5090 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:06.947158  4876 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.880345 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:07.098726 31826 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:07.098829 31748 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:23:07.099764 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2095) took 1762 ms. Trace:
I20260709 12:23:07.100237 31748 rpcz_store.cc:276] 0709 12:23:05.337462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.062154 (+1724692us) service_pool.cc:224] Handling call
0709 12:23:07.098575 (+ 36421us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.099399 (+   824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.099679 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:06.874641 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:06.898790  5009 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:06.969271  5085 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.644s	user 0.011s	sys 0.002s
I20260709 12:23:06.933040  5083 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.912813 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:06.981663  5093 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.534s	user 0.007s	sys 0.004s
I20260709 12:23:07.111093  5014 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:07.111209  5093 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.114010  5093 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:06.939592  4776 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.965881  5077 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:07.114984  5077 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:07.116732  4776 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:07.014181 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:06.945338 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.029402 31822 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:07.118891 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 104781us
I20260709 12:23:07.119694 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:07.120551 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 118675us
W20260709 12:23:07.045717  5134 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.871s	user 0.012s	sys 0.002s
I20260709 12:23:07.045625  5162 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:06.950103  4785 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.048363 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:06.991426  5125 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:06.961584  5184 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 5, Time since last communication: 0.053s
I20260709 12:23:07.050150 31749 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:23:07.032474 31818 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.052209 31752 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:07.128706 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2696) took 1792 ms. Trace:
I20260709 12:23:07.129110 31749 rpcz_store.cc:276] 0709 12:23:05.336307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.049249 (+1712942us) service_pool.cc:224] Handling call
0709 12:23:07.049678 (+   429us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.128409 (+ 78731us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.128647 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.129917 31752 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:23:07.130343 31749 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 13. Current term is 14. Ops: [13.7-13.7]
I20260709 12:23:07.055994 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:07.005267  5098 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.120026 31792 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:07.063730 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:07.029091  5094 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:07.144994  5141 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:07.145766  5141 consensus_queue.cc:1243] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:06.975804 31820 raft_consensus.cc:2410] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
W20260709 12:23:07.117044  5166 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.683s	user 0.008s	sys 0.002s
I20260709 12:23:07.050958  5074 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:07.147439  5166 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.148114  5074 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:07.148525  5074 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.148890  5074 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:07.149953  5166 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:06.862287 31574 rpcz_store.cc:269] 0709 12:22:44.183449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:44.183719 (+   270us) service_pool.cc:224] Handling call
0709 12:22:44.747034 (+563315us) spinlock_profiling.cc:243] Waited 25 ms on lock 0x7b54002828d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3f090 00007fe8fcdddf3a 00007fe902b93e9c 00007fe902c9204e 00007fe9011532a4 00007fe901153232 00007fe901153161 00007fe9011530dc 00007fe901152392 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:06.635153 (+21888119us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:22:59.924024 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:00.335810 (+411786us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:00.347545 (+ 11735us) write_op.cc:435] Acquired schema lock
0709 12:23:00.356365 (+  8820us) tablet.cc:662] Decoding operations
0709 12:23:00.634749 (+278384us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:00.634812 (+    63us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:00.634844 (+    32us) tablet.cc:685] Acquiring locks for 15 operations
0709 12:23:00.636310 (+  1466us) tablet.cc:701] Row locks acquired
0709 12:23:00.636332 (+    22us) write_op.cc:260] PREPARE: finished
0709 12:23:00.636502 (+   170us) write_op.cc:270] Start()
0709 12:23:00.636634 (+   132us) write_op.cc:276] Timestamp: P: 1783599780636442 usec, L: 0
0709 12:23:00.636668 (+    34us) op_driver.cc:348] REPLICATION: starting
0709 12:23:00.637455 (+   787us) log.cc:844] Serialized 5994 byte log entry
0709 12:23:01.783236 (+1145781us) op_driver.cc:464] REPLICATION: finished
0709 12:23:01.934219 (+150983us) write_op.cc:301] APPLY: starting
0709 12:23:01.934406 (+   187us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:02.212554 (+278148us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:02.212729 (+   175us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:02.333978 (+121249us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:02.334439 (+   461us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=30,key_file_lookups=30,delta_file_lookups=0,mrs_lookups=0
0709 12:23:02.334517 (+    78us) write_op.cc:312] APPLY: finished
0709 12:23:02.445959 (+111442us) log.cc:844] Serialized 139 byte log entry
0709 12:23:02.446218 (+   259us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:02.446667 (+   449us) write_op.cc:454] Released schema lock
0709 12:23:02.446749 (+    82us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:02.447356 (+   607us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:02.447393 (+    37us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2814756,"reported_tablets":50,"spinlock_wait_cycles":57549056,"child_traces":[["op",{"apply.queue_time_us":150491,"apply.run_cpu_time_us":29997,"apply.run_wall_time_us":513221,"cfile_cache_hit":62,"cfile_cache_hit_bytes":66797,"num_ops":15,"prepare.queue_time_us":799177,"prepare.run_cpu_time_us":23573,"prepare.run_wall_time_us":1156257,"raft.queue_time_us":283563,"raft.run_cpu_time_us":9797,"raft.run_wall_time_us":150465,"replication_time_us":1146354,"thread_start_us":1278050,"threads_started":4,"wal-append.queue_time_us":584170,"wal-append.run_cpu_time_us":35824,"wal-append.run_wall_time_us":2203573}]]}
I20260709 12:23:07.070868 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:07.150946  5098 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.152530 32376 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 16 index: 14. (index mismatch)
I20260709 12:23:07.071574 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.072155  5144 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:07.072892  5011 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:07.080077  5004 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:07.086724  5123 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.088160  4859 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:07.011068  5075 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.091759  5096 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:07.030992  5169 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:07.092374  5100 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:07.156620  5084 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:07.156870  4324 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:06.928489  4865 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.102514 31824 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:23:07.047751  5008 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:07.109128  5009 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:07.108960  5139 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.108697 31570 catalog_manager.cc:5697] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 5 to 12, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 12 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:07.156231  5125 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:07.110535  4570 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.707s	user 0.011s	sys 0.000s
I20260709 12:23:07.026125 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2695) took 1694 ms. Trace:
I20260709 12:23:07.205050 31744 rpcz_store.cc:276] 0709 12:23:05.335113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.025310 (+1690197us) service_pool.cc:224] Handling call
0709 12:23:07.025631 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.025819 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.025841 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:07.025881 (+    40us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:07.025933 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.025988 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.026075 (+    87us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.122931 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2097) took 1785 ms. Trace:
W20260709 12:23:07.122979 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 127591us
I20260709 12:23:07.179967  5100 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.176371  5084 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:07.130610 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2098) took 1792 ms. Trace:
I20260709 12:23:07.130987 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2099) took 1792 ms. Trace:
I20260709 12:23:07.211095 31749 rpcz_store.cc:276] 0709 12:23:05.338568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.129843 (+1791275us) service_pool.cc:224] Handling call
0709 12:23:07.130141 (+   298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.130739 (+   598us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.130936 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:07.131060 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 75165us
I20260709 12:23:07.131603 31717 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:07.214913 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 83413.2us
I20260709 12:23:07.210093 31752 rpcz_store.cc:276] 0709 12:23:05.338312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.129340 (+1791028us) service_pool.cc:224] Handling call
0709 12:23:07.129694 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.130351 (+   657us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.130559 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.224593 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2701) took 1476 ms. Trace:
I20260709 12:23:07.131846  5134 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.143096  5083 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:07.143271 31746 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:07.144316  5162 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:07.107378  5081 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.736s	user 0.011s	sys 0.000s
I20260709 12:23:07.154289 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:07.155565 31792 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:07.098083  4739 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:07.133507 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 85230.3us
I20260709 12:23:07.135095 31818 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:07.237028 31744 rpcz_store.cc:276] 0709 12:23:05.748411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.223179 (+1474768us) service_pool.cc:224] Handling call
0709 12:23:07.223731 (+   552us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.223980 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.224034 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:07.224109 (+    75us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:07.224195 (+    86us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.224290 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.224498 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.221212 31748 rpcz_store.cc:276] 0709 12:23:05.338010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.101332 (+1763322us) service_pool.cc:224] Handling call
0709 12:23:07.122396 (+ 21064us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.122580 (+   184us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.122604 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:07.122669 (+    65us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:07.122710 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.122764 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.122874 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.238574 31744 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:07.132601 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:07.132673 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:07.239888 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 107367us
I20260709 12:23:07.239920 31822 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:07.133682  5093 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:07.240494 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:07.161705 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 237697us
I20260709 12:23:07.163825 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:07.236060 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 80622.4us
I20260709 12:23:07.129706  5085 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:07.063970  5155 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.155014  5090 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:07.064800  5133 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.253382 32369 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 12:23:07.096906 31823 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:23:07.246011  5085 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.254557  5085 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:07.254962 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:07.256294 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:07.256042  4896 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:07.257086 31823 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:07.257279  4896 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:07.257949 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:07.178411 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 149061us
I20260709 12:23:07.258966 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:07.259241 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:07.249356  5184 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:07.159772  5014 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:07.260268  5014 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:07.260035  5184 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.158012  5011 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:07.261056 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:07.261878  5185 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:07.261694  5011 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.267747 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:07.027282  5160 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.170595  5166 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:07.220618 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:07.270386  5177 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.706s	user 0.000s	sys 0.003s
I20260709 12:23:07.226572 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:07.315510  5190 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:07.316110  5190 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:07.317303  5160 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:07.317080  5190 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.318819 31827 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:07.166534  4846 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.319597  5125 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:07.242847  5096 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:07.136274  5010 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.203936  4324 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:07.204595  5008 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:07.207784  4570 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:07.245406 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 809) took 1215 ms. Trace:
I20260709 12:23:07.339618 31744 rpcz_store.cc:276] 0709 12:23:06.029549 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.238065 (+1208516us) service_pool.cc:224] Handling call
0709 12:23:07.238394 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.245095 (+  6701us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.245340 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.242506 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:07.248337 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:07.222397  5084 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:07.226802  5144 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:07.159433  5004 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:07.264779 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:07.269863  4873 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.262925  5148 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:07.262362  5185 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:07.269349  4948 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:07.161643  4859 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:07.270543  5151 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.267135 32371 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 13 index: 10. (index mismatch)
I20260709 12:23:07.282498 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:07.249763  5090 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:07.264905  5099 consensus_queue.cc:1175] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "039d66a6c85e43c3936bf5017501b344" responder_term: 19 status { last_received { term: 16 index: 13 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: [18.14-18.14]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:23:07.263919  4739 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:07.279656 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:07.263270 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:07.274034 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:07.290748 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:07.276887 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:07.291253  5164 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:07.293095  5134 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:07.294588 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.250500  5169 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.251976  5161 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:07.264406 31677 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:07.229931  5100 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.243980  5162 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:07.296334  5188 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:07.241160  5081 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.296758  5146 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.278221  5123 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:07.271561  5102 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:07.276503  5072 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:07.531643 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:07.533068 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:07.300352 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 972) took 1419 ms. Trace:
I20260709 12:23:07.300577  5074 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.280016  5194 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:07.301585 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:07.534170 31750 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:07.302947  5014 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.303835  4865 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:07.306277 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.539799 31671 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:23:07.273842  5136 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:07.272343  5094 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.268417  5098 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:07.270390 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:07.296202 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:07.303844 31746 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
W20260709 12:23:07.313436 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 92982us
I20260709 12:23:07.192766  5141 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:07.313616 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 810) took 1283 ms. Trace:
I20260709 12:23:07.564785 31748 rpcz_store.cc:276] 0709 12:23:06.029885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.238762 (+1208877us) service_pool.cc:224] Handling call
0709 12:23:07.313118 (+ 74356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.313300 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.313324 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:07.313364 (+    40us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:07.313391 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.313445 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.313559 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.228714 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2102) took 1352 ms. Trace:
I20260709 12:23:07.566473 31749 rpcz_store.cc:276] 0709 12:23:05.876492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.227391 (+1350899us) service_pool.cc:224] Handling call
0709 12:23:07.227851 (+   460us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.228129 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.228173 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:07.228245 (+    72us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:07.228313 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.228400 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.228617 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.568444 31749 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:07.232508 31752 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:07.249564  5187 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:07.322221 31821 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:07.247879  5093 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:07.321947 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 112339us
I20260709 12:23:07.580904 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:07.320787  4846 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:07.343135 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 100708us
I20260709 12:23:07.342522 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:07.402284  5169 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:07.361128 31818 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:07.352708 31673 raft_consensus.cc:2410] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 17 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:23:07.361920 31676 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 17: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:07.362607 32374 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 15: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:07.314908 31570 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 10 to 11, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 11 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:23:07.363979 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 104823us
I20260709 12:23:07.322304  5166 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:07.582861  5093 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:07.354065 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:07.381644 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.382694 31823 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 13 index: 9. (index mismatch)
I20260709 12:23:07.590157 31818 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 10. Current term is 11. Ops: []
I20260709 12:23:07.263777  5155 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:07.611200 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2096) took 2273 ms. Trace:
I20260709 12:23:07.611660 31751 rpcz_store.cc:276] 0709 12:23:05.337748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.068122 (+1730374us) service_pool.cc:224] Handling call
0709 12:23:07.068560 (+   438us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:07.098820 (+ 30260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.098854 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:07.107228 (+  8374us) log.cc:844] Serialized 52 byte log entry
0709 12:23:07.231859 (+124631us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:07.396311 (+164452us) log.cc:844] Serialized 14 byte log entry
0709 12:23:07.537421 (+141110us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.537499 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:07.610964 (+ 73465us) raft_consensus.cc:1685] finished
0709 12:23:07.610985 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.611138 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":948,"raft.queue_time_us":141813,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":38803,"thread_start_us":235384,"threads_started":2,"wal-append.queue_time_us":207570}
I20260709 12:23:07.612782  5184 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:07.613775 31751 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:07.614367 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2105) took 1071 ms. Trace:
I20260709 12:23:07.533710 31753 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:07.614882 31753 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:07.615645 31753 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.616600 31753 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:07.617870  5096 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:07.397364 31825 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:07.618489  5096 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.533699 32376 rpcz_store.cc:276] 0709 12:23:05.880604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:05.880856 (+   252us) service_pool.cc:224] Handling call
0709 12:23:05.891761 (+ 10905us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.152908 (+1261147us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.300057 (+147149us) spinlock_profiling.cc:243] Waited 1.26 s on lock 0x7b5400544f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:07.300272 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2898533120}
I20260709 12:23:07.621031  5096 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:07.621613 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:07.537460 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:07.361340  5148 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:07.589493  5091 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:07.624019  5148 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.360746  5164 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:07.362838  5161 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.364495  5090 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:07.363435  5127 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.362721  5144 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:07.365295  5188 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:07.569602 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2704) took 1586 ms. Trace:
I20260709 12:23:07.351168  5008 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.581138 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2710) took 1465 ms. Trace:
I20260709 12:23:07.370294  4739 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.369805  5019 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:07.538050  5194 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.587177 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:07.394304 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1947) took 1517 ms. Trace:
I20260709 12:23:07.648327 31753 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:07.670604 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:07.592069 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:07.592715 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:07.672232 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.635473  5161 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:07.589185  5166 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.579403  5187 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.593550  5083 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.004s
I20260709 12:23:07.595883  5081 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:07.596464  5010 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:07.597329  5146 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:07.366545  5151 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.672945  5194 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:07.598076  5185 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.603804  5100 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:07.603921  5074 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:07.604668 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.604789 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:07.605276  5004 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.703938  5004 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:07.583573  5141 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.607121 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.706965 31674 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:07.707571 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.708024 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.708340 31671 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:07.708659  5150 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.709189  5150 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:07.365803  5162 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.709538 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.545451  5094 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:07.711289  4785 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:07.624032 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:07.712415 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 812) took 1134 ms. Trace:
I20260709 12:23:07.712631 31670 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:07.712875 31746 rpcz_store.cc:276] 0709 12:23:06.578343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.923209 (+344866us) service_pool.cc:224] Handling call
0709 12:23:07.712247 (+789038us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b5400213ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:07.712356 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":259101440}
I20260709 12:23:07.627079 31751 rpcz_store.cc:276] 0709 12:23:06.542654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.613278 (+1070624us) service_pool.cc:224] Handling call
0709 12:23:07.613590 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.614131 (+   541us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.614317 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.714398 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:07.715018 31751 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:07.715345 31676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1954) took 1348 ms (client timeout 1766 ms). Trace:
W20260709 12:23:07.715827 31676 rpcz_store.cc:269] 0709 12:23:06.367087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.401180 (+ 34093us) service_pool.cc:224] Handling call
0709 12:23:07.715041 (+1313861us) spinlock_profiling.cc:243] Waited 927 ms on lock 0x7b54003917d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:07.715189 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2131504896}
I20260709 12:23:07.368975  4948 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.717458 31676 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:07.363248  5077 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.718124 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1949) took 1741 ms. Trace:
I20260709 12:23:07.627574 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:07.719272 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 91765.3us
I20260709 12:23:07.719466 31671 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:07.720052 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:07.721405  4865 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:07.721827  4865 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:07.722218  4865 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:07.723059 31678 raft_consensus.cc:2410] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:23:07.724156 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2806) took 1527 ms. Trace:
I20260709 12:23:07.724480 31678 rpcz_store.cc:276] 0709 12:23:06.196263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.932463 (+736200us) service_pool.cc:224] Handling call
0709 12:23:07.723945 (+791482us) spinlock_profiling.cc:243] Waited 604 ms on lock 0x7b54002535d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:07.724094 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1389378432}
I20260709 12:23:07.726294 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2799) took 1697 ms. Trace:
I20260709 12:23:07.561827 31675 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:07.625463  5148 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:07.638113 31750 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:07.638881 31823 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:07.642594 31752 rpcz_store.cc:276] 0709 12:23:05.983045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.231797 (+1248752us) service_pool.cc:224] Handling call
0709 12:23:07.232255 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.569290 (+337035us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.569531 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.701313  5188 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.339536  4324 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:07.345630  5167 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.648514  5127 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:07.656879 31749 rpcz_store.cc:276] 0709 12:23:06.115654 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.567777 (+1452123us) service_pool.cc:224] Handling call
0709 12:23:07.568204 (+   427us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.568923 (+   719us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.581082 (+ 12159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.701041 31570 catalog_manager.cc:5697] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 10 to 13, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 13 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:07.580924 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:07.658723 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 388382us
I20260709 12:23:07.545020  5072 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:07.667507 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 86658.8us
I20260709 12:23:07.671689 31753 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:23:07.673234 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.592971 32371 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
W20260709 12:23:07.675024 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 83024.5us
I20260709 12:23:07.730906 31819 raft_consensus.cc:2368] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15.
I20260709 12:23:07.732044 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 811) took 1701 ms. Trace:
I20260709 12:23:07.743556 31745 rpcz_store.cc:276] 0709 12:23:06.030173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.532970 (+1502797us) service_pool.cc:224] Handling call
0709 12:23:07.533257 (+   287us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.731106 (+197849us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.731141 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:07.731189 (+    48us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:07.731235 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.731815 (+   580us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b54004033d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:07.731860 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.731982 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":454519680}
I20260709 12:23:07.732767 31748 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
W20260709 12:23:07.676690 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 89598.1us
I20260709 12:23:07.736099 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:07.718494 31676 rpcz_store.cc:276] 0709 12:23:05.976998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.716830 (+1739832us) service_pool.cc:224] Handling call
0709 12:23:07.717279 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.717881 (+   602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.718068 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.628405  4859 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.755776  4859 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:07.642434  5144 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:07.759138 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2803) took 1690 ms. Trace:
I20260709 12:23:07.759595 31673 rpcz_store.cc:276] 0709 12:23:06.069136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.820859 (+751723us) service_pool.cc:224] Handling call
0709 12:23:07.758889 (+938030us) spinlock_profiling.cc:243] Waited 410 ms on lock 0x7b54001d5954. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:07.759068 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":943421696}
I20260709 12:23:07.657092  5174 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.689628 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2794) took 1662 ms. Trace:
I20260709 12:23:07.365078  4873 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.672053  5075 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:07.732592 31570 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:07.783968  4948 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:07.784693 31570 catalog_manager.cc:5697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 3 to 12, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 12 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:07.696419 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:07.787474 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:07.680032  5155 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:07.788949 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 957) took 1123 ms. Trace:
I20260709 12:23:07.789620  5077 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:07.789798 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53833.9us
I20260709 12:23:07.790015 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2712) took 1673 ms. Trace:
I20260709 12:23:07.678786  5012 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:07.701149 31747 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:23:07.702805 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.790477 31744 rpcz_store.cc:276] 0709 12:23:06.116294 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.584458 (+1468164us) service_pool.cc:224] Handling call
0709 12:23:07.584870 (+   412us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.585172 (+   302us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.585212 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.585668 (+   456us) log.cc:844] Serialized 26 byte log entry
0709 12:23:07.686272 (+100604us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:07.686949 (+   677us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.687027 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:07.789749 (+102722us) raft_consensus.cc:1685] finished
0709 12:23:07.789776 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.789919 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":18110,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":104,"spinlock_wait_cycles":225408,"thread_start_us":82590,"threads_started":1,"wal-append.queue_time_us":140926}
W20260709 12:23:07.730674  5009 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.533s	user 0.010s	sys 0.000s
I20260709 12:23:07.792711 31744 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:07.607416 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 792) took 1277 ms. Trace:
I20260709 12:23:07.702775 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:07.620366  4846 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.795006 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:07.795205 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.729866 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:07.796193 31752 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 13: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:07.796387 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2109) took 1252 ms. Trace:
I20260709 12:23:07.796917 31744 rpcz_store.cc:276] 0709 12:23:06.543726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.792068 (+1248342us) service_pool.cc:224] Handling call
0709 12:23:07.792381 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.794208 (+  1827us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.796327 (+  2119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.797405 31752 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:23:07.798032 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2112) took 1253 ms. Trace:
I20260709 12:23:07.686797  5187 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:07.735428 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:07.801121 31675 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:07.630772  5184 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:07.669330 31669 rpcz_store.cc:276] 0709 12:23:05.877171 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.029479 (+1152308us) service_pool.cc:224] Handling call
0709 12:23:07.029818 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.030069 (+   251us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:07.226019 (+195950us) log.cc:844] Serialized 14 byte log entry
0709 12:23:07.368929 (+142910us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:07.369016 (+    87us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:07.369094 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.369214 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.394203 (+ 24989us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32426,"prepare.queue_time_us":260684,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":10079,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":139426,"thread_start_us":308667,"threads_started":2,"wal-append.queue_time_us":318168}
I20260709 12:23:07.746613 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:07.733393 31822 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:23:07.746727 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:07.802700 31672 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:07.732437 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:07.726612 31678 rpcz_store.cc:276] 0709 12:23:06.028591 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.725445 (+1696854us) service_pool.cc:224] Handling call
0709 12:23:07.725782 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.725975 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.725999 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:07.726040 (+    41us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:07.726070 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.726123 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:07.726242 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.731655  5167 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:07.754739  5207 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:23:07.765502 31746 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:23:07.768204 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:07.769364 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.773731 31677 rpcz_store.cc:276] 0709 12:23:06.027105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.569058 (+541953us) service_pool.cc:224] Handling call
0709 12:23:07.689449 (+1120391us) spinlock_profiling.cc:243] Waited 694 ms on lock 0x7b54002553d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:07.689574 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1596995328}
I20260709 12:23:07.778446  5133 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:07.779775  4873 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:07.792761 31751 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:07.730155  5188 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:07.722146 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:07.885125 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 163040us
I20260709 12:23:07.789484 31819 rpcz_store.cc:276] 0709 12:23:06.665359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.732481 (+ 67122us) service_pool.cc:224] Handling call
0709 12:23:07.788714 (+1056233us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b5400197dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:07.788858 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1901707648}
I20260709 12:23:07.729178 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:07.798368 31752 rpcz_store.cc:276] 0709 12:23:06.544528 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.796797 (+1252269us) service_pool.cc:224] Handling call
0709 12:23:07.797171 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.797783 (+   612us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.797980 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.794492 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:07.793805 31753 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:07.764344  5206 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.079s	user 0.000s	sys 0.003s
I20260709 12:23:07.741053 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:07.889266 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.702087  5084 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:07.894431 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 153436us
I20260709 12:23:07.900560 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:07.805541 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:07.903532 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 98065.9us
I20260709 12:23:07.693406  5085 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:07.856467 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.905850 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:07.878528 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 131995us
I20260709 12:23:07.789059  5155 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:07.809266 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:07.879284 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:07.726887  5146 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:07.793712 32373 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:07.763737  5019 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:07.882097  5083 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election won for term 17
W20260709 12:23:07.887703 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 97142.1us
I20260709 12:23:07.930817  5019 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:07.793332 32369 rpcz_store.cc:276] 0709 12:23:06.329902 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:06.330070 (+   168us) service_pool.cc:224] Handling call
0709 12:23:06.331977 (+  1907us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.253877 (+921900us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.607162 (+353285us) spinlock_profiling.cc:243] Waited 921 ms on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:07.607359 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2118409088}
I20260709 12:23:07.894333 31752 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:07.895397  4873 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:07.779844  4324 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:07.914227 31678 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:07.907225 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:07.916707 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:07.958170 31819 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:07.797139  5011 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:07.909225 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:07.918536 31818 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:07.962620 31676 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:07.963418 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1957) took 1595 ms. Trace:
I20260709 12:23:07.963761 31676 rpcz_store.cc:276] 0709 12:23:06.367918 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.931402 (+1563484us) service_pool.cc:224] Handling call
0709 12:23:07.962419 (+ 31017us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.963136 (+   717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.963347 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.807499  5184 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:07.799100  5152 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:07.807735  4948 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:07.798941  5016 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.128s
I20260709 12:23:07.877691  5209 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.884130 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:07.795291  5127 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:07.881489  5167 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.886780  5077 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:07.940659 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:08.025658 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:07.888295 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2107) took 1345 ms. Trace:
I20260709 12:23:08.029323 31746 rpcz_store.cc:276] 0709 12:23:06.543237 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.764838 (+1221601us) service_pool.cc:224] Handling call
0709 12:23:07.765150 (+   312us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.888022 (+122872us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.888235 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.879119 31745 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:23:07.946967 31751 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: [15.10-17.11]
I20260709 12:23:07.947304 32374 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:07.689898  5164 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:07.908413  5139 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.959056 31678 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:07.960363 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53260.9us
I20260709 12:23:07.969051  5146 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:07.921116  5134 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:07.998934  5036 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.972807 31819 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:08.082775 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:07.796840  5174 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.018707 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 891) took 1462 ms. Trace:
I20260709 12:23:08.085328 31676 rpcz_store.cc:276] 0709 12:23:06.556009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.964524 (+1408515us) service_pool.cc:224] Handling call
0709 12:23:08.018078 (+ 53554us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.018278 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.018310 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.018354 (+    44us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:08.018393 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.018452 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.018619 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.017227  5081 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:08.028642 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 88116.9us
I20260709 12:23:07.946617 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2720) took 1025 ms. Trace:
I20260709 12:23:08.031208 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2113) took 1448 ms. Trace:
I20260709 12:23:08.104575 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:08.104806 31745 rpcz_store.cc:276] 0709 12:23:06.582554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.878537 (+1295983us) service_pool.cc:224] Handling call
0709 12:23:07.878898 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.030807 (+151909us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.031129 (+   322us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.104212 31752 rpcz_store.cc:276] 0709 12:23:06.921276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.893553 (+972277us) service_pool.cc:224] Handling call
0709 12:23:07.894046 (+   493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.946340 (+ 52294us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.946550 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:07.952759  5155 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:07.917613  5075 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 12:23:08.038751 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:08.042321 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:07.910779  5009 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:07.969527  4324 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:07.968694  5074 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:08.044785 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 890) took 1489 ms. Trace:
I20260709 12:23:08.047518 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.151999 31678 rpcz_store.cc:276] 0709 12:23:06.555734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.955943 (+1400209us) service_pool.cc:224] Handling call
0709 12:23:07.956434 (+   491us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.044480 (+ 88046us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.044720 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.153976 31678 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:08.018322  5190 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:08.095031 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 813) took 1516 ms. Trace:
I20260709 12:23:08.024686 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:08.084515 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.098971 32374 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:08.035845  5091 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:08.158895 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.027366  5125 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:08.121520 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 201123us
I20260709 12:23:08.126567 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:08.044962  5146 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:08.045243  5077 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:08.180326 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 892) took 1620 ms. Trace:
I20260709 12:23:08.181175 31676 rpcz_store.cc:276] 0709 12:23:06.559729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.130067 (+1570338us) service_pool.cc:224] Handling call
0709 12:23:08.179567 (+ 49500us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.179829 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.179883 (+    54us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.179922 (+    39us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:08.179968 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.180042 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.180235 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:08.144598 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 102417us
I20260709 12:23:08.040824  5151 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:07.807556  4846 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.160982 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:08.165309 32371 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:08.027802  5127 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.093066  5162 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:08.102432  5134 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:08.185640 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 161101us
I20260709 12:23:08.187379 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:08.187428 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.190812 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 895) took 1595 ms. Trace:
I20260709 12:23:08.229779 31678 rpcz_store.cc:276] 0709 12:23:06.595131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.153243 (+1558112us) service_pool.cc:224] Handling call
0709 12:23:08.153724 (+   481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.162469 (+  8745us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.190721 (+ 28252us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.043795  5152 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:07.891937 31570 catalog_manager.cc:5697] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 9 to 11, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 11 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:08.130138  5166 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:08.047899  5139 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.036113  5167 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:08.207516 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 301768us
I20260709 12:23:08.135087  5083 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:08.223387 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 118945us
I20260709 12:23:08.261912 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:08.262275 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:08.263759 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:08.262342  5083 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.264525  5091 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:08.264923 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:08.165041  4776 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:08.161728  5190 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:08.265563  4776 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:08.265733  5190 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:08.017778  5209 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.177052  5146 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:08.161195  5195 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:08.123706  5081 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.268165  5209 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:08.268821  5209 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:08.096912  5036 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.187940  4948 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:08.265066  5091 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:08.248229 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:08.186295  5074 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:08.218569  5009 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:08.269610  5209 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:08.275565 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:08.271596 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.282276  5036 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:08.282413  5200 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:23:08.283070 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.283131  4846 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:08.283944  4846 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
W20260709 12:23:08.284076  5212 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.286s	user 0.001s	sys 0.000s
I20260709 12:23:08.287102  5084 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:08.287956  5085 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:08.288899  5094 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.579s	user 0.000s	sys 0.008s
I20260709 12:23:08.289377  5094 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:08.290735  4785 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.579s	user 0.008s	sys 0.002s
I20260709 12:23:08.290915  4570 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:08.291757 31671 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 13 index: 9. (term mismatch)
I20260709 12:23:08.292156 31671 pending_rounds.cc:85] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 8
I20260709 12:23:08.292339 31671 pending_rounds.cc:107] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 12.9
I20260709 12:23:08.292529 31671 raft_consensus.cc:2889] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:08.279273  5072 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:08.295745  5072 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:08.296212  5072 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:08.296602 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.297254  5072 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.297216  4776 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.299738  5174 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:08.300597  5091 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.6, Last appended by leader: 6, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.295265  5074 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.302433  5127 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:08.269670  5081 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:08.304133  4785 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:08.303980  5146 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.305397 31822 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:08.306227  5081 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Stepping down as leader of term 10
I20260709 12:23:08.188618  5125 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.306787  5081 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 10 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:08.279855  5219 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.308815  5219 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.310981  4895 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:08.311513  4895 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:08.312121  5167 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:08.270201  5162 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.043s
W20260709 12:23:08.297187 32343 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 2.125s (SENT) [suppressed 8 similar messages]
I20260709 12:23:08.314386 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2670) took 1075 ms. Trace:
I20260709 12:23:08.315603 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1951) took 1949 ms. Trace:
I20260709 12:23:08.311246  5099 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:23:08.248464  5150 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.539s	user 0.008s	sys 0.004s
I20260709 12:23:08.316489 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2807) took 2041 ms. Trace:
I20260709 12:23:08.163641  4324 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.314899 31827 rpcz_store.cc:276] 0709 12:23:07.238516 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.609221 (+370705us) service_pool.cc:224] Handling call
0709 12:23:07.609600 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.609979 (+   379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.610009 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.610541 (+   532us) log.cc:844] Serialized 26 byte log entry
0709 12:23:07.685375 (+ 74834us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:07.685417 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.685496 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:08.314122 (+628626us) raft_consensus.cc:1685] finished
0709 12:23:08.314144 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.314318 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3424,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":86,"thread_start_us":41357,"threads_started":1,"wal-append.queue_time_us":258523}
I20260709 12:23:08.307878  5081 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.315975 31673 rpcz_store.cc:276] 0709 12:23:06.366258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.760579 (+1394321us) service_pool.cc:224] Handling call
0709 12:23:07.760987 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:07.761376 (+   389us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:07.761410 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:07.761950 (+   540us) log.cc:844] Serialized 26 byte log entry
0709 12:23:07.973250 (+211300us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:07.973315 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:07.973395 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:08.315388 (+341993us) raft_consensus.cc:1685] finished
0709 12:23:08.315412 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.315545 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1562,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":90,"thread_start_us":210851,"threads_started":1,"wal-append.queue_time_us":475168}
I20260709 12:23:08.339434 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 817) took 1325 ms. Trace:
I20260709 12:23:08.340047 31748 rpcz_store.cc:276] 0709 12:23:07.013611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.338202 (+1324591us) service_pool.cc:224] Handling call
0709 12:23:08.338736 (+   534us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.338973 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.339011 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.339070 (+    59us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:08.339116 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.339189 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.339354 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.342139 31748 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
W20260709 12:23:08.352079 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.227240  5100 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.088s
I20260709 12:23:08.330137 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2115) took 1727 ms. Trace:
I20260709 12:23:08.382076 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:08.332937 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2114) took 1750 ms. Trace:
I20260709 12:23:08.383129 31751 rpcz_store.cc:276] 0709 12:23:06.582891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.880844 (+1297953us) service_pool.cc:224] Handling call
0709 12:23:07.881219 (+   375us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:08.030844 (+149625us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.332516 (+301672us) spinlock_profiling.cc:243] Waited 65.5 ms on lock 0x7b54004047d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:08.332817 (+   301us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":150573440}
W20260709 12:23:08.315172  5004 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.611s	user 0.011s	sys 0.000s
W20260709 12:23:08.386914 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.387730  5167 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:08.388497  4859 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.632s	user 0.008s	sys 0.001s
I20260709 12:23:08.388238  5167 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:08.391546 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.394598  5094 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:08.392582 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:08.395861 31673 raft_consensus.cc:2368] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15.
I20260709 12:23:08.396911 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:08.291301  4570 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:08.390467  5167 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:08.317837 31671 rpcz_store.cc:276] 0709 12:23:06.274988 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.747703 (+1472715us) service_pool.cc:224] Handling call
0709 12:23:07.782482 (+ 34779us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.294612 (+512130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.310360 (+ 15748us) spinlock_profiling.cc:243] Waited 509 ms on lock 0x7b5400181fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:08.316402 (+  6042us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1170436864}
I20260709 12:23:08.249006  5010 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20260709 12:23:08.273283  4739 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:08.280326  5209 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:08.323531  5161 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.637s	user 0.005s	sys 0.005s
I20260709 12:23:08.399639 31673 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:08.401192 31818 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:08.394089 31751 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:23:08.401772  5099 consensus_queue.cc:1243] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 16.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
W20260709 12:23:08.409039 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 248159us
I20260709 12:23:08.451625 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 898) took 1813 ms. Trace:
I20260709 12:23:08.454605 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:08.455209 31673 rpcz_store.cc:276] 0709 12:23:06.639069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.398052 (+1758983us) service_pool.cc:224] Handling call
0709 12:23:08.399453 (+  1401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.451326 (+ 51873us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.451557 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.456403 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:08.456739 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:08.411262 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:08.413517 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 816) took 1400 ms. Trace:
I20260709 12:23:08.457864 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:08.461146  5167 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:08.467991 31822 raft_consensus.cc:2368] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15.
W20260709 12:23:08.436231 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.336004  5150 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.439864  5195 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:08.466032  5187 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.579s	user 0.008s	sys 0.004s
I20260709 12:23:08.484323  5150 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:08.487283 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2106) took 1944 ms. Trace:
I20260709 12:23:08.487740 31753 rpcz_store.cc:276] 0709 12:23:06.542921 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.747314 (+1204393us) service_pool.cc:224] Handling call
0709 12:23:07.793612 (+ 46298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.887332 (+ 93720us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.487060 (+599728us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b54001ee6d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:08.487217 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105564032}
I20260709 12:23:08.489486 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 894) took 1914 ms. Trace:
I20260709 12:23:08.489912 31675 rpcz_store.cc:276] 0709 12:23:06.575822 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.141242 (+1565420us) service_pool.cc:224] Handling call
0709 12:23:08.141703 (+   461us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.150957 (+  9254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.151006 (+    49us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.151066 (+    60us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:08.151114 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.489230 (+338116us) spinlock_profiling.cc:243] Waited 8.96 ms on lock 0x7b54001824d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:08.489281 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.489423 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20600576}
I20260709 12:23:08.491348 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:08.491938 31675 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:08.500756 31752 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
W20260709 12:23:08.440346 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 197691us
I20260709 12:23:08.502859 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2116) took 1645 ms. Trace:
I20260709 12:23:08.462913  5219 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:08.517380  5167 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:08.518402  5093 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.877s	user 0.008s	sys 0.003s
W20260709 12:23:08.447659 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.537449 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 240268us
I20260709 12:23:08.452497 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:08.539011 31751 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:23:08.539625  5164 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:08.408304 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2118) took 1109 ms. Trace:
I20260709 12:23:08.460534  5215 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:08.416545 32342 leader_election.cc:400] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15.
I20260709 12:23:08.457950 31746 rpcz_store.cc:276] 0709 12:23:07.013338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.333838 (+1320500us) service_pool.cc:224] Handling call
0709 12:23:08.412873 (+ 79035us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.413090 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.413133 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.413189 (+    56us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:08.413236 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.413308 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.413443 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.465230 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:08.417920 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.542492 32342 leader_election.cc:403] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:23:08.473244 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:08.544977 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:08.438973 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 195338us
I20260709 12:23:08.380674  5019 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.474263 31819 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:08.451676  5209 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:08.503440 31750 rpcz_store.cc:276] 0709 12:23:06.857448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.895777 (+1038329us) service_pool.cc:224] Handling call
0709 12:23:07.931406 (+ 35629us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.931727 (+   321us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:08.216990 (+285263us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.449519 (+232529us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.502358 (+ 52839us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.502436 (+    78us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:08.502501 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.502596 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.502782 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60411,"prepare.queue_time_us":110328,"prepare.run_cpu_time_us":99,"prepare.run_wall_time_us":105,"raft.queue_time_us":10308,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":14070,"thread_start_us":463926,"threads_started":2}
W20260709 12:23:08.548470 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 83334.6us
I20260709 12:23:08.549353 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2810) took 1537 ms. Trace:
I20260709 12:23:08.549727 31672 rpcz_store.cc:276] 0709 12:23:07.012328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.053887 (+ 41559us) service_pool.cc:224] Handling call
0709 12:23:08.549184 (+1495297us) spinlock_profiling.cc:243] Waited 648 ms on lock 0x7b5400229f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:08.549291 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1491376896}
I20260709 12:23:08.492771 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:08.551211 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:08.431420  5077 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.509384 31570 catalog_manager.cc:5697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 10 to 12, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 12 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:23:08.481483  5090 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.838s	user 0.007s	sys 0.004s
I20260709 12:23:08.550031  4876 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15.
I20260709 12:23:08.511869  5155 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.558004 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:08.558691 31750 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:08.558518  5234 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.007s
I20260709 12:23:08.559310  5234 consensus_queue.cc:1243] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 13.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:08.514791  4570 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.559466  5219 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:08.560117  4570 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:08.560426  5234 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:08.560817  5234 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:08.561089  5219 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 15.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:08.562038  5219 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:08.562798  5227 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:23:08.562671  5232 consensus_queue.cc:1175] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "039d66a6c85e43c3936bf5017501b344" responder_term: 15 status { last_received { term: 14 index: 11 } last_committed_idx: 11 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:23:08.563423  5227 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:08.564324  5141 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:08.505101  5187 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.564404  5227 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.564682 31570 catalog_manager.cc:5697] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 7 to 8, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 8 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:23:08.547322  5194 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.814s	user 0.010s	sys 0.001s
W20260709 12:23:08.527527  4865 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.805s	user 0.011s	sys 0.000s
I20260709 12:23:08.529345  5072 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:08.442016 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:08.583477 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 141603us
I20260709 12:23:08.584558 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:08.532128  4324 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:08.463301  5161 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.470237  5184 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:08.605962 31750 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:08.540421 31748 rpcz_store.cc:276] 0709 12:23:07.298998 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.341290 (+1042292us) service_pool.cc:224] Handling call
0709 12:23:08.341696 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.342558 (+   862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.408217 (+ 65659us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:08.543476 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 142789us
I20260709 12:23:08.478349 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:08.535827  5148 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.807s	user 0.009s	sys 0.001s
I20260709 12:23:08.547328 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:08.610425 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:08.551828 31751 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:08.551245 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 900) took 1759 ms. Trace:
I20260709 12:23:08.616086 31678 rpcz_store.cc:276] 0709 12:23:06.791800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.550450 (+1758650us) service_pool.cc:224] Handling call
0709 12:23:08.550764 (+   314us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.550946 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.550971 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.551011 (+    40us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:08.551040 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.551090 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.551179 (+    89us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.617753 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:08.440408 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 174587us
I20260709 12:23:08.533344  5146 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:08.542309  5093 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.623104 31673 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:23:08.623901 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1963) took 1501 ms. Trace:
I20260709 12:23:08.624236 31673 rpcz_store.cc:276] 0709 12:23:07.122158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.614614 (+1492456us) service_pool.cc:224] Handling call
0709 12:23:08.615107 (+   493us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.623595 (+  8488us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.623826 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.625403 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.625810 31673 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19. Ops: [16.11-18.12]
I20260709 12:23:08.626906 31678 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:08.555337 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:08.628837 31752 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:08.554262 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 901) took 1697 ms. Trace:
W20260709 12:23:08.557169  5188 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.664s	user 0.009s	sys 0.003s
I20260709 12:23:08.557824  5090 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.558964 31674 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:08.561775  5234 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.639021  5234 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:08.516594  5139 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:08.524468  4873 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.559s	user 0.006s	sys 0.003s
I20260709 12:23:08.523200  4859 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:23:08.526648  5144 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.768s	user 0.009s	sys 0.001s
I20260709 12:23:08.572108 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:08.454185  5134 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.574133 31820 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:08.574155 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.652530 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:08.655577 31818 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 15: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:08.581110  5230 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:08.582386 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:08.587369 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.586021 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:08.556088  5169 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.966s	user 0.006s	sys 0.006s
I20260709 12:23:08.524541  5190 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.586812  4865 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.592418  5019 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:08.593560  5187 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:08.558046  5215 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:08.594676  5235 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:08.595175  5195 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:08.596484 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 55847.1us
I20260709 12:23:08.597790 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 897) took 1959 ms. Trace:
W20260709 12:23:08.600109 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.610440  5141 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 12:23:08.603616 31669 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:23:08.534853 31753 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:23:08.592677 31570 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 10 to 11, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 11 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:08.619947 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:08.582520  5100 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.629272 31675 rpcz_store.cc:276] 0709 12:23:06.856412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.553148 (+1696736us) service_pool.cc:224] Handling call
0709 12:23:08.553449 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.553631 (+   182us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.553655 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.553701 (+    46us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:08.553742 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.553805 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.553905 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.631888 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:08.635088 31819 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:08.636166 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1966) took 1508 ms. Trace:
I20260709 12:23:08.637569 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:08.637988 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2814) took 1328 ms. Trace:
I20260709 12:23:08.640292 31825 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 10. (term mismatch)
I20260709 12:23:08.687431 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 821) took 1065 ms. Trace:
I20260709 12:23:08.690469 31825 pending_rounds.cc:85] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 9
I20260709 12:23:08.639015  5126 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 12:23:08.690953 31825 pending_rounds.cc:107] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 16.10
I20260709 12:23:08.671358  5085 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.643635 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:08.605571  5133 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.723s	user 0.011s	sys 0.000s
I20260709 12:23:08.686614 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:08.637225 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:08.646014 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:08.639384 32343 leader_election.cc:400] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15.
W20260709 12:23:08.710779 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 79633.9us
I20260709 12:23:08.710894 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:08.648140 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.605352  4948 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:08.582253  5150 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.061s
I20260709 12:23:08.651731 31823 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:08.658147 31749 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:08.609325  5151 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:08.659643 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 77355.4us
W20260709 12:23:08.667793 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.672703 31677 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:08.639904  5232 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:08.583057  5194 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.676148 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.750204 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:08.677728 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 122593us
I20260709 12:23:08.640239  5208 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:08.793787 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.680480 31751 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
W20260709 12:23:08.680546 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 94619.9us
I20260709 12:23:08.650879  4873 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:23:08.600998 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.684122 31676 rpcz_store.cc:276] 0709 12:23:06.638321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.236894 (+1598573us) service_pool.cc:224] Handling call
0709 12:23:08.275532 (+ 38638us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.315632 (+ 40100us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.315670 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.315714 (+    44us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:08.315747 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.597500 (+281753us) spinlock_profiling.cc:243] Waited 39.9 ms on lock 0x7b5400181ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:08.597560 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.597711 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":91771136}
I20260709 12:23:08.671645  5008 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:08.691377 31825 raft_consensus.cc:2889] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:08.642571 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:08.688719 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1960) took 1787 ms. Trace:
I20260709 12:23:08.835350 32374 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:08.835687 31669 rpcz_store.cc:276] 0709 12:23:06.901487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.602867 (+1701380us) service_pool.cc:224] Handling call
0709 12:23:08.603234 (+   367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.686815 (+ 83581us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.688659 (+  1844us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.836880 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:08.837603 31669 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
W20260709 12:23:08.690151 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52688us
I20260709 12:23:08.839694 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:08.702899 31673 rpcz_store.cc:276] 0709 12:23:07.311887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.625098 (+1313211us) service_pool.cc:224] Handling call
0709 12:23:08.625500 (+   402us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:08.637678 (+ 12178us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.637929 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.704650 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.841874 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.687959 31750 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:08.705036 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:08.689033 31675 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:08.843470 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2815) took 1531 ms. Trace:
I20260709 12:23:08.843850 31675 rpcz_store.cc:276] 0709 12:23:07.312307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.688489 (+1376182us) service_pool.cc:224] Handling call
0709 12:23:08.688805 (+   316us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.843160 (+154355us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.843408 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.705328 32376 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:23:08.690845 31753 rpcz_store.cc:276] 0709 12:23:07.622498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.533963 (+911465us) service_pool.cc:224] Handling call
0709 12:23:08.534467 (+   504us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.687157 (+152690us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.687374 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.845638 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:08.846300 31753 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:08.689553 31678 rpcz_store.cc:276] 0709 12:23:07.127418 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.617755 (+1490337us) service_pool.cc:224] Handling call
0709 12:23:08.618213 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.635858 (+ 17645us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.636098 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:08.847913 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:08.689431 31745 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:08.849529 31745 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:08.851003 31745 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:08.711788 31752 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:08.712633  5240 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
W20260709 12:23:08.713230  5229 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:23:08.651856  5225 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.079s
I20260709 12:23:08.719702  5090 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:08.722827  5085 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.724478 31570 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 9 to 11, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 11 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:08.688194  5075 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.704556  5009 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:08.671180  5144 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.727592  5161 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:08.735591  5093 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:08.686534  5221 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:08.738549 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:08.737978  5215 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.684403  5235 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:08.887457 31821 raft_consensus.cc:2410] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:23:08.740458  5126 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Stepping down as leader of term 14
W20260709 12:23:08.743662 31820 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2767) took 1763 ms (client timeout 2123 ms). Trace:
I20260709 12:23:08.744169 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:08.739588 31826 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:08.752513 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.744169 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:08.889741 32369 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 15: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:08.744377 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 100886us
I20260709 12:23:08.890524 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:08.864439  5200 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:08.825886 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.824577  5151 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:08.827819 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 77730.9us
I20260709 12:23:08.828147 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:08.829888 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 118240us
I20260709 12:23:08.772539 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:08.911618 31670 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: [15.10-17.11]
I20260709 12:23:08.911949 31818 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:08.901146 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 965) took 1157 ms. Trace:
I20260709 12:23:08.887766  5221 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:08.913787 31821 rpcz_store.cc:276] 0709 12:23:07.744375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.744499 (+   124us) service_pool.cc:224] Handling call
0709 12:23:08.738381 (+993882us) spinlock_profiling.cc:243] Waited 1.38 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:23:08.893581 (+155200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3179264}
I20260709 12:23:08.889731  5126 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:08.882678  5169 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 12:23:08.742816  4948 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:08.906379  5127 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:08.742568  5190 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:08.689826  5188 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.168s
I20260709 12:23:08.931573 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.753264  4739 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:08.770526  5019 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:08.882138 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:08.830229  5230 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:08.708488  5100 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.834141  5139 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:08.836966  5232 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:08.888067 31820 rpcz_store.cc:269] 0709 12:23:06.980176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.147434 (+167258us) service_pool.cc:224] Handling call
0709 12:23:07.282337 (+134903us) spinlock_profiling.cc:243] Waited 9.17 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:23:08.743470 (+1461133us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b540043d5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:08.743601 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2789443200}
I20260709 12:23:08.873085  5004 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.900241 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:08.904320 31748 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:08.825726 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 179878us
I20260709 12:23:08.862658  5146 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.031s
I20260709 12:23:08.908057 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:08.908816 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:08.909530 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:08.743206  5187 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:08.911661 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:08.972188 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.972242 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.928876  5152 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.934278 31752 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:08.980094 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.980651  5187 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:08.984328 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 72733.3us
I20260709 12:23:08.923974 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:08.837895  5208 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.922899 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:08.927698 31753 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:23:08.866314 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:08.931196  5164 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.047s
W20260709 12:23:08.801648 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.927070  5209 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.882488  5075 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.941737 32372 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:08.950448 31818 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:08.772229  4865 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:08.920257  5221 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:08.965430 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:08.966446 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 84426.4us
W20260709 12:23:08.999516  5243 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:23:08.967880 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:08.996208  5081 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:08.903712 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:08.974351 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:08.915727 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.992452 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 68562.7us
I20260709 12:23:08.964169  4739 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:09.050668 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.995239 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 82962.7us
W20260709 12:23:08.998980 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:08.998358 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:08.974395  5230 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:08.977617  5004 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:08.973950  5161 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:09.041962 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:09.059162 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.059684  5241 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:09.060179  5241 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:09.060575  5241 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:09.060881 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:09.039319  5174 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:08.981665  5187 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:09.074507 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.043517 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:09.044152 31748 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
W20260709 12:23:08.995719  5074 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.549s	user 0.014s	sys 0.038s
W20260709 12:23:09.046154 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 80823.8us
I20260709 12:23:09.087051 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:09.088845 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:08.999452  5011 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.031s	user 0.013s	sys 0.001s
I20260709 12:23:08.861406 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:09.054694 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:08.978436  5166 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 12:23:09.092175 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:09.040053 31751 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:09.037729  5190 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:08.996616  5231 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:08.941390  4948 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:09.117189 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1962) took 2215 ms. Trace:
I20260709 12:23:09.039660  5127 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:09.118536  5127 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.117686 31671 rpcz_store.cc:276] 0709 12:23:06.902058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.611420 (+1709362us) service_pool.cc:224] Handling call
0709 12:23:08.611859 (+   439us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.612288 (+   429us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.612321 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:08.612855 (+   534us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.675068 (+ 62213us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:08.675113 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.675378 (+   265us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:09.116892 (+441514us) raft_consensus.cc:1685] finished
0709 12:23:09.116917 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.117089 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":125,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"thread_start_us":61746,"threads_started":1,"wal-append.queue_time_us":222163}
I20260709 12:23:09.118978  5127 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:08.382309 31747 rpcz_store.cc:276] 0709 12:23:06.602997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.881871 (+1278874us) service_pool.cc:224] Handling call
0709 12:23:07.922351 (+ 40480us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.922670 (+   319us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:08.109559 (+186889us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.329727 (+220168us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.329788 (+    61us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:08.329835 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.329906 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.330055 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":39627,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":16680,"raft.queue_time_us":577,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":49943,"thread_start_us":405556,"threads_started":2}
I20260709 12:23:09.119580 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:09.120222 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:09.120765 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.121384 31671 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:23:09.121418 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:09.122232 31747 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:09.122366 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:09.061189  5247 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:23:09.057946  5232 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:08.962991 31570 catalog_manager.cc:5697] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 5 to 10, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 10 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:09.125885  5144 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:09.067744 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:09.067031  5096 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:09.076027  5208 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:09.126806  5125 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:08.155004 31744 rpcz_store.cc:276] 0709 12:23:06.579043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.797811 (+1218768us) service_pool.cc:224] Handling call
0709 12:23:07.798084 (+   273us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:07.798286 (+   202us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:07.883832 (+ 85546us) log.cc:844] Serialized 14 byte log entry
0709 12:23:08.094612 (+210780us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:08.094715 (+   103us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:08.094775 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.094846 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:08.094971 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16749,"prepare.queue_time_us":85256,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":16505,"raft.queue_time_us":27561,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":20102,"thread_start_us":210331,"threads_started":1}
I20260709 12:23:09.132892 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:09.080840  5161 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.133597 31744 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:09.133908  5161 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:09.078044  5155 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:09.079958  4570 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.519s	user 0.010s	sys 0.000s
I20260709 12:23:08.987561  5152 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:09.061447  5228 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.096585  5085 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:09.097445  5249 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:09.098237  5168 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:09.098878 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:09.100069  5148 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:09.100381  5250 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:09.101763  4776 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:09.102566  5004 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:09.102736  5099 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.103091 32374 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:09.104264 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1959) took 2291 ms. Trace:
I20260709 12:23:09.104727  5100 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:09.106649  5185 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:09.109308 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:09.112243  5075 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:09.050006  5091 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:08.996622  5126 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:09.135079  5208 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:09.164490  5099 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:09.173636  5126 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:09.113260 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.112980  5174 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:09.113359  5094 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:09.080296  5194 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:08.977841  5235 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.115317  5230 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.123124 31671 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 15: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:09.126966 31676 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:09.071041  4865 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:23:09.127542 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:09.183173 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.118547 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 1973) took 1498 ms. Trace:
I20260709 12:23:09.183790 31674 rpcz_store.cc:276] 0709 12:23:07.619811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.707737 (+ 87926us) service_pool.cc:224] Handling call
0709 12:23:09.118328 (+1410591us) spinlock_profiling.cc:243] Waited 850 ms on lock 0x7b54002e6b54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:09.118482 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1954938624}
I20260709 12:23:09.185035 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:09.128310 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:09.186133 31752 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:09.126034 31570 catalog_manager.cc:5697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 10 to 13, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 13 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:09.079416  5151 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:09.142832 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.190147 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:09.191864 31752 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
W20260709 12:23:09.140244 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 278897us
W20260709 12:23:09.135463 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 46738.6us
I20260709 12:23:09.144294 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 983) took 1242 ms. Trace:
I20260709 12:23:09.141057 31821 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:09.194197 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.141906  5168 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.150677  5148 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:09.151233  5152 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:09.151820 31826 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2762) took 2188 ms (client timeout 1841 ms). Trace:
I20260709 12:23:09.192675 32371 rpcz_store.cc:276] 0709 12:23:07.901715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.901839 (+   124us) service_pool.cc:224] Handling call
0709 12:23:07.931095 (+ 29256us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.221576 (+290481us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.144010 (+922434us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b54004cfed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:09.144227 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":516745088}
I20260709 12:23:09.196823 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.156309 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:09.187484 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:09.165032 31669 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:09.198786 31827 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:09.146214  5146 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.034s
I20260709 12:23:09.199530 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.199888 31671 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 15: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:09.200965 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.202057 31674 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
W20260709 12:23:09.196285 31826 rpcz_store.cc:269] 0709 12:23:06.963317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.131659 (+168342us) service_pool.cc:224] Handling call
0709 12:23:09.151563 (+2019904us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b54001978d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:09.151690 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3317627264}
I20260709 12:23:09.168391 31751 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:09.159896  4785 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:09.176023 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:09.134608 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:09.209051 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:09.209815 31744 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 17: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:23:09.086076  5074 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.014s	sys 0.039s
I20260709 12:23:09.152356  4895 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:09.198024 31747 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:09.156383 31672 rpcz_store.cc:276] 0709 12:23:06.812496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.550681 (+1738185us) service_pool.cc:224] Handling call
0709 12:23:08.576961 (+ 26280us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:08.577285 (+   324us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:08.577329 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:08.583608 (+  6279us) log.cc:844] Serialized 26 byte log entry
0709 12:23:08.915087 (+331479us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:08.915144 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:08.915237 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:09.104006 (+188769us) raft_consensus.cc:1685] finished
0709 12:23:09.104031 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:09.104193 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":263915,"threads_started":1,"wal-append.queue_time_us":370846}
I20260709 12:23:09.205506 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:09.205612 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.208057 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.177762  5094 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 12 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:09.210908 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:09.182684  4865 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.215719 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:09.217442 31674 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:09.221276  5099 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:09.224829 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:23:09.226706 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50701.8us
I20260709 12:23:09.241124  5155 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:09.246976 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.276408 31570 catalog_manager.cc:5697] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 9 to 13. New cstate: current_term: 13 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:09.277923 31672 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:09.279296 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.279640 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:09.250828 31669 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 15: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:09.281632 31673 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:23:09.282215 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2817) took 1494 ms. Trace:
I20260709 12:23:09.234568  5241 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.282788 31669 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:09.283183 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2673) took 2043 ms. Trace:
I20260709 12:23:09.283530 31825 rpcz_store.cc:276] 0709 12:23:07.239263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.620251 (+380988us) service_pool.cc:224] Handling call
0709 12:23:07.620550 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.834876 (+1214326us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.282958 (+448082us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54003c96d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:09.283126 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2344337664}
I20260709 12:23:09.284263 31669 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: [16.10-16.10]
I20260709 12:23:09.284982  5241 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:09.286244 31669 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:09.286546 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1948) took 3309 ms. Trace:
I20260709 12:23:09.286937 31670 rpcz_store.cc:276] 0709 12:23:05.976709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.713498 (+1736789us) service_pool.cc:224] Handling call
0709 12:23:07.713934 (+   436us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:08.912178 (+1198244us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.286276 (+374098us) spinlock_profiling.cc:243] Waited 1.18 s on lock 0x7b54001b35d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:09.286487 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2712589056}
I20260709 12:23:09.197506  5148 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:09.211359  5152 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:09.210932  5188 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:09.197278  5168 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:09.290517 31747 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:09.282557 31673 rpcz_store.cc:276] 0709 12:23:07.788153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.280995 (+1492842us) service_pool.cc:224] Handling call
0709 12:23:09.281285 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.281972 (+   687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.282163 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.254966 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.345394 31671 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
W20260709 12:23:09.346112 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:09.346587 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.346470 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:09.347193 31671 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:09.348178 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:09.247678 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:09.258463 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1972) took 1638 ms. Trace:
I20260709 12:23:09.303653 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:09.360373 31674 rpcz_store.cc:276] 0709 12:23:07.619556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.216959 (+1597403us) service_pool.cc:224] Handling call
0709 12:23:09.217261 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.258215 (+ 40954us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.258406 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:09.361243  5190 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:09.286841  5221 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.253284 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:09.376873 31753 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:09.256060  4865 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:09.275633 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.338091 31676 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
W20260709 12:23:09.352536 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.196326  5096 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.101s
I20260709 12:23:09.188539  5166 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.100s
I20260709 12:23:09.349218 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:09.392117 31672 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:09.354681 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.255806  5227 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:09.268004  4785 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:09.356164  5014 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:09.356643  4739 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:09.399066  5136 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.415344  5136 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:09.417791  5136 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:09.418591  5220 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.421263 31826 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:09.422981 31746 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:23:09.424661 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 68552.2us
I20260709 12:23:09.357090 32371 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: [13.10-13.10]
I20260709 12:23:09.251469  5094 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.365038 31752 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:09.252343 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:09.428397 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 176134us
I20260709 12:23:09.429174 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:09.369256 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2126) took 1747 ms. Trace:
I20260709 12:23:09.431506 31749 rpcz_store.cc:276] 0709 12:23:07.622253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.732155 (+109902us) service_pool.cc:224] Handling call
0709 12:23:09.369012 (+1636857us) spinlock_profiling.cc:243] Waited 855 ms on lock 0x7b54001ed2d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:09.369165 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1965490048}
I20260709 12:23:09.375139 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1953) took 3008 ms. Trace:
I20260709 12:23:09.432363 31671 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:09.420444 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:09.432760 31677 rpcz_store.cc:276] 0709 12:23:06.366825 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.880268 (+1513443us) service_pool.cc:224] Handling call
0709 12:23:07.880553 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:08.749886 (+869333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.374866 (+624980us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b5400254ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:09.375073 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1821251968}
I20260709 12:23:09.233349  4948 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:09.377715 31717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.017s	sys 0.032s
W20260709 12:23:09.435638 31717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.017s	sys 0.032s
I20260709 12:23:09.436285 31744 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:09.380573 31819 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:09.381207 31670 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:09.384469 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:09.438362 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:09.438870 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:09.439244 31819 raft_consensus.cc:2410] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:23:09.384720 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:09.270905  5162 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:09.393157 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 117692us
I20260709 12:23:09.393858 31753 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:23:09.350015  5188 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.456357 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:09.402822 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 48296.7us
I20260709 12:23:09.457008 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:09.354373  4895 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:09.401250 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:09.155488  5228 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.399048  5227 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:09.399472  5141 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:09.442833  5083 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:09.419914  4785 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:09.443518  5200 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:09.452035 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:09.453557 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:09.456562 31821 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:09.280692  5099 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:09.456482 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:09.345836  5074 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:09.404426  5184 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:09.486537 32376 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: [15.13-15.13]
I20260709 12:23:09.460217 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:09.458837 31676 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:09.458837 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:09.466733 31750 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.029s	sys 0.068s
W20260709 12:23:09.467901 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:09.490950 31750 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.615s	user 0.029s	sys 0.068s
I20260709 12:23:09.491142 31753 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 17: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:09.269464  5174 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:09.495333 31753 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:23:09.419269  5220 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:09.463564 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:09.476559 31819 raft_consensus.cc:2368] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15.
I20260709 12:23:09.459456  5188 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:09.466166  5255 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:09.482244 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.458007  5136 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:09.469738 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:09.461643  4873 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
I20260709 12:23:09.467833  5100 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.034s
I20260709 12:23:09.446087 31752 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:09.471026  5155 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:09.497560 31821 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 14. (term mismatch)
I20260709 12:23:09.476150  5200 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:09.483467  5072 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.896s	user 0.009s	sys 0.005s
I20260709 12:23:09.478506  5099 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.502228 31750 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:09.499420  5256 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:23:09.461795  5230 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:23:09.517753 31821 pending_rounds.cc:85] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 13
I20260709 12:23:09.530454 31821 pending_rounds.cc:107] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:23:09.528610 31678 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:09.510697 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.511827 31753 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:09.502826 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:09.519094 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:09.532560 31752 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:09.477758  5083 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:09.476375  5232 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.031s
I20260709 12:23:09.511603  5081 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:09.494206 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:09.475839  5127 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:09.522851  5141 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:09.521541 31819 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 6. Preceding OpId from leader: term: 13 index: 7. (index mismatch)
I20260709 12:23:09.553643  5151 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:09.555912  5221 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:09.555974  5151 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:09.557358  5261 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:23:09.557899  5220 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:09.560804  5209 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:09.561276 31745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:09.561947 31745 raft_consensus.cc:606] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:09.562368 31745 raft_consensus.cc:614] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:09.500429  5187 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.064s
I20260709 12:23:09.535706 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.511973  5166 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:09.486536  5162 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:09.539331 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:09.539752 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:09.519551 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.541661 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:09.581537 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45914.1us
I20260709 12:23:09.583086 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.542323  5096 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.547336  5094 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:09.488960  5074 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.523940  5200 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:09.523986  4739 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:09.527191  5099 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:09.530970 31821 raft_consensus.cc:2889] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:09.567374 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:09.503132  5136 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:09.570657 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 117547us
I20260709 12:23:09.530537  5155 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:09.536787  5152 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:09.582402 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.581233 31753 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
W20260709 12:23:09.593820 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54676.1us
W20260709 12:23:09.614413 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47139.2us
W20260709 12:23:09.614880 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 95390.3us
I20260709 12:23:09.534548  4785 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.550874  5194 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:09.566618  5166 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.558483  5151 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:09.567603  5081 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:09.613994 31826 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:09.593596  5019 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:09.586731  5209 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:09.615211 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:09.598543 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:09.685451 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 70327.4us
I20260709 12:23:09.618288 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:09.636931  5161 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.502s	user 0.013s	sys 0.000s
W20260709 12:23:09.688004 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 69830.5us
I20260709 12:23:09.688640  5161 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.688995 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.689640  5221 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:09.602080  5096 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:09.690447  5221 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:09.691665  5187 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:09.692068  5221 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260709 12:23:09.614051  5136 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:09.692601  5146 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:09.602145  5098 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:09.690397 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.634318  4948 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:09.635020  5152 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:09.695976 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2816) took 1908 ms. Trace:
I20260709 12:23:09.696177  5168 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:09.696398 31672 rpcz_store.cc:276] 0709 12:23:07.787865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.280856 (+1492991us) service_pool.cc:224] Handling call
0709 12:23:09.389209 (+108353us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.400675 (+ 11466us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.695781 (+295106us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b5400229a50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:09.695915 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":248316800}
I20260709 12:23:09.697074 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:09.622670 31751 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:23:09.619166 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:09.604238  5075 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:09.698086  5075 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:09.622372 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:09.698396 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:09.623049 31747 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:23:09.592797  5255 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.036s
I20260709 12:23:09.699059  5075 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.699656 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:09.700245 31751 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:09.624248  5126 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20260709 12:23:09.701128  5126 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:09.701476  5126 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:09.639812  5194 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:09.629168  5125 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.015s
I20260709 12:23:09.645035 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2728) took 1089 ms. Trace:
I20260709 12:23:09.707322 31746 rpcz_store.cc:276] 0709 12:23:08.555141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.683007 (+127866us) service_pool.cc:224] Handling call
0709 12:23:09.644740 (+961733us) spinlock_profiling.cc:243] Waited 580 ms on lock 0x7b54001a5ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:09.644939 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1333392640}
I20260709 12:23:09.708386 31746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
W20260709 12:23:09.645781 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.708952 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:09.708954 31746 raft_consensus.cc:606] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:09.652765  5228 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:09.601790  5004 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.030s
I20260709 12:23:09.616070  5241 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
W20260709 12:23:09.654968 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 381667us
I20260709 12:23:09.634214  5166 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:09.658263  5258 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.052s	user 0.000s	sys 0.002s
I20260709 12:23:09.663779  5148 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.665036  5215 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:09.667320  5155 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.669181 31749 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:23:09.672224  5127 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:09.673830  5074 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:09.621693  4873 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:09.676199  5227 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.13, Last appended by leader: 13, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:09.680225  5083 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:09.681365 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.682114  5235 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:09.682875  4570 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.682811  5081 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:09.689774  5091 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.513s	user 0.002s	sys 0.007s
I20260709 12:23:09.690883 31825 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:23:09.692277  5096 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:09.632635 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:09.860412 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 227829us
I20260709 12:23:09.815909  4873 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:09.861747 31718 leader_election.cc:400] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15.
I20260709 12:23:09.607002  4739 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:09.863511  4739 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.864827  4739 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:09.866889  5209 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:09.628257  5008 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:09.703538 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:09.721400 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 102322us
I20260709 12:23:09.862419 31718 leader_election.cc:403] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:23:09.870662  5151 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 14. Current term is 15.
I20260709 12:23:09.871501  5151 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:09.730814 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:23:09.873049 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 142526us
I20260709 12:23:09.873744  5008 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:09.873946 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:09.874426  5008 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:09.741866 31744 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 12. (index mismatch)
I20260709 12:23:09.696033  5219 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:09.826421 31751 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:09.827016 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.830766 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.833490 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.835543 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:09.720268  5146 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:23:09.955973 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 77163.5us
I20260709 12:23:09.957666 32372 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:09.958361 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:09.818984  5161 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:09.739840  5019 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:09.841738 31749 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:09.829308  5074 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:09.969003 31749 raft_consensus.cc:606] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:09.960523 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:09.828085 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:09.861676  4873 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:09.703284 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:09.969695 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:09.973167 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:09.973553 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 270327us
I20260709 12:23:09.879287 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:09.706806  5141 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.152s
I20260709 12:23:09.879607 31672 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:09.875581  5008 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.909166 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:09.953052 31673 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:23:09.976440 32376 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:09.977658 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:09.954594 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.983428 31676 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:23:09.880458 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 14 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:09.921200 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:09.959990 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:09.739045  5254 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:09.747187  5215 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:09.730113  5134 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.126s
I20260709 12:23:09.967811 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:09.971052 31825 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:09.999889 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:09.969663 31749 raft_consensus.cc:614] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:09.961128  5148 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:09.977677 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 98674.5us
I20260709 12:23:09.959127 31671 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:10.009392 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:09.698402  5168 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:09.990077 32371 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 16: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:09.962468  5096 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.962985  5125 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:09.903076  5227 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:09.989818  5241 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:09.984807 31826 raft_consensus.cc:2376] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17. Re-sending same reply.
I20260709 12:23:10.031118 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:09.992161 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:09.902513  5185 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.728s	user 0.011s	sys 0.002s
I20260709 12:23:09.971166 31677 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:09.855634  5220 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.143s
I20260709 12:23:10.033004 31749 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:23:10.012172 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:10.012337 32372 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:09.905663  5091 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:10.000094 32369 raft_consensus.cc:2410] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:23:10.036586 31744 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
W20260709 12:23:10.038476 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.012092 31673 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:10.040047 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:10.040581 31750 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:23:10.043622 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:10.045526 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:09.904107  5209 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.097s
I20260709 12:23:10.055603 31746 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:10.048401 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:10.056757 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:10.057337 31746 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:09.858206  5188 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:09.856098  4570 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.050321 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:10.053035 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:10.049721 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:09.962141  5133 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.074s
I20260709 12:23:09.980361  5146 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:10.001780  5255 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.993510  5254 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:09.968247  5234 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.080s
I20260709 12:23:09.842303  5126 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.143702 31676 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:23:10.062500 31671 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:10.209355 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:10.210357 31671 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:10.113809 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:10.212365 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:10.115814 31753 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:09.999889  4876 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:10.071226 31822 raft_consensus.cc:2410] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:23:10.172096 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:10.150947 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.113300 32376 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:23:10.112875 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:10.174817 31673 raft_consensus.cc:2410] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 16 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:23:10.073941 31820 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 11: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:10.219561 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 47578.6us
I20260709 12:23:10.220600 31642 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:09.991483  5151 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:10.201612  5255 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:10.203641  5187 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.511s	user 0.008s	sys 0.004s
I20260709 12:23:10.143361  5168 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:09.997782  5074 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.029644  5221 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.223196 31749 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:10.030225  4948 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.012856  4785 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.168s
I20260709 12:23:10.232371 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:10.049373  5019 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.257195 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:10.258373 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:10.200835 31677 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
W20260709 12:23:10.267321 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 153619us
I20260709 12:23:10.270385 31820 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260709 12:23:10.290781 31669 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.657s	user 0.012s	sys 0.052s
I20260709 12:23:10.266203  5241 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:10.272603  4865 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.867s	user 0.013s	sys 0.000s
I20260709 12:23:10.273262  5187 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:10.210210  5254 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:10.275059  5081 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:10.280303 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:10.281363 31824 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:23:10.198405 31746 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:09.979604  5194 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:10.289158 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56880.1us
I20260709 12:23:10.340333 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:10.029971  5016 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.208s
I20260709 12:23:10.287159  5091 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.292320 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:10.216197  5167 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.300218  5164 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:10.312251  5098 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.619s	user 0.003s	sys 0.006s
I20260709 12:23:10.348523 31819 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
W20260709 12:23:10.270825  5260 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
I20260709 12:23:10.054664  5127 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.229s
W20260709 12:23:10.300756 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.047658  5152 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.222s
I20260709 12:23:10.065440  5219 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:10.259003  4776 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.262645  5268 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.314157 31820 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 12. Current term is 13. Ops: [12.7-12.7]
I20260709 12:23:10.338498 31821 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:10.284435  5184 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:10.051008  5096 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:10.291471 31669 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.012s	sys 0.052s
I20260709 12:23:10.048568  5162 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.258s
I20260709 12:23:10.047885  5161 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.262s
I20260709 12:23:10.351181 31676 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [16.10-18.11]
I20260709 12:23:10.279270  5075 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.065s
I20260709 12:23:10.367992 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:10.306656  5221 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.299685  4948 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.337420  4785 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.062s
I20260709 12:23:10.304531  5168 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.360882  4859 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.310838  5019 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.310724  4865 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:10.426178 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.380587  5219 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:10.427241 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.365310  5098 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:10.342489  5016 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:23:10.361647  5091 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.364058  5167 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.299994  5215 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:10.411543 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 177952us
I20260709 12:23:10.442132 31751 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:23:10.425724 31826 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:10.445982 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78088.8us
I20260709 12:23:10.342445  5187 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.054s
I20260709 12:23:10.371018  5164 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.393447  4776 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.397662  5268 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.465227 31827 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:23:10.435557  5219 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:10.406869  5283 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:10.483669 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:10.458395 32369 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
W20260709 12:23:10.503346  5272 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:23:10.504155  5019 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:10.504946  5164 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:10.435504  4859 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.484230  4876 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:10.507931 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 217033us
I20260709 12:23:10.529107 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:10.466359  5284 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:10.494336  5134 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:10.398561  5146 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.464509  5282 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:10.507561 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 166008us
I20260709 12:23:10.507439 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:10.484082 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.476614  5152 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:10.532969 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.577s	user 0.004s	sys 0.009s
I20260709 12:23:10.493352  5188 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:10.486902 32341 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.005s	sys 0.018s
W20260709 12:23:10.717795 32341 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.005s	sys 0.018s
W20260709 12:23:10.533720 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 241165us
I20260709 12:23:10.544485  5286 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:10.551386  5255 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:10.551920  4865 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:10.552166  5280 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:23:10.447242  5241 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:10.551489  5285 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:23:10.552754  5136 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.860s	user 0.011s	sys 0.000s
I20260709 12:23:10.552837  5300 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.750167  5136 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.750532  5300 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.752740  5136 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:10.753098  5300 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:10.517848  5281 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:10.753824  5281 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:10.754849  5281 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.756247  5281 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:10.553998  5074 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:10.507385  5215 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:10.554659  5099 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.933s	user 0.008s	sys 0.002s
W20260709 12:23:10.763056 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2789) took 2053 ms (client timeout 1608 ms). Trace:
I20260709 12:23:10.479142  5290 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.062s
W20260709 12:23:10.557516  5148 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.528s	user 0.009s	sys 0.002s
I20260709 12:23:10.530370 32372 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:10.560243  5268 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:10.561249  5200 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.944s	user 0.008s	sys 0.001s
I20260709 12:23:10.777415  5200 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.780197  5200 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:10.780946 31671 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
W20260709 12:23:10.565179  4739 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.700s	user 0.006s	sys 0.005s
I20260709 12:23:10.565387  5096 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:10.566792  5168 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:10.785406  5126 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
W20260709 12:23:10.570871  4895 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.106s	user 0.007s	sys 0.007s
I20260709 12:23:10.571101  5302 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.787456  5302 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.790818 32374 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
W20260709 12:23:10.572154  4873 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.572s	user 0.010s	sys 0.000s
W20260709 12:23:10.517692  5094 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.916s	user 0.006s	sys 0.006s
I20260709 12:23:10.576176 31826 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:10.517417  4570 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:10.541635  5266 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.279s	user 0.001s	sys 0.000s
I20260709 12:23:10.529497  5273 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:23:10.701695 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 207203us
W20260709 12:23:10.715595 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.004s	sys 0.009s
I20260709 12:23:10.715956 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:10.716888 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 187881us
W20260709 12:23:10.804499 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 88625.1us
I20260709 12:23:10.805181 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:10.718324 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 210967us
I20260709 12:23:10.805894 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:10.719789 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 497396us
I20260709 12:23:10.727067 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:10.735076  5293 raft_consensus.cc:2764] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
W20260709 12:23:10.736529 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.729534  5286 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.724983  5279 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:10.730377 31749 raft_consensus.cc:606] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:10.814992 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:10.744385  5161 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:10.753402 31677 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:10.714916  5134 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:10.720194  5166 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.889s	user 0.010s	sys 0.001s
I20260709 12:23:10.555081  4948 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:10.725706  5164 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:10.763628 31822 rpcz_store.cc:269] 0709 12:23:08.709369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:08.752145 (+ 42776us) service_pool.cc:224] Handling call
0709 12:23:10.760536 (+2008391us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54002080d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:10.762966 (+  2430us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2686897024}
I20260709 12:23:10.763118  5099 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.799093 31678 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 13 index: 7. (term mismatch)
W20260709 12:23:10.793880 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 837941us
I20260709 12:23:10.825757 31678 pending_rounds.cc:85] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 6
I20260709 12:23:10.826066 31678 pending_rounds.cc:107] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 12.7
I20260709 12:23:10.826321 31678 raft_consensus.cc:2889] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:10.802682 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:10.810076 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:10.815330 31749 raft_consensus.cc:614] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:10.732990 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:10.793111  4873 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.790480  5302 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:10.816987 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:10.813584  5221 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:10.818423 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:10.853235 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:10.817953 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:10.814781  5279 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:10.815543  5290 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260709 12:23:10.798422  5273 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:10.859393 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:10.822979  5223 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:10.833046  5277 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.463s	user 0.000s	sys 0.003s
I20260709 12:23:10.833364  5099 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:10.759202  4876 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.843233 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:10.819254 32377 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: [16.14-16.14]
I20260709 12:23:10.819300  5134 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:10.848469 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:10.821990  5152 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:10.821211  4739 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:10.883121  5152 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:10.883224  4739 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:10.884369  5152 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.820587  5096 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:10.853210 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:10.856127 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:10.843030  5167 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:10.830054  5308 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:10.864907 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:10.898598 31744 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:10.834694  5016 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.023s
I20260709 12:23:10.868247 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 831) took 1378 ms. Trace:
W20260709 12:23:10.904518 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51403.9us
I20260709 12:23:10.903960 31752 rpcz_store.cc:276] 0709 12:23:09.489856 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.535819 (+ 45963us) service_pool.cc:224] Handling call
0709 12:23:09.536175 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:09.551753 (+ 15578us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:09.551794 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:09.565063 (+ 13269us) log.cc:844] Serialized 26 byte log entry
0709 12:23:09.705811 (+140748us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:09.705871 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:09.705995 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:10.868011 (+1162016us) raft_consensus.cc:1685] finished
0709 12:23:10.868033 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:10.868174 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11721,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":5714,"thread_start_us":140244,"threads_started":1,"wal-append.queue_time_us":483128}
I20260709 12:23:10.869555 31820 raft_consensus.cc:2410] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 19 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
I20260709 12:23:10.823789  5161 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.854998  5094 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:10.863066  5072 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 5, Time since last communication: 0.009s
I20260709 12:23:10.860056  4570 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.013s
I20260709 12:23:10.872745  4873 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:10.886948  5235 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.049s	user 0.012s	sys 0.001s
I20260709 12:23:10.867942  5223 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.823649  5166 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.859602  5290 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:10.878856  5219 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.853082  5100 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:10.843796  5091 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:10.855216  5282 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:23:10.859361  5188 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:10.895053  5279 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.908393 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:10.957562 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 49312.4us
I20260709 12:23:10.859721  5302 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:23:10.915189 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 59150.6us
I20260709 12:23:10.884617  4739 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.913889  5155 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:10.919303 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:10.942019 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:10.948832 31823 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: [13.9-13.9]
I20260709 12:23:10.972714 31820 raft_consensus.cc:2410] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:23:10.914693  5167 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:10.952210 31818 raft_consensus.cc:2410] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 18 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 9.
I20260709 12:23:10.938125  5008 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:10.866012  5221 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:10.961362  5267 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.568s	user 0.001s	sys 0.000s
I20260709 12:23:10.906952  5096 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:10.941013  5290 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 10 FOLLOWER]: Advancing to term 11
I20260709 12:23:10.940958  5293 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:10.974349 31819 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:23:10.982930 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:10.982954 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:10.988485 31820 raft_consensus.cc:2410] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 16 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:23:10.917757  5094 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:10.992302  5227 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.943s	user 0.011s	sys 0.000s
I20260709 12:23:10.994565 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2748) took 1372 ms. Trace:
I20260709 12:23:11.000388 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:10.984169  5014 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.002949  5014 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.001518  5155 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.004326  5155 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:11.005661  5254 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.658s	user 0.004s	sys 0.008s
I20260709 12:23:10.973292  5125 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.029s
I20260709 12:23:10.979821  5188 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:10.917757  5273 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.997910  5219 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:10.987711  5169 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.006590  5155 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:11.009806  5166 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:11.007438  5188 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.985054  5221 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.012725  5221 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.012406  5134 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.018136  5155 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.000504 31745 rpcz_store.cc:276] 0709 12:23:09.622180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.622694 (+   514us) service_pool.cc:224] Handling call
0709 12:23:09.623052 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:09.639001 (+ 15949us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:09.639265 (+   264us) log.cc:844] Serialized 14 byte log entry
0709 12:23:10.395259 (+755994us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:10.396024 (+   765us) log.cc:844] Serialized 26 byte log entry
0709 12:23:10.396332 (+   308us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:10.396373 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.396459 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:10.960892 (+564433us) spinlock_profiling.cc:243] Waited 4.1 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:10.994227 (+ 33335us) raft_consensus.cc:1685] finished
0709 12:23:10.994333 (+   106us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:10.994488 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10265,"prepare.queue_time_us":598,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":38,"raft.queue_time_us":109555,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":63486,"spinlock_wait_cycles":9441024,"thread_start_us":755515,"threads_started":2,"wal-append.queue_time_us":703566}
I20260709 12:23:10.980814  5302 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:11.000614 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:10.971293  4570 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:11.018199 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:11.023680  4570 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.024214  4570 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:11.024694 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:11.020958 31820 raft_consensus.cc:2410] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:23:10.983911  5008 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.022015 31744 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:11.022089 32376 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:11.026556 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:11.028102 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:11.026521  5081 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:10.992209  5096 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:11.028810 31820 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:11.037333 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:11.008847  5169 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.029461  5136 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:11.031705  5302 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.041119  5169 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:11.019232  5161 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:11.037379  5008 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:11.029294 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:11.030030  5081 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:11.048171  5151 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:11.048712  5081 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:11.050019  5262 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:11.050666  5262 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:11.046082 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.051605  5262 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.055753 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:11.055954 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:11.026790  5286 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.060137 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:11.031188  5094 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.006168  5254 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:10.984788  5167 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.042780  5223 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:11.048719  5151 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:11.077154  5151 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.078004  5161 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.078310 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:11.066843 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1992) took 1030 ms. Trace:
W20260709 12:23:11.079934 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.082536 31677 rpcz_store.cc:276] 0709 12:23:10.036211 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.348877 (+312666us) service_pool.cc:224] Handling call
0709 12:23:10.349204 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.819957 (+470753us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.066504 (+246547us) spinlock_profiling.cc:243] Waited 404 ms on lock 0x7b540036f1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:11.066762 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":929105280}
I20260709 12:23:11.037046  5155 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.087775 32371 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:11.080488 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:11.019429  5293 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.095166  5281 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:11.095945  5281 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:11.097082  5293 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.098862  5295 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:11.099124  5227 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:11.099373  5295 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:11.086205 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.101322 31825 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
W20260709 12:23:11.105682 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.062793  5134 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:11.112152 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.113072 31673 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:11.115193 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:11.116928 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.078670  5151 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:11.080641  5286 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.079985  5169 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:11.124455 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.125293 31673 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:11.101001  5281 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:11.078722  5161 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.127333 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:11.128300  5221 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:11.128715  5221 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:11.128954  5161 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:11.129097  5221 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:11.122802  5297 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.136721 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:11.100402  5295 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.135249  5297 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.122496  4739 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:11.143115 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1987) took 1808 ms. Trace:
I20260709 12:23:11.139005  5254 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:11.240032 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:11.241164 31673 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:11.143582 31669 rpcz_store.cc:276] 0709 12:23:09.334398 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.348531 (+ 14133us) service_pool.cc:224] Handling call
0709 12:23:09.348775 (+   244us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:09.616805 (+268030us) raft_consensus.cc:1534] Early marking committed up to index 8
0709 12:23:10.386456 (+769651us) log.cc:844] Serialized 14 byte log entry
0709 12:23:10.716153 (+329697us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:10.716220 (+    67us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:10.716274 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.142851 (+426577us) spinlock_profiling.cc:243] Waited 268 ms on lock 0x7b540017d9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:11.142899 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.143054 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21921,"prepare.queue_time_us":651893,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":22791,"raft.run_cpu_time_us":97,"raft.run_wall_time_us":98,"spinlock_wait_cycles":615866624,"thread_start_us":1060138,"threads_started":2,"wal-append.queue_time_us":387095}
I20260709 12:23:11.249240 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.251883 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.254573 31669 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:11.259796 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:11.261559 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.262748 31669 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:23:11.265578 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:11.266779 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.267501 31669 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
W20260709 12:23:11.242035  5325 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:23:11.239462  4739 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.270910 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:11.268571 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.140419  5126 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.126609  5286 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.257349  5169 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:11.255751  5162 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.257192  5209 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:11.271301  5295 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:11.273751  4739 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:11.273655  5223 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:11.272197  5281 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:11.275348  5126 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:11.290791  5274 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.942s	user 0.001s	sys 0.001s
W20260709 12:23:11.292217  5289 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.736s	user 0.001s	sys 0.001s
I20260709 12:23:11.293921  4570 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.273911  5152 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:11.299872 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2749) took 1677 ms. Trace:
I20260709 12:23:11.297923 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.306879 31673 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:11.308590 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2842) took 1473 ms. Trace:
I20260709 12:23:11.309216 31675 rpcz_store.cc:276] 0709 12:23:09.835579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.960575 (+124996us) service_pool.cc:224] Handling call
0709 12:23:09.961006 (+   431us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.259600 (+298594us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:10.259632 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:10.275767 (+ 16135us) log.cc:844] Serialized 26 byte log entry
0709 12:23:10.399778 (+124011us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:10.399846 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.301712 (+901866us) spinlock_profiling.cc:243] Waited 263 ms on lock 0x7b54001815d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:11.301748 (+    36us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.301782 (+    34us) raft_consensus.cc:1685] finished
0709 12:23:11.301790 (+     8us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.308488 (+  6698us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":34702,"raft.queue_time_us":121638,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":31443,"spinlock_wait_cycles":605815040,"thread_start_us":123458,"threads_started":1,"wal-append.queue_time_us":197808}
I20260709 12:23:11.300367 31748 rpcz_store.cc:276] 0709 12:23:09.622527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.630361 (+  7834us) service_pool.cc:224] Handling call
0709 12:23:09.707766 (+ 77405us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:09.708771 (+  1005us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:09.708800 (+    29us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:09.712242 (+  3442us) log.cc:844] Serialized 52 byte log entry
0709 12:23:10.310745 (+598503us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:10.382238 (+ 71493us) log.cc:844] Serialized 14 byte log entry
0709 12:23:11.082025 (+699787us) spinlock_profiling.cc:243] Waited 3.25 ms on lock 0x7b18000e52a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:23:11.297209 (+215184us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.297317 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.297381 (+    64us) raft_consensus.cc:1685] finished
0709 12:23:11.299605 (+  2224us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.299804 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":44550,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":67,"raft.queue_time_us":158677,"raft.run_cpu_time_us":381,"raft.run_wall_time_us":29711,"spinlock_wait_cycles":7479680,"thread_start_us":666780,"threads_started":2,"wal-append.queue_time_us":609636}
I20260709 12:23:11.297940  5281 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.295277  5223 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:11.299835  5152 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.328069 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1995) took 1290 ms. Trace:
I20260709 12:23:11.315235 31673 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:23:11.341566 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1994) took 1304 ms. Trace:
I20260709 12:23:11.278815  5169 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.341979 31673 rpcz_store.cc:276] 0709 12:23:10.036848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.314271 (+1277423us) service_pool.cc:224] Handling call
0709 12:23:11.314769 (+   498us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.341158 (+ 26389us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.341496 (+   338us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:11.278560  5162 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.340664 31675 rpcz_store.cc:276] 0709 12:23:10.037120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.326791 (+1289671us) service_pool.cc:224] Handling call
0709 12:23:11.327293 (+   502us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.327571 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.327612 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:11.327678 (+    66us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:11.327729 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.327804 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.327964 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:11.356894 31673 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:11.357126 31675 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:23:11.357667 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1996) took 1320 ms. Trace:
I20260709 12:23:11.357695 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1999) took 1319 ms. Trace:
I20260709 12:23:11.358332 31675 rpcz_store.cc:276] 0709 12:23:10.038427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.356504 (+1318077us) service_pool.cc:224] Handling call
0709 12:23:11.356778 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.357495 (+   717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.357639 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:11.358180 31673 rpcz_store.cc:276] 0709 12:23:10.037416 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.356370 (+1318954us) service_pool.cc:224] Handling call
0709 12:23:11.356703 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.357375 (+   672us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.357600 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:11.360277 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 919) took 1190 ms. Trace:
W20260709 12:23:11.360921 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.280543  5209 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.362699 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 920) took 1153 ms. Trace:
I20260709 12:23:11.360654 31675 rpcz_store.cc:276] 0709 12:23:10.170345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.359447 (+1189102us) service_pool.cc:224] Handling call
0709 12:23:11.359750 (+   303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.359927 (+   177us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.359953 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:11.359998 (+    45us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:11.360031 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.360084 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.360222 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:11.363582 31673 rpcz_store.cc:276] 0709 12:23:10.209489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.360147 (+1150658us) service_pool.cc:224] Handling call
0709 12:23:11.360444 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.361653 (+  1209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.361684 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:11.361736 (+    52us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:11.361772 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.361837 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.362483 (+   646us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:11.367828  5167 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.339883  5223 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:11.329813  5161 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.330291  5281 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:11.342453  5152 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:11.385561 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2843) took 1546 ms. Trace:
I20260709 12:23:11.367014  5155 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.380664  5141 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.340409  5290 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.394318  5290 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:11.391587  5141 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.377696  5081 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.386076 31672 rpcz_store.cc:276] 0709 12:23:09.839616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.975737 (+136121us) service_pool.cc:224] Handling call
0709 12:23:10.014533 (+ 38796us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.015050 (+   517us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:10.015097 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:10.048731 (+ 33634us) log.cc:844] Serialized 26 byte log entry
0709 12:23:10.439797 (+391066us) raft_consensus.cc:1645] Marking committed up to 6
0709 12:23:10.439860 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.439957 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.291503 (+851546us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:11.380448 (+ 88945us) raft_consensus.cc:1685] finished
0709 12:23:11.380475 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.385468 (+  4993us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":155520,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":11421,"spinlock_wait_cycles":36275584,"thread_start_us":161624,"threads_started":1,"wal-append.queue_time_us":338715}
I20260709 12:23:11.406140  5286 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:11.408969  5221 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:11.411168  5094 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.411419  5155 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.411834  5094 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.382305  5161 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.417199 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:11.456822  4739 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.417574 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:11.506362 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2705) took 1482 ms. Trace:
I20260709 12:23:11.465461 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:11.481408 32373 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:23:11.512053 32372 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
W20260709 12:23:11.451985 31826 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2704) took 1428 ms (client timeout 1520 ms). Trace:
I20260709 12:23:11.463696 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 853) took 1393 ms. Trace:
W20260709 12:23:11.506251 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 89148.3us
I20260709 12:23:11.464016 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:11.450371 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1990) took 1414 ms. Trace:
I20260709 12:23:11.470721 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.443564 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2799) took 1483 ms. Trace:
I20260709 12:23:11.434765 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2832) took 1610 ms. Trace:
W20260709 12:23:11.456295 31824 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2768) took 4463 ms (client timeout 2168 ms). Trace:
I20260709 12:23:11.460927 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:11.468678 32374 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 849) took 2121 ms (client timeout 2455 ms). Trace:
I20260709 12:23:11.511202 31827 rpcz_store.cc:276] 0709 12:23:10.024200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.310956 (+286756us) service_pool.cc:224] Handling call
0709 12:23:10.349577 (+ 38621us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.495956 (+146379us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.506081 (+1010125us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x7b54000fad54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:11.506298 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":265119616}
W20260709 12:23:11.513213 31826 rpcz_store.cc:269] 0709 12:23:10.023847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.030822 (+  6975us) service_pool.cc:224] Handling call
0709 12:23:11.451815 (+1420993us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b54004466d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:11.451924 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":905900288}
I20260709 12:23:11.471724 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2755) took 1429 ms. Trace:
I20260709 12:23:11.513721 31671 rpcz_store.cc:276] 0709 12:23:10.035629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.213232 (+177603us) service_pool.cc:224] Handling call
0709 12:23:10.213602 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.781401 (+567799us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.449970 (+668569us) spinlock_profiling.cc:243] Waited 567 ms on lock 0x7b540022b354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:11.450281 (+   311us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1304160128}
I20260709 12:23:11.513700 32378 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:11.514178 31821 rpcz_store.cc:276] 0709 12:23:09.960566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.960680 (+   114us) service_pool.cc:224] Handling call
0709 12:23:11.443380 (+1482700us) spinlock_profiling.cc:243] Waited 358 ms on lock 0x7b54001e9bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:11.443496 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":823343360}
I20260709 12:23:11.432461 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35284 (request call id 42) took 1220 ms. Trace:
I20260709 12:23:11.514329 31678 rpcz_store.cc:276] 0709 12:23:09.824578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.826233 (+  1655us) service_pool.cc:224] Handling call
0709 12:23:09.832015 (+  5782us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.826795 (+994780us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.434501 (+607706us) spinlock_profiling.cc:243] Waited 967 ms on lock 0x7b54001ff4d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:11.434701 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2223166208}
I20260709 12:23:11.515446 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:11.530273 31752 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:11.515729 31744 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:23:11.531512 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:11.524507 31750 rpcz_store.cc:276] 0709 12:23:10.042092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.044445 (+  2353us) service_pool.cc:224] Handling call
0709 12:23:10.044872 (+   427us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:10.321235 (+276363us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:10.339128 (+ 17893us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:10.343716 (+  4588us) log.cc:844] Serialized 52 byte log entry
0709 12:23:10.693773 (+350057us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:10.794091 (+100318us) log.cc:844] Serialized 14 byte log entry
0709 12:23:10.797848 (+  3757us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.471285 (+673437us) spinlock_profiling.cc:243] Waited 259 ms on lock 0x7b54004056d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:11.471354 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.471423 (+    69us) raft_consensus.cc:1685] finished
0709 12:23:11.471532 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.471668 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43880,"prepare.queue_time_us":108307,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":40,"raft.queue_time_us":165023,"raft.run_cpu_time_us":395,"raft.run_wall_time_us":23511,"spinlock_wait_cycles":596414336,"thread_start_us":347542,"threads_started":2,"wal-append.queue_time_us":390623}
I20260709 12:23:11.531558 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:11.531888 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:11.532300 31744 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:23:11.516868 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:11.404807  5081 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.533866 31753 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:23:11.454226  5330 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.087s	user 0.001s	sys 0.000s
W20260709 12:23:11.454391  5298 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.901s	user 0.003s	sys 0.001s
I20260709 12:23:11.545728 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.517168 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:11.445477 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:11.548017 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:11.527436 31749 rpcz_store.cc:276] 0709 12:23:10.212503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.222997 (+ 10494us) service_pool.cc:224] Handling call
0709 12:23:11.432219 (+1209222us) spinlock_profiling.cc:243] Waited 445 ms on lock 0x7b5400403dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdcf853 00007fe901d9d60a 00007fe8fb3237d4 00007fe8fb323762 00007fe8fb323691 00007fe8fb32360c 00007fe8fb3228c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:11.432357 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1023543040}
I20260709 12:23:11.514734 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:11.548461 31746 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:11.548919 31752 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:11.537385 31745 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:11.513898 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:11.469503  5333 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 12:23:11.514935 31824 rpcz_store.cc:269] 0709 12:23:06.992899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:07.207857 (+214958us) service_pool.cc:224] Handling call
0709 12:23:11.456080 (+4248223us) spinlock_profiling.cc:243] Waited 2.64 s on lock 0x7b5400379e54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:11.456187 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6082020736}
I20260709 12:23:11.540611 31826 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
I20260709 12:23:11.555559 31826 pending_rounds.cc:85] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 8
I20260709 12:23:11.555830 31826 pending_rounds.cc:107] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260709 12:23:11.556070 31826 raft_consensus.cc:2889] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:11.515604 32374 rpcz_store.cc:269] 0709 12:23:09.347403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.347903 (+   500us) service_pool.cc:224] Handling call
0709 12:23:11.468437 (+2120534us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b54006dd654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:11.468571 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3316103424}
I20260709 12:23:11.542438 31753 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
W20260709 12:23:11.429168  5271 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.080s	user 0.005s	sys 0.011s
I20260709 12:23:11.563253 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:11.547976 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:11.455575  5279 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:11.513341 32369 rpcz_store.cc:276] 0709 12:23:10.070343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.070534 (+   191us) service_pool.cc:224] Handling call
0709 12:23:11.463495 (+1392961us) spinlock_profiling.cc:243] Waited 229 ms on lock 0x7b540055da54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:11.463608 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":527198336}
I20260709 12:23:11.453359  5209 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.470294  5126 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:11.475067  5302 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.458381  5136 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:11.455183  4570 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.561079 31750 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:11.414167  5094 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.432080  5169 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.566742 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.473281  5286 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:11.457185  5162 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.446214  5141 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.474824  5290 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.453425  5168 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:11.560804  5167 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.020s
I20260709 12:23:11.555120  5096 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 12:23:11.586932  5215 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:11.596079  4776 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.412204  5155 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.509895  4865 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:11.610498 31827 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:23:11.460350  4859 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:11.479871  5014 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:11.633900 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.635177  5209 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.637595 31821 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:11.560766 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.697599 31673 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:11.643740 31671 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:11.698900 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.699225 31675 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 11.
I20260709 12:23:11.699447 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.699703 31673 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:11.700103 31671 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:23:11.701265 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:11.701707  5081 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:11.644781 31749 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:11.703022 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:11.703558 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:11.703963  5151 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.578s	user 0.010s	sys 0.007s
I20260709 12:23:11.704572  5215 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:11.643829 31677 raft_consensus.cc:2410] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
W20260709 12:23:11.651420 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:11.705160  5215 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:11.675048  5290 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.698346 31678 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:11.674402  5232 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.670986  5286 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.635816  5302 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.702488 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.636001  5126 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.632606  5187 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:11.670045 31746 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:11.695466 31827 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:11.596645  5241 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.078s
I20260709 12:23:11.706213  5215 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.671187  5168 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:11.732807 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.738554 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2706) took 1714 ms. Trace:
I20260709 12:23:11.738924 31745 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:23:11.739277 31819 rpcz_store.cc:276] 0709 12:23:10.024524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.349507 (+324983us) service_pool.cc:224] Handling call
0709 12:23:10.385235 (+ 35728us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.995288 (+610053us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.738128 (+742840us) spinlock_profiling.cc:243] Waited 587 ms on lock 0x7b54001671d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:11.738456 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1349131136}
I20260709 12:23:11.740579 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:11.709774 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:11.707677 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.746583 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.713683 31750 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:23:11.706591  5152 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.715397  5126 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.724617 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.753144 31673 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:11.719244 32378 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:11.725031 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:11.755869 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:11.706965 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2844) took 1863 ms. Trace:
I20260709 12:23:11.757304  5126 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:11.715675  5187 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.724370  5169 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:11.757444 31670 rpcz_store.cc:276] 0709 12:23:09.844103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.984532 (+140429us) service_pool.cc:224] Handling call
0709 12:23:10.029707 (+ 45175us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.318144 (+288437us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:10.318204 (+    60us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:10.345059 (+ 26855us) log.cc:844] Serialized 26 byte log entry
0709 12:23:10.487912 (+142853us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:10.487974 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.488080 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.453797 (+965717us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:11.687769 (+233972us) raft_consensus.cc:1685] finished
0709 12:23:11.687792 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.706891 (+ 19099us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":142006,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":10537,"spinlock_wait_cycles":42963328,"thread_start_us":386908,"threads_started":2,"wal-append.queue_time_us":207775}
I20260709 12:23:11.759598 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.758425  5169 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:11.759904  5187 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.760720 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:11.719172  5275 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.348s	user 0.006s	sys 0.002s
I20260709 12:23:11.760946  5169 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.762030 31670 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:11.762317  5169 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:11.720247  5008 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.678s	user 0.017s	sys 0.000s
I20260709 12:23:11.760030 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.766830 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2838) took 1932 ms. Trace:
I20260709 12:23:11.767345 31673 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 11.
I20260709 12:23:11.767596 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.721958  4570 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:23:11.763209  5075 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:11.751005 31752 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:11.712451 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:11.766115 31678 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:23:11.788381 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:11.697975  4776 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:23:11.788399 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 76049.5us
I20260709 12:23:11.717540 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:11.767236 31674 rpcz_store.cc:276] 0709 12:23:09.834470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.834615 (+   145us) service_pool.cc:224] Handling call
0709 12:23:09.834987 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:09.837705 (+  2718us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:09.837739 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:09.838293 (+   554us) log.cc:844] Serialized 26 byte log entry
0709 12:23:10.063651 (+225358us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:10.063706 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:10.139338 (+ 75632us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:11.652703 (+1513365us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:11.763705 (+111002us) raft_consensus.cc:1685] finished
0709 12:23:11.763731 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.766752 (+  3021us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2309,"raft.queue_time_us":3161,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":18164,"spinlock_wait_cycles":31153024,"thread_start_us":222690,"threads_started":1,"wal-append.queue_time_us":468308}
I20260709 12:23:11.718876  5241 consensus_queue.cc:1243] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:11.721176  5081 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.046s
I20260709 12:23:11.791735 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:11.709024  5161 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:11.818329 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:11.809128 31745 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:11.724695 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:11.827105 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:11.830298  5273 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:11.732430  5209 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.816028  5187 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:11.789714  4776 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:11.763598  5281 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.716333  5223 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.823204  4859 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:11.787142  5008 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.802780  5152 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:11.831445  5161 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:11.786324  5125 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.052s
I20260709 12:23:11.838089  5162 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.709355  5232 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.837805  5014 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260709 12:23:11.815502  5282 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.041s
I20260709 12:23:11.859339  5286 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:11.838871 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:11.839358  4865 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:11.828287  5168 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.848384  5290 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:11.866669  5161 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.856174  5187 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:11.860908 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:11.910017 31748 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
W20260709 12:23:11.896097 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57345.1us
I20260709 12:23:11.899027 31673 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:11.912387 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:11.913609  5096 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:11.863193  4859 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.902570 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 1991) took 1866 ms. Trace:
I20260709 12:23:11.915036 31676 rpcz_store.cc:276] 0709 12:23:10.035924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.344503 (+308579us) service_pool.cc:224] Handling call
0709 12:23:10.346614 (+  2111us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:10.394124 (+ 47510us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.902173 (+1508049us) spinlock_profiling.cc:243] Waited 4.17 ms on lock 0x7b54002b0a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:11.902479 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9586560}
I20260709 12:23:11.891465  5014 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:11.903932 31675 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:11.915859 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:11.914214 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:11.953305 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:11.954385 31748 raft_consensus.cc:2368] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17.
I20260709 12:23:11.955475 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:11.960904 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:11.907799  5184 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:11.887272  5162 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.907899  5161 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:11.917279  5187 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:11.945591  5014 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:11.915237  5241 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:11.952616  5168 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:11.936432  4859 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:11.962927  5290 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:11.971014  5310 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:11.971048  5184 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:11.903549  4776 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:11.903337  4865 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:11.969555  5286 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:23:12.012224 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2150) took 1959 ms. Trace:
W20260709 12:23:12.009684 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 318151us
I20260709 12:23:11.982899  5155 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:12.033927 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 119808us
I20260709 12:23:12.074322  5167 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:12.075109  5167 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:12.077936  5167 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:12.080824  5167 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:12.081780  5167 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:12.083209 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:12.084857 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 96429.3us
W20260709 12:23:12.087664 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.035878 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:12.090936 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:12.091840 31748 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:12.091918 31671 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:12.093446 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:12.097043 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:12.099200 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:11.991420  5152 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Leader election won for term 11
I20260709 12:23:12.111409 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:12.093508 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:12.101456 31676 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:11.980695 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2816) took 1020 ms. Trace:
I20260709 12:23:12.103938 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:12.089119 31751 rpcz_store.cc:276] 0709 12:23:10.052977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:10.053388 (+   411us) service_pool.cc:224] Handling call
0709 12:23:10.053626 (+   238us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:10.466563 (+412937us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.011925 (+1545362us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b5400404cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:12.012146 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":892439808}
I20260709 12:23:12.122601 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:12.111647 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:12.117118 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.128749 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 11 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:12.126042 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:12.135861 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:12.142854 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:12.145113 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:12.037438  5241 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:12.151072 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:12.143864 31827 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:12.167846 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:12.118242 31822 rpcz_store.cc:276] 0709 12:23:10.960858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.591835 (+630977us) service_pool.cc:224] Handling call
0709 12:23:11.592308 (+   473us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.592602 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:11.592921 (+   319us) log.cc:844] Serialized 14 byte log entry
0709 12:23:11.980161 (+387240us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:11.980236 (+    75us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:11.980312 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.980451 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:11.980615 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1315,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":122,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":9443,"thread_start_us":386706,"threads_started":1}
I20260709 12:23:12.175443 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:12.150094  4570 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.027s
I20260709 12:23:12.180485  5281 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:12.180848  4570 consensus_queue.cc:1243] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 18.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:23:12.181360  4570 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:12.176750  5223 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:12.150120  5008 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:12.183374 31748 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:12.182814  4570 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:12.183706 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 923) took 1460 ms. Trace:
W20260709 12:23:12.184357 31753 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.043s	sys 0.058s
W20260709 12:23:12.195452 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.044s	sys 0.092s
W20260709 12:23:12.195700 31753 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.044s	sys 0.059s
W20260709 12:23:12.196128 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.045s	sys 0.092s
I20260709 12:23:12.197062 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 995) took 1329 ms. Trace:
I20260709 12:23:12.195564 31669 rpcz_store.cc:276] 0709 12:23:10.723087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.366218 (+643131us) service_pool.cc:224] Handling call
0709 12:23:11.380624 (+ 14406us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:11.381095 (+   471us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.381143 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:11.388006 (+  6863us) log.cc:844] Serialized 26 byte log entry
0709 12:23:11.867056 (+479050us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:11.867098 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.867169 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:12.183294 (+316125us) raft_consensus.cc:1685] finished
0709 12:23:12.183317 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.183605 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":852,"raft.queue_time_us":110,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":5003,"thread_start_us":391206,"threads_started":1,"wal-append.queue_time_us":689957}
I20260709 12:23:12.197553 31826 rpcz_store.cc:276] 0709 12:23:10.867235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.557127 (+689892us) service_pool.cc:224] Handling call
0709 12:23:11.736085 (+178958us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.736305 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.736333 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:11.736379 (+    46us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:11.736414 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.736480 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.196847 (+460367us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b540037ad50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:12.196995 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":410821504}
I20260709 12:23:12.203720 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:12.204684 31671 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:12.208096 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.110282  5096 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.124770  5075 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:12.117309  5155 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.008915  5310 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:12.138738  5307 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:12.227737 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2795) took 3122 ms. Trace:
I20260709 12:23:12.209254  4776 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:12.234895 31818 rpcz_store.cc:276] 0709 12:23:09.091261 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.091831 (+   570us) service_pool.cc:224] Handling call
0709 12:23:12.213367 (+3121536us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54000fbc54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:12.213633 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4197745536}
I20260709 12:23:12.240583 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:12.242167 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:12.246939 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:12.183763  5008 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:12.182502  5281 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:12.224485  4570 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:12.249095  5215 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s	user 0.007s	sys 0.031s
I20260709 12:23:12.313001  5286 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:23:12.243969  5234 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.000s	sys 0.038s
I20260709 12:23:12.318382 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:12.318775  5174 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.322578  5302 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:12.327600  5162 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:12.332535  5339 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:12.333679  5151 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.373169  5348 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:12.389226  5297 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:12.391387  4739 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:12.199632  5184 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:12.407261  5232 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:12.418334  5227 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:12.418982  5169 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:12.419200  5328 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:12.419924  4865 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:12.233819  5334 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.367s	user 0.001s	sys 0.000s
I20260709 12:23:12.215122  5209 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:12.224211  5221 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:23:12.450838  5099 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.945s	user 0.017s	sys 0.039s
W20260709 12:23:12.534107  5099 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.029s	user 0.017s	sys 0.039s
I20260709 12:23:12.238104  5307 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.239400  5155 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:12.461758  5215 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.738s	user 0.007s	sys 0.031s
I20260709 12:23:12.523459  5227 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:12.498839  5339 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:12.499694  5279 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.905s	user 0.013s	sys 0.011s
W20260709 12:23:12.466542  5234 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.870s	user 0.004s	sys 0.041s
I20260709 12:23:12.467188  5286 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:12.491753  5174 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:12.554724  5286 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:12.556901  5174 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:12.463065  5281 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:12.560529  5161 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.582s	user 0.002s	sys 0.007s
I20260709 12:23:12.561188  5161 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:12.506803  5167 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:12.497660  5310 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:12.527201  5169 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:12.563382  5281 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:12.548106  5168 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.552s	user 0.011s	sys 0.000s
I20260709 12:23:12.569939  5286 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:12.515407  5348 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:12.573045  5286 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:12.573159  5348 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:12.577831  5273 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.740s	user 0.012s	sys 0.001s
I20260709 12:23:12.580580  5155 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:12.551441  5339 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:12.552649  5187 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.536s	user 0.009s	sys 0.002s
I20260709 12:23:12.581880  5187 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:12.583559  5169 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:12.584758  5187 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:12.537606  5307 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:12.585894  5227 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:12.588168  5161 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:12.529753  5328 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:12.592834  5328 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:12.601300  5307 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:12.643684 31641 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.860s	user 0.024s	sys 0.034s
W20260709 12:23:12.850364 31641 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.067s	user 0.024s	sys 0.035s
I20260709 12:23:12.853403  5307 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:12.856103  5155 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:23:12.644493 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.862s	user 0.021s	sys 0.037s
W20260709 12:23:12.858012 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.075s	user 0.022s	sys 0.037s
I20260709 12:23:12.858350  5310 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:12.858551 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 1.07597e+06us
I20260709 12:23:12.859447  5310 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:12.861378 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:12.407017 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:12.416313 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:12.408748 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:12.867019 31670 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:23:12.412670 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:12.645372 31672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.789s	user 0.020s	sys 0.030s
I20260709 12:23:12.442610 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:12.870445 31672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.014s	user 0.021s	sys 0.031s
I20260709 12:23:12.871465 31673 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 12.
I20260709 12:23:12.422281 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:12.872290 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2854) took 1536 ms. Trace:
I20260709 12:23:12.873308 31675 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:12.875005  5100 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:12.877277  5348 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:12.458153 31676 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:12.878173  5348 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:12.378168 31677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 924) took 1566 ms (client timeout 1508 ms). Trace:
W20260709 12:23:12.879916 31677 rpcz_store.cc:269] 0709 12:23:10.811489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.085805 (+274316us) service_pool.cc:224] Handling call
0709 12:23:12.377807 (+1292002us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b54000474d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:12.378021 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1232677760}
I20260709 12:23:12.467865 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:12.881717  5348 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:12.882397 31746 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:12.872974 31672 rpcz_store.cc:276] 0709 12:23:11.336286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.401748 (+ 65462us) service_pool.cc:224] Handling call
0709 12:23:11.425855 (+ 24107us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.426188 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:11.856313 (+430125us) log.cc:844] Serialized 14 byte log entry
0709 12:23:12.871496 (+1015183us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:12.871627 (+   131us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:12.871757 (+   130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.871901 (+   144us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.872150 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":565117,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":351,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":268,"thread_start_us":1444125,"threads_started":2}
I20260709 12:23:12.885404 31746 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:12.403769 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:12.425805 31749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.712s	user 0.006s	sys 0.040s
I20260709 12:23:12.448658 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:12.892238 31749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.178s	user 0.006s	sys 0.040s
I20260709 12:23:12.456732 31748 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:12.433948 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:12.185716 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:12.662165 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2775) took 1009 ms. Trace:
W20260709 12:23:12.652884 31747 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.007s	sys 0.031s
I20260709 12:23:12.454152 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:12.387694 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:12.457360 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:12.405055 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2698) took 2997 ms. Trace:
I20260709 12:23:12.998068 31672 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: [16.11-19.13]
I20260709 12:23:12.677660 31827 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:12.679612 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2818) took 1717 ms. Trace:
I20260709 12:23:12.704197  4570 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:12.462313  4785 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.641s	user 0.027s	sys 0.055s
W20260709 12:23:13.016700 31790 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.503s	user 0.009s	sys 0.019s
I20260709 12:23:12.714118  4739 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:13.017807 31744 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.795s	user 0.018s	sys 0.027s
W20260709 12:23:13.018355 31744 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.834s	user 0.018s	sys 0.027s
I20260709 12:23:13.017750  4739 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:12.493167  5011 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:12.501605  5014 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.503s	user 0.008s	sys 0.000s
I20260709 12:23:12.731191  5081 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:12.526773  5094 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.008s	sys 0.036s
I20260709 12:23:12.234361  5096 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.041218 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2777) took 1364 ms. Trace:
I20260709 12:23:13.041889  5096 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:12.514986  5151 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:12.847950  4846 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.045087  4846 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.047838  4846 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:12.857092  5155 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:12.862039  5273 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.002s
I20260709 12:23:12.867143 31674 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:12.863008  5279 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:13.055269  5279 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:13.056931  5155 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:13.057466  5279 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:12.886442 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2159) took 1178 ms. Trace:
I20260709 12:23:12.865378  5184 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:12.950366 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.782s	user 0.013s	sys 0.024s
W20260709 12:23:12.958937 31747 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.983s	user 0.010s	sys 0.032s
I20260709 12:23:12.955367 31751 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
W20260709 12:23:12.955075 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.716s	user 0.015s	sys 0.015s
W20260709 12:23:12.971025 31825 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.806s	user 0.008s	sys 0.023s
W20260709 12:23:12.981002 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 120012us
W20260709 12:23:12.985584 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.010s	sys 0.027s
I20260709 12:23:12.986236 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2778) took 1224 ms. Trace:
I20260709 12:23:12.989884  5352 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:12.985386 31717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.003s	sys 0.022s
I20260709 12:23:12.985216  5100 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:12.996821  4876 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.575s	user 0.007s	sys 0.028s
I20260709 12:23:12.997049 31822 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
W20260709 12:23:12.997248  5290 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.512s	user 0.003s	sys 0.011s
W20260709 12:23:13.080902 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.945s	user 0.015s	sys 0.024s
W20260709 12:23:13.081030 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.884s	user 0.011s	sys 0.027s
W20260709 12:23:13.080709 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.859s	user 0.015s	sys 0.017s
I20260709 12:23:12.997978 31823 rpcz_store.cc:276] 0709 12:23:09.422912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:09.629973 (+207061us) service_pool.cc:224] Handling call
0709 12:23:09.630348 (+   375us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:10.972717 (+1342369us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.404680 (+1431963us) spinlock_profiling.cc:243] Waited 1.32 s on lock 0x7b54001c93d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:12.404942 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3031601536}
I20260709 12:23:12.676388 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2814) took 1831 ms. Trace:
W20260709 12:23:12.985603  5152 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.732s	user 0.019s	sys 0.017s
W20260709 12:23:12.992976 31826 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.713s	user 0.010s	sys 0.008s
I20260709 12:23:12.952155 31753 rpcz_store.cc:276] 0709 12:23:11.652577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.652715 (+   138us) service_pool.cc:224] Handling call
0709 12:23:11.653070 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:11.671919 (+ 18849us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.671970 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:11.672680 (+   710us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.196400 (+523720us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:12.196447 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.196525 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:12.376308 (+179783us) spinlock_profiling.cc:243] Waited 1.18 ms on lock 0x7b4800232758. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:23:12.661085 (+284777us) raft_consensus.cc:1685] finished
0709 12:23:12.661163 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.661886 (+   723us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18358,"raft.queue_time_us":27290,"raft.run_cpu_time_us":8107,"raft.run_wall_time_us":692505,"spinlock_wait_cycles":2712832,"thread_start_us":523320,"threads_started":1,"wal-append.queue_time_us":874440}
I20260709 12:23:13.015125 31752 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:13.015447 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2156) took 1283 ms. Trace:
I20260709 12:23:13.015826  5232 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:13.015717 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:13.016968  4785 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.199s	user 0.028s	sys 0.055s
W20260709 12:23:12.730180  4859 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.723s	user 0.009s	sys 0.002s
I20260709 12:23:12.458461  5008 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:13.085805 31750 rpcz_store.cc:276] 0709 12:23:11.707585 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.433603 (+726018us) service_pool.cc:224] Handling call
0709 12:23:12.987183 (+553580us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:13.101648 31750 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:13.101289  5008 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:13.093425  5152 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.909s	user 0.020s	sys 0.017s
I20260709 12:23:13.103183  5152 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:13.103492 31678 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
W20260709 12:23:13.105291 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 979271us
I20260709 12:23:13.104921  5152 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:13.093502 31826 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.863s	user 0.011s	sys 0.008s
I20260709 12:23:13.108330  5096 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.109153  5096 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.109918 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:13.110930 32377 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:13.111969  5096 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:13.115032 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.042900 31749 rpcz_store.cc:276] 0709 12:23:11.676753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.704641 (+ 27888us) service_pool.cc:224] Handling call
0709 12:23:11.704961 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:11.713199 (+  8238us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:11.713531 (+   332us) log.cc:844] Serialized 14 byte log entry
0709 12:23:12.893333 (+1179802us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:12.893485 (+   152us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:12.893601 (+   116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.040854 (+147253us) spinlock_profiling.cc:243] Waited 7.93 ms on lock 0x7b54002f6cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:13.040908 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.041131 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":660,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":1539,"raft.run_cpu_time_us":745,"raft.run_wall_time_us":43370,"spinlock_wait_cycles":18242432,"thread_start_us":1179032,"threads_started":1,"wal-append.queue_time_us":775757}
I20260709 12:23:13.115899 32373 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:13.117275 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:13.117426 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.118335 32371 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:13.090713 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2166) took 1380 ms. Trace:
I20260709 12:23:13.120167 31751 rpcz_store.cc:276] 0709 12:23:11.710379 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.185441 (+475062us) service_pool.cc:224] Handling call
0709 12:23:13.090527 (+905086us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:12.993001  5293 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.004s	sys 0.008s
W20260709 12:23:13.121291  5293 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.005s	sys 0.009s
I20260709 12:23:12.992259  5307 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:23:12.846915 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.122246 31675 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:23:13.102562 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2168) took 1385 ms. Trace:
I20260709 12:23:13.123277 31750 rpcz_store.cc:276] 0709 12:23:11.716815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.100616 (+1383801us) service_pool.cc:224] Handling call
0709 12:23:13.101268 (+   652us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.102168 (+   900us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.102449 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:12.991067  5338 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.015s	sys 0.014s
I20260709 12:23:12.862560  5310 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:13.061794  5094 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.091s	user 0.008s	sys 0.037s
W20260709 12:23:13.063270 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.063505  5169 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.063407  4846 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.142091 31670 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 12:23:13.141942  4846 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.142479  5353 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:13.143105  5307 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.144649 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2176) took 1028 ms. Trace:
I20260709 12:23:13.144829  4846 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:13.063958  5081 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.064872  5151 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:13.145658  5307 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:13.065083 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 577302us
W20260709 12:23:13.146696  5338 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.815s	user 0.015s	sys 0.015s
I20260709 12:23:13.147862 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:13.148202 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:13.148672 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:13.148584  5169 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.130734 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2160) took 1414 ms. Trace:
I20260709 12:23:13.065701 31818 raft_consensus.cc:2410] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:23:13.065867  5373 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:23:13.077800 31717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.909s	user 0.003s	sys 0.023s
I20260709 12:23:13.070065 31746 rpcz_store.cc:276] 0709 12:23:11.708422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.884125 (+1175703us) service_pool.cc:224] Handling call
0709 12:23:12.885013 (+   888us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:12.885957 (+   944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.886324 (+   367us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:13.000201 31821 rpcz_store.cc:276] 0709 12:23:10.961649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.697923 (+736274us) service_pool.cc:224] Handling call
0709 12:23:11.708099 (+ 10176us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:11.708547 (+   448us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.708587 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:11.709196 (+   609us) log.cc:844] Serialized 26 byte log entry
0709 12:23:12.013271 (+304075us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:12.013316 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:12.013390 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:12.678936 (+665546us) raft_consensus.cc:1685] finished
0709 12:23:12.678983 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.679381 (+   398us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1242,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":112,"thread_start_us":303709,"threads_started":1,"wal-append.queue_time_us":797801}
I20260709 12:23:13.074421 31748 rpcz_store.cc:276] 0709 12:23:11.728014 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.184197 (+456183us) service_pool.cc:224] Handling call
0709 12:23:12.947173 (+762976us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:13.078680 32342 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.013s	sys 0.013s
W20260709 12:23:13.088483 31825 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.973s	user 0.008s	sys 0.025s
I20260709 12:23:13.077816  5011 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.090801  5352 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:13.098846 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 877615us
I20260709 12:23:13.115247  4859 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.121873 31749 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
W20260709 12:23:13.036701 31719 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.759s	user 0.009s	sys 0.039s
W20260709 12:23:13.084758  4876 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.008s	sys 0.029s
I20260709 12:23:13.145107 31750 rpcz_store.cc:276] 0709 12:23:12.116669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.143346 (+1026677us) service_pool.cc:224] Handling call
0709 12:23:13.143796 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.144158 (+   362us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:13.144186 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:13.144350 (+   164us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:13.144408 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.144469 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.144586 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:13.153836 31822 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:13.151142 31752 rpcz_store.cc:276] 0709 12:23:11.708656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.923025 (+1214369us) service_pool.cc:224] Handling call
0709 12:23:12.980974 (+ 57949us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.091374 (+110400us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.123234 (+ 31860us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:13.165668 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2762) took 1642 ms. Trace:
I20260709 12:23:13.081826 31824 rpcz_store.cc:276] 0709 12:23:10.845123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.555609 (+710486us) service_pool.cc:224] Handling call
0709 12:23:11.555916 (+   307us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:11.556133 (+   217us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.556160 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:11.579413 (+ 23253us) log.cc:844] Serialized 26 byte log entry
0709 12:23:11.997604 (+418191us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:11.997659 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:11.997739 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:12.675646 (+677907us) raft_consensus.cc:1685] finished
0709 12:23:12.675715 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:12.676083 (+   368us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":417754,"threads_started":1,"wal-append.queue_time_us":920092}
W20260709 12:23:13.162351 32342 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.845s	user 0.013s	sys 0.014s
W20260709 12:23:13.169572 31719 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.962s	user 0.010s	sys 0.040s
I20260709 12:23:13.174780 31672 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:23:13.174921 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2171) took 1447 ms. Trace:
I20260709 12:23:13.177541 31749 rpcz_store.cc:276] 0709 12:23:11.717839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.121012 (+1403173us) service_pool.cc:224] Handling call
0709 12:23:13.121539 (+   527us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.165186 (+ 43647us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.165558 (+   372us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":9472}
I20260709 12:23:13.155308  5373 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:13.198854 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.173161  4846 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:13.155028  5232 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.167496 31747 rpcz_store.cc:276] 0709 12:23:11.515702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.516103 (+   401us) service_pool.cc:224] Handling call
0709 12:23:11.516412 (+   309us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:11.931469 (+415057us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:11.931519 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:11.959896 (+ 28377us) log.cc:844] Serialized 52 byte log entry
0709 12:23:13.090779 (+1130883us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:13.093383 (+  2604us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.093537 (+   154us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.151513 (+ 57976us) raft_consensus.cc:1685] finished
0709 12:23:13.151533 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.158515 (+  6982us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":115688,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":129,"thread_start_us":1512575,"threads_started":2,"wal-append.queue_time_us":982640}
I20260709 12:23:13.094013  5100 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.095687  5014 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.169083  5338 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:13.220646 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46889.8us
I20260709 12:23:13.233876 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:13.202561  5373 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:13.229743 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 1.06197e+06us
W20260709 12:23:13.238016 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 920450us
W20260709 12:23:13.245589 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 112161us
I20260709 12:23:13.255496  5169 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:13.256021 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:13.260294 32370 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:23:13.260560  5014 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:13.261451  5307 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:13.261900  5307 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.262120  4859 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:13.262279  5307 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:13.262771  5282 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.105s
I20260709 12:23:13.263448 31820 raft_consensus.cc:2368] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12.
I20260709 12:23:13.237948  5096 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:13.264150  5187 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.264784  5096 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.269099  5100 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:13.264870  5187 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:13.271057 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.277530  5232 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:13.275703 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 1001) took 1063 ms. Trace:
I20260709 12:23:13.285861 31818 rpcz_store.cc:276] 0709 12:23:12.212618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.241918 (+ 29300us) service_pool.cc:224] Handling call
0709 12:23:13.275256 (+1033338us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b5400447ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:13.275626 (+   370us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1837043456}
I20260709 12:23:13.270525 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2165) took 1560 ms. Trace:
I20260709 12:23:13.291249 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.271891  5187 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:13.300885  5099 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:13.301888  5099 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.304857  5099 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:13.306053  5355 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.241s	user 0.001s	sys 0.001s
I20260709 12:23:13.305949 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 999) took 1197 ms. Trace:
I20260709 12:23:13.306885 31791 leader_election.cc:400] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17.
I20260709 12:23:13.306834 31826 rpcz_store.cc:276] 0709 12:23:12.108974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.202465 (+ 93491us) service_pool.cc:224] Handling call
0709 12:23:12.202813 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.217339 (+ 14526us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.217381 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.230105 (+ 12724us) log.cc:844] Serialized 26 byte log entry
0709 12:23:13.107589 (+877484us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:13.107669 (+    80us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.107792 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.305547 (+197755us) raft_consensus.cc:1685] finished
0709 12:23:13.305578 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.305846 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14058,"raft.queue_time_us":14254,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":12583,"thread_start_us":876844,"threads_started":1,"wal-append.queue_time_us":943068}
W20260709 12:23:13.308604 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 102776us
I20260709 12:23:13.309613 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:13.309864 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:13.310326 31826 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:13.310506  5011 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:13.316376 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.317387 31824 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:13.321368 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1000) took 1171 ms. Trace:
I20260709 12:23:13.321858 31823 rpcz_store.cc:276] 0709 12:23:12.184437 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.320204 (+1135767us) service_pool.cc:224] Handling call
0709 12:23:13.320673 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.321004 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:13.321033 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:13.321095 (+    62us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:13.321125 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.321177 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.321282 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:13.322755 32371 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:13.323160 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 173197us
I20260709 12:23:13.325374 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.326086  5293 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.328104  5014 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:13.350854  5014 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.322819  5234 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.073s
I20260709 12:23:13.353466  5008 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:13.353914  5014 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:13.325781 31822 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:13.356135 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2161) took 1647 ms. Trace:
I20260709 12:23:13.339622 31826 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:13.356637 31753 rpcz_store.cc:276] 0709 12:23:11.708900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.093871 (+1384971us) service_pool.cc:224] Handling call
0709 12:23:13.094593 (+   722us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:13.095340 (+   747us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:13.095375 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:13.096138 (+   763us) log.cc:844] Serialized 52 byte log entry
0709 12:23:13.205296 (+109158us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:13.205351 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.207298 (+  1947us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.355799 (+148501us) raft_consensus.cc:1685] finished
0709 12:23:13.355837 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.356043 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3780,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":149,"thread_start_us":108563,"threads_started":1,"wal-append.queue_time_us":257482}
I20260709 12:23:13.341612 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2715) took 1236 ms. Trace:
I20260709 12:23:13.350564  5293 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.359409 31825 rpcz_store.cc:276] 0709 12:23:12.105541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.105684 (+   143us) service_pool.cc:224] Handling call
0709 12:23:12.106163 (+   479us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:12.114619 (+  8456us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:12.114688 (+    69us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:12.115327 (+   639us) log.cc:844] Serialized 26 byte log entry
0709 12:23:13.247911 (+1132584us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:13.278159 (+ 30248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.295778 (+ 17619us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.325160 (+ 29382us) raft_consensus.cc:1685] finished
0709 12:23:13.325182 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.341537 (+ 16355us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7894,"raft.queue_time_us":8058,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":14813,"thread_start_us":1046938,"threads_started":1,"wal-append.queue_time_us":1195610}
I20260709 12:23:13.361406  5293 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:13.362188 31825 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:23:13.345726 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:13.365092  5363 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:13.346514  5152 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:13.366972  5338 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:13.367815  5338 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:13.369011  5338 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.295454 31745 rpcz_store.cc:276] 0709 12:23:11.710090 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.448272 (+738182us) service_pool.cc:224] Handling call
0709 12:23:13.270159 (+821887us) spinlock_profiling.cc:243] Waited 376 ms on lock 0x7b54000b81d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:13.270325 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":864341376}
I20260709 12:23:13.347609 31821 raft_consensus.cc:2410] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 14 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 6.
I20260709 12:23:13.371425  4739 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:13.339397  5352 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:13.373136  4776 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.934s	user 0.017s	sys 0.010s
I20260709 12:23:13.383034  5364 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.383543  4739 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:13.384019  5364 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.384217  4739 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.384346  5352 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:13.383178  5365 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.386600  4739 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:13.387111  5361 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.389794  5361 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.392578  5361 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:13.367138 31825 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:23:13.349694 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:13.298851 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.318403 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.398293 32371 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:13.422937 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:13.396814  4776 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:13.387588  5151 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.394840  5364 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:13.445669 31824 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:13.388279  5352 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.424525  5014 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:13.462771 31671 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.198s	user 0.019s	sys 0.010s
I20260709 12:23:13.463336 31671 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:13.464011 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2866) took 1309 ms. Trace:
I20260709 12:23:13.464391 31671 rpcz_store.cc:276] 0709 12:23:12.154554 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.203444 (+ 48890us) service_pool.cc:224] Handling call
0709 12:23:13.463915 (+1260471us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:13.466095 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:13.468500 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:13.469739 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 120183us
I20260709 12:23:13.472390 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:13.475050 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:13.474852  5151 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.476140 31672 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 15 because replica has last-logged OpId of term: 13 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 12 index: 6.
I20260709 12:23:13.456135  5152 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:13.477633  5151 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:13.478277 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 168617us
I20260709 12:23:13.388495  5365 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.480317 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:13.480724  5365 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:13.482523  5293 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
W20260709 12:23:13.486963 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.483666  5293 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.003s
I20260709 12:23:13.487134 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.487802 31825 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:23:13.500090 31822 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:13.496289 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:13.477757  5152 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:13.510762  5011 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:13.511386  5011 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.511840  5011 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:13.516902 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 11 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:13.482615 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:13.521639 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:13.522782 31819 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:13.525980  5014 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.526194  5293 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:13.525368 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:13.518525 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:13.527793 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2766) took 2011 ms. Trace:
I20260709 12:23:13.534781 31750 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:13.502993  4785 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:13.541793 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:13.532483  5152 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:13.551874 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:13.552397  5352 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:13.554737 31751 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:23:13.555446 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2172) took 1837 ms. Trace:
I20260709 12:23:13.555769 31751 rpcz_store.cc:276] 0709 12:23:11.718110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.121028 (+1402918us) service_pool.cc:224] Handling call
0709 12:23:13.528456 (+407428us) spinlock_profiling.cc:243] Waited 1.1 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:23:13.538325 (+  9869us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.555151 (+ 16826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.555383 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2568832}
I20260709 12:23:13.533865  5293 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:13.570089  4895 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:13.570892  4895 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:13.572115  4895 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.565376  5096 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:13.576632  5360 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:13.501634  4776 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.583045 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.584470  4776 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:13.585325 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:13.586961 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:13.595780 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:13.534952 31744 rpcz_store.cc:276] 0709 12:23:11.516796 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.533397 (+ 16601us) service_pool.cc:224] Handling call
0709 12:23:11.533869 (+   472us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:12.006314 (+472445us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:12.061913 (+ 55599us) log.cc:844] Serialized 14 byte log entry
0709 12:23:13.443600 (+1381687us) log.cc:844] Serialized 14 byte log entry
0709 12:23:13.462717 (+ 19117us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:13.463608 (+   891us) log.cc:844] Serialized 52 byte log entry
0709 12:23:13.463811 (+   203us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:13.464079 (+   268us) log.cc:844] Serialized 14 byte log entry
0709 12:23:13.490611 (+ 26532us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.490706 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:13.490761 (+    55us) raft_consensus.cc:1685] finished
0709 12:23:13.490777 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:13.527710 (+ 36933us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32148,"prepare.queue_time_us":58479,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":6978,"raft.queue_time_us":188557,"raft.run_cpu_time_us":743,"raft.run_wall_time_us":89681,"spinlock_wait_cycles":10047872,"thread_start_us":1446998,"threads_started":3,"wal-append.queue_time_us":1225896}
I20260709 12:23:13.602419 31671 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:13.587371  5293 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:13.567844  5184 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:13.541276  5014 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.600867  5081 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:13.629192 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.634958 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:13.641551 31672 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:23:13.644496 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:13.647256 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.647724 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.648882 32378 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:13.650391 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.651034 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.652119 31825 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
I20260709 12:23:13.652388 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.653048 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.653283 31818 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:13.654551 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.654816  5081 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.655411 31818 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:13.655481  5081 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.656800 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.658206  5081 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:13.659415 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 996) took 1551 ms. Trace:
I20260709 12:23:13.659473 31818 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
W20260709 12:23:13.662979 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 166826us
I20260709 12:23:13.675617 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:13.640807 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:13.681439 32378 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:23:13.682523 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.695740 32373 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:13.683665 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.654388 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:13.598940 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49364.4us
I20260709 12:23:13.699842 32377 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 14: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:13.693143 31818 pending_rounds.cc:85] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 10
I20260709 12:23:13.692895 31827 rpcz_store.cc:276] 0709 12:23:12.108108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.166308 (+ 58200us) service_pool.cc:224] Handling call
0709 12:23:12.187304 (+ 20996us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.031976 (+844672us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.659070 (+627094us) spinlock_profiling.cc:243] Waited 490 ms on lock 0x7b54001987d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:13.659347 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1126042880}
I20260709 12:23:13.705498 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:13.703296 31818 pending_rounds.cc:107] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 12:23:13.707026 31818 raft_consensus.cc:2889] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:13.707281 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.708971 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:13.717413 31718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.520s (SENT) [suppressed 7 similar messages]
W20260709 12:23:13.719118  5357 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.674s	user 0.000s	sys 0.004s
W20260709 12:23:13.719859 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 194592us
I20260709 12:23:13.720801 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:13.723443 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:13.724866 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:13.718982 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.722292 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:13.728935 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:13.713922  5307 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.732606 31674 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:13.738300 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.742987 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.745283 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 999) took 2687 ms. Trace:
I20260709 12:23:13.745675 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:13.746744 32376 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:13.747663 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.748739 32377 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:13.749764 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.751235 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.752043  5081 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:13.752398 32377 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:13.752751  5014 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:13.751367 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.753556 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.736159  5184 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:13.759240 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.745718 32370 rpcz_store.cc:276] 0709 12:23:11.052777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.052929 (+   152us) service_pool.cc:224] Handling call
0709 12:23:11.055123 (+  2194us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:13.260949 (+2205826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.699753 (+438804us) spinlock_profiling.cc:243] Waited 2.2 s on lock 0x7b54004cfed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:13.740244 (+ 40491us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5070069504}
I20260709 12:23:13.718341  5011 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.751133 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.767128 32372 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:13.770573 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.744071  4776 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:13.769933 32373 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:13.764926 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:13.781237 31820 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2717) took 1662 ms (client timeout 1673 ms). Trace:
I20260709 12:23:13.788079 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:13.782009 31820 rpcz_store.cc:269] 0709 12:23:12.107455 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.128366 (+ 20911us) service_pool.cc:224] Handling call
0709 12:23:13.769495 (+1641129us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54002675d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:13.769734 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2575998848}
W20260709 12:23:13.804852 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 82668.3us
I20260709 12:23:13.806185 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:13.807478  5014 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:13.807974  5014 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:13.808382  5014 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:13.815615 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.816316 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:13.817459 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.818403 31820 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:13.818754  4785 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:13.819761 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.821358 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.821573  5307 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:13.821456  5014 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.762357  5184 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:13.823016  5184 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:13.817664 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.824927 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.823101 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.826673 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:13.827976 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.827361 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:13.829479 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.830338 32370 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:13.830992 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.826131 31825 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:13.832408 31827 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 15: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:13.829551 31824 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:13.822238 31820 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:13.841980 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:13.841988 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:13.843266 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 68983.1us
I20260709 12:23:13.849298 31671 raft_consensus.cc:2410] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 17 because replica has last-logged OpId of term: 15 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:23:13.856688 32375 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 14: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:13.856941 31744 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
W20260709 12:23:13.852433  5373 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.548s	user 0.003s	sys 0.009s
I20260709 12:23:13.875865 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:13.839309  5360 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:13.882860 31752 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:23:13.880215 32378 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:13.871958  5008 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:13.856099  5352 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 12:23:13.831534  5014 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:13.878779  5373 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:13.869110  5187 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.024s
I20260709 12:23:13.902876 31823 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:13.903810  5187 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:13.903888  5373 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:13.904258  5187 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:13.905441  5187 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:13.908993  5187 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:13.910434 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:13.909396 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:13.916976 31674 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:13.851790  5234 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.040s
W20260709 12:23:13.938185  5353 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.795s	user 0.003s	sys 0.012s
I20260709 12:23:13.943209  5353 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:13.911855  5008 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.949360 31750 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:13.953899 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2824) took 1526 ms. Trace:
I20260709 12:23:13.953680  5152 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:13.954579 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:13.948071  5234 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:13.955112 31744 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:23:13.956146 31822 rpcz_store.cc:276] 0709 12:23:12.427016 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.355946 (+928930us) service_pool.cc:224] Handling call
0709 12:23:13.356376 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.500653 (+144277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.953552 (+452899us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b540037a854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:13.953812 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":314645376}
W20260709 12:23:13.969310 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 98363.9us
I20260709 12:23:13.970592 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:13.971365  5014 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:13.973028  5011 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:13.973150 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:13.971865 32369 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:13.935148  4785 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:23:13.977983 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 100834us
I20260709 12:23:13.979180 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:13.980196 31750 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
W20260709 12:23:13.979943 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 104988us
I20260709 12:23:13.982584 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:13.984788 31748 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 17: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:13.960014  5234 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:13.996218 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 278842us
W20260709 12:23:14.001252 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.001360 31675 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:13.979526  4785 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:14.000159  5184 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.021631 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.025261 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.033222 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:14.035589  5409 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:14.037655 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:14.039513  5408 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:14.040088  5408 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.040493  5408 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:14.042906 31748 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
 from {username='slave'} at 127.0.0.1:35284
W20260709 12:23:14.045418 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.047021 31748 raft_consensus.cc:606] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Received request to transfer leadership to TS b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:14.049227 31748 raft_consensus.cc:614] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:14.051414 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 80437.2us
I20260709 12:23:13.990272  5373 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:14.015445  4785 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:14.059168 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:14.078248 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.057886 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 15 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.017558  5174 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.071s
I20260709 12:23:14.027967  5014 consensus_queue.cc:1243] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:14.090749 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.090580 31674 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
W20260709 12:23:14.078603 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 183797us
I20260709 12:23:14.093375 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:14.098932 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 125932us
I20260709 12:23:14.108819 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2823) took 1682 ms. Trace:
I20260709 12:23:14.115747 31823 rpcz_store.cc:276] 0709 12:23:12.426178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.405477 (+979299us) service_pool.cc:224] Handling call
0709 12:23:13.452707 (+ 47230us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:13.903365 (+450658us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.108446 (+205081us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b540043d5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:14.108728 (+   282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1034818304}
I20260709 12:23:14.090319 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:14.118538 31827 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:14.122380 31745 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 15 because replica has last-logged OpId of term: 13 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 9.
W20260709 12:23:14.092954  5396 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:23:14.127213  5014 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:14.128957  5014 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:14.055138  5011 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
I20260709 12:23:14.131893  5014 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.081259  5353 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.133373  5014 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:14.133255 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:14.128856 31675 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:14.128870 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:14.135876 31818 raft_consensus.cc:2410] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 17 because replica has last-logged OpId of term: 15 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 12.
I20260709 12:23:14.095191  5096 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:14.133308  5353 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:14.143777 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 173330us
I20260709 12:23:14.144941  5353 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:14.145448 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:14.145826  5361 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:14.146417  5361 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:14.146898  5361 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:14.149070  5290 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:14.149184 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:14.149785  5290 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.151329 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:14.153031  5414 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:14.153549  5414 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.153924  5414 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:14.154062  5290 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:14.155876  5184 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:14.124476 31718 connection.cc:256] RPC call timeout handler was delayed by -1.3648680579999564s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1723s for a response
I20260709 12:23:14.158663  5365 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:14.160523 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:14.161929 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:14.162549  5290 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.165562 31675 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:14.165660  5290 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.163270 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:14.179227  5011 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:14.180482  5310 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:14.181005 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:14.225878 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 62723.6us
I20260709 12:23:14.181573  5184 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.187099  5151 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:14.189713  5152 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:14.192597 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.193742 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.090272  5234 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.210848 31750 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:14.217253 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.124311  5363 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:14.233918 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.234114 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:14.174470  5361 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.240887 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.241657 31670 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:14.241756 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:14.118494  5293 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.240962 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.249581 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:14.250274 31673 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:14.248987 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.251731 31672 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:23:14.252960  5363 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:14.236050  5391 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.335s	user 0.003s	sys 0.008s
I20260709 12:23:14.217202  5011 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:14.230101  5407 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.179698  5414 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.263221 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.262176  5361 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:14.280483 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:14.253803  5184 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.266240 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.264017  5152 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:14.266885 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:14.269264 31825 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:14.270581  5293 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:14.280099 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:14.264858 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.234094  5310 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:14.294927 31670 raft_consensus.cc:2410] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 18 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:23:14.287904 31827 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:23:14.290940 31677 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 14
I20260709 12:23:14.292637 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:14.297977 31672 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:14.294133 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 129109us
I20260709 12:23:14.293421 31821 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:14.294018 31669 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:14.248190  5096 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.254807  5412 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:14.301440 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:23:14.309168 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 128263us
I20260709 12:23:14.303596  5187 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:14.303637 31678 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:14.303822  5363 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.322818 31674 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:14.298938  5081 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:14.330981 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:14.328272 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:14.336786  5008 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:14.337273 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.306025  5415 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 12:23:14.338757 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.340224 31670 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:14.341825 31670 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:23:14.342213 31670 pending_rounds.cc:85] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 11
I20260709 12:23:14.342459 31670 pending_rounds.cc:107] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:23:14.301983  5293 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.344134 31670 raft_consensus.cc:2889] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:14.309296  5152 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:14.356623 31678 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
W20260709 12:23:14.335039  5401 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.102s	user 0.005s	sys 0.000s
I20260709 12:23:14.360144 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:14.366519 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 86509.6us
I20260709 12:23:14.283746  5407 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.343189  5353 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
W20260709 12:23:14.379897 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51830.2us
I20260709 12:23:14.317480  5282 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.046s
I20260709 12:23:14.358734  5307 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:14.326049  5096 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:14.389346 31819 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2714) took 2278 ms (client timeout 1834 ms). Trace:
W20260709 12:23:14.391989 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 69476.3us
I20260709 12:23:14.396888 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.376488  5412 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:23:14.395367 31819 rpcz_store.cc:269] 0709 12:23:12.105011 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:12.105187 (+   176us) service_pool.cc:224] Handling call
0709 12:23:14.366778 (+2261591us) spinlock_profiling.cc:243] Waited 1.22 s on lock 0x7b540030e8d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:14.383532 (+ 16754us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2811973888}
I20260709 12:23:14.399391 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:14.411294 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2826) took 1983 ms. Trace:
I20260709 12:23:14.411918 31826 rpcz_store.cc:276] 0709 12:23:12.427557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.357154 (+929597us) service_pool.cc:224] Handling call
0709 12:23:13.357600 (+   446us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:13.358052 (+   452us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:13.358086 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:13.358731 (+   645us) log.cc:844] Serialized 26 byte log entry
0709 12:23:13.599902 (+241171us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:13.599948 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:13.600017 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:14.173122 (+573105us) spinlock_profiling.cc:243] Waited 1.72 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:14.393947 (+220825us) raft_consensus.cc:1685] finished
0709 12:23:14.393971 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.411222 (+ 17251us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4164,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":131,"spinlock_wait_cycles":3951360,"thread_start_us":233005,"threads_started":1,"wal-append.queue_time_us":484031}
I20260709 12:23:14.417068 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.369032  5293 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.418121 31792 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:14.427616 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:14.409457  5412 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.434417 31673 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:23:14.435930  5415 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:14.366153  5011 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.436447  5415 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.436849  5415 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:14.436969  5011 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:14.425341  5008 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.014s
I20260709 12:23:14.440402  5008 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:14.440802  5008 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.441171  5008 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:14.434440  4876 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.444024  4876 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.446182  4876 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:14.447222  5418 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:23:14.447427  5187 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 12:23:14.447950 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.448235  5187 consensus_queue.cc:1243] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:14.448568 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2784) took 1019 ms. Trace:
I20260709 12:23:14.448639  5373 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:14.449196  5373 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:14.449071 31749 rpcz_store.cc:276] 0709 12:23:13.429315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.429525 (+   210us) service_pool.cc:224] Handling call
0709 12:23:13.483906 (+ 54381us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:13.959839 (+475933us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:13.959886 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:13.991398 (+ 31512us) log.cc:844] Serialized 26 byte log entry
0709 12:23:14.108739 (+117341us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:14.108814 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.108935 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:14.448356 (+339421us) raft_consensus.cc:1685] finished
0709 12:23:14.448378 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.448508 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":529528,"raft.queue_time_us":471262,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":4448,"thread_start_us":63949,"threads_started":1,"wal-append.queue_time_us":215496}
I20260709 12:23:14.450170  5373 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.451021  5152 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.451490 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:14.452423  5174 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:14.438269 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:23:14.434671 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 100619us
I20260709 12:23:14.453879 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:14.451548  5373 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:14.466457 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:14.482081 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.487129 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:14.490059 31677 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:23:14.481457 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:14.490689 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 113010us
I20260709 12:23:14.454166  5412 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:14.508260 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 281554us
I20260709 12:23:14.512243 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:14.509109 31672 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:14.514703 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:14.515828 31672 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:14.495064 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:14.479601  5187 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:14.520537 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:14.479756  5366 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:14.536763 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:14.531749 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.520223 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:14.540096 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:14.406219  5310 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.491698  5406 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:14.542716 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.544168 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:14.484268  5421 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:14.551920 32372 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:14.556690  5282 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.574043 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:14.626959  5429 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.534920  5408 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.517769  5184 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:14.534171  4895 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:14.538465  5187 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:14.521708  5414 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:14.560776 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:14.618116 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 151764us
W20260709 12:23:14.667446 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 204109us
I20260709 12:23:14.564930 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:14.674465 32373 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:14.545523  4776 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.096s
I20260709 12:23:14.582091  5290 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:14.553076 31677 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:23:14.629432 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.679231 31744 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:14.562124 31745 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:23:14.571913 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:14.524273  5422 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.104s	user 0.000s	sys 0.001s
I20260709 12:23:14.552129  5360 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:23:14.558774  5420 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.196s	user 0.000s	sys 0.001s
W20260709 12:23:14.560261  5416 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.192s	user 0.000s	sys 0.001s
I20260709 12:23:14.562067 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:14.619865  5096 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:14.616324  5415 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.698486 31677 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 13 index: 10. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:14.698755 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.668633  5081 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:14.657439 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.663141 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:14.552160  5423 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:23:14.666397 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:14.678333 31674 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
W20260709 12:23:14.666105  4785 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.562s	user 0.007s	sys 0.006s
I20260709 12:23:14.689651 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:14.629333  5014 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.057s
I20260709 12:23:14.677215  5187 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:14.670459  5293 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:14.695912 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.575013 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:14.702407 31746 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:14.704785 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.712347 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:14.708618 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 243290us
W20260709 12:23:14.710412 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 226481us
I20260709 12:23:14.707021 31753 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:14.716691 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:14.689172  5409 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.023s
I20260709 12:23:14.718918 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:14.719475 31747 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:14.722620 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:14.717813 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 222837us
I20260709 12:23:14.692770  5428 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:14.724979 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:14.725579 31744 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:14.726728 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:14.727093 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:14.727396 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.691409  5282 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.730785 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.731113 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.736516 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.737088 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.740397 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.722182  5014 consensus_queue.cc:1243] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 16.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:14.716156  5184 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.020s
I20260709 12:23:14.724747  5428 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:14.784036  5428 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:14.784168  5184 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:14.784693  5184 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:14.785560  5184 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.788005  5408 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:14.791474 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2785) took 1300 ms. Trace:
I20260709 12:23:14.792131  5234 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:14.791976 31751 rpcz_store.cc:276] 0709 12:23:13.491537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:13.556810 (+ 65273us) service_pool.cc:224] Handling call
0709 12:23:13.557217 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:13.984121 (+426904us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:13.984153 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:14.011771 (+ 27618us) log.cc:844] Serialized 26 byte log entry
0709 12:23:14.215986 (+204215us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:14.216031 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:14.216105 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:14.791219 (+575114us) raft_consensus.cc:1685] finished
0709 12:23:14.791241 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:14.791408 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":504445,"raft.queue_time_us":424936,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":12132,"thread_start_us":135554,"threads_started":1,"wal-append.queue_time_us":258919}
I20260709 12:23:14.793202  5428 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.732016  5352 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:14.801653  5352 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:14.801771 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.802213  5352 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:14.740628 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:14.752063 31752 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:23:14.803395 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.804025 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.804592 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.805248 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:14.805861 31752 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:14.753203  5008 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.808620  5008 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:14.809573 31673 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:23:14.711277  5429 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.754684  5187 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.756054 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:14.759819  5415 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:14.761194  5011 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:14.763191  5430 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:14.763988 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.764678 31744 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:14.768879 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:14.768970  5014 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:14.769937 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:14.713840  4785 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.771344 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:14.824784 31826 raft_consensus.cc:2410] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 20 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
I20260709 12:23:14.772025 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:14.772593 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:14.772677 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:14.825417 31821 raft_consensus.cc:2376] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16. Re-sending same reply.
I20260709 12:23:14.774981 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:14.825897 31827 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:14.775836  5431 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:14.742487  5414 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:23:14.839178 31745 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:14.776407  5290 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:14.782528 31747 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:14.782819  5432 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:14.783119 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:14.747411  5081 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.822516 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:14.823880 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.774789 31823 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
W20260709 12:23:14.827626 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 71736.3us
I20260709 12:23:14.827916  5184 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:14.831028  5429 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:14.831149 31670 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:14.827519 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 108703us
I20260709 12:23:14.870536 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:14.865801 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:14.832140  4785 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:14.872404 31819 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:14.832939  5411 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:14.880832 31750 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:14.337592 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:14.833664  5430 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:14.819321 31748 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
W20260709 12:23:14.834443 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 93882.7us
I20260709 12:23:14.822957  4776 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:14.835021 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 12 candidate_status { last_received { term: 10 index: 6 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:14.885428 31821 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 12: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:14.837371 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:14.838979 31818 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 15: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:14.813941  5174 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:14.887203 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:14.888096 31823 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:14.870043 31827 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: [17.9-17.9]
W20260709 12:23:14.892416 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 159887us
I20260709 12:23:14.874120 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:14.875888  5361 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:14.883421 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:14.884485 31669 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:14.823896  5152 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:14.907068 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:14.910488 31822 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:23:14.921777 31747 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:14.924578 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:14.873013  5338 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:14.884027  5310 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:14.927024 31749 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:14.881016  5411 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:14.928540  5352 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:14.929888  5431 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.014s
I20260709 12:23:14.944583 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:14.951001 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:14.954427 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:23:14.955719 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51884.3us
I20260709 12:23:14.957800 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:14.886749  5430 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:14.982661 31744 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:14.962833 31750 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:14.955837 32369 raft_consensus.cc:1217] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Deduplicated request from leader. Original: 13.10->[15.11-15.11]   Dedup: 15.11->[]
I20260709 12:23:14.984695 32369 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 12:23:14.955911 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:14.985430 32375 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:14.869912  5014 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:14.899402  5282 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:14.910610  5152 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:14.996628 32374 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 17: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:14.997265 32373 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 14: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:14.999656 31746 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: [16.9-18.10]
I20260709 12:23:14.911595  5361 consensus_queue.cc:1243] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 17.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260709 12:23:15.007650 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 49938.7us
I20260709 12:23:15.008461 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:15.012537 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:14.910486  5364 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:14.929080  5432 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:14.929829  4785 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:14.928952  5412 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.019359 31671 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:15.018452 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:14.966342  5373 raft_consensus.cc:2764] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election decision vote started in defunct term 13: won
I20260709 12:23:15.016326 31745 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:14.942826  5338 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:15.027258 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.011206  5187 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:15.002027  5310 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:15.014230  5152 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:15.031706 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 73852.5us
I20260709 12:23:15.034772  5352 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:15.020121  5406 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.015s
I20260709 12:23:15.044545 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:15.045516 31745 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:15.045822 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:15.047500 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:15.048388 31821 raft_consensus.cc:2410] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:23:15.051271 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:15.052858 32377 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:23:15.002478  5099 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 12:23:15.055747 31752 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:15.022761  4785 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:14.999879  5366 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.035s
I20260709 12:23:15.059368 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:15.059521 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:15.058917 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:15.031744  5234 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:15.017542  5365 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 10.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 12:23:15.068487 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:15.068604 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:15.024652  5282 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 12:23:15.094998 31818 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:15.059845  5436 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:15.096051 31752 raft_consensus.cc:2410] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:23:15.071060  4785 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:15.073474  5414 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election won for term 12
I20260709 12:23:15.065639  4776 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:15.116178 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:15.076710  5234 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.112501  5432 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:15.157517 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:15.162563 31677 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:23:15.164431 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:15.164881 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:15.161487 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:15.174047 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:15.190009 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:15.115886  5352 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.076s
W20260709 12:23:15.191612 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.109750  5187 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:15.114560  5363 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:15.177920  5151 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.173802  5234 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:15.163122  5447 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:15.133607  5428 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:15.210507 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:15.163147  5407 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:15.178385  5414 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:15.221710 32375 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:15.198815  5365 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:15.165990  5360 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.046s
W20260709 12:23:15.232144 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 204982us
I20260709 12:23:15.234894 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:15.236069 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62140.3us
I20260709 12:23:15.236953 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:15.243436 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:15.192885  5415 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:23:15.220001  5364 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:15.245041 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:15.096879  5411 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.116156  5338 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.224354  5432 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:15.257105 31823 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:15.231865  5365 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:15.260582 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:15.261080 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:15.240073  5455 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:15.269397 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50561.7us
I20260709 12:23:15.246498  5373 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.272969 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:15.219163  5363 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.259619  5081 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:15.260758  5014 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:15.259938  5452 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:15.215978  5310 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.229166  5151 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.276854 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:15.279456 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:15.277895  5433 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:15.245467  5447 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.280856  5360 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:15.280645  5364 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 16.14, Last appended by leader: 14, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:15.287787  5446 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:23:15.287297  5414 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.6, Last appended by leader: 6, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.267490  5365 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:15.211359  4776 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:15.333575  5448 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.117s	user 0.000s	sys 0.001s
I20260709 12:23:15.277287 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:15.335315 31749 raft_consensus.cc:2410] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 19 because replica has last-logged OpId of term: 18 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:23:15.336524 31749 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: [15.12-15.12]
I20260709 12:23:15.278323  5411 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:15.340163  5443 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.341s	user 0.002s	sys 0.002s
I20260709 12:23:15.298000  5412 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:15.288429  5441 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.210s	user 0.001s	sys 0.000s
I20260709 12:23:15.343223 31751 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:15.344133  5363 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:15.347720  5447 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:15.305358 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:15.289691  5419 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:15.278151  5081 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:15.309815 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:15.360761 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 61861us
I20260709 12:23:15.307126  5187 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:15.314240  5456 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:23:15.316918  5460 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:23:15.308564  5008 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:15.319939  5151 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:15.321448 31818 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:15.376652 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:15.377070 31677 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:15.377889 31671 pending_rounds.cc:85] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 10
I20260709 12:23:15.378013 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:15.378335 31671 pending_rounds.cc:107] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 17.11
I20260709 12:23:15.324558 31718 leader_election.cc:400] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 11 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12.
I20260709 12:23:15.378674 31671 raft_consensus.cc:2889] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:15.325044  5453 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:15.278488  5373 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.294595  5455 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.309823  5360 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:15.330869 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:15.318413  5432 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.360113 31745 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:15.378156 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:15.298199  5452 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:15.382944 31825 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:23:15.399240 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 74846.7us
I20260709 12:23:15.362898  5282 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:15.407523 31825 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:15.312228  5463 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.092s
I20260709 12:23:15.383960  5456 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:15.420136 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 183232us
I20260709 12:23:15.422897 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.423810 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.424765 31820 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:15.402125  5352 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:15.403585  5436 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260709 12:23:15.427412  5452 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.428064  4776 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:15.429426 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:15.455413  5450 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.438s	user 0.000s	sys 0.002s
I20260709 12:23:15.425853 31825 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.456949  5187 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.403164  5360 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:15.430912  5427 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:23:15.429919 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.425889  5456 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:15.385057  5407 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 12:23:15.439530  5234 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.443276  5365 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.446312  5430 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:15.480165 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57339.5us
I20260709 12:23:15.458927 31825 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:15.464296 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.466077 31748 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:15.384274  5419 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.484445 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.486766 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.416846  5008 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.054s
I20260709 12:23:15.464843  5463 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:15.486578  5081 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.495338  5352 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.503752 31751 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:15.496737  5011 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:15.505950 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.496891  5152 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.498656  5419 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:15.503048 31753 raft_consensus.cc:2368] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17.
I20260709 12:23:15.516345 31676 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 17: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:15.468017  4776 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:15.518930 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.518260  5466 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:15.492311  5447 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:15.458967  5453 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:23:15.534401 31820 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:15.552237 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.517593 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.542815 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:23:15.555339 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.555392 32372 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:15.543125 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:15.556507 32375 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:15.517354  5011 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:15.522989  5081 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:15.528600  5447 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:15.566391 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.547618 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:15.521087  5465 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:15.573079 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:15.557212  5361 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:15.541844  5152 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:15.568222  5430 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:15.588505 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:15.543514 31747 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:15.565485 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.598392 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.567395  5363 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:15.543864  5365 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:15.530252  4785 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.597769 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:15.575670  5453 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:15.601974 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:15.576939  5447 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:15.545249  5455 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:15.613431 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:15.580312  5008 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:15.584698  5361 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:15.612716  5465 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:15.642227 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.616313 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.620806  5011 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.623479 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:15.543290  5360 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.626240 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:15.581107  5436 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:15.639787 32372 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:15.624634 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:15.650141 31669 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:23:15.650404 31823 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:23:15.521095  5352 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.653393 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:15.613979  5430 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:15.611093  5453 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.593743  5373 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:15.645275  5361 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:15.640079  5411 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:15.640370  5477 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:15.666021 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:15.651399  5436 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:15.655309  5363 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:15.654899  5414 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:15.671833 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:15.681707 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.659153  5011 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:15.660380  5365 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.027s
I20260709 12:23:15.657735  5467 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:15.663920  5453 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:15.687371 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:15.639485  5428 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 12:23:15.689150 31678 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:15.683650  4785 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:15.702962  5290 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:15.703482  5290 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:15.704442  5290 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.705538  5290 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:15.709632  5467 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:15.710227  5467 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:15.697022  5373 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:15.711220  5373 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:15.660665  5407 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:15.712265  5407 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:15.713292  5338 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:15.715242 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:15.715904 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.720407  5234 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:15.682466  5440 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.669s	user 0.002s	sys 0.000s
I20260709 12:23:15.712306  5373 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.722980  5234 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.723582  5234 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:15.691525 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49389.6us
I20260709 12:23:15.671046  5477 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:15.725143 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:15.725651  5477 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:15.725878 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:15.659403  5430 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:15.726706  5430 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:15.694233 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:15.729841  5465 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:15.680203  5436 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:15.715776  5428 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:15.731361  5428 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:15.680358  5411 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:15.732474  5428 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.733078  5411 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 14.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.734719  5428 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:15.726056 32374 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:15.692252 31823 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:15.743292 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49133us
I20260709 12:23:15.743292 31825 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:15.749902 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.754004 31678 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:15.757946 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.750736 31792 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:15.760761 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.762220 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:15.762678 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:15.763162 31678 raft_consensus.cc:2410] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:23:15.713840  5407 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.765321 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:15.769963 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:15.761581  5293 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20260709 12:23:15.762995  5187 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election lost for term 14. Reason: could not achieve majority
I20260709 12:23:15.761807  5096 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:15.777755  5432 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:15.793470  5361 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.795259  5234 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:15.799088  5361 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:15.802208 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 946) took 1535 ms. Trace:
I20260709 12:23:15.810330 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:15.769624  5466 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:15.808822 31670 rpcz_store.cc:276] 0709 12:23:14.267253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.419259 (+152006us) service_pool.cc:224] Handling call
0709 12:23:14.430156 (+ 10897us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:14.874082 (+443926us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.801730 (+927648us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b5400181ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:15.802075 (+   345us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":921643392}
I20260709 12:23:15.818670  5414 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:15.819269  5414 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:15.765499  5465 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:23:15.876013 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:15.884135 31744 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:15.892580 31744 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:23:15.782555  4776 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.088s
I20260709 12:23:15.902578 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2835) took 1134 ms. Trace:
I20260709 12:23:15.903227 31822 rpcz_store.cc:276] 0709 12:23:14.768420 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.768577 (+   157us) service_pool.cc:224] Handling call
0709 12:23:15.902281 (+1133704us) spinlock_profiling.cc:243] Waited 90.1 ms on lock 0x7b5400196ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:15.902486 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":207258368}
I20260709 12:23:15.871531  5234 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.876750  5352 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:15.914925 31669 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:15.882006  5473 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:15.931885 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2790) took 1313 ms. Trace:
I20260709 12:23:15.953163  5430 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:15.955776  5430 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:15.956496 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 206666us
I20260709 12:23:15.958776  5456 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.917521  5234 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:15.962945 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2791) took 1305 ms. Trace:
I20260709 12:23:15.973249 31753 rpcz_store.cc:276] 0709 12:23:14.658048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.721339 (+ 63291us) service_pool.cc:224] Handling call
0709 12:23:15.962737 (+1241398us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b54002959d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:15.962867 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1759481088}
I20260709 12:23:15.932539 31746 rpcz_store.cc:276] 0709 12:23:14.618519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:14.760459 (+141940us) service_pool.cc:224] Handling call
0709 12:23:14.983364 (+222905us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:15.007007 (+ 23643us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.931580 (+924573us) spinlock_profiling.cc:243] Waited 222 ms on lock 0x7b54003e3ad0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:15.931774 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":511621376}
I20260709 12:23:15.920548  5361 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:15.982437  5234 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:15.984206  5361 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:15.985539  5361 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:15.991407  5427 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:15.992027  5427 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:15.993064  5427 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:15.996970 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2812) took 1072 ms. Trace:
I20260709 12:23:15.997545 31747 rpcz_store.cc:276] 0709 12:23:14.924312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.540077 (+615765us) service_pool.cc:224] Handling call
0709 12:23:15.540332 (+   255us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.590897 (+ 50565us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.996605 (+405708us) spinlock_profiling.cc:243] Waited 3 ms on lock 0x7b54002968d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:15.996882 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6903040}
I20260709 12:23:16.000021 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:16.021471 31753 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:15.970036  4776 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.022835 31749 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:23:16.030210  5427 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:16.038965 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:16.030529 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:16.043219 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:16.044613 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:16.049053 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.050885 31746 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:16.053189 31677 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: [17.9-17.9]
I20260709 12:23:16.054651 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:16.057740 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:16.056666  5413 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:15.998006  5436 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:16.074703 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.076071 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 13 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:16.077023 32376 raft_consensus.cc:2410] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 17 because replica has last-logged OpId of term: 14 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 11.
I20260709 12:23:16.060199  5412 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.018s
I20260709 12:23:16.078233  5451 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:23:16.078475 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 75570.1us
I20260709 12:23:16.042905  5373 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.021082  5430 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
W20260709 12:23:16.091104 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.091539 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:16.094336 31747 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:23:15.980628  5414 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.078521  5412 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:16.071553  5363 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:16.103657 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:16.128872 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:16.044545  4776 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.130689 31821 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:16.136574 31818 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:16.137126 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:16.138037 31822 raft_consensus.cc:2368] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21.
I20260709 12:23:16.062058  5465 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.045s
I20260709 12:23:16.140086 31826 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:16.147730 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:16.079804  5447 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.098547  5451 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:16.152029 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:23:16.135641  5486 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.165s	user 0.000s	sys 0.001s
W20260709 12:23:16.156989 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.160897  5187 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:16.162053  5414 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:16.154757  5363 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:16.164144  5451 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.166280  4776 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:16.166795 32373 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:16.081096  5413 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.117393  5412 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:16.147874  5014 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.084s
I20260709 12:23:16.283447  5363 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:16.284886  5494 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
I20260709 12:23:16.284251  5412 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.285815  5412 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:16.286343  5493 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.196s	user 0.000s	sys 0.000s
I20260709 12:23:16.287205 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:16.289342  5360 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:16.299849  5428 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.564s	user 0.013s	sys 0.000s
I20260709 12:23:16.139463  5096 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:16.172205 31821 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: []
W20260709 12:23:16.167457  5474 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.051s	user 0.000s	sys 0.001s
W20260709 12:23:16.257658 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 270430us
I20260709 12:23:16.099431  5373 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.169261  5187 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:16.220037  5174 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.187299  5465 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.300638  5456 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:16.318571  5352 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.299906  5436 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:16.318478  5465 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.319408 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:16.319229  5352 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.312158 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:16.320645 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:16.320796 31824 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
W20260709 12:23:16.314292 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 215944us
I20260709 12:23:16.321228 31824 pending_rounds.cc:85] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 12
I20260709 12:23:16.321631 31824 pending_rounds.cc:107] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 12:23:16.321890 31824 raft_consensus.cc:2889] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:16.307628  5011 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.329610 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:16.325836 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:16.331851 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:16.334231 31674 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:23:16.337881  5438 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.338474  5438 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.340991  5438 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:16.341526  5438 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:23:16.342121 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.341719 31674 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:23:16.333473 32373 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:23:16.321116  5096 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:16.347505 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:16.322964  4776 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:16.351353 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 866) took 1333 ms. Trace:
I20260709 12:23:16.355820 31745 rpcz_store.cc:276] 0709 12:23:15.018435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.654605 (+636170us) service_pool.cc:224] Handling call
0709 12:23:15.664709 (+ 10104us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.664945 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:15.664972 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:15.670469 (+  5497us) log.cc:844] Serialized 26 byte log entry
0709 12:23:15.744896 (+ 74427us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:15.744938 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.745003 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:16.351098 (+606095us) raft_consensus.cc:1685] finished
0709 12:23:16.351131 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.351295 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":62387,"threads_started":1,"wal-append.queue_time_us":207593}
W20260709 12:23:16.351936 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 307396us
I20260709 12:23:16.347505 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:16.324710  5436 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.354938 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:16.276806  5467 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.359611 31827 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:23:16.365043 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:16.366009 31673 raft_consensus.cc:2410] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:23:16.344046  5427 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:16.344174  5455 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:16.370105  5467 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:16.372956 31792 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:16.377496 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:16.358558  5429 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:16.388609  5429 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:16.389645  5429 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.366943  5436 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.393189  5436 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:16.395077  5453 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.677s	user 0.005s	sys 0.008s
I20260709 12:23:16.395648  5453 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.355638  4776 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:16.383419 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:16.399008 31820 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:23:16.331799  5360 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.038s
I20260709 12:23:16.363160  5008 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 12:23:16.330595  5174 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.387728 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:16.373401  5427 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:16.400842 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 230403us
I20260709 12:23:16.431059 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:16.431602 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:16.431782 32373 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:16.344712  5011 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:16.404515 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 346635us
I20260709 12:23:16.437583 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:16.405231 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1016) took 1337 ms. Trace:
I20260709 12:23:16.443341 31819 rpcz_store.cc:276] 0709 12:23:15.073688 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.160725 (+ 87037us) service_pool.cc:224] Handling call
0709 12:23:15.904826 (+744101us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:15.914992 (+ 10166us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:15.915026 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:15.915080 (+    54us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:15.915133 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:15.915228 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.405047 (+489819us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b5400196ed0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:16.405170 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1710395648}
I20260709 12:23:16.408913 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2798) took 1665 ms. Trace:
I20260709 12:23:16.445412 31749 rpcz_store.cc:276] 0709 12:23:14.743066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.338197 (+595131us) service_pool.cc:224] Handling call
0709 12:23:15.338467 (+   270us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:16.029559 (+691092us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.408509 (+378950us) spinlock_profiling.cc:243] Waited 681 ms on lock 0x7b54000b8bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:16.408825 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1566602624}
I20260709 12:23:16.447257 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:16.449525 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:16.422127 31827 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:16.399458 31825 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:16.426393  5413 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:16.427083 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:16.430524 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:16.457584 32375 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:16.415686 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:16.459239 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.452256 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:16.461012 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:16.437271  5427 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:16.454109 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:16.455637 31675 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 10. (term mismatch)
I20260709 12:23:16.423312  5414 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:16.468201 31673 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 10 index: 6. Preceding OpId from leader: term: 12 index: 7. (index mismatch)
I20260709 12:23:16.461778 31670 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
W20260709 12:23:16.470813 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.471892 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:16.481899 31749 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:16.453615 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:16.472397 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:16.475281 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:16.485376 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:16.486275 31677 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:23:16.487074 31819 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:16.436214  5436 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:16.476040 31750 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 14 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:16.477927 31675 pending_rounds.cc:85] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 9
I20260709 12:23:16.496007 31675 pending_rounds.cc:107] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 18.10
I20260709 12:23:16.496547 31675 raft_consensus.cc:2889] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:16.440718  5360 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:23:16.507500 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 75976.8us
I20260709 12:23:16.508450 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:16.511014  5465 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:16.512764 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 156849us
I20260709 12:23:16.511432  5352 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:16.500653  5490 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.331s	user 0.004s	sys 0.010s
I20260709 12:23:16.384786  5187 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:16.504185  5516 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.084s	user 0.002s	sys 0.000s
I20260709 12:23:16.561563  5427 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.460381  5096 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.562886 31671 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:23:16.563836  5507 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:16.564422  5507 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.564826  5373 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:16.513325  5008 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:16.482153  5419 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:16.568519  5187 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:16.569715  5011 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:16.499696 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:16.529594  5447 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:16.571501 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 71937.9us
W20260709 12:23:16.546802  5338 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.833s	user 0.016s	sys 0.000s
I20260709 12:23:16.572924 31752 raft_consensus.cc:2410] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:23:16.500979  5360 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:16.574246  5360 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.575506  5360 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:16.578233 31819 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:16.579043  5413 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:16.579376  5452 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:16.579496  5413 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:16.580070  5413 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:16.580477 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:16.583958  5506 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:16.585713  5174 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:16.587317 31670 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 11. Current term is 12. Ops: []
I20260709 12:23:16.548270  5455 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:16.502084  5436 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:16.589057 31745 raft_consensus.cc:2410] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:23:16.529668 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:16.530011 31825 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:16.531981  5414 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:16.601665 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 73059.6us
I20260709 12:23:16.602560 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:16.551802  5151 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:16.617295  5151 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:16.485378 31746 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:23:16.618394  5151 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.619704  5151 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:16.587327  5407 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.547124 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2864) took 1047 ms. Trace:
I20260709 12:23:16.620150 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:16.591435 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:16.620177  5428 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:16.620407  5407 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.623598  5407 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:16.559528 32375 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
W20260709 12:23:16.510512  5477 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.748s	user 0.006s	sys 0.001s
I20260709 12:23:16.623951  4776 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.597113 31826 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:16.604578  5414 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.604920 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:16.605350  5436 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.606113 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:16.627936 31825 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:16.602312  5447 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.604979  5008 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.606351 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:16.612746 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2203) took 1484 ms. Trace:
I20260709 12:23:16.619400 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:16.560204  5473 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.635s	user 0.015s	sys 0.000s
I20260709 12:23:16.625368  4776 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:16.631309 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:16.631132 31752 rpcz_store.cc:276] 0709 12:23:15.129069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.268065 (+138996us) service_pool.cc:224] Handling call
0709 12:23:16.612445 (+1344380us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400294fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:16.612651 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2931368320}
I20260709 12:23:16.619240 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:16.632915 32376 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.503s	user 0.009s	sys 0.000s
I20260709 12:23:16.633332 32376 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:16.620370 31820 rpcz_store.cc:276] 0709 12:23:15.499886 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.590691 (+ 90805us) service_pool.cc:224] Handling call
0709 12:23:15.612327 (+ 21636us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:16.399630 (+787303us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.546776 (+147146us) spinlock_profiling.cc:243] Waited 786 ms on lock 0x7b54003c8cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:16.547039 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1808334720}
I20260709 12:23:16.645452 32369 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:16.645766 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:16.642989  5473 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:16.661959 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:16.638054 31749 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:16.671727 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:16.673197 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:16.673934 31749 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:16.624749  5477 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.684635 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 15 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:16.674979 31748 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:16.668906 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:16.701632 31826 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:23:16.687834 31821 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
W20260709 12:23:16.695880 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.705826 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:16.694563 31825 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:16.627331  5506 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 12:23:16.707207 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:16.699072 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.708514 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:16.643664  5430 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:16.709182 31675 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:16.710861 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:16.713109 31675 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:16.647863  5414 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:16.661208  5174 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:16.720467 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:16.723047 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 77428us
I20260709 12:23:16.724025 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:16.725756 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:16.683619  5338 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:16.729652 31751 raft_consensus.cc:2410] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 18 because replica has last-logged OpId of term: 16 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:23:16.729679 31820 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 13. (term mismatch)
I20260709 12:23:16.730798 31820 pending_rounds.cc:85] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 12
I20260709 12:23:16.731088 31820 pending_rounds.cc:107] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 15.13
I20260709 12:23:16.731388 31820 raft_consensus.cc:2889] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:16.732869 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:16.735503 31670 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:16.736804 31672 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:23:16.699307  5432 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.648572  5008 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.643966  5447 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:16.712898  5436 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:16.683650  5413 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.754691 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:16.726924  5507 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:16.729753  5174 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.735586  5419 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:23:16.759244 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 270450us
I20260709 12:23:16.762025 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:16.766602  5500 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:23:16.740212  5435 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:23:16.768265  5361 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.035s	sys 0.048s
I20260709 12:23:16.802603 31678 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 18 because replica has last-logged OpId of term: 16 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 13.
I20260709 12:23:16.802017 31826 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:16.802717 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:16.785614 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:16.806193 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:16.810164 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:16.811506 31827 raft_consensus.cc:2410] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 18 because replica has last-logged OpId of term: 16 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 9.
W20260709 12:23:16.801667  5486 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:23:16.806845 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:16.827544 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:16.829104 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:16.755975  5187 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:16.800832  5373 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:16.767464  5501 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:16.769778  5174 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:16.841706 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:16.851051 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2211) took 1095 ms. Trace:
I20260709 12:23:16.877759  4776 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:16.877523 31824 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:23:16.877465 31744 rpcz_store.cc:276] 0709 12:23:15.755470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.893860 (+138390us) service_pool.cc:224] Handling call
0709 12:23:15.894313 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:15.950123 (+ 55810us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:15.950160 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:15.967770 (+ 17610us) log.cc:844] Serialized 26 byte log entry
0709 12:23:16.322775 (+355005us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:16.322823 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:16.322883 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:16.850794 (+527911us) raft_consensus.cc:1685] finished
0709 12:23:16.850817 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:16.850975 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23175,"raft.queue_time_us":27199,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":1227,"thread_start_us":303286,"threads_started":1,"wal-append.queue_time_us":490661}
I20260709 12:23:16.879878  4776 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:16.880424  4776 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.880862  4776 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:16.890515  4776 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.843487  5435 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:16.752602  5432 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.821575  5466 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:16.769563  5419 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:16.857009  5451 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:16.806193  5361 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.035s	sys 0.048s
W20260709 12:23:16.855775 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 87696.2us
I20260709 12:23:16.845265  5373 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:16.810377  5465 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.062s
I20260709 12:23:16.910594 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:16.913566 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:16.915966 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:16.893007  4776 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:16.881338  5414 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:16.876065  5455 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:16.908885 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:16.894160  5427 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:16.920215 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.895272  5502 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:16.898959  5466 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:16.905817  5419 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:16.898742  5513 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:16.915122  5373 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:16.899560  5152 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:23:16.890653  5517 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.051s
I20260709 12:23:16.917666  5452 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:16.918306  5435 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:16.921591  5414 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.924511  5456 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:16.961550 31670 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:23:16.913372  5451 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.980229  5465 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:16.964977  5414 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:16.983021 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:16.980811  5465 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:16.958925  5452 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:16.994886 31676 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:16.996704 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:17.004910 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:16.982093  5456 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:17.023680 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 354265us
I20260709 12:23:16.918459  5429 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:16.998005  5452 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:16.990121  5290 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.029s
I20260709 12:23:17.010325  5152 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:17.008630  5413 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:17.008404  5008 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:17.055722 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 450968us
I20260709 12:23:17.061470  5427 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.061638  5432 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:17.072809  5522 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:23:17.042497  5430 raft_consensus.cc:2764] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:23:16.899262  5501 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:16.937633  5502 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:17.058535  5429 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:17.110205 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.120610 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 293264us
I20260709 12:23:17.129127  4776 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.001s
I20260709 12:23:16.979809  5466 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.079493  5517 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:16.954381  5513 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:17.083112  5427 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.136369 31675 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:17.140184 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:17.146713  5432 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.012s
I20260709 12:23:17.147647 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:17.148711  5465 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:17.148850  5531 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 7, Time since last communication: 0.016s
I20260709 12:23:17.150918  5501 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:17.126530  4785 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.153388  5502 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:17.154572  5514 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
I20260709 12:23:17.156908  5521 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.174270 31674 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:17.180094 31677 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:17.192647 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:17.209244 31675 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:17.123989  5187 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:17.209829 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:17.211267 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:17.217209 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:17.219950 31675 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:17.128150  5429 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:17.209600  5525 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.128s	user 0.003s	sys 0.001s
I20260709 12:23:17.192389  5518 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:17.227401 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:17.180028  5517 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:17.244524 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 442800us
I20260709 12:23:17.212029  5352 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.026s
I20260709 12:23:17.200839  4785 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.211817  5533 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 12:23:17.232465  5465 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:17.235216  5502 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:17.244933 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 195255us
I20260709 12:23:17.192692  5456 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 16.13, Last appended by leader: 13, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:17.260914 31677 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:17.243979  5501 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:17.262838 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:17.212337  5521 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.268105  5520 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:17.265206  5096 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:23:17.275450  5529 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:23:17.276417  5513 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:17.279274  5466 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:17.252969  5465 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:17.237010  5360 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.019s
I20260709 12:23:17.299386  5427 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:17.251199  5152 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:17.300587  5152 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.303102  5517 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:17.303035  5373 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:17.282371  5187 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:17.305591  5373 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:17.310361 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:17.282547 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 59462.1us
I20260709 12:23:17.234891  5518 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:17.298705  5519 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.169s	user 0.000s	sys 0.001s
I20260709 12:23:17.264379  5501 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:17.285876  5520 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:17.308952  5096 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:17.319084  5438 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:17.327737  5521 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:17.320407  5360 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 13.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.008s
I20260709 12:23:17.323311  5455 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.329242  5455 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.308660  5465 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:17.331892  5520 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:17.332873 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2208) took 1580 ms. Trace:
I20260709 12:23:17.333266  5520 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:17.267422  5502 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:17.335837  4785 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:17.336367  5502 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:17.328085  5096 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:17.329591  5497 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.928s	user 0.005s	sys 0.000s
I20260709 12:23:17.314262  5539 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.009s
I20260709 12:23:17.324805  5518 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:17.336665 31718 leader_election.cc:400] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21.
I20260709 12:23:17.341125  5174 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:17.343760  5151 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260709 12:23:17.344463  5151 consensus_queue.cc:1243] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 12.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260709 12:23:17.345310 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:17.337749 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 75146.4us
W20260709 12:23:17.338383 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 88833.2us
I20260709 12:23:17.333266 31751 rpcz_store.cc:276] 0709 12:23:15.752641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.762404 (+  9763us) service_pool.cc:224] Handling call
0709 12:23:17.332675 (+1570271us) spinlock_profiling.cc:243] Waited 959 ms on lock 0x7b5400064354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:17.332813 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2206508416}
I20260709 12:23:17.350337 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:17.321506  5501 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:17.325614 31570 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42070 (request call id 224) took 39371 ms (client timeout 15000 ms). Trace:
I20260709 12:23:17.353022 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:17.341877  5436 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.586s	user 0.010s	sys 0.004s
I20260709 12:23:17.363255 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:17.363862 32374 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:17.356298 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:17.368738 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:17.371094 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:17.370245 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:17.384334 32375 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:17.371731  5290 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:17.337146  5455 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:17.372365 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:17.375607 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 874) took 1028 ms. Trace:
I20260709 12:23:17.378448  5447 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:17.374317 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:17.386829 31745 rpcz_store.cc:276] 0709 12:23:16.347062 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.364248 (+ 17186us) service_pool.cc:224] Handling call
0709 12:23:17.375356 (+1011108us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b54004051d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:17.375533 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":479567872}
I20260709 12:23:17.388408  5428 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:17.390002  5435 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260709 12:23:17.404474 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:17.404695 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:17.392549 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:17.345129  5174 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:17.395825  5453 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:17.408099 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:17.411536 31676 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:17.412091 32373 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:17.406621 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:17.409106 32378 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 17: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:17.415059 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:17.383857  5518 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:17.380854  5338 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Stepping down as leader of term 11
I20260709 12:23:17.424173 31676 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:17.425611 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:17.426009 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:17.408752  5447 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:17.437393 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:17.438474 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:17.443042  5455 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.444015 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:17.385301  5414 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.445924 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:17.444131 31745 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:17.422556  5187 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:17.466739 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:17.463094 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:17.422439  5432 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.421012  5338 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:17.470078 31749 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:17.422235  5427 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:17.471937 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:17.475342 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:17.485882 31751 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:17.479393 31752 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 6. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:23:17.487388 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:17.488622 31752 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:23:17.435134  5438 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:17.481199 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:17.485824 31820 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:23:17.494238 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:17.436116  5361 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:17.478127 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:17.483896 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:17.495841 31749 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 17: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:17.420761  5502 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.053s
I20260709 12:23:17.451007  5290 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:23:17.493384 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 132059us
I20260709 12:23:17.431951  5546 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:17.486843  5452 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:17.496953 31745 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:23:17.529067 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:17.497177 31744 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:17.422703  5518 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.499270 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:17.506232 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67837.4us
I20260709 12:23:17.528796 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:17.533039 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:17.521209  5561 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.522918 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:17.467113  5187 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:17.497812 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:17.535892 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 10 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:17.538439 32377 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:17.539178 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:17.549501 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:17.554414 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:17.556944 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:17.491302  5427 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:17.466126  5352 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.063s
I20260709 12:23:17.563026 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:17.458062  5447 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:17.467254  5455 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.518019  5456 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:17.528111  5290 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:17.490530  5432 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.583091 31745 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:17.565644  5373 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:17.567580 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:17.597445 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:17.552421  5187 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:17.560025  5453 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:17.604691 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:17.604653  5534 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:17.605070  5555 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:23:17.606925  5547 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:17.607473  5556 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.039s
I20260709 12:23:17.608415  5550 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:17.609694 31744 raft_consensus.cc:2410] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 15 because replica has last-logged OpId of term: 13 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 10 index: 8.
I20260709 12:23:17.647686  5556 consensus_queue.cc:1243] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 18.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:17.610807  5541 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 12:23:17.611132  5546 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:17.611585 31752 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:17.611532  5549 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:17.614217  5447 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:17.614705  5545 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.246s
I20260709 12:23:17.617321  5373 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:17.639846  5455 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:17.619930  5548 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:23:17.561187  5361 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:17.561955  5427 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:17.621651  5551 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:23:17.595516  5290 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:17.624317  5432 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:17.625093 31822 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:17.584398  5561 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:17.638355 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:17.635507 32370 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:17.645753 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:17.701447  5438 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:17.692639 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 878) took 1149 ms. Trace:
I20260709 12:23:17.692868 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:17.642805  5419 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.682s	user 0.004s	sys 0.008s
I20260709 12:23:17.636720  5293 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:17.707751 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:17.648097 31574 catalog_manager.cc:5697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 10 to 15, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 15 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:17.717267 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:17.631091  5453 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:17.568349  5338 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.692023  5546 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:17.761564 31745 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:17.762177 31753 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 17: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:17.694275  5534 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.764981 31752 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 19 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:23:17.934672 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1026) took 1502 ms. Trace:
I20260709 12:23:17.900187 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:17.765292 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48154.4us
I20260709 12:23:17.943940 31675 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 19 because replica has last-logged OpId of term: 17 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 8.
I20260709 12:23:17.864928 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 953) took 1832 ms. Trace:
I20260709 12:23:17.892243 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:17.844154 31818 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:17.761425 31750 rpcz_store.cc:276] 0709 12:23:16.542875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.543240 (+   365us) service_pool.cc:224] Handling call
0709 12:23:16.543643 (+   403us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.567487 (+ 23844us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:16.567529 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:16.567583 (+    54us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:16.567638 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.692288 (+1124650us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b5400065754. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:17.692362 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.692552 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54186240}
I20260709 12:23:17.860978 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:17.854678  5414 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:17.816769  5518 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:17.726819  5447 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.223s
I20260709 12:23:17.812053 31574 catalog_manager.cc:5697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 11 to 17, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 17 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:17.941898 32369 rpcz_store.cc:276] 0709 12:23:16.429503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.430069 (+   566us) service_pool.cc:224] Handling call
0709 12:23:16.645198 (+215129us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:16.645901 (+   703us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.932092 (+1286191us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54006dd650. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:17.932279 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":473639168}
I20260709 12:23:17.953053 31574 catalog_manager.cc:5697] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 7 to 15, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 15 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:17.957309  5534 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:17.957732  5451 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:17.777712  5419 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:17.960290  5419 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:17.761022  5555 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260709 12:23:17.806679  5538 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.283s	user 0.001s	sys 0.000s
I20260709 12:23:17.963064 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2750) took 1611 ms. Trace:
I20260709 12:23:17.963477 31823 rpcz_store.cc:276] 0709 12:23:16.351317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.351442 (+   125us) service_pool.cc:224] Handling call
0709 12:23:16.366049 (+ 14607us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:16.367698 (+  1649us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:16.588324 (+220626us) spinlock_profiling.cc:243] Waited 17.3 ms on lock 0x7b1800258b80. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:23:16.913212 (+324888us) log.cc:844] Serialized 14 byte log entry
0709 12:23:17.226568 (+313356us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:17.227055 (+   487us) log.cc:844] Serialized 26 byte log entry
0709 12:23:17.227188 (+   133us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:17.230136 (+  2948us) log.cc:844] Serialized 14 byte log entry
0709 12:23:17.238484 (+  8348us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.238563 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:17.394407 (+155844us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b1800258b80. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:17.919373 (+524966us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b1800258b80. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:17.962841 (+ 43468us) raft_consensus.cc:1685] finished
0709 12:23:17.962865 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:17.963000 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24847,"prepare.queue_time_us":38541,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":28541,"raft.run_cpu_time_us":469,"raft.run_wall_time_us":15688,"spinlock_wait_cycles":48528896,"thread_start_us":558383,"threads_started":2,"wal-append.queue_time_us":303058}
I20260709 12:23:17.728348  5096 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.238s
I20260709 12:23:17.968281  5453 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.971288  5546 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.972522  5546 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:17.892724  5569 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:17.975760  5569 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.976425  5559 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:17.976874  5559 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:17.887447  5501 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.501s	user 0.008s	sys 0.002s
I20260709 12:23:17.977850  5501 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:17.977737  5559 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.979152 32369 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 14 index: 11. (index mismatch)
I20260709 12:23:17.979313  5559 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:17.945601 31671 rpcz_store.cc:276] 0709 12:23:16.032153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.033434 (+  1281us) service_pool.cc:224] Handling call
0709 12:23:16.412210 (+378776us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:23:16.412532 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:16.563311 (+150779us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:17.864608 (+1301297us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b54001d5954. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:17.864853 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":347627648}
I20260709 12:23:17.944051 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:17.986186 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:17.366549 31570 rpcz_store.cc:269] 0709 12:22:37.954550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:37.954846 (+   296us) service_pool.cc:224] Handling call
0709 12:23:17.325353 (+39370507us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:10.560126 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:10.560978 (+   852us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:10.561046 (+    68us) write_op.cc:435] Acquired schema lock
0709 12:23:10.561069 (+    23us) tablet.cc:662] Decoding operations
0709 12:23:10.563696 (+  2627us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:10.563830 (+   134us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:10.563868 (+    38us) tablet.cc:685] Acquiring locks for 16 operations
0709 12:23:10.565501 (+  1633us) tablet.cc:701] Row locks acquired
0709 12:23:10.681885 (+116384us) write_op.cc:260] PREPARE: finished
0709 12:23:10.725948 (+ 44063us) write_op.cc:270] Start()
0709 12:23:10.763968 (+ 38020us) write_op.cc:276] Timestamp: P: 1783599790714407 usec, L: 0
0709 12:23:10.766246 (+  2278us) op_driver.cc:348] REPLICATION: starting
0709 12:23:10.831192 (+ 64946us) log.cc:844] Serialized 6366 byte log entry
0709 12:23:11.403427 (+572235us) op_driver.cc:464] REPLICATION: finished
0709 12:23:12.114132 (+710705us) write_op.cc:301] APPLY: starting
0709 12:23:12.134211 (+ 20079us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:12.589774 (+455563us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:12.589916 (+   142us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:12.908610 (+318694us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:12.955175 (+ 46565us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=32,key_file_lookups=32,delta_file_lookups=0,mrs_lookups=0
0709 12:23:12.955294 (+   119us) write_op.cc:312] APPLY: finished
0709 12:23:12.984341 (+ 29047us) log.cc:844] Serialized 148 byte log entry
0709 12:23:13.090989 (+106648us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:13.096769 (+  5780us) write_op.cc:454] Released schema lock
0709 12:23:13.096852 (+    83us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:13.097563 (+   711us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:13.097596 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2138275,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":631296,"apply.run_cpu_time_us":53968,"apply.run_wall_time_us":1031706,"cfile_cache_hit":66,"cfile_cache_hit_bytes":70975,"num_ops":16,"prepare.queue_time_us":196508,"prepare.run_cpu_time_us":19082,"prepare.run_wall_time_us":396285,"raft.queue_time_us":302471,"raft.run_cpu_time_us":68673,"raft.run_wall_time_us":413611,"replication_time_us":619288,"thread_start_us":784306,"threads_started":5,"wal-append.queue_time_us":365772,"wal-append.run_cpu_time_us":135904,"wal-append.run_wall_time_us":2404665}]]}
I20260709 12:23:17.984385 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:18.013688 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 69707.2us
I20260709 12:23:18.038261 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:18.041465 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:18.042650  4776 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:18.043131 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:18.043949  4776 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:18.044445 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:17.928997  5465 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.596s	user 0.012s	sys 0.000s
I20260709 12:23:18.045897 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:18.045280 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 103373us
I20260709 12:23:18.050295 32370 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
W20260709 12:23:18.046036 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 485310us
I20260709 12:23:18.051584 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:18.052109 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:18.052098 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67850us
I20260709 12:23:18.053000 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:17.728899  5545 consensus_queue.cc:1243] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 13.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:17.815598  5452 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:17.931484  5512 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.056350 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:18.057677 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.058816 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:17.980731  5520 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.647s	user 0.009s	sys 0.004s
I20260709 12:23:17.745898  5456 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:23:18.067166 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.069543 32371 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:23:18.058075 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:18.092900 31753 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:18.061628  5583 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:18.050149  5447 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:18.053517  5465 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.055956  5518 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:18.057693  5544 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:18.075338  5512 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.078519  5569 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:18.045552  5534 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:18.083367 31750 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:18.056280  5014 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:18.081110 31574 catalog_manager.cc:5697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 14 to 15, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 15 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:18.064342 31752 raft_consensus.cc:2410] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 18 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:23:18.071224 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.071574  5541 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.153793 31674 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:23:18.070601  5373 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.070251  5419 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.071339 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:18.071947 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:17.786877  5432 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:18.096575 31745 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:18.027709  5096 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:18.097618 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:18.093302 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:18.071642  5361 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:17.728890  5455 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:18.036424  5501 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:18.106997  5456 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.068598 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:18.106359 32370 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 13. Preceding OpId from leader: term: 17 index: 15. (index mismatch)
I20260709 12:23:18.108507  4776 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:17.794639  5293 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.146348 31643 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:18.147624 31819 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:18.055981  5549 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:18.150835 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:18.097535 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:18.189581 31671 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:18.157533 31676 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:18.187723 32369 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:18.184414 31574 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 11 to 14, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 14 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:18.192320 31574 catalog_manager.cc:5697] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 13 to 16, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 16 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } }
I20260709 12:23:18.195156 32374 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:18.095291  5290 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:17.909024  5438 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.173641 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2067) took 1038 ms. Trace:
I20260709 12:23:18.174134 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:18.197460 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 51169.3us
I20260709 12:23:18.201896 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:18.145605  5544 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:18.185853 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:18.203591 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:18.206251 31747 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:18.207013 31676 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:18.157513  5430 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.200304 31669 rpcz_store.cc:276] 0709 12:23:17.135258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.141216 (+  5958us) service_pool.cc:224] Handling call
0709 12:23:17.141621 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:17.218349 (+ 76728us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:17.248442 (+ 30093us) log.cc:844] Serialized 14 byte log entry
0709 12:23:17.436760 (+188318us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:17.438254 (+  1494us) log.cc:844] Serialized 26 byte log entry
0709 12:23:17.438410 (+   156us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:17.438448 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.067979 (+629531us) spinlock_profiling.cc:243] Waited 9.38 ms on lock 0x7b4800080058. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:23:18.173247 (+105268us) spinlock_profiling.cc:243] Waited 58.8 ms on lock 0x7b5400046fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:18.173317 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:18.173377 (+    60us) raft_consensus.cc:1685] finished
0709 12:23:18.173395 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:18.173567 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":51285,"prepare.queue_time_us":29802,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":1981,"raft.queue_time_us":19298,"raft.run_cpu_time_us":1943,"raft.run_wall_time_us":635368,"spinlock_wait_cycles":156769024,"thread_start_us":134329,"threads_started":1,"wal-append.queue_time_us":152259}
I20260709 12:23:18.210407 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:18.210284 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:18.217805 32371 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 11. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:18.219864 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 2869) took 1458 ms. Trace:
I20260709 12:23:18.172019  5465 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:18.220374 31750 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:18.176719  5512 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.221949 31574 catalog_manager.cc:5697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 10 to 18. New cstate: current_term: 18 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:18.220405 31826 rpcz_store.cc:276] 0709 12:23:16.761682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.761801 (+   119us) service_pool.cc:224] Handling call
0709 12:23:18.219668 (+1457867us) spinlock_profiling.cc:243] Waited 31.8 ms on lock 0x7b54001969d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:18.219806 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":73253248}
I20260709 12:23:18.226177 31747 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:18.204131  5429 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:18.226899  5432 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:18.227044  5360 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:18.217913  4785 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:18.212937  5452 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:18.218099  5571 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:18.229027  5453 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:18.232971  5290 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:18.211534 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 118314us
I20260709 12:23:18.200730  5455 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.184854  5456 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.177018  5561 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:18.209946  5544 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:18.287842  5360 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:18.221153  5430 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.221319  5427 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.267269  5081 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:23:18.291177 31574 catalog_manager.cc:5697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 12 to 15, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 15 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:18.296849  5530 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:18.297375  5530 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:18.264322  5571 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.297905  5571 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:18.219607  5451 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:18.165216  5419 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.262893 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:18.221549  5507 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:18.188271  5373 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.318243  5373 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.318683  5477 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:18.289049  5453 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.164716  5361 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.324793  5361 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:18.325184  5518 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:18.331840  5453 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.162441  5541 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:18.256243  5187 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.622s	user 0.010s	sys 0.007s
I20260709 12:23:18.277421 31671 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:18.281150 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:18.332224 32369 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:18.285087 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:18.217347  5310 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.048s
I20260709 12:23:18.217818  5412 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 12:23:18.254364  5429 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.295920 31574 catalog_manager.cc:5697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 12 to 13, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 13 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:18.302834 31673 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 15: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:18.291337  5413 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.013s
I20260709 12:23:18.291774  5360 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.293708  5427 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.298219  5530 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.304198  5293 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:18.305884  5589 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.071s	user 0.003s	sys 0.000s
I20260709 12:23:18.311110  5451 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:18.311787  5590 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:23:18.313009  5419 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.314185  5549 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:18.314350  5455 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.275377  5452 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:18.338734 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53740.2us
I20260709 12:23:18.339126 31669 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:23:18.340653 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:18.400777 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:18.401327 31671 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:18.352069 31574 catalog_manager.cc:5697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:18.354506 31676 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 12:23:18.356257 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 93431.8us
I20260709 12:23:18.334996  5187 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.408308  5432 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:18.408903  5432 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.409895  5187 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.356046  5429 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:18.411082  5428 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:18.411499  5428 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:18.412314 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:18.412437  5428 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 13.7, Last appended by leader: 7, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.413087 32369 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:18.413545  5428 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:18.414418  5419 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:18.393090  5451 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:18.416546 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.417824  5507 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:18.418265  5507 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:18.356025  5453 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:18.418843  5453 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:18.419286  5453 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:18.421633  5008 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:18.407431  5534 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:18.423044  5534 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.423491  5429 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.424736 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 964) took 1809 ms. Trace:
I20260709 12:23:18.425386  5534 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.453480 31820 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:18.410831  5407 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:18.415004  5419 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.426434  5429 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.429425  5544 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.429579 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:18.397976  5452 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.398452  5293 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.485481  5293 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.486981  5544 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.487864  5293 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:18.431774  5586 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.433388  5587 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:18.488796  5586 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.434110 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:18.407882  5414 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:18.490923  5414 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.491377  5586 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.406876  5473 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:18.435825 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:18.437848  5541 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:18.364346  5361 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.390552  5373 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.369326  5559 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:23:18.439414 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.443878 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:18.507459 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 63641.4us
I20260709 12:23:18.508847 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:18.509930  5544 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:18.510705  5544 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.513564  5544 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:18.514230  5544 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:18.516557 31744 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:23:18.517238 31748 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:18.517474 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:18.517949 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:18.518656  5593 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:18.518893 31753 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:23:18.407225 31574 catalog_manager.cc:5697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 11 to 14, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 14 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:18.521155 31574 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 11 to 16, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 16 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:18.452432 31823 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:18.516615 31750 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:18.523381 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.524325 31675 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:23:18.524580  5593 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:18.528977 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:18.529748 31750 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:18.425169 31678 rpcz_store.cc:276] 0709 12:23:16.615341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:16.618790 (+  3449us) service_pool.cc:224] Handling call
0709 12:23:18.424486 (+1805696us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54001810d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:18.424665 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":388503424}
I20260709 12:23:18.447762 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:23:18.470672 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 221476us
W20260709 12:23:18.482288 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.452123  5428 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:23:18.470230  5518 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.017s
I20260709 12:23:18.490424 32370 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:18.483644 32374 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:18.432425  5412 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.483176  5512 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:18.550714 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:18.551496 31750 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:23:18.553360 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:18.554324 31746 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:18.481995 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:18.501644 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:18.445396 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:18.483963  5427 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:18.535810 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:18.537050 31642 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:18.538812 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:18.597440 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:18.543277 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:18.599692 31745 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:18.598786 31748 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:18.601044 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 64117.6us
I20260709 12:23:18.601891 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:18.600625 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 64939.5us
I20260709 12:23:18.603767 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:18.604789 31678 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:18.504097  5559 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.599117 31747 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:18.499253 31671 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:18.599737 31753 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:18.525401  5593 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:18.606037 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:18.606403 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:18.574059 31820 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:23:18.607004 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:18.613468 32372 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:18.613804 32370 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:18.608989 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.614851 31669 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:18.608294 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:18.608534 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:18.608925 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 163616us
I20260709 12:23:18.547243 31574 catalog_manager.cc:5697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 10 to 13, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 13 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:18.610598 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:18.613699 31823 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:18.609377 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:18.624755 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:18.620361 31574 catalog_manager.cc:5697] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 11 to 15. New cstate: current_term: 15 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:18.625504 32374 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:18.535213  5414 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:18.635051 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.638641 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:18.642149 31574 catalog_manager.cc:5697] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 8 to 15, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 15 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:23:18.597360  5578 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:23:18.629895  5559 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:18.651052 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:18.653458  5566 raft_consensus.cc:2764] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:23:18.653779  5081 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:18.654251  5081 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.654661  5081 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:18.655014 31822 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:18.657672 31825 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:18.658859  5419 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:18.599895  5412 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.624614  5407 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:18.662266  5412 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:18.626415  5429 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.533403  5544 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.610823  5512 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.503821  5361 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.556535  4785 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:18.669834 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:18.705031  4785 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.632769  5455 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:18.635721  5545 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:23:18.671908 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.673957 31827 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:23:18.674955  5430 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:18.676570 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:18.677999  5588 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.285s	user 0.000s	sys 0.001s
I20260709 12:23:18.671947 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:18.679929 31819 pending_rounds.cc:85] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 10
I20260709 12:23:18.722954 31819 pending_rounds.cc:107] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 13.11
I20260709 12:23:18.692282 31820 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 12:23:18.693079 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.723243 31676 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:18.502710  5473 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.697062 31574 catalog_manager.cc:5697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 5 to 15, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 15 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:18.609071  5008 raft_consensus.cc:2764] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:23:18.701805 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:18.703693 32372 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
W20260709 12:23:18.734751 32343 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.689s (SENT) [suppressed 3 similar messages]
I20260709 12:23:18.609706  5427 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.678436  5593 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.610288  5432 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:18.708353 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51808.7us
I20260709 12:23:18.714519 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:18.752236 31825 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:18.752866 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:18.753283 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:18.720549 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:18.723277 31819 raft_consensus.cc:2889] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:18.724504 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:18.754961 31818 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:18.732231 31574 catalog_manager.cc:5697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 9 to 10, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 10 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:18.525700  5373 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.735301 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:18.750468 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.764991 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.632766  5571 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:18.771154 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 69435.5us
I20260709 12:23:18.783936 31750 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:18.732760  5455 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:18.768283 31574 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:18.769263 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.792356 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:18.709888  5429 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.765750 31678 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:23:18.773872 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.797973 31818 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 9. (term mismatch)
I20260709 12:23:18.798487 31818 pending_rounds.cc:85] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 8
I20260709 12:23:18.798785 31818 pending_rounds.cc:107] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 12:23:18.799074 31818 raft_consensus.cc:2889] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:18.768424 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:18.793866 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:18.799943 31825 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:18.798079 31677 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:18.632496  5507 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.798650 31669 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 12.
I20260709 12:23:18.801376 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.802806 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.804183 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:18.788646 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:18.811039 31750 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:18.770399 31672 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:18.817178 32371 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:23:18.717085  5411 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 12:23:18.823652 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.805490 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.824462 31678 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:18.825349 31669 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 19: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:18.832613 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.833578 31677 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:18.835865 31677 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
W20260709 12:23:18.838181 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 227660us
I20260709 12:23:18.839963 31678 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:18.833571 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:18.842346 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.847756 31672 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:18.787698  5430 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:18.850343 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:18.840085 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:18.846923 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:18.787256  5597 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:18.853679 31669 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 17 index: 14. (index mismatch)
I20260709 12:23:18.871719 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:18.829779  5517 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:18.875255 31669 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:18.879026 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:18.882493 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:18.887401 31792 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:18.889004 31826 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 7. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:23:18.845283  5518 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:18.843273  5536 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:18.895953 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 853) took 1164 ms. Trace:
I20260709 12:23:18.896059 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:18.897115 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:18.855890  5507 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:18.855480  5011 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:18.897944 32375 rpcz_store.cc:276] 0709 12:23:17.721738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.948253 (+226515us) service_pool.cc:224] Handling call
0709 12:23:17.950035 (+  1782us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:17.950920 (+   885us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:17.950952 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:17.985858 (+ 34906us) log.cc:844] Serialized 52 byte log entry
0709 12:23:18.263902 (+278044us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:18.263957 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.264033 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:18.677236 (+413203us) spinlock_profiling.cc:243] Waited 6.35 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:18.875119 (+197883us) raft_consensus.cc:1685] finished
0709 12:23:18.875141 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:18.885934 (+ 10793us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":113,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":79,"spinlock_wait_cycles":14610176,"thread_start_us":191555,"threads_started":1,"wal-append.queue_time_us":338711}
I20260709 12:23:18.901275 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:18.907853 31825 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:18.904749 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 295450us
I20260709 12:23:18.848939  5563 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.037s
I20260709 12:23:18.903621 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:18.871591  5411 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:18.856019  4785 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:18.887902  5517 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.887539  5566 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:18.920149 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:18.850193  5310 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.055s
I20260709 12:23:18.896776  5502 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:18.931022 32375 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:18.931439 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:18.932065 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:18.932276 31826 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:23:18.907004 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:18.937184 32375 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:18.911103  5465 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:18.906661  5011 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:18.967866 32370 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:18.949792 31824 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:18.885972  5576 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.040s
I20260709 12:23:18.921092  5505 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:18.915572  5411 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.879364  5430 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.923005  5518 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:18.925117  5517 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:18.903257  5414 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.050s
I20260709 12:23:18.985210 31820 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:18.985893  5361 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:18.987182  5011 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.988613  5011 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:18.988992  5473 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:18.989689  5473 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:18.990140  5473 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:18.991098  5473 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.993324 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:18.953043  5465 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:18.929221  4785 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 12:23:18.894148  5293 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.042s
I20260709 12:23:18.911335  5187 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:23:18.909521  5536 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:18.983484 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:18.941059  5455 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 10, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:18.936333  5360 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:18.975929  5407 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:18.990471 31746 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:18.999135 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:18.983433  5545 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:19.038877  5545 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.039316  5545 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:18.979974  5505 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.043493  5505 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:19.028733 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:19.047662 32375 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:23:19.030409 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 295657us
I20260709 12:23:18.982165  5411 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:18.940215  5008 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.057s
I20260709 12:23:19.004050  5544 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:19.023455  5465 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:18.995635  5512 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:19.022827  5373 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:19.050673 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.068871 31669 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260709 12:23:19.069635 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:19.080600 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 292419us
I20260709 12:23:19.049422  5360 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:19.062191  5599 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.263s	user 0.000s	sys 0.001s
W20260709 12:23:19.076859  5601 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.250s	user 0.001s	sys 0.000s
I20260709 12:23:19.125331  5081 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:19.095130  5429 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:19.076243  5361 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.123036  5571 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:19.123589 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.124801  5607 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.126711  5361 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.127620  5607 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.129586  5361 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:19.130014  5607 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:19.093909  5518 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:19.093735  5407 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:19.131105  5532 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:19.131655  5532 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:19.132510  5532 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:19.133805  5532 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:19.093011  5512 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:23:19.134934  5081 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:19.136267  5603 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.219s	user 0.000s	sys 0.001s
I20260709 12:23:19.138057  5559 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:19.138893  5531 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:19.139353  5531 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:19.139461 31670 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:19.140209  5531 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:19.140816  5559 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:19.142695 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:19.144409 31672 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: []
W20260709 12:23:19.145462  5453 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.726s	user 0.010s	sys 0.001s
I20260709 12:23:19.114305  5430 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:19.149252 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:19.106383  5596 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.253s	user 0.001s	sys 0.001s
I20260709 12:23:19.093937  5360 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:19.163158 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:19.165956 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.145891  5453 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.177755 31822 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:19.178898  5453 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:19.192727  5411 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:19.194581 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.164681  5438 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:23:19.216662 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 313088us
I20260709 12:23:19.219908 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:19.175884  5506 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:19.161608  5571 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:23:19.202564  5536 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:19.237703 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 153419us
I20260709 12:23:19.250452 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:19.251771 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 362167us
I20260709 12:23:19.272557 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:19.273671 31747 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:19.279347 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:19.280406 31750 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:19.291397 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:19.175477  5174 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.068s
I20260709 12:23:19.301707  5427 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:19.302294  5174 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:19.302837  5174 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:19.303866  5174 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:19.305158  5174 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:19.292220  5452 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:19.242403  5187 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:19.288849  5506 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:19.307585  5187 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:19.308096  5187 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:19.308269 31827 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:19.307242  5590 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:23:19.310019 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:19.311939 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:19.315733 31821 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:19.323448 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:19.314854 31827 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:23:19.328439 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2917) took 2182 ms. Trace:
I20260709 12:23:19.328912 31670 rpcz_store.cc:276] 0709 12:23:17.131178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.131446 (+   268us) service_pool.cc:224] Handling call
0709 12:23:18.426095 (+1294649us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.139917 (+713822us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.141414 (+  1497us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b54001810d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:19.306415 (+165001us) spinlock_profiling.cc:243] Waited 1.29 s on lock 0x7b54001810d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:19.313739 (+  7324us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4616525568}
I20260709 12:23:19.329911 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:19.331229 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:19.332800 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:19.333446 31827 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:19.334268 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:19.335311 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44017.8us
I20260709 12:23:19.335939 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:19.343801 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 172841us
I20260709 12:23:19.348356  5586 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:23:19.351645 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 437124us
I20260709 12:23:19.353049 31827 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:19.348753 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:19.362015 31745 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:19.365773 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:19.367089 31827 raft_consensus.cc:2410] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 17 because replica has last-logged OpId of term: 15 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:23:19.243733  5541 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.095s
I20260709 12:23:19.353377 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:19.375820 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:19.339919 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:19.407573 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:19.381680 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2088) took 1024 ms. Trace:
I20260709 12:23:19.420434 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:19.421716 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 14 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:19.419267 31674 rpcz_store.cc:276] 0709 12:23:18.357723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.357924 (+   201us) service_pool.cc:224] Handling call
0709 12:23:18.386855 (+ 28931us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:18.387096 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:18.727668 (+340572us) log.cc:844] Serialized 14 byte log entry
0709 12:23:19.381334 (+653666us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:19.381388 (+    54us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:19.381435 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.381504 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:19.381617 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13701,"prepare.queue_time_us":503252,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":70736,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":108,"thread_start_us":924577,"threads_started":3,"wal-append.queue_time_us":512393}
I20260709 12:23:19.422195 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:19.340965  5452 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:19.354158  5502 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:19.423785 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 83979.2us
I20260709 12:23:19.438982 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:19.446889 31818 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:19.423171 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:19.422364  5637 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:19.450107 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:19.451675 32369 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: [13.10-14.11]
I20260709 12:23:19.374722  5151 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:19.462937  5151 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:19.463335  5151 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:19.469159  5611 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.130s	user 0.000s	sys 0.000s
I20260709 12:23:19.471839  5411 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:19.369336  5621 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:19.426940  5452 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.308329  5506 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:19.423552  5586 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:19.370937  5606 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.132s
W20260709 12:23:19.502805 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:19.495477 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.018s	sys 0.024s
W20260709 12:23:19.512041 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.020s	sys 0.026s
W20260709 12:23:19.514135 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 520837us
I20260709 12:23:19.462152  5545 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.479246  5360 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:19.490759  5507 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.584s	user 0.009s	sys 0.002s
I20260709 12:23:19.486879  5466 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:19.494665  5452 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:19.494505  5586 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.557665  5626 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:19.490006  5621 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:19.580674  5619 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.068s	user 0.001s	sys 0.000s
I20260709 12:23:19.575321  5541 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:19.578505  5624 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:19.586534  5586 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:19.613209  5517 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.591s	user 0.011s	sys 0.000s
W20260709 12:23:19.652419 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.565867 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:19.655014  5563 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.726s	user 0.013s	sys 0.007s
W20260709 12:23:19.655463  5563 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.730s	user 0.013s	sys 0.008s
I20260709 12:23:19.595584  5627 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:19.613724  5518 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.609843  5626 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:19.617201  5466 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:19.631106  5624 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.672385 31827 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:19.619459  5630 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:19.672559 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:19.683543 31819 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:19.678900 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 228848us
W20260709 12:23:19.692711 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.580317  5502 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.699000 31744 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:19.699602 31744 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:19.655984  5544 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.024s
I20260709 12:23:19.700636 31744 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.702170 31744 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:19.712630 31744 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
W20260709 12:23:19.708513 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 298940us
I20260709 12:23:19.649981  5541 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.673718  5626 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:19.740233  5530 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:19.740818  5451 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:19.741719  5451 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.680227  5624 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:19.660696  5517 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:19.716512  5505 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.672s	user 0.009s	sys 0.005s
W20260709 12:23:19.716732  5465 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.640s	user 0.011s	sys 0.000s
I20260709 12:23:19.780846 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:19.673406  5518 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:19.784427 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:19.786656 31826 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 15. Ops: []
I20260709 12:23:19.691146  5630 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.755304  5451 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:19.716210  5187 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:19.796468  5633 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:23:19.772389  5465 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:19.772446  5505 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:19.809402  5451 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:23:19.819765  5634 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 12:23:19.831176 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:19.811563  5650 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.815421  5360 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:19.819931  5545 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:19.835196 31821 raft_consensus.cc:2368] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19.
I20260709 12:23:19.789835  5466 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:19.837958 31824 raft_consensus.cc:2368] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21.
I20260709 12:23:19.822994  5651 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:19.833029  5451 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:19.839712  5174 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.534s	user 0.011s	sys 0.004s
I20260709 12:23:19.846768  5621 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:19.838739  5650 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.863023  5567 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:19.863585  5567 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:19.864845  5465 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:19.864552  5567 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:19.869809  5649 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:19.870385  5649 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:19.870873  5649 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:19.869494  5151 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:19.854835  5451 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:19.881026 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:19.881733 31744 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
W20260709 12:23:19.872762  5643 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.089s	user 0.000s	sys 0.001s
I20260709 12:23:19.885684 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:19.886358 31744 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:19.888243  5636 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:23:19.854195  5651 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:19.896199 31744 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
W20260709 12:23:19.886327  5639 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.153s	user 0.001s	sys 0.001s
W20260709 12:23:19.906400 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 122089us
I20260709 12:23:19.907473 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:19.896461  5517 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:19.908711 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:19.909356 31746 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:19.911548 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:19.915235 31673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2079) took 2023 ms (client timeout 2054 ms). Trace:
W20260709 12:23:19.924377 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.919060 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:19.917814 31746 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:19.916230 31672 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:19.905406  5187 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:19.904024  5662 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:19.920605  5592 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:23:19.929224  5455 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:19.948230 31670 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:19.931389  5505 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:19.919793  5666 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
W20260709 12:23:19.922932 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 629810us
I20260709 12:23:19.955520 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:19.959458  5651 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:19.964690  5452 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.965207  5649 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.935928  5427 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.967410  5427 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.903678  5541 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:19.969192 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2240) took 1905 ms. Trace:
I20260709 12:23:19.969465  5541 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:19.970193  5427 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:19.922499 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2091) took 1324 ms. Trace:
I20260709 12:23:19.970042  5541 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.970846 31675 rpcz_store.cc:276] 0709 12:23:18.597942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.598103 (+   161us) service_pool.cc:224] Handling call
0709 12:23:18.598410 (+   307us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:18.598614 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:18.895959 (+297345us) log.cc:844] Serialized 14 byte log entry
0709 12:23:19.211223 (+315264us) log.cc:844] Serialized 14 byte log entry
0709 12:23:19.234227 (+ 23004us) log.cc:844] Serialized 14 byte log entry
0709 12:23:19.288928 (+ 54701us) log.cc:844] Serialized 14 byte log entry
0709 12:23:19.310720 (+ 21792us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b180010aec0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:19.921644 (+610924us) spinlock_profiling.cc:243] Waited 30.2 ms on lock 0x7b180010aec0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:23:19.921900 (+   256us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:19.921996 (+    96us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:19.922086 (+    90us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.922213 (+   127us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:19.922406 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":80572,"prepare.queue_time_us":357657,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":4067,"raft.queue_time_us":722536,"raft.run_cpu_time_us":649,"raft.run_wall_time_us":31434,"spinlock_wait_cycles":221078784,"thread_start_us":576892,"threads_started":2,"wal-append.queue_time_us":573798}
I20260709 12:23:19.951552  5455 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:19.972760  5361 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:19.973239  5361 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:19.973212  5455 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:19.972613  5541 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:19.928594 31673 rpcz_store.cc:269] 0709 12:23:17.891761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.891919 (+   158us) service_pool.cc:224] Handling call
0709 12:23:19.914939 (+2023020us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b5400046ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:19.915143 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":813409152}
I20260709 12:23:19.985344  5649 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:19.987221  5630 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:19.988040  5455 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:19.990442  5455 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:19.953923  5646 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.007s
I20260709 12:23:19.926158  5518 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:19.953646  5650 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:19.979192  5563 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:19.979487  5544 consensus_queue.cc:1243] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 16.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
W20260709 12:23:19.979279 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 413542us
I20260709 12:23:19.969653 31749 rpcz_store.cc:276] 0709 12:23:18.063936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.064072 (+   136us) service_pool.cc:224] Handling call
0709 12:23:18.064415 (+   343us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:18.197354 (+132939us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:18.197407 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:18.200076 (+  2669us) log.cc:844] Serialized 52 byte log entry
0709 12:23:18.338041 (+137965us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:18.365976 (+ 27935us) log.cc:844] Serialized 14 byte log entry
0709 12:23:18.366249 (+   273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:18.366351 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:18.397698 (+ 31347us) spinlock_profiling.cc:243] Waited 996 us on lock 0x7b18003f0520. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:19.794180 (+1396482us) spinlock_profiling.cc:243] Waited 8.09 ms on lock 0x7b18003f0520. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:19.968949 (+174769us) raft_consensus.cc:1685] finished
0709 12:23:19.968972 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:19.969123 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13373,"prepare.queue_time_us":27697,"prepare.run_cpu_time_us":77,"prepare.run_wall_time_us":3342,"raft.queue_time_us":15722,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":206,"spinlock_wait_cycles":20890240,"thread_start_us":258000,"threads_started":2,"wal-append.queue_time_us":223722}
I20260709 12:23:19.980602 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 970) took 1188 ms. Trace:
W20260709 12:23:19.981536 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:19.982070  5627 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:19.982860 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:20.019338 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2929) took 1265 ms. Trace:
I20260709 12:23:19.983879  5545 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:19.996934 31821 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:20.025717 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:19.935964 31718 leader_election.cc:400] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19.
W20260709 12:23:20.027283 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 93599.3us
I20260709 12:23:20.029901 31670 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:20.030385 31670 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:20.034224 31821 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:20.035305 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:20.036082 31821 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:20.038209 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:20.039063 32370 raft_consensus.cc:2368] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19.
I20260709 12:23:20.037110 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:20.001559 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 98059.8us
I20260709 12:23:20.040907 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:20.002349 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2769) took 1070 ms. Trace:
I20260709 12:23:20.043167 31826 rpcz_store.cc:276] 0709 12:23:18.932065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.933394 (+  1329us) service_pool.cc:224] Handling call
0709 12:23:18.933785 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.792656 (+858871us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.002021 (+209365us) spinlock_profiling.cc:243] Waited 853 ms on lock 0x7b5400166cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:20.002266 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1960878848}
I20260709 12:23:20.044523 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:20.045123 31826 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:20.003289 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:20.046069 31819 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:20.004297 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2770) took 1029 ms. Trace:
I20260709 12:23:20.005225 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:20.022490 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:20.047155 31827 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:20.047567 31675 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:20.011688 31678 rpcz_store.cc:276] 0709 12:23:18.791737 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.884041 (+ 92304us) service_pool.cc:224] Handling call
0709 12:23:18.884430 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:18.884701 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:19.204367 (+319666us) log.cc:844] Serialized 14 byte log entry
0709 12:23:19.935782 (+731415us) log.cc:844] Serialized 14 byte log entry
0709 12:23:19.980249 (+ 44467us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:19.980304 (+    55us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:19.980346 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.980413 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:19.980539 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":650129,"prepare.queue_time_us":203304,"prepare.run_cpu_time_us":122,"prepare.run_wall_time_us":13672,"raft.queue_time_us":658761,"spinlock_wait_cycles":76018432,"thread_start_us":426215,"threads_started":2,"wal-append.queue_time_us":145450}
I20260709 12:23:20.046955 31824 rpcz_store.cc:276] 0709 12:23:18.974602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.975024 (+   422us) service_pool.cc:224] Handling call
0709 12:23:20.004116 (+1029092us) spinlock_profiling.cc:243] Waited 801 ms on lock 0x7b54004461d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:20.004245 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1841998976}
I20260709 12:23:19.999604  5151 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:20.051689 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:20.052740 31749 raft_consensus.cc:2368] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15.
I20260709 12:23:20.047155 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:20.048939 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:20.054864 31824 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:20.055560 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:20.056094 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:20.056569 31821 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:20.059226 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:20.041556 31670 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.063835 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:20.064610 31827 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:20.066545 31675 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:20.069093 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:20.071244 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:20.072095 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:20.073225 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.073455 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:19.953085  5662 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.030350  4785 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 12:23:20.030643  5569 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.047s
W20260709 12:23:20.126693 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.077445 31718 leader_election.cc:400] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19.
I20260709 12:23:20.129117 31718 leader_election.cc:403] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
W20260709 12:23:20.130543 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.109295 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 887) took 1087 ms. Trace:
W20260709 12:23:20.125185 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 294064us
I20260709 12:23:20.120710 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1036) took 1708 ms. Trace:
I20260709 12:23:20.143641 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:20.138366 32371 rpcz_store.cc:276] 0709 12:23:18.411810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.415895 (+  4085us) service_pool.cc:224] Handling call
0709 12:23:18.416287 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:18.817685 (+401398us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.120290 (+1302605us) spinlock_profiling.cc:243] Waited 401 ms on lock 0x7b54004c5c54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:20.120614 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":921268992}
I20260709 12:23:19.974171  5361 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:20.146839 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 69588us
W20260709 12:23:20.150223 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 207816us
I20260709 12:23:20.153697 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:20.156301 32372 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:20.153033 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:20.161661 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:20.051920  5627 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:20.164369 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:20.118748  5663 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:23:20.170495 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:20.117091  5628 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:23:20.180542 32372 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:20.172322 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:20.182976 32375 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:20.149657  5646 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19.
I20260709 12:23:20.165684  5455 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:20.187399  5455 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.188438 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:20.094554  5641 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:20.189263 32374 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 19: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:20.189523  5641 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:20.189750  5455 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:20.189993  5641 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:20.190194 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:20.193356  5567 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:20.154922  5174 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:20.116768  4785 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:20.195542  4785 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:20.149328  5518 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.197917  5518 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.199810 32375 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:20.200732  5518 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:20.203049 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:20.177678 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:20.117749  5531 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:20.205827 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:20.247709 32370 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:20.102578  5662 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:20.101649  5632 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:20.172257 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:20.249009 31748 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:20.145371  5512 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:20.252924 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:20.230675 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:20.207320 31678 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:23:20.204129  5435 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:23:20.259379 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 98785.9us
I20260709 12:23:20.231943 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:20.229449  5430 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.237221  5518 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:20.133584 32378 rpcz_store.cc:276] 0709 12:23:19.021697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.022006 (+   309us) service_pool.cc:224] Handling call
0709 12:23:19.051717 (+ 29711us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:19.066898 (+ 15181us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.070872 (+  3974us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:19.094219 (+ 23347us) log.cc:844] Serialized 52 byte log entry
0709 12:23:19.656350 (+562131us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:19.730431 (+ 74081us) log.cc:844] Serialized 14 byte log entry
0709 12:23:20.108301 (+377870us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b18002598a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:23:20.108599 (+   298us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.108715 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.108795 (+    80us) raft_consensus.cc:1685] finished
0709 12:23:20.108938 (+   143us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.109194 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3582,"prepare.queue_time_us":173310,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":44,"raft.queue_time_us":387616,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":7052,"spinlock_wait_cycles":24453504,"thread_start_us":546573,"threads_started":2,"wal-append.queue_time_us":583059}
I20260709 12:23:20.243921 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:20.114833  5151 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:20.256501 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:20.083595 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:20.100026  5586 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.260151 31674 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:20.266571 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:20.271351 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:20.268389 31826 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 14: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:20.051177  5530 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:20.278318  5411 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:20.283808 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2892) took 1554 ms. Trace:
I20260709 12:23:20.289047 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2772) took 1303 ms. Trace:
I20260709 12:23:20.379240 31818 rpcz_store.cc:276] 0709 12:23:18.985843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.986225 (+   382us) service_pool.cc:224] Handling call
0709 12:23:18.986598 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.447567 (+460969us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.288637 (+841070us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b54001969d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:20.288935 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1057757568}
W20260709 12:23:20.290120 31676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2092) took 1682 ms (client timeout 1634 ms). Trace:
I20260709 12:23:20.295346 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 971) took 1503 ms. Trace:
I20260709 12:23:20.382228 31671 rpcz_store.cc:276] 0709 12:23:18.791994 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.127993 (+335999us) service_pool.cc:224] Handling call
0709 12:23:20.293085 (+1165092us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.293350 (+   265us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:20.293430 (+    80us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:20.293496 (+    66us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:20.293544 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.293627 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.295030 (+  1403us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54001b2bd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:20.295239 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2678510464}
I20260709 12:23:20.297083  5595 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:20.380813 31676 rpcz_store.cc:269] 0709 12:23:18.607893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.608290 (+   397us) service_pool.cc:224] Handling call
0709 12:23:20.289815 (+1681525us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54001b2bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:20.289979 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":244441472}
I20260709 12:23:20.303910  5576 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:20.313959 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:20.316000 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:20.389853 31827 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:20.317592 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:20.391099 31824 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:20.392015 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:20.392791 31824 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 20: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:20.331310  5361 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:20.264586  5631 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.409921 31744 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: [18.14-18.14]
I20260709 12:23:20.412423 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:20.413164 31824 raft_consensus.cc:2368] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19.
I20260709 12:23:20.352007  5571 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.353070 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1044) took 1712 ms. Trace:
I20260709 12:23:20.246583  5412 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.355441 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:20.377628 31822 rpcz_store.cc:276] 0709 12:23:18.728956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.729224 (+   268us) service_pool.cc:224] Handling call
0709 12:23:18.751599 (+ 22375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.178354 (+426755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.283369 (+1105015us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b54004470d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:20.283700 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":979338624}
I20260709 12:23:20.385710 31820 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: [17.10-19.11]
I20260709 12:23:20.260815  5546 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.258942  5011 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:20.205210  5627 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.387784 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:20.050192  5563 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.223s
I20260709 12:23:20.395138 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:20.397387 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:20.332628 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:20.400646 31821 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:20.399031 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:20.263363  5632 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.269977  5435 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:20.357570  5658 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:20.414518 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:20.424504 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:20.426791  5649 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:20.397374  5411 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.357993  5438 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:20.411134  5466 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.411372  5518 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:20.438880  5543 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.169s
I20260709 12:23:20.481494  5506 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.455397 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:20.506601 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51360.7us
I20260709 12:23:20.420442 31823 rpcz_store.cc:276] 0709 12:23:18.641166 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.672028 (+ 30862us) service_pool.cc:224] Handling call
0709 12:23:18.672400 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:18.818254 (+145854us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:18.818296 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:18.818818 (+   522us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.203172 (+384354us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:19.241008 (+ 37836us) log.cc:844] Serialized 14 byte log entry
0709 12:23:19.256364 (+ 15356us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.888900 (+632536us) spinlock_profiling.cc:243] Waited 5.67 ms on lock 0x7b4800130158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:23:20.183643 (+294743us) spinlock_profiling.cc:243] Waited 16.2 ms on lock 0x7b18004ab5a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:20.325981 (+142338us) spinlock_profiling.cc:243] Waited 104 ms on lock 0x7b5400447ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:20.326078 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.326195 (+   117us) raft_consensus.cc:1685] finished
0709 12:23:20.326395 (+   200us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.352958 (+ 26563us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48242,"prepare.queue_time_us":37473,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":53,"raft.queue_time_us":92803,"raft.run_cpu_time_us":8922,"raft.run_wall_time_us":1086020,"spinlock_wait_cycles":289540608,"thread_start_us":315937,"threads_started":1,"wal-append.queue_time_us":495382,"wal-append.run_cpu_time_us":2512,"wal-append.run_wall_time_us":1002517}
I20260709 12:23:20.456343 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:20.426954 31748 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:20.463128 31671 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:20.270138  5662 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.150s
I20260709 12:23:20.262795  5512 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.518151  5435 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.520414  5512 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:20.520987  5642 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.756s	user 0.000s	sys 0.002s
I20260709 12:23:20.501458  5338 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 11 FOLLOWER]: Advancing to term 12
I20260709 12:23:20.465705 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:20.522950  5665 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.261s	user 0.001s	sys 0.000s
I20260709 12:23:20.524658 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:20.525465 31823 raft_consensus.cc:2368] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21.
I20260709 12:23:20.525810 31670 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:20.465689 31819 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:23:20.410740  5549 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.526908  5549 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.397718  5576 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:20.529335  5549 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:20.528828  5576 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.270287  5430 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.532559  5430 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:20.416210  5530 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:20.535359  5530 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.536626  5583 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.537279  5583 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.473358 31750 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:23:20.357312  5151 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.426460  5545 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:20.553232  5545 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.554080 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2768) took 1629 ms. Trace:
I20260709 12:23:20.473634 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:20.482251 31827 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:20.554474 31825 rpcz_store.cc:276] 0709 12:23:18.924458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.924682 (+   224us) service_pool.cc:224] Handling call
0709 12:23:18.968711 (+ 44029us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:19.221820 (+253109us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:19.221870 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:19.222504 (+   634us) log.cc:844] Serialized 26 byte log entry
0709 12:23:19.654754 (+432250us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:19.660944 (+  6190us) log.cc:844] Serialized 14 byte log entry
0709 12:23:19.661165 (+   221us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:19.661243 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:20.520341 (+859098us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:20.545191 (+ 24850us) raft_consensus.cc:1685] finished
0709 12:23:20.553855 (+  8664us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.554015 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10583,"prepare.queue_time_us":8655,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":68,"raft.queue_time_us":267950,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":7821,"spinlock_wait_cycles":8522624,"thread_start_us":590923,"threads_started":2,"wal-append.queue_time_us":386351}
I20260709 12:23:20.427682  5646 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:20.556895 31825 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:20.557231  5646 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.558597  5630 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:20.559026  5630 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.559381  5630 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:20.559772  5646 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:20.483693 31818 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:20.447798  5541 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.556770  5512 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:20.562278  5512 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:20.487604 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:20.570209 31670 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:20.571719 31822 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:23:20.470821  5618 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:20.498546 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 101276us
I20260709 12:23:20.457962  5412 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.426142 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:20.448019  5571 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.517247 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:20.539480 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 125064us
I20260709 12:23:20.578264 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:20.526300 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:20.580662 31823 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:20.486676  5658 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:20.562022  5541 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.542083  5505 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:20.546324 31752 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:20.497730  5438 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.552299 31751 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 7. (term mismatch)
W20260709 12:23:20.555817 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 82255.9us
I20260709 12:23:20.584623 31751 pending_rounds.cc:85] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 6
I20260709 12:23:20.584895 31751 pending_rounds.cc:107] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 11.7
I20260709 12:23:20.585119 31751 raft_consensus.cc:2889] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:20.586711  5549 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:20.583492  5646 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:20.577348  5618 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.588631  5571 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:20.588757  5151 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:20.590109 31674 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:23:20.590780  5618 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:20.590194  5669 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260709 12:23:20.591142  5412 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:20.561412  5310 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:20.356441  5452 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:20.464850  5360 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:20.592496  5512 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.592932  5310 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.593333  5360 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.556671 31827 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:23:20.472958  5546 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.420015  5631 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.604030 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:20.605213 31671 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 15: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:20.505074  5466 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.481387  5427 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.033s
W20260709 12:23:20.574962 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 148908us
I20260709 12:23:20.479740  5011 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.472568  5532 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.031s
I20260709 12:23:20.542675  5455 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:20.581576 31792 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:20.540167  5576 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:20.582283 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 65110.8us
I20260709 12:23:20.592087  5635 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:20.595247 32378 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:20.496446  5649 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.588678 31750 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:20.577814  5566 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 16 current term: 16 is higher.
I20260709 12:23:20.600621 31824 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:20.609134 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:20.569648 31748 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:20.606276  5014 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.009s
I20260709 12:23:20.580353  5545 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:20.581516  5647 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:20.505316  5518 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.611120  5549 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:20.651532  5549 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:20.655372  5676 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.159s	user 0.000s	sys 0.000s
I20260709 12:23:20.657109 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:20.657867 31750 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
W20260709 12:23:20.621205 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.621430  5310 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:20.616515 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:20.661468 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:20.663038 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:20.663457 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:20.663683 31671 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:20.540589  5506 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:20.614909  5624 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.837s	user 0.010s	sys 0.008s
I20260709 12:23:20.632640 31826 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:20.634117 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:20.634599 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 53121us
W20260709 12:23:20.680016 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 46046.2us
I20260709 12:23:20.680995 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:20.681572 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.613626  5618 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:20.641920 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:20.600556  5641 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.646646 31744 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:20.646490  5625 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:20.642870  5672 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
W20260709 12:23:20.657903  5673 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:23:20.666986 31642 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:20.647780  5647 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.744470 31753 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:23:20.623600 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:20.668556 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:20.675277 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58896.4us
I20260709 12:23:20.748307 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:20.678053 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:20.749848 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 71921.3us
I20260709 12:23:20.750890 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:20.688150 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:20.696861  5677 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
W20260709 12:23:20.699499  5451 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.813s	user 0.011s	sys 0.000s
I20260709 12:23:20.701447  5466 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:20.703521  5438 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:20.703912  5626 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.928s	user 0.009s	sys 0.004s
I20260709 12:23:20.706314  5011 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:20.709524  5632 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:20.788179 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:20.789005 31750 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:20.711205  5546 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:20.711714  5646 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:20.712036  5411 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:20.714782 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:20.715608  5452 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:20.716307  5338 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:20.716919  5505 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:20.641520  5658 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.722417  5649 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:20.718160  5455 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.005s
I20260709 12:23:20.698364  5518 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:20.724561  5631 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:20.727818  5541 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:20.640899  5662 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:20.732383 31823 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
W20260709 12:23:20.733630 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 91789.2us
I20260709 12:23:20.641391  5586 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:20.742655 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 75782.5us
I20260709 12:23:20.667081 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:20.956804 31748 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 12.
I20260709 12:23:20.639029  5151 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.099s
I20260709 12:23:20.760455 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:20.975814 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 215449us
I20260709 12:23:20.762506 31675 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:20.740561  5567 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.547s	user 0.011s	sys 0.000s
I20260709 12:23:20.739240  5625 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:20.737937  5014 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:20.739540  5532 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:20.989178  5532 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.989832  5532 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:20.762549  5430 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:20.765483 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:20.739420  5672 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:20.992806  5672 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:20.756950  5447 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:20.839504 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:20.754707  5569 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:23:20.720489  5674 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:23:20.730392  5624 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.755333  5647 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:20.784340  5627 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:20.867499 31753 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: [14.11-14.11]
I20260709 12:23:20.800707  5646 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.874913 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2773) took 1888 ms. Trace:
I20260709 12:23:20.877135 31750 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
W20260709 12:23:20.884773 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:20.743654  5618 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:20.973971 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2219) took 3395 ms. Trace:
I20260709 12:23:20.976673 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:20.997444  5641 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:20.999615  5618 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:20.999598  5151 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:21.000717  5151 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:21.001605  5151 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.002786  5361 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:21.003651  5151 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:21.005096 31747 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:21.007179  5627 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:21.007937  5627 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:21.008913  5627 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.012060 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:20.863755  5505 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.013995  5466 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:21.014725  5518 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:21.015368  5518 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:21.015412  5338 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:21.003800 31820 rpcz_store.cc:276] 0709 12:23:18.986191 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.995846 (+  9655us) service_pool.cc:224] Handling call
0709 12:23:18.996237 (+   391us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:20.457690 (+1461453us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.874467 (+416777us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54001671d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:20.874815 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3194572672}
I20260709 12:23:20.977875 31642 leader_election.cc:400] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15.
I20260709 12:23:21.017583 31642 leader_election.cc:403] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:21.017366  5630 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.018558 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:21.020002 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:20.764004  5635 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.105s
I20260709 12:23:20.983103 31823 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:20.985067  5625 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:20.989075  5014 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:21.021641  5635 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:21.022265  5014 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:20.993515  5360 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:20.757159  5593 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:20.997622  5646 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:20.787461  5451 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.023427 31669 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
W20260709 12:23:20.869661  5677 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:23:20.996997 31751 rpcz_store.cc:276] 0709 12:23:17.482347 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.486940 (+  4593us) service_pool.cc:224] Handling call
0709 12:23:17.487241 (+   301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.585560 (+3098319us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.875771 (+290211us) spinlock_profiling.cc:243] Waited 3.05 s on lock 0x7b54000b81d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:20.876018 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7017771264}
I20260709 12:23:20.976558  5662 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:20.789186  5502 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.189s
I20260709 12:23:21.036406 32372 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 12:23:21.037909 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 323138us
I20260709 12:23:21.072628 31827 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: [18.9-18.9]
I20260709 12:23:20.870358  5452 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:21.053866 31675 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
W20260709 12:23:21.012096  5679 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 12:23:20.955341 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:21.003938  5361 consensus_queue.cc:1243] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 18.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:23:21.067050 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:21.067744 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 308478us
I20260709 12:23:21.069015 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:21.066447 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2097) took 1044 ms. Trace:
I20260709 12:23:21.076957 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:21.098089 31669 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:21.099409 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:20.797240  5626 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.020699  5630 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:21.116564 31670 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:21.094164 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 977) took 1940 ms. Trace:
I20260709 12:23:21.054781  5569 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.117404 31672 rpcz_store.cc:276] 0709 12:23:19.154525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:19.156682 (+  2157us) service_pool.cc:224] Handling call
0709 12:23:19.916008 (+759326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:19.934491 (+ 18483us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.093840 (+1159349us) spinlock_profiling.cc:243] Waited 759 ms on lock 0x7b5400046ad0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:21.094069 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1745287424}
I20260709 12:23:21.118834 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:21.101105 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 145845us
I20260709 12:23:21.097229 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:21.050701  5618 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 15
I20260709 12:23:21.052469  5563 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:21.040704  5518 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:21.054544  5675 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:21.106799 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.109220 31746 raft_consensus.cc:2368] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18.
I20260709 12:23:21.050056  5530 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:23:21.110440  5531 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.851s	user 0.012s	sys 0.000s
I20260709 12:23:21.097127 31673 rpcz_store.cc:276] 0709 12:23:20.021876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.022112 (+   236us) service_pool.cc:224] Handling call
0709 12:23:20.041970 (+ 19858us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:20.071973 (+ 30003us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:20.072003 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:20.072601 (+   598us) log.cc:844] Serialized 52 byte log entry
0709 12:23:20.265360 (+192759us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:20.265422 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.265523 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:21.066170 (+800647us) raft_consensus.cc:1685] finished
0709 12:23:21.066192 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.066376 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":44824,"raft.queue_time_us":30790,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":95541,"thread_start_us":166129,"threads_started":1,"wal-append.queue_time_us":366831}
I20260709 12:23:21.100126  5452 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:21.131448 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:21.132264 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:21.149238 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.149636 31748 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:23:21.078475  5662 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:20.019704 31677 rpcz_store.cc:276] 0709 12:23:18.753754 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:18.852417 (+ 98663us) service_pool.cc:224] Handling call
0709 12:23:18.853043 (+   626us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:18.853283 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:19.341465 (+488182us) log.cc:844] Serialized 14 byte log entry
0709 12:23:20.016607 (+675142us) log.cc:844] Serialized 14 byte log entry
0709 12:23:20.018948 (+  2341us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:20.019004 (+    56us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:20.019063 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:20.019136 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:20.019273 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":338732,"prepare.queue_time_us":156659,"prepare.run_cpu_time_us":127,"prepare.run_wall_time_us":4049,"raft.queue_time_us":327712,"spinlock_wait_cycles":47490432,"thread_start_us":800434,"threads_started":3,"wal-append.queue_time_us":478009}
I20260709 12:23:21.155818 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:21.156883 31677 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:21.079353  5502 consensus_queue.cc:1243] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:20.982182  5631 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:21.096210 31751 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:21.161932 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:21.147029 31819 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:21.116050  5546 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:21.154286 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:21.164341 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:21.165391  5683 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:23:21.169039 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:21.152729  5452 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:21.180070 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:21.168419 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 154971us
I20260709 12:23:21.108865  5586 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.052s
I20260709 12:23:21.107482  5455 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.047s
I20260709 12:23:21.118317  5541 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 12:23:21.181624 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:21.193216 31748 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:21.196496 31748 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260709 12:23:21.196982 31748 pending_rounds.cc:85] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 11
I20260709 12:23:21.197250 31748 pending_rounds.cc:107] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 12:23:21.197526 31748 raft_consensus.cc:2889] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:21.161793  5681 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260709 12:23:21.185176 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:21.152984  5534 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.206444 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:21.210291 31752 raft_consensus.cc:606] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:21.210873 31752 raft_consensus.cc:614] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:21.139057  5569 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.170322  5546 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.169775  5624 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:21.174166  5647 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:21.220507 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 123371us
I20260709 12:23:21.222088 31718 leader_election.cc:400] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19.
I20260709 12:23:21.224023 31718 leader_election.cc:400] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21.
I20260709 12:23:21.181720  5466 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:21.224411 31718 leader_election.cc:403] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:23:21.227916 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:21.146786  5675 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.200205  5506 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:21.217506  5690 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:21.169564  5631 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.242813 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:21.220947  5546 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:21.222028  5534 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.258764  5675 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:21.259061  5647 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:21.259568  5670 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.744s	user 0.008s	sys 0.000s
I20260709 12:23:21.229184  5455 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:23:21.281980 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 96860.7us
I20260709 12:23:21.267865 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:21.279671 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2782) took 1451 ms. Trace:
I20260709 12:23:21.284762 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:21.285847 31677 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:21.286876 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:21.268950 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.271755 31669 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:21.260082  5451 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:21.248790  5690 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.263232  5151 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:21.303887 31750 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:21.287755 31672 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:21.289680 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:21.267064 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.270440  5447 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:21.269497  5692 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:21.305704 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.306100 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:21.309156 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 154945us
I20260709 12:23:21.284962 31826 rpcz_store.cc:276] 0709 12:23:19.828810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.481907 (+653097us) service_pool.cc:224] Handling call
0709 12:23:20.495766 (+ 13859us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.679088 (+183322us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.279391 (+600303us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b5400379454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:21.279589 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":313862016}
W20260709 12:23:21.291215  5680 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.192s	user 0.001s	sys 0.001s
I20260709 12:23:21.326865 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.329136 31818 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
W20260709 12:23:21.332284 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 89534.4us
I20260709 12:23:21.282262  5455 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:21.310120 31744 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
W20260709 12:23:21.311194 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.311517 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.333987 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:21.351886 31820 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:23:21.352775 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:21.309516  5695 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:21.327875  5452 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.371508  5695 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:21.372110 31752 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:21.372768  5695 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.373270  5014 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.373665  5452 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:21.374260  5695 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:21.379647  5624 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:21.380208  5624 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.380577  5624 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:21.381575  5690 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:21.328157  5692 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.385161  5647 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:21.385470  5692 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:21.308929  5687 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21.
W20260709 12:23:21.357985  5686 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 12:23:21.385828  5647 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.388262  5647 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:21.389084  5662 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.390619 31821 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:21.391772  5626 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:21.357609  5697 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:21.330283  5466 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 13.7, Last appended by leader: 7, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.396308  5662 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:21.367384  5151 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:21.397974  5151 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:21.379617  5014 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:21.399005  5151 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.399710 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2103) took 1324 ms. Trace:
I20260709 12:23:21.400264 31675 rpcz_store.cc:276] 0709 12:23:20.075462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.083113 (+  7651us) service_pool.cc:224] Handling call
0709 12:23:21.399462 (+1316349us) spinlock_profiling.cc:243] Waited 493 ms on lock 0x7b54001ff4d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:21.399627 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1134475520}
I20260709 12:23:21.398388  5618 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 14. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15.
I20260709 12:23:21.403115 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1054) took 1564 ms. Trace:
I20260709 12:23:21.403680 31824 rpcz_store.cc:276] 0709 12:23:19.838369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.533280 (+694911us) service_pool.cc:224] Handling call
0709 12:23:20.533617 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:20.639425 (+105808us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.402743 (+763318us) spinlock_profiling.cc:243] Waited 66.8 ms on lock 0x7b5400197dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:21.403027 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":153640192}
W20260709 12:23:21.410410 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55862.1us
I20260709 12:23:21.410981  5534 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:21.412055 31673 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:21.412856  5647 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:21.422812 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2109) took 1189 ms. Trace:
I20260709 12:23:21.423501 31674 rpcz_store.cc:276] 0709 12:23:20.233391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.414946 (+181555us) service_pool.cc:224] Handling call
0709 12:23:21.414416 (+999470us) spinlock_profiling.cc:243] Waited 66.9 ms on lock 0x7b540036f1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:21.422690 (+  8274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":153827712}
W20260709 12:23:21.424371 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.362337  5569 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:21.311543  5338 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader election won for term 13
I20260709 12:23:21.367954  5583 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:21.355199  5455 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:21.434988 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.436435 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.437740 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.437851 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 872) took 1049 ms. Trace:
I20260709 12:23:21.392572  4785 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.439175 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.439541  4785 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:21.439951 31823 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:21.440838 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:21.441339 31823 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:21.367552  5631 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:21.427136 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.444319  5505 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:21.444949 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:21.445535 31822 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:21.425288  5586 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:21.448987 31745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b416bf5304744b14b72501e697b8c951"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:21.449674 31745 raft_consensus.cc:606] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:21.450043 31745 raft_consensus.cc:614] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:21.455428 31672 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:21.455888 31672 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:21.438244 32378 rpcz_store.cc:276] 0709 12:23:20.388607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.389026 (+   419us) service_pool.cc:224] Handling call
0709 12:23:21.437653 (+1048627us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b54000127d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:21.437757 (+   104us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":248169856}
W20260709 12:23:21.465539  5689 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.186s	user 0.004s	sys 0.000s
W20260709 12:23:21.432185 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:21.469174 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.485225 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.488802 31818 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 15 index: 11. (term mismatch)
I20260709 12:23:21.489470 31818 pending_rounds.cc:85] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 10
I20260709 12:23:21.482393 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.490161 31824 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:21.438342 31826 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:21.496735 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:21.497505 31747 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
W20260709 12:23:21.498111 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 473323us
I20260709 12:23:21.499346 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:21.499606 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:21.500864 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:21.501610 31745 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:21.496804 31818 pending_rounds.cc:107] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 14.11
I20260709 12:23:21.503208 31818 raft_consensus.cc:2889] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:21.505316 31822 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:21.510141 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:21.511319 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:21.519465 31744 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:23:21.497917  5451 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.520396  5451 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.513300 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:21.521596 31752 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:21.513517 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:21.510926  5361 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.522804 31745 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 19: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:21.523159  5361 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.524623  5518 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.525697  5361 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:21.449061  5455 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.527529 31823 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:21.542579 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.546816 31745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:21.499651  5586 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:21.555368  5576 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.918s	user 0.012s	sys 0.003s
I20260709 12:23:21.556479 31677 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:21.557220 31745 raft_consensus.cc:606] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:21.557374 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.557688 31745 raft_consensus.cc:614] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:21.481467  5338 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:21.507800  5583 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:21.500828  5625 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.546952  5451 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:21.508121  5624 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:21.514320 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 188181us
I20260709 12:23:21.512681  5506 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 12:23:21.566059  5447 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:21.569284  5518 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:21.583112  5597 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.571296  5567 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:21.609513  5597 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.612135  5597 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:21.609975  5506 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:21.612931  5506 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:21.613958  5506 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.615098 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:21.615510  5506 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:21.616423  5698 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:21.619724  5586 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 13.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.627965 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.599498  5568 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:21.633836 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:21.634714 31744 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:21.640254 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 130204us
I20260709 12:23:21.572052  5583 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.599777  5451 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.646203 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:21.647569 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:21.650516 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:21.651433  5568 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:21.652911  5506 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 12:23:21.653337  5532 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.663s	user 0.011s	sys 0.003s
I20260709 12:23:21.654376  5624 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:21.654817  5338 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 13, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.655305  5583 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:21.656282  5455 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:21.635557  5567 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:21.657395  5625 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:21.658725 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:21.658676  5545 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:23:21.657150 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 525120us
I20260709 12:23:21.721627 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:21.660223  5466 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:21.662998 31674 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:23:21.664532 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:21.734143 31678 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:21.665244 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:21.667378  5546 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:21.667939 31746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2845) took 1626 ms (client timeout 1713 ms). Trace:
I20260709 12:23:21.668721 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:21.748195 31746 rpcz_store.cc:269] 0709 12:23:20.041671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.058850 (+ 17179us) service_pool.cc:224] Handling call
0709 12:23:21.667736 (+1608886us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400294fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:21.667881 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2411954560}
I20260709 12:23:21.671862  5451 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.675706 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1055) took 1837 ms. Trace:
I20260709 12:23:21.750041 31819 rpcz_store.cc:276] 0709 12:23:19.838782 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.536535 (+697753us) service_pool.cc:224] Handling call
0709 12:23:20.539228 (+  2693us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.163279 (+624051us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.675381 (+512102us) spinlock_profiling.cc:243] Waited 607 ms on lock 0x7b540030edd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:21.675622 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1397208320}
I20260709 12:23:21.647792 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:21.683065 31749 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:21.751554 31749 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:21.683229 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:21.683948 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.685878 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2113) took 1106 ms. Trace:
I20260709 12:23:21.753373 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:21.662285 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:21.756505 31745 raft_consensus.cc:2368] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
W20260709 12:23:21.696486 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 197229us
I20260709 12:23:21.696890 31744 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
W20260709 12:23:21.699023 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48592.4us
I20260709 12:23:21.702101 31826 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:23:21.686672  5698 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.027s
I20260709 12:23:21.735469 31675 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:21.701059  5360 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:21.705519  5627 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:23:21.671140  5646 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.601s	user 0.012s	sys 0.000s
I20260709 12:23:21.758549 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:21.753624 31671 rpcz_store.cc:276] 0709 12:23:20.579589 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.664281 (+ 84692us) service_pool.cc:224] Handling call
0709 12:23:20.664632 (+   351us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:20.679229 (+ 14597us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:20.744013 (+ 64784us) log.cc:844] Serialized 14 byte log entry
0709 12:23:21.081796 (+337783us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:21.082467 (+   671us) log.cc:844] Serialized 52 byte log entry
0709 12:23:21.082620 (+   153us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:21.082890 (+   270us) log.cc:844] Serialized 14 byte log entry
0709 12:23:21.098744 (+ 15854us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.098825 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:21.407820 (+308995us) spinlock_profiling.cc:243] Waited 9.65 ms on lock 0x7b1800406900. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:21.464774 (+ 56954us) spinlock_profiling.cc:243] Waited 8.95 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:21.655926 (+191152us) spinlock_profiling.cc:243] Waited 7.36 ms on lock 0x7b1800406900. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe900ca958f 00007fe8f9ce8571 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd
0709 12:23:21.685663 (+ 29737us) raft_consensus.cc:1685] finished
0709 12:23:21.685687 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.685819 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":330622,"prepare.queue_time_us":65830,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":5474,"raft.queue_time_us":40789,"raft.run_cpu_time_us":546,"raft.run_wall_time_us":72647,"spinlock_wait_cycles":59735424,"thread_start_us":261025,"threads_started":1,"wal-append.queue_time_us":439489}
W20260709 12:23:21.771659 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 123960us
I20260709 12:23:21.778134 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:21.778255 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:21.773432 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:21.780658 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:21.752431 31749 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:21.782186 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:21.782569 31749 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:21.782572 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:21.747700  4785 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:21.704627  5532 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.783994 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:21.784677 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:21.787933 31671 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:21.789458 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:21.789477 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:21.790351 31671 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:21.790755 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:21.791468 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:21.792572 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:21.792663 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:21.793438 32373 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:21.794188 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:21.794557 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:21.795305 32370 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:21.795674 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:21.799259 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:21.790302 32378 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:21.798573 31671 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:21.786675 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:21.802590 31675 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:21.803608 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:21.800225 31674 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:23:21.772506  5627 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.807564 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2787) took 1767 ms. Trace:
I20260709 12:23:21.807777 31671 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:23:21.809211 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:21.777829  5566 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:21.777137  5631 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.783700  5618 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20260709 12:23:21.772218  5576 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.045s
I20260709 12:23:21.799398  5675 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:21.776592  5693 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.064s
I20260709 12:23:21.812671 32369 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 15 index: 11. (index mismatch)
I20260709 12:23:21.813634 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:21.819742 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:21.820436 31749 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 12. (term mismatch)
I20260709 12:23:21.816551 31825 rpcz_store.cc:276] 0709 12:23:20.040230 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.557703 (+517473us) service_pool.cc:224] Handling call
0709 12:23:20.558044 (+   341us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:20.567777 (+  9733us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:20.567836 (+    59us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:20.607633 (+ 39797us) log.cc:844] Serialized 52 byte log entry
0709 12:23:20.981898 (+374265us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:21.218836 (+236938us) log.cc:844] Serialized 14 byte log entry
0709 12:23:21.806527 (+587691us) log.cc:844] Serialized 14 byte log entry
0709 12:23:21.807249 (+   722us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.807327 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:21.807369 (+    42us) raft_consensus.cc:1685] finished
0709 12:23:21.807380 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:21.807507 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":115243,"prepare.queue_time_us":477109,"prepare.run_cpu_time_us":118,"prepare.run_wall_time_us":16989,"raft.queue_time_us":71050,"raft.run_cpu_time_us":617,"raft.run_wall_time_us":27294,"thread_start_us":949889,"threads_started":4,"wal-append.queue_time_us":562760}
I20260709 12:23:21.845546 31749 pending_rounds.cc:85] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 11
I20260709 12:23:21.846148 31749 pending_rounds.cc:107] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 17.12
I20260709 12:23:21.846596 31749 raft_consensus.cc:2889] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:21.847435 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2261) took 1106 ms. Trace:
I20260709 12:23:21.847719 31749 rpcz_store.cc:276] 0709 12:23:20.740837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.740998 (+   161us) service_pool.cc:224] Handling call
0709 12:23:20.741361 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.682958 (+941597us) spinlock_profiling.cc:243] Waited 240 ms on lock 0x7b54003affa0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:23:21.847136 (+164178us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:21.847387 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":552669440}
I20260709 12:23:21.826475 31671 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:21.779500  5646 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.799847  5709 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:21.837311 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.820575  5566 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.815717 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:21.830860 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:21.857183 31674 raft_consensus.cc:2410] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 13 because replica has last-logged OpId of term: 12 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 11 index: 7.
I20260709 12:23:21.840039 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.841059 31675 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:21.858209 31819 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:21.853890 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:21.855994 31823 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:21.853890 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.859288 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:21.859771 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.860420 31823 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:21.860523 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.861270 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:21.864413 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:21.865368 32370 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 19 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:23:21.820466  4785 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:21.868491  5711 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:23:21.868058  4785 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:21.870100  5706 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:23:21.875049 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:21.852269  5546 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:21.857249  5709 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.861718  5566 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:21.880908 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.818920  5627 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:21.824828  5631 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:21.835611  5713 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:21.822183  5530 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:23:21.876004 31824 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:21.894935 31671 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:21.894686 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:21.894981 31750 pending_rounds.cc:85] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 9
I20260709 12:23:21.819322  5568 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:21.829979  5567 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:21.901054 32378 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:21.919844 31750 pending_rounds.cc:107] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 12:23:21.920127 31825 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:21.921589 31750 raft_consensus.cc:2889] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:21.911988  5435 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.390s	user 0.013s	sys 0.003s
I20260709 12:23:21.908051  5709 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:21.858178  5506 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:21.900009 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:21.855463  5649 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 18 current term: 18 is higher.
I20260709 12:23:21.907161 32370 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:23:21.914497  5713 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:21.926523  5435 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.405s	user 0.013s	sys 0.003s
W20260709 12:23:21.948086 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48223.8us
I20260709 12:23:21.948580 31670 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:23:21.948633 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:21.909940  5563 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:21.935894  5618 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:21.922380  5568 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:21.976653 31670 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:23:21.952931  5563 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.936336  5691 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.913939  5360 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.033s
I20260709 12:23:21.983608 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:21.939406  5506 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:21.940397  5713 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:21.989892  5451 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:21.990269  5714 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
I20260709 12:23:21.990739  5532 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:21.993301  5360 consensus_queue.cc:1243] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 20.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:23:21.987699  5506 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:22.002337  5713 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.006052 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:22.001904 31672 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.007467  5713 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:21.994151  5629 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:21.998582  5646 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:22.007580 31672 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:22.008324  5629 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.010664  5629 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:21.976298  5530 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Leader pre-election won for term 13
W20260709 12:23:21.995322  5717 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:23:21.977773  5618 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:21.979755  5568 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:21.981097  5563 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:22.024874  5506 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.027639 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:21.984447  5546 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:22.040508 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:22.018491  5530 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:22.031255  5722 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
W20260709 12:23:22.030658  5544 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 18 current term: 18 is higher.
I20260709 12:23:21.984241  5691 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.056106  5530 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:22.076874 32375 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:23:22.090435 32370 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:22.107709 31744 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:22.094383  5715 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.096503  5567 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.073208  5546 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.064116  5618 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.124282  5691 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:22.128527 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:22.130440  5473 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:22.133886  4785 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:22.109786  5631 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:22.135746 31744 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 11. (term mismatch)
I20260709 12:23:22.136135 31744 pending_rounds.cc:85] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 10
I20260709 12:23:22.136399 31744 pending_rounds.cc:107] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:23:22.136659 31744 raft_consensus.cc:2889] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:22.134544  4785 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.143142 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2911) took 1458 ms. Trace:
I20260709 12:23:22.143620 31827 rpcz_store.cc:276] 0709 12:23:20.684883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.685027 (+   144us) service_pool.cc:224] Handling call
0709 12:23:20.685432 (+   405us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:21.073207 (+387775us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.142858 (+1069651us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b540030ded4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:22.143083 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":889856256}
I20260709 12:23:22.147286  4785 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:22.133476  5606 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.133708  5569 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.149637  5618 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:22.117725  5567 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.162057  5606 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.115612  5715 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.137864  5593 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.025s
I20260709 12:23:22.218221  5567 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:22.224743 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:22.221047  5593 consensus_queue.cc:1243] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 17.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:23:22.164553  5569 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.215451  5506 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:22.230568  5724 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
W20260709 12:23:22.254078  5728 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:23:22.257694 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:22.259557  5546 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:22.261698  5569 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:22.263087 31671 raft_consensus.cc:2368] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:22.263566  5709 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.264155 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:22.264858 32377 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:22.265458 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:22.265877 31791 leader_election.cc:400] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:22.266242 31791 leader_election.cc:403] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 12:23:22.266621 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:22.270100 31751 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:22.271209 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 870) took 2111 ms. Trace:
I20260709 12:23:22.271624 32372 rpcz_store.cc:276] 0709 12:23:20.160025 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.182563 (+ 22538us) service_pool.cc:224] Handling call
0709 12:23:20.183633 (+  1070us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.070667 (+887034us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.270907 (+1200240us) spinlock_profiling.cc:243] Waited 852 ms on lock 0x7b54006473d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:22.271117 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1959316736}
I20260709 12:23:22.272114 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2272) took 1325 ms. Trace:
I20260709 12:23:22.272541 31751 rpcz_store.cc:276] 0709 12:23:20.946507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.161566 (+215059us) service_pool.cc:224] Handling call
0709 12:23:22.271945 (+1110379us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54000b8bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:22.272054 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2546370816}
I20260709 12:23:22.273053 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:22.266000  5566 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.277261 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:22.278324 31670 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:22.269112  5593 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:22.282918  5715 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:22.286556 31670 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:23:22.287976 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:22.289062 31670 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 15: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:22.290032 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:22.291255 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:22.291546 31669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2940) took 1215 ms (client timeout 1573 ms). Trace:
I20260709 12:23:22.292050 31670 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
W20260709 12:23:22.292006 31669 rpcz_store.cc:269] 0709 12:23:21.075731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.099028 (+ 23297us) service_pool.cc:224] Handling call
0709 12:23:22.291331 (+1192303us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54001d6354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:22.291486 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":393682688}
I20260709 12:23:22.293283 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:22.293977 31669 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:22.297571 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:22.298712 31670 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:22.298790 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 2949) took 1168 ms. Trace:
I20260709 12:23:22.300217 31673 rpcz_store.cc:276] 0709 12:23:21.130466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.204602 (+ 74136us) service_pool.cc:224] Handling call
0709 12:23:22.298487 (+1093885us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b5400181fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:22.298709 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":397492992}
I20260709 12:23:22.293339 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:22.302263 31675 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 19: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:22.329444 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:22.330523 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:22.331435 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:22.331017  5627 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.311314  5606 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:22.297555  5624 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.332012  5693 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:22.342417 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:22.367775 31827 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 21: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:22.367698 31669 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:22.343282  5432 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:22.369941  5432 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:22.370342  5432 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:22.363373  5373 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:22.359670  5624 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.327080  5607 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:22.370541  5658 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:22.341117  5709 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:22.371325  5373 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.341627  5627 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:22.334306  5566 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:22.338466  5530 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.339895  5568 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.384598  5607 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.398065  5568 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:22.388481  5373 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:22.396015  5624 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:22.401181 31677 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:22.398334  5607 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:22.416651 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:22.413565 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:22.420825  5738 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:22.421420  5738 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:22.422072  5632 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:22.422516  5632 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:22.423363  5632 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.424330  5632 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:22.425212  5715 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 12.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:22.422529  5738 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.426559  5567 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:22.427004  5567 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:22.428253 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:22.428902 31674 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:22.432206 31674 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:23:22.435089 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:22.427989  5567 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.437006  5624 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:22.437443  5624 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.437472  5567 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:22.437919  5624 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:22.442808  5506 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:22.431213  5712 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:22.444888  5712 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.445282  5712 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:22.447907  5672 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:22.451346 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:22.452131 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.453187 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:22.454697 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2953) took 1314 ms. Trace:
I20260709 12:23:22.455106 31672 rpcz_store.cc:276] 0709 12:23:21.140727 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.305238 (+164511us) service_pool.cc:224] Handling call
0709 12:23:21.329227 (+ 23989us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:22.001463 (+672236us) spinlock_profiling.cc:243] Waited 28.5 ms on lock 0x7b4c00337600. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb0a1a20 00007fe8fb0b2a29 00007fe8fb078644 00007fe8fb09ff2d 00007fe8fb09fb93 00007fe8fcdd018b 00007fe8fcdd0cb5 00007fe8fcdcbf30 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416
0709 12:23:22.020630 (+ 19167us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:22.020681 (+    51us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:22.020747 (+    66us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:22.020805 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.454391 (+433586us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54001ff9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:22.454452 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:22.454609 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3479,"raft.queue_time_us":3599,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":115,"spinlock_wait_cycles":355452160}
I20260709 12:23:22.457748  5473 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:22.458356  5473 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:22.459455  5473 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.460964  5473 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:22.462613  5566 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:22.464288 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 892) took 1323 ms. Trace:
W20260709 12:23:22.467900 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 797899us
I20260709 12:23:22.399988  5563 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:22.470818 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.471475  5690 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.472083  5455 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:22.472617  5455 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.431414  5568 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:23:22.470715  5586 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.479856 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:22.464727 31748 rpcz_store.cc:276] 0709 12:23:21.141118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.198549 (+ 57431us) service_pool.cc:224] Handling call
0709 12:23:21.221462 (+ 22913us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.221694 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:23:21.500628 (+278934us) log.cc:844] Serialized 14 byte log entry
0709 12:23:22.463683 (+963055us) spinlock_profiling.cc:243] Waited 23.4 ms on lock 0x7b18002562a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:23:22.463888 (+   205us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:22.463948 (+    60us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:22.464008 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.464088 (+    80us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:22.464227 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6766,"prepare.queue_time_us":363491,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":2256,"raft.queue_time_us":865025,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":122,"spinlock_wait_cycles":53741824,"thread_start_us":350115,"threads_started":2,"wal-append.queue_time_us":146711}
W20260709 12:23:22.487498 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51546.9us
I20260709 12:23:22.492256 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:22.492399 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:22.496140 31824 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:22.501077 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:22.505477 31819 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:22.512154 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:22.482602  5502 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.515494 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:22.518143 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.518695 31823 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:23:22.519445 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:22.519776 32372 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:23:22.521173 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:22.521442 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:22.517268 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:22.523950 31823 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:22.524940 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:22.526983 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:22.481753  5690 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.530772 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:22.531421 31820 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:22.532284 31822 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:22.532428 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:22.533226 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:22.535184 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:22.536213 32377 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
W20260709 12:23:22.538425 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 64922.8us
I20260709 12:23:22.539582 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:22.551142 31824 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:22.556727 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:22.557057 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.555332 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:22.560902 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.561864 31753 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:22.563426 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:22.487737  5586 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.566115 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:22.567829 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 88045.5us
I20260709 12:23:22.596143 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:22.596833 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:22.569904 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.598420 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:22.598690 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:22.599174 31744 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:22.570981 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.599884 31749 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:22.572999 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.600700 31746 raft_consensus.cc:2410] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 20 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:23:22.506749  5544 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.046s
I20260709 12:23:22.575683  5373 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.626349 32373 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 17: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:22.628475 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:22.577559 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.626752 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.627540 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:22.626120 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:22.577803  5690 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:22.556363  5641 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:22.583284 32369 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
W20260709 12:23:22.588140 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 466788us
I20260709 12:23:22.635612 32370 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 13.
I20260709 12:23:22.524693  5518 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
I20260709 12:23:22.530395  5563 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:22.555934  5530 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:22.591470  5455 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:22.523974  5502 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.592679  5541 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:22.592592 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:22.644312 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.644583 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:22.644404 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:22.645280 31746 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 21 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:23:22.645351 32374 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:22.593441 31751 raft_consensus.cc:2368] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15.
I20260709 12:23:22.593330  5435 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:22.594394 31753 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:22.652266  5502 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:22.652837 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.653532 31753 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:22.594964 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2909) took 1912 ms. Trace:
I20260709 12:23:22.654749 31821 rpcz_store.cc:276] 0709 12:23:20.682459 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:20.682661 (+   202us) service_pool.cc:224] Handling call
0709 12:23:20.745388 (+ 62727us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:21.391095 (+645707us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.594680 (+1203585us) spinlock_profiling.cc:243] Waited 645 ms on lock 0x7b54001987d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:22.594896 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1483350400}
I20260709 12:23:22.655377 31826 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:22.577215  5545 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:22.656152  5545 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.656595  5545 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:22.630686 31752 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:22.657196 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.658218 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2276) took 1026 ms. Trace:
I20260709 12:23:22.658757 32377 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:22.567236  5649 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:23:22.578496  5736 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
I20260709 12:23:22.665118 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:22.665084 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 72579.6us
I20260709 12:23:22.629899 32373 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:22.627627 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.640913 31823 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:22.642321 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:22.666425 31752 raft_consensus.cc:2368] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18.
I20260709 12:23:22.647423 31746 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:23:22.658728 31747 rpcz_store.cc:276] 0709 12:23:21.631839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.631971 (+   132us) service_pool.cc:224] Handling call
0709 12:23:21.632384 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:22.467800 (+835416us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:22.523784 (+ 55984us) log.cc:844] Serialized 14 byte log entry
0709 12:23:22.609362 (+ 85578us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:22.610131 (+   769us) log.cc:844] Serialized 26 byte log entry
0709 12:23:22.610292 (+   161us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:22.615496 (+  5204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.615609 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:22.657929 (+ 42320us) raft_consensus.cc:1685] finished
0709 12:23:22.657956 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:22.658143 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":834913,"prepare.queue_time_us":118911,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":50,"raft.queue_time_us":835398,"raft.run_cpu_time_us":256,"raft.run_wall_time_us":15271,"thread_start_us":140378,"threads_started":2,"wal-append.queue_time_us":125031}
I20260709 12:23:22.641357  5430 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:22.648317 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:22.649030 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:22.649518  5373 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:22.609961 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.654418 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:22.596179  5646 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:22.643787  5541 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:22.681668 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:22.682981 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.629488  5568 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:22.666998 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:22.657292  5690 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:22.684867 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.671726 32369 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:23:22.685781 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.675341 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.678007 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:22.680241 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:22.689920 31749 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:22.690898 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:22.692189 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47920.2us
I20260709 12:23:22.695559 31752 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:23:22.654474  5432 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.699038 31747 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 15 index: 8. (index mismatch)
I20260709 12:23:22.701087  5432 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:22.701789  5569 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:22.702318  5569 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:22.702771  5569 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:22.706013  5741 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:23:22.706875  5675 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 16.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:22.708098  5675 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:22.708479  5675 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:22.710716  5690 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:22.710911 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 112566us
I20260709 12:23:22.711288  5690 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:22.711998 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:22.712529  5618 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:23:22.713248  5618 consensus_queue.cc:1243] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 18.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
W20260709 12:23:22.716243 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.717075 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:22.718287  5532 raft_consensus.cc:2764] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:23:22.718609  5586 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:22.719832  5568 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.718456 31642 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:22.721989  5726 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.546s	user 0.003s	sys 0.000s
I20260709 12:23:22.722844  5563 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:22.725425 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:22.685179  5541 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:22.728060  5563 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:22.733045 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:22.665661  5649 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:22.778927 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 290308us
I20260709 12:23:22.779789 31675 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:22.740509 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:22.781500 31677 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 12. Current term is 13. Ops: []
I20260709 12:23:22.782915 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:22.694067 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:22.783749 31678 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:22.784204 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 90175.5us
I20260709 12:23:22.784956 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:22.785347 32373 pending_rounds.cc:85] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 13
I20260709 12:23:22.785673 32373 pending_rounds.cc:107] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 18.14
I20260709 12:23:22.785955 32373 raft_consensus.cc:2889] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:22.786422 32375 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:23:22.738898 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:22.747007 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:22.773136 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:22.796489 31822 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:22.673324  5361 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:22.785002 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:22.798211 31669 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:22.709379  5675 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:22.802802 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62376us
I20260709 12:23:22.811280 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:22.811990 31823 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
I20260709 12:23:22.804188 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:22.805285 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:22.816439 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:22.818948 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:22.820053  5712 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:22.820971  5646 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:22.821693  5532 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:22.823822 32372 raft_consensus.cc:2368] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16.
I20260709 12:23:22.824075 31820 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:23:22.835865 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:22.744614  5625 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
W20260709 12:23:22.865681 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50367.4us
I20260709 12:23:22.835255 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:22.722543  5568 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:22.878094 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:22.867364 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54820.2us
I20260709 12:23:22.738875  5597 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:22.739274  5151 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:22.822414  5532 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:22.803001  5649 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:22.868225 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 63025.3us
W20260709 12:23:22.911346 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:22.798924  5618 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:22.747766  5435 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:22.875471 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 95490.3us
W20260709 12:23:22.877410 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 76177.4us
I20260709 12:23:22.933518 31792 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:22.878779 31749 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:22.879472 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:22.895565 31642 leader_election.cc:400] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15.
I20260709 12:23:22.914084 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:22.934932 31642 leader_election.cc:403] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Cancelling election due to peer responding with higher term
I20260709 12:23:22.937114 31746 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:22.915627 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:22.813124  5695 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:22.828800  5361 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:22.829650  5583 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:22.930441 31822 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:22.877413  5625 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:22.919262 31678 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:22.945191 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:22.823094  5699 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.110s
I20260709 12:23:22.823239  5687 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.090s
I20260709 12:23:22.877664  5646 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:23:22.946736 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 51418us
I20260709 12:23:22.955711 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:22.955904 31749 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:22.948900 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:22.812958  5563 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:22.804267  5629 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.107s
I20260709 12:23:22.894620  5597 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:22.823479  5544 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:22.953233 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:22.872008  5678 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.075s
I20260709 12:23:22.931141  5618 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:22.945745  5725 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:23:22.953251  5532 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election lost for term 14. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15.
W20260709 12:23:22.967499  5672 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.519s	user 0.011s	sys 0.000s
I20260709 12:23:22.975919  5629 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:22.977015  5729 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.758s	user 0.000s	sys 0.003s
W20260709 12:23:22.977768  5742 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:23:23.003855 31675 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:22.969259  5747 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:23.007376 31792 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:22.972347  5563 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:22.984510 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:23.005700 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:23.008500 31749 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:22.975245  5373 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 12.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:23.015661  5627 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:23.016438  5373 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Stepping down as leader of term 12
I20260709 12:23:22.990396 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:23.016868  5373 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:22.908951  5593 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
I20260709 12:23:22.954108  5641 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:23.000734  5583 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:23.022938  5597 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.024190  5597 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:23.017831 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:23.030067  5649 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:23.030797  5624 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.592s	user 0.008s	sys 0.003s
I20260709 12:23:23.031612 31753 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:23:23.031960  5632 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.607s	user 0.011s	sys 0.002s
I20260709 12:23:22.955169  5646 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:23.006983 31671 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:23.033299  5646 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:23.033226  5361 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.034552 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:23.035378 31671 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 14: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:23.036263 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:23.005263  5678 consensus_queue.cc:1243] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 15.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:23.003923  5695 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:23.015390  5739 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:23.036803 31671 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:23.000216  5432 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:23.031248  5624 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.046440 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:23.044574 31669 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:23:23.071709 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 66109.2us
I20260709 12:23:22.991524  5618 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.017809  5373 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.073127 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:23.066390 31749 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
W20260709 12:23:23.038772  5727 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.819s	user 0.004s	sys 0.000s
I20260709 12:23:23.044678  5576 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.045995  5747 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:23.039263  5338 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:23.048280  5430 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:23.043396  5593 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:23.045467  5698 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:23.081207  5373 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:23.037844  5545 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.095448  5624 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:23.111899 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.066846  5432 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:23.071942  5502 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:23.068259  5151 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:23.084304  5699 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:23.102739  5473 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.641s	user 0.011s	sys 0.004s
I20260709 12:23:23.098097  5747 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:23.100595  5338 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:23.105269  5506 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:23.043601  5695 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.101563  5576 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.111137  5593 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.117007  5432 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.131645  5506 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:23.134279  5658 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.748s	user 0.009s	sys 0.008s
I20260709 12:23:23.135092  5563 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:23.136961  5545 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:23.127574  5699 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.131531  5690 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:23.198290  5749 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.280s	user 0.001s	sys 0.000s
I20260709 12:23:23.163605  5630 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.185868  5699 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:23.187530  5576 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:23.170258  5338 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:23.281754 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:23.284966 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 329172us
I20260709 12:23:23.287873 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 982) took 1624 ms. Trace:
I20260709 12:23:23.292174 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:23.295920 31747 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:23.297520 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:23.299220 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:23.304684 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.306406 31819 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:23:23.306661  5361 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:23.309850 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 236634us
I20260709 12:23:23.347308  5691 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:23.346966  5361 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.348069 31791 leader_election.cc:400] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:23.348518  5691 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.311041  5566 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.351253  5566 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:23.223382  5569 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.520s	user 0.010s	sys 0.004s
I20260709 12:23:23.355892  5569 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:23.312064 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.190937  5593 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:23.364579  5739 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:23.365325 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:23.313422 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:23.208022  5675 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:23.200225  5541 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.372953 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:23.170724  5151 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.313897 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:23.207371  5630 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.188683  5506 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.386279  5506 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:23.387213 31818 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:23.191753  5625 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:23.206393  5745 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.330s	user 0.002s	sys 0.000s
I20260709 12:23:23.198997  5646 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.393945 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2280) took 1756 ms. Trace:
I20260709 12:23:23.394747  5646 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:23.199832  5690 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:23.206800  5712 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:23.184777  5750 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:23:23.234076  5618 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.417742 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:23.419207 31747 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 22 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:23:23.419965  5618 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:23.420197 31747 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
W20260709 12:23:23.197129  5755 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.164s	user 0.000s	sys 0.002s
I20260709 12:23:23.135393  5747 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.421320 31747 raft_consensus.cc:606] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:23.297122 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 417549us
I20260709 12:23:23.318710  5658 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:23.422606  5658 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:23.423215 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:23.424937 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:23.425356 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:23.322594  5563 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:23.333125 31676 rpcz_store.cc:276] 0709 12:23:21.663631 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.683654 (+ 20023us) service_pool.cc:224] Handling call
0709 12:23:21.683995 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:21.684406 (+   411us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:21.922970 (+238564us) log.cc:844] Serialized 14 byte log entry
0709 12:23:22.111921 (+188951us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:22.114734 (+  2813us) log.cc:844] Serialized 26 byte log entry
0709 12:23:22.114859 (+   125us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:22.114881 (+    22us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.114930 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:23.038096 (+923166us) spinlock_profiling.cc:243] Waited 6.13 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:23.286897 (+248801us) raft_consensus.cc:1685] finished
0709 12:23:23.287143 (+   246us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.287568 (+   425us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":348235,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":1581,"raft.queue_time_us":5014,"raft.run_cpu_time_us":171,"raft.run_wall_time_us":174,"spinlock_wait_cycles":14097792,"thread_start_us":421676,"threads_started":2,"wal-append.queue_time_us":229039}
I20260709 12:23:23.339246 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:23.343861 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:23.345141 31678 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:23.333936 31819 pending_rounds.cc:85] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 11
I20260709 12:23:23.362569 31672 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:23.365126 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:23.207221  5432 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.435501 31746 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:23.437139 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 93685.9us
I20260709 12:23:23.365574  5739 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.382321 31753 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:23.383549 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:23.383353  5545 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:23.382979  5597 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:23.358855  5696 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:23.443398 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.444084  5696 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.445467 31752 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:23.447472  5696 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:23.235003  5649 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:23.394276 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:23.394459 31745 rpcz_store.cc:276] 0709 12:23:21.637489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.066712 (+429223us) service_pool.cc:224] Handling call
0709 12:23:22.069501 (+  2789us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:22.509438 (+439937us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:22.509469 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:22.515796 (+  6327us) log.cc:844] Serialized 26 byte log entry
0709 12:23:22.660737 (+144941us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:22.660779 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.660844 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:23.205802 (+544958us) spinlock_profiling.cc:243] Waited 4.63 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:23.393241 (+187439us) raft_consensus.cc:1685] finished
0709 12:23:23.393294 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.393734 (+   440us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":468385,"raft.queue_time_us":443025,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":7621,"spinlock_wait_cycles":11046016,"thread_start_us":144424,"threads_started":1,"wal-append.queue_time_us":359803}
I20260709 12:23:23.398768 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:23.451507 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53109.4us
I20260709 12:23:23.452731  5545 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:23.453503  5545 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:23.427824 31819 pending_rounds.cc:107] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 16.12
I20260709 12:23:23.458912 31819 raft_consensus.cc:2889] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:23.431823  5563 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:23.463204  5563 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.464510  5563 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:23.468259  5586 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:23.374145  5541 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:23.405928  5566 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:23.358400  5338 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:23.478063  5361 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:23.479084  5762 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:23.480118  5762 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:23.382969 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:23.365156  5569 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:23.438421 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:23.195583  5748 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.269s	user 0.000s	sys 0.002s
I20260709 12:23:23.483557 31749 raft_consensus.cc:2410] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 22 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:23:23.486358 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2279) took 1849 ms. Trace:
I20260709 12:23:23.487309 31750 rpcz_store.cc:276] 0709 12:23:21.637175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.876283 (+239108us) service_pool.cc:224] Handling call
0709 12:23:21.894713 (+ 18430us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:22.472610 (+577897us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:22.472645 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:22.473243 (+   598us) log.cc:844] Serialized 52 byte log entry
0709 12:23:22.638822 (+165579us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:22.638891 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:22.639000 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:23.194701 (+555701us) spinlock_profiling.cc:243] Waited 16.5 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:23.485526 (+290825us) raft_consensus.cc:1685] finished
0709 12:23:23.485575 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.486078 (+   503us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":549739,"raft.queue_time_us":570382,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":13630,"spinlock_wait_cycles":38003584,"thread_start_us":165151,"threads_started":1,"wal-append.queue_time_us":329083}
W20260709 12:23:23.490356 31818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2815) took 1807 ms (client timeout 1942 ms). Trace:
W20260709 12:23:23.490926 31818 rpcz_store.cc:269] 0709 12:23:21.682394 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:21.683716 (+  1322us) service_pool.cc:224] Handling call
0709 12:23:23.489987 (+1806271us) spinlock_profiling.cc:243] Waited 1.63 s on lock 0x7b540043d5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:23.490193 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3755666816}
W20260709 12:23:23.495666 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57470.7us
I20260709 12:23:23.497184 31642 leader_election.cc:400] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18.
I20260709 12:23:23.497581 31642 leader_election.cc:403] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:23.498689  5641 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18.
I20260709 12:23:23.499853  5641 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:23.424394  5658 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.320401 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:23.514576 32371 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:23.516251  5747 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.434365 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:23.506182  5625 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.374356 31826 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.452677 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:23.520252  5586 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:23.521062 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 68619.9us
I20260709 12:23:23.521646  5569 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:23.522524  5569 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:23.399889 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 350413us
I20260709 12:23:23.527586  5738 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.528682  5738 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.524112  5569 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.534312 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:23.534926  5593 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:23.535650 31818 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:23.535985  5593 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.539875  5576 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.541213 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:23.540927  5576 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.539963  5641 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:23.543540  5641 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.548465  5473 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.549546  5473 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.481844  5762 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.572580  5586 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.581210 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2929) took 1071 ms. Trace:
I20260709 12:23:23.580785  5586 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.582763  5502 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.583645  5502 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.581941 31819 rpcz_store.cc:276] 0709 12:23:22.510012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.510172 (+   160us) service_pool.cc:224] Handling call
0709 12:23:22.510668 (+   496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:23.459686 (+949018us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.571522 (+111836us) spinlock_profiling.cc:243] Waited 794 ms on lock 0x7b540037ad54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:23.581102 (+  9580us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1827298176}
W20260709 12:23:23.610746 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.630812 31676 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 13 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
W20260709 12:23:23.628306 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 58537.1us
I20260709 12:23:23.636097 31752 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:23.584805 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:23.649027 31671 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.602s	user 0.009s	sys 0.006s
I20260709 12:23:23.649641 31671 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:23.650725 31674 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
W20260709 12:23:23.653626 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 303313us
I20260709 12:23:23.654373  5630 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:23.655501  5641 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:23.656651  5739 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:23.655556  5432 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:23.640544 31744 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:23.575055  5618 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:23.655421  5738 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:23.674413 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:23.666517  5566 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:23:23.691294 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 257146us
I20260709 12:23:23.672955  5473 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:23.698036 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:23.602051  5545 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:23.697304 31820 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
W20260709 12:23:23.703053 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 69953.4us
I20260709 12:23:23.703205 31819 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:23:23.705770  5586 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:23.705953 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.706905  5641 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:23.707453  5641 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:23.707875  5641 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:23.709838  5618 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.710753  5618 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.711257 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:23.713218 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:23.713624  5618 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:23.715309  5593 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:23.715695  5586 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:23.716243  5696 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:23.720388  5696 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:23.720916  5373 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.606s	user 0.014s	sys 0.003s
I20260709 12:23:23.720970  5696 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:23.719281 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:23.727309  5455 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:23.727946 31818 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.728415  5624 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:23.727348 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2292) took 1078 ms. Trace:
I20260709 12:23:23.732004 31748 rpcz_store.cc:276] 0709 12:23:22.648587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:22.658197 (+  9610us) service_pool.cc:224] Handling call
0709 12:23:22.678058 (+ 19861us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:22.678513 (+   455us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:22.772755 (+ 94242us) log.cc:844] Serialized 14 byte log entry
0709 12:23:23.196532 (+423777us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:23.627483 (+430951us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:23.724091 (+ 96608us) log.cc:844] Serialized 26 byte log entry
0709 12:23:23.724320 (+   229us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:23.724360 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:23.724475 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:23.726484 (+  2009us) raft_consensus.cc:1685] finished
0709 12:23:23.726623 (+   139us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:23.727131 (+   508us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25506,"prepare.queue_time_us":107115,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":6778,"raft.queue_time_us":49893,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":58986,"spinlock_wait_cycles":33290880,"thread_start_us":599139,"threads_started":2,"wal-append.queue_time_us":230596}
I20260709 12:23:23.691378  5630 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.044s
W20260709 12:23:23.714313 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 198340us
I20260709 12:23:23.737358 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:23.738797 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:23.688616  5502 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:23.751559 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:23.752542 31752 raft_consensus.cc:606] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:23.753114 31752 raft_consensus.cc:614] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:23.752571 31825 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:23.763972 31826 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
W20260709 12:23:23.763569 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 50525.8us
I20260709 12:23:23.769413 32369 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 19 because replica has last-logged OpId of term: 17 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:23:23.751646 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:23.773898 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.753285 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:23.779425 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.780623 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:23.781857 32376 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.787075  5597 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:23.787752  5597 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:23.788890  5597 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.773451 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:23.793286  5713 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:23.794085  5713 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 12:23:23.796348 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.797896 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:23.799396  5567 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:23.799921  5567 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:23.801070  5567 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:23.803013  5691 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:23.806440 31678 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:23.818095 31827 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:23:23.818305 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.819998 31671 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:23:23.821171 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 68045.7us
W20260709 12:23:23.822655 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 151046us
I20260709 12:23:23.828277  5563 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:23.828886  5563 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:23.829420  5563 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:23.831038 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:23.828962  5455 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.005s
I20260709 12:23:23.834030 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:23.835108 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:23.835924 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:23.836108  5566 raft_consensus.cc:2764] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
I20260709 12:23:23.837066  5566 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:23.837760 31642 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:23.838404 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.839192  5672 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:23.834514  5455 consensus_queue.cc:1243] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 21.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:23:23.843534 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:23.849730  5567 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.861150  5576 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:23.805148  5506 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:23.783603  5618 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:23.835933 32370 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:23.916122 32372 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:23.919698 32376 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:23.909140 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:23.912504 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:23.939675 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:23.943532  5586 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:23.944036  5586 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:23.944445  5586 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:23.942349 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:23.954013 31818 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
W20260709 12:23:23.958379 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 303724us
I20260709 12:23:23.947027 31576 catalog_manager.cc:5697] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 7 to 9. New cstate: current_term: 9 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:23.985133 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:23.987141 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 149537us
I20260709 12:23:23.988341  5630 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:23.988574 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:23.989547  5672 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:23:23.990686 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 63110.4us
I20260709 12:23:23.990201 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:23.995636 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:23.996029  5630 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:23.997576  5691 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
W20260709 12:23:24.001510 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 175789us
I20260709 12:23:23.947146  5695 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.035s
I20260709 12:23:23.970127  5576 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:23.998468  5691 consensus_queue.cc:1243] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 17.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:23.955225  5618 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.019893  5690 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:24.025364  5361 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:24.025996  5361 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.026464  5361 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:24.019860  5506 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.028293  5506 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.032168  5506 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:24.033818 31576 catalog_manager.cc:5697] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 9 to 10, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 10 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:23:24.041924 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 99719.3us
W20260709 12:23:24.044634 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.049479  5586 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.055104  5618 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:24.009438  5338 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.030s
W20260709 12:23:24.059036 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53021.6us
W20260709 12:23:24.044914  5747 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.528s	user 0.010s	sys 0.018s
I20260709 12:23:24.054957  5566 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260709 12:23:24.063128  5569 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.012s
W20260709 12:23:24.076553 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 86474.5us
W20260709 12:23:24.072175  5625 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.553s	user 0.005s	sys 0.016s
W20260709 12:23:24.089876 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.094882  5691 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:24.097945  5691 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260709 12:23:24.098309  5586 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:24.110982 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.115320  5151 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:23:24.115779  5675 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.674s	user 0.014s	sys 0.009s
I20260709 12:23:24.116042  5772 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.103s
I20260709 12:23:24.094305 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:24.122332  5151 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:24.129357 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 333038us
I20260709 12:23:24.147193 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:24.152851 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58586.5us
I20260709 12:23:24.155583 31576 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 11 to 13. New cstate: current_term: 13 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:24.157759 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:24.157927 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.087447  5641 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.160097 31827 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:24.121340  5563 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.182803  5576 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:24.137478  5151 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.203399  5576 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:24.204298  5151 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:24.205631  5563 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:24.206034  5576 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:24.206650  5546 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:24.183117 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.207595 31826 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:24.207429  5546 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.210132  5546 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:24.201807  5696 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.215200  5696 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:24.152963  5747 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:24.168686  5625 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 12:23:24.192694 31752 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:23:24.196822 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:24.198747  5675 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:23:24.201817 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 128392us
I20260709 12:23:24.218211  5506 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.221571  5779 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.232811 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.257042 31744 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 19 because replica has last-logged OpId of term: 17 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 14.
I20260709 12:23:24.258136  5779 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.260702  5779 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:24.233538 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.234879 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:24.238030  5361 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.133287  5569 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.108s
I20260709 12:23:24.241856  5641 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:24.265367 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.242250 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:24.245934  5672 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.191063 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:24.270953 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74281.2us
W20260709 12:23:24.272560 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 81807us
I20260709 12:23:24.267954 31576 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 7 to 12, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 12 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:24.264328  5675 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:24.284052 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.283170 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:24.285077 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 277508us
I20260709 12:23:24.292243  5762 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:24.302842 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:24.303033  5545 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:24.303572 31820 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:23:24.303167  5762 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.303676  5545 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.304257  5545 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:24.306147  5762 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:24.307027 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.308020 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:24.309810 31676 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:24.307155 31677 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:24.311131  5788 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:24.307142 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:24.311838 32373 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:24.312776 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:24.313450 31827 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:24.311765  5788 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:24.314944 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.315979  5788 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.317008  5696 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.317600  5696 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:24.326391 31827 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:24.284897  5625 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.329959 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.333544 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.289137  5576 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:23:24.338971 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46376.6us
I20260709 12:23:24.342142 31752 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:24.318475  5695 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:24.341348 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:24.351984 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.352486  5696 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:24.352900 31752 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:24.360944  5545 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.362066 31671 pending_rounds.cc:85] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 14
I20260709 12:23:24.362421 31671 pending_rounds.cc:107] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20260709 12:23:24.362918  5759 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:24.363562  5759 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.366919  5618 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.367620  5618 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.368669 31818 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 18 index: 11. (index mismatch)
I20260709 12:23:24.369370  5641 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:24.369939  5641 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:24.370112  5618 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:24.352128 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:24.371320  5563 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:24.371887  5563 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:24.372128 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:24.372854  5563 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.281818  5506 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.377144 31820 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:24.376937 31576 catalog_manager.cc:5697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 12 to 13, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 13 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:24.390601  5641 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.392211 31671 raft_consensus.cc:2889] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:24.272215  5672 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.379310  5506 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:24.380121 31822 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:23:24.380622 32373 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:24.382764  5361 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:24.389817 31574 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42072 (request call id 243) took 13000 ms (client timeout 15000 ms). Trace:
W20260709 12:23:24.411831 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 70638.4us
I20260709 12:23:24.392249 31576 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 9 to 12, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 12 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:24.343981  5691 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:24.394263 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 93539.3us
I20260709 12:23:24.331043  5747 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:24.413134 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:24.340981  5569 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:24.337291  5576 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:24.418113 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.426136 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:24.366549  5759 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:24.436666 31822 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:23:24.439560  5691 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.426105  5563 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:24.441108  5545 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:24.441874  5545 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.442173  5691 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:24.442363  5545 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.444890  5545 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:24.445485  5545 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:24.427594  5576 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:24.449251 31747 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:24.450901 31820 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
I20260709 12:23:24.415714  5695 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.456943 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:24.457352 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:24.457932 32370 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:24.457998  5695 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:24.458899 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.417846  5566 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:24.471179 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.472121 31753 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:24.475324 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.476691 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:24.478292 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.480751 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:24.481799 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.413446  5625 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.431598  5747 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:24.445935  5506 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:24.487568 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.490268 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.469313  5566 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:24.459985  5790 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:24.490131 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.500239 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.500976 31747 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:24.498665 31820 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:24.502837 31822 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14. Ops: []
I20260709 12:23:24.503839 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:24.506309  5506 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.509101  5506 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:24.502501 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:24.512084 32371 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:24.513200 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.516153  5618 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:24.516857  5618 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:24.520457  5641 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:24.521965 31751 pending_rounds.cc:85] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 9
I20260709 12:23:24.523507 31751 pending_rounds.cc:107] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 17.10
I20260709 12:23:24.523813 31751 raft_consensus.cc:2889] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:24.524266 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:24.500217  5566 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:24.524838 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:24.523128 31675 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:24.525565 31672 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:23:24.526253 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.531033 31747 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:24.531108 31745 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:24.526497 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:24.525558 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:24.535190 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.537580 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.538522 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:24.539451 31818 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:24.540395 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.541766 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:24.541654  5658 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:24.544128 31819 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: [15.11-15.11]
I20260709 12:23:24.546559 31674 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:24.545624 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:24.489281 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0374701630003074s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1912s for a response
I20260709 12:23:24.446007  5545 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.546693 32370 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:24.428171  5569 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.559705 31822 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:24.566088 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:24.569331 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:24.570915 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:24.572227 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:24.572533 31745 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:24.574000 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:24.583413 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:24.584548 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:24.585083 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:24.585960 32369 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:24.586603 31748 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:24.587747 31826 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:24.588845  5597 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:24.523263  5618 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.590039  5793 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:24.477550  5675 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.590744  5793 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.593704 31749 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:24.599179 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:24.600111 31674 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:24.603575 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.604449  5793 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:24.604658 31746 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: [15.8-15.8]
I20260709 12:23:24.606864 32371 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:24.611449 31827 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 10 index: 8. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:24.575033  5658 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.615202 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:24.615228 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.605748  5576 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.576628  5747 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.622246 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:24.625823 31672 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:24.635215 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:24.640626 31818 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:23:24.606410  5593 raft_consensus.cc:2764] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:23:24.607101  5691 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.620885 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:24.671088 32370 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:24.567276  5790 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:24.692137 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 120036us
I20260709 12:23:24.695330 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:24.642688  5787 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:24.734427 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:24.735527 32375 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:24.736688 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:24.708267  5569 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:24.710979  5790 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:24.712215 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:24.670558  5658 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.744247 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:24.719340  5625 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:24.722891  5695 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:24.723296  5672 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:24.723726  5361 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:24.724167 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.687036  5630 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:24.729481 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 110868us
I20260709 12:23:24.684347  5691 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.731499 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:24.733335 32378 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 12:23:24.411640 31574 rpcz_store.cc:269] 0709 12:23:11.272377 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:11.274897 (+  2520us) service_pool.cc:224] Handling call
0709 12:23:24.252880 (+12977983us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:19.783832 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:19.866598 (+ 82766us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:19.866709 (+   111us) write_op.cc:435] Acquired schema lock
0709 12:23:19.866740 (+    31us) tablet.cc:662] Decoding operations
0709 12:23:19.963033 (+ 96293us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:19.963140 (+   107us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:19.963169 (+    29us) tablet.cc:685] Acquiring locks for 18 operations
0709 12:23:19.964556 (+  1387us) tablet.cc:701] Row locks acquired
0709 12:23:19.964617 (+    61us) write_op.cc:260] PREPARE: finished
0709 12:23:19.964815 (+   198us) write_op.cc:270] Start()
0709 12:23:19.965038 (+   223us) write_op.cc:276] Timestamp: P: 1783599799964734 usec, L: 0
0709 12:23:19.965063 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 12:23:19.965843 (+   780us) log.cc:844] Serialized 7217 byte log entry
0709 12:23:21.257814 (+1291971us) spinlock_profiling.cc:243] Waited 28.4 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:21.652586 (+394772us) op_driver.cc:464] REPLICATION: finished
0709 12:23:21.882923 (+230337us) write_op.cc:301] APPLY: starting
0709 12:23:21.913588 (+ 30665us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:22.395346 (+481758us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:22.395392 (+    46us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:22.982712 (+587320us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:23.046013 (+ 63301us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=36,key_file_lookups=36,delta_file_lookups=0,mrs_lookups=0
0709 12:23:23.046067 (+    54us) write_op.cc:312] APPLY: finished
0709 12:23:23.046493 (+   426us) log.cc:844] Serialized 164 byte log entry
0709 12:23:23.208202 (+161709us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:23.208399 (+   197us) write_op.cc:454] Released schema lock
0709 12:23:23.208455 (+    56us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:23.208990 (+   535us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:23.209039 (+    49us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":8568,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":197922,"apply.run_cpu_time_us":45100,"apply.run_wall_time_us":1325999,"cfile_cache_hit":74,"cfile_cache_hit_bytes":79331,"num_ops":18,"prepare.queue_time_us":397022,"prepare.run_cpu_time_us":154987,"prepare.run_wall_time_us":600506,"raft.queue_time_us":276713,"raft.run_cpu_time_us":53646,"raft.run_wall_time_us":165791,"replication_time_us":1687338,"spinlock_wait_cycles":65284352,"thread_start_us":1246901,"threads_started":5,"wal-append.queue_time_us":695238,"wal-append.run_cpu_time_us":20731,"wal-append.run_wall_time_us":3144938}]]}
I20260709 12:23:24.739854 31822 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
W20260709 12:23:24.734347  5151 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.530s	user 0.006s	sys 0.006s
I20260709 12:23:24.753122 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:24.691649  5793 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:24.777686 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:24.784003 31823 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
W20260709 12:23:24.760319 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 334310us
I20260709 12:23:24.762732 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.742080  5593 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:24.775419 31672 raft_consensus.cc:2410] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 19 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:23:24.775205 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:24.793262 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.800424 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:24.792006 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.811851 32369 raft_consensus.cc:2368] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18.
I20260709 12:23:24.812280 31676 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:24.817976 31822 raft_consensus.cc:2368] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18.
I20260709 12:23:24.822870 31670 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:23:24.752219  5566 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.827643 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:24.828939 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.824693 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:24.830348 32375 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:24.828447 31672 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:24.831547 31674 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:24.828585 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:24.832367 32369 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:24.828367 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:24.840606 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:24.786052  5793 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:24.829707 31675 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 13.
W20260709 12:23:24.850893 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 87804.3us
I20260709 12:23:24.856714 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:24.779186  5695 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.859040 31669 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:24.765475  5361 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.860869  5792 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.071s
W20260709 12:23:24.864576  5800 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:23:24.869316  5566 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:24.801060  5772 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.051s
I20260709 12:23:24.776847  5788 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.060s
I20260709 12:23:24.811693  5762 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:23:24.873368 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:24.811983  5151 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:23:24.801249  5696 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:23:24.836232  5473 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 14. Reason: could not achieve majority
I20260709 12:23:24.789858  5593 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:24.917681 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:24.899209 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:24.855898  5793 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:24.904184 31678 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:24.787202  5630 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:24.852746  5576 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:24.878613  5779 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.014s	sys 0.037s
I20260709 12:23:24.915803 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:24.932377 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:24.886317  5545 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:24.939256 32371 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
W20260709 12:23:24.938992 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 153110us
W20260709 12:23:24.948721 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 364244us
I20260709 12:23:24.964766 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:24.965584 31678 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
W20260709 12:23:24.942611  5563 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.501s	user 0.006s	sys 0.007s
I20260709 12:23:24.992190 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:24.946709  5779 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.583s	user 0.015s	sys 0.037s
I20260709 12:23:24.986064 31674 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:23:24.985471 31824 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:24.937844  5772 consensus_queue.cc:1243] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 14.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
W20260709 12:23:24.997298 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 121631us
I20260709 12:23:24.998600 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:25.003319 31673 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.000s	sys 0.022s
W20260709 12:23:25.003813 31673 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.001s	sys 0.023s
I20260709 12:23:24.991099  5545 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:25.022153 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:25.029389 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:25.031903 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 68041.5us
I20260709 12:23:25.056083 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:25.055619 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57137.1us
I20260709 12:23:25.021103  5792 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:25.062961 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:25.064098 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:25.064972 32377 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
W20260709 12:23:25.065479  5810 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
W20260709 12:23:25.066499  5798 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260709 12:23:25.075730  5473 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:23:25.076296  5808 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:23:25.076459  5473 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:25.077613  5473 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:25.078992 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64774us
I20260709 12:23:25.054090  5759 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:25.033167  5597 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:25.026909  5772 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.042s
I20260709 12:23:25.068336  5658 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:25.073841  5545 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:23:25.147245 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 82581.8us
W20260709 12:23:25.154876  5813 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.060s	user 0.001s	sys 0.000s
I20260709 12:23:25.159286  5593 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:25.143132  5625 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:25.165169  5625 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:25.166965  5833 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:25.118786  5792 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
I20260709 12:23:25.169046  5792 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:25.171900  5804 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.086s
I20260709 12:23:25.130332  5691 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:25.175406 32370 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:25.176391 32378 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: [13.10-15.11]
I20260709 12:23:25.179309  5630 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:25.181452  5695 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:25.130947  5772 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 13.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:23:25.183542  5772 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:25.184003  5772 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:25.185210  5772 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:25.188910 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 189553us
I20260709 12:23:25.191017 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:25.196480  5816 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:25.198504 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:25.142781 31792 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.593s (SENT) [suppressed 7 similar messages]
I20260709 12:23:25.208736  5690 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Stepping down as leader of term 13
I20260709 12:23:25.222194  5545 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.222891  5772 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:25.224481 31643 leader_election.cc:400] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18.
I20260709 12:23:25.254894 31643 leader_election.cc:403] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:25.225591  5576 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:25.226819  5805 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.227263  5566 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:25.232981 31671 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:25.218379 32375 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:25.214421 32369 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
W20260709 12:23:25.243669 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 94790.7us
I20260709 12:23:25.259253 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:25.253556 31821 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:25.170281  5792 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.082s
I20260709 12:23:25.272898 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2953) took 1108 ms. Trace:
I20260709 12:23:25.279946 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:25.265080  5545 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:25.286734 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 62409.3us
I20260709 12:23:25.292384  5566 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.293704  5545 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:25.294786  5576 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:25.295548  5838 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:25.296967  5576 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:25.302960  5597 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:25.275539  5827 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:25.305120  5804 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18.
W20260709 12:23:25.307387 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 226349us
I20260709 12:23:25.288275 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:25.309360 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:25.253222  5690 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 13 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:25.279050  5747 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:25.327805 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:25.288699  5762 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.044s
I20260709 12:23:25.290532  5790 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:25.336340 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:25.297004  5838 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:25.294616  5566 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:25.334697 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:25.279415 31825 rpcz_store.cc:276] 0709 12:23:24.164502 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.165635 (+  1133us) service_pool.cc:224] Handling call
0709 12:23:24.214268 (+ 48633us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.214752 (+   484us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.214789 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.215479 (+   690us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.630165 (+414686us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:24.949513 (+319348us) log.cc:844] Serialized 14 byte log entry
0709 12:23:25.272350 (+322837us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.272473 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.272554 (+    81us) raft_consensus.cc:1685] finished
0709 12:23:25.272573 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.272789 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":70294,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":118986,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":8092,"thread_start_us":1004663,"threads_started":3,"wal-append.queue_time_us":570604}
I20260709 12:23:25.337633 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:25.322966  5675 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:23:25.344194 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 145945us
I20260709 12:23:25.345075 31643 leader_election.cc:400] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18.
I20260709 12:23:25.278596  5805 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.336647  5361 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:25.354557 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:25.339990  5804 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:25.343806  5576 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:25.369486 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:25.343520  5675 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.373078 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:25.369002  5563 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:25.396059 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 312071us
I20260709 12:23:25.398619 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:25.398891 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:25.399439 32369 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:25.346081  5641 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.028s
I20260709 12:23:25.339885  5790 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.408672 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:25.374927  5675 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:25.342373  5747 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.370184  5361 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:25.409816 32370 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:25.402426 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:25.411602 32373 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 12:23:25.411914 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 269161us
I20260709 12:23:25.375130  5842 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:25.412040 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:25.403116  5696 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:25.376729  5804 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:25.399935  5576 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:25.410677 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:25.422443 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:25.374182  5473 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.041s
I20260709 12:23:25.400463  5690 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 13.9, Last appended by leader: 9, Current term: 13, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:25.427111  5618 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:25.427560 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:25.428218  5690 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:25.428247 31822 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 12 index: 8. Preceding OpId from leader: term: 14 index: 9. (index mismatch)
I20260709 12:23:25.429086 31823 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:25.428416 31824 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:25.430217  5593 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:25.430696  5593 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:25.431058  5593 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:25.432129  5805 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:25.427641  5618 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:25.394004  5563 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:25.434429 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:25.436164 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:25.437160 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:25.436755 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:25.446197 32377 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:25.444401 31824 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:25.420423  5842 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:25.443485 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:25.421653  5696 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:25.433956  5545 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:25.453043 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:25.437906  5361 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:25.457371 31676 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:23:25.451350 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:25.473727 31677 raft_consensus.cc:2410] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 18 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 8.
I20260709 12:23:25.437579  5618 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:25.433629  5804 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.477182  5597 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:23:25.478401  5845 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:23:25.447345  5838 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 17.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.012s
I20260709 12:23:25.479453  5838 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:25.479629  5841 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:25.479961  5838 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:25.480589  5841 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:25.481045  5838 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.481707  5841 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.484084  5833 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:23:25.485102  5824 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.408s	user 0.000s	sys 0.005s
I20260709 12:23:25.451200  5827 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:25.462083  5759 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:25.464092 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:25.491113 31825 raft_consensus.cc:2410] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:23:25.464999 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:25.421170  5691 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:25.467938 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 2989) took 1170 ms. Trace:
I20260709 12:23:25.468773 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:25.477380  5843 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:23:25.467798  5696 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:25.498346 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:25.491830 31827 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:25.449079  5805 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:25.487922 31823 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 16 index: 12. (index mismatch)
I20260709 12:23:25.456133  5842 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:25.498545 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:25.459278 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:25.482527  5838 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:25.505344 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 46162.6us
I20260709 12:23:25.524495 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:25.506034  5852 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:25.474102  5576 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:25.474115  5361 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:25.493595 31673 rpcz_store.cc:276] 0709 12:23:24.297231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.297481 (+   250us) service_pool.cc:224] Handling call
0709 12:23:24.308682 (+ 11201us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.004250 (+695568us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.004311 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.027873 (+ 23562us) log.cc:844] Serialized 26 byte log entry
0709 12:23:25.142930 (+115057us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:25.142979 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:25.143920 (+   941us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.467621 (+323701us) raft_consensus.cc:1685] finished
0709 12:23:25.467655 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.467858 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29533,"raft.queue_time_us":57522,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":180,"thread_start_us":738453,"threads_started":2,"wal-append.queue_time_us":288273}
I20260709 12:23:25.499557  5691 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:25.519232  5811 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.451s	user 0.002s	sys 0.000s
W20260709 12:23:25.519905 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96043.8us
I20260709 12:23:25.521005 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:25.520910 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:25.504072 31676 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:25.551977 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:25.493930  5759 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:25.503005  5805 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:25.549183 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 101078us
I20260709 12:23:25.564622 31678 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 13.
I20260709 12:23:25.565631 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:25.567376  5832 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:25.572114 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:25.573292 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:25.554507  5792 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:25.560266  5361 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:25.561004  5827 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:25.561125  5805 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:25.585626 31678 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:25.574239  5713 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:25.576810  5690 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:25.586488  5844 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:25.621078 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60587.6us
I20260709 12:23:25.625761 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:25.624094 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:25.591259  5151 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
I20260709 12:23:25.650166 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 888) took 1161 ms. Trace:
I20260709 12:23:25.633180  5597 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.631111  5762 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:25.632203  5792 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:25.649647  5713 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:25.651719  5790 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:25.671173 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2837) took 1076 ms. Trace:
I20260709 12:23:25.660346  5563 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:25.654928  5690 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:25.671598 31821 rpcz_store.cc:276] 0709 12:23:24.595817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.667073 (+ 71256us) service_pool.cc:224] Handling call
0709 12:23:25.670997 (+1003924us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b540043e9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:25.671110 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":476956544}
I20260709 12:23:25.650797 32372 rpcz_store.cc:276] 0709 12:23:24.485139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.493757 (+  8618us) service_pool.cc:224] Handling call
0709 12:23:24.505306 (+ 11549us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.543153 (+ 37847us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.543194 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.543729 (+   535us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.813366 (+269637us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:24.813413 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.813500 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.493445 (+679945us) spinlock_profiling.cc:243] Waited 7.99 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:25.638809 (+145364us) raft_consensus.cc:1685] finished
0709 12:23:25.638840 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.642004 (+  3164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37292,"raft.queue_time_us":38630,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":7890,"spinlock_wait_cycles":18370304,"thread_start_us":269181,"threads_started":1,"wal-append.queue_time_us":463603}
I20260709 12:23:25.661825  5827 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:25.674958  5713 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:25.685937 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2159) took 1158 ms. Trace:
I20260709 12:23:25.663378  5597 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.690569 31669 rpcz_store.cc:276] 0709 12:23:24.545720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.559868 (+ 14148us) service_pool.cc:224] Handling call
0709 12:23:25.682277 (+1122409us) spinlock_profiling.cc:243] Waited 283 ms on lock 0x7b5400046fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:25.682445 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":651778944}
I20260709 12:23:25.691349  5858 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
I20260709 12:23:25.695393  5690 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.668020  5849 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.070s
I20260709 12:23:25.731017 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:25.682191  5846 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:25.717933  5815 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.622s	user 0.001s	sys 0.001s
I20260709 12:23:25.718600 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:25.719425  5809 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.657s	user 0.002s	sys 0.000s
I20260709 12:23:25.719269  5658 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:25.720124 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:25.724713 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:25.726202 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49753.7us
I20260709 12:23:25.726047 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:25.668206  5762 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:25.735699 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:25.666242  5792 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.770351 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:25.687629  5566 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:25.769380 31826 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:25.743613 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:25.775578 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:25.775840 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:25.728688  5691 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:25.777679 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:25.781154 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2314) took 1265 ms. Trace:
I20260709 12:23:25.781638 31748 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:25.779919 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 13 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:25.784806 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:25.681079  5827 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:25.792191 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 61282.6us
I20260709 12:23:25.795164 31820 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:25.796376 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:25.797988  5597 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:25.803910  5762 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:25.789440  5506 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:25.804730  5506 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:25.781575 31753 rpcz_store.cc:276] 0709 12:23:24.515384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.518772 (+  3388us) service_pool.cc:224] Handling call
0709 12:23:24.519167 (+   395us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.519713 (+   546us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:24.519749 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.520300 (+   551us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.761691 (+241391us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:24.762761 (+  1070us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.762864 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.718623 (+955759us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:25.769815 (+ 51192us) raft_consensus.cc:1685] finished
0709 12:23:25.769843 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:25.781071 (+ 11228us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1403,"raft.run_cpu_time_us":99,"raft.run_wall_time_us":100,"spinlock_wait_cycles":31504768,"thread_start_us":240893,"threads_started":1,"wal-append.queue_time_us":468132}
I20260709 12:23:25.798034 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:25.808280 31669 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:25.810990 31826 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:25.771466  5851 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:25.815973 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:25.814411 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:25.816917 31676 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:23:25.817916 31751 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:25.820586 31748 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:25.789669  5566 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:25.744823  5852 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.072s
I20260709 12:23:25.760658  5618 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.831633 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:25.779100  5658 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.835191 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:25.832603 31745 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 12.
I20260709 12:23:25.802109  5805 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:25.805271  5506 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:25.759872  5846 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:25.846480  5846 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:25.843528 31752 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:25.853211 31673 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:25.885561 31748 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:25.892663 31750 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
W20260709 12:23:25.895284 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 63910.4us
I20260709 12:23:25.802160  5593 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:25.897631 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:25.827486  5864 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:25.908102 31750 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:25.910058 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:25.911998 31753 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:25.914290 31672 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:25.820616  5675 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.931180 31675 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 16 index: 10. (index mismatch)
I20260709 12:23:25.931625  5862 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:25.932062  5850 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:25.936575  5792 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:25.925107  5597 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.013s
I20260709 12:23:25.832195  5713 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:25.940268 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:25.940912 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:25.941390  5860 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
I20260709 12:23:25.941560  5871 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:25.942162 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:25.942482  5863 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:23:25.944047  5593 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:26.009495 31750 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:25.979547  5858 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.010396  5858 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:25.944049  5625 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:25.954083 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:25.953908 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:25.955906 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:25.957389 32370 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:25.987344 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:25.827364  5851 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:25.990839 31748 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
W20260709 12:23:25.977190  5793 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.025s	user 0.008s	sys 0.007s
I20260709 12:23:25.990410 31826 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
W20260709 12:23:25.997677 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 373659us
I20260709 12:23:25.999650  5618 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:25.929315  5864 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:25.977975  5641 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:26.005560  5855 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.358s	user 0.000s	sys 0.004s
I20260709 12:23:25.945675 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:26.011698 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:26.025861 31823 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:26.026952 32372 raft_consensus.cc:2376] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18. Re-sending same reply.
I20260709 12:23:26.012765  5871 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.036911 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:26.038034 31748 raft_consensus.cc:2410] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:23:26.038596 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:26.040130 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:26.041560 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 96025.8us
I20260709 12:23:26.042235 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:26.040699 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:26.044409  5864 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.035133  5630 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:25.990370  5759 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.049772  5858 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.050287  5759 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.050518  5597 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:26.034066  5871 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:26.051049  5597 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.051553  5597 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:25.989848  5850 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.047376  5630 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.056391  5675 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:26.057518 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:26.057829 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:26.058377 31753 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:26.058871  5838 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:26.059324 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:26.059501  5658 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:26.059661  5838 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:26.055004  5850 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:26.060824 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.062402 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.059952 31753 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:26.063493 31826 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:25.993646  5862 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.065568 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:26.030474  5696 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.512s	user 0.012s	sys 0.000s
I20260709 12:23:26.069260 31678 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:25.990471  5506 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.051059 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:26.075909 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:26.077217 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:26.079419 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:26.079648  5869 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:23:26.080281 32374 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:26.080286  5827 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:26.083199  5854 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.342s	user 0.003s	sys 0.000s
I20260709 12:23:26.088624 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:26.087891 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.090683 31826 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:26.036595  5625 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:26.012036  5566 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.077521  5618 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:26.097981 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:26.034432  5793 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.101514 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:26.115984 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:26.129737 32377 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:26.129849 32378 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:26.129776 31792 leader_election.cc:336] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 2.216s (SENT)
I20260709 12:23:26.135654 31750 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:26.135808 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:26.085875  5576 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:26.121728 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:26.141631 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:26.089774  5675 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:26.073825  5641 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:26.116449  5872 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:23:26.147271 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 13 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:26.157193 32373 raft_consensus.cc:2376] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16. Re-sending same reply.
I20260709 12:23:26.082818  5838 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.156198 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:26.161764 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.163898 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.163774 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:26.167821  5846 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:26.167939  5833 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.169193  5870 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.116274  5856 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:26.173195 31751 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:26.077025  5696 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.114166  5830 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.128535  5618 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.094771  5858 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:26.172772 31822 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 13.
I20260709 12:23:26.197716  5870 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.217062  5506 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.219350 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.116180  5545 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:26.220587  5545 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:26.223037 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:26.166047  5597 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.226464  5597 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:26.231472 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 101741us
I20260709 12:23:26.096773  5759 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.075s
I20260709 12:23:26.150223  5805 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.099347  5625 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.234862 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.191689  5846 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.210157 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:26.113755  5849 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.087s
I20260709 12:23:26.114646  5827 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.074s
I20260709 12:23:26.201879 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:26.192335  5695 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:26.198027  5856 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:26.116115  5869 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.081s
I20260709 12:23:26.219558  5630 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:26.157550  5675 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.205628  5862 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.228677  5793 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.274359 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:26.275395 31823 raft_consensus.cc:2368] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21.
I20260709 12:23:26.276387 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.277086 31823 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:26.260138  5805 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:26.278599  5805 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.280781  5805 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.254710  5695 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:26.277850 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:26.285488  5772 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.020s	user 0.006s	sys 0.006s
I20260709 12:23:26.211449  5618 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:26.250936  5830 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.289075  5830 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:26.185863  5361 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.572s	user 0.004s	sys 0.009s
I20260709 12:23:26.235015  5625 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.206770 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2312) took 1694 ms. Trace:
I20260709 12:23:26.290899  5361 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.291308  5641 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.292352 31746 raft_consensus.cc:2410] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 20 because replica has last-logged OpId of term: 18 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 8.
I20260709 12:23:26.291201 31744 rpcz_store.cc:276] 0709 12:23:24.512738 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:24.514334 (+  1596us) service_pool.cc:224] Handling call
0709 12:23:24.589684 (+ 75350us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:24.590189 (+   505us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:24.590227 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:24.595438 (+  5211us) log.cc:844] Serialized 26 byte log entry
0709 12:23:24.967650 (+372212us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:24.967691 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:24.967762 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:25.168069 (+200307us) raft_consensus.cc:1685] finished
0709 12:23:25.168091 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.199307 (+1031216us) spinlock_profiling.cc:243] Waited 74.9 ms on lock 0x7b5400000d50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:26.206671 (+  7364us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":670,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":113,"spinlock_wait_cycles":172366208,"thread_start_us":292188,"threads_started":1,"wal-append.queue_time_us":530790,"wal-append.run_cpu_time_us":2006,"wal-append.run_wall_time_us":1015219}
I20260709 12:23:26.293164  5361 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.246991 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:26.243158 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:26.294025 31824 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:26.211517  5566 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.298833  5641 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.298928  5566 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.299551  5641 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.301362  5566 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:26.301697  5641 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.302722  5641 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:26.252956 31751 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:26.258947 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:26.246805  5151 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:26.254096 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:26.303727 31677 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:26.303946 31673 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:26.189630  5838 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.303265  5641 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.306053  5838 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:26.262602 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:26.307111 31670 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:26.306933  5641 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.263123 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.240978  5880 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:26.308701  5880 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:26.309127  5880 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:26.209585  5833 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.311051  5861 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:26.312992  5833 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.313082  5861 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:26.311127  5747 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:26.247704  5846 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:26.262377  5856 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:26.265787  5473 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:26.270929 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:26.359505 31669 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:26.272382  5675 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:26.270419 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 68620.6us
I20260709 12:23:26.360571 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:26.272608 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:26.361673 31671 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:26.285755 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:26.190819 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:26.208324  5842 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.680s	user 0.013s	sys 0.000s
I20260709 12:23:26.322144 31824 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:26.327070 32377 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
W20260709 12:23:26.351982 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 254046us
W20260709 12:23:26.292368  5881 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
W20260709 12:23:26.364125 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 173447us
I20260709 12:23:26.369853 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:26.370534 31824 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:26.370841 32342 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:26.371673 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:26.375313 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 132280us
I20260709 12:23:26.380342  5151 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260709 12:23:26.380846  5151 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:26.381223  5151 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:26.382817 31718 leader_election.cc:400] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21.
I20260709 12:23:26.383194 31718 leader_election.cc:403] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:23:26.384075  5833 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21.
I20260709 12:23:26.359135  5830 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:23:26.385659 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:26.388870 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:26.389691 31823 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 17
I20260709 12:23:26.394021 32378 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:26.405689 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:26.373701 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:26.420998 31824 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:26.452802 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.422816 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:26.422261  5625 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:26.431550 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:26.429117 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:26.441761 32375 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:23:26.385030  5790 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:26.386159  5597 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:26.375159  5827 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:26.460235 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:26.422686 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 51039.4us
I20260709 12:23:26.479065 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:26.427179  5787 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:26.465318 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.468657 31717 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:26.471365 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 65741.1us
I20260709 12:23:26.441746  5869 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:26.484133 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:26.488879 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.427249  5506 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 12:23:26.469275  5864 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:26.413342  5842 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.525065  5625 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:26.525223 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46269.1us
I20260709 12:23:26.488955  5827 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.526301 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:26.477274  5862 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:26.481472  5593 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:26.529502 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:26.523711 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.529821  5864 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.482213  5790 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.533867  5842 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:26.444144  5545 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.431540  5871 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.534950 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:26.489778  5772 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:26.556546 32378 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:26.476719  5473 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.055s
I20260709 12:23:26.482010  5787 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:26.559664 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 192770us
I20260709 12:23:26.562682 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:26.569847 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:26.571182 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 87080.4us
I20260709 12:23:26.581772 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:26.584108 31674 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:26.585594 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:26.589536 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 215945us
I20260709 12:23:26.590258 32369 raft_consensus.cc:2368] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14.
I20260709 12:23:26.533890  5790 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:26.568912  5597 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:26.572367  5849 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:26.491528  5869 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.584975  5787 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:26.613437 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 126965us
I20260709 12:23:26.615767 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:26.598258  5882 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.595186  5858 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:26.618594 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 204484us
I20260709 12:23:26.600831  5566 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:26.602320  5849 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.600996  5836 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:26.609531  5597 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:26.615590  5675 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:26.656850 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:23:26.661399 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 46586.8us
I20260709 12:23:26.666862  5597 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 14.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.669603  5827 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.671854  5871 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.673949  5887 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:26.674592  5887 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.675021  5887 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:26.676540  5884 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 12:23:26.631376  5849 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:26.685189 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 99670.1us
I20260709 12:23:26.706084 32377 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:26.654901  5566 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:26.715257 32377 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:26.710621 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:26.717088 31824 raft_consensus.cc:2368] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20.
I20260709 12:23:26.654861  5576 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:26.610838  5618 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.728760 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:26.729444 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3008) took 1115 ms. Trace:
I20260709 12:23:26.714125 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:26.729530 32373 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:26.628834  5858 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.730316 31678 rpcz_store.cc:276] 0709 12:23:25.614487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.614672 (+   185us) service_pool.cc:224] Handling call
0709 12:23:25.628318 (+ 13646us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.069848 (+441530us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.729108 (+659260us) spinlock_profiling.cc:243] Waited 441 ms on lock 0x7b5400229f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:26.729355 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1013542656}
W20260709 12:23:26.737844  5885 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.252s	user 0.001s	sys 0.000s
W20260709 12:23:26.739400  5894 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
W20260709 12:23:26.741000  5896 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:23:26.741619  5883 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:26.742233 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:26.743068 31821 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:26.743685  5869 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:26.745220  5897 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.121s	user 0.000s	sys 0.000s
W20260709 12:23:26.748231  5907 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.082s	user 0.002s	sys 0.000s
W20260709 12:23:26.750334  5889 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.151s	user 0.000s	sys 0.001s
I20260709 12:23:26.694790  5903 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:26.755075  5903 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.757514 31821 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:23:26.757783  5903 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:26.728509  5904 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.759073  5904 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.760129  5875 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:26.760591  5875 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:26.683764  5675 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:26.761533  5875 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.627537  5882 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.690876 31823 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:26.767717 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.782060 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:26.770262 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:26.795367 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:26.697351 31717 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:26.798403 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2168) took 1309 ms. Trace:
I20260709 12:23:26.678803  5790 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.056s
I20260709 12:23:26.799110 31672 rpcz_store.cc:276] 0709 12:23:25.488644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.489482 (+   838us) service_pool.cc:224] Handling call
0709 12:23:26.798092 (+1308610us) spinlock_profiling.cc:243] Waited 415 ms on lock 0x7b540036f6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:26.798310 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":954329344}
W20260709 12:23:26.765166  5899 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
W20260709 12:23:26.763588  5886 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:23:26.762620  5875 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:26.765201  5618 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:26.817150 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:26.712790  5842 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.083s
I20260709 12:23:26.739436  5803 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:26.814743 32378 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:26.767372  5864 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:26.816183 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 118940us
I20260709 12:23:26.832886 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:26.708637  5880 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.824327 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:26.834968 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:26.835834 31824 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:26.761541  5904 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.802613  5576 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:26.814365  5563 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.837276 31822 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: [16.12-16.12]
I20260709 12:23:26.852476  5871 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:26.836485 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 2972) took 1278 ms. Trace:
I20260709 12:23:26.822958  5909 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 12 FOLLOWER]: Advancing to term 13
I20260709 12:23:26.832336  5545 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:26.812151  5903 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:26.872555  5903 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.865796 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:26.864122 31825 rpcz_store.cc:276] 0709 12:23:25.558556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.558769 (+   213us) service_pool.cc:224] Handling call
0709 12:23:25.570760 (+ 11991us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:25.571054 (+   294us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:25.575206 (+  4152us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.125236 (+550030us) spinlock_profiling.cc:243] Waited 2.04 ms on lock 0x7b18004e2620. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:23:26.835936 (+710700us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:26.836018 (+    82us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:26.836099 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.836201 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.836403 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3362,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":44,"raft.queue_time_us":350556,"spinlock_wait_cycles":4694272,"thread_start_us":688158,"threads_started":2,"wal-append.queue_time_us":1158848}
I20260709 12:23:26.809187  5846 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:26.894124 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2844) took 1172 ms. Trace:
I20260709 12:23:26.898583 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:26.762748  5675 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.863306  5842 consensus_queue.cc:1243] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 20.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:23:26.906358 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:26.907627 31824 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:26.908218 31753 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:26.909284 31824 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:23:26.886972  5563 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.836603  5566 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.14, Last appended by leader: 14, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.929814 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:26.926748 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:26.919598 31744 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:26.923573 31822 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:26.934165 31822 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:26.875665  5864 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:26.886797  5793 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:23:26.961099 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:26.941020 31750 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:23:26.887328  5576 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:26.975523 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:26.980926 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:26.981827 31824 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:23:26.981932 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:26.894996 31819 rpcz_store.cc:276] 0709 12:23:25.721987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.722148 (+   161us) service_pool.cc:224] Handling call
0709 12:23:25.722611 (+   463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.723096 (+   485us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.723151 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:26.315867 (+592716us) spinlock_profiling.cc:243] Waited 3.67 ms on lock 0x7b2c0007dbc8. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:23:26.316342 (+   475us) log.cc:844] Serialized 26 byte log entry
0709 12:23:26.557590 (+241248us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:26.585989 (+ 28399us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.796515 (+210526us) spinlock_profiling.cc:243] Waited 2.95 ms on lock 0x7b18003a4800. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:23:26.891846 (+ 95331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.893678 (+  1832us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:26.893752 (+    74us) raft_consensus.cc:1685] finished
0709 12:23:26.893777 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.894037 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4089,"prepare.queue_time_us":31059,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":249,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":5655,"spinlock_wait_cycles":15242880,"thread_start_us":334776,"threads_started":2,"wal-append.queue_time_us":432398}
I20260709 12:23:26.982614 31674 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
W20260709 12:23:26.983662 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 148365us
I20260709 12:23:26.986280 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1087) took 1133 ms. Trace:
I20260709 12:23:26.882820  5871 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:26.982697 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:26.986774 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 405039us
I20260709 12:23:26.905169  5803 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:27.028462 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:26.986697 31820 rpcz_store.cc:276] 0709 12:23:25.852906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.896496 (+ 43590us) service_pool.cc:224] Handling call
0709 12:23:25.896925 (+   429us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:25.897365 (+   440us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:25.897401 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:25.927815 (+ 30414us) log.cc:844] Serialized 26 byte log entry
0709 12:23:26.478513 (+550698us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:26.478575 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:26.478683 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:26.986064 (+507381us) raft_consensus.cc:1685] finished
0709 12:23:26.986085 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:26.986223 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1713,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":6061,"thread_start_us":457600,"threads_started":1,"wal-append.queue_time_us":632335}
W20260709 12:23:26.921252  5151 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.540s	user 0.013s	sys 0.001s
I20260709 12:23:26.943169  5864 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:26.919756  5903 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:26.951082 31822 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.029814 31669 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:27.067834 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:26.983016  5793 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:26.947360  5842 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:26.913704  5892 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.197s	user 0.000s	sys 0.000s
I20260709 12:23:26.904613  5910 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:27.070803 31822 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:26.923424  5827 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.123s
I20260709 12:23:27.073197 32374 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:23:26.988116  5625 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:23:27.074301 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45945.5us
I20260709 12:23:27.082049 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:27.083457 31792 leader_election.cc:336] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.593s (SENT)
I20260709 12:23:26.984946  5904 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:26.940193  5841 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:26.923906  5869 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.147s
I20260709 12:23:27.068012  5864 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:26.989832  5790 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.087s
I20260709 12:23:27.073760  5793 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:27.090534  5912 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:27.098786 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:27.131351 31822 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:27.132455 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:27.099584  5911 raft_consensus.cc:2764] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:23:27.102173  5909 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Leader pre-election lost for term 13. Reason: could not achieve majority
I20260709 12:23:27.071522  5151 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.106841  5882 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:27.122859 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.135118  5927 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.140290 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:27.108429  5846 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.079336  5842 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.131057  5827 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:27.157398 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:27.158231 31822 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:27.167757 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:27.170449  5871 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:27.134931  5912 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:27.175822  5920 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
I20260709 12:23:27.175664  5151 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.176250  5921 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:27.176682  5916 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:23:27.178000  5876 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.164s	user 0.004s	sys 0.000s
I20260709 12:23:27.179339  5905 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:27.108072  5576 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.157013  5858 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.156792  5473 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:27.152962  5827 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:27.153525  5658 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.028s
W20260709 12:23:27.190835  5856 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.814s	user 0.010s	sys 0.002s
I20260709 12:23:27.158756  5911 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.036s
I20260709 12:23:27.202294  5827 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.204124  5658 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:27.192493  5916 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:27.152305  5927 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.148082  5803 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.206511  5858 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:27.215591  5658 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:27.221312  5916 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:27.255519  5908 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:27.256129  5908 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.256487  5908 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:27.259267  5926 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
W20260709 12:23:27.260589  5787 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.639s	user 0.012s	sys 0.000s
I20260709 12:23:27.269698  5695 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:27.272087  5914 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.138s	user 0.000s	sys 0.001s
I20260709 12:23:27.266027  5842 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:27.260970  5827 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:27.276156  5695 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:27.281322 32378 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
W20260709 12:23:27.281796 31745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2326) took 1549 ms (client timeout 1902 ms). Trace:
W20260709 12:23:27.282267 31745 rpcz_store.cc:269] 0709 12:23:25.732580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.733706 (+  1126us) service_pool.cc:224] Handling call
0709 12:23:27.281590 (+1547884us) spinlock_profiling.cc:243] Waited 59.4 ms on lock 0x7b54000b81d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:27.281703 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":136577920}
I20260709 12:23:27.283083 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:27.283540  5932 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.076s	user 0.000s	sys 0.000s
I20260709 12:23:27.207870  5856 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.288868  5563 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:27.290356  5696 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:27.281370 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:27.246237  5858 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.303512 32373 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:27.316278 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:27.316993 32370 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:27.275624  5851 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:27.307714  5887 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.632s	user 0.012s	sys 0.000s
I20260709 12:23:27.325659 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:27.273664  5870 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:27.308903  5849 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.513s	user 0.012s	sys 0.000s
I20260709 12:23:27.327996 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:27.320822  5929 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.071s	user 0.000s	sys 0.002s
W20260709 12:23:27.339012 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 157001us
I20260709 12:23:27.320271  5909 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.338326 31718 leader_election.cc:400] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14.
I20260709 12:23:27.345207  5927 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:27.301882  5787 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.325620  5869 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.347522 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:27.315820  5804 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 14.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:23:27.334700  5870 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:27.354331  5842 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:23:27.359848 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0858010629999626s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2116s for a response
I20260709 12:23:27.335839  5856 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.361789 32369 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:27.362689 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:27.363591  5793 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:27.372821  5933 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:23:27.372977  5912 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.331808  5887 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:27.367463  5931 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.152s	user 0.000s	sys 0.001s
I20260709 12:23:27.371902 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:27.379863 32373 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:27.380060 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:27.386114 32369 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
W20260709 12:23:27.390399 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 89527.5us
I20260709 12:23:27.390797 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:27.334887  5658 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.352890  5593 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.023s
I20260709 12:23:27.368435  5842 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.336191  5849 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.352967  5844 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.037s
I20260709 12:23:27.356916  5869 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.404129  5925 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:27.404644  5925 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:27.392649  5856 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.411391  5856 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.412032  5887 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.401856  5846 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:27.413627  5856 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:27.414460  5849 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.401995  5841 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.016s
I20260709 12:23:27.391371  5836 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:27.417886  5658 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:27.407413 32373 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:27.390504  5830 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Leader pre-election won for term 14
I20260709 12:23:27.353766  5909 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.410750  5787 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.421693 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 917) took 1179 ms. Trace:
I20260709 12:23:27.359953  5916 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.398227  5842 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:27.440573  5888 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.765s	user 0.000s	sys 0.002s
I20260709 12:23:27.389461  5870 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.424223  5830 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:27.466722 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.467437  5830 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:27.467855 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2328) took 1733 ms. Trace:
I20260709 12:23:27.468226 31746 rpcz_store.cc:276] 0709 12:23:25.734153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.743393 (+  9240us) service_pool.cc:224] Handling call
0709 12:23:27.467631 (+1724238us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b5400294ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:27.467797 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1164716800}
I20260709 12:23:27.468911  5870 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.448762  5793 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.445693  5827 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:27.470688  5827 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.475009  5869 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.480183 31718 leader_election.cc:400] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20.
I20260709 12:23:27.480666 31718 leader_election.cc:403] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:23:27.481829 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:27.451439  5938 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:23:27.453013  5858 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:27.453398 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.423730  5846 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:27.465848 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2324) took 1733 ms. Trace:
I20260709 12:23:27.466678  5916 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:27.469482 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.483461 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:27.484426 31677 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:27.509112 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.427623 32371 rpcz_store.cc:276] 0709 12:23:26.242740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.242897 (+   157us) service_pool.cc:224] Handling call
0709 12:23:26.259950 (+ 17053us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:26.268000 (+  8050us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:26.268035 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:27.283953 (+1015918us) spinlock_profiling.cc:243] Waited 973 us on lock 0x7b180044ede0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240
0709 12:23:27.284984 (+  1031us) spinlock_profiling.cc:243] Waited 53.9 ms on lock 0x7b180044ede0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a
0709 12:23:27.285522 (+   538us) log.cc:844] Serialized 26 byte log entry
0709 12:23:27.354533 (+ 69011us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:27.420946 (+ 66413us) log.cc:844] Serialized 14 byte log entry
0709 12:23:27.421256 (+   310us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.421374 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:27.421432 (+    58us) raft_consensus.cc:1685] finished
0709 12:23:27.421451 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.421617 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2189,"prepare.queue_time_us":69291,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":64,"raft.queue_time_us":5950,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":34453,"spinlock_wait_cycles":128077312,"thread_start_us":124966,"threads_started":2,"wal-append.queue_time_us":100824}
I20260709 12:23:27.511059 31676 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:27.456235  5937 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.110s	user 0.001s	sys 0.000s
I20260709 12:23:27.504766 31752 rpcz_store.cc:276] 0709 12:23:25.732114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.744499 (+ 12385us) service_pool.cc:224] Handling call
0709 12:23:25.775157 (+ 30658us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:25.847203 (+ 72046us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.465480 (+1618277us) spinlock_profiling.cc:243] Waited 68.1 ms on lock 0x7b54001ee1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:27.465758 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":156709120}
I20260709 12:23:27.514179 31673 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:27.515736 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.516044 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.516479 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:27.521682 31744 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:27.517145 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:27.518018 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:27.518564  5909 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.522799 31745 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:27.520391 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:27.525619 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.527423 31747 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:27.526142 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:27.496347  5793 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:27.530393 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:27.534133 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.536531 31669 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:27.496621  5870 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:27.542177 31673 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:27.543198 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:27.547369 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:27.546399 32372 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:27.495450  5864 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.532279  5857 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:27.557291 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:27.559587 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:27.558207 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:27.563210 31676 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:27.540251  5844 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.565508 31675 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:23:27.567075 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66924.1us
I20260709 12:23:27.569123 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:27.571074 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:27.571739 31752 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:27.572045 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:27.553822  5945 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.539391  5909 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.028s
I20260709 12:23:27.573590 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:27.574339 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:27.563710  5911 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:27.577041 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:27.581007 31746 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:23:27.585861 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:27.586678 31824 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:27.591087 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:27.591583 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.573308  5940 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:27.577699  5827 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.594460 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:27.405524  5925 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.570214  5844 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.610081 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:27.588466  5911 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:27.612677 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.594452  5869 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:27.626879 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57560.6us
I20260709 12:23:27.596246  5940 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:27.584532  5945 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.639156  5940 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:27.641083  5945 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:27.641547  5913 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:27.641866  5945 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:27.634501  5844 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:27.642868 31827 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:27.642853  5830 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.644009  5945 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:27.644546  5945 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:27.644922  5945 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:27.645068 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2914) took 1652 ms. Trace:
I20260709 12:23:27.645382  5864 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.645848 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2321) took 1914 ms. Trace:
I20260709 12:23:27.645625 31751 rpcz_store.cc:276] 0709 12:23:25.993259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.039114 (+ 45855us) service_pool.cc:224] Handling call
0709 12:23:27.644905 (+1605791us) spinlock_profiling.cc:243] Waited 95.2 ms on lock 0x7b54003e3ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:27.645000 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":219062528}
I20260709 12:23:27.647011 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.647687 31751 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:27.646420 31749 rpcz_store.cc:276] 0709 12:23:25.731198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.731405 (+   207us) service_pool.cc:224] Handling call
0709 12:23:25.767341 (+ 35936us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:26.065042 (+297701us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:26.065423 (+   381us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.251551 (+186128us) log.cc:844] Serialized 14 byte log entry
0709 12:23:26.356352 (+104801us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:26.356427 (+    75us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:26.356486 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:27.645614 (+1289128us) spinlock_profiling.cc:243] Waited 297 ms on lock 0x7b54002f67d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:27.645661 (+    47us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:27.645779 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":125068,"prepare.queue_time_us":8497,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":86,"raft.queue_time_us":106939,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":21421,"spinlock_wait_cycles":735239168,"thread_start_us":183651,"threads_started":1,"wal-append.queue_time_us":418045}
I20260709 12:23:27.648612 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.649489 31820 raft_consensus.cc:2368] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14.
I20260709 12:23:27.649785 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.651386  5793 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20.
I20260709 12:23:27.651743  5922 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:27.652313  5922 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:27.651811  5911 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 13.10, Last appended by leader: 11, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.653211  5922 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.654917  5869 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.657191  5869 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:27.655119 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.659161 31746 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:27.661223 31752 pending_rounds.cc:85] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 8
I20260709 12:23:27.661049 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:27.661590 31752 pending_rounds.cc:107] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 13.9
I20260709 12:23:27.661885 31752 raft_consensus.cc:2889] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:27.662339 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:27.664465 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:27.645121  5830 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:27.666819  5830 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:27.667164  5830 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:27.668031  5830 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.672730 31820 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 14 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:23:27.674363 31745 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
W20260709 12:23:27.676549  5947 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.112s	user 0.002s	sys 0.000s
I20260709 12:23:27.706144 31669 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:27.638341  5842 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.681797  5945 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.577443  5846 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.688114  5869 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.691859  5903 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:27.688973 31746 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:23:27.747984 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:27.694850  5713 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:23:27.697141 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:27.752653 31750 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:27.697724  5905 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.501s	user 0.007s	sys 0.003s
I20260709 12:23:27.698076  5925 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:27.698997 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:27.699259 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:27.699798 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:27.701001 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:27.762529 31673 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:27.701633  5832 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:27.746146  5922 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:27.677179  5946 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:23:27.720266 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:27.787050 31753 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:27.789279 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:27.790272 31746 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:27.793501 31750 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:27.795372 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:27.740581 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:27.801062 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:27.804158  5952 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:23:27.764034  5905 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
W20260709 12:23:27.742843 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 181931us
I20260709 12:23:27.687909 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.749347 31824 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:27.809480 31745 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 16: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:27.750617 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:27.751608 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:27.755343 31675 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:23:27.761642 31826 raft_consensus.cc:2368] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18.
I20260709 12:23:27.709213 31748 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
W20260709 12:23:27.764647 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 65755.7us
I20260709 12:23:27.783141  5576 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:23:27.781196  5803 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:27.796209 31674 raft_consensus.cc:2368] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18.
I20260709 12:23:27.763751  5832 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:27.814953 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:27.779875  5713 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:27.810987 31677 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:27.836735 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:27.821184 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:27.825881 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.751717  5869 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.832985 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:27.842345 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 13 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:27.843502 31675 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 13: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:27.840420 31750 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:27.845189 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:27.847153 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:27.847913 31677 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:27.847765 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:27.850219 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:27.849331 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:27.758188  5908 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.856515 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:27.857348 31671 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:27.859066 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:27.853601 31825 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:27.876746 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:27.877660 31753 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:27.860011 31826 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:27.878360 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.880144 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:27.841055 31673 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:23:27.879962 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:27.883036 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:27.815500  5793 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:27.917198 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:27.916368 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:27.919435 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:27.869083 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:27.841570  5842 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:27.861281  5851 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:23:27.937577 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 68582.4us
I20260709 12:23:27.940688 31673 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:27.941718 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:27.942000 31718 leader_election.cc:400] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14.
I20260709 12:23:27.943490 31718 leader_election.cc:403] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 13 election: Cancelling election due to peer responding with higher term
I20260709 12:23:27.917940 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.942559 31824 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:27.881668  5887 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:27.885017  5849 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:27.884471  5675 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:27.943600 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:27.885162  5844 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:27.834918  5857 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.099s
I20260709 12:23:27.936643  5576 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:27.925249  5658 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:27.962564 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:28.007354 31753 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:23:27.927408  5949 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:27.995441 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:27.973732  5909 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 13 FOLLOWER]: Advancing to term 14
W20260709 12:23:28.012521 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 68988.8us
I20260709 12:23:27.977864  5849 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:27.939639  5856 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:27.927497  5960 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.111s
I20260709 12:23:27.957963 31673 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:28.024186 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:27.951059  5827 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:27.955930  5945 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:28.038445 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:28.038533 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:28.000232  5851 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:27.882432  5803 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.936478  5804 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.084s
I20260709 12:23:27.951582  5954 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.121s
I20260709 12:23:27.929745  5793 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:27.948163  5593 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.083s
W20260709 12:23:28.056106 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 93678.2us
I20260709 12:23:28.095543 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:28.012949  5916 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:28.096642  5916 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:27.975041  5905 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.061s
I20260709 12:23:28.062261 31753 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 20: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:28.018249  5576 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:28.029553 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:28.045660 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 468652us
I20260709 12:23:28.098960  5576 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:28.085940 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:28.044325  5841 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:28.099748 31673 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 13 FOLLOWER]: Advancing to term 14
I20260709 12:23:28.055986  5849 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:28.099110 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:28.106853 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:28.058086  5844 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:28.099289 31671 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:28.135838 31677 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:28.069211  5827 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.108484 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:28.110144  5956 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:23:28.110026  5506 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:28.110773  5965 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:28.113276  5803 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:28.116873 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 13 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:27.952441  5832 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:28.121755  5793 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:28.122789 31750 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:23:28.122859  5747 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:28.108345 31753 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:28.140200 31749 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:28.069814  5856 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:28.034842  5940 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:28.146910 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:28.095144  5960 consensus_queue.cc:1243] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 18.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:23:28.176407 31673 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:28.145227  5827 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:28.092242  5851 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:28.034170  5949 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:28.167410 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:28.222362 31749 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:28.172953 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:28.177968 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:28.155771 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:28.186314  5793 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:28.226104  5793 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:28.226415 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:28.226776  5793 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:28.227893 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:28.186581  5908 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:28.191965 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:28.230515  5851 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:28.192142  5779 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:28.231184  5779 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:28.231408 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:28.232134 32371 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:28.232486 31673 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:23:28.232216  5779 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.233696  5779 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:28.142103  5593 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 12:23:28.153160  5965 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.196381 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 11 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:28.153644  5506 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:28.204533 31670 raft_consensus.cc:2410] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 17 because replica has last-logged OpId of term: 16 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 11.
I20260709 12:23:28.206247  5856 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:28.209039 31818 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:28.210666  5940 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:28.177870  5832 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:28.156836  5864 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:28.215519 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:28.224220 31674 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:28.224433 31677 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:28.234622 31673 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:28.237409 31749 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:23:28.255937 31744 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 17: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:28.270777 31753 raft_consensus.cc:2410] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 21 because replica has last-logged OpId of term: 18 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 13.
I20260709 12:23:28.277465 31674 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:28.277976 31746 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:28.279539 31749 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:28.283259 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:28.256331  5911 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.012s	sys 0.008s
I20260709 12:23:28.284998 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:28.285789 31746 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:28.287109 32342 leader_election.cc:400] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18.
I20260709 12:23:28.288290 31671 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 18 index: 16. (index mismatch)
I20260709 12:23:28.288595 32342 leader_election.cc:403] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:23:28.285933 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:28.294786  5950 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:28.295493  5950 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:28.251689  5965 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:28.297881  5950 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:28.282699  5566 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:28.303186  5566 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:28.304872 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:28.305321 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.305965  5566 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:28.304682 31818 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:28.307653 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:28.308543 31747 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:28.311251 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 14 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:28.313426  5790 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:28.315162 31748 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:23:28.317265  5950 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.318048  5950 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:28.321349 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:28.321679  5950 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:28.322919  5641 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:28.324663 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:28.325587 31745 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:28.327071  5869 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:28.328831  5922 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.546s	user 0.011s	sys 0.000s
I20260709 12:23:28.326534  5909 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election lost for term 13. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 13. Current term is 14.
W20260709 12:23:28.335762  5901 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 12:23:28.336913  5880 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:28.343019 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:28.343926 31746 raft_consensus.cc:2376] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21. Re-sending same reply.
I20260709 12:23:28.351689  5965 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.352211 31675 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 12. (term mismatch)
I20260709 12:23:28.352643 31675 pending_rounds.cc:85] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 11
I20260709 12:23:28.352896 31675 pending_rounds.cc:107] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 16.12
W20260709 12:23:28.353037 31672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3020) took 1429 ms (client timeout 1863 ms). Trace:
W20260709 12:23:28.353643 31672 rpcz_store.cc:269] 0709 12:23:26.923925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.924077 (+   152us) service_pool.cc:224] Handling call
0709 12:23:28.352838 (+1428761us) spinlock_profiling.cc:243] Waited 571 ms on lock 0x7b540036f6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:28.352974 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1314347904}
I20260709 12:23:28.356570  5506 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.357326 31745 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:28.357749  5506 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:28.359797 31753 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:28.353138 31675 raft_consensus.cc:2889] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:28.361478 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:28.314033  5790 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.366652  5790 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:28.367472  5790 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18.
W20260709 12:23:28.368122 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.610s	user 0.020s	sys 0.020s
I20260709 12:23:28.368409 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:28.368866 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.020s	sys 0.020s
I20260709 12:23:28.367703  5949 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:28.370568 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:28.369289 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 617151us
W20260709 12:23:28.289498  5911 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.013s	sys 0.008s
I20260709 12:23:28.380434  5880 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:28.382803 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:28.388849 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:28.291375  5887 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:28.396811 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:28.399696 32343 leader_election.cc:400] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18.
I20260709 12:23:28.368525  5803 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:23:28.387566 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:28.405627 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 305735us
I20260709 12:23:28.381524  5965 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:28.420699 31745 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 12:23:28.420955 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:28.403644  5803 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.390161  5804 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.029s
I20260709 12:23:28.409847  5841 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:28.424428 31747 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:28.413537  5911 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:28.426236  5965 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.410006  5675 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.029s
I20260709 12:23:28.447562  5473 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:28.432355  5803 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:28.424021  5976 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:28.454440  5846 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:28.473798  5790 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:28.450489  5641 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:28.467792  5973 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:28.398394 31677 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:28.483803 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:28.498648  5973 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.501490  5973 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:28.506865 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.507972  5830 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:28.482447  5846 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.511129  5849 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:28.511591 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.513860  5473 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:28.514535  5473 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:28.515530  5846 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:28.480571  5965 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.519263 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:28.521555  5986 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:28.493614  5927 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:28.468636  5618 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.018s
I20260709 12:23:28.533932  5545 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:28.465688  5911 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.534615  5545 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:28.539680 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:28.540069 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:28.542248 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:28.471235  5576 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:28.543035 31819 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:28.545282  5576 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:28.554085 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:28.556969 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.513504  5849 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:28.565234 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 81512.4us
I20260709 12:23:28.567999 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:28.569752 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:28.572381 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:28.573315 31821 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:28.577692 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:28.587152 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:28.600695 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:28.604066 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:28.605513 31820 raft_consensus.cc:2410] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:23:28.593611 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:28.647153 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:28.483439  5790 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.650760 31820 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 12:23:28.656347 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 377146us
I20260709 12:23:28.593541  5846 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:28.658502 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:28.661623 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:28.678056 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:28.665513 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:28.582719  5986 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:28.661296 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:28.679237 32369 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:28.664664 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 95006us
I20260709 12:23:28.667316 32372 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:28.671887 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:28.668835 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:28.641990  5713 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.679533 31824 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:28.593135  5983 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:28.684348 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:28.683836 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:28.688627 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:28.596272  5909 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.076s
I20260709 12:23:28.695281 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:28.688110  5713 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.699399  5473 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:28.687789  5975 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:28.702383  5975 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.703464 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:28.704398 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 315638us
I20260709 12:23:28.681659 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:28.705265 32369 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
W20260709 12:23:28.705814 31717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.018s	sys 0.030s
I20260709 12:23:28.666081  5984 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:28.716105 31676 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:28.707257 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:28.711050 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:28.712339 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1100) took 1727 ms. Trace:
I20260709 12:23:28.682579  5846 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:28.709345 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:28.696519  5696 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:23:28.712625 31717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.526s	user 0.018s	sys 0.030s
I20260709 12:23:28.717164 32374 rpcz_store.cc:276] 0709 12:23:26.985186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:26.985457 (+   271us) service_pool.cc:224] Handling call
0709 12:23:27.034023 (+ 48566us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:27.078977 (+ 44954us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.708217 (+1629240us) spinlock_profiling.cc:243] Waited 34.7 ms on lock 0x7b540055da54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:28.712280 (+  4063us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":79707904}
W20260709 12:23:28.718493 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 538676us
I20260709 12:23:28.702853  5975 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:28.706398  5973 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Leader election won for term 14
I20260709 12:23:28.710150  5903 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.708508  5695 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:28.651095  5841 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:28.712113  5545 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:28.740437 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58835.9us
I20260709 12:23:28.723661  5846 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:28.745668 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:28.727806  5965 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:28.749022 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 352741us
I20260709 12:23:28.775352 32374 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:23:28.733841  5851 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:28.725664  5793 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:28.783257 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:28.783821 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:28.687927  5986 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.786789 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:28.836820 32378 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:28.750247 31748 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
I20260709 12:23:28.788774 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:28.837729 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:28.792451 32372 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
W20260709 12:23:28.838080 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49384.7us
I20260709 12:23:28.737619  5695 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:28.792455 31747 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:28.839627 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:28.793581  5851 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:28.793846 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:28.775811  5151 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:28.747936  5696 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:28.714423  5985 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.082s
I20260709 12:23:28.799335  5713 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:28.802806  5950 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:28.803431  5949 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 12:23:28.805138  5979 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
I20260709 12:23:28.748629  5945 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:28.811702  5803 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.821116  5790 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:28.822029 32375 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:28.781981  5992 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.828927 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:28.740916  5903 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.787309  5973 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:28.831393  5922 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.735556  5576 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
I20260709 12:23:28.851302  5151 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:28.835314 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:28.835522  5916 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:28.844563 31819 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:28.834290  5793 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:28.861836 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:28.780121  5887 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:28.830974  5830 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:28.883703 31820 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:28.852417  5695 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:28.929385 31747 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
W20260709 12:23:28.872684  5832 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.597s	user 0.005s	sys 0.008s
I20260709 12:23:28.850006  5880 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:28.834219 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 89362.1us
I20260709 12:23:28.893103 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:28.941823 32374 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:28.943619 32378 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:28.845615  5675 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:28.920320 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:28.853530  5976 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:23:28.950867 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:28.876808  5849 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:28.948559 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:28.889340  5827 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.668s	user 0.013s	sys 0.000s
I20260709 12:23:28.953940 31676 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:28.891167  5945 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:28.953852 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 92179.3us
I20260709 12:23:28.955566 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:28.956585 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:28.953850  5979 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.066s	user 0.000s	sys 0.001s
I20260709 12:23:28.961092 32373 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:28.942616  5790 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:28.887692  5911 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:28.891438  5856 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:28.965687 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:28.889140  5992 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.961745  5995 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:29.001649  5790 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.021384  5151 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.006480  5851 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:28.890584  5922 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:29.012880 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:28.871173  5950 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:28.965256  5827 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.034114  5790 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:29.064882  5827 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:29.036629 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 934) took 1074 ms. Trace:
I20260709 12:23:28.966848  5867 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:29.066067  5867 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:29.065570 31749 rpcz_store.cc:276] 0709 12:23:27.962060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.280682 (+318622us) service_pool.cc:224] Handling call
0709 12:23:28.281092 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:28.285303 (+  4211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:28.285337 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:28.384793 (+ 99456us) log.cc:844] Serialized 26 byte log entry
0709 12:23:28.741038 (+356245us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:28.741086 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.741169 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:29.036333 (+295164us) raft_consensus.cc:1685] finished
0709 12:23:29.036365 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:29.036550 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":126,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":76546,"thread_start_us":326926,"threads_started":1,"wal-append.queue_time_us":556233}
I20260709 12:23:29.067773  5867 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.048098  5875 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:29.069227  5875 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:29.071869  5803 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:29.070607  5875 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.040881  5151 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:29.031512  5856 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.084586  5903 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:29.048913  5973 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 12.9, Last appended by leader: 9, Current term: 14, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.093840  5875 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:29.096913  5992 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:29.100121 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 134469us
I20260709 12:23:29.102478 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1017) took 1341 ms. Trace:
I20260709 12:23:29.062371 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:28.974740 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 14 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:29.121023 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:29.108340 31675 rpcz_store.cc:276] 0709 12:23:27.761563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.844293 (+ 82730us) service_pool.cc:224] Handling call
0709 12:23:28.351835 (+507542us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:28.361472 (+  9637us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:29.102183 (+740711us) spinlock_profiling.cc:243] Waited 507 ms on lock 0x7b5400046ad0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:29.102388 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1166291200}
I20260709 12:23:28.947811  5830 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:29.128497 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 153863us
I20260709 12:23:29.133047 31748 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:29.134137 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:29.135243 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:28.994421  5984 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:29.137008 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:29.144917 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:29.147089 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:29.055557  5994 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.093s	user 0.000s	sys 0.000s
I20260709 12:23:29.141396 32374 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:29.022094  5921 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:29.149848 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:29.163036 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:29.151255 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:29.152951 31670 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 12:23:29.153151 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:29.140744 31673 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [17.11-18.12]
I20260709 12:23:29.030793  5850 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:29.021659  5949 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:29.026209  5944 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.182204 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:29.157444 32377 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:23:29.158288 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:29.045852  5473 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:29.166872 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:29.021245  5985 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.134s
I20260709 12:23:29.137261  5995 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:29.057765  5922 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:29.061326  5950 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:29.176859 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:29.184283 31676 raft_consensus.cc:2368] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:29.140107  5593 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader election won for term 14
W20260709 12:23:29.185971 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 458914us
I20260709 12:23:29.148540  5849 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:29.185810  5473 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:29.146531  5805 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:29.027217  5911 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.149s
I20260709 12:23:29.218381 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:28.968318  5696 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.222781 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:29.219571 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:29.161846  5921 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.162257  5984 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:29.255636 31670 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:29.258430  5473 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:29.260896  5975 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.523s	user 0.012s	sys 0.000s
I20260709 12:23:29.181608  5887 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.262912 31748 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
W20260709 12:23:29.265256 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.269819  5985 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.302971 32378 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:29.270502  5630 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:29.275205  5913 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:23:29.282542 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63111.2us
I20260709 12:23:29.287367  5849 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:29.048779  5945 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.229993  5593 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:29.209067  5944 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.203471  5949 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.303916 32369 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 12:23:29.310002  5989 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.456s	user 0.000s	sys 0.001s
I20260709 12:23:29.048358  5880 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.290488  5986 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:29.307932 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:29.331276 32374 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:29.292522  5921 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:29.338789 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:23:29.355716 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47899.9us
W20260709 12:23:29.309376  5974 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.769s	user 0.006s	sys 0.001s
I20260709 12:23:29.310158  5951 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:29.357039 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:29.311861  5473 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:29.302599  5975 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.365700 31818 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:29.301661  5695 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.369773 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2950) took 1007 ms. Trace:
I20260709 12:23:29.327042  5856 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:29.371728 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 255626us
I20260709 12:23:29.371662 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:29.332683  5981 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:23:29.376017 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 102486us
I20260709 12:23:29.377137 32374 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 14. Preceding OpId from leader: term: 18 index: 15. (index mismatch)
W20260709 12:23:29.378257 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.374255 31744 rpcz_store.cc:276] 0709 12:23:28.362779 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.364812 (+  2033us) service_pool.cc:224] Handling call
0709 12:23:28.365331 (+   519us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:28.365796 (+   465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:28.365824 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:28.366582 (+   758us) log.cc:844] Serialized 52 byte log entry
0709 12:23:28.711331 (+344749us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:28.776546 (+ 65215us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.777285 (+   739us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.777369 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:29.369522 (+592153us) raft_consensus.cc:1685] finished
0709 12:23:29.369544 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:29.369701 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":332,"prepare.queue_time_us":64065,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":24245,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":155,"thread_start_us":344120,"threads_started":1,"wal-append.queue_time_us":520428}
I20260709 12:23:29.305905  5985 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.309542  5997 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.216s
I20260709 12:23:29.385720 31678 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:29.400484 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:29.401957 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:29.403482  5991 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.609s	user 0.005s	sys 0.000s
I20260709 12:23:29.405539 32372 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 12:23:29.403513 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:29.358909  5995 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:23:29.403936  6000 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
I20260709 12:23:29.373670  5805 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:29.367916  5951 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:29.379874  5986 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.382462  5850 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:29.308890  5630 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.416591  5593 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 12.7, Last appended by leader: 7, Current term: 14, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.418725  5999 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.006s
I20260709 12:23:29.407085  5830 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:29.421741 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1014) took 1677 ms. Trace:
I20260709 12:23:29.421963 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2949) took 1059 ms. Trace:
I20260709 12:23:29.384406  5976 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:29.422185 31669 rpcz_store.cc:276] 0709 12:23:27.744475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.744732 (+   257us) service_pool.cc:224] Handling call
0709 12:23:27.754047 (+  9315us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:27.760654 (+  6607us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:27.815246 (+ 54592us) log.cc:844] Serialized 14 byte log entry
0709 12:23:28.139492 (+324246us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:28.154237 (+ 14745us) log.cc:844] Serialized 26 byte log entry
0709 12:23:28.182203 (+ 27966us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:28.182261 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:28.182374 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:29.151681 (+969307us) spinlock_profiling.cc:243] Waited 4.39 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:29.421410 (+269729us) raft_consensus.cc:1685] finished
0709 12:23:29.421526 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:29.421675 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":53893,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":20155,"raft.queue_time_us":3657,"raft.run_cpu_time_us":317,"raft.run_wall_time_us":27276,"spinlock_wait_cycles":10101504,"thread_start_us":221986,"threads_started":1,"wal-append.queue_time_us":534061}
I20260709 12:23:29.422320 31746 rpcz_store.cc:276] 0709 12:23:28.362461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.362901 (+   440us) service_pool.cc:224] Handling call
0709 12:23:28.363259 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:28.363659 (+   400us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:28.363718 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:28.364300 (+   582us) log.cc:844] Serialized 26 byte log entry
0709 12:23:28.659658 (+295358us) spinlock_profiling.cc:243] Waited 8.08 ms on lock 0x7b180042e260. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:23:29.023404 (+363746us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:29.023433 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:29.023490 (+    57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:29.421667 (+398177us) raft_consensus.cc:1685] finished
0709 12:23:29.421698 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:29.421905 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":111,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":125,"spinlock_wait_cycles":18588928,"thread_start_us":363577,"threads_started":1,"wal-append.queue_time_us":941635}
I20260709 12:23:29.429890 31746 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:29.424404 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:29.432116 31744 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:29.413509  5856 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.030s
I20260709 12:23:29.379788  5984 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:29.447146  6003 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:23:29.447796  5695 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:29.449074  6011 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:23:29.412114  5759 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.041s
I20260709 12:23:29.428040  5850 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:29.453948  5846 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.663s	user 0.010s	sys 0.000s
I20260709 12:23:29.401145  5867 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:23:29.427853  5986 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.457897  5975 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:29.460410  6006 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
I20260709 12:23:29.471022 31744 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:23:29.472772 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:29.460887  5630 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:29.491590 31669 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
W20260709 12:23:29.494429  6005 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:23:29.498447 31751 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:29.494253 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:29.386552  5981 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.501857 31670 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 19: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:29.454420  5846 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.514384  5985 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:29.540038  5981 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:29.541541  5846 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:29.542444  5905 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:23:29.551510 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:29.570530 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.541564  5949 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:29.546833  5905 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:29.579665  5792 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.786s	user 0.008s	sys 0.009s
I20260709 12:23:29.569422  5986 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:29.595176  5875 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.501s	user 0.008s	sys 0.008s
I20260709 12:23:29.608670  5905 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:29.584138  5792 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.790s	user 0.008s	sys 0.009s
I20260709 12:23:29.613303  5473 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:29.620580  5473 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.615288  5841 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:29.621719  5841 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:29.622750 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.614100  5630 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.624155  5630 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.594310  5921 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.635560 31826 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:23:29.626583  5630 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:29.640053 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2881) took 2044 ms. Trace:
I20260709 12:23:29.642541 31825 rpcz_store.cc:276] 0709 12:23:27.595647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:27.596030 (+   383us) service_pool.cc:224] Handling call
0709 12:23:29.639806 (+2043776us) spinlock_profiling.cc:243] Waited 241 ms on lock 0x7b540037a854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:29.639950 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":554859776}
I20260709 12:23:29.644490  5905 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:29.637707  5473 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:29.667743  5841 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:29.637115  5921 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:29.673238 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 926) took 1036 ms. Trace:
I20260709 12:23:29.673682 32375 rpcz_store.cc:276] 0709 12:23:28.637080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.646044 (+  8964us) service_pool.cc:224] Handling call
0709 12:23:29.673048 (+1027004us) spinlock_profiling.cc:243] Waited 141 ms on lock 0x7b54005947d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:29.673174 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":324412672}
W20260709 12:23:29.668872  5675 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.711s	user 0.014s	sys 0.000s
I20260709 12:23:29.699965  5675 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:29.684880 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:29.686024 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:29.703852 32376 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:29.686415 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:29.705942 32378 raft_consensus.cc:2410] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 19 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:23:29.706461 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:29.711202 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.711536 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:29.713249  5151 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.713932  5151 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.716576  5151 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:29.718923  5793 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:29.686105 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:29.719734  5792 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.720283  5792 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.721369 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:29.721925 32372 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:29.723340 31750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 929) took 1924 ms (client timeout 2055 ms). Trace:
W20260709 12:23:29.723771 31750 rpcz_store.cc:269] 0709 12:23:27.798910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.038225 (+239315us) service_pool.cc:224] Handling call
0709 12:23:29.723153 (+1684928us) spinlock_profiling.cc:243] Waited 48.5 ms on lock 0x7b5400064854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:29.723270 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":111626496}
I20260709 12:23:29.687345 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:29.726497 32375 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:29.686550 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:29.718844 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 496111us
I20260709 12:23:29.730026 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:29.730116 32372 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:29.729866 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:29.733930  5792 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:29.732337 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:29.737267 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 14 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:29.707829  5675 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:29.739796 31822 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:23:29.746042 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:29.746189 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:29.747502 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:29.747813 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:29.748240 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:29.749356 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:29.749459 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:29.750939 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:29.751538 31824 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:29.757992 31819 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:23:29.750429 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:29.764432 31820 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:29.771723 31824 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:29.741127 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:29.782438 31822 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 19. Ops: []
I20260709 12:23:29.787899  5850 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.789911  5833 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:29.790738  5833 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:29.793836  5833 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:29.798370 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:29.800797  5985 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:29.802857  5830 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:29.803462  5830 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:29.804563  5830 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.806205  5830 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:29.802049  5985 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:29.808398  6012 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:29.810585  5984 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:29.811121  5984 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:29.812110  5984 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.813824  5984 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:29.729210  5630 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.816865  5630 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:29.821390  5990 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.027s	user 0.008s	sys 0.003s
I20260709 12:23:29.824940  5951 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:29.828595  6009 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.443s	user 0.000s	sys 0.001s
I20260709 12:23:29.828390 31746 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: []
W20260709 12:23:29.840708 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 99654.5us
W20260709 12:23:29.847282 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:29.868225 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 138459us
W20260709 12:23:29.868916 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 138979us
I20260709 12:23:29.740288  5950 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:29.871017 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 82659.9us
I20260709 12:23:29.731670  5921 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:29.833768  5983 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:29.874083 31750 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:23:29.876494 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 928) took 2078 ms. Trace:
I20260709 12:23:29.878279 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:29.738186  5913 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:29.880410 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:29.887890 31751 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:29.886474 31753 rpcz_store.cc:276] 0709 12:23:27.798572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.277039 (+478467us) service_pool.cc:224] Handling call
0709 12:23:28.277359 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:28.360485 (+ 83126us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:29.876241 (+1515756us) spinlock_profiling.cc:243] Waited 82.1 ms on lock 0x7b54004038d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:29.876431 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":188742528}
I20260709 12:23:29.889158 31792 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:29.887007 31748 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:29.894141 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:29.781800  5975 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:29.897846 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:29.898813 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:29.899452 31753 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:29.894825 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:29.902115 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:29.899461 31747 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:23:29.826558  5951 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:29.880918  5792 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:29.913832 32374 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: [13.10-16.11]
I20260709 12:23:29.915994 31717 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:29.916663 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:29.886667  5986 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:29.922192 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:29.883247  5983 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:29.925060 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:29.918416  5833 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:29.927660  6004 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:29.928150  5779 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:29.928314  5696 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:29.929230  5870 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:29.884064  5950 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.934012 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:29.935660 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2956) took 1192 ms. Trace:
I20260709 12:23:29.887372  5695 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:29.939522 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:29.943064 31717 connection.cc:256] RPC call timeout handler was delayed by -1.341749438000079s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18630000000000002s for a response
I20260709 12:23:29.900321  5975 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.951442 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:29.978341 31824 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:29.911826  5841 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:29.905352  5945 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:23:29.952412  6004 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.959945 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:30.005345 31819 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
W20260709 12:23:29.930888 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:29.966569 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:29.966902 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 194112us
I20260709 12:23:29.888103  5921 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:29.969981 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:29.921460  5986 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:29.914683  5792 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:29.974970 31752 rpcz_store.cc:276] 0709 12:23:28.743161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:28.745286 (+  2125us) service_pool.cc:224] Handling call
0709 12:23:29.728326 (+983040us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:29.728529 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:29.728554 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:29.728596 (+    42us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:29.728632 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:29.728689 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:29.935491 (+206802us) spinlock_profiling.cc:243] Waited 944 ms on lock 0x7b5400064850. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:29.935601 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2170206464}
I20260709 12:23:30.083928 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 14 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:30.084798 31752 raft_consensus.cc:2376] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14. Re-sending same reply.
I20260709 12:23:29.924319  5850 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:29.981033 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:29.909067  5804 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.053s
I20260709 12:23:29.982707 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:29.911545  5793 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:29.911691  5988 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 12:23:29.953382 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:30.062975  5827 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:29.957506  5151 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:29.955483  5779 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:30.059234 31744 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:30.129791 31744 raft_consensus.cc:606] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:30.130403 31744 raft_consensus.cc:614] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:29.987843 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:30.132200 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.051954  6004 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:29.954022  5858 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 12:23:30.092926 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:30.096925  5792 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:29.984447  5975 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:30.094405 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:30.114660 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 205769us
I20260709 12:23:29.964267  6012 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 11.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.110s
I20260709 12:23:29.983541  5841 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:30.115696 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:29.993180 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53789.5us
I20260709 12:23:30.185711 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.186173 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:30.186610 31674 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:30.185415 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:30.081496 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.187844 31675 raft_consensus.cc:2410] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 17 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:23:30.059891  5695 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:30.188093 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:30.189782  5695 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.087427  5986 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:30.139377 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 151631us
I20260709 12:23:30.142285 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:30.095489 32374 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:29.974792  5696 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:30.123071  6003 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.065s	user 0.001s	sys 0.000s
I20260709 12:23:30.170941 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:30.116647  5793 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.175498 31677 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:30.076566  5913 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:30.223062  6026 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.244s	user 0.000s	sys 0.001s
I20260709 12:23:30.223686 31746 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:23:30.223419  5913 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:30.226511 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 56080.3us
W20260709 12:23:30.227890 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:30.178293 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 224998us
I20260709 12:23:30.229725 31677 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:30.198777 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:30.176522  5151 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:30.208063  5793 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:30.216271  6027 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.226s	user 0.001s	sys 0.000s
I20260709 12:23:30.217680  5696 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:30.219218  5944 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:30.220072  6041 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:30.092573  5905 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:30.225684 31753 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 19: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:30.227591 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:30.273213  5804 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.273824  5804 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.175132  5846 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
W20260709 12:23:30.281881 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54351.9us
I20260709 12:23:30.173749  5630 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:30.247022 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:30.247202 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.289572 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:30.294783 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:30.295886 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.297166 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:30.299655 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.310873 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.311640 31670 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:23:30.247635 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:30.194031  5875 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:30.282613  6038 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.051s
I20260709 12:23:30.326948  5875 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.073096  5983 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.183631  5880 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:30.310499  5641 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:30.329265  5875 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.329500  5641 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.329864  5875 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:23:30.197877  5841 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:30.331745  5641 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.250119  5988 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:30.332402  5988 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.332801  5988 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:30.334046  5827 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:30.201555  5925 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.334548  5827 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:30.334916  5925 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:30.263285  6031 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:23:30.263029 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:30.263610  6043 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:30.265362  5921 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:30.279126 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.355757 31669 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:30.356978 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.358286 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.279215 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:30.280352 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51304us
I20260709 12:23:30.361292 31674 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:30.312536  6030 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 12:23:30.280934  5803 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:30.281136 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:30.237061 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:30.364154 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 127177us
I20260709 12:23:30.282694 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:30.315559 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.310247  5944 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:30.312979  5759 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.270578  5905 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:30.331799  5804 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.396649  5566 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:30.397307  5566 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:30.397891 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.253980  5151 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.337649 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:30.314620 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 222926us
I20260709 12:23:30.403409 31672 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:30.404729 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:30.348692 31676 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:30.280471 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:30.367489 31818 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 8. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 12:23:30.369910 31669 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:23:30.407878 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.314949 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.408454 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.370461 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:30.409469 31676 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:30.382550 32372 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
W20260709 12:23:30.383073 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 100492us
I20260709 12:23:30.351027  6038 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:30.384681 32371 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:30.348054  5981 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.381744  5759 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.388103  5846 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:30.388641  5921 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:30.389074  5844 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.391815  5986 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.393499  6004 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.398103  5566 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.396711 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.373510 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:30.413429 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 2965) took 1322 ms. Trace:
I20260709 12:23:30.406252 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.383234 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:30.398582 31670 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:30.482923 31672 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:30.415746 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.417131 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:30.418118 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:30.418879 31717 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.863s (SENT) [suppressed 10 similar messages]
I20260709 12:23:30.419782 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.411991  5875 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:30.384436  5856 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.097s
I20260709 12:23:30.371869  5949 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:30.436774  5950 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.440539  5921 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:30.439639 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:30.537163 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 248598us
I20260709 12:23:30.442126 31674 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:30.444914  6052 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.396215  5905 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:30.471985 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 347701us
I20260709 12:23:30.415495 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:30.484102 32371 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
W20260709 12:23:30.484632 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 65810.2us
I20260709 12:23:30.481243 31675 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
W20260709 12:23:30.485411 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 69742.6us
I20260709 12:23:30.482472 31749 rpcz_store.cc:276] 0709 12:23:29.090899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.091118 (+   219us) service_pool.cc:224] Handling call
0709 12:23:29.091609 (+   491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:29.707071 (+615462us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:29.707104 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:29.707632 (+   528us) log.cc:844] Serialized 26 byte log entry
0709 12:23:29.998834 (+291202us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:29.998909 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:29.999041 (+   132us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:30.413229 (+414188us) raft_consensus.cc:1685] finished
0709 12:23:30.413249 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.413373 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":614984,"raft.queue_time_us":620873,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":95,"thread_start_us":290672,"threads_started":1,"wal-append.queue_time_us":379008}
I20260709 12:23:30.384788  5925 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.508168 31669 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:30.524482 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.527194 31672 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:30.522297 31751 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:30.508106  5875 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:30.372398  5944 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.529845 31821 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:23:30.522581  5913 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:30.445122  5803 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.537856 31820 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:30.545014 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.528781  5949 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:30.440927  5846 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.439502  5792 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.564064  5944 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:30.573414 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.545846 31747 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 19. Ops: []
I20260709 12:23:30.430074  6038 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.422986  6048 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.085s
I20260709 12:23:30.544680 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:30.563674 31676 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:30.545269  5830 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.002s
I20260709 12:23:30.569746 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:30.437670  5981 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.538620  5921 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:30.474537  5844 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.542042  5566 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:30.624670  5844 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.624747  6012 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:30.570873  6036 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:30.625980  6036 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.627060  5805 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.627616  5805 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.628584  6036 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:30.629702  5805 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.570119  5803 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.596230 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.632077  5803 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.632401 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.632745  5792 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.633567  5792 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:30.633749 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.636117  5759 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:30.637182  6004 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:30.637885  6004 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:30.638382  5827 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.597489  5151 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:30.639503  5151 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:30.639914  5151 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:30.607005  5630 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.640331 31671 raft_consensus.cc:2376] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19. Re-sending same reply.
I20260709 12:23:30.640710  5827 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:30.640630  5630 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.641430  5827 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:23:30.642513 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 98014.5us
I20260709 12:23:30.633805 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3012) took 1477 ms. Trace:
I20260709 12:23:30.643699  5986 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:30.573307  5951 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.071s
I20260709 12:23:30.552045  6052 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.612190  5779 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.643352 31826 rpcz_store.cc:276] 0709 12:23:29.156910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.157123 (+   213us) service_pool.cc:224] Handling call
0709 12:23:29.167993 (+ 10870us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:29.641053 (+473060us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.633469 (+992416us) spinlock_profiling.cc:243] Waited 467 ms on lock 0x7b54004475d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:30.633724 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1074652416}
I20260709 12:23:30.575824  5949 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:30.647917 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:30.660423  5151 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.598518  5850 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:30.571795  5841 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.621850  5975 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.667816 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.676599  5151 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:30.668612  6056 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.053s	user 0.002s	sys 0.000s
I20260709 12:23:30.678965 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:30.671181  6057 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.054s	user 0.000s	sys 0.003s
I20260709 12:23:30.674932  5779 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:30.630376  5988 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:30.674144  6054 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.063s	user 0.000s	sys 0.003s
W20260709 12:23:30.674113  6051 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.071s	user 0.000s	sys 0.002s
I20260709 12:23:30.672099  5844 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:30.671936  6045 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:30.706067 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:30.718807 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.649281  6055 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.053s
I20260709 12:23:30.684157  5833 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:30.708703  5841 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:30.690845  5695 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:30.716025  5844 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:30.716847 31744 raft_consensus.cc:2368] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21.
I20260709 12:23:30.703239  5630 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:30.717329  5988 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:30.720642 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.723335  5695 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:30.725229  5695 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:30.725519  6053 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:23:30.726449  5793 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.727854  6052 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.728760  5793 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.731623  5844 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 17.15, Last appended by leader: 15, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.719007  6045 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:30.735126  6045 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:30.735564  5988 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.736204  5988 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.738570  5988 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:30.727881  5975 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.739017 31751 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:23:30.726310  5981 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:30.745747 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60637.7us
W20260709 12:23:30.722733 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 182626us
I20260709 12:23:30.749001 31750 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:23:30.749181 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.758144 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:30.759516 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:30.767902 31751 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:30.768422 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.770140 31749 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:30.766402 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.771970 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.774863 32377 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [18.13-18.13]
I20260709 12:23:30.777702 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:30.782609 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:30.783558 32375 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:30.783861  5151 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:30.784507 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:30.785831 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:30.787712  5151 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:30.788254  5151 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:30.788964 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:30.789089  5151 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.788264 31753 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:30.789839 31671 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:30.791093 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.792141 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.793696  6045 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.795492 32371 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: [13.10-16.11]
I20260709 12:23:30.796398  6045 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.797106  6045 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:30.782035  5833 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:30.798965 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.799825  6045 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:30.768854  5696 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:23:30.803680 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:30.809407 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.813709 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:30.816920 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:30.817461 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:30.817878 31669 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:30.818799 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.818879 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:30.819674 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:30.820422 31672 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:30.804872 32377 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:23:30.764986  5927 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.764618  5887 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.821113 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:30.779247  6052 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:23:30.823341 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 255278us
I20260709 12:23:30.832450 31672 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:30.838013 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:30.833856 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.839128 31676 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:23:30.839407 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:30.800994  5792 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:30.842540 31791 leader_election.cc:400] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:30.844853 31791 leader_election.cc:403] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 12:23:30.844779 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.770956  5975 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.060s
I20260709 12:23:30.846096 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:30.847524 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:30.876807 31820 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:30.847888 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.855412 31749 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:30.877331 31824 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:30.877825 31749 raft_consensus.cc:606] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:30.878618 31749 raft_consensus.cc:614] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:30.837167  5641 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.036s
I20260709 12:23:30.836486  5887 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.870488 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:30.883837 31826 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:30.807278  5833 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:30.792162  5880 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.043s
W20260709 12:23:30.885181 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 315507us
I20260709 12:23:30.896298 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:30.899159 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:30.899874 31822 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:30.900020 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.897614 31818 raft_consensus.cc:2410] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:23:30.904860 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.861977 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:30.882113 31669 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:30.882788 31671 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:30.875731 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:30.910661 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:30.912424 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 50552.7us
I20260709 12:23:30.915720 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.920225 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.896915  6061 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:30.922972  6041 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:30.909801  5981 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:30.898845  5846 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:30.838065  5927 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.929841  5695 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:30.931488  5921 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:30.932955 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.934885  6038 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:30.901177  6052 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:23:30.935982 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48527.8us
I20260709 12:23:30.981420 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:30.934906  5985 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:30.982568 31820 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:30.940119 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:30.984359 31825 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:30.987293 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:30.988054 31821 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:30.989245 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:30.989739  6044 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:30.991719 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:30.992496 31748 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:30.994787 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3023) took 1283 ms. Trace:
I20260709 12:23:30.995244 31827 rpcz_store.cc:276] 0709 12:23:29.711088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.719716 (+  8628us) service_pool.cc:224] Handling call
0709 12:23:29.720073 (+   357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:29.736303 (+ 16230us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:29.736348 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:29.737590 (+  1242us) log.cc:844] Serialized 26 byte log entry
0709 12:23:30.626212 (+888622us) spinlock_profiling.cc:243] Waited 24.5 ms on lock 0x7b18004bf8e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:23:30.992064 (+365852us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:30.992106 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:30.992179 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:30.994376 (+  2197us) raft_consensus.cc:1685] finished
0709 12:23:30.994401 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:30.994692 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15831,"raft.queue_time_us":19967,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":6945,"spinlock_wait_cycles":56350080,"thread_start_us":365652,"threads_started":1,"wal-append.queue_time_us":1177345}
I20260709 12:23:30.944451  5887 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:30.883581  5792 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:31.002661  5792 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.003762 31821 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:23:30.949618  5927 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:30.949739 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 101974us
I20260709 12:23:31.008095 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:30.951932 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:31.009341  5975 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:31.010821 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:30.952653 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 77034.5us
I20260709 12:23:31.013315 31745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:31.014197 31745 raft_consensus.cc:606] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:31.014696 31745 raft_consensus.cc:614] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:30.970137 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:31.019713  6077 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:31.021806 31745 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:31.019881 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:31.076965 32371 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 11 index: 9. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
W20260709 12:23:31.077572 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57812.4us
I20260709 12:23:31.024216 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:30.948837  5827 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:31.078851  5973 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:31.078994  5827 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:31.080258  5827 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.026793  6012 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.083357  6012 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.085801  6012 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:30.990438  6044 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:31.030435  6055 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:30.974498 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:31.034569  5618 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:31.039183  5927 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:31.042248 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 940) took 1057 ms. Trace:
W20260709 12:23:31.049844 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.053107 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:31.054381 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:31.032076  5856 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:31.056766 32372 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:31.057361  6062 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:31.057266 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:31.086562  5905 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:31.057279 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:31.065578  5641 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:31.086969  6055 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.066246  5880 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:31.068459 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 943) took 1072 ms. Trace:
I20260709 12:23:30.994282 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:31.073653 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:31.174849 31744 rpcz_store.cc:276] 0709 12:23:29.996487 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.186176 (+189689us) service_pool.cc:224] Handling call
0709 12:23:31.068124 (+881948us) spinlock_profiling.cc:243] Waited 403 ms on lock 0x7b5400000d54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:31.068365 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":927127296}
I20260709 12:23:31.175348 32374 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:31.075511 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:31.055462  5887 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:31.094029 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:31.094621 32377 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:31.199277 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:31.092796 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 118328us
I20260709 12:23:31.106313 31825 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
W20260709 12:23:31.200723 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 106799us
I20260709 12:23:30.897179  6059 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:31.100855 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:31.092056 31747 rpcz_store.cc:276] 0709 12:23:29.984774 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.195256 (+210482us) service_pool.cc:224] Handling call
0709 12:23:30.195590 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:30.585554 (+389964us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.041986 (+456432us) spinlock_profiling.cc:243] Waited 350 ms on lock 0x7b54001ee6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:31.042171 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":804992256}
I20260709 12:23:31.176684 32376 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:31.179566 31673 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
W20260709 12:23:31.207552 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 106796us
I20260709 12:23:30.983897 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:31.208410 31820 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:30.972810  6061 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:30.955552  5779 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:30.955440  6063 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:31.202943 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:31.214558 31751 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:31.215833 31824 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:31.223378 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:31.229969 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:31.216951 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:31.192613 31826 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:31.099537 32370 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:30.968436  5995 raft_consensus.cc:2764] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:23:31.173257 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:31.216549 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:31.220911 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:31.087083  6044 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:31.226529 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:31.271524 32373 raft_consensus.cc:2368] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:31.272071 32372 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 17 index: 11. (index mismatch)
I20260709 12:23:31.227063 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:31.234457 31826 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 11. (term mismatch)
I20260709 12:23:31.274680 31826 pending_rounds.cc:85] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 10
I20260709 12:23:31.275215 31826 pending_rounds.cc:107] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:23:31.275996 31826 raft_consensus.cc:2889] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:30.979988  5951 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:31.245018  5151 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.094327  5913 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:31.261410 31820 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:31.213806  5833 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:31.268754 31823 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:31.113797  5832 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.160s
W20260709 12:23:31.278143 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 104965us
I20260709 12:23:31.199545  5593 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:31.215052  6012 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:31.091818  5927 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:31.283703  5151 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.214304  6055 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:31.118281  5641 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.269214  5576 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:31.243136  5985 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.071s
I20260709 12:23:31.277750  5630 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.283229  5841 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:31.290084  6052 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:31.282935  5618 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:23:30.972697  5981 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.271s
I20260709 12:23:31.219242  5856 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:31.268886 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 66052.7us
I20260709 12:23:31.304348 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:31.116191  6062 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.306677 31744 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:31.227265  5473 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:31.304379 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:31.359392  5576 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:31.308959 32374 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:31.312668 31818 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:31.312736 32377 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:31.282735  5830 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:31.205700  5880 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.213161  5916 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.087s
I20260709 12:23:31.360170 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:31.366400 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:31.373291 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3026) took 1007 ms. Trace:
W20260709 12:23:31.297780  5944 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.699s	user 0.009s	sys 0.002s
I20260709 12:23:31.361155  5844 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:23:31.380555 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 134446us
W20260709 12:23:31.364864  6053 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:23:31.385601  5151 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:31.381326 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:31.384639 31677 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b416bf5304744b14b72501e697b8c951"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:23:31.419062 31677 raft_consensus.cc:606] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:31.419196  5985 raft_consensus.cc:2764] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
I20260709 12:23:31.419560 31677 raft_consensus.cc:614] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:31.390242 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 86116.8us
I20260709 12:23:31.398519  5506 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:31.423728  5695 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:31.397998  5473 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.425653 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:31.426769  5641 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:31.391222 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:31.406796  5830 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:31.427528  5830 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:31.427567 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:31.374838  5630 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.429059 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:31.429646 31753 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:23:31.429833 31678 raft_consensus.cc:2368] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:31.430132  5630 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:31.289274 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:31.431427  5921 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:31.378552  5841 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.433905  5506 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:31.435001  5506 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.434392  5576 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.437170  5576 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:31.438604 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1117) took 1974 ms. Trace:
I20260709 12:23:31.406208 31819 rpcz_store.cc:276] 0709 12:23:30.366076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.367132 (+  1056us) service_pool.cc:224] Handling call
0709 12:23:31.025986 (+658854us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:23:31.026170 (+   184us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.283164 (+256994us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.283204 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:31.283247 (+    43us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:31.283275 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.373014 (+ 89739us) spinlock_profiling.cc:243] Waited 257 ms on lock 0x7b540037ad54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:31.373065 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.373203 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":594839424}
I20260709 12:23:31.379578  5827 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.364121  6034 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:31.378284  6052 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:31.412669  5545 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:31.498929  5545 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.499406  5545 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:31.412492  5944 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.359090  5925 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.032s
I20260709 12:23:31.426703 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:31.426523  5473 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:31.363871  5805 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:31.408934 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:31.441010 31744 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:31.442032  5921 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.439152 32369 rpcz_store.cc:276] 0709 12:23:29.464434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:29.465240 (+   806us) service_pool.cc:224] Handling call
0709 12:23:29.465542 (+   302us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:29.477650 (+ 12108us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:23:29.500738 (+ 23088us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.336751 (+836013us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b18003ea7c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:23:30.440155 (+103404us) log.cc:844] Serialized 14 byte log entry
0709 12:23:30.715843 (+275688us) spinlock_profiling.cc:243] Waited 7.36 ms on lock 0x7b48002313d8. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:23:30.989484 (+273641us) spinlock_profiling.cc:243] Waited 2.53 ms on lock 0x7b18003ea7c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:31.023414 (+ 33930us) spinlock_profiling.cc:243] Waited 4.44 ms on lock 0x7b18003ea7c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:23:31.284831 (+261417us) log.cc:844] Serialized 14 byte log entry
0709 12:23:31.286342 (+  1511us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.361957 (+ 75615us) log.cc:844] Serialized 26 byte log entry
0709 12:23:31.362244 (+   287us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:31.362285 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:31.362364 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:31.436575 (+ 74211us) spinlock_profiling.cc:243] Waited 7.95 ms on lock 0x7b18003ea7c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:31.438300 (+  1725us) raft_consensus.cc:1685] finished
0709 12:23:31.438333 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:31.438521 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31263,"prepare.queue_time_us":24608,"prepare.run_cpu_time_us":131,"prepare.run_wall_time_us":3096,"raft.queue_time_us":262056,"raft.run_cpu_time_us":11135,"raft.run_wall_time_us":1258218,"spinlock_wait_cycles":56892544,"thread_start_us":347465,"threads_started":2,"wal-append.queue_time_us":1057193}
I20260709 12:23:31.390015  5856 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.468936 31677 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
W20260709 12:23:31.471391 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60365.6us
I20260709 12:23:31.478500 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:31.534128 31670 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:31.480594 31674 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
W20260709 12:23:31.482218  6066 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 12:23:31.540512 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:31.487756  6077 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.542541  6077 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.545032  5999 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:31.546200  5999 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:31.543813 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:31.546690  5833 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:31.541587  6044 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.549392  5833 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:23:31.491140  6004 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.853s	user 0.008s	sys 0.008s
I20260709 12:23:31.551021  6004 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:31.551849  6004 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:31.553668  6004 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:31.554198  6004 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:31.495086  5844 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:31.555547  5844 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:31.494728  5827 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.497563  5875 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.16, Last appended by leader: 16, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.502343 31678 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 12 index: 7. Preceding OpId from leader: term: 14 index: 8. (index mismatch)
I20260709 12:23:31.433945  5841 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:31.380940  5696 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:31.524278  5630 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:31.530507  6038 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 12:23:31.526571 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:31.479878 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:31.501998  6052 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:31.550073  5833 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:31.534966  5805 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:31.538415 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 129577us
I20260709 12:23:31.566915 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:31.567682 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:31.583612 31819 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:31.539235 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:31.570823 31671 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:31.584971 31675 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:31.558616 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3054) took 1285 ms. Trace:
I20260709 12:23:31.589663 31673 rpcz_store.cc:276] 0709 12:23:30.273557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:30.275065 (+  1508us) service_pool.cc:224] Handling call
0709 12:23:31.558386 (+1283321us) spinlock_profiling.cc:243] Waited 478 ms on lock 0x7b54002553d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:31.558535 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1100281216}
I20260709 12:23:31.490460  6012 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:31.562121 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:31.574209 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:31.580554 31642 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:31.547809 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:31.585863 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:31.609237 31791 leader_election.cc:400] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:31.609660 31791 leader_election.cc:403] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 12:23:31.613608 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:31.623934 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 16 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:31.626437 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 12 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:31.628754 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:31.630510 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:31.631628 31747 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:31.632454 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:31.634910 31748 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:31.635740 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:31.637241 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:31.638691 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 127544us
I20260709 12:23:31.641512  5151 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:31.642184  5151 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:31.642935 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 95215.5us
I20260709 12:23:31.645838 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:31.646929 31827 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:31.636160 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:31.662447 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49285.7us
I20260709 12:23:31.640188 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:31.666378 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:31.564847  6044 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:31.561825  5696 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.687032  5913 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:31.688252  6004 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.689049  5696 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:31.546099  6077 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:31.560838  5875 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:31.691160  6080 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.203s	user 0.004s	sys 0.000s
I20260709 12:23:31.691779 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:31.693104  5944 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:31.579540  5576 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:31.560423  5827 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:31.562609  5630 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:31.633834  5805 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:31.695081  5630 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:31.695449  5805 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:31.696702 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:31.696405  5630 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.698213  5630 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:31.563756  5841 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:31.691052  6045 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:31.702576  6045 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:31.703044  6045 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:31.703434 31753 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 11. (term mismatch)
I20260709 12:23:31.703848 31753 pending_rounds.cc:85] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 10
I20260709 12:23:31.704111 31753 pending_rounds.cc:107] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 16.11
W20260709 12:23:31.704322  5949 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.029s	user 0.009s	sys 0.002s
I20260709 12:23:31.705044  5949 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.700985  5576 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.704380 31753 raft_consensus.cc:2889] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:31.717626 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 108624us
I20260709 12:23:31.710398 31748 raft_consensus.cc:2368] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24.
I20260709 12:23:31.724449 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:31.725106 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.725289 31751 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:31.726737 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:23:31.727146 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60897.4us
I20260709 12:23:31.728848 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:31.729797 32374 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:31.711324  5151 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.566671  5566 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.607750  5916 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.546693  5999 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:31.611534  6012 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:31.693617  5850 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:31.635316  5641 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:31.514314  5981 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.200s
I20260709 12:23:31.569641  5833 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.567835  5905 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.188s
I20260709 12:23:31.537199  5921 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.690455  5696 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.062s
W20260709 12:23:31.799664 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:31.747304  5841 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:31.751842  5988 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:31.835224  6062 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:31.831575  5927 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:31.760836  6063 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.076s
I20260709 12:23:31.836834  5545 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.838527  5856 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:31.843086  5473 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:31.843314  5880 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:31.843683  5473 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:31.762199  5566 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.844661  5473 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.846257  5473 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:31.848663  5983 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:31.857175  5576 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:31.769946  5916 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.864530  5988 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:31.882133 31820 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.019s	sys 0.010s
W20260709 12:23:31.885054 31820 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.020s	sys 0.010s
I20260709 12:23:31.888228 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:31.913224 31673 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: [17.15-18.16]
I20260709 12:23:31.891077  5916 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:31.875669  6063 consensus_queue.cc:1243] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 19.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:31.893225  5641 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:31.894105  5921 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:31.896449  5949 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:31.870311  5999 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.939435  5999 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:31.898623  5830 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.944588  5830 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:31.881387  6093 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.055s	user 0.001s	sys 0.001s
I20260709 12:23:31.833817 31751 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:31.904534  5545 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:31.948273  6086 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.461s	user 0.000s	sys 0.004s
I20260709 12:23:31.907675  6045 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:31.907575  5844 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.914861 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:31.992489  5995 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.006s	sys 0.014s
I20260709 12:23:31.932394 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:31.932821 31675 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 12 index: 9. Preceding OpId from leader: term: 14 index: 10. (index mismatch)
I20260709 12:23:31.955369 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:31.956760 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:31.963461 31825 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.530s	user 0.001s	sys 0.038s
I20260709 12:23:31.872483  6012 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:31.928934  5696 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:31.922394  5593 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:31.996112 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.612s	user 0.006s	sys 0.015s
I20260709 12:23:32.017201 31672 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:32.047269 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.664s	user 0.007s	sys 0.015s
I20260709 12:23:32.047226  5593 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:32.047878 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 665893us
I20260709 12:23:31.924252 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:31.955171  6091 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:32.049026  6091 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:32.050726 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:32.051034  5870 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:23:32.035621 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 80389.7us
I20260709 12:23:32.051489  5870 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.041726 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:32.051885  5870 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:32.052202 31673 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:32.053421 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:32.054284 31673 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:32.042547 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:32.055536 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:32.056137 31675 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:32.057443 31826 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:23:32.051498  6091 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:32.036192 31825 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.003s	sys 0.040s
I20260709 12:23:32.060631 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:32.036921  5988 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.057449  5593 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.066159  6063 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:32.067248  6063 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20260709 12:23:32.068471  6052 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.501s	user 0.004s	sys 0.007s
I20260709 12:23:32.068943  6052 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:32.017815  6064 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:23:32.061165 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:32.049340 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 125226us
I20260709 12:23:32.076972 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:32.077276 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 881691us
I20260709 12:23:32.065287  6045 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:32.079653 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:32.083050 31792 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:32.084419 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:32.087235 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:32.088954 31672 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:32.090313 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:32.081633 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:32.093139 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:32.100091 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:31.956333  5641 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.103675  5641 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:32.078763  5830 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:32.105530 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:32.095144 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:32.091769 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.142485 32342 leader_election.cc:400] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21.
I20260709 12:23:32.143464 32342 leader_election.cc:403] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:23:32.101377 31792 leader_election.cc:400] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
W20260709 12:23:32.208943 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 107710us
W20260709 12:23:32.210067 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:32.040237  5995 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.007s	sys 0.014s
I20260709 12:23:32.091706 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:32.033699  5944 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.109258 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:32.003587  5858 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:32.119865 32373 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:32.333350  5875 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21.
I20260709 12:23:32.249209  5593 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:32.120105  5849 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.047183  5696 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.251418  5844 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:32.251910  6052 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:32.131568  6091 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:32.132436  5913 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:32.257797 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:32.120010  5981 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.139s
I20260709 12:23:32.353914  6106 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:32.095412  5792 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.172s
I20260709 12:23:32.096333  6059 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:32.096827  5921 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:32.440871  5921 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:23:32.441174  6096 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.302s	user 0.001s	sys 0.000s
I20260709 12:23:32.441453  5921 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:32.356407  5944 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:32.444573  5944 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:32.261842  5992 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:32.446880  5992 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:32.448242  5992 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:32.449800  5992 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:32.274256 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.033s	sys 0.040s
W20260709 12:23:32.450505 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.785s	user 0.033s	sys 0.040s
W20260709 12:23:32.450955 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 785770us
I20260709 12:23:32.095229  6101 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:32.262431  5630 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.547s	user 0.008s	sys 0.002s
W20260709 12:23:32.332197 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 189947us
I20260709 12:23:32.426250  6091 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.353283 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:32.366053 31818 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.659s	user 0.024s	sys 0.019s
I20260709 12:23:32.409801 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:32.460759 31818 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.755s	user 0.024s	sys 0.019s
I20260709 12:23:32.425285 32373 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:32.427644 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:32.258846  6092 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:32.110718  6063 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:23:32.419612  5779 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.002s	sys 0.017s
I20260709 12:23:32.470126 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:32.415889  5841 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:23:32.452329 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 194625us
I20260709 12:23:32.131963  5641 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:32.407595  5858 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.483894 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:32.472723 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:32.470672 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:32.468906 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.470145 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:32.485852 31674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.028s	sys 0.018s
W20260709 12:23:32.486243 31674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.028s	sys 0.018s
I20260709 12:23:32.475054  6063 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.435415 31675 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
W20260709 12:23:32.475539 31821 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.019s	sys 0.070s
I20260709 12:23:32.418648  5849 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:32.486661 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 478070us
I20260709 12:23:32.506371  6034 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:32.509785 31821 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.643s	user 0.020s	sys 0.071s
W20260709 12:23:32.473999  5779 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.002s	sys 0.018s
I20260709 12:23:32.510774 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3033) took 1295 ms. Trace:
W20260709 12:23:32.510666 31678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.584s	user 0.009s	sys 0.012s
I20260709 12:23:32.463871  6105 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:32.469247  6091 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:32.520169 31822 rpcz_store.cc:276] 0709 12:23:31.215074 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.215234 (+   160us) service_pool.cc:224] Handling call
0709 12:23:31.215547 (+   313us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.268674 (+ 53127us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.268704 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.282738 (+ 14034us) log.cc:844] Serialized 26 byte log entry
0709 12:23:31.590233 (+307495us) spinlock_profiling.cc:243] Waited 6.17 ms on lock 0x7b180041b780. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:23:31.997170 (+406937us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:32.476464 (+479294us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.487259 (+ 10795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.487336 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.487378 (+    42us) raft_consensus.cc:1685] finished
0709 12:23:32.510511 (+ 23133us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.510703 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":56840,"prepare.queue_time_us":265938,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":24,"raft.queue_time_us":83692,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":12711,"spinlock_wait_cycles":14190464,"thread_start_us":799282,"threads_started":2,"wal-append.queue_time_us":1153159}
I20260709 12:23:32.517179 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:32.520388 31678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.009s	sys 0.012s
I20260709 12:23:32.438889  6106 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:32.523865 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53320.7us
I20260709 12:23:32.527459 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:32.528072 31822 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:32.476073  5841 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:32.538094 31748 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:32.531613 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:32.533224 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:32.533499 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:32.535194 31750 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: [18.16-18.16]
I20260709 12:23:32.535617 31672 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:32.542352 31746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:32.543222 31746 raft_consensus.cc:606] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:32.523300  6110 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:32.545449  6110 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.511835  6063 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:32.524685  6034 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:32.475879  5545 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.054s
I20260709 12:23:32.472913  6045 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.054s
I20260709 12:23:32.514530  5916 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:32.514747  5630 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:32.473323  6092 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:32.522773  5833 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:32.509034  5641 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.463959  6101 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:32.551970  6059 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:32.529057  6065 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.052s
I20260709 12:23:32.516247  5995 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.049s
W20260709 12:23:32.581691  5870 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.529s	user 0.013s	sys 0.000s
I20260709 12:23:32.588294  6105 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:32.613091 31822 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:23:32.603192  5995 consensus_queue.cc:1243] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 15.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:32.623458 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:32.583571  5630 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.606504  5981 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:32.589394  5833 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:32.642066 32369 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.019s	sys 0.082s
I20260709 12:23:32.629036 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:32.635767  6104 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
I20260709 12:23:32.623131  5944 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:32.604969  5870 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:32.631910  5858 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:32.674121 31746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:32.675331 31746 raft_consensus.cc:606] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:32.682263 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:32.683563  5630 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:32.683696 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:32.685561 31678 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:32.685775  5833 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:32.686619 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:32.687343 31678 raft_consensus.cc:2376] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20. Re-sending same reply.
I20260709 12:23:32.688683 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:32.689277  6120 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:23:32.690017 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:32.690976  6004 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:32.691385  6004 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:32.691601 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:23:32.691779  6004 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:32.694168 31672 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:23:32.690289 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:32.636334  5696 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:32.601016  5841 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.643198  5545 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:32.701543 31825 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.012s	sys 0.016s
I20260709 12:23:32.596427  6059 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:32.690276  6113 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
W20260709 12:23:32.732239 31825 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.598s	user 0.013s	sys 0.017s
W20260709 12:23:32.688745  6115 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
I20260709 12:23:32.771112  5593 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:32.771739  5593 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.650688  5981 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.696708  6110 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:32.702855  5833 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:32.773783 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2936) took 1104 ms. Trace:
W20260709 12:23:32.767465  6112 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:23:32.766947  5841 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:32.790076  5593 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:32.779022  5696 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:32.800130  5151 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:32.800436  5695 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:32.764190  5473 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:32.780406 31818 rpcz_store.cc:276] 0709 12:23:31.669864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.670395 (+   531us) service_pool.cc:224] Handling call
0709 12:23:31.670783 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.705139 (+ 34356us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.705179 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.705823 (+   644us) log.cc:844] Serialized 26 byte log entry
0709 12:23:32.461206 (+755383us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:32.461480 (+   274us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.461669 (+   189us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.461742 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.773470 (+311728us) raft_consensus.cc:1685] finished
0709 12:23:32.773501 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.773694 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45515,"prepare.queue_time_us":508,"prepare.run_wall_time_us":52,"raft.queue_time_us":47846,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":34292,"spinlock_wait_cycles":10592512,"thread_start_us":755015,"threads_started":1,"wal-append.queue_time_us":859796}
I20260709 12:23:32.695616  5944 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:32.707011  5921 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.819182  5151 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.758644  5545 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.823302  5695 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:32.831064 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:32.835227  5981 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:32.835614  5695 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:32.841275  5999 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.901s	user 0.011s	sys 0.000s
I20260709 12:23:32.847213  6059 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:32.848708  5545 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:32.850346  5870 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:32.851406  5641 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:32.816108  5849 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:32.808001  5833 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:32.853353  5833 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.824293  5696 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.857558  5696 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:32.858351  5983 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.009s	user 0.011s	sys 0.003s
I20260709 12:23:32.787226  5630 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:32.860610 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 378989us
I20260709 12:23:32.757472 31678 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:32.867657 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:32.814235  6106 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:32.869217 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:32.872299 31818 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:32.870656  5151 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:32.870862 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 382261us
I20260709 12:23:32.873723 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:32.877241 31669 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:32.822861  5846 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:32.883937 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:32.891129 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:32.893357 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:32.861236  5985 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:32.898121 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 369226us
W20260709 12:23:32.930084 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 46251.4us
I20260709 12:23:32.939855 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:32.867457  5983 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:32.940793 31823 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
W20260709 12:23:32.942893 32377 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.006s	sys 0.011s
W20260709 12:23:32.946689 32377 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.007s	sys 0.011s
I20260709 12:23:32.939855 31818 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:32.943876  5985 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.952912 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:32.954115 31824 raft_consensus.cc:2410] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
W20260709 12:23:32.955818 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.942341  6101 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:32.956815 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 952) took 1310 ms. Trace:
I20260709 12:23:32.952117 31674 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 14. Preceding OpId from leader: term: 19 index: 16. (index mismatch)
I20260709 12:23:32.960522 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:32.927583  6092 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:32.870551  5630 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:32.965523 31823 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:32.970203 31674 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:32.956230 31678 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:32.946959  5779 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:32.945171  6106 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:32.965158 32369 rpcz_store.cc:276] 0709 12:23:31.645367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.645707 (+   340us) service_pool.cc:224] Handling call
0709 12:23:31.646213 (+   506us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:31.646792 (+   579us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.646835 (+    43us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:31.721209 (+ 74374us) log.cc:844] Serialized 78 byte log entry
0709 12:23:32.095270 (+374061us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:32.651003 (+555733us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.955090 (+304087us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.955190 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:32.955248 (+    58us) raft_consensus.cc:1685] finished
0709 12:23:32.955266 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:32.955468 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":483743,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":815,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":10558,"thread_start_us":1092189,"threads_started":3,"wal-append.queue_time_us":688162}
I20260709 12:23:32.953542  6059 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:32.974848 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:32.973326 31678 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:23:32.977607  5981 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:33.006346  5981 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:32.948837  5995 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.024s
I20260709 12:23:32.954453  5858 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 12:23:32.964257  5849 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.050s
W20260709 12:23:33.013572  5992 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.551s	user 0.011s	sys 0.000s
I20260709 12:23:33.014250  5992 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:33.017284 31678 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:32.951460  5983 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:32.929072  5846 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.020987 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:32.953882  6109 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 18.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 12:23:33.024669  5981 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:33.024425  6063 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.034538 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:33.036350 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.074200 31818 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:32.975865  5779 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:33.106140 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 222496us
I20260709 12:23:33.110440 31822 raft_consensus.cc:2368] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
W20260709 12:23:33.113417 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.113247 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:33.015589  6059 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.140604  6059 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:33.141321  6059 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:33.114730  6092 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.114118 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:33.142792  6092 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.118976  6061 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:33.144694  6061 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.145388  6092 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:33.146826 31823 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:33.117049  6106 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:33.147888  6106 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:33.148769  6106 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.150085  6106 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:33.152305  6111 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.675s	user 0.002s	sys 0.000s
I20260709 12:23:33.154592 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2937) took 1484 ms. Trace:
I20260709 12:23:33.154685  5618 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:33.155297  5618 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:33.155668  6138 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:33.155890  5618 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:33.156317  6138 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.160683 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3074) took 1246 ms. Trace:
I20260709 12:23:33.161656  6138 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:33.161187 31671 rpcz_store.cc:276] 0709 12:23:31.913982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.510881 (+596899us) service_pool.cc:224] Handling call
0709 12:23:32.511272 (+   391us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:32.524446 (+ 13174us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:32.524477 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:32.525027 (+   550us) log.cc:844] Serialized 26 byte log entry
0709 12:23:32.945253 (+420226us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:32.945293 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.945360 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:33.160421 (+215061us) raft_consensus.cc:1685] finished
0709 12:23:33.160444 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.160612 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12799,"raft.queue_time_us":86772,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":11053,"thread_start_us":419838,"threads_started":1,"wal-append.queue_time_us":633601}
I20260709 12:23:33.121549  5995 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:33.167057  5995 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 17 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:33.168288  5995 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.122020  5849 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:33.170316  5849 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.173273  5849 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:33.026379  5992 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:33.175333  6034 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.596s	user 0.010s	sys 0.002s
I20260709 12:23:33.175928  6034 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.177160  5992 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:33.178517  6061 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:33.149935 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3087) took 1052 ms. Trace:
I20260709 12:23:33.183017 31672 rpcz_store.cc:276] 0709 12:23:32.097126 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.352938 (+255812us) service_pool.cc:224] Handling call
0709 12:23:33.149657 (+796719us) spinlock_profiling.cc:243] Waited 74.8 ms on lock 0x7b5400229f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:33.149847 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":172024576}
W20260709 12:23:33.187563  6139 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:23:33.189661  6063 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:33.190907 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:33.123751  5913 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.689s	user 0.013s	sys 0.000s
W20260709 12:23:33.123315 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.239475 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:33.240517 31827 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:33.125034  5630 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:33.127051  6143 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.127777  6145 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:33.073830  6131 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:33.130041  6091 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.583s	user 0.009s	sys 0.000s
I20260709 12:23:33.131095 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:33.131441  5983 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:33.132122  5944 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:33.132673  5921 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:33.109549  5593 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:33.136897  5846 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:33.142297 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:33.155161 31819 rpcz_store.cc:276] 0709 12:23:31.670232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.671092 (+   860us) service_pool.cc:224] Handling call
0709 12:23:31.671429 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.719975 (+ 48546us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.720026 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.720722 (+   696us) log.cc:844] Serialized 26 byte log entry
0709 12:23:32.097002 (+376280us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:32.097050 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.097124 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:33.151692 (+1054568us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:33.154252 (+  2560us) raft_consensus.cc:1685] finished
0709 12:23:33.154275 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.154525 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":48096,"raft.queue_time_us":73044,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":8875,"spinlock_wait_cycles":54194048,"thread_start_us":301786,"threads_started":1,"wal-append.queue_time_us":654004}
I20260709 12:23:33.120646  5779 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:33.293802 31576 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 237) took 69670 ms (client timeout 15000 ms). Trace:
W20260709 12:23:33.294291 31576 rpcz_store.cc:269] 0709 12:22:23.623350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:23.628734 (+  5384us) service_pool.cc:224] Handling call
0709 12:23:33.293652 (+69664918us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:25.125258 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:25.192850 (+ 67592us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:25.192924 (+    74us) write_op.cc:435] Acquired schema lock
0709 12:23:25.192953 (+    29us) tablet.cc:662] Decoding operations
0709 12:23:25.194146 (+  1193us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:25.194279 (+   133us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:25.194323 (+    44us) tablet.cc:685] Acquiring locks for 6 operations
0709 12:23:25.306392 (+112069us) tablet.cc:701] Row locks acquired
0709 12:23:25.306451 (+    59us) write_op.cc:260] PREPARE: finished
0709 12:23:25.311762 (+  5311us) write_op.cc:270] Start()
0709 12:23:25.337150 (+ 25388us) write_op.cc:276] Timestamp: P: 1783599805306567 usec, L: 0
0709 12:23:25.342283 (+  5133us) op_driver.cc:348] REPLICATION: starting
0709 12:23:25.420669 (+ 78386us) log.cc:844] Serialized 2466 byte log entry
0709 12:23:26.736978 (+1316309us) op_driver.cc:464] REPLICATION: finished
0709 12:23:27.078015 (+341037us) write_op.cc:301] APPLY: starting
0709 12:23:27.091323 (+ 13308us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:27.247121 (+155798us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:27.281086 (+ 33965us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:27.327161 (+ 46075us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:27.375070 (+ 47909us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=12,key_file_lookups=12,delta_file_lookups=0,mrs_lookups=0
0709 12:23:27.392821 (+ 17751us) write_op.cc:312] APPLY: finished
0709 12:23:27.406254 (+ 13433us) log.cc:844] Serialized 65 byte log entry
0709 12:23:27.406502 (+   248us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:27.406622 (+   120us) write_op.cc:454] Released schema lock
0709 12:23:27.406712 (+    90us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:27.410285 (+  3573us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:27.410335 (+    50us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":145934,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":294564,"apply.run_cpu_time_us":13103,"apply.run_wall_time_us":340583,"cfile_cache_hit":26,"cfile_cache_hit_bytes":29195,"num_ops":6,"prepare.queue_time_us":489142,"prepare.run_cpu_time_us":21919,"prepare.run_wall_time_us":496698,"raft.queue_time_us":491062,"raft.run_cpu_time_us":13321,"raft.run_wall_time_us":152682,"replication_time_us":1385403,"thread_start_us":771340,"threads_started":4,"wal-append.queue_time_us":166017,"wal-append.run_cpu_time_us":22192,"wal-append.run_wall_time_us":2855645}]]}
I20260709 12:23:33.291323  5593 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.301430  5593 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:33.303656 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:33.307471 31811 tablet_service.cc:1558] Processing DeleteTablet for tablet f6404cb9c2174326b06cbbb0d27d43f2 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:56358
W20260709 12:23:33.308393  6117 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.745s	user 0.004s	sys 0.000s
I20260709 12:23:33.244843  6130 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.309980  6130 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.310897  5566 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:33.312418  6130 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:33.239112  5841 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:33.312875  5618 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.313302  5841 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.291132 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:33.211000 31791 leader_election.cc:400] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24.
I20260709 12:23:33.215762 31672 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [17.10-18.11]
I20260709 12:23:33.217213 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:33.319320 31676 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:33.218678 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:33.243775 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:33.244493 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.320761 31752 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
W20260709 12:23:33.320863 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 77159.7us
I20260709 12:23:33.321645 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:33.321707 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.244541 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.323284 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.246313 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.247637 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.324433 31750 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:33.325165 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.251345 31747 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:33.329090 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.329360 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.252276 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:33.244210 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:33.263178 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:33.330261 31752 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:33.264720 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.331198 31827 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
W20260709 12:23:33.331552 32341 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.014s	sys 0.031s
I20260709 12:23:33.266477 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.254312  6110 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.190837  6034 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:33.325289 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.248921 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.325842 31791 leader_election.cc:403] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Cancelling election due to peer responding with higher term
I20260709 12:23:33.327466 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:33.266695 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:33.331908 32341 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.605s	user 0.014s	sys 0.031s
I20260709 12:23:33.332108 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:33.333159 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 89041.3us
I20260709 12:23:33.345525 31823 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:33.346943 31819 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:33.255295  5870 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.252475  5913 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:23:33.265965  6004 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.574s	user 0.012s	sys 0.000s
I20260709 12:23:33.267581  5695 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.265188  6142 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:33.368108 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1137) took 1317 ms. Trace:
I20260709 12:23:33.407171 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:33.407981 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 196367us
I20260709 12:23:33.263904  6092 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.409833 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.252660  6063 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.157s
I20260709 12:23:33.307528  5992 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:33.350216  5793 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:33.327466  5593 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.413630 32375 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:33.457892 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:33.418300 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:33.458894  6144 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:23:33.459013 31746 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:33.418143  5779 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election lost for term 23. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24.
I20260709 12:23:33.418313  6061 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:33.459980 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.460110  6061 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.460620 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:33.461107 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:33.461404 31744 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:33.265168  6143 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.408344 31827 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:33.331044  6091 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.441702 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:33.462234 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.470981 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.437821 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:33.443274 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:33.390950 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.347944  5846 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:33.494364  5846 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:33.445205 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.422659 32377 rpcz_store.cc:276] 0709 12:23:32.050671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.050830 (+   159us) service_pool.cc:224] Handling call
0709 12:23:32.051124 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.051323 (+   199us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:23:32.410964 (+359641us) log.cc:844] Serialized 14 byte log entry
0709 12:23:33.367661 (+956697us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:33.367724 (+    63us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:33.367772 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.367862 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.368022 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":483609,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":40,"raft.queue_time_us":208040,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":126,"thread_start_us":1301162,"threads_started":3,"wal-append.queue_time_us":666724}
I20260709 12:23:33.350822  6138 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:33.495399 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:33.444729 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:33.496218 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.496837 32378 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:33.498703 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:33.472683 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.518584 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.475780 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:33.520411 31827 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:33.521317 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:33.522368 32376 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:33.523162 31751 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:33.524331 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:33.510032 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 72282.9us
I20260709 12:23:33.526713 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:33.520123 31749 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:33.528190 32371 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:23:33.531297 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.532187 32375 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:23:33.417775  5870 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.464619  5618 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:33.548750 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.267904  6131 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.404827  6110 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.542063 32374 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: [19.14-19.14]
I20260709 12:23:33.561939 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2235) took 1602 ms. Trace:
I20260709 12:23:33.563480  6110 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:33.562486 31673 rpcz_store.cc:276] 0709 12:23:31.959877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.529944 (+570067us) service_pool.cc:224] Handling call
0709 12:23:32.530292 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:32.961229 (+430937us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:32.962452 (+  1223us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:32.966403 (+  3951us) log.cc:844] Serialized 26 byte log entry
0709 12:23:33.327264 (+360861us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:33.327302 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.327370 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:33.561696 (+234326us) raft_consensus.cc:1685] finished
0709 12:23:33.561729 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.561879 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":71735,"spinlock_wait_cycles":10540416,"thread_start_us":727916,"threads_started":2,"wal-append.queue_time_us":545014}
I20260709 12:23:33.564878 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:33.567306 31744 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:33.577085 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:33.455046  5841 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:33.437179  6004 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.470937  5992 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:33.549727 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:33.541922  6143 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:33.547372 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:33.535423  5833 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.510s	user 0.015s	sys 0.039s
W20260709 12:23:33.539798 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 505376us
I20260709 12:23:33.454751  6092 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.440585  5695 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.462316  6061 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:33.608625 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 2942) took 1489 ms. Trace:
I20260709 12:23:33.526327  5846 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:33.560726 32375 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 16. Preceding OpId from leader: term: 19 index: 16. (term mismatch)
I20260709 12:23:33.561098  6091 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:33.574163 31753 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:33.565949 31750 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:33.540551  6063 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:33.465878  5593 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:33.575052 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 461887us
W20260709 12:23:33.607530  5833 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.017s	sys 0.040s
I20260709 12:23:33.573403  5983 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:33.555605  5779 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.066s
I20260709 12:23:33.535084 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.440191  6142 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.547412  5913 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:33.546057  6135 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.640221  6135 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:33.649580  6103 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.217s	user 0.000s	sys 0.002s
I20260709 12:23:33.654845  5695 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:33.636005  5841 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:33.656437 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3052) took 1445 ms (client timeout 1676 ms). Trace:
I20260709 12:23:33.656842  5841 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.659176  6063 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:33.660619  6063 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.604574 31819 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:33.609218 31818 rpcz_store.cc:276] 0709 12:23:32.119552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.952597 (+833045us) service_pool.cc:224] Handling call
0709 12:23:33.608367 (+655770us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54003c8cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:33.608544 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":275592064}
I20260709 12:23:33.609766 31746 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:33.555877  6138 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.612214 32373 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:33.617329 32375 pending_rounds.cc:85] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 15
W20260709 12:23:33.701009 31824 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3036) took 2485 ms (client timeout 2780 ms). Trace:
W20260709 12:23:33.657132 31822 rpcz_store.cc:269] 0709 12:23:32.211470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.628714 (+417244us) service_pool.cc:224] Handling call
0709 12:23:33.656094 (+1027380us) spinlock_profiling.cc:243] Waited 450 ms on lock 0x7b54000fc654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:33.656334 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1036076800}
I20260709 12:23:33.737555 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.739617 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.740156 31819 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:33.740850 32375 pending_rounds.cc:107] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 18.16
I20260709 12:23:33.741374 32375 raft_consensus.cc:2889] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:33.741714 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:33.665215 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 950) took 1134 ms. Trace:
I20260709 12:23:33.742990 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.743464 31751 rpcz_store.cc:276] 0709 12:23:32.530983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.531322 (+   339us) service_pool.cc:224] Handling call
0709 12:23:33.664872 (+1133550us) spinlock_profiling.cc:243] Waited 984 ms on lock 0x7b54000b90d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:33.665114 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2262597376}
I20260709 12:23:33.698282 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3040) took 2405 ms. Trace:
I20260709 12:23:33.745155 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.621969 31827 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:33.625841 31748 raft_consensus.cc:2368] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20.
I20260709 12:23:33.564989  6145 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:33.570956 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.748204  6135 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:33.737634 31824 rpcz_store.cc:269] 0709 12:23:31.215977 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.229614 (+ 13637us) service_pool.cc:224] Handling call
0709 12:23:33.700618 (+2471004us) spinlock_profiling.cc:243] Waited 1.72 s on lock 0x7b5400379454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:33.700853 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3962223488}
W20260709 12:23:33.632869 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.750219 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.751137 31824 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:33.753041 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.753135  5913 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:33.754004  5913 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.754657 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.757244  5913 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:33.758049  5913 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.759115 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.637666 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:33.582849  5630 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:33.761937 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:33.762125 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.745038 31820 rpcz_store.cc:276] 0709 12:23:31.292689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.293032 (+   343us) service_pool.cc:224] Handling call
0709 12:23:31.293587 (+   555us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:31.308348 (+ 14761us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:31.916621 (+608273us) log.cc:844] Serialized 14 byte log entry
0709 12:23:31.917700 (+  1079us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:31.918003 (+   303us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:31.918061 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.696009 (+1777948us) spinlock_profiling.cc:243] Waited 3.9 ms on lock 0x7b180045a600. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419
0709 12:23:33.697826 (+  1817us) spinlock_profiling.cc:243] Waited 14.5 ms on lock 0x7b540043d5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:33.697905 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.698134 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":954366,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":33,"raft.queue_time_us":407,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":33814,"spinlock_wait_cycles":42255360,"thread_start_us":545616,"threads_started":1}
I20260709 12:23:33.766722 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:33.769428 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 312937us
I20260709 12:23:33.770185 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:33.761129 31824 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:33.773356 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3034) took 2558 ms. Trace:
I20260709 12:23:33.773829 31818 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 17 index: 12. (index mismatch)
I20260709 12:23:33.774544 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2940) took 1663 ms. Trace:
I20260709 12:23:33.744199 31819 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:23:33.775266 31818 rpcz_store.cc:276] 0709 12:23:32.111431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.773084 (+1661653us) service_pool.cc:224] Handling call
0709 12:23:33.773473 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.774280 (+   807us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.774488 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":23296}
I20260709 12:23:33.636274  5921 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:33.782416 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2943) took 1652 ms. Trace:
I20260709 12:23:33.671422  5841 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:33.720999  6012 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:33.789618 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:33.680593  6131 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:33.661324  5985 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:33.775148  6065 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:33.769289  6152 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:33.791710  6148 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.116s	user 0.001s	sys 0.000s
I20260709 12:23:33.639297  5983 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.775341  6141 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:33.796386  6138 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.820398  5545 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:33.820868  5545 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:33.821766  6138 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:33.777772  5913 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.788908  5921 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.803648  6065 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:33.776458  6145 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.816901  6141 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:33.807924  6152 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:33.861063  6145 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:33.856300  6141 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:33.624737 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:33.915684 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:33.582901 31673 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 19: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:33.608104 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:33.915684 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2241) took 1775 ms. Trace:
W20260709 12:23:33.920477 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 312374us
I20260709 12:23:33.919703 31674 rpcz_store.cc:276] 0709 12:23:32.140551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.975746 (+835195us) service_pool.cc:224] Handling call
0709 12:23:32.976249 (+   503us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:33.179396 (+203147us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:33.179434 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:33.180048 (+   614us) log.cc:844] Serialized 26 byte log entry
0709 12:23:33.609431 (+429383us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:33.609494 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.609588 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:33.914904 (+305316us) raft_consensus.cc:1685] finished
0709 12:23:33.914983 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.915402 (+   419us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72325,"raft.queue_time_us":66064,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":43633,"thread_start_us":556212,"threads_started":2,"wal-append.queue_time_us":676725}
W20260709 12:23:33.923908 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.515s	user 0.001s	sys 0.018s
W20260709 12:23:33.924453 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.001s	sys 0.018s
W20260709 12:23:33.924367 31669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1033) took 2036 ms (client timeout 2030 ms). Trace:
W20260709 12:23:33.924926 31669 rpcz_store.cc:269] 0709 12:23:31.887722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.887891 (+   169us) service_pool.cc:224] Handling call
0709 12:23:33.923895 (+2036004us) spinlock_profiling.cc:243] Waited 930 ms on lock 0x7b54001b2bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:33.924137 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2140101888}
I20260709 12:23:33.930199 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:33.932794 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:33.935123 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:33.936148 31719 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.007s	sys 0.012s
I20260709 12:23:33.972503 31822 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:23:33.623027 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:33.998876 31749 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:23:33.999919 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:34.000278 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 65637.8us
I20260709 12:23:34.001104 32377 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:34.001492 31749 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [18.11-18.11]
I20260709 12:23:34.002888  6110 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:34.007511 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2941) took 1895 ms. Trace:
I20260709 12:23:34.007840  5921 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:34.008222 31818 rpcz_store.cc:276] 0709 12:23:32.111701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.776223 (+1664522us) service_pool.cc:224] Handling call
0709 12:23:33.776566 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.005252 (+228686us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.005327 (+    75us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:34.005431 (+   104us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:34.005495 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.007072 (+  1577us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.007279 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.011634 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:34.012960  6163 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:23:34.014645 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:33.774134 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1137) took 1700 ms. Trace:
I20260709 12:23:33.781778 31827 raft_consensus.cc:2410] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 20 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:23:33.773792 31821 rpcz_store.cc:276] 0709 12:23:31.215384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.216195 (+   811us) service_pool.cc:224] Handling call
0709 12:23:31.216505 (+   310us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:31.363529 (+147024us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.363564 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:31.364034 (+   470us) log.cc:844] Serialized 26 byte log entry
0709 12:23:31.794821 (+430787us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:32.520100 (+725279us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.520377 (+   277us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.759648 (+1239271us) spinlock_profiling.cc:243] Waited 144 ms on lock 0x7b54002080d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:33.759723 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:33.759792 (+    69us) raft_consensus.cc:1685] finished
0709 12:23:33.773138 (+ 13346us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.773297 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2100,"prepare.queue_time_us":468052,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":4795,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":10528,"spinlock_wait_cycles":332322560,"thread_start_us":1077410,"threads_started":2,"wal-append.queue_time_us":561444,"wal-append.run_cpu_time_us":2911,"wal-append.run_wall_time_us":1664656}
I20260709 12:23:33.783095 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:34.048022 32374 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:34.048143 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.794580  5992 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.049916 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:34.050751 31820 rpcz_store.cc:276] 0709 12:23:32.073550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.772813 (+1699263us) service_pool.cc:224] Handling call
0709 12:23:33.773449 (+   636us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.773693 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:33.773734 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:33.773808 (+    74us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:33.773857 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.773934 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.774063 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2432}
I20260709 12:23:34.052515 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:34.053447 31820 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:33.849117  6065 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:34.057402  6045 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.436s	user 0.008s	sys 0.006s
W20260709 12:23:34.058065  6045 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.468s	user 0.009s	sys 0.006s
I20260709 12:23:34.057403 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:34.069146 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:34.075172 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:34.077272 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:33.969631  5913 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:33.969470  5983 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:33.941579 32369 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.633s	user 0.025s	sys 0.068s
I20260709 12:23:33.972504 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:33.973179 31562 catalog_manager.cc:5028] TS 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): tablet f6404cb9c2174326b06cbbb0d27d43f2 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
I20260709 12:23:33.974841 31819 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:33.976666 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3043) took 2625 ms. Trace:
W20260709 12:23:33.976992  6052 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.415s	user 0.003s	sys 0.006s
I20260709 12:23:33.781566  5630 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.978372 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:33.978399 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:33.980120  5593 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:33.984050  5887 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:33.943482 31824 rpcz_store.cc:276] 0709 12:23:32.130568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.777271 (+1646703us) service_pool.cc:224] Handling call
0709 12:23:33.779161 (+  1890us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.779422 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:33.779461 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:33.779524 (+    63us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:33.779570 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:33.779644 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.782322 (+  2678us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:33.981964 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:33.931481 31676 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:33.636937 31745 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
W20260709 12:23:33.934320 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.105562 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:33.995661 31719 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.577s	user 0.007s	sys 0.012s
I20260709 12:23:34.014123 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:34.010653 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:34.115115  6135 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:23:34.116362 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 102470us
I20260709 12:23:34.117475  6136 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:34.118065  6136 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:34.075335  5870 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.119692  5870 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:34.119791  6136 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:33.821696  5545 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 18.15, Last appended by leader: 15, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.122043 32375 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:34.078296 31827 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:34.124243  6136 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:34.125069 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3055) took 1249 ms. Trace:
I20260709 12:23:34.126789  6166 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:34.127712  6166 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.129537 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.070567 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:34.097949 31562 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 125021us
I20260709 12:23:34.131742 31820 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:34.100011  6168 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:34.094303  5887 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:34.131179 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2944) took 2000 ms. Trace:
W20260709 12:23:34.135078 32369 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.827s	user 0.027s	sys 0.070s
I20260709 12:23:34.125517 31827 rpcz_store.cc:276] 0709 12:23:32.876154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.123525 (+1247371us) service_pool.cc:224] Handling call
0709 12:23:34.124108 (+   583us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.124601 (+   493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.124635 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:34.124705 (+    70us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:34.124743 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.124824 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.124973 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.135617 31819 rpcz_store.cc:276] 0709 12:23:32.130972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.781136 (+1650164us) service_pool.cc:224] Handling call
0709 12:23:33.781712 (+   576us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.130774 (+349062us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.131087 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:33.841024  5793 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:34.138003 31672 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:34.119563 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:34.125949  6151 consensus_queue.cc:1175] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5206fd0087404d2da964a5d985a04ff4" responder_term: 18 status { last_received { term: 18 index: 15 } last_committed_idx: 15 last_received_current_leader { term: 18 index: 15 } }
W20260709 12:23:34.144625  6052 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.596s	user 0.004s	sys 0.006s
I20260709 12:23:34.130561  6166 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.137804  5887 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:34.159525 31716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.548s	user 0.029s	sys 0.024s
I20260709 12:23:34.166736  5995 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:34.173365  5846 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.544s	user 0.008s	sys 0.010s
I20260709 12:23:34.084821 31825 rpcz_store.cc:276] 0709 12:23:31.349245 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:31.431519 (+ 82274us) service_pool.cc:224] Handling call
0709 12:23:31.431895 (+   376us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:31.432283 (+   388us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:31.432315 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:31.432943 (+   628us) log.cc:844] Serialized 52 byte log entry
0709 12:23:32.059219 (+626276us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:32.755570 (+696351us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.759099 (+  3529us) log.cc:844] Serialized 14 byte log entry
0709 12:23:32.759380 (+   281us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:32.759489 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:33.648686 (+889197us) spinlock_profiling.cc:243] Waited 12.5 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:33.662394 (+ 13708us) spinlock_profiling.cc:243] Waited 2.41 ms on lock 0x7b1800391a20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:33.944962 (+282568us) raft_consensus.cc:1685] finished
0709 12:23:33.945035 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:33.974363 (+ 29328us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23979,"prepare.queue_time_us":277414,"prepare.run_cpu_time_us":115,"prepare.run_wall_time_us":119,"raft.queue_time_us":13141,"raft.run_cpu_time_us":422,"raft.run_wall_time_us":46718,"spinlock_wait_cycles":34396800,"thread_start_us":1247123,"threads_started":2,"wal-append.queue_time_us":905482}
I20260709 12:23:34.158910 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:34.178802  6092 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.181275  5880 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:23:34.183965 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 406827us
I20260709 12:23:34.186780 31820 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: [16.11-18.12]
W20260709 12:23:34.202421 31716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.030s	sys 0.024s
I20260709 12:23:34.186240 31567 catalog_manager.cc:5697] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 10 to 16, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 16 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:34.199421  5545 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.178763  5981 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.214154 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3057) took 1326 ms. Trace:
I20260709 12:23:34.214867 31820 rpcz_store.cc:276] 0709 12:23:32.884409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.147092 (+1262683us) service_pool.cc:224] Handling call
0709 12:23:34.186205 (+ 39113us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:34.210093 (+ 23888us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.210419 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.216681 31821 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
W20260709 12:23:34.223883 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 88207.9us
I20260709 12:23:34.231029 32375 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 15.
I20260709 12:23:34.238888  6091 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:34.239542 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:34.240039  6091 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:34.242848 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 72696.2us
I20260709 12:23:34.243374 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:34.244092 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:34.245584 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:34.247326 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:34.250113  6012 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:34.250787  6152 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:34.250948  6092 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:34.257690  5641 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:34.236068 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.209539  5846 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.257884  6092 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:34.266520 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.270782  6092 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:34.271997 31567 catalog_manager.cc:5697] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 5 to 12, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 12 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:34.274969  5759 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:34.275445  6142 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.263419 31676 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:34.262492 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:34.278498 31823 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3053) took 2066 ms (client timeout 2355 ms). Trace:
W20260709 12:23:34.279881 31823 rpcz_store.cc:269] 0709 12:23:32.211775 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.977993 (+766218us) service_pool.cc:224] Handling call
0709 12:23:34.278055 (+1300062us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b5400207bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:34.278292 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":275085696}
I20260709 12:23:34.282045  6065 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.283679 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:34.284473 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:34.264750 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:34.266095  6091 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.264602  5880 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.287453  5630 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:34.287667  5880 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:34.323315 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.275594  5759 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.324489 31674 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:34.324790  5759 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:34.324385  5793 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.326117 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.344324 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.290563  5870 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:34.291061  5695 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:34.343175 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:34.347028 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3001) took 1096 ms. Trace:
I20260709 12:23:34.347549 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:34.348929  5630 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:34.349924  5630 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:34.349722  5759 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.351346  5630 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.291219  6091 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:34.292021  5949 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:34.292855  6140 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:34.295533 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3076) took 2375 ms. Trace:
I20260709 12:23:34.355005  6140 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:34.355402  5844 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:34.355083 31675 rpcz_store.cc:276] 0709 12:23:31.999826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.514503 (+514677us) service_pool.cc:224] Handling call
0709 12:23:32.514888 (+   385us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:32.563481 (+ 48593us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:32.563511 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:32.563552 (+    41us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:32.563587 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.290069 (+1726482us) spinlock_profiling.cc:243] Waited 48.4 ms on lock 0x7b5400391cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:34.290131 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.290425 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":111248640}
I20260709 12:23:34.356243  6140 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.357385 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.357841  6140 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:34.355159  5949 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:34.359174  5949 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:34.360966  5759 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:34.365317 31673 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:34.346426 31744 raft_consensus.cc:2368] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20.
I20260709 12:23:34.367226 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:34.347566 31747 rpcz_store.cc:276] 0709 12:23:33.250587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.358860 (+108273us) service_pool.cc:224] Handling call
0709 12:23:33.359437 (+   577us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:33.629798 (+270361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:33.630331 (+   533us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:33.974761 (+344430us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x7b2c00001158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:23:33.975936 (+  1175us) log.cc:844] Serialized 26 byte log entry
0709 12:23:34.034413 (+ 58477us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:34.200390 (+165977us) log.cc:844] Serialized 14 byte log entry
0709 12:23:34.227543 (+ 27153us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.235449 (+  7906us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:34.235511 (+    62us) raft_consensus.cc:1685] finished
0709 12:23:34.255883 (+ 20372us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.346803 (+ 90920us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22739,"prepare.queue_time_us":118242,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":41,"raft.queue_time_us":206229,"raft.run_cpu_time_us":451,"raft.run_wall_time_us":41286,"spinlock_wait_cycles":3022720,"thread_start_us":438171,"threads_started":3,"wal-append.queue_time_us":134777}
I20260709 12:23:34.367825 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2254) took 1064 ms. Trace:
I20260709 12:23:34.368320 31673 rpcz_store.cc:276] 0709 12:23:33.302976 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.932139 (+629163us) service_pool.cc:224] Handling call
0709 12:23:34.367627 (+435488us) spinlock_profiling.cc:243] Waited 354 ms on lock 0x7b5400181ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:34.367755 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":813909376}
I20260709 12:23:34.368808 31821 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:34.298115 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.369554 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2950) took 1025 ms. Trace:
I20260709 12:23:34.370075 31676 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:34.371394 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:34.373152 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 12 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.299134 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:34.297015 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:34.299633 31753 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: [17.11-17.11]
I20260709 12:23:34.301457 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:34.301976 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:34.216665  5981 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.378512 31678 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:23:34.378696 31751 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:34.380460 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.381434 31673 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:34.381790  5981 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:34.382622 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.382835 31826 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
W20260709 12:23:34.384450 31826 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3049) took 2307 ms (client timeout 1928 ms). Trace:
I20260709 12:23:34.384537 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:34.384873 31826 rpcz_store.cc:269] 0709 12:23:32.076666 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:32.105270 (+ 28604us) service_pool.cc:224] Handling call
0709 12:23:34.384128 (+2278858us) spinlock_profiling.cc:243] Waited 2.27 s on lock 0x7b54000fbc54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:34.384346 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5231982848}
I20260709 12:23:34.385430 31673 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:34.288784  6065 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.303385 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:34.389006 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:34.389530 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 86235.8us
I20260709 12:23:34.390671 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:34.391769  5844 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:34.392462 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:34.282179 31823 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 12. (term mismatch)
I20260709 12:23:34.393007  5844 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:34.393347 31823 pending_rounds.cc:85] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 11
I20260709 12:23:34.393726 31823 pending_rounds.cc:107] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260709 12:23:34.304742 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:34.361109 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:34.372884  6141 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.510s	user 0.005s	sys 0.012s
I20260709 12:23:34.305846 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2946) took 2050 ms. Trace:
I20260709 12:23:34.370098 31821 rpcz_store.cc:276] 0709 12:23:33.343718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.229491 (+885773us) service_pool.cc:224] Handling call
0709 12:23:34.368531 (+139040us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.369279 (+   748us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.369485 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.309669 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.261286 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 934) took 1016 ms. Trace:
I20260709 12:23:34.315719  6169 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:34.316668  5846 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.317443 31752 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [21.12-21.12]
W20260709 12:23:34.317587 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 198487us
I20260709 12:23:34.258414  5641 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:34.319764  6004 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:34.322760 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 86573.2us
I20260709 12:23:34.329253 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.335906 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.336374  6092 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.338945  5696 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.340615  6105 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:34.341315  6151 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.342083  6168 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:34.296335 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.343123 31567 catalog_manager.cc:5697] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 15 to 17, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 17 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:34.289809 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:34.425993 31826 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
I20260709 12:23:34.360595  5949 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.296515 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:34.363198 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:34.365212 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:34.322973 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 69177us
I20260709 12:23:34.299445 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 2993) took 1053 ms. Trace:
I20260709 12:23:34.376111 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.245608  5832 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:34.305138 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3050) took 2228 ms. Trace:
I20260709 12:23:34.394062 31823 raft_consensus.cc:2889] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:34.308354  6142 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:34.408282 31824 rpcz_store.cc:276] 0709 12:23:32.255067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.107479 (+1852412us) service_pool.cc:224] Handling call
0709 12:23:34.206912 (+ 99433us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.305377 (+ 98465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.305422 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:34.305493 (+    71us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:34.305542 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.305616 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.305769 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.409703 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2256) took 1106 ms. Trace:
I20260709 12:23:34.420303 31678 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:34.393612  5844 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:34.426477 31826 pending_rounds.cc:85] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 10
I20260709 12:23:34.424353 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1140) took 1108 ms. Trace:
I20260709 12:23:34.436966 31826 pending_rounds.cc:107] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 12:23:34.410190 32369 rpcz_store.cc:276] 0709 12:23:33.240213 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.240408 (+   195us) service_pool.cc:224] Handling call
0709 12:23:33.240737 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:33.241087 (+   350us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:33.241118 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:33.241600 (+   482us) log.cc:844] Serialized 26 byte log entry
0709 12:23:34.217042 (+975442us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:34.224616 (+  7574us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.233429 (+  8813us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:34.233484 (+    55us) raft_consensus.cc:1685] finished
0709 12:23:34.233498 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.256678 (+ 23180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":66178,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"thread_start_us":828121,"threads_started":1,"wal-append.queue_time_us":598545}
I20260709 12:23:34.417665  5641 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.430611 31676 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:34.429774 31748 rpcz_store.cc:276] 0709 12:23:33.245791 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.245984 (+   193us) service_pool.cc:224] Handling call
0709 12:23:34.298888 (+1052904us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x7b54002f67d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:34.299181 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":693310720}
I20260709 12:23:34.439218 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2257) took 1134 ms. Trace:
I20260709 12:23:34.433295 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2248) took 1213 ms. Trace:
I20260709 12:23:34.433809 31671 rpcz_store.cc:276] 0709 12:23:33.303571 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.308980 (+1005409us) service_pool.cc:224] Handling call
0709 12:23:34.409616 (+100636us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.424000  5695 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:34.431034 31822 rpcz_store.cc:276] 0709 12:23:32.077067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.780274 (+1703207us) service_pool.cc:224] Handling call
0709 12:23:33.780702 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:33.997559 (+216857us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.304727 (+307168us) spinlock_profiling.cc:243] Waited 163 ms on lock 0x7b54001671d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:34.305047 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":373789568}
I20260709 12:23:34.437506 31826 raft_consensus.cc:2889] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:34.437071 31827 rpcz_store.cc:276] 0709 12:23:33.316094 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.146814 (+830720us) service_pool.cc:224] Handling call
0709 12:23:34.167862 (+ 21048us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.168420 (+   558us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:34.168891 (+   471us) log.cc:844] Serialized 14 byte log entry
0709 12:23:34.423914 (+255023us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:34.423986 (+    72us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:34.424044 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.424132 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.424283 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":849,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":63,"raft.queue_time_us":9880,"thread_start_us":243574,"threads_started":1}
I20260709 12:23:34.426487  5981 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:34.451086 31675 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:34.445940 31677 rpcz_store.cc:276] 0709 12:23:33.219583 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.295732 (+1076149us) service_pool.cc:224] Handling call
0709 12:23:34.433217 (+137485us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.448491 31673 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:23:34.451162 31678 rpcz_store.cc:276] 0709 12:23:33.303832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.300861 (+997029us) service_pool.cc:224] Handling call
0709 12:23:34.438541 (+137680us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.453038  5981 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:34.453236  6168 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.454998  6105 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.466068 31824 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:23:34.466920 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2962) took 1023 ms. Trace:
I20260709 12:23:34.467279 31824 rpcz_store.cc:276] 0709 12:23:33.443428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.432868 (+989440us) service_pool.cc:224] Handling call
0709 12:23:34.465529 (+ 32661us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.466545 (+  1016us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.466816 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.467962  6105 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.467746  6151 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.468317  5870 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:34.469110 31824 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:34.469532 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2947) took 2214 ms. Trace:
I20260709 12:23:34.470252  6151 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:34.417922  6141 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.445178  5844 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:34.433106  5832 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.470968 31674 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
W20260709 12:23:34.460417 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 68110.5us
I20260709 12:23:34.470664 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:34.450687  6091 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:34.475297 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:34.477763 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 114778us
I20260709 12:23:34.460224 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2959) took 1043 ms. Trace:
I20260709 12:23:34.481905 31827 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:23:34.493901 31823 rpcz_store.cc:276] 0709 12:23:33.416842 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.280926 (+864084us) service_pool.cc:224] Handling call
0709 12:23:34.281483 (+   557us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.459856 (+178373us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.460144 (+   288us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:34.482719 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:34.488211 31821 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:34.488289 31822 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:34.477983 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:34.495937 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2960) took 1077 ms. Trace:
I20260709 12:23:34.481618 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:34.459968  5981 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:34.488915 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2961) took 1070 ms. Trace:
I20260709 12:23:34.500368 31672 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:34.469925 31818 rpcz_store.cc:276] 0709 12:23:32.255390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.115661 (+1860271us) service_pool.cc:224] Handling call
0709 12:23:34.116346 (+   685us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.117007 (+   661us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.117039 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:34.117761 (+   722us) log.cc:844] Serialized 26 byte log entry
0709 12:23:34.219312 (+101551us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:34.219379 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.219473 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:34.469276 (+249803us) raft_consensus.cc:1685] finished
0709 12:23:34.469300 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.469456 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1954,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":218,"thread_start_us":100912,"threads_started":1,"wal-append.queue_time_us":308699}
I20260709 12:23:34.499601 31826 rpcz_store.cc:276] 0709 12:23:33.418374 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.409387 (+991013us) service_pool.cc:224] Handling call
0709 12:23:34.413176 (+  3789us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.495560 (+ 82384us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.495857 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:34.508116 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.504864 31820 rpcz_store.cc:276] 0709 12:23:33.418676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.421677 (+1003001us) service_pool.cc:224] Handling call
0709 12:23:34.475812 (+ 54135us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.488562 (+ 12750us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.488589 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:34.488638 (+    49us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:34.488670 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.488727 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.488857 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4992}
I20260709 12:23:34.459599  5949 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:34.516148  6141 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.516290 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:34.517294  6141 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:34.517742  6141 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.465286  6168 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:34.518121  6141 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:34.518476  6168 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:34.518971  6168 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.519327  6168 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:34.487334  6136 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:34.486832  5832 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.479732  5695 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:34.526892 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 55813.2us
W20260709 12:23:34.526438 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 43838.8us
I20260709 12:23:34.544477 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:34.545267 31717 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:34.547017  6142 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:34.547096  5545 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:34.527726 31671 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:34.541909  5759 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:34.553630  5759 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.554054  5759 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:34.558840  5870 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:34.559886  6063 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:34.560657  6063 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.563059  6158 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:34.465101 31567 catalog_manager.cc:5697] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 11 to 18, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 18 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:34.564754  5695 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.569371  6168 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.526085  6136 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:34.571087  6136 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.571941 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3008) took 1221 ms. Trace:
I20260709 12:23:34.572359 31752 rpcz_store.cc:276] 0709 12:23:33.350185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.365050 (+ 14865us) service_pool.cc:224] Handling call
0709 12:23:33.365453 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.426344 (+1060891us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.548734 (+122390us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b54003e30d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:34.571852 (+ 23118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2187747072}
I20260709 12:23:34.574677 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.575558 31671 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:34.577790 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.564538 31567 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 13 to 19, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 19 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:34.571455  5981 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.479712  5887 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.587420 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:34.588948 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:34.597519 31675 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:34.582146 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:34.462859  5696 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.597919 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:34.598778 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:34.579566  6092 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:34.612147 31825 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:34.616818  5545 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:34.626129  5545 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:34.626580  5545 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.626996 31671 raft_consensus.cc:2376] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18. Re-sending same reply.
I20260709 12:23:34.628901 31567 catalog_manager.cc:5697] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:34.630983 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:34.615098  6012 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:34.631381 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1141) took 1314 ms. Trace:
I20260709 12:23:34.632407 31643 leader_election.cc:400] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
I20260709 12:23:34.631714 31567 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 12 to 14, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 14 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:34.632826 31643 leader_election.cc:403] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:23:34.634892 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 86682us
I20260709 12:23:34.636133 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:34.633930 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:34.632068 31819 rpcz_store.cc:276] 0709 12:23:33.316458 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.146884 (+830426us) service_pool.cc:224] Handling call
0709 12:23:34.188347 (+ 41463us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.188871 (+   524us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.188899 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:34.189571 (+   672us) log.cc:844] Serialized 26 byte log entry
0709 12:23:34.395463 (+205892us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:34.395495 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.395576 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:34.631078 (+235502us) raft_consensus.cc:1685] finished
0709 12:23:34.631101 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.631291 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":205422,"threads_started":1,"wal-append.queue_time_us":416325}
I20260709 12:23:34.650357 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.651294 31674 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:34.654803 31675 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:23:34.587556  6152 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 12:23:34.648669  6012 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:34.643455  6105 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
I20260709 12:23:34.643911  6091 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:23:34.672441 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 85210.2us
I20260709 12:23:34.674750  5887 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.675431  6151 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:34.676623  6158 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:34.677232  6158 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.677862  6158 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:34.674826  5870 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.006s
I20260709 12:23:34.682753  5805 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:34.684043  5805 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.686875  6063 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:34.687532  6063 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:34.689641  5981 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:34.690388  5981 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:34.690747  5981 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:34.692487 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58753.4us
I20260709 12:23:34.654673  6110 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:34.679723 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:34.674178 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:34.585876  5759 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:34.720736 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 46696.2us
I20260709 12:23:34.694610 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:34.659355  5995 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:23:34.730119 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50596.6us
I20260709 12:23:34.731410 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:34.733083 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:34.733587  5846 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:34.734694  5995 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:34.734795  6012 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.735781  6092 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:23:34.735318  5995 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:34.737540  6012 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:34.738224  6012 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:34.734007  5151 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:34.691745  5981 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.655136  6142 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.066s
I20260709 12:23:34.674336  5696 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.735806 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:34.754333 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:34.670691  6052 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:34.787091 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:34.801998 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66300.8us
W20260709 12:23:34.779299 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 84776.1us
W20260709 12:23:34.801960 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 105254us
I20260709 12:23:34.719554  6110 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.777290 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2391) took 1345 ms. Trace:
I20260709 12:23:34.827095 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:34.827981 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 396734us
W20260709 12:23:34.830435 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 100945us
I20260709 12:23:34.833709 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:34.763860 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:34.786856  5696 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.040s
I20260709 12:23:34.786586  5870 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:34.738605  6012 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.824458  6091 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:34.718264  5949 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.107s
I20260709 12:23:34.779516  6105 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:34.802163  6065 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:34.774366  6168 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.775418  5759 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.743362  6158 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.844285  5696 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:34.758252  5832 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.746981  6140 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:34.847725  6012 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:34.767588  5805 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.828402  6110 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:34.839179  6169 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:34.861783 31823 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:34.820772  5545 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:34.912797 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 87032.7us
I20260709 12:23:34.781987  6141 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:34.865358 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 78379.1us
I20260709 12:23:34.913826 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:34.746423  5995 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.852288 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:34.866333  5151 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:34.924185  5151 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:34.822216  5844 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.754684  6142 consensus_queue.cc:1243] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 16.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260709 12:23:34.902791 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 51045.9us
I20260709 12:23:34.828732 31746 rpcz_store.cc:276] 0709 12:23:33.432205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:33.938753 (+506548us) service_pool.cc:224] Handling call
0709 12:23:34.262330 (+323577us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:34.262936 (+   606us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:34.262980 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:34.263778 (+   798us) log.cc:844] Serialized 26 byte log entry
0709 12:23:34.599887 (+336109us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:34.599958 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:34.600087 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:34.754150 (+154063us) raft_consensus.cc:1685] finished
0709 12:23:34.754175 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:34.776850 (+ 22675us) spinlock_profiling.cc:243] Waited 251 ms on lock 0x7b54000b81d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:34.777105 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2115,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":98,"spinlock_wait_cycles":576862592,"thread_start_us":334714,"threads_started":1,"wal-append.queue_time_us":488134}
I20260709 12:23:34.864845  5696 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:34.849021  6052 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.863267  6162 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:23:34.933493  6158 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.934401  6158 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:34.934871  6158 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:34.935118 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 84102us
I20260709 12:23:34.936822  6136 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.937655 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:34.938863  5880 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:34.939133  6141 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:34.939278  6136 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.939352  5880 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.940049  5880 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:34.940474  5846 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:34.941191  6136 raft_consensus.cc:2764] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election decision vote started in defunct term 19: won
I20260709 12:23:34.937482  6092 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:34.943804  6092 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:34.944268  6092 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:34.945307  6092 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.948704  6092 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.956427  5696 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.958040  5696 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:34.919775  6169 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:34.960234  6169 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:34.963104 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.967083  5844 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.967355  6105 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:34.968040  6105 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:34.968525  6105 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:34.932101  6168 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:34.751847 31567 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 11 to 15, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 15 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:34.913623  6065 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:34.969560  6168 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.019822 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:35.020907  6168 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:34.871241  5641 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:35.022698 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:35.023689 31750 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:35.023913  5641 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:35.025291 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:35.028254 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:35.029824 31751 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:35.034539 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:35.036732  5545 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:34.931660 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 67937.8us
I20260709 12:23:34.986534 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:34.987191 31669 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19. Ops: []
W20260709 12:23:34.987885 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:34.987941 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:34.986156  5880 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:34.990137 31675 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:34.995683  5151 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.000277  5832 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:35.048790 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:35.051011 31675 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:23:35.004848  5995 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:35.012084 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:35.016701 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:35.054059 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:35.057021 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 402011us
I20260709 12:23:34.942049  5846 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:35.050565 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:35.066540 31673 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:23:35.079102 31676 raft_consensus.cc:2410] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 18 because replica has last-logged OpId of term: 15 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 13 index: 7.
I20260709 12:23:35.073766  5832 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 12:23:35.068744  6065 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:35.067128  6168 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:35.089319  5832 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:23:35.089725 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.090044  5995 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:35.089753  5832 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:35.091213  5995 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:35.091804  5995 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:35.092254  5832 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.092775  5995 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.095359  5151 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:35.078624 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:35.066269  6012 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:34.931133  6140 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.048779  6110 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:35.083472 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.053772  6158 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.105111 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:35.106187 31670 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:35.106395 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:35.094244  5832 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:35.086330  5880 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:35.115792 31750 raft_consensus.cc:2368] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20.
I20260709 12:23:35.117122  6110 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:35.103777 31823 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:35.150043 31820 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 20: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:35.072993  5630 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.109076  6052 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:35.182499 31675 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:35.198096 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 93106.8us
I20260709 12:23:35.186175 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:35.038435 31567 catalog_manager.cc:5697] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 8 to 19. New cstate: current_term: 19 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:35.176276  6012 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:35.213546 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:35.196522  5695 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.004s	sys 0.008s
W20260709 12:23:35.225804 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 149955us
I20260709 12:23:35.227133 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:35.228833  6194 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:35.229336  6194 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:35.229712  6194 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:35.094260  5995 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:35.231530 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:35.232210  6091 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:35.232610  5870 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:35.234594  5981 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:35.234983  5949 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:35.235575  6140 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:35.237028  6162 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:35.238358  6133 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:23:35.232336 31673 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:35.203079  6052 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:23:35.256691 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.262746 31670 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
W20260709 12:23:35.228286  5695 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.004s	sys 0.009s
I20260709 12:23:35.212703  6152 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:35.180383  5630 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.286892  6196 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:35.287220 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.287771  6168 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.250466  6140 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:35.304986 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:35.306198 31751 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:35.293787 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:35.307195 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 369647us
I20260709 12:23:35.247427  6052 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:35.249604  5793 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:35.288626 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:35.297675  6196 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:35.308748 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 371737us
I20260709 12:23:35.327870 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.327574 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:35.327447 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:35.330691 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:35.339028 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 153017us
I20260709 12:23:35.310361 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:35.333976 31745 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:35.348343 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:35.349354 32375 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:35.351342 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:35.352558 31749 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:35.343746 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.358235 31747 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:23:35.367025 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:35.369594 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:35.310500  6052 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:35.309890  5793 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:35.372366 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:35.375027 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:35.375500 31751 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:35.375860 31823 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:23:35.378890 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:35.379932 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:35.380367 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.380002 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:35.380002 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:35.350739  6158 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:35.384325 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:35.385237 31752 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:35.327260  5870 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.062s
I20260709 12:23:35.394670 32372 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:35.394676 31675 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:35.398183 32370 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:35.398437 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:35.401660 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:35.405623 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:35.406067 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.407770 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:35.327255  6168 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.408495 31672 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:35.410435 32371 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:35.410171 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:35.414587 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.416906 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.355134  5832 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:23:35.420291 31675 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:35.354780  6091 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
W20260709 12:23:35.433434 31677 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.020s	sys 0.023s
W20260709 12:23:35.434764 31677 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.020s	sys 0.023s
I20260709 12:23:35.440260 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:35.348091  6152 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.444847  5875 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.448648  5875 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.453905  6168 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:35.407413  6141 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:35.455116  6141 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.457340  6141 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:35.458277 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 74515.8us
I20260709 12:23:35.462895 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:35.462931 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 102961us
I20260709 12:23:35.471750 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.437539  6092 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:35.471465 31827 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:35.475318 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.478504 31676 raft_consensus.cc:2410] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:23:35.479727 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 15 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:35.480235 32370 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:35.480129  6201 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.035s
I20260709 12:23:35.480758 32373 raft_consensus.cc:2410] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 11.
I20260709 12:23:35.482514 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.483153  6141 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:35.484181 31675 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:23:35.485214  5545 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:35.485852  5545 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:35.486976  5545 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.487747  5949 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:35.488215  5949 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:35.489090  5949 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.507503  6092 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:35.529579 31671 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:35.540848 31671 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:35.541230  6092 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.541769 31671 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.528264  6196 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:35.521615 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:35.543738 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0702427199994418s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16890000000000002s for a response
W20260709 12:23:35.545681 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0298322309995456s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.199s for a response
W20260709 12:23:35.562858 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 89193.2us
I20260709 12:23:35.547312 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:35.442984 31567 catalog_manager.cc:5697] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 8 to 17. New cstate: current_term: 17 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:35.480018  5875 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:35.565905 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 91507.8us
I20260709 12:23:35.521950 31676 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:35.496873  5949 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:35.599475 31671 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:35.599947 32370 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: [19.16-19.16]
I20260709 12:23:35.622104 31823 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
W20260709 12:23:35.538184  5696 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.580s	user 0.014s	sys 0.002s
W20260709 12:23:35.624751  5151 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.529s	user 0.007s	sys 0.006s
I20260709 12:23:35.598834  5983 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:35.630003 32372 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:35.631313 31744 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:23:35.632714 31673 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:35.633333 31753 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:35.594880  5844 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:35.635628 31567 catalog_manager.cc:5697] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 7 to 16, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 16 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
W20260709 12:23:35.636529 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 226417us
I20260709 12:23:35.636813  5844 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.637758 32342 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:35.639285  5844 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:35.639385 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:35.598194  6142 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:35.644429 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:35.644276  6142 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.637717 31567 catalog_manager.cc:5697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 13 to 19, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 19 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:35.634928  5983 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:35.646946 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:35.647682 32370 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:35.647774  5983 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.647905 31567 catalog_manager.cc:5697] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 10 to 17. New cstate: current_term: 17 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:35.649003  5983 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:23:35.650769 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.653759  6052 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.634438  5875 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.654982  5875 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.657358  5875 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:35.618932  6012 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.657827  6052 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:35.658146  6012 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.660924  6012 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:35.661721 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:35.661877  5995 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:35.662477  6142 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:35.636073 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 82248.8us
I20260709 12:23:35.662601 32372 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:35.644446 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:35.663726 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:35.669023 31671 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:23:35.670746 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:35.649621 31567 catalog_manager.cc:5697] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 15 to 18, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 18 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.635007 31672 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:35.675290 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3113) took 1311 ms. Trace:
I20260709 12:23:35.684590 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:35.636546  6196 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:35.670284  6012 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:35.668990  6168 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:35.772176  6214 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:23:35.758296 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 72515.7us
I20260709 12:23:35.772693  6214 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:35.773097  6214 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:35.758728 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 93996.9us
I20260709 12:23:35.755965  6052 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:35.674146  5875 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:35.775579  5875 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:35.775985  5875 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:35.780324 31567 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 16 to 17, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 17 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.672263  5759 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:35.680191 31677 rpcz_store.cc:276] 0709 12:23:34.364376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:34.452814 (+ 88438us) service_pool.cc:224] Handling call
0709 12:23:34.493699 (+ 40885us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:34.512984 (+ 19285us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:34.871491 (+358507us) log.cc:844] Serialized 14 byte log entry
0709 12:23:35.674899 (+803408us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:35.674962 (+    63us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:35.675013 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:35.675090 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:35.675234 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":239041,"prepare.queue_time_us":129246,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"thread_start_us":872808,"threads_started":2,"wal-append.queue_time_us":612443}
I20260709 12:23:35.783434  5759 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:35.784001  5759 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:35.784754 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.785748  6141 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:35.757787  6199 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.787891  6199 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.790371 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:35.667251  5630 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:35.670215  5641 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.087s
I20260709 12:23:35.761036  6110 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:35.791989  6110 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:35.792409  6110 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:35.795327 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 150963us
I20260709 12:23:35.770257  6196 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.674547  6202 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:23:35.769834  6105 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.801s	user 0.008s	sys 0.004s
I20260709 12:23:35.792472  5805 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:23:35.793298 31792 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.786s (SENT) [suppressed 4 similar messages]
W20260709 12:23:35.796288 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 150656us
I20260709 12:23:35.791146 32375 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
W20260709 12:23:35.802557 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:35.669683  6194 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.814781  6183 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.815464  6183 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.816270 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.817802  6183 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:35.817912 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.819252 31673 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:35.820750 31677 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:23:35.809602 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:35.822248 32371 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:35.806324  6052 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:35.799566  5805 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:35.824211 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:35.826995 32370 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:35.827714  5151 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:35.828119  5151 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:35.828498  5151 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:35.828424 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:35.828469 32372 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:35.830283 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:35.830816 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:35.831131 31820 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:35.800482  6152 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:35.835781 31678 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:35.804221  6168 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:35.833772 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:35.806782  6202 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.840263  5805 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.841617  5805 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:35.782476 31567 catalog_manager.cc:5697] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 13 to 15, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 15 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:35.842829  6110 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.844398  6110 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:35.844210 31567 catalog_manager.cc:5697] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 9 to 14, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 14 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.842895  5832 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:35.846958  6199 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:35.847257  5832 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:35.847733  6196 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:35.848390  5832 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.15, Last appended by leader: 15, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.801584  5844 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.031s
I20260709 12:23:35.852998  5844 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:35.853367  5844 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:35.840334  6052 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.801231  5949 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.031s
I20260709 12:23:35.856323  5949 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:35.856679  5949 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:35.857496  5949 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:35.855510  6190 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.679s	user 0.005s	sys 0.001s
I20260709 12:23:35.799952  6105 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.865998 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:35.869175  5151 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.877291 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:35.838549  6091 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:35.851282  5875 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:35.889398 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 52460.1us
I20260709 12:23:35.891278  5875 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:35.831597  5696 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.045s
I20260709 12:23:35.891934 32369 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:35.880155 32372 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:35.894160  6105 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:35.894990  6202 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:35.854197  5844 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.896178  5844 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:35.896884  5793 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.897960  5793 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:35.846347 31567 catalog_manager.cc:5697] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 18 to 19, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 19 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:35.883922  6194 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:35.887144  6168 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.889209  5983 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:35.895147 31670 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:23:35.880704 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 87431.6us
I20260709 12:23:35.912027 32369 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:35.912170  5151 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:35.927034 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 120227us
W20260709 12:23:35.927147 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 87232.2us
I20260709 12:23:35.929840 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:35.930924 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:35.932837 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 55678.3us
I20260709 12:23:35.933652 31642 leader_election.cc:400] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20.
I20260709 12:23:35.934157 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:35.934490 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:35.934820 31824 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:35.935214 31827 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:35.935663 31642 leader_election.cc:400] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20.
I20260709 12:23:35.932482  6091 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:35.938138  6091 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.943462 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:35.944242 31822 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:35.944528 31824 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:35.945288 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:35.945442 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:35.934176  6202 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:35.946776  6202 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:35.947203  6202 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:35.947384  5695 raft_consensus.cc:993] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:23:35.951002 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:35.951596 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:35.952301  6004 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:35.952744  6004 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:35.953588  6004 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.951676 32372 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:35.894515  5759 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:35.932348 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:35.964967 31819 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:35.965806 32342 leader_election.cc:400] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20.
I20260709 12:23:35.967981 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:35.968616 31821 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:35.968806 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:35.970778  6105 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:35.963068  5983 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:35.974532  6105 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:35.964409  6141 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:35.975602  6141 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:35.976099  5759 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:35.976748  6105 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:35.976619  6141 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:35.979265 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 50853.5us
I20260709 12:23:35.980463 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:35.980700  6218 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:35.981882 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:35.982209  6218 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:35.977035 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:35.958199  6004 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:35.988265 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.996078 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:35.983605 31819 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:35.999030 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:35.999917 31671 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 19: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:35.995818 31670 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:36.002328 32343 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:35.966825  5641 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:36.020594 31752 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:36.013906 31822 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:23:36.021690 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:36.022181 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:35.954552  6214 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.022086 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:36.023420 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:36.042210 32369 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:36.021737 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:36.042500 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.019029 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:36.026093 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:36.028358 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:36.029289 31744 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:36.029865 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:35.992084  5759 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:36.031946  5832 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:36.026456 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:36.043334 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:36.047338 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:36.047873 31820 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:36.048089 31747 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:36.034041 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:35.991752  5695 raft_consensus.cc:1081] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Signalling peer 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:23:36.035076 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:36.020377  6199 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:35.976768  6183 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.044s
I20260709 12:23:36.012913  6213 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:23:36.066838 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.007959  5641 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:36.039495 31567 catalog_manager.cc:5697] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 13 to 16, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 16 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:35.983244  6218 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:35.992278  6222 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:36.019403 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:36.030349  5983 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:36.042618 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:36.082564 31749 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 23 because replica has last-logged OpId of term: 22 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 14.
I20260709 12:23:36.042109 31827 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:23:36.045861 31748 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:36.033390 31677 raft_consensus.cc:2368] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20.
I20260709 12:23:36.084383 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:36.085201 31827 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:36.019979 32372 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
W20260709 12:23:36.053481 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.013880  6194 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:36.057369 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:36.059665 32371 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:36.058292 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:36.015462  6052 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:36.087942 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:36.089205  5151 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:36.089861  5151 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:36.093225 31818 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:23:36.097302 31821 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:36.098733 31821 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:36.099869 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:36.101747 31826 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:36.103261 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:36.108083 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:36.078861  5641 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:36.063354  6213 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:36.110713 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:36.115967 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:36.111811 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:36.115958 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:36.092485  6052 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:36.152093 32373 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:36.155539 31826 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:36.156785 31824 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:23:36.153208 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:36.158051 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:36.160215  5880 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:36.160920  5880 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:36.161477  5880 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:36.164259  5949 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:36.126955 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:36.155522  5151 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.101017  5805 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:36.053884  5759 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.128902 32372 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
W20260709 12:23:36.131160 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:36.173717 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 63131us
I20260709 12:23:36.132483 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:36.075927  6199 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:36.134122  6225 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:23:36.120007  6213 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:36.177525 31751 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:36.139362  6214 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:36.157068  6052 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:36.145823 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:36.099992  6169 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.171577 31753 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:36.165732 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:36.175357 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.136381  6218 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.110545  6222 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:36.183451 31822 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:36.186316 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:36.190284 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:36.198799 31826 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:36.198568 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:36.173040  5805 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.194808 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:36.195154  5992 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:36.195482  6183 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:36.222034 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:36.224205 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:36.202920  6218 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:36.204321 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:36.207517 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:36.200412  6169 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.215436 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:36.217111 31825 raft_consensus.cc:2410] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:23:36.218043 31820 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:36.228449 31822 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:36.219565 31819 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:36.231398 31819 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:36.195884 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:36.232607 31753 raft_consensus.cc:2410] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:23:36.224447 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:36.226171 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:36.225191 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:36.232772 31820 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: [15.11-17.12]
I20260709 12:23:36.220618  5992 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.238219 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:36.193151 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:36.238965 31747 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:36.246815  5992 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:36.247093 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:36.247789 31753 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:36.200866  6196 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:36.220990  5805 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:36.252705 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:36.258728 31753 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:36.227303  5995 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:36.226816  6183 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:36.231007  6133 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:36.231422  5983 raft_consensus.cc:2764] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election decision vote started in defunct term 18: won
W20260709 12:23:36.266985 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47028.5us
I20260709 12:23:36.263537 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:36.236805  6194 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:36.284459 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:36.284518 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:36.285336 31751 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:36.285475 31753 raft_consensus.cc:2410] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 21 because replica has last-logged OpId of term: 19 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 13.
I20260709 12:23:36.267638  6183 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:36.287235 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:36.273626  6194 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:36.293917  6194 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:36.266521  6133 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:36.309221 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:36.310213 31751 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:23:36.268440  6196 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.313861  6196 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:36.297176  5695 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.315104  5695 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.324581 31827 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:36.325860 31818 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:23:36.324954 31747 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
W20260709 12:23:36.332130 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 107745us
I20260709 12:23:36.303982  6133 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:36.353057 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:36.353878 31825 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:23:36.311403  5759 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:36.357944 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:36.361701 32373 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:36.365581 31751 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:36.366782  5995 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:36.330260  6152 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:36.413720 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:36.423485 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:36.435249 31753 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:36.426044 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:36.443320 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.444938 31825 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:36.445971 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 158806us
I20260709 12:23:36.447449 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:36.448575 32369 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:36.383468  5759 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:36.457387 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:36.457638 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:36.461016 31827 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:23:36.368839  5630 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.078s
W20260709 12:23:36.493183  6241 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.123s	user 0.001s	sys 0.000s
I20260709 12:23:36.501708  5545 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:36.478812  6226 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.477077  5995 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.435642  6199 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:36.509020 31822 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:36.512998 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:36.513821  6140 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:36.515026 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:36.515374 31818 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:36.381464  5880 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.517943 31752 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:36.443912  6227 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:36.461347  5759 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:36.519624  6202 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.572s	user 0.012s	sys 0.001s
I20260709 12:23:36.539678 31827 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 12:23:36.527590 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 70320.1us
I20260709 12:23:36.504823  5846 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:23:36.537542  6242 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.098s	user 0.001s	sys 0.000s
I20260709 12:23:36.559374 31822 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 15 index: 11. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:36.521765  5545 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:36.561692 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.576848 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:36.534276  6196 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:36.536746  5832 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:36.515690  6223 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.048s
W20260709 12:23:36.584380 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.548346  5846 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:36.603631 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:36.607182 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.534400  6226 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.556768  6202 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.587937  6223 consensus_queue.cc:1243] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 15.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:36.582335  6196 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:36.580668  6065 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:36.549333  6227 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.592142  6199 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:36.621840 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:36.622036 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:36.621325  6152 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:36.625692 31672 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
W20260709 12:23:36.627585  6245 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.082s	user 0.003s	sys 0.000s
W20260709 12:23:36.628065  6239 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.236s	user 0.001s	sys 0.000s
I20260709 12:23:36.629568  6169 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:36.629726  5805 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.630177  5983 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:36.630785  5983 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:36.631708  5983 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.633963 32369 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:36.634802  5983 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:36.635850  6222 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:36.637183  6194 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.638836  5880 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:36.639000  5805 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:36.639534  6194 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:36.641006 31822 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:41204
I20260709 12:23:36.641757 31822 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:23:36.642144 31822 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:36.643028  6227 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:36.644681 31823 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
I20260709 12:23:36.619266  6196 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:36.647716  6249 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.064s	user 0.000s	sys 0.003s
I20260709 12:23:36.648779  5793 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.649468  5793 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.615057  6243 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.651206  6243 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.591655  5832 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.657253  5832 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:36.657954  5832 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.653708  6243 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:36.660849 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.666841  5846 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.658480  5832 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.670331 31749 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
I20260709 12:23:36.618220  6065 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:36.627657  6152 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:36.638936 31826 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:36.677436 31678 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:23:36.670567 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:36.683797 31669 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
W20260709 12:23:36.740000 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 69491.2us
W20260709 12:23:36.768352 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 146402us
I20260709 12:23:36.770613 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:36.772827 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:36.767227 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.770185 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:36.696372 31822 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:36.682142  5641 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.533s	user 0.010s	sys 0.002s
I20260709 12:23:36.679692  6065 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:36.678387  6202 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:36.652174  5793 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:36.783762 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:36.716130 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:36.771623  6227 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.772689  6222 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:36.793941 31669 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:36.795581 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:36.797497 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:36.808388  5151 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:36.810913 31822 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:36.790659 32342 leader_election.cc:400] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20.
I20260709 12:23:36.817593 32342 leader_election.cc:403] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Cancelling election due to peer responding with higher term
I20260709 12:23:36.781572  5949 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:36.673728  5832 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:36.815392 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:36.782218  5875 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:36.821197 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:36.827157 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:36.832554 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:36.835834  5151 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:36.683521  6183 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.681274  6196 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.807902  6162 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:36.786237  5870 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.049s
W20260709 12:23:36.843019 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 217307us
I20260709 12:23:36.835916 31670 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:36.656867  6213 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.714737  5759 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.809470  5641 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.846531 31676 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:36.676621  6226 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:36.677476  5545 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.12, Last appended by leader: 12, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.781289  6168 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:23:36.851508 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:36.844072 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 329090us
I20260709 12:23:36.834198  5793 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:36.861670 31826 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:23:36.823714  6255 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:36.810665  6201 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:36.873971 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 76583.2us
I20260709 12:23:36.874818 31677 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:36.850507 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:36.839278  5832 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.878273 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:36.888679 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:36.851552  6052 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.650s	user 0.010s	sys 0.002s
I20260709 12:23:36.846567  5983 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:36.892279 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:36.824038  6227 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.900148 31678 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:36.884073 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:36.886992 31642 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:36.853993  6141 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election lost for term 19. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20.
I20260709 12:23:36.854147  5949 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:36.904701 31676 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 18 index: 13. (index mismatch)
W20260709 12:23:36.866236  5992 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.619s	user 0.014s	sys 0.000s
I20260709 12:23:36.868137  6140 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.898236  6052 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.903976 31675 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:23:36.889154 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:36.842482  6133 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.910931  6222 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:36.911358  6169 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:23:36.882911  6252 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:23:36.905875 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:36.898926  6226 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:36.900568  5875 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:36.898481  6199 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:36.905892  5844 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:36.932492 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:36.853734  6162 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.906225  6012 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:23:36.934386 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45336.7us
I20260709 12:23:36.905490  6254 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.030s
W20260709 12:23:37.009385 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78632.1us
W20260709 12:23:37.011013 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 124145us
I20260709 12:23:37.013559 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:36.912251  6222 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.899557  5832 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.897620  6201 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:36.940280  5844 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:36.913085  5992 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:36.947361  5875 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:23:37.027711 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 82141.5us
I20260709 12:23:37.039155 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:37.015127  6202 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.039815 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:36.940156  6226 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:37.044153 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:36.955541  6133 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:36.947243  6262 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:36.917883  6140 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.035807 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:37.036248 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 103864us
I20260709 12:23:37.019073  5995 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:37.043689 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:37.044893 31826 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:37.045605 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:37.046265 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 147086us
I20260709 12:23:37.046489 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:37.082448 32370 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:37.054723 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:37.083654 31820 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:23:37.059842 32375 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:23:37.060453 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:37.085207 32369 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:23:36.906788  6255 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.049261 32378 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:23:37.071321 32373 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:37.033021  6247 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:37.018859  6168 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 12:23:36.944943  6199 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.026422  5949 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.037s
I20260709 12:23:37.034612  5844 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:37.076395 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:37.071769  6226 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.072875  5875 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.116544 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:37.116895 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 285088us
I20260709 12:23:37.094127  6168 consensus_queue.cc:1243] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 16.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
W20260709 12:23:37.124967 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 48700.3us
I20260709 12:23:37.126791 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:37.095660  6140 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.073485  6202 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.117900  6247 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:37.119436  6196 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:37.138167 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:37.138178 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:37.121012  6183 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:37.150261 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 136795us
W20260709 12:23:37.130990  6218 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.899s	user 0.012s	sys 0.001s
I20260709 12:23:37.120201  5983 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:37.159789 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:37.151237 31642 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:37.160984 32372 raft_consensus.cc:2368] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:37.135489  5641 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:37.149823  6012 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:37.135706  6213 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.122972  5875 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:37.174530 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.177120 31746 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:37.182478 31792 leader_election.cc:400] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:37.182969 31792 leader_election.cc:403] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
W20260709 12:23:37.185094 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 68622.2us
I20260709 12:23:37.170070  6196 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
W20260709 12:23:37.195463 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 365172us
W20260709 12:23:37.181571  6260 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.233s	user 0.001s	sys 0.001s
I20260709 12:23:37.171334  6227 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.178524  6052 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.207608 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:37.211187 31674 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 19: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:37.198957  5983 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:37.189038  6269 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.177667  6004 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.031s
I20260709 12:23:37.198163  6196 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:23:37.232115 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 94061.5us
I20260709 12:23:37.184329  6091 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.031s
I20260709 12:23:37.214841  6012 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.247476 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:37.251410 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:37.239627  6227 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:37.253159  6285 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:37.242063  6196 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:37.238528  5832 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:37.256922 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:37.259971 32372 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:23:37.215006  6270 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.153s
I20260709 12:23:37.240724  6162 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.246151  5992 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.256686  6012 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.265515 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:37.263283  6286 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.003s
I20260709 12:23:37.255077  6227 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.280323 31822 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:37.277298 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:37.284430  6281 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:37.285477  6272 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:23:37.286317 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:37.287515  6227 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:37.219280  6247 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.291648  5875 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.292222  6278 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:37.273084  6199 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:37.292799  6268 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.157s	user 0.001s	sys 0.001s
I20260709 12:23:37.295586 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:37.299731 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:37.300959 32372 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:23:37.302812 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:37.242581  6269 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.308974 31717 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:37.270160  6283 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.279445  6285 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:37.312589  6283 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.264362  5870 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 12:23:37.285274  6281 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.307268  6278 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:37.331420 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:37.333068 31824 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
W20260709 12:23:37.335289 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.310966  6199 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:37.323030  6202 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.332895  6091 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.334908  6278 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.334362  6289 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:37.339862  6199 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.364289  6269 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.365247  5545 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:37.354092  5875 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:37.367355  6271 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.165s	user 0.002s	sys 0.000s
I20260709 12:23:37.367053  5875 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.357427  6278 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:37.369535 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 2992) took 1001 ms. Trace:
I20260709 12:23:37.358304  6091 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.370002 31823 rpcz_store.cc:276] 0709 12:23:36.368395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.382424 (+ 14029us) service_pool.cc:224] Handling call
0709 12:23:36.382781 (+   357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:36.645126 (+262345us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.369218 (+724092us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b5400379454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:37.369461 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":601633664}
W20260709 12:23:37.373920  6065 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.559s	user 0.010s	sys 0.000s
W20260709 12:23:37.376540  6263 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.459s	user 0.001s	sys 0.000s
I20260709 12:23:37.379740  6287 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:37.380597  6227 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.383846  5844 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.427089  5844 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.386020  6281 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.386144 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:37.428188  6287 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:37.428376 31819 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:37.428790  6287 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:37.388820 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:37.358927  6152 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.586s	user 0.013s	sys 0.001s
I20260709 12:23:37.430967  6152 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.431854  6283 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:37.432192 31827 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:37.432265  6281 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:37.432868  6281 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.433285  6281 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:37.433008  6196 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.359200  6199 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:37.383898  5846 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:37.422698  6012 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:37.423177  6065 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.424394 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:37.446945 32369 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:37.434857 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:37.378356  6092 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:37.441409 31821 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:37.442312 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:37.471342 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:37.473394 32371 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
W20260709 12:23:37.476754  6238 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.099s	user 0.007s	sys 0.000s
W20260709 12:23:37.479293  6235 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.223s	user 0.006s	sys 0.014s
I20260709 12:23:37.480106  6280 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:37.480806  6280 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:37.484368 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:37.460760  6092 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:37.511142 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:37.512477 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:37.485399  6286 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:37.518445 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:37.519268 31818 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:37.518522 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:37.522845  6291 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:23:37.528048 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 276665us
I20260709 12:23:37.516047  5844 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:37.532713 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:37.533962 31825 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:37.512262  6247 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:37.532230  6065 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.538983  6292 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:37.539522  6012 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:37.539557  5844 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.547757  6091 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:37.489795  6269 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.043s
I20260709 12:23:37.518810  6286 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.506147  6226 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.042s
I20260709 12:23:37.527361  5870 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:37.560242  6303 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.012s
I20260709 12:23:37.561090  5983 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:37.549819  6280 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:37.562161  6012 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:37.551714  6292 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.563952  6227 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.565032  6196 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:37.555296  5844 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:37.566819  6286 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:37.552189  6247 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:37.616024 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57956.3us
I20260709 12:23:37.638082 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:37.691619 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:37.620620 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 580881us
I20260709 12:23:37.623584 31823 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
I20260709 12:23:37.627224 31820 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:37.628512 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:37.693158 31818 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:37.629172 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:37.687951 31822 pending_rounds.cc:85] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 10
I20260709 12:23:37.622268  5870 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.620323  5805 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:37.692564 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:37.692924 31823 pending_rounds.cc:85] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 12
I20260709 12:23:37.703605 31823 pending_rounds.cc:107] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 18.13
I20260709 12:23:37.696391 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:37.619984  6214 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:37.698807 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:37.699153 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 547984us
I20260709 12:23:37.624087  6254 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:37.702222 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:37.704134 31823 raft_consensus.cc:2889] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:37.694113 31822 pending_rounds.cc:107] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 15.11
I20260709 12:23:37.705382 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:37.712826 31822 raft_consensus.cc:2889] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:37.708395 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:37.625710  6012 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:37.629024  6158 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:37.705247  6248 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.104s	user 0.002s	sys 0.016s
I20260709 12:23:37.707700 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:37.698881  6279 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:23:37.711555 31820 pending_rounds.cc:85] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 10
I20260709 12:23:37.731537 31820 pending_rounds.cc:107] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 16.11
I20260709 12:23:37.731822 31820 raft_consensus.cc:2889] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:37.732059 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:37.712745 31745 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:23:37.737128  6276 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:37.737818  6276 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:37.708067  5695 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:37.738940  6276 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.741422  6201 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.742776  6105 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:37.743252 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:37.705796 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:37.744889  5870 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.687227  6262 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:37.745599  5805 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:37.746090  5805 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.746192  6262 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.748168 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:37.708106 31749 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:37.694873  6247 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:37.749202 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:37.746493  5805 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:37.752493  6258 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.753098  6258 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.705960  6133 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:37.708143  6270 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:37.755824  6270 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:37.756654  6300 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.194s	user 0.000s	sys 0.002s
I20260709 12:23:37.758093 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:37.758385 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3016) took 1744 ms. Trace:
I20260709 12:23:37.758906 31746 rpcz_store.cc:276] 0709 12:23:36.013614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.016086 (+  2472us) service_pool.cc:224] Handling call
0709 12:23:36.027968 (+ 11882us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:37.184961 (+1156993us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.758128 (+573167us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b5400064854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:37.758322 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2640156032}
I20260709 12:23:37.694162  5995 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:37.706532  6214 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:37.720683  6301 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.109s	user 0.000s	sys 0.007s
W20260709 12:23:37.721642  6296 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.166s	user 0.000s	sys 0.001s
I20260709 12:23:37.846665 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:37.723533 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:37.724587 31676 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 15. Preceding OpId from leader: term: 20 index: 16. (index mismatch)
I20260709 12:23:37.726861  6218 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:37.727114 31677 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:23:37.728034  6267 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.577s	user 0.005s	sys 0.002s
I20260709 12:23:37.723578  6251 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.743358  6105 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:37.745086 31753 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:37.759357 32373 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:37.717288 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:37.713021 31747 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:37.763572 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:37.764119 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:37.762562 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:37.624225  6278 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.767619 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:37.882491  6169 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:37.787919 31825 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:37.883129 31670 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:37.883234  6169 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.883917 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:37.884169  6222 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:37.884677 31825 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:37.884806  6251 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.885707 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:37.885879  6169 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:37.791673 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:37.886729 31749 raft_consensus.cc:606] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Received request to transfer leadership to TS b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:37.806404 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 103376us
I20260709 12:23:37.887817 31678 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 15: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:37.887959 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:37.758322  6270 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:37.892246  5151 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:37.708900  5983 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:37.746208  6162 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:37.787437 32370 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:37.833911 31746 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:23:37.834623 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 983) took 1198 ms. Trace:
I20260709 12:23:37.849488 31675 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
W20260709 12:23:37.849012 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 82203.8us
I20260709 12:23:37.864200 31677 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:37.864890 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:37.869887 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 107414us
I20260709 12:23:37.896049 31745 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:37.872466  6259 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.845659  6256 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:37.884670  6222 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.873380  6202 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.873796  5641 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.874250  5759 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.975554  5759 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.977954  5759 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:37.874683  6252 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.978930  5995 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.979393  6252 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.980161  6278 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:37.867512  6226 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:37.980916  6226 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:37.981356  6226 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:37.981667  6252 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.982925  6152 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:37.982864  6012 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:37.984336  6012 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:37.632220  6292 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:37.887224 31749 raft_consensus.cc:614] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:37.984717  5805 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.876181  6218 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:37.897557 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:37.993091 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:37.899232 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:37.872928  6243 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:37.900429 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:38.000746 31827 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:37.766827  6158 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.630211  6092 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:37.935452 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47576us
I20260709 12:23:37.888213  6140 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.961009 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:37.971228  5875 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.943776  6222 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:23:37.972581  6281 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.539s	user 0.010s	sys 0.015s
I20260709 12:23:37.973496 31670 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:37.973698  6105 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.974278  6258 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:37.974650  6202 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.003981  6252 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:37.982744 31677 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.950795  6311 raft_consensus.cc:2764] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:23:38.045399  6218 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:37.863580  6254 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:37.978395 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:38.078032 31824 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:38.079268 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:37.869103 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:38.080299 31674 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:23:37.998138 31823 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
W20260709 12:23:37.998108 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 99000.9us
I20260709 12:23:37.994542 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:37.999646 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:37.999889 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3143) took 1327 ms. Trace:
I20260709 12:23:37.902323  6251 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:38.082804 31672 rpcz_store.cc:276] 0709 12:23:36.672060 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.675129 (+  3069us) service_pool.cc:224] Handling call
0709 12:23:36.675572 (+   443us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:36.815295 (+139723us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:23:36.836932 (+ 21637us) log.cc:844] Serialized 14 byte log entry
0709 12:23:37.018781 (+181849us) log.cc:844] Serialized 14 byte log entry
0709 12:23:37.574571 (+555790us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b18000d7ce0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:23:37.618369 (+ 43798us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:37.626303 (+  7934us) log.cc:844] Serialized 26 byte log entry
0709 12:23:37.626539 (+   236us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:23:37.626565 (+    26us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.626614 (+    49us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:37.999651 (+373037us) raft_consensus.cc:1685] finished
0709 12:23:37.999675 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:37.999817 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":952999,"prepare.queue_time_us":84829,"prepare.run_cpu_time_us":128,"prepare.run_wall_time_us":8879,"raft.queue_time_us":817515,"raft.run_cpu_time_us":545,"raft.run_wall_time_us":24058,"spinlock_wait_cycles":3566848,"thread_start_us":55671,"threads_started":1,"wal-append.queue_time_us":264290}
I20260709 12:23:38.086711 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:38.087379 31752 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:37.939328  5870 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.135s
I20260709 12:23:38.088265 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:38.088908  5870 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:38.089277  5870 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:38.089394 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:38.090342 31677 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:38.090253  5870 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.090998 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:38.091616  5870 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:38.091912  6251 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:38.092315  6251 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.092662  6251 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:37.996088 31751 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:38.075649  6152 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.095073  6152 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.096038 31751 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:23:38.096571  6276 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:38.097227  6152 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:38.097340  6276 consensus_queue.cc:1243] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:37.901759  6162 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:38.097910  6276 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:38.098271  6276 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:38.098492  6162 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.099056  6276 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.099867  6162 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:38.100274  6276 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:37.902205 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:37.952970 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62948.6us
I20260709 12:23:38.103410 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:38.104534 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:38.104789  6258 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:38.105268  6258 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:38.105639  6258 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:37.895093 32371 rpcz_store.cc:276] 0709 12:23:36.632815 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.632945 (+   130us) service_pool.cc:224] Handling call
0709 12:23:36.633267 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:37.475182 (+841915us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.831137 (+355955us) spinlock_profiling.cc:243] Waited 840 ms on lock 0x7b54006c0554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:37.831357 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1931637120}
W20260709 12:23:38.107093  6287 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.678s	user 0.014s	sys 0.000s
I20260709 12:23:38.006469 31746 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:38.107645  6223 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:38.107577  6287 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.108224  6223 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:38.109144  6223 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.109918  6287 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:38.102489 31825 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:38.064383  6281 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.112783  6281 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:37.953491 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:37.905834  6196 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:38.002602 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:38.116067 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 113604us
I20260709 12:23:37.902243  6199 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.089995 31752 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:38.117448 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:38.013928  5805 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:38.119210 31821 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:38.119475  6202 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:38.120424  6196 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.120682  6254 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.123566  5759 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:38.124607  5759 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:38.124788 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:38.125108  5759 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:38.126259 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:38.127182  5880 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:38.127678  5880 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.127847  6183 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:38.128149  5880 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:38.128577  6183 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.130860  6244 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:38.131388 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:38.131911 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:38.132588 31746 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:38.133492  6169 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:38.134111  6158 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:38.134475  6169 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:38.134717 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:38.123596 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:38.015621 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:38.137262 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 183913us
W20260709 12:23:38.137972 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:38.021055  6304 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:23:38.021623  6243 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.023867  5844 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:38.024925  6299 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.465s	user 0.003s	sys 0.000s
I20260709 12:23:38.025130 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:38.025731 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:38.026722 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:38.029495  6247 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.033625  6286 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:37.902896  6294 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:37.971647  6255 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:38.059980 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:38.060295 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:37.899946  6065 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.107s
I20260709 12:23:37.909649  5992 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.133s
I20260709 12:23:37.978536  5641 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.072743 31827 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:23:38.153715 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:38.153971 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:38.083719 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:38.005343 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:38.101730  5151 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:37.967679  6259 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.155414  5151 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.155689 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:38.135133  6169 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:38.135473 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:38.135612  5949 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:38.014477 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:38.137194 31748 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:38.140852 31824 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:38.141479 32373 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:38.138064 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:38.151077 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.154430 31827 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:38.073968 31818 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:38.156484 32370 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:38.157385 31745 raft_consensus.cc:2368] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22.
I20260709 12:23:38.146162  6202 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:38.161022 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:38.049649  6140 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.142896  6234 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:38.248790 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:38.210860 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:38.249630 31827 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:38.248857 32375 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 17 index: 13. (index mismatch)
I20260709 12:23:38.147112  5983 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:38.211527 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57644.1us
I20260709 12:23:38.161015 31677 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:23:38.161346 31819 raft_consensus.cc:2368] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19.
W20260709 12:23:38.160432 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 155187us
I20260709 12:23:38.217280 31744 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:38.219110 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:38.223120 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:38.224097 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:38.224624  6303 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:38.226946 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:38.216670 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:38.214813 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.014042  6214 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.229424 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:38.141201  6287 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:38.159016 31752 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
W20260709 12:23:38.237648 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.270452 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:38.270457 32374 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:38.154206  6278 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.227635  6286 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:38.270777 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:38.248960  6234 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.252480  5151 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:38.272053 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:38.126165  5759 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.275104 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:38.146452  6133 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:38.228233  6202 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.276149 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:38.306972 31676 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 19: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:38.276772 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:38.308145 31823 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
W20260709 12:23:38.278839 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.311034 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:38.311786 32375 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:38.286219 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:38.287446 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 70873.3us
I20260709 12:23:38.286715  6234 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:38.276837  6255 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:38.319563  6255 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.278967 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:38.321633  6247 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:38.321858  6255 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:38.287762  6286 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:38.323483  6243 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:38.324503  6169 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.325835  6259 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:38.288170  6262 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:38.327414  6262 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:38.327726  6196 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:38.284843  6092 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:38.328336  6252 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:38.328447  6262 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.328992  6252 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.276014  6287 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:38.289034 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:38.331107  6287 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:38.331957 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:38.333005 31642 leader_election.cc:400] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22.
I20260709 12:23:38.289494 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:38.335425 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46014us
I20260709 12:23:38.336577 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:38.336858  6270 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:38.337338  5545 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:38.290110 31818 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:38.337956  5545 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.338366  5545 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:38.289649  6285 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:38.339040  6012 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:38.281078  6278 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.339382 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:38.337728 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:38.290807 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:38.304479 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:38.277762 32378 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:38.339047 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:38.318562  5844 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:38.399129 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:38.401257 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:38.402894 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:38.337308  6270 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:38.337742 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:38.406754 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 69091.9us
I20260709 12:23:38.407670 32343 leader_election.cc:400] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19.
I20260709 12:23:38.407979 32343 leader_election.cc:403] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Cancelling election due to peer responding with higher term
I20260709 12:23:38.352725 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:38.409312 31752 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:38.269414  6315 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:38.348788 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:38.216503  5949 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:38.390380 31824 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:38.299111  5641 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:38.390946 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53280us
I20260709 12:23:38.391608 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:38.453256 31821 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:38.392577 31674 raft_consensus.cc:2410] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:23:38.392848 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:38.455222 31827 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:38.393543 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 87871.9us
I20260709 12:23:38.405516 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:38.360316 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:38.379086  6259 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:38.457365 32375 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:38.457414 31823 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:23:38.160562  6281 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.227s
I20260709 12:23:38.275493  5995 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:38.291978  6202 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:38.322546  6255 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:38.323279  6286 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:38.379832  6201 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.379082  5695 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:38.448752 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 100061us
I20260709 12:23:38.457553 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3004) took 1170 ms. Trace:
I20260709 12:23:38.454380 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:38.281672  6251 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.481277 31826 rpcz_store.cc:276] 0709 12:23:37.265284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.265405 (+   121us) service_pool.cc:224] Handling call
0709 12:23:37.265742 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:37.352745 (+ 87003us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:37.352768 (+    23us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:37.360412 (+  7644us) log.cc:844] Serialized 26 byte log entry
0709 12:23:37.461241 (+100829us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:37.461307 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:37.461420 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:38.024322 (+562902us) spinlock_profiling.cc:243] Waited 58.9 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:38.406064 (+381742us) raft_consensus.cc:1685] finished
0709 12:23:38.406091 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:38.435810 (+ 29719us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":27139,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"spinlock_wait_cycles":135459712,"thread_start_us":176508,"threads_started":2,"wal-append.queue_time_us":157074}
I20260709 12:23:38.460366 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:38.484284 31673 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:38.303929  5983 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.301363  6199 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:38.406324  6256 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:38.152444  6294 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.465374 32374 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:38.388101  6247 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:38.500150 31677 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:38.475809  6259 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:38.405124  6303 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:38.506850 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:38.507630 31827 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:38.507812 31673 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:38.283385  6140 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:38.284741  6226 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.451571  5992 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:38.461918  5844 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:38.525043 31822 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:38.480671 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:38.378765  6196 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.091s
I20260709 12:23:38.530977 31819 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:38.396996  6162 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.078s
I20260709 12:23:38.461428 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:38.465324  5641 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:38.334508  6222 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.478617  6286 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:38.510872 31824 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:38.538826 31821 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:38.501013  6316 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:23:38.542670 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 404624us
I20260709 12:23:38.543845 31824 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:38.526244 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45643.7us
W20260709 12:23:38.547608 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 261478us
I20260709 12:23:38.548516 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:38.528505 31827 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:38.530094 31677 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:38.513830 31823 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:38.468116  5995 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:38.555207 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.512694  6252 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:38.486678  6201 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.530458  6140 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:38.490021  5695 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.524672  5844 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.492602  6258 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.570874 32375 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:38.557629  6328 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.124s
I20260709 12:23:38.535418  5846 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:23:38.638423 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 89979.5us
I20260709 12:23:38.565063 31824 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:23:38.680063 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:38.748632 32378 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:38.748523 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:38.633455  6254 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:38.641916 31819 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:23:38.500155  5793 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:23:38.548250  6316 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:38.669535 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 208160us
W20260709 12:23:38.644209  6321 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.109s	user 0.001s	sys 0.000s
I20260709 12:23:38.625830 31826 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:38.733137  5844 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:38.643350  6318 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:23:38.749274  5846 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:38.548573  5641 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.635007 31822 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:38.493945  6270 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:38.574975  6303 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:38.670879  6259 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.708050 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:38.808316 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 100363us
I20260709 12:23:38.466423  5545 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.644251  6183 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:38.754931 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:38.777858 31826 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:38.534305  5992 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:38.708997 31673 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:38.757229  6158 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.787073 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:38.786208 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:38.832357 31677 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 25 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:23:38.791153 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:38.792604 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 323616us
I20260709 12:23:38.779860  6278 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:38.778448  5880 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.650s	user 0.013s	sys 0.000s
I20260709 12:23:38.843158 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:38.785781  5846 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:38.845964 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:38.780256  6316 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:38.819226 32375 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:38.835943 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:38.837307 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46256.3us
I20260709 12:23:38.838172  5545 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:38.856212 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:38.857650  5545 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:38.861385 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:38.862573 32370 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:38.839074 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3097) took 1077 ms. Trace:
I20260709 12:23:38.865658 31821 rpcz_store.cc:276] 0709 12:23:37.761335 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.493184 (+731849us) service_pool.cc:224] Handling call
0709 12:23:38.538584 (+ 45400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:38.539325 (+   741us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.838779 (+299454us) spinlock_profiling.cc:243] Waited 44.9 ms on lock 0x7b5400207bd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:38.838989 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103325568}
I20260709 12:23:38.870064 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:38.810067 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:38.807991  6315 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:38.875949 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 66017.2us
W20260709 12:23:38.876338 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.877442 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:38.878160 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:38.883657 31744 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.609s	user 0.012s	sys 0.000s
I20260709 12:23:38.884232 31744 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:38.823208  6105 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:38.888705 32375 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:38.839962  6092 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:38.890946  6337 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:38.892150  6348 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:38.897089 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:38.900655  6169 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:38.898764  6345 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:38.910372 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:38.926126  5805 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:38.813732  6256 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.061s
I20260709 12:23:38.908973  5793 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.070s
W20260709 12:23:38.878113  6234 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.556s	user 0.008s	sys 0.004s
I20260709 12:23:38.869263  6278 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:23:38.958532 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:38.956466 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 100370us
I20260709 12:23:38.908165  6251 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:38.846297  6158 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.922684  6294 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:38.962728  6347 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.060s	user 0.001s	sys 0.004s
I20260709 12:23:38.971352 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3163) took 1033 ms. Trace:
W20260709 12:23:38.972879  6292 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.988s	user 0.013s	sys 0.000s
I20260709 12:23:38.974078  6258 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:38.923437  6222 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:38.950320  6310 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:23:38.864444  5880 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:38.911512  6226 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:38.910588  6092 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:38.917320  6337 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:38.911211 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:38.950181  6270 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:38.889357  6328 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.074s
I20260709 12:23:39.086982  6201 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:38.951287  6335 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:38.962950  6012 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:39.080025 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:39.119894 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.120694 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:38.971355  6091 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:39.071372  6317 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.559s	user 0.005s	sys 0.000s
W20260709 12:23:39.078053 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 342941us
I20260709 12:23:39.090179 31675 rpcz_store.cc:276] 0709 12:23:37.938341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.971719 (+ 33378us) service_pool.cc:224] Handling call
0709 12:23:37.972081 (+   362us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:37.972419 (+   338us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:37.972456 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:37.972919 (+   463us) log.cc:844] Serialized 26 byte log entry
0709 12:23:38.143822 (+170903us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:38.143847 (+    25us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.143920 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:38.971015 (+827095us) raft_consensus.cc:1685] finished
0709 12:23:38.971047 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:38.971262 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2545,"raft.run_cpu_time_us":81,"raft.run_wall_time_us":81,"thread_start_us":170569,"threads_started":1,"wal-append.queue_time_us":488341}
I20260709 12:23:38.917035  6105 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.073787  6169 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:39.081873  6322 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.449s	user 0.001s	sys 0.000s
I20260709 12:23:39.107345 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:39.110517 31818 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:23:39.137917 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:38.845619  6183 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:39.084924  6278 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:39.087577  6345 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:39.144824  6345 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.087177  6303 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:39.147351  6303 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.094990  5832 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 12:23:39.077477  5805 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.153573  5805 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.121433 31669 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:39.154713 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:39.155198 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:39.155927 31821 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:39.155964 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:39.156718 31669 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:39.156985  6336 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
I20260709 12:23:39.078576  5992 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.158355  6169 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:39.124979  6326 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.330s	user 0.000s	sys 0.001s
I20260709 12:23:39.112890  6337 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:39.139114  6278 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:39.090514  6315 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:39.143116  6340 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
I20260709 12:23:39.099293  6292 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.080740  6234 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.163142 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:39.169029  6286 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.615s	user 0.009s	sys 0.003s
I20260709 12:23:39.163968 31670 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 21 because replica has last-logged OpId of term: 20 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 10.
I20260709 12:23:39.158592  6158 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.160643 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:39.222399 31673 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:39.173619 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:39.224895 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:39.225112 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:39.175207 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:39.220490  5870 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.128s	user 0.007s	sys 0.006s
I20260709 12:23:39.227808 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:39.228509 31821 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:23:39.229449  6334 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.265s	user 0.000s	sys 0.001s
I20260709 12:23:39.176503 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:39.131507  6091 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.231374 31676 raft_consensus.cc:2368] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
I20260709 12:23:39.178047  6105 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:39.181447  6343 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:23:39.181159  5846 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.185317 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3162) took 1276 ms. Trace:
I20260709 12:23:39.275085 31678 rpcz_store.cc:276] 0709 12:23:37.908900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:37.909167 (+   267us) service_pool.cc:224] Handling call
0709 12:23:37.909569 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:37.938398 (+ 28829us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:37.938428 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:37.971322 (+ 32894us) log.cc:844] Serialized 26 byte log entry
0709 12:23:38.409138 (+437816us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:38.409182 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.409260 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:38.762499 (+353239us) spinlock_profiling.cc:243] Waited 5.05 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:39.184989 (+422490us) raft_consensus.cc:1685] finished
0709 12:23:39.185013 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.185237 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":28769,"raft.run_cpu_time_us":118,"raft.run_wall_time_us":118,"spinlock_wait_cycles":11622784,"thread_start_us":437395,"threads_started":1,"wal-append.queue_time_us":488535}
I20260709 12:23:39.267695 31821 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:23:39.189011 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:39.191776 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:39.192535 31820 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:23:39.279168 31672 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:39.195310 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:39.278350 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:39.119125  5695 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.159721 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:39.201079  6243 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:39.291088 31677 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:39.205389  6234 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.206597 32371 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 19: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:39.207192  5875 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.209548  5151 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:39.293900  5151 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:39.294763  5151 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.295955  5151 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:39.212049  5995 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.213090  6292 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:39.213709 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:39.254782  6254 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:39.215570 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:39.303750 31674 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:39.215977 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:39.216476  6286 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:39.186384 31818 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:23:39.186098  6359 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.056s
I20260709 12:23:39.161228  6092 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.158854  5641 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:39.220019 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:39.124784  6012 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.235494 31827 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:39.238754 31791 leader_election.cc:400] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19.
W20260709 12:23:39.313316 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74824.7us
I20260709 12:23:39.249730 31669 raft_consensus.cc:2410] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:23:39.254142 31676 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:39.257563 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:39.316020 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58565.1us
I20260709 12:23:39.317580 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:39.261101 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.189601  5880 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.124169  6335 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.257073 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:39.280074 32373 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:39.140122  6252 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:39.291400 31673 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:39.235889  6262 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.299389 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:39.300078 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.298557 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:39.329795 31820 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:39.214653  5949 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.301837 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:39.306875 31672 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:39.278898  6158 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.308380 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:39.336992 31818 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
W20260709 12:23:39.307536 31824 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.016s	sys 0.009s
I20260709 12:23:39.309284 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:39.337762 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:39.338419 31674 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:39.338754 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:39.309017 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:39.319892 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 62868.8us
W20260709 12:23:39.340893 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:39.337782 31824 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.564s	user 0.016s	sys 0.009s
I20260709 12:23:39.321897 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3055) took 1187 ms. Trace:
I20260709 12:23:39.323473 31827 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:39.324932 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:39.344429 31827 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:39.327181 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:39.332028 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:39.346006 31676 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:39.346930 31819 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:23:39.348146 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:39.348766 31672 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:23:39.333474 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:39.335965 31821 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:39.350999 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:39.343225 31746 rpcz_store.cc:276] 0709 12:23:38.134238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.135303 (+  1065us) service_pool.cc:224] Handling call
0709 12:23:38.135630 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:38.136026 (+   396us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:38.404486 (+268460us) log.cc:844] Serialized 14 byte log entry
0709 12:23:38.892780 (+488294us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:38.893476 (+   696us) log.cc:844] Serialized 26 byte log entry
0709 12:23:38.893666 (+   190us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:38.893708 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:38.893789 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:39.228828 (+335039us) spinlock_profiling.cc:243] Waited 80.3 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:39.321588 (+ 92760us) raft_consensus.cc:1685] finished
0709 12:23:39.321681 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.321836 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":387864,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":8312,"raft.queue_time_us":68001,"raft.run_cpu_time_us":351,"raft.run_wall_time_us":33084,"spinlock_wait_cycles":184785152,"thread_start_us":709925,"threads_started":3,"wal-append.queue_time_us":421584}
I20260709 12:23:39.353168 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:39.329016  6105 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:39.354962 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.326568  6234 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:39.330761  6292 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.328390  6335 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.364971 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:39.375118 31674 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:23:39.330770  5995 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:39.301867  5875 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.382490 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:39.383400 31819 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 19
I20260709 12:23:39.371729 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.371923 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:39.372051 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:39.385774 31747 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:39.386449 31821 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:39.302294  6243 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.368084 31827 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 15.8, Last appended by leader: 8, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.405807 31820 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:23:39.415930 31827 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:39.414431 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:39.417037 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.417702 31752 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:23:39.419202 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:39.366357  6183 raft_consensus.cc:2764] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election decision vote started in defunct term 16: won
I20260709 12:23:39.356508  6091 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.428409 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3028) took 1227 ms. Trace:
I20260709 12:23:39.327540  6262 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:39.418953  5995 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:39.426599  6333 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.090s
I20260709 12:23:39.424017  6356 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:39.367122  6234 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:23:39.431134 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53999us
I20260709 12:23:39.438198 31827 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:39.371575  5880 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:39.435271 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:39.385326  5949 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:39.437052 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:39.440660 31752 raft_consensus.cc:606] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:39.441038 31752 raft_consensus.cc:614] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:39.437309 31678 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:39.441632 31827 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:39.346139  6158 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:39.444291 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.440327 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:39.413254  6303 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:39.366935  6270 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.063s
W20260709 12:23:39.451778 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.452566 31747 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:23:39.460996 31820 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:39.462509 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.462620 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:39.463891 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.463932 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:39.465871 31752 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:39.467109 31747 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:39.470454 31752 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 24. Ops: [22.16-22.16]
I20260709 12:23:39.470515 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.415524  6335 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.038s
I20260709 12:23:39.474443 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.387089  6292 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:39.478741 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.489988 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:39.459264  6202 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.972s	user 0.012s	sys 0.006s
I20260709 12:23:39.482774 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:39.494983 31750 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:39.499967 31678 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
W20260709 12:23:39.474598  6287 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.137s	user 0.014s	sys 0.000s
I20260709 12:23:39.524720 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.525609 31751 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:23:39.487465  6183 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:39.539146 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.490849  5880 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:39.439795  6234 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:39.486918 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 63575.4us
I20260709 12:23:39.501341  6328 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:39.473380 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.492496 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.492703 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.481174 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:39.511415  6365 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:23:39.453330  6356 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.543292 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.551326 31751 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:39.510617 31750 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:39.486032 31819 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:39.547940 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.552295 31753 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:39.551600 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:39.553952 31752 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:39.552541 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:39.548596 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 67504.5us
I20260709 12:23:39.555402 31752 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:39.556628 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:39.556970 31820 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:23:39.558164 31819 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:39.468055  6303 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.560053 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:39.564368 31751 raft_consensus.cc:2410] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:23:39.567292 31750 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:39.568511 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:39.568943  6158 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.569154 31749 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:39.569950  5875 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:39.571063  6012 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:39.574996 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.575418  6243 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.544051  5880 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:39.576179  5880 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:39.576268  6286 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:39.576514  6278 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:39.560235 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 120000us
I20260709 12:23:39.537688  5846 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.559895  6091 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.018s
W20260709 12:23:39.579298 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:39.579914 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:39.580654 31749 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:39.580763  5151 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:39.527758  6202 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.588478  6202 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.589615 31671 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:39.589706 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.590811 31749 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:39.591639 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.592492 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.592798 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:39.593446 31749 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:39.540007  5995 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:39.560460 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 120191us
I20260709 12:23:39.599255 31752 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:23:39.548502  6328 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:39.555023  6316 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.678s	user 0.000s	sys 0.012s
I20260709 12:23:39.602116 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.602787 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:39.549584  6254 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:39.530407  6287 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.602867 31752 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:39.610931 31748 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: [15.8-17.9]
I20260709 12:23:39.607553  5151 consensus_queue.cc:1243] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 21.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:23:39.611863 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:39.610852 31749 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:23:39.604655  6328 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:23:39.616291  6355 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.301s	user 0.001s	sys 0.001s
I20260709 12:23:39.607105 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.614286 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.625728 31749 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:39.601226  6243 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader election won for term 15
I20260709 12:23:39.630472 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:39.633078 31749 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:39.633534 31746 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:39.634804 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.636484 31749 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:39.643342 31748 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:39.644570 31750 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:23:39.651808 31749 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:39.653828 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.655475 31749 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:39.634786  6012 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.608436  6316 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.640484  5846 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:39.647449  6243 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:39.702486  6364 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.268s	user 0.000s	sys 0.002s
I20260709 12:23:39.723587 32377 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:39.717877 31674 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:39.731058  5983 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:39.725142 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:39.731735  5983 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:39.731986 31674 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:39.735558 31674 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:23:39.736728 31674 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
W20260709 12:23:39.737608 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3100) took 1901 ms (client timeout 2109 ms). Trace:
W20260709 12:23:39.741114 31822 rpcz_store.cc:269] 0709 12:23:37.835861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.083175 (+247314us) service_pool.cc:224] Handling call
0709 12:23:39.737431 (+1654256us) spinlock_profiling.cc:243] Waited 407 ms on lock 0x7b54001c93d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:39.737548 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":935857408}
I20260709 12:23:39.738422 31823 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: []
I20260709 12:23:39.742609 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:39.659983  6223 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:39.722752  5846 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.739018  5759 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:39.726397  5844 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.926s	user 0.005s	sys 0.005s
I20260709 12:23:39.753017 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:39.757318 31822 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:23:39.707089 31748 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:23:39.760085 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:39.717485  6287 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:39.762399 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:39.716203  6303 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:39.719199  6316 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:39.769335 31822 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:23:39.767390 31750 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:39.773101 31753 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:39.719352  6262 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:39.702286  6012 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.751758  6255 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:23:39.792634  6254 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:39.793563  6244 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:23:39.794059  6337 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.618s	user 0.017s	sys 0.003s
I20260709 12:23:39.763553  5805 raft_consensus.cc:2764] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:23:39.795114  6214 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:23:39.789961  6140 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.057s	user 0.014s	sys 0.000s
I20260709 12:23:39.791061  6303 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.008s
I20260709 12:23:39.790568 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.786240  5995 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.022s
I20260709 12:23:39.801863 31747 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:39.826335 31748 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 19: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:39.737653  6243 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 15, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.757725  6315 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 12:23:39.732297  5983 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:39.805217  6356 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:39.806388  6278 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:39.767951  6276 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.039s
I20260709 12:23:39.767920  6333 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:39.807500 31678 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:39.808209 31669 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:39.760391  6223 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:39.760607  5844 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.785200  6105 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
W20260709 12:23:39.849447 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 119178us
I20260709 12:23:39.814780  5759 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.025s
I20260709 12:23:39.838376 31750 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:39.826622  6140 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election lost for term 18. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19.
I20260709 12:23:39.819336  6337 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.844549  6356 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:39.864835 31745 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 19: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:39.866137 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:39.868369 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:39.839581  6234 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.853040  6333 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:39.878016 32374 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:39.870839 31752 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:39.864642  6105 consensus_queue.cc:1243] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 21.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:23:39.864928  6356 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:39.852062  5880 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.896634  5844 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:39.896466  6356 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.894142  6223 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.898043  6356 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:39.908227 31673 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:23:39.884235  6385 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:23:39.888039  6383 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:39.914551 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3021) took 1773 ms. Trace:
I20260709 12:23:39.946182 31823 rpcz_store.cc:276] 0709 12:23:38.141608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.562172 (+420564us) service_pool.cc:224] Handling call
0709 12:23:39.738237 (+1176065us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:39.742797 (+  4560us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.914126 (+171329us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54001c93d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:39.914455 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2696625536}
I20260709 12:23:39.947687 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:39.948390 31823 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
W20260709 12:23:39.919538  6381 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:23:39.922075 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:39.952208  6379 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:23:39.952332 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 960) took 1229 ms. Trace:
I20260709 12:23:39.953039 32374 rpcz_store.cc:276] 0709 12:23:38.723142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.723303 (+   161us) service_pool.cc:224] Handling call
0709 12:23:39.952055 (+1228752us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400546354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:39.952188 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2568214912}
I20260709 12:23:39.886837  6328 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.957511 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:39.960983  6331 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:39.961730  6331 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:39.962827 31745 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:39.925014 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:39.963757 31827 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:39.923904  6337 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.927155 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:39.967090 32373 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:39.967715 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:39.968042 31749 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:39.968341 31750 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:39.428952 31824 rpcz_store.cc:276] 0709 12:23:38.200887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.747411 (+546524us) service_pool.cc:224] Handling call
0709 12:23:38.747741 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:38.747973 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:23:39.341914 (+593941us) log.cc:844] Serialized 14 byte log entry
0709 12:23:39.428017 (+ 86103us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:39.428072 (+    55us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:23:39.428121 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.428197 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:39.428344 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":148647,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":463,"thread_start_us":654268,"threads_started":2}
I20260709 12:23:39.925413  5983 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:39.930959 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:39.971025 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:39.932727  6333 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.939245 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2280) took 1171 ms. Trace:
I20260709 12:23:39.961266  6247 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:39.966419 31745 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:39.973037 31669 rpcz_store.cc:276] 0709 12:23:38.768189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.441276 (+673087us) service_pool.cc:224] Handling call
0709 12:23:39.807990 (+366714us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:39.843273 (+ 35283us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.938982 (+ 95709us) spinlock_profiling.cc:243] Waited 345 ms on lock 0x7b54002e7f50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:39.939163 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":793689344}
I20260709 12:23:39.930464  6234 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:39.945147  6369 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.582s	user 0.000s	sys 0.003s
I20260709 12:23:39.971652 31744 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:39.926793 32378 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:39.962724  6331 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 14.9, Last appended by leader: 9, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:39.961050  6012 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:39.977102  6331 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:39.977236 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:39.977560  6328 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.973302  6247 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:39.934401  6292 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:39.979398  6247 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:39.981971 32373 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:23:39.971043 31820 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:39.978466  5880 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:39.966471 31827 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:39.990458 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:39.993988 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:39.995042 31827 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:39.997520 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:40.002485 32378 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:23:40.009045 31750 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:40.012534 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:40.017148 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:40.013932 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:40.016362 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:40.021320 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:40.022265 32374 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:40.023620 32369 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:23:40.026280 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:40.028331 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:40.029382 31750 raft_consensus.cc:2368] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
I20260709 12:23:40.041142 32374 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:40.043576 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:40.044487 31827 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:40.032622 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:40.048678 31748 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:40.049401 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:40.053123  6345 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:40.031277  6092 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:40.028407  5880 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:40.059696 31827 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:40.031942  6202 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.023s
I20260709 12:23:40.053632  6345 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.061134 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:40.061300 31820 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:40.014343  6276 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
I20260709 12:23:40.032485  6252 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:40.070844 31824 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:40.075706 31748 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:23:40.077618 32376 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:40.059109  5695 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:40.081092 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:40.083386 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.088107 31827 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:40.089731 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:40.084887 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:40.060359  5880 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.093497  5880 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:40.090802  6252 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:40.099535  6252 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:40.105962  6278 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:40.072464  6345 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:40.106745  6278 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:40.107822  6278 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:40.070290  6281 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:40.082022  6377 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.112560  6377 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:40.129194 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:40.130545 31827 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 19: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:40.117259  5695 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:40.134881 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:40.137220 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:40.139089  5870 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:40.143352  6292 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:40.143982  6292 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:40.144932  6292 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:40.145308  5846 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:40.139145  5695 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:40.146201  5870 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:40.147070 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 996) took 2004 ms. Trace:
I20260709 12:23:40.147279  5870 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:40.147473 32372 rpcz_store.cc:276] 0709 12:23:38.142757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.142962 (+   205us) service_pool.cc:224] Handling call
0709 12:23:38.505384 (+362422us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:38.692124 (+186740us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:38.692172 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:38.740225 (+ 48053us) log.cc:844] Serialized 26 byte log entry
0709 12:23:39.075173 (+334948us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:39.077367 (+  2194us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:39.077496 (+   129us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:39.220994 (+143498us) raft_consensus.cc:1685] finished
0709 12:23:39.221021 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:40.146771 (+925750us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b5400499150. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:40.147006 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":37698,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":69,"spinlock_wait_cycles":797873920,"thread_start_us":479175,"threads_started":2,"wal-append.queue_time_us":359552}
I20260709 12:23:40.153956  6359 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:23:40.158418 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 290105us
I20260709 12:23:40.082731  6287 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.054s
I20260709 12:23:40.159560 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:40.167449 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:40.172981  5151 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:40.174132 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:40.176991 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:40.178572 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:40.182163 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:40.182674 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:40.154497  6359 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:40.137056  6281 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:40.187148 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:40.185945  6396 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:23:40.173926  5949 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.014s
W20260709 12:23:40.202881 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.181445  6303 raft_consensus.cc:2764] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:23:40.181495  6383 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:40.181710  6247 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:40.188494  6385 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:23:40.231459 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 58347.4us
I20260709 12:23:40.171770  5846 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.025s
I20260709 12:23:40.237742  5151 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:40.261925  5151 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 19.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:40.264616 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 77568us
I20260709 12:23:40.265544 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:40.267112 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:40.270823 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:40.274053 32372 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:40.209709  6222 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:40.234277  6391 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:40.274190  6303 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
W20260709 12:23:40.305984 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 64482.7us
I20260709 12:23:40.317554  6091 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:40.318089  6091 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:40.319047  6091 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:40.320705  6091 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:40.286000  6251 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:40.259680  6247 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:40.257687  6383 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:40.304761  6385 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:23:40.334215  6395 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 12:23:40.336236  6391 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:23:40.360671 32376 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 999) took 2201 ms (client timeout 1640 ms). Trace:
I20260709 12:23:40.361018  6391 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:40.361555  6391 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:40.365650 31670 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2276) took 1600 ms (client timeout 1858 ms). Trace:
I20260709 12:23:40.366246  6385 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:40.373313  6385 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:40.335533  6303 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:40.355310 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:40.375149 31819 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:40.290577  6281 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:40.279167  6359 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:40.359331  5759 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:40.296391  6222 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:40.366166 31670 rpcz_store.cc:269] 0709 12:23:38.764945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.094813 (+329868us) service_pool.cc:224] Handling call
0709 12:23:40.365424 (+1270611us) spinlock_profiling.cc:243] Waited 26.1 ms on lock 0x7b54002553d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:40.365586 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":60143872}
I20260709 12:23:40.368942  6345 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:40.340822  6251 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:40.371554  6356 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:40.371656  6247 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:40.361130 32376 rpcz_store.cc:269] 0709 12:23:38.159668 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.159837 (+   169us) service_pool.cc:224] Handling call
0709 12:23:40.360455 (+2200618us) spinlock_profiling.cc:243] Waited 1.76 s on lock 0x7b5400427f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:40.360567 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4037822208}
I20260709 12:23:40.377125  6328 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:40.392563 31751 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:40.379082  6359 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:40.405874 32371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 994) took 2272 ms (client timeout 1629 ms). Trace:
W20260709 12:23:40.406519 32371 rpcz_store.cc:269] 0709 12:23:38.133761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.135149 (+  1388us) service_pool.cc:224] Handling call
0709 12:23:40.405527 (+2270378us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b5400646c54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:40.405747 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2412627712}
I20260709 12:23:40.378954  6303 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 15.12, Last appended by leader: 12, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:40.380502 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:40.411880 32378 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 19: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:40.380623  5880 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:40.414800  5880 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:40.383608 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.392336 31819 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:23:40.383991  5759 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:40.400146 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 133102us
I20260709 12:23:40.343641  6337 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.045s
I20260709 12:23:40.423058 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:40.427889 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:40.433626 31819 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:23:40.404764  6342 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:40.410620  6392 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 12:23:40.438220 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:40.444504 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:40.444585 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:40.418339  6252 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:40.453532 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:40.393503  6356 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:40.461117 31642 leader_election.cc:400] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
I20260709 12:23:40.463398 31642 leader_election.cc:403] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:23:40.464385 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:40.465021 31750 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:40.409103  6328 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:40.439989  6352 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:40.464668 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:40.452553  6247 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:23:40.477664 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:40.461658  6092 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:23:40.494803 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.440229  6316 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:23:40.461709  6405 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:40.472515  5844 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:40.472685  6333 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:40.481011  6331 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.503s	user 0.011s	sys 0.003s
I20260709 12:23:40.479596  6276 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.016s
I20260709 12:23:40.480880  6292 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:40.442967  6342 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:40.477684  5759 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:40.686911 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 251461us
I20260709 12:23:40.494926  6352 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:40.477279  6218 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.123s
I20260709 12:23:40.510094  6405 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:40.600374  6401 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
W20260709 12:23:40.601636  6398 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.307s	user 0.000s	sys 0.001s
I20260709 12:23:40.511776  6162 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:40.517952  5844 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:40.455838  6404 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.063s
I20260709 12:23:40.610858 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1082) took 1333 ms. Trace:
I20260709 12:23:40.811899 31678 rpcz_store.cc:276] 0709 12:23:39.276960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.848551 (+571591us) service_pool.cc:224] Handling call
0709 12:23:39.848976 (+   425us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:39.865380 (+ 16404us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:39.865442 (+    62us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:39.870369 (+  4927us) log.cc:844] Serialized 26 byte log entry
0709 12:23:40.056668 (+186299us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:40.056712 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.056782 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:40.608699 (+551917us) raft_consensus.cc:1685] finished
0709 12:23:40.608729 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:40.610770 (+  2041us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15908,"raft.queue_time_us":17099,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":1150,"thread_start_us":155091,"threads_started":1,"wal-append.queue_time_us":286489}
I20260709 12:23:40.621781  6292 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:40.625335 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2444) took 1089 ms. Trace:
W20260709 12:23:40.599404  6407 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
W20260709 12:23:40.709076 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.472966  6287 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.053s
I20260709 12:23:40.522759  6410 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:40.467957  6252 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:40.478420  6234 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.146s
W20260709 12:23:40.799995 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 355615us
I20260709 12:23:40.610286  5880 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.691346  6276 consensus_queue.cc:1243] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 21.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:23:40.820457 31753 rpcz_store.cc:276] 0709 12:23:39.536078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.553447 (+ 17369us) service_pool.cc:224] Handling call
0709 12:23:40.625052 (+1071605us) spinlock_profiling.cc:243] Waited 116 ms on lock 0x7b54000b95d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:40.625237 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":267113344}
I20260709 12:23:40.619313 31750 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:40.800194  6385 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:40.796802  6405 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:40.857052 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:40.891991 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48331.4us
I20260709 12:23:40.815356  6222 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:40.826371  6292 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:40.856045  5151 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:40.813812  5844 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:40.805274  6409 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.808037  6244 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:40.829560  5759 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:40.818423  6162 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:40.902259  5151 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:40.917107 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:40.908111  6276 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:40.855751  6410 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:40.862671  6345 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:40.941272  6276 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:40.887140  6404 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:40.930961  6091 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:40.861948  5880 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:40.983916 31674 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: []
W20260709 12:23:40.985050 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:40.985762 31826 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3034) took 2167 ms (client timeout 1804 ms). Trace:
W20260709 12:23:40.986851 31826 rpcz_store.cc:269] 0709 12:23:38.818470 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.845633 (+ 27163us) service_pool.cc:224] Handling call
0709 12:23:40.985469 (+2139836us) spinlock_profiling.cc:243] Waited 484 ms on lock 0x7b54000fad54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:40.985667 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1112808320}
I20260709 12:23:40.921595  6244 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:40.991237  5151 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:40.856002 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:40.992837 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 136910us
I20260709 12:23:40.939386  6410 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:41.028679 31671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3165) took 2924 ms (client timeout 1759 ms). Trace:
W20260709 12:23:41.029357 31671 rpcz_store.cc:269] 0709 12:23:38.059508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.059696 (+   188us) service_pool.cc:224] Handling call
0709 12:23:40.983320 (+2923624us) spinlock_profiling.cc:243] Waited 1.44 s on lock 0x7b54003917d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:40.983530 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3308629760}
I20260709 12:23:40.943756  6356 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:41.030536 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.009s	sys 0.019s
W20260709 12:23:41.030984 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.009s	sys 0.019s
W20260709 12:23:41.031334 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 553773us
W20260709 12:23:41.045324 32372 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.029s	sys 0.018s
W20260709 12:23:41.046149 32372 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.532s	user 0.029s	sys 0.018s
I20260709 12:23:40.942119  6404 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:40.945369  6091 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:41.087358 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 399628us
I20260709 12:23:41.003858  6385 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:41.103645 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:41.105396 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:41.106428 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:41.032140 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:41.062340  6287 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.075037  6356 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:41.076403  6413 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.158s	user 0.000s	sys 0.000s
I20260709 12:23:41.077132  6162 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:41.079236  6417 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.298s	user 0.001s	sys 0.000s
I20260709 12:23:41.081004  6342 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:40.942297  6276 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:41.121183 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1168) took 2234 ms. Trace:
I20260709 12:23:40.937067  6352 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:40.925247  6409 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:41.137749 32374 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.755s	user 0.007s	sys 0.028s
W20260709 12:23:41.146171 32374 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.763s	user 0.008s	sys 0.028s
W20260709 12:23:41.118952  6418 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.111s	user 0.001s	sys 0.000s
I20260709 12:23:41.049705  6328 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:41.141528 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 109501us
I20260709 12:23:41.142910 32377 rpcz_store.cc:276] 0709 12:23:38.868323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:38.868561 (+   238us) service_pool.cc:224] Handling call
0709 12:23:38.905804 (+ 37243us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:39.729114 (+823310us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.101969 (+1372855us) spinlock_profiling.cc:243] Waited 817 ms on lock 0x7b54004cfed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:41.102164 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1880215424}
I20260709 12:23:41.150229 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:41.153976 31747 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 18: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:41.154714 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:41.121726  6133 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.163983 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3075) took 1198 ms. Trace:
I20260709 12:23:41.164417 31747 rpcz_store.cc:276] 0709 12:23:39.965950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.976753 (+ 10803us) service_pool.cc:224] Handling call
0709 12:23:41.150076 (+1173323us) spinlock_profiling.cc:243] Waited 7.93 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:23:41.163908 (+ 13832us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":18232064}
W20260709 12:23:41.163964 31790 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.542s	user 0.017s	sys 0.044s
W20260709 12:23:41.165740 31790 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.544s	user 0.017s	sys 0.044s
I20260709 12:23:41.121788  6331 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.162041 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:41.167375  6331 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:41.168495  6337 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:41.168951  6337 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.169382  6337 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:41.171358 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.029s	sys 0.033s
W20260709 12:23:41.171809 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.651s	user 0.029s	sys 0.033s
I20260709 12:23:41.167881 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:41.170230  6331 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:41.179081  6331 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.179798  6331 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:41.180622  6437 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:41.181332  6437 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.181900  6437 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:41.182439  6385 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:41.183837  6385 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:41.187348  6337 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:41.188709  6434 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.197615  6281 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:41.131035  6421 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.081s
W20260709 12:23:41.199079  6412 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.147s	user 0.001s	sys 0.000s
I20260709 12:23:41.121526  6247 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 13.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.045s
W20260709 12:23:41.202903  6406 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.868s	user 0.007s	sys 0.008s
I20260709 12:23:41.202533  5844 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:41.207535  6259 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:23:41.215963 32342 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 2.148s (SENT) [suppressed 6 similar messages]
I20260709 12:23:41.151101  6251 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:41.147671  6422 raft_consensus.cc:2764] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election decision vote started in defunct term 19: won
I20260709 12:23:41.238705 31826 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:23:41.163636  6345 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:41.267299 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1200) took 1501 ms. Trace:
I20260709 12:23:41.141353  6287 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:41.144220  6356 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:41.190479  6303 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:41.283825 31821 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 12:23:41.261294 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 154909us
I20260709 12:23:41.296099 31822 rpcz_store.cc:276] 0709 12:23:39.766169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.771144 (+  4975us) service_pool.cc:224] Handling call
0709 12:23:39.810151 (+ 39007us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:39.810493 (+   342us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:39.827194 (+ 16701us) log.cc:844] Serialized 14 byte log entry
0709 12:23:40.362212 (+535018us) spinlock_profiling.cc:243] Waited 1.78 ms on lock 0x7b18004cc900. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:23:40.453158 (+ 90946us) log.cc:844] Serialized 14 byte log entry
0709 12:23:41.204767 (+751609us) spinlock_profiling.cc:243] Waited 2.24 ms on lock 0x7b18004cc900. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:23:41.266211 (+ 61444us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:41.266293 (+    82us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:41.266361 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.267038 (+   677us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.267215 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7588,"prepare.queue_time_us":16496,"prepare.run_cpu_time_us":102,"prepare.run_wall_time_us":3208,"raft.queue_time_us":183468,"raft.run_cpu_time_us":384,"raft.run_wall_time_us":189154,"spinlock_wait_cycles":58835456,"thread_start_us":138406,"threads_started":2,"wal-append.queue_time_us":860433}
I20260709 12:23:41.295951  6345 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:41.132268  6244 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:41.191624  6133 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:41.284373  6199 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:41.364143 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:41.368659 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 200858us
I20260709 12:23:41.368844  6258 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:23:41.209774  6292 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:41.371057 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:41.290854  6259 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:41.382210 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:41.370570  6287 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:41.361347  6345 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:41.445165 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:41.434409  6292 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:41.412264  6337 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:41.263051  6434 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:41.430850  6303 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:41.476408 31745 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:23:41.413851  6199 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:41.426288 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2290) took 2566 ms. Trace:
I20260709 12:23:41.523792 31674 rpcz_store.cc:276] 0709 12:23:38.860362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.744143 (+883781us) service_pool.cc:224] Handling call
0709 12:23:40.983558 (+1239415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.984691 (+  1133us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.425962 (+441271us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b54003917d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:41.426190 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2849365248}
I20260709 12:23:41.432782  6375 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:41.546006  6375 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:41.522099  6437 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:41.547288  6333 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.938s	user 0.012s	sys 0.000s
I20260709 12:23:41.547113  6375 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:41.548661  6375 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:41.548967  6437 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:41.424124  6400 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.095s	user 0.000s	sys 0.013s
W20260709 12:23:41.439733  6415 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.617s	user 0.012s	sys 0.005s
I20260709 12:23:41.300997  6252 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:41.210474  6410 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:41.407670  6439 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.441865  6247 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:41.424562  5844 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:41.415467  6091 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:41.441030  5759 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.507s	user 0.009s	sys 0.000s
I20260709 12:23:41.443403  6259 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:41.407053  6438 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:41.432360  6133 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:41.428257  6391 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.066s	user 0.007s	sys 0.004s
W20260709 12:23:41.484810 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 102697us
I20260709 12:23:41.598558 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:41.444231  6423 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.508s	user 0.000s	sys 0.004s
W20260709 12:23:41.485826  6406 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.106s	user 0.000s	sys 0.000s
I20260709 12:23:41.521485 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3083) took 1547 ms. Trace:
W20260709 12:23:41.432081  6399 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.103s	user 0.000s	sys 0.012s
I20260709 12:23:41.547807  6333 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
I20260709 12:23:41.307049  6331 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:41.552399 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:41.667987 31670 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:41.553267 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:41.553580 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:41.669376 31673 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:41.486655  5870 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:41.515978  5880 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:41.560571 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:41.561565 31678 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 19 index: 15. (index mismatch)
I20260709 12:23:41.564152 31677 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:41.565683 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:41.520123  6218 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:41.523486  6303 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:41.673853 31671 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:23:41.585067 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:41.675280 31676 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:41.587756 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:41.677305 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:41.677999 31827 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:41.679414 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:41.680183 31827 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:41.681250 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:41.682160 31827 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
W20260709 12:23:41.613085 31643 connection.cc:256] RPC call timeout handler was delayed by -1.9323754280003413s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2109s for a response
W20260709 12:23:41.615372 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 399350us
I20260709 12:23:41.622370 31821 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:41.622781 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:41.686800 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:41.623431 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:41.620282 31745 rpcz_store.cc:276] 0709 12:23:39.973698 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.978806 (+  5108us) service_pool.cc:224] Handling call
0709 12:23:41.475639 (+1496833us) spinlock_profiling.cc:243] Waited 6.92 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:23:41.475997 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:41.521166 (+ 45169us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.521424 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15926528}
I20260709 12:23:41.588488 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:41.688526 31820 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:41.588954 31823 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:41.683452 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:41.678310 31671 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:41.693703 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:41.694772 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:41.584282  6247 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:41.599833  6410 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:41.700430 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:41.703020 31827 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:23:41.708215 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:41.725586 31672 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:41.717494 31675 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:41.718662 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:41.709707 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:41.727851 31821 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:41.670943  6259 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:41.713611 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:41.720937 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3081) took 1747 ms. Trace:
I20260709 12:23:41.625607  6404 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:41.580801  6439 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:41.673367  5870 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:41.729512 31749 rpcz_store.cc:276] 0709 12:23:39.973158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.973885 (+   727us) service_pool.cc:224] Handling call
0709 12:23:39.984424 (+ 10539us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:40.049388 (+ 64964us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:40.049424 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:40.060115 (+ 10691us) log.cc:844] Serialized 26 byte log entry
0709 12:23:40.169378 (+109263us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:40.169460 (+    82us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.169571 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:40.323306 (+153735us) spinlock_profiling.cc:243] Waited 4.86 ms on lock 0x7b4800232758. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:23:41.423259 (+1099953us) spinlock_profiling.cc:243] Waited 2.21 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:41.720582 (+297323us) raft_consensus.cc:1685] finished
0709 12:23:41.720621 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.720844 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79694,"raft.queue_time_us":45948,"raft.run_cpu_time_us":2459,"raft.run_wall_time_us":291467,"spinlock_wait_cycles":16267648,"thread_start_us":58318,"threads_started":1,"wal-append.queue_time_us":157346}
I20260709 12:23:41.699594  6234 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.619433  5844 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.099s
I20260709 12:23:41.719403  6331 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.710382  6278 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.703624  6420 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.621433  6391 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:41.694064  6218 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:41.728431  6435 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.766976  6162 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.741740  5880 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.738421  5870 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:41.718727  6342 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.050s
W20260709 12:23:41.753576 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 49300.2us
W20260709 12:23:41.751731  6405 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.848s	user 0.010s	sys 0.002s
I20260709 12:23:41.793699 31821 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:41.738462  6333 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:41.796087 31672 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:23:41.832835 31824 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:41.800945 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3080) took 1833 ms. Trace:
I20260709 12:23:41.789367  6303 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:41.797489  6359 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.388s	user 0.016s	sys 0.000s
I20260709 12:23:41.791446  5759 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:41.753034  6446 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:23:41.926332 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 212907us
I20260709 12:23:41.833405 31746 rpcz_store.cc:276] 0709 12:23:39.967344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.972623 (+  5279us) service_pool.cc:224] Handling call
0709 12:23:39.984409 (+ 11786us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:40.054841 (+ 70432us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:40.054870 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:40.055372 (+   502us) log.cc:844] Serialized 26 byte log entry
0709 12:23:40.189055 (+133683us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:40.189116 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:40.189216 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:41.431233 (+1242017us) spinlock_profiling.cc:243] Waited 4.53 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:41.789913 (+358680us) raft_consensus.cc:1685] finished
0709 12:23:41.789939 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:41.800878 (+ 10939us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":72382,"raft.queue_time_us":71618,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":13970,"spinlock_wait_cycles":10469632,"thread_start_us":98029,"threads_started":1,"wal-append.queue_time_us":184451}
W20260709 12:23:41.896170 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 62092.6us
I20260709 12:23:41.796447  6251 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:41.761004  6234 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:41.768076  6252 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:23:41.791733  6331 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:41.965572 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:41.969031 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:41.799751  6420 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:41.826236  6162 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:42.028319 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62860.1us
I20260709 12:23:41.797036  6278 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:41.803930  6435 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:41.831748  5880 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:41.844250  6405 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:42.099718  6451 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:23:42.102273  6405 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:41.888309  6247 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.067556  6234 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:42.075356  6445 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.364s	user 0.001s	sys 0.000s
W20260709 12:23:42.076563  6450 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.174s	user 0.000s	sys 0.000s
W20260709 12:23:42.077672  6448 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.184s	user 0.000s	sys 0.000s
I20260709 12:23:42.033720  6270 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.078059  6455 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:23:42.079449  6449 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.183s	user 0.000s	sys 0.000s
W20260709 12:23:42.082790 31703 heartbeater.cc:646] Failed to heartbeat to 127.29.201.254:37495 (4 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.29.201.254:37495 timed out after 15.000s (SENT)
I20260709 12:23:42.085105  6331 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.047240  6303 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:41.970768  6434 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:42.087148  6342 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:42.089519  6218 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.092587  6278 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:42.093717  6391 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:41.971633  6446 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:41.924618  6333 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:42.055099 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:42.106909 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.107614 31670 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:42.112841  6162 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.039548  6292 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.144716  6342 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:42.037068  6251 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.141814  6270 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:42.125893  6252 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:23:42.165654 31675 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: [17.13-17.13]
I20260709 12:23:42.039192  6133 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:23:42.170849 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 43500.9us
W20260709 12:23:42.164454 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.138353  6276 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.018s
I20260709 12:23:42.205137  6252 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.191560  6133 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:42.145622  6331 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.022s
I20260709 12:23:42.139981  6455 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:42.149134  6434 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.027s
I20260709 12:23:41.971267  5759 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:42.223344  6133 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:42.282902 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:42.297740 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:42.293339 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:42.236637  6410 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.531s	user 0.011s	sys 0.000s
W20260709 12:23:42.218568  6345 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.664s	user 0.014s	sys 0.000s
I20260709 12:23:42.299280  6385 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:23:42.275012  6454 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.117s	user 0.001s	sys 0.000s
I20260709 12:23:42.306191 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:42.307243 31670 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 9. Preceding OpId from leader: term: 15 index: 10. (index mismatch)
I20260709 12:23:42.274952  6105 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:42.310130 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:42.312561 31823 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:23:42.309847  6105 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:42.316779  6460 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:42.317445  6105 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:42.318542  6105 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:42.323401  6453 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.324030  6453 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.326877  6460 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:42.327656 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:42.328359 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.327888  6460 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.329563 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:42.330314 32371 raft_consensus.cc:2368] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
I20260709 12:23:42.331136 32377 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:42.297328  6391 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:42.330016  6234 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:42.326414  6453 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.333781 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:42.234524  6439 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:42.177120  6292 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:42.155663  6375 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.607s	user 0.010s	sys 0.004s
I20260709 12:23:42.380209 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:42.177857  6342 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:42.382699 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.229244  6252 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.383450 31821 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:42.386698 31821 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:42.396471 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.397918 32371 raft_consensus.cc:2410] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:23:42.341343  6410 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.406859 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.408895 32377 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:42.410121  6294 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.410673  6294 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.412874  6294 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:42.387486 31821 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.416560 31821 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:42.426031 31821 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 22 index: 11. (term mismatch)
I20260709 12:23:42.426671 31821 pending_rounds.cc:85] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 10
I20260709 12:23:42.427353 31821 pending_rounds.cc:107] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 21.11
I20260709 12:23:42.428306 31821 raft_consensus.cc:2889] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:42.427783 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:42.430249 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:42.430915 31670 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:42.309361  6345 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.437089 31670 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:23:42.437955 31670 raft_consensus.cc:606] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:42.313975  6385 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.389017  6439 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:42.442411 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:42.445036 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 513200us
I20260709 12:23:42.445147 31820 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:23:42.445299 31821 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:42.446182 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.438496 31670 raft_consensus.cc:614] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:23:42.416072  6259 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.677s	user 0.011s	sys 0.003s
I20260709 12:23:42.449033  6352 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:23:42.437927  6420 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.452507  6345 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:42.453286  6456 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:23:42.453311  6294 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:42.454097  6435 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:42.454914  6333 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.454169  6294 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:42.455694  5759 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:42.456440  6251 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.457301  6292 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:42.457522  6294 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:42.458750  6402 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 2.103s	user 0.014s	sys 0.003s
I20260709 12:23:42.459139  6385 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:42.427070  6410 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.460708 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3049) took 2163 ms. Trace:
I20260709 12:23:42.461503  6410 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:42.449067 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:42.359118  6391 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:42.448174 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.454452  6439 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:42.473875 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:42.474090 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.474808 31821 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:42.474952 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:42.446867 32377 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:42.461072 31825 rpcz_store.cc:276] 0709 12:23:40.297555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.297770 (+   215us) service_pool.cc:224] Handling call
0709 12:23:40.298108 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.310885 (+ 12777us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:40.342566 (+ 31681us) log.cc:844] Serialized 14 byte log entry
0709 12:23:40.440157 (+ 97591us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:40.440215 (+    58us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:40.440266 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.460478 (+2020212us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b54003ca5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:42.460526 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:42.460650 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":32353,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":1787,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":20623,"spinlock_wait_cycles":28613504,"thread_start_us":97222,"threads_started":1,"wal-append.queue_time_us":488870,"wal-append.run_cpu_time_us":2533,"wal-append.run_wall_time_us":1310620}
I20260709 12:23:42.455754  6446 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:42.482020 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:42.483212 32371 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:42.482283 31673 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:42.462605  6455 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.479018  6234 raft_consensus.cc:2764] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:23:42.485168  6463 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:42.477864  6410 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.480952  6420 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:42.489673 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.497615 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.497766 31825 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:23:42.498368 32371 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:42.488200 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:42.487037 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 126126us
I20260709 12:23:42.453511  6133 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:42.495340 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.487795 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:42.507009 31670 raft_consensus.cc:2376] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20. Re-sending same reply.
I20260709 12:23:42.508487 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.512384 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:42.512530 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:42.513103 31821 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:42.513528 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.489105  6385 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:42.514271 32372 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:42.514714 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.517135 32374 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:23:42.515744 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.517783 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:42.518285 32373 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:42.518368 31670 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:23:42.520273 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:42.494781  6391 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:42.531217  5846 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:42.531764  5846 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:42.532132 31746 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:23:42.532735  5846 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 17.13, Last appended by leader: 13, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:42.533707 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.534050  5846 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:42.529290 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:42.535355 31747 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: []
W20260709 12:23:42.536036  6414 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.648s	user 0.011s	sys 0.021s
I20260709 12:23:42.536099  6133 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.536455 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.537140 32376 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:42.538028 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.540197  6345 raft_consensus.cc:2764] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:23:42.540951  6345 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:23:42.508030  6464 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.546265  6464 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.547469 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:42.547852 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:42.548493 31745 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:42.548599  6464 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.550019  6439 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:42.499507  6410 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.550765 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:42.550982 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:42.551599 31750 raft_consensus.cc:2376] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20. Re-sending same reply.
I20260709 12:23:42.552317  6410 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:42.552325 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:42.553337 31744 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:42.553575 31753 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:23:42.507879  6465 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:42.498600  6294 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.497386  6420 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:42.527613 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:42.528079 31821 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:42.529358 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:42.547518 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.564577 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.564749 31820 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:23:42.555083 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.565271 31821 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:42.556813 32373 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:42.529722 31643 leader_election.cc:400] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
I20260709 12:23:42.563549 31670 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:23:42.567325 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:42.567659 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:42.569052 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:42.569262 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:42.570775 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:42.571775  6251 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:42.572315  6251 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.572763  6251 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:42.570062 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:42.581383 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.582103 31821 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:42.578163 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:42.583082 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:42.583684 31821 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:42.583827 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 67590.4us
I20260709 12:23:42.588886 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:42.601047 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:42.604676 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:42.609916 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.610618 31820 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:42.555045  6385 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:42.607065  6452 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:42.619135  6452 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:42.624728  5870 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.733s	user 0.011s	sys 0.000s
I20260709 12:23:42.625226  5870 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.588218  6439 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.626997  6439 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:42.565384  6247 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 21.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.064s
I20260709 12:23:42.632805 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:42.633551 31671 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:42.635893  5870 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.636125 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:42.637661 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:42.640992 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.641173  6383 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:42.641628  6247 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:42.642033  6247 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:42.642853  6247 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.644258  6247 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:42.645165 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1090) took 1092 ms. Trace:
I20260709 12:23:42.565896  6420 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:42.581261  6455 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:23:42.649672 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 93053us
I20260709 12:23:42.579041  6459 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:42.645592 31677 rpcz_store.cc:276] 0709 12:23:41.552362 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.552491 (+   129us) service_pool.cc:224] Handling call
0709 12:23:41.552782 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:41.680339 (+127557us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.644879 (+964540us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b540022b854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:42.645096 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":25465088}
I20260709 12:23:42.653607 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:42.655025 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:42.634390 31670 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:42.588441  6133 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:42.617584  6252 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:42.659471 31717 connection.cc:256] RPC call timeout handler was delayed by -2.2929189259998566s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21760000000000002s for a response
I20260709 12:23:42.659641 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:42.562389  6259 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.085s
I20260709 12:23:42.641253  6391 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:42.608449  6352 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:42.657326 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:42.684478 31670 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:42.657729 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.685326 31820 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:42.650791 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:42.613613  5759 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:42.649068 31821 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:42.689625 31671 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:42.690411 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.692051 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:42.692850 31821 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:42.606765  6223 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.060s
I20260709 12:23:42.581113  6222 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.076s
I20260709 12:23:42.566092  6294 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.658769  6133 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:42.711845 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.662847  6455 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:42.662963  5846 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.662868  6278 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.687796  6352 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.689440  5759 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.726466  6292 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.734828  6133 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:42.663820  6439 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:42.754173 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.712787 31820 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
W20260709 12:23:42.715080 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.756083 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:42.718730  6252 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.721135 31821 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:42.698472  6460 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:42.756400 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:42.723225 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.762090 32372 raft_consensus.cc:2368] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23.
I20260709 12:23:42.725428  6294 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:42.672729  6342 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:42.740041 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.748832 31745 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:23:42.743821  6352 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:42.757651 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.775024 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:42.744796 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 104170us
I20260709 12:23:42.758244 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:42.705588 31717 connection.cc:256] RPC call timeout handler was delayed by -1.8388269669999318s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1804s for a response
I20260709 12:23:42.745428  5759 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:42.756479 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.768778 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:42.768262  6234 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:23:42.809118 31717 connection.cc:256] RPC call timeout handler was delayed by -1.7056269669997164s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1952s for a response
I20260709 12:23:42.688090  6385 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:42.766690  6276 raft_consensus.cc:2764] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election decision vote started in defunct term 17: won
W20260709 12:23:42.813477 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55510.9us
I20260709 12:23:42.859506 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.743146  6278 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:42.849822 31717 connection.cc:256] RPC call timeout handler was delayed by -1.5868158960001892s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1504s for a response
I20260709 12:23:42.814079 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:42.865257 31717 connection.cc:256] RPC call timeout handler was delayed by -1.5202269669998714s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2158s for a response
W20260709 12:23:42.819253 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.851890 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.740469  5846 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.860584 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:23:42.865758 31717 connection.cc:256] RPC call timeout handler was delayed by -1.4878158960000292s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16140000000000002s for a response
I20260709 12:23:42.754328  6292 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:42.869145 31717 connection.cc:256] RPC call timeout handler was delayed by -1.4410269669997433s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22460000000000002s for a response
W20260709 12:23:42.869570 31717 connection.cc:256] RPC call timeout handler was delayed by -1.2406286380000893s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21040000000000003s for a response
W20260709 12:23:42.869863 31717 connection.cc:256] RPC call timeout handler was delayed by -1.2266529309999896s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1759s for a response
W20260709 12:23:42.870412 31717 connection.cc:256] RPC call timeout handler was delayed by -1.1794286379999903s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21720000000000003s for a response
I20260709 12:23:42.769189  6259 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.096s
I20260709 12:23:42.765892  6252 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:42.870853 31717 connection.cc:256] RPC call timeout handler was delayed by -1.1355173259998992s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1787s for a response
W20260709 12:23:42.876546 31717 connection.cc:256] RPC call timeout handler was delayed by -1.0914173259998279s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1836s for a response
I20260709 12:23:42.851382  6223 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
W20260709 12:23:42.876981 31717 connection.cc:256] RPC call timeout handler was delayed by -1.0869173259998206s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1841s for a response
I20260709 12:23:42.853053  6234 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.775913  6446 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.090s
W20260709 12:23:42.877386 31717 connection.cc:256] RPC call timeout handler was delayed by -1.0612158960002489s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2088s for a response
I20260709 12:23:42.769670  6460 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:42.852886  6331 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.023s
I20260709 12:23:42.864482  6391 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.883984 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.887045 31821 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
W20260709 12:23:42.887131 31717 connection.cc:256] RPC call timeout handler was delayed by -1.0536173259997668s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1878s for a response
W20260709 12:23:42.887960 31717 connection.cc:256] RPC call timeout handler was delayed by -1.0036158960001558s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21520000000000003s for a response
W20260709 12:23:42.889536  6472 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.074s	user 0.000s	sys 0.000s
I20260709 12:23:42.889346 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:42.890247 31821 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:42.892320  6460 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:42.893173  6105 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.574s	user 0.009s	sys 0.001s
I20260709 12:23:42.895253  5846 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.878114  6214 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:42.896162  6214 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:42.896649  6214 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:23:42.897369  6467 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.194s	user 0.001s	sys 0.000s
I20260709 12:23:42.898916 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:42.898933 31745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:42.876533  6333 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:42.900305 31745 raft_consensus.cc:606] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:42.901731 31821 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:42.928442 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:42.929786 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.900059 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:42.881477  6234 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:42.937661  6444 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.237s	user 0.001s	sys 0.000s
I20260709 12:23:42.863785  6455 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:42.943598 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1014) took 2703 ms. Trace:
I20260709 12:23:42.944089 32369 rpcz_store.cc:276] 0709 12:23:40.240386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:40.240589 (+   203us) service_pool.cc:224] Handling call
0709 12:23:40.241020 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:40.409729 (+168709us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:23:40.651005 (+241276us) log.cc:844] Serialized 14 byte log entry
0709 12:23:40.946123 (+295118us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:40.946177 (+    54us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:23:40.946221 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.943329 (+1997108us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b54006dd654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:42.943388 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:42.943536 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":477506,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":112,"raft.run_cpu_time_us":242,"raft.run_wall_time_us":53089,"spinlock_wait_cycles":387256192,"thread_start_us":515487,"threads_started":2,"wal-append.queue_time_us":467208,"wal-append.run_cpu_time_us":1442,"wal-append.run_wall_time_us":1086767}
I20260709 12:23:42.947352 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3142) took 1247 ms. Trace:
I20260709 12:23:42.947826 31824 rpcz_store.cc:276] 0709 12:23:41.707607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.710972 (+  3365us) service_pool.cc:224] Handling call
0709 12:23:41.740709 (+ 29737us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:41.833301 (+ 92592us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.947084 (+1113783us) spinlock_profiling.cc:243] Waited 91.9 ms on lock 0x7b54001c8ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:42.947287 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":211322624}
I20260709 12:23:42.949172 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:42.949846  6252 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.950031  6287 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:42.922520  6303 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.760s	user 0.013s	sys 0.000s
I20260709 12:23:42.903021  6391 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.815050  6453 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.071s
I20260709 12:23:42.900056  6223 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:42.950718  6287 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:42.961756 31827 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 9. (term mismatch)
I20260709 12:23:42.982568 31827 pending_rounds.cc:85] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 8
I20260709 12:23:42.982934 31827 pending_rounds.cc:107] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 17.9
I20260709 12:23:42.983350 31827 raft_consensus.cc:2889] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:42.967305 31821 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
W20260709 12:23:42.954449 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:42.922971  6333 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.949877 31824 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:42.976689 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:42.989642 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3201) took 1392 ms. Trace:
I20260709 12:23:42.991044 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1208) took 1747 ms. Trace:
I20260709 12:23:42.991689 31818 rpcz_store.cc:276] 0709 12:23:41.243287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.243407 (+   120us) service_pool.cc:224] Handling call
0709 12:23:41.243751 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:41.367428 (+123677us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:41.367462 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:41.368047 (+   585us) log.cc:844] Serialized 26 byte log entry
0709 12:23:41.515105 (+147058us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:41.515157 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:41.515244 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:42.936921 (+1421677us) spinlock_profiling.cc:243] Waited 8.12 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:42.986361 (+ 49440us) raft_consensus.cc:1685] finished
0709 12:23:42.986400 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:42.990968 (+  4568us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":142590,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"spinlock_wait_cycles":18670976,"thread_start_us":269721,"threads_started":2,"wal-append.queue_time_us":208861}
I20260709 12:23:42.995105 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:42.978469 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44012.9us
I20260709 12:23:42.960448  6391 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:42.932045  5846 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:42.988459  6223 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.000799  6287 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:43.003556  6303 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:43.005517 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3199) took 1409 ms. Trace:
I20260709 12:23:42.963986  6278 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:42.990577  6333 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:42.965415  6292 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:43.016187 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:42.958458  6377 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:43.025242 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:43.021476  5846 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:42.960556  6438 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:43.034227 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 220207us
I20260709 12:23:43.034776 32374 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
W20260709 12:23:43.043336 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.035336 31826 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260709 12:23:43.037592 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.043736 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:43.045532 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:43.006110  5759 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.028587 31677 rpcz_store.cc:276] 0709 12:23:41.596226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.660969 (+1064743us) service_pool.cc:224] Handling call
0709 12:23:42.661268 (+   299us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:42.661756 (+   488us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:42.661783 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:42.674662 (+ 12879us) log.cc:844] Serialized 26 byte log entry
0709 12:23:42.752273 (+ 77611us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:42.752315 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.752385 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:43.005208 (+252823us) raft_consensus.cc:1685] finished
0709 12:23:43.005280 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:43.005445 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12041,"raft.queue_time_us":1570,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":16963,"thread_start_us":37635,"threads_started":1,"wal-append.queue_time_us":189491}
I20260709 12:23:43.049765 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:43.050177 31717 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:43.050420 31677 raft_consensus.cc:2376] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18. Re-sending same reply.
I20260709 12:23:43.051110  5151 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:43.051460 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.051709 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:43.020107  6469 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:43.052134 31677 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:43.056754 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.040123  6222 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:43.046747 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:43.038434 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 138463us
I20260709 12:23:43.052608 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:43.044441  6278 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:43.044808  5846 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:43.057885 31677 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:43.049563  6105 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:43.064730 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:43.091509 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:43.103852 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 404323us
I20260709 12:23:43.038015  6352 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.107847 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:43.089627  6251 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.516s	user 0.014s	sys 0.000s
I20260709 12:23:43.043040  6133 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.109231 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:43.052748  6420 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.066104 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:43.061463  6222 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:43.126530 32369 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:43.053790  6162 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.051s
W20260709 12:23:43.156796  6484 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:23:43.116503 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:43.158845 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.159961 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.160522 31677 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:43.161317 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:43.161846 31677 raft_consensus.cc:2376] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18. Re-sending same reply.
I20260709 12:23:43.053290  6438 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.096560  6105 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:43.095513  6453 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:43.126291  6251 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.132293  6133 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:43.132432  6352 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:43.132876 31670 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:43.089898  6337 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:23:43.117894  6485 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:43.128203  6460 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:43.112577 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:43.155171 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:43.155588 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:43.118932 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52920.3us
I20260709 12:23:43.181718 31753 raft_consensus.cc:2376] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23. Re-sending same reply.
I20260709 12:23:43.163081 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.110056  6278 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:43.096060  6487 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:43.107249  6342 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:43.164811  6105 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:43.153550  5846 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.170248 32377 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:43.172403 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:43.174934 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:43.175227  6251 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:43.169606  6453 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:43.133296  6292 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:43.185730 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:43.183609 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:43.187402 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:43.192564 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 80073.9us
I20260709 12:23:43.128618  6222 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:43.212984 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:43.215327 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:43.188812  6247 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.544s	user 0.010s	sys 0.002s
I20260709 12:23:43.198491 31746 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 19: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:43.198181 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:43.221061 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:43.186681  6460 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:43.206694 31745 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 18.
I20260709 12:23:43.189132  6294 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.210767 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:43.212494 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:43.228101 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:43.197348  6278 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:43.210752  6162 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:43.239918  6162 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.240401  6162 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:43.240447 31673 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3192) took 1646 ms (client timeout 1614 ms). Trace:
I20260709 12:23:43.241245  6435 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:43.241659  6435 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:43.217437  6385 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:43.242983  6385 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:43.243372  6385 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:43.235056  6287 raft_consensus.cc:2764] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:23:43.235935 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:43.197705  6487 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:43.236832  6337 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:43.238749  6464 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:43.248387 31746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:43.238855  6342 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:43.248992  5151 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:43.239970  6278 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:43.240938 31673 rpcz_store.cc:269] 0709 12:23:41.593916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.670795 (+ 76879us) service_pool.cc:224] Handling call
0709 12:23:43.240242 (+1569447us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b540036f6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:43.240373 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1734832512}
I20260709 12:23:43.249513  5151 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.242588  6435 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.230996 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:43.235832  6243 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:23:43.234411 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:43.253723 31751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2452) took 3656 ms (client timeout 2140 ms). Trace:
I20260709 12:23:43.234061  6294 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.197619  6489 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:43.257490 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:43.246325 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3136) took 1646 ms. Trace:
I20260709 12:23:43.211176  6292 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.260694 31747 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:43.248996 31746 raft_consensus.cc:606] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:43.227764  6460 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:43.250478 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.263029 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.266693  5846 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:43.266805  6294 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:43.233142  6453 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:43.252519  6464 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:43.289323  6464 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.274912 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.291769 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.252592  6345 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.219590  6251 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:43.185029  6485 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.255134 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:43.254124 31751 rpcz_store.cc:269] 0709 12:23:39.590381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:39.592222 (+  1841us) service_pool.cc:224] Handling call
0709 12:23:43.246794 (+3654572us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b54002963d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:43.246889 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1838172928}
I20260709 12:23:43.258329 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.258358 31823 rpcz_store.cc:276] 0709 12:23:41.599581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.690861 (+ 91280us) service_pool.cc:224] Handling call
0709 12:23:41.691301 (+   440us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:42.313155 (+621854us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.246093 (+932938us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b54002675d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:43.246267 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1427436416}
I20260709 12:23:43.259824 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.244562  6223 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:43.317615  5151 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:43.319509 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.319998 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.319028 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:43.324673 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67259.7us
I20260709 12:23:43.268004  6420 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:43.280547 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:43.328842 32369 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:43.247383  6133 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:43.294919 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.332970 31745 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:43.299855  5846 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:43.309139 31753 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:43.296851 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.313508 31744 raft_consensus.cc:2368] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23.
I20260709 12:23:43.337621 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.309618 31749 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:43.315737 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:43.300987 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3184) took 1745 ms. Trace:
I20260709 12:23:43.338835 31750 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:43.310727 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.342211 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:43.342864 31673 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:43.278154 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:43.268951  6292 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:43.344420 31823 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:43.346573 31823 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:23:43.320966 31752 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:23:43.260083  5949 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:43.325547 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:43.279676 31747 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:23:43.248670  6259 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.054s
I20260709 12:23:43.270694  5759 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:43.269229  6342 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:43.268591  6337 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:43.298285  6464 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:43.331000 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.298513  6435 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:43.292730  6489 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:43.335114 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 89620.7us
W20260709 12:23:43.322688 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 86829.8us
I20260709 12:23:43.316308  6247 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:43.351040 31746 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:43.385011 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:43.338719 31676 rpcz_store.cc:276] 0709 12:23:41.555657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.727287 (+171630us) service_pool.cc:224] Handling call
0709 12:23:41.728567 (+  1280us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:41.743790 (+ 15223us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:41.743826 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:41.744357 (+   531us) log.cc:844] Serialized 26 byte log entry
0709 12:23:42.966376 (+1222019us) spinlock_profiling.cc:243] Waited 2.16 ms on lock 0x7b18003ee3c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:23:43.248025 (+281649us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:43.248067 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.249444 (+  1377us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:43.292878 (+ 43434us) raft_consensus.cc:1685] finished
0709 12:23:43.292909 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:43.300914 (+  8005us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18237,"raft.queue_time_us":18364,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":15521,"spinlock_wait_cycles":4974336,"thread_start_us":281450,"threads_started":1,"wal-append.queue_time_us":1522279}
I20260709 12:23:43.357662 31744 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 16 index: 13. (index mismatch)
I20260709 12:23:43.358217 31747 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:23:43.359292 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.387284 31753 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:23:43.361088 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.362102 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.363088 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.390015 31752 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:43.383960 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.397935 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:43.398339 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.395043 31747 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:23:43.397710  6487 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.400595 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.401274 31744 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
W20260709 12:23:43.402596  6490 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.210s	user 0.001s	sys 0.000s
I20260709 12:23:43.404327  6498 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:43.408289  6470 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.408941  6470 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.351388  5846 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:43.412572  6485 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:43.413148  5846 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.350410  5949 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:43.414930  5949 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:43.354398  6420 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.065s
I20260709 12:23:43.419163 32376 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 15. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:23:43.420418 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:43.421547 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3196) took 1826 ms. Trace:
I20260709 12:23:43.397426 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:43.421940 31674 rpcz_store.cc:276] 0709 12:23:41.595246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.216790 (+621544us) service_pool.cc:224] Handling call
0709 12:23:42.217218 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:42.231473 (+ 14255us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:42.270034 (+ 38561us) log.cc:844] Serialized 14 byte log entry
0709 12:23:42.471367 (+201333us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:42.471430 (+    63us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:42.472185 (+   755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.421302 (+949117us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b540022a954. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:43.421363 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:43.421488 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":38345,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":69,"raft.queue_time_us":266,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":5652,"spinlock_wait_cycles":31985536,"thread_start_us":184434,"threads_started":1,"wal-append.queue_time_us":245191}
I20260709 12:23:43.395963 31752 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 22 index: 11. (index mismatch)
W20260709 12:23:43.427830 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.371083 31673 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:43.430369 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:43.300772  6345 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.409714  6105 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.436326 31751 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:23:43.440909 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.447113 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:43.447754 32374 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:43.447885 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:43.447726 32369 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:43.358186  6287 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:23:43.449893 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.453258 31749 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:43.453800 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.372324  6438 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:43.451156 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.472440 32369 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:43.478422 31750 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:43.443080  6092 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:43.484685  6092 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:43.487383  6270 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:43.489418 31751 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:43.446260  6133 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:43.491685 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:43.443506  6439 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:43.511071 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.518572 31676 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:23:43.518786 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 72582.3us
I20260709 12:23:43.444964  6247 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:43.510358  6214 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.599s	user 0.013s	sys 0.000s
I20260709 12:23:43.526228 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.530232 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:43.535935  6345 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:43.537189  6470 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:43.525524  6500 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:43.510449  6012 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.544246 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:43.534310  6391 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.556610  6356 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:43.559152 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.488642  6337 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.560694 31744 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:43.576272 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:43.559597 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.579448 31751 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:43.578466 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.583676 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:43.562537  6333 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.508s	user 0.007s	sys 0.004s
I20260709 12:23:43.583837 31746 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:43.589838 31747 raft_consensus.cc:606] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:43.587770 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:43.596429 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 52304.1us
I20260709 12:23:43.614516 31746 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 13 index: 7. Preceding OpId from leader: term: 18 index: 9. (index mismatch)
I20260709 12:23:43.520527  6439 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.619601  6505 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:43.620409 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.512741  6342 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.577934  6500 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.631656 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.708976 31822 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:43.634140  6247 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:43.715413 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:43.716110 31822 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:23:43.717672  6292 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.486375  6489 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.642983  6133 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.574762  6259 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.052s
I20260709 12:23:43.651400  6251 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:43.578984  6499 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.092s
I20260709 12:23:43.582794  6391 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.561764  6214 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.654444 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.730895 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.655735 31818 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:23:43.656189 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:43.656631 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3185) took 2100 ms. Trace:
I20260709 12:23:43.732532 31675 rpcz_store.cc:276] 0709 12:23:41.555936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.728005 (+172069us) service_pool.cc:224] Handling call
0709 12:23:42.165364 (+437359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:42.175889 (+ 10525us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.656412 (+1480523us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b54002e7a50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:43.656565 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1003209600}
I20260709 12:23:43.657234 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3200) took 2060 ms. Trace:
I20260709 12:23:43.734954 31670 rpcz_store.cc:276] 0709 12:23:41.596533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.688648 (+1092115us) service_pool.cc:224] Handling call
0709 12:23:42.696695 (+  8047us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.175931 (+479236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.656938 (+481007us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b540017cfd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:43.657162 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1002674944}
I20260709 12:23:43.657719 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.658666 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:43.736824 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78277.1us
I20260709 12:23:43.737350 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:43.738201 31820 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:43.581321  6012 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:43.631716  6492 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.297s	user 0.001s	sys 0.000s
W20260709 12:23:43.659838 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.744056 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:42.990506 31669 rpcz_store.cc:276] 0709 12:23:41.596844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.689035 (+1092191us) service_pool.cc:224] Handling call
0709 12:23:42.699734 (+ 10699us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:42.699959 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:42.738225 (+ 38266us) log.cc:844] Serialized 14 byte log entry
0709 12:23:42.989271 (+251046us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:42.989348 (+    77us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:42.989403 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:42.989480 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:42.989588 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12884,"prepare.queue_time_us":39865,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":7867,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":157,"thread_start_us":242494,"threads_started":1}
I20260709 12:23:43.748050 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.748219 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.748746 31821 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:23:43.663585 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1019) took 1161 ms. Trace:
I20260709 12:23:43.749723 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:43.750116 32377 rpcz_store.cc:276] 0709 12:23:42.501715 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.502811 (+  1096us) service_pool.cc:224] Handling call
0709 12:23:42.517015 (+ 14204us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.197523 (+680508us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.663342 (+465819us) spinlock_profiling.cc:243] Waited 652 ms on lock 0x7b54004bb254. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:43.663526 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1499550080}
I20260709 12:23:43.663482  6105 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:43.752152 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:43.600965  6333 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.676100  6222 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.759928 32372 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:43.683985 31792 leader_election.cc:400] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23.
I20260709 12:23:43.761809 31792 leader_election.cc:403] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:43.685633 31744 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 19 index: 14. (index mismatch)
I20260709 12:23:43.692852 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:43.589434  6356 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.694765 31676 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:43.729671  6251 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:43.734229 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.731019  6452 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:43.665246  6409 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:43.753711 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.758735  6012 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:43.611109  6162 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.724129  6505 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:43.726729 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.729947  6478 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.054s
I20260709 12:23:43.779547 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.802057 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.802595 31744 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
W20260709 12:23:43.781216 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 97383.3us
I20260709 12:23:43.805025 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:43.784277 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.802861 31676 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:43.786163  6247 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:23:43.810109 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1210) took 1285 ms. Trace:
I20260709 12:23:43.810246 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:43.810514 31826 rpcz_store.cc:276] 0709 12:23:42.524603 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.045111 (+520508us) service_pool.cc:224] Handling call
0709 12:23:43.045603 (+   492us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.045986 (+   383us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.046014 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.049682 (+  3668us) log.cc:844] Serialized 26 byte log entry
0709 12:23:43.228606 (+178924us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:43.228650 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.228717 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:43.809892 (+581175us) raft_consensus.cc:1685] finished
0709 12:23:43.809913 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:43.810055 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1057,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":7942,"thread_start_us":162559,"threads_started":1,"wal-append.queue_time_us":212904}
I20260709 12:23:43.812647 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:43.746228  6276 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:43.788870 31677 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:43.796527 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.779114 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:43.798247 31820 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:23:43.807721 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:43.752446 32373 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:43.809425 31753 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:43.817384 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.820106 31820 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:23:43.814262 31822 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:43.824352 31821 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 20: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:43.826026 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:43.826402 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.831477 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:43.824826 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:43.833269 31826 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:43.839980 31822 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 21: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:43.840399 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:43.840858 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.834663 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.842042 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.842312 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.842764 31822 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:43.843457 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.803767  6012 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.838038 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.844448 31820 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:43.845330 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 13 index: 7 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.846195 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.848721 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:43.835021 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3151) took 1294 ms. Trace:
I20260709 12:23:43.844185  6012 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.850479 31823 rpcz_store.cc:276] 0709 12:23:42.541051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.354196 (+813145us) service_pool.cc:224] Handling call
0709 12:23:43.354557 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.355050 (+   493us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.355079 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.392268 (+ 37189us) log.cc:844] Serialized 26 byte log entry
0709 12:23:43.542278 (+150010us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:43.558921 (+ 16643us) log.cc:844] Serialized 14 byte log entry
0709 12:23:43.834508 (+275587us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.834777 (+   269us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:43.834827 (+    50us) raft_consensus.cc:1685] finished
0709 12:23:43.834843 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:43.834962 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16439,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":16611,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":25023,"thread_start_us":349642,"threads_started":2,"wal-append.queue_time_us":212890}
I20260709 12:23:43.852294  6012 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:43.787287  5949 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:43.854264 31669 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
W20260709 12:23:43.833775  6303 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.799s	user 0.009s	sys 0.005s
I20260709 12:23:43.855315 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1101) took 1357 ms. Trace:
I20260709 12:23:43.862423 31676 rpcz_store.cc:276] 0709 12:23:42.498273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.824634 (+1326361us) service_pool.cc:224] Handling call
0709 12:23:43.854847 (+ 30213us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.855015 (+   168us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.855038 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:43.855076 (+    38us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:43.855107 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.855159 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:43.855260 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:43.856091 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.852597 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:43.847074 31820 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:23:43.830256  6105 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.809321  6409 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:43.785846  6356 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.859946 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1102) took 1356 ms. Trace:
I20260709 12:23:43.854595 31678 raft_consensus.cc:2410] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:23:43.858366 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1218) took 1197 ms. Trace:
I20260709 12:23:43.824082  5759 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:43.823908  6276 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.876143 31826 rpcz_store.cc:276] 0709 12:23:42.661077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.857530 (+1196453us) service_pool.cc:224] Handling call
0709 12:23:43.857832 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.857994 (+   162us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.858017 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:43.858058 (+    41us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:43.858096 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.858192 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:43.858308 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:43.843575 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:43.866695 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:43.871470 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:43.837387  6223 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:43.876822 31751 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:43.878829 31669 rpcz_store.cc:276] 0709 12:23:42.503700 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.853764 (+1350064us) service_pool.cc:224] Handling call
0709 12:23:43.854083 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.859679 (+  5596us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.859892 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:43.863013  6469 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:43.879267 31823 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:23:43.834092  6421 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.038s
I20260709 12:23:43.869071  6303 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:43.877380 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1103) took 1372 ms. Trace:
I20260709 12:23:44.011498 31676 rpcz_store.cc:276] 0709 12:23:42.505270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.863530 (+1358260us) service_pool.cc:224] Handling call
0709 12:23:43.876814 (+ 13284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.877018 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.877043 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:43.877091 (+    48us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:43.877131 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.877193 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:43.877316 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:43.833428  6316 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 12:23:43.883656 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3161) took 1167 ms. Trace:
I20260709 12:23:44.013020 31823 rpcz_store.cc:276] 0709 12:23:42.716304 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.878356 (+1162052us) service_pool.cc:224] Handling call
0709 12:23:43.878795 (+   439us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.883374 (+  4579us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.883601 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:43.830668  6475 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.038s
W20260709 12:23:43.874691  6512 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
W20260709 12:23:43.906455 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:43.866298  6470 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:43.865379 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3152) took 1324 ms. Trace:
W20260709 12:23:43.880403  6455 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.935s	user 0.016s	sys 0.010s
I20260709 12:23:43.882968  6469 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:43.871316  6464 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:43.879155 31822 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:23:43.868185  6495 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:43.880559  6276 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:43.880685  6223 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:44.024370 31820 rpcz_store.cc:276] 0709 12:23:42.541406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.846531 (+1305125us) service_pool.cc:224] Handling call
0709 12:23:43.846849 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.864988 (+ 18139us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.865319 (+   331us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:43.835911  6251 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:44.035096 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:43.883523  6516 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
W20260709 12:23:44.015949  6234 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.025s	user 0.010s	sys 0.004s
I20260709 12:23:44.044148 31822 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:44.114189 31822 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
W20260709 12:23:44.045527  6092 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.560s	user 0.012s	sys 0.000s
I20260709 12:23:44.045992  6223 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:44.039445  6421 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:44.039726  6500 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:43.875370  6105 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:44.053475 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:44.025166  6385 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.776s	user 0.012s	sys 0.000s
I20260709 12:23:44.109550  6439 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:44.037658  6455 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.089s	user 0.017s	sys 0.010s
W20260709 12:23:44.115389  6511 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.312s	user 0.000s	sys 0.004s
I20260709 12:23:44.055495  6469 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:44.140674 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.110543  6516 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:44.142405 31746 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:44.140373 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 14 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.119479  6162 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:44.121806  6391 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:44.040577  6470 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:44.035712  5995 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.106s
I20260709 12:23:44.158286 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.110121  6495 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:43.894006  6505 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:44.144243  6333 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:44.122044  6092 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:44.148294  6251 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:44.124627  6421 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:44.193718 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:44.151526  6516 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:44.243520 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.119251  6234 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:44.256695 31746 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:44.139078  6385 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:44.280903 31749 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:44.184381 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:44.237111  6514 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:44.321213 32375 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 18: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:44.324149 31824 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:44.327296 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3140) took 2660 ms. Trace:
I20260709 12:23:44.327912 31827 rpcz_store.cc:276] 0709 12:23:41.667108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.719295 (+ 52187us) service_pool.cc:224] Handling call
0709 12:23:41.719657 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:42.983925 (+1264268us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.326888 (+1342963us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b5400166cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:44.327193 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2855352064}
I20260709 12:23:44.329887 31827 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:44.378002 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.275501  5949 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:44.341637 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:44.391651  6453 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.067s	user 0.008s	sys 0.004s
I20260709 12:23:44.279021  6475 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.118s
I20260709 12:23:44.367987  6514 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:44.503770 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1105) took 1985 ms. Trace:
I20260709 12:23:44.507897 31676 rpcz_store.cc:276] 0709 12:23:42.518161 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.020892 (+1502731us) service_pool.cc:224] Handling call
0709 12:23:44.047959 (+ 27067us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.112916 (+ 64957us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.112945 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.113347 (+   402us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.338964 (+225617us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:44.339006 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.339083 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.502598 (+163515us) raft_consensus.cc:1685] finished
0709 12:23:44.502726 (+   128us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.503304 (+   578us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":64542,"raft.queue_time_us":67395,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":6146,"spinlock_wait_cycles":15616,"thread_start_us":184240,"threads_started":1,"wal-append.queue_time_us":251963}
W20260709 12:23:44.391743 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 390166us
I20260709 12:23:44.419613 31674 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:23:44.514302 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3207) took 1913 ms. Trace:
W20260709 12:23:44.514394 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 172685us
I20260709 12:23:44.514813 31674 rpcz_store.cc:276] 0709 12:23:42.601001 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.411896 (+1810895us) service_pool.cc:224] Handling call
0709 12:23:44.412358 (+   462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.511774 (+ 99416us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.514030 (+  2256us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.431753 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1206) took 3193 ms. Trace:
I20260709 12:23:44.519405 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:44.517045 31819 rpcz_store.cc:276] 0709 12:23:41.242566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.242759 (+   193us) service_pool.cc:224] Handling call
0709 12:23:41.243149 (+   390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:41.443833 (+200684us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:41.443865 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:41.444375 (+   510us) log.cc:844] Serialized 26 byte log entry
0709 12:23:41.783135 (+338760us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:41.783224 (+    89us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.427105 (+2643881us) spinlock_profiling.cc:243] Waited 176 ms on lock 0x7b54002080d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:44.427169 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.427217 (+    48us) raft_consensus.cc:1685] finished
0709 12:23:44.427229 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.431671 (+  4442us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24561,"raft.queue_time_us":25399,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":13346,"spinlock_wait_cycles":403721600,"thread_start_us":262799,"threads_started":1,"wal-append.queue_time_us":343205,"wal-append.run_cpu_time_us":2105,"wal-append.run_wall_time_us":1002107}
I20260709 12:23:44.532514  6092 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:44.534588  6105 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:44.534880 32377 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: [19.12-19.12]
W20260709 12:23:44.464463  6460 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.166s	user 0.008s	sys 0.003s
I20260709 12:23:44.542521  6536 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:44.542789  6460 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:44.470569  6516 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:44.420221  5949 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.554953  5949 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:23:44.450249  6528 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
W20260709 12:23:44.431993  6515 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.415s	user 0.001s	sys 0.000s
W20260709 12:23:44.449159  6519 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.327s	user 0.000s	sys 0.001s
I20260709 12:23:44.559073  6439 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:23:44.456526  6522 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.270s	user 0.000s	sys 0.001s
I20260709 12:23:44.418473  6391 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:44.563279  6391 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:44.460755  6524 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.162s	user 0.000s	sys 0.000s
I20260709 12:23:44.564038  6391 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:44.563295 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.566362 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.568778 31752 raft_consensus.cc:2410] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
W20260709 12:23:44.538153 31672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s	user 0.000s	sys 0.006s
W20260709 12:23:44.571254 31672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.000s	sys 0.007s
I20260709 12:23:44.572928 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3189) took 2980 ms. Trace:
I20260709 12:23:44.570621 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.575124  6434 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:23:44.575567 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1216) took 1963 ms. Trace:
I20260709 12:23:44.576043 31821 rpcz_store.cc:276] 0709 12:23:42.611605 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.855164 (+1243559us) service_pool.cc:224] Handling call
0709 12:23:43.855467 (+   303us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:43.855822 (+   355us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:43.855850 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:43.856322 (+   472us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.012769 (+156447us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:44.012815 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.012885 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.575045 (+562160us) raft_consensus.cc:1685] finished
0709 12:23:44.575084 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.575352 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1082,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":5702,"thread_start_us":148136,"threads_started":1,"wal-append.queue_time_us":314038}
I20260709 12:23:44.577798 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.578029 31819 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:23:44.439983  6475 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:44.578869 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3070) took 1186 ms. Trace:
I20260709 12:23:44.579434  6475 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:44.423272  6489 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.580960  6475 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:44.581632  6489 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:44.413735  6356 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:44.578742 31752 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:44.585263  6489 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:44.406210  6523 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.144s	user 0.001s	sys 0.000s
W20260709 12:23:44.449092  6517 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.393s	user 0.000s	sys 0.001s
W20260709 12:23:44.455365  6526 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.172s	user 0.001s	sys 0.000s
I20260709 12:23:44.461325  6514 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:44.588018 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.423084  6453 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:44.423614 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.600214  6453 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:44.600294 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1097) took 2117 ms. Trace:
I20260709 12:23:44.589409  6385 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:44.590188  6495 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:44.601367 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.591362  6234 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:44.461611  6470 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:44.579442 31819 rpcz_store.cc:276] 0709 12:23:43.392255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.521465 (+1129210us) service_pool.cc:224] Handling call
0709 12:23:44.522195 (+   730us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.578462 (+ 56267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.578789 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:44.603457 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62435.2us
I20260709 12:23:44.604321  6495 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.605108  6495 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:44.605702 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3169) took 1173 ms. Trace:
I20260709 12:23:44.607178 31819 rpcz_store.cc:276] 0709 12:23:43.432017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.603434 (+1171417us) service_pool.cc:224] Handling call
0709 12:23:44.604769 (+  1335us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.605345 (+   576us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.605370 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:44.605435 (+    65us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:44.605467 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.605522 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.605627 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.606266 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:44.607977  6470 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:44.609344 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.611044 31752 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:44.615352 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.600848 31671 rpcz_store.cc:276] 0709 12:23:42.482518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.701428 (+218910us) service_pool.cc:224] Handling call
0709 12:23:42.701759 (+   331us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:42.702866 (+  1107us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:42.702908 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:42.703541 (+   633us) log.cc:844] Serialized 26 byte log entry
0709 12:23:43.488609 (+785068us) spinlock_profiling.cc:243] Waited 20.4 ms on lock 0x7b180014c9e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:23:43.811138 (+322529us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:43.811201 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.811300 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.599714 (+788414us) raft_consensus.cc:1685] finished
0709 12:23:44.599751 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.600109 (+   358us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":705,"raft.queue_time_us":10162,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":4965,"spinlock_wait_cycles":46904064,"thread_start_us":322291,"threads_started":1,"wal-append.queue_time_us":1168425}
I20260709 12:23:44.547806  6536 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:23:44.612260  6516 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:44.613533 31819 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
W20260709 12:23:44.609967 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 90875.1us
I20260709 12:23:44.602290 31744 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:44.573410 31672 rpcz_store.cc:276] 0709 12:23:41.592910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:41.891417 (+298507us) service_pool.cc:224] Handling call
0709 12:23:41.906870 (+ 15453us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:41.907109 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:23:41.937491 (+ 30382us) log.cc:844] Serialized 14 byte log entry
0709 12:23:43.974812 (+2037321us) spinlock_profiling.cc:243] Waited 3.98 ms on lock 0x7b18003451c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:23:44.572109 (+597297us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:44.572266 (+   157us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:23:44.572423 (+   157us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.572577 (+   154us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.572829 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":56867,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":192,"raft.run_cpu_time_us":297,"raft.run_wall_time_us":298,"spinlock_wait_cycles":9156992,"thread_start_us":596703,"threads_started":1,"wal-append.queue_time_us":2489861}
I20260709 12:23:44.626225 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1104) took 2116 ms. Trace:
I20260709 12:23:44.625556 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3073) took 1174 ms. Trace:
I20260709 12:23:44.628979 31819 rpcz_store.cc:276] 0709 12:23:43.445515 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.612145 (+1166630us) service_pool.cc:224] Handling call
0709 12:23:44.612682 (+   537us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.618874 (+  6192us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.620099 (+  1225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.624320 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.618359  6536 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:44.633322 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:44.643764  6385 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.635329  6536 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:44.645098 31671 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:44.644788  6385 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:44.646031 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2301) took 1645 ms. Trace:
I20260709 12:23:44.646452 31671 rpcz_store.cc:276] 0709 12:23:43.000151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.632758 (+1632607us) service_pool.cc:224] Handling call
0709 12:23:44.644828 (+ 12070us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.645620 (+   792us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.645949 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.649031 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3079) took 1165 ms. Trace:
I20260709 12:23:44.648885 31671 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:23:44.650095 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2316) took 1619 ms. Trace:
I20260709 12:23:44.650580 31671 rpcz_store.cc:276] 0709 12:23:43.031158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.647792 (+1616634us) service_pool.cc:224] Handling call
0709 12:23:44.648355 (+   563us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.649701 (+  1346us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.649997 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.649479 31826 rpcz_store.cc:276] 0709 12:23:43.483411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.647633 (+1164222us) service_pool.cc:224] Handling call
0709 12:23:44.648107 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.648533 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.648561 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:44.648620 (+    59us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:44.648671 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.648725 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.648853 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.635444  5949 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:44.664356 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2474) took 1309 ms. Trace:
I20260709 12:23:44.665000 31750 rpcz_store.cc:276] 0709 12:23:43.355116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.355459 (+   343us) service_pool.cc:224] Handling call
0709 12:23:44.663789 (+1308330us) spinlock_profiling.cc:243] Waited 79 ms on lock 0x7b54004047d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:44.664063 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":181750016}
I20260709 12:23:44.669545 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.671511  6470 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:44.672425  6470 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:44.673252 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:35284 (request call id 58) took 1316 ms. Trace:
I20260709 12:23:44.673671 31747 rpcz_store.cc:276] 0709 12:23:43.357122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.386948 (+ 29826us) service_pool.cc:224] Handling call
0709 12:23:44.672850 (+1285902us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54002ac6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdcf853 00007fe901d9d60a 00007fe8fb3237d4 00007fe8fb323762 00007fe8fb323691 00007fe8fb32360c 00007fe8fb3228c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:44.673019 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":437075712}
I20260709 12:23:44.674100 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3092) took 1219 ms. Trace:
I20260709 12:23:44.672762 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3168) took 1291 ms. Trace:
I20260709 12:23:44.675765 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.677026  6470 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:44.628463 31669 rpcz_store.cc:276] 0709 12:23:42.507389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.879952 (+1372563us) service_pool.cc:224] Handling call
0709 12:23:43.880233 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.055888 (+175655us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:44.122315 (+ 66427us) log.cc:844] Serialized 14 byte log entry
0709 12:23:44.391876 (+269561us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:44.391942 (+    66us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:44.391998 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.622973 (+230975us) spinlock_profiling.cc:243] Waited 175 ms on lock 0x7b54003921d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:44.623080 (+   107us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.624086 (+  1006us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":64094,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":4243,"raft.queue_time_us":233,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":4350,"spinlock_wait_cycles":403421568,"thread_start_us":269013,"threads_started":1,"wal-append.queue_time_us":201103}
I20260709 12:23:44.675299 31822 rpcz_store.cc:276] 0709 12:23:43.381280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.118005 (+736725us) service_pool.cc:224] Handling call
0709 12:23:44.118333 (+   328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.263841 (+145508us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.263884 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.264469 (+   585us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.424289 (+159820us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:44.424359 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.672104 (+247745us) spinlock_profiling.cc:243] Waited 145 ms on lock 0x7b54000fc654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:44.672199 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.672280 (+    81us) raft_consensus.cc:1685] finished
0709 12:23:44.672301 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.672597 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1145,"raft.run_cpu_time_us":194,"raft.run_wall_time_us":8867,"spinlock_wait_cycles":333604608,"thread_start_us":159392,"threads_started":1,"wal-append.queue_time_us":183456}
I20260709 12:23:44.683311  6475 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:44.685418 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1032) took 1221 ms. Trace:
I20260709 12:23:44.682036 31669 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 11. (index mismatch)
I20260709 12:23:44.683943 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1029) took 1290 ms. Trace:
I20260709 12:23:44.690418 32372 rpcz_store.cc:276] 0709 12:23:43.393741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.394348 (+   607us) service_pool.cc:224] Handling call
0709 12:23:43.394728 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.760323 (+365595us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.683449 (+923126us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:44.683741 (+   292us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":839456128}
I20260709 12:23:44.687862 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 1187) took 1258 ms. Trace:
I20260709 12:23:44.674466 31749 rpcz_store.cc:276] 0709 12:23:43.454584 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.455566 (+   982us) service_pool.cc:224] Handling call
0709 12:23:43.481789 (+ 26223us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.281612 (+799823us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.673611 (+391999us) spinlock_profiling.cc:243] Waited 799 ms on lock 0x7b54002959d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:44.673929 (+   318us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1837183360}
I20260709 12:23:44.651842  6385 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:44.685914 32373 rpcz_store.cc:276] 0709 12:23:43.464179 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.464567 (+   388us) service_pool.cc:224] Handling call
0709 12:23:43.482250 (+ 17683us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.815364 (+333114us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.684943 (+869579us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b5400499154. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:44.685239 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":620807040}
I20260709 12:23:44.695271 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3221) took 1561 ms. Trace:
I20260709 12:23:44.692414 32375 rpcz_store.cc:276] 0709 12:23:43.429505 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.430059 (+   554us) service_pool.cc:224] Handling call
0709 12:23:44.687375 (+1257316us) spinlock_profiling.cc:243] Waited 890 ms on lock 0x7b54005c30d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:44.687644 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2046479104}
I20260709 12:23:44.699397 31669 rpcz_store.cc:276] 0709 12:23:43.133752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.678840 (+1545088us) service_pool.cc:224] Handling call
0709 12:23:44.680691 (+  1851us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.688861 (+  8170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.695106 (+  6245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.704581 32370 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:44.708621 31642 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:44.710893 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 79488.3us
I20260709 12:23:44.711366  6012 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:44.711704  6516 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:44.713775 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3075) took 1261 ms. Trace:
I20260709 12:23:44.466992  6469 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:44.715149  6516 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:44.716452  5880 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:44.716918  6214 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:44.711881  6012 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:44.720793  6012 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:44.722314  6292 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:44.722584  6012 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:44.723673 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:44.698644 31751 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:44.725410 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:44.725980 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.727743 31751 raft_consensus.cc:2368] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23.
I20260709 12:23:44.714342 31819 rpcz_store.cc:276] 0709 12:23:43.452882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.629716 (+1176834us) service_pool.cc:224] Handling call
0709 12:23:44.700314 (+ 70598us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.713317 (+ 13003us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.713345 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:44.713439 (+    94us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:23:44.713485 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.713551 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.713675 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.726450 31747 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:44.729364 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.727659 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:44.699441  6460 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:44.737525 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.738427 31751 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:44.742726 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.745873 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.760229 31744 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:44.760355  6294 raft_consensus.cc:2764] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:23:44.762080 31744 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:44.768802 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:44.771114  6441 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:23:44.772682  6223 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.639s	user 0.015s	sys 0.005s
I20260709 12:23:44.773420  6223 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:44.767241 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:44.779783 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1024) took 1021 ms. Trace:
I20260709 12:23:44.780254 31744 rpcz_store.cc:276] 0709 12:23:43.743849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.761157 (+1017308us) service_pool.cc:224] Handling call
0709 12:23:44.761720 (+   563us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.765384 (+  3664us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.765602 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.784556 31672 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 16. (term mismatch)
I20260709 12:23:44.785845 31669 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:23:44.787576 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3228) took 1642 ms. Trace:
I20260709 12:23:44.788283 31669 rpcz_store.cc:276] 0709 12:23:43.145423 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.728837 (+1583414us) service_pool.cc:224] Handling call
0709 12:23:44.757920 (+ 29083us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.787314 (+ 29394us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.787519 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.784993 31672 pending_rounds.cc:85] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 15
I20260709 12:23:44.790380 31750 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:44.791235 31744 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:23:44.793311 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3104) took 1079 ms. Trace:
I20260709 12:23:44.802014 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:44.803231  6487 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.803995  6487 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:44.807027 31747 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:23:44.807835 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3109) took 1025 ms. Trace:
I20260709 12:23:44.808166 31747 rpcz_store.cc:276] 0709 12:23:43.782781 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.794442 (+1011661us) service_pool.cc:224] Handling call
0709 12:23:44.802822 (+  8380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.807537 (+  4715us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.807779 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:44.811228 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 105645us
I20260709 12:23:44.813469  6223 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:44.815871 31677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2302) took 1815 ms (client timeout 1935 ms). Trace:
W20260709 12:23:44.817348 31745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2471) took 1494 ms (client timeout 1574 ms). Trace:
W20260709 12:23:44.817781 31745 rpcz_store.cc:269] 0709 12:23:43.322932 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.333629 (+ 10697us) service_pool.cc:224] Handling call
0709 12:23:44.816934 (+1483305us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b54000b86d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:44.817146 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1347408128}
I20260709 12:23:44.820506  6501 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:44.821878  6501 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:23:44.813576 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.824425 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3061) took 2303 ms. Trace:
I20260709 12:23:44.824558  6507 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 15.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:23:44.824950 31818 rpcz_store.cc:276] 0709 12:23:42.520530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.994038 (+473508us) service_pool.cc:224] Handling call
0709 12:23:43.655371 (+661333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.731704 (+ 76333us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.823971 (+1092267us) spinlock_profiling.cc:243] Waited 661 ms on lock 0x7b54003c96d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:44.824221 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1520176384}
I20260709 12:23:44.825413  6469 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:44.820916 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:44.795163 31752 rpcz_store.cc:276] 0709 12:23:43.713686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.782393 (+1068707us) service_pool.cc:224] Handling call
0709 12:23:44.792509 (+ 10116us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.792836 (+   327us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.792871 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:44.792946 (+    75us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:44.792986 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.793052 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.793208 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.826309  6507 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:44.830107 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:44.835474 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:44.852579 31745 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:44.862327 31744 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:23:44.831056 31674 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:44.840215 32376 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: [16.13-16.13]
I20260709 12:23:44.804001 31750 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 15 index: 9. (index mismatch)
I20260709 12:23:44.794427 31672 pending_rounds.cc:107] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 21.16
I20260709 12:23:44.813469 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:44.848348 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2477) took 1492 ms. Trace:
I20260709 12:23:44.874995 31672 raft_consensus.cc:2889] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:44.837920 31748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2466) took 1564 ms (client timeout 1967 ms). Trace:
I20260709 12:23:44.826566 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:44.878454 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3111) took 1040 ms. Trace:
I20260709 12:23:44.879395 31750 rpcz_store.cc:276] 0709 12:23:43.837619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.801352 (+963733us) service_pool.cc:224] Handling call
0709 12:23:44.803478 (+  2126us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.873630 (+ 70152us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.878396 (+  4766us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.880973  5759 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:44.881680  5759 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:44.833067  6505 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.050s
I20260709 12:23:44.883854  6505 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20260709 12:23:44.848583  6247 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.018s	user 0.010s	sys 0.014s
I20260709 12:23:44.869061  6507 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:44.886822  6469 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:44.830978 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:44.887871  6469 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:44.887910  6507 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:44.889827  6507 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:44.838106 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:44.891112 32378 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:44.891873 31745 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:23:44.890569  6452 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.002s
I20260709 12:23:44.875757 31753 rpcz_store.cc:276] 0709 12:23:43.355959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.387903 (+ 31944us) service_pool.cc:224] Handling call
0709 12:23:43.443748 (+ 55845us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:43.815794 (+372046us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.847661 (+1031867us) spinlock_profiling.cc:243] Waited 365 ms on lock 0x7b54000b95d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:44.848097 (+   436us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":840443264}
I20260709 12:23:44.886997  6247 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23.
I20260709 12:23:44.893222  6452 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:44.893682  6452 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:44.893924 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3232) took 1728 ms. Trace:
I20260709 12:23:44.878052 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3112) took 1002 ms. Trace:
I20260709 12:23:44.894907 31744 rpcz_store.cc:276] 0709 12:23:43.871036 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.804841 (+933805us) service_pool.cc:224] Handling call
0709 12:23:44.805275 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.872854 (+ 67579us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.873069 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:44.877631 31748 rpcz_store.cc:269] 0709 12:23:43.273810 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.291454 (+ 17644us) service_pool.cc:224] Handling call
0709 12:23:44.837419 (+1545965us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b54002f6cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:44.837676 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2544478592}
W20260709 12:23:44.896507 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 83202us
W20260709 12:23:44.816430 31677 rpcz_store.cc:269] 0709 12:23:43.000447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.234146 (+233699us) service_pool.cc:224] Handling call
0709 12:23:44.815376 (+1581230us) spinlock_profiling.cc:243] Waited 535 ms on lock 0x7b5400253fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:44.815582 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1229555072}
I20260709 12:23:44.877470 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3085) took 1337 ms. Trace:
I20260709 12:23:44.898849 31819 rpcz_store.cc:276] 0709 12:23:43.533826 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.730288 (+1196462us) service_pool.cc:224] Handling call
0709 12:23:44.779489 (+ 49201us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.788267 (+  8778us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.788302 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:44.799131 (+ 10829us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:44.870738 (+ 71607us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.870892 (+   154us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.871141 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.890010 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:44.894440 31673 rpcz_store.cc:276] 0709 12:23:43.165044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.892551 (+1727507us) service_pool.cc:224] Handling call
0709 12:23:44.893014 (+   463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.893418 (+   404us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.893453 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:44.893522 (+    69us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:44.893572 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.893630 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:44.893741 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.902976 31818 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:44.904558 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:23:44.897011  6303 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:44.910025 31750 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:23:44.916137  6487 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:44.848419 31746 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.582s	user 0.005s	sys 0.011s
I20260709 12:23:44.852912  6222 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:44.925786 31747 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:23:44.928874 32378 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:44.929091 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:44.929114 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3216) took 2257 ms. Trace:
I20260709 12:23:44.890517  6469 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:44.933687 31672 rpcz_store.cc:276] 0709 12:23:42.671607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.630931 (+1959324us) service_pool.cc:224] Handling call
0709 12:23:44.728770 (+ 97839us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.915882 (+187112us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.928989 (+ 13107us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.925169  6460 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.949885 31746 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:44.953204 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:44.951675  6475 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:44.963855 32378 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
W20260709 12:23:44.964061 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 143353us
I20260709 12:23:44.966768 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:44.967352 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 90719us
I20260709 12:23:44.967495 31752 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:44.952939  6516 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:23:44.988029 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:44.989157 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1020) took 1261 ms. Trace:
I20260709 12:23:44.992056 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:44.989691 31746 rpcz_store.cc:276] 0709 12:23:43.701141 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.186457 (+485316us) service_pool.cc:224] Handling call
0709 12:23:44.960593 (+774136us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:45.000062 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 65868.8us
I20260709 12:23:45.002261 31673 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:45.010715  6469 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:45.010922 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:45.011626  6469 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.014999 31819 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:45.019109 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3233) took 1849 ms. Trace:
I20260709 12:23:45.019599 31673 rpcz_store.cc:276] 0709 12:23:43.169544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.925537 (+1755993us) service_pool.cc:224] Handling call
0709 12:23:44.962299 (+ 36762us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.015063 (+ 52764us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.019032 (+  3969us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:44.949075  6452 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:44.962245  6391 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.028014 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:45.014045  6469 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:44.990824  6487 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260709 12:23:45.032562 31751 raft_consensus.cc:1217] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Deduplicated request from leader. Original: 20.15->[21.16-21.16]   Dedup: 21.16->[]
I20260709 12:23:45.034438 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3083) took 1487 ms. Trace:
I20260709 12:23:45.037341 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:45.040362 31672 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:45.041028 31751 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [21.16-21.16]
I20260709 12:23:45.035903 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:45.039688 31826 rpcz_store.cc:276] 0709 12:23:43.533125 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.653452 (+1120327us) service_pool.cc:224] Handling call
0709 12:23:44.740043 (+ 86591us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.751238 (+ 11195us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:44.925385 (+174147us) log.cc:844] Serialized 14 byte log entry
0709 12:23:45.019662 (+ 94277us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:45.019768 (+   106us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:45.019829 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.019928 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.020077 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3136,"prepare.queue_time_us":287278,"raft.queue_time_us":15164,"thread_start_us":224125,"threads_started":2}
I20260709 12:23:45.048420 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:45.051172 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52259.9us
I20260709 12:23:45.052325 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:45.050300 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:45.055578  6501 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:45.056562 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:45.057875 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:45.059777  6276 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.936s	user 0.008s	sys 0.012s
W20260709 12:23:45.060524 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.060269  6391 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:45.060948 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 127286us
I20260709 12:23:45.007509  6460 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.060256  6276 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.062573 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:45.062961 31675 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:45.063108  6536 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.063779  6391 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:45.064203  6391 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.064087 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:45.064774  6391 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:45.065654  6536 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:45.028560  6475 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.044102 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2319) took 1619 ms. Trace:
I20260709 12:23:45.068550 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:45.068833 31672 rpcz_store.cc:276] 0709 12:23:43.424621 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.948987 (+1524366us) service_pool.cc:224] Handling call
0709 12:23:45.010107 (+ 61120us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.043841 (+ 33734us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.044045 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:45.018935  6012 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:45.069237 31826 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:45.070087 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3084) took 1536 ms. Trace:
W20260709 12:23:45.091495 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.079165 31822 rpcz_store.cc:276] 0709 12:23:43.533534 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.685277 (+1151743us) service_pool.cc:224] Handling call
0709 12:23:44.686159 (+   882us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:44.688268 (+  2109us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.688396 (+   128us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:44.743529 (+ 55133us) log.cc:844] Serialized 52 byte log entry
0709 12:23:44.800817 (+ 57288us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:44.800871 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.859850 (+ 58979us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.069867 (+210017us) raft_consensus.cc:1685] finished
0709 12:23:45.069889 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.070032 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":445,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"thread_start_us":8875,"threads_started":1,"wal-append.queue_time_us":149397}
I20260709 12:23:45.097299 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3229) took 1951 ms. Trace:
I20260709 12:23:45.040009  6303 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:45.100121 31669 rpcz_store.cc:276] 0709 12:23:43.145729 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.809781 (+1664052us) service_pool.cc:224] Handling call
0709 12:23:44.810205 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.857479 (+ 47274us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:23:44.998513 (+141034us) log.cc:844] Serialized 14 byte log entry
0709 12:23:45.096901 (+ 98388us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:45.096951 (+    50us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:23:45.097000 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.097077 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.097239 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":152132,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":5036,"thread_start_us":167620,"threads_started":2}
I20260709 12:23:45.068583  6475 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:45.106966 31818 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:45.078022  6276 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:45.110466 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46518.2us
I20260709 12:23:45.100886  6247 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:23:45.122290 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 68358us
I20260709 12:23:45.128533 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:45.128785 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:45.111370  6303 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:45.138284 31818 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:45.148317 31822 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:45.154974 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:45.157323 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:45.158593  6012 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:45.158835  6475 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:45.159425  6303 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.164068  6485 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:45.169214 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:45.170680  5844 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.171381  5844 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.172071 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:45.173719 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:45.176187  6501 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:45.177136  6501 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:45.181322  5759 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.183481  6012 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:45.187119  6438 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.187950 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:45.187873  6438 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.188455 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 977) took 1004 ms. Trace:
I20260709 12:23:45.188850 32376 rpcz_store.cc:276] 0709 12:23:44.184044 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.184590 (+   546us) service_pool.cc:224] Handling call
0709 12:23:44.247672 (+ 63082us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.891376 (+643704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.188097 (+296721us) spinlock_profiling.cc:243] Waited 592 ms on lock 0x7b5400427f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:45.188372 (+   275us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1361168128}
I20260709 12:23:45.190871  6092 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:45.191257 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3215) took 2564 ms. Trace:
I20260709 12:23:45.191720 31674 rpcz_store.cc:276] 0709 12:23:42.646468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.608501 (+1962033us) service_pool.cc:224] Handling call
0709 12:23:44.830761 (+222260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.874259 (+ 43498us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.191026 (+316767us) spinlock_profiling.cc:243] Waited 221 ms on lock 0x7b5400253fd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:45.191188 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":508900224}
I20260709 12:23:45.192301  6092 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:45.192793  6092 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.193154  6092 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:45.194430  6495 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:45.194958  5949 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:45.222085 31826 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:45.190366  6438 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:45.137843  6452 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:45.230484 31819 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
W20260709 12:23:45.237673 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 88214.7us
I20260709 12:23:45.195396  5949 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:45.173754  5844 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:45.247243 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:45.252293 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:45.253502 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 79934.1us
W20260709 12:23:45.245249  6555 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:23:45.267479 31822 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:45.268000 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:45.247999  5759 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:45.271173 31826 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:45.276546 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:45.279858 31822 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 24 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 11.
I20260709 12:23:45.277451 31818 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:45.252789  6247 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:45.283037 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 93360.6us
I20260709 12:23:45.291494 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2317) took 2251 ms. Trace:
I20260709 12:23:45.292048 31675 rpcz_store.cc:276] 0709 12:23:43.040373 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.807412 (+767039us) service_pool.cc:224] Handling call
0709 12:23:45.291177 (+1483765us) spinlock_profiling.cc:243] Waited 1.25 s on lock 0x7b540017c0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:45.291398 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2868524928}
I20260709 12:23:45.293365 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3066) took 1901 ms. Trace:
I20260709 12:23:45.293625 31676 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:45.293831 31827 rpcz_store.cc:276] 0709 12:23:43.391308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.333783 (+942475us) service_pool.cc:224] Handling call
0709 12:23:44.334219 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.335810 (+  1591us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.335853 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.399043 (+ 63190us) log.cc:844] Serialized 26 byte log entry
0709 12:23:45.180081 (+781038us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b180036f7c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:23:45.252313 (+ 72232us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:45.252378 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.258851 (+  6473us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.292850 (+ 33999us) raft_consensus.cc:1685] finished
0709 12:23:45.292874 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.293165 (+   291us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1149,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":31280,"spinlock_wait_cycles":10181376,"thread_start_us":56702,"threads_started":1,"wal-append.queue_time_us":891674}
I20260709 12:23:45.298656  6247 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.301244  6247 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:45.301975 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:45.303687 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:45.306021 31753 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: [17.11-19.12]
I20260709 12:23:45.307475 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:45.308370 31753 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:45.307855  5949 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.309459  5949 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:45.308224 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:45.304858 31748 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 19: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:45.313162 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:45.309459 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:45.319942 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1211) took 2794 ms. Trace:
I20260709 12:23:45.321480 31826 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: []
I20260709 12:23:45.322513 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1226) took 1490 ms. Trace:
I20260709 12:23:45.324079 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:45.324595 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:45.320429 31824 rpcz_store.cc:276] 0709 12:23:42.524989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.327620 (+802631us) service_pool.cc:224] Handling call
0709 12:23:43.332236 (+  4616us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.324858 (+992622us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.319496 (+994638us) spinlock_profiling.cc:243] Waited 990 ms on lock 0x7b5400197dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:45.319748 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2275867008}
I20260709 12:23:45.322960 31826 rpcz_store.cc:276] 0709 12:23:43.831813 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.319340 (+1487527us) service_pool.cc:224] Handling call
0709 12:23:45.320757 (+  1417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.322110 (+  1353us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.322435 (+   325us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:45.335858 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:45.340590 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:45.340875 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:45.341719 32375 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:23:45.341593 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:45.342499 32376 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 21: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:45.342506 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:45.342931 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:45.347605 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:45.348068 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3165) took 1967 ms. Trace:
I20260709 12:23:45.348871 31748 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:23:45.348495  6391 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.353571 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:45.355867 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:45.356475 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.357511 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:45.359360 31818 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:45.349709  6276 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:23:45.362242  6251 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.100s	user 0.014s	sys 0.005s
I20260709 12:23:45.362913  6516 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.363478  6516 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:45.370119  6554 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.230s	user 0.000s	sys 0.001s
W20260709 12:23:45.372155  6483 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 2.426s	user 0.004s	sys 0.017s
W20260709 12:23:45.373492  6559 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.211s	user 0.001s	sys 0.000s
I20260709 12:23:45.348551 31820 rpcz_store.cc:276] 0709 12:23:43.380327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.042866 (+662539us) service_pool.cc:224] Handling call
0709 12:23:44.043204 (+   338us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:44.088912 (+ 45708us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.088956 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:44.113006 (+ 24050us) log.cc:844] Serialized 52 byte log entry
0709 12:23:45.171287 (+1058281us) spinlock_profiling.cc:243] Waited 4.66 ms on lock 0x7b18003d1380. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:23:45.283779 (+112492us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:45.312527 (+ 28748us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.335680 (+ 23153us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.347563 (+ 11883us) raft_consensus.cc:1685] finished
0709 12:23:45.347587 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.347893 (+   306us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45277,"raft.queue_time_us":100476,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":32135,"spinlock_wait_cycles":10716032,"thread_start_us":81451,"threads_started":1,"wal-append.queue_time_us":1220901}
I20260709 12:23:45.375149 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3148) took 2870 ms. Trace:
I20260709 12:23:45.375535 31825 rpcz_store.cc:276] 0709 12:23:42.504790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:42.504921 (+   131us) service_pool.cc:224] Handling call
0709 12:23:42.505300 (+   379us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:42.736909 (+231609us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:42.736945 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:42.744608 (+  7663us) log.cc:844] Serialized 26 byte log entry
0709 12:23:42.868199 (+123591us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:42.875378 (+  7179us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:43.416280 (+540902us) spinlock_profiling.cc:243] Waited 231 ms on lock 0x7b54000fbc54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:43.416348 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.371461 (+1955113us) spinlock_profiling.cc:243] Waited 20.5 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:45.374712 (+  3251us) raft_consensus.cc:1685] finished
0709 12:23:45.374734 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.374944 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7257,"raft.queue_time_us":1297,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":2759,"spinlock_wait_cycles":578774272,"thread_start_us":55904,"threads_started":1,"wal-append.queue_time_us":177773}
I20260709 12:23:45.377848 31825 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:45.378274 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:45.379112 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 70955.4us
I20260709 12:23:45.380149  6563 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:45.380795  6563 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:45.381716  6557 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 12:23:45.382360 31744 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
W20260709 12:23:45.377902  6531 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.941s	user 0.000s	sys 0.003s
I20260709 12:23:45.383574 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3101) took 1671 ms. Trace:
I20260709 12:23:45.383967 31751 rpcz_store.cc:276] 0709 12:23:43.712422 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.752900 (+1040478us) service_pool.cc:224] Handling call
0709 12:23:44.790384 (+ 37484us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:45.072369 (+281985us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.383297 (+310928us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b54002945d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:45.383510 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":489499520}
I20260709 12:23:45.379055 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3234) took 2209 ms. Trace:
I20260709 12:23:45.385581 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3164) took 2005 ms. Trace:
I20260709 12:23:45.386343  6497 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:45.386794  6497 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:45.386693 31823 rpcz_store.cc:276] 0709 12:23:43.380013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.039112 (+659099us) service_pool.cc:224] Handling call
0709 12:23:44.042035 (+  2923us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.088213 (+ 46178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.088271 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.117071 (+ 28800us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.339309 (+222238us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:44.339351 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.339415 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:44.630365 (+290950us) spinlock_profiling.cc:243] Waited 2.89 ms on lock 0x7b4800130158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:23:45.360509 (+730144us) spinlock_profiling.cc:243] Waited 2.26 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:45.384235 (+ 23726us) raft_consensus.cc:1685] finished
0709 12:23:45.384258 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.385271 (+  1013us) spinlock_profiling.cc:243] Waited 2.63 ms on lock 0x7b54003c8cd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:45.385414 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":45691,"raft.queue_time_us":46410,"raft.run_cpu_time_us":13365,"raft.run_wall_time_us":531259,"spinlock_wait_cycles":17872128,"thread_start_us":188629,"threads_started":1,"wal-append.queue_time_us":294006}
I20260709 12:23:45.388082  6497 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.389740  6497 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:45.414516 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:45.418654 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:45.420071 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:45.421563 31748 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:45.417418 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1225) took 1798 ms. Trace:
I20260709 12:23:45.380107 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:45.423044 31822 rpcz_store.cc:276] 0709 12:23:43.619452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.317214 (+1697762us) service_pool.cc:224] Handling call
0709 12:23:45.416484 (+ 99270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.416918 (+   434us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:45.416962 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:45.417038 (+    76us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:45.417088 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.417170 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.417334 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:45.394455  6434 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:45.397734  6244 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:45.360677 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:45.430013 31753 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:45.382073  6563 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:45.403062 31678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2305) took 2401 ms (client timeout 2221 ms). Trace:
I20260709 12:23:45.386029 31677 rpcz_store.cc:276] 0709 12:23:43.169878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.917524 (+1747646us) service_pool.cc:224] Handling call
0709 12:23:44.918131 (+   607us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.931674 (+ 13543us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.931717 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.932324 (+   607us) log.cc:844] Serialized 26 byte log entry
0709 12:23:45.113369 (+181045us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:45.113463 (+    94us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.114100 (+   637us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.372703 (+258603us) spinlock_profiling.cc:243] Waited 66.1 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:45.378705 (+  6002us) raft_consensus.cc:1685] finished
0709 12:23:45.378728 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.378989 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7052,"raft.queue_time_us":15652,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":8016,"spinlock_wait_cycles":152013696,"thread_start_us":144978,"threads_started":1,"wal-append.queue_time_us":204626}
I20260709 12:23:45.406085 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:45.407270  6507 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.517s	user 0.011s	sys 0.005s
I20260709 12:23:45.407436 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3119) took 1254 ms. Trace:
W20260709 12:23:45.408020 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 50580.4us
I20260709 12:23:45.408607  6495 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:45.412710  6487 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:23:45.413029 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 181544us
I20260709 12:23:45.413691 31824 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:23:45.438601 31747 rpcz_store.cc:276] 0709 12:23:44.153365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.808972 (+655607us) service_pool.cc:224] Handling call
0709 12:23:44.865822 (+ 56850us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.932925 (+ 67103us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.407214 (+474289us) spinlock_profiling.cc:243] Waited 56.4 ms on lock 0x7b54000b86d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:45.407357 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":129621376}
W20260709 12:23:45.435451 31678 rpcz_store.cc:269] 0709 12:23:43.001339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.655844 (+654505us) service_pool.cc:224] Handling call
0709 12:23:45.402589 (+1746745us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b54001ff9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:45.402841 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":280453888}
I20260709 12:23:45.441494 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:45.413959  6558 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
W20260709 12:23:45.390769 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.416070  6391 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:45.414593  6516 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:45.429546  6434 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:23:45.449939 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 69979.8us
I20260709 12:23:45.459229 32378 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
W20260709 12:23:45.467188 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 146816us
I20260709 12:23:45.429275  6092 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.459738  6487 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:45.493590 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.496273 31825 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 21. Current term is 22. Ops: []
W20260709 12:23:45.496883 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.462262  6495 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:23:45.494864 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 53492.5us
I20260709 12:23:45.465591  6251 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:45.503593 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:45.504830  6553 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.361s	user 0.001s	sys 0.001s
I20260709 12:23:45.548718  6434 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.497891  6563 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:45.563218  6434 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:45.553058 31825 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:23:45.553435 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:45.502496  6251 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:45.567454  6251 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:45.571445  6575 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:45.573369  6564 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:45.573848  6564 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:45.574975  6564 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.577030  6564 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:45.570461 31822 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
W20260709 12:23:45.587865 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58840.7us
I20260709 12:23:45.588729  6303 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:45.589496  6303 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.590600  6247 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:45.591276  6247 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.592657  6303 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:45.593287  6247 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:45.605026 31748 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:23:45.604434  6247 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:45.607688  6247 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.609246 31753 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:45.608044  6434 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:45.610529  6218 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:45.611214 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 146501us
I20260709 12:23:45.610819  6501 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:45.614838  6276 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.615139 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:45.615572 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:45.596613  6092 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:45.506218  6391 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 15.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.107s
I20260709 12:23:45.609078  6516 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:45.624102  6218 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.624511 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:45.613550  6507 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.619694  5759 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.625047 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:45.625329  6501 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.622704 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 17 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:45.621049 31678 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:23:45.626086  6276 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.636040 31674 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 22 because replica has last-logged OpId of term: 19 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 15.
W20260709 12:23:45.640386 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 111708us
I20260709 12:23:45.640877 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 974) took 1944 ms. Trace:
I20260709 12:23:45.638444 31822 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:23:45.641299 32370 rpcz_store.cc:276] 0709 12:23:43.692003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.692138 (+   135us) service_pool.cc:224] Handling call
0709 12:23:44.695170 (+1003032us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:44.705241 (+ 10071us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.630784 (+925543us) spinlock_profiling.cc:243] Waited 997 ms on lock 0x7b54005c30d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:45.635634 (+  4850us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2291982080}
I20260709 12:23:45.646034 31669 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:45.648800 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:45.649495 32372 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:23:45.649837 31825 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:45.653208 31673 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 16 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:45.653097 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:45.647215 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:45.643677  6507 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.661346  6501 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:45.661922  6252 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:45.662324  6252 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:45.663009  6247 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:45.663244  6507 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:45.652688  5759 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.665884  5759 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:45.663272  6252 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.666759 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:45.667137 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:45.659471  6434 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:45.667546 31678 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:45.667496  6252 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:45.669657 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:45.670750 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:45.654866  6460 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:45.672381  6485 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:45.677840  6485 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:45.678944  6485 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:45.681285 31672 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:23:45.683106 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45431.1us
I20260709 12:23:45.687088  6218 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:45.691553 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:45.696642 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3212) took 3070 ms. Trace:
I20260709 12:23:45.706221 31676 rpcz_store.cc:276] 0709 12:23:42.625783 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.539641 (+1913858us) service_pool.cc:224] Handling call
0709 12:23:45.293441 (+753800us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:45.294117 (+   676us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.696409 (+402292us) spinlock_profiling.cc:243] Waited 713 ms on lock 0x7b540017c0d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:45.696570 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1640280320}
I20260709 12:23:45.698834 31818 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:45.699453  6455 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:45.684128 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:45.709910 31678 raft_consensus.cc:2410] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 22 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:23:45.652724  6487 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.721524 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:45.670487  6495 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 9, Time since last communication: 0.033s
I20260709 12:23:45.719072 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:45.722252 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:45.729751 31822 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
W20260709 12:23:45.723997 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.710825  6247 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:45.733906 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:45.740633 31672 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:45.745645 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:45.763190 31791 leader_election.cc:400] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23.
I20260709 12:23:45.765147 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:45.766366  6012 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:45.767050  6012 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:45.767606  6012 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:45.788981 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:45.739439  6247 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:45.803601  6247 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:45.746739 31678 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
W20260709 12:23:45.903314 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:45.908233 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 265508us
I20260709 12:23:45.909106 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:45.906848  6391 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:45.911458 31672 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:45.913054 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:45.912135 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:23:45.915200  6391 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.917814  6391 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:45.917913  6540 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.960s	user 0.002s	sys 0.000s
W20260709 12:23:45.919262  6530 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.482s	user 0.002s	sys 0.005s
I20260709 12:23:45.919976 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1223) took 2495 ms. Trace:
I20260709 12:23:45.920390 31821 rpcz_store.cc:276] 0709 12:23:43.424860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.577946 (+1153086us) service_pool.cc:224] Handling call
0709 12:23:44.578456 (+   510us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:44.636535 (+ 58079us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:44.636583 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:44.637585 (+  1002us) log.cc:844] Serialized 26 byte log entry
0709 12:23:44.700578 (+ 62993us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:44.700641 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:44.700731 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:45.917314 (+1216583us) spinlock_profiling.cc:243] Waited 3.45 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:45.919568 (+  2254us) raft_consensus.cc:1685] finished
0709 12:23:45.919609 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:45.919909 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3108,"raft.queue_time_us":3387,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":3877,"spinlock_wait_cycles":7924480,"thread_start_us":62430,"threads_started":1,"wal-append.queue_time_us":160516}
W20260709 12:23:45.923538 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 158614us
I20260709 12:23:45.924513 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:45.927821  6391 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:45.927630 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:45.929785 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:45.931273 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:45.932623 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:45.932731 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:45.927527  6501 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:45.947372 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:45.949069  6501 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:45.950342  6501 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:45.952054  6556 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.789s	user 0.010s	sys 0.007s
W20260709 12:23:45.955974 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 639341us
I20260709 12:23:45.962188  6460 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:45.962602  6460 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:45.962651 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:45.956669  6303 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:45.963649  6303 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:45.964775  6303 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:45.965955 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:45.966169  6303 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:45.958738 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3174) took 2109 ms. Trace:
I20260709 12:23:45.959024  6292 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:23:45.967473 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:45.967891 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:45.967041 31824 rpcz_store.cc:276] 0709 12:23:43.849690 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:45.346453 (+1496763us) service_pool.cc:224] Handling call
0709 12:23:45.346938 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:45.441705 (+ 94767us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:45.958406 (+516701us) spinlock_profiling.cc:243] Waited 66.5 ms on lock 0x7b54002675d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:45.958675 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":152835840}
I20260709 12:23:45.968343 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:45.967344 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:45.972265 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:45.959631  6342 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:45.960285  6092 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:45.961182  6244 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.523s	user 0.008s	sys 0.003s
I20260709 12:23:45.912561  6276 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:45.916158 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:45.944199  6252 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:45.992944 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:45.943358  6391 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:45.992946 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:45.996243 31747 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:45.989437 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:45.977876  6342 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:45.987465  6563 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:46.010035  6498 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:46.010536  6498 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.010938  6498 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:45.999789  6391 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:46.029872 31747 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:45.985275  6455 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.030s
I20260709 12:23:45.939303  6251 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.034909 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:46.038883 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:46.038846 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.040433 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.015244  6496 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.002162 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 15 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.007426  6223 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:45.996137  6092 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.051836 31672 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:23:46.007349  6252 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:46.006243  6563 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:46.057178 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.038199  6391 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:46.060101 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:46.061568 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.061944 31676 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:46.062354  6563 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:46.080458 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.067152  6276 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:46.054972  6507 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.026s
I20260709 12:23:46.073165  6292 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:46.057559  6223 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:46.098183 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 92563.9us
W20260709 12:23:46.082490  6497 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.645s	user 0.022s	sys 0.001s
I20260709 12:23:46.096190 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.054767  6496 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.108566 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:46.127322 31670 raft_consensus.cc:2410] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
W20260709 12:23:46.128405 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 477861us
I20260709 12:23:46.130298  5759 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:46.131428 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:46.132001  5759 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:46.132673  5759 consensus_queue.cc:1243] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 22.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:23:46.134768  6292 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:46.132341 31678 raft_consensus.cc:2410] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 20 because replica has last-logged OpId of term: 19 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 11.
I20260709 12:23:46.136497 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.136708  6421 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:46.136102  6292 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.137274  6421 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:46.138535  6452 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:23:46.138988  6452 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:46.139990  6452 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:46.137248 31678 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:46.143606 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.144212 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:46.144467 31678 raft_consensus.cc:2410] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 20 because replica has last-logged OpId of term: 19 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 14.
I20260709 12:23:46.145416 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.145783 31674 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:23:46.145993 31678 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:46.155575 31678 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:23:46.157773 31677 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:46.160323 31673 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 11. (term mismatch)
I20260709 12:23:46.160792 31673 pending_rounds.cc:85] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 10
I20260709 12:23:46.160983  6223 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:46.098353  6516 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:46.163617  6460 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.165455  6460 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:46.166847 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:46.167752  5949 raft_consensus.cc:2764] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:23:46.168030 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.168859  5949 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:46.169845 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:46.169930 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.170665  5949 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:46.170840 31669 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:46.170729 31675 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:23:46.171176  5949 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:46.171605  5949 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:46.176331  6092 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:46.177345  6252 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.178857 31678 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
W20260709 12:23:46.179831 31670 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2311) took 3174 ms (client timeout 2718 ms). Trace:
W20260709 12:23:46.180272 31670 rpcz_store.cc:269] 0709 12:23:43.005027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:43.736173 (+731146us) service_pool.cc:224] Handling call
0709 12:23:46.179560 (+2443387us) spinlock_profiling.cc:243] Waited 2.38 s on lock 0x7b540022b354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:46.179743 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5472147200}
I20260709 12:23:46.181012  6251 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:46.087189  6434 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.094s
I20260709 12:23:46.181649 31671 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:23:46.081925  6342 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:46.161060 31673 pending_rounds.cc:107] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:23:46.182880 31673 raft_consensus.cc:2889] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:46.079435  6495 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.034s
I20260709 12:23:46.081140  6012 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:46.141429  6452 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:23:46.208329 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 204394us
I20260709 12:23:46.208392 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3220) took 3075 ms. Trace:
W20260709 12:23:46.209682 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 198168us
I20260709 12:23:46.214808 31671 rpcz_store.cc:276] 0709 12:23:43.133478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:44.692821 (+1559343us) service_pool.cc:224] Handling call
0709 12:23:46.181309 (+1488488us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:46.182028 (+   719us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:46.208157 (+ 26129us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b540022b350. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:46.208327 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3323502976}
I20260709 12:23:46.141459  6247 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.222342 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:46.221742 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 230026us
I20260709 12:23:46.232810 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:46.240022 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 231829us
W20260709 12:23:46.240775 31717 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.823s (SENT) [suppressed 37 similar messages]
W20260709 12:23:46.241961 31717 leader_election.cc:336] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 2.000s (SENT)
I20260709 12:23:46.255730 31825 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
W20260709 12:23:46.259464 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.261991 31677 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
W20260709 12:23:46.262472 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 184926us
I20260709 12:23:46.266748 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:46.267990  6251 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:23:46.273088 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 257454us
I20260709 12:23:46.274724 31745 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 17 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:46.269943  6092 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:46.269985  6251 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.304100 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:46.307327 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:46.308068 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 75390.3us
W20260709 12:23:46.310653 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:46.311210 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 89645.6us
I20260709 12:23:46.312317 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.313480 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:46.314462 32378 raft_consensus.cc:2410] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 21 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:23:46.319730 31674 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
W20260709 12:23:46.323380 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 117633us
W20260709 12:23:46.311251 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.300854  6251 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:46.271950  6421 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 12:23:46.283435  6276 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 12:23:46.269745  6485 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.054s
I20260709 12:23:46.299535  6218 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 12:23:46.362957 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:46.356995 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 56987.5us
I20260709 12:23:46.362672 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:46.330919  6496 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:46.331832  5949 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.354668  6223 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 19.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.022s
I20260709 12:23:46.359874  6497 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.019s
I20260709 12:23:46.374656  6487 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:46.371057  6507 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.019s
I20260709 12:23:46.404014 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.407209 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:46.385857  6548 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:46.421531  6092 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:23:46.423964 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 111765us
I20260709 12:23:46.426178  6589 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:46.421725 32374 raft_consensus.cc:2410] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 23 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 15.
I20260709 12:23:46.426654  6589 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:46.386817  6496 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:46.427017 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:46.427485  6496 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:46.427588  6589 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.428280 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:46.428504  6496 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.429932  6496 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:46.431468  6591 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.112s	user 0.000s	sys 0.001s
W20260709 12:23:46.432906  6584 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 12:23:46.429030  6589 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:46.434116  6598 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 12:23:46.435119  6590 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:23:46.444597  6498 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.378165  6501 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:23:46.447770  6391 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.410753  6012 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:46.410102  6247 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:46.450613  6391 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:46.472240 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:46.457607  6303 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:46.455402  6012 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.473533  6303 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.474452 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.474879 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.471571 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.472429 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:46.460623 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:46.461015 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:46.461648 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:46.478482 31818 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:46.462502 31824 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:46.384807  6602 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Stepping down as leader of term 17
W20260709 12:23:46.458040 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:46.489529 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.490250 31821 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:46.491035 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:46.490449 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.493098 31747 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
W20260709 12:23:46.498381 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.481434 31676 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:46.462603 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:46.415118  6575 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:46.514566 31819 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:23:46.510085 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.473941  6012 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.513124 31751 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:46.389263  6587 raft_consensus.cc:2764] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
W20260709 12:23:46.464699  6582 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 12:23:46.525640 31825 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
W20260709 12:23:46.535612 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 107415us
I20260709 12:23:46.458937  6247 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:46.474000  6303 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:46.525856 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.536444 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:46.518549 31747 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:23:46.527743 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 141774us
I20260709 12:23:46.528726 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:46.530052 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.533209  6599 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:46.533797  6012 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:46.533627  6595 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.121s
W20260709 12:23:46.534837  6592 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
I20260709 12:23:46.477378  6487 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:46.532276 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.536919 31750 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:23:46.539112 31826 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:23:46.490504  6602 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:46.539810 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.571509 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.564352 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:46.573215 31744 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:46.535254  6497 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.459861  6092 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.102s
I20260709 12:23:46.585907 31753 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:46.586377 31753 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:46.516157  6575 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:46.585325 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:46.589692 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:46.596187 31750 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:46.593714 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 154602us
I20260709 12:23:46.594455 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:46.574107  6421 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:46.603430 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 276681us
I20260709 12:23:46.570377  6247 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:46.570964  6608 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:23:46.609270 31745 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: [21.12-23.13]
I20260709 12:23:46.610864 32342 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.578835  5844 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.035s
I20260709 12:23:46.619876 31745 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 12:23:46.599707  6587 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:46.619956 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.574026  6251 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.617239 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.625140 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.619580 31753 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.626504 31825 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 20 index: 12. (term mismatch)
I20260709 12:23:46.589231  6223 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 12:23:46.625383 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.633215 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.606280  6498 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:46.637108 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:46.639192 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 283801us
I20260709 12:23:46.627063 31825 pending_rounds.cc:85] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 11
I20260709 12:23:46.639233 31745 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:23:46.640056 31825 pending_rounds.cc:107] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 12:23:46.640407 31825 raft_consensus.cc:2889] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:46.639659 31747 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:23:46.642681 31744 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:23:46.634752 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:46.646977 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:46.647704 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:46.648003 31753 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:46.648511 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.649355 31674 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:46.650666 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.651512 31745 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:46.620425  6421 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:46.652382 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:46.653785 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.653950  6587 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:46.655486 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:46.647750 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.658965 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.659231  6331 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:46.659734 31675 raft_consensus.cc:2410] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:23:46.659278  6516 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:46.661463 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:46.662173 31745 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:46.668210 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:46.610728  6599 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:46.669750  6599 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:46.670713  6599 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 14.8, Last appended by leader: 8, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:46.675457  6600 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.060s	user 0.002s	sys 0.000s
I20260709 12:23:46.675809  6455 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
W20260709 12:23:46.676579  6563 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.566s	user 0.010s	sys 0.001s
W20260709 12:23:46.677273  6460 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.511s	user 0.014s	sys 0.000s
I20260709 12:23:46.677063  6563 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:46.669258 31747 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:46.676772 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:46.680258 31823 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:46.680470 31745 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:46.681865 31669 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:46.615022  6247 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:46.684862 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:46.596733  6497 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:46.688956 31753 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:23:46.690973 31671 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:46.692472 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.706277 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:46.713824 31751 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
W20260709 12:23:46.719398 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.690380  6607 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:23:46.716343  6460 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:46.719549  6611 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.720244  6460 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.723730  6460 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:46.720948  6587 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.686556  6548 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:46.709578  6575 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:46.727679  6575 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:46.728389  6434 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:46.729120  6434 consensus_queue.cc:1243] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 22.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
W20260709 12:23:46.728832  6223 consensus_queue.cc:1175] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 20 status { last_received { term: 19 index: 14 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:23:46.729879  6614 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:23:46.720755  6611 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:46.731447  6614 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:46.720391 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 118928us
I20260709 12:23:46.732374  6614 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.732875 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.733775  6611 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:46.721055 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.734539 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.742453 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:46.743266 31753 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:46.747623 31745 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:46.748097 31745 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:46.735966  6455 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:46.736002  6497 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:46.730563  6251 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:46.691512  6489 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.057s
I20260709 12:23:46.686874  6602 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:46.723124  6342 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:46.759796 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45276.9us
W20260709 12:23:46.796285 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61812us
I20260709 12:23:46.797797 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.759930  6421 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:46.734169  6612 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:23:46.763980 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:46.759299 31745 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.782176 31750 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:46.787348 31753 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
W20260709 12:23:46.783636 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 176342us
I20260709 12:23:46.758800  6616 raft_consensus.cc:2764] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
W20260709 12:23:46.795131 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 86951.6us
I20260709 12:23:46.748636  6607 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.041s
I20260709 12:23:46.809327 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:46.789324  6223 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
W20260709 12:23:46.829210 31791 leader_election.cc:336] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.935s (SENT)
I20260709 12:23:46.829778 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.803398  6594 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:46.831279 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:46.835043 31745 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:46.790493  6497 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20260709 12:23:46.819401  6251 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.851125  6487 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.852149  6487 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:46.838438  6496 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.858819  6620 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.021s
I20260709 12:23:46.859153  6496 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:46.862136  6496 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:46.832489  6594 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:46.834504  6507 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:46.867902 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:46.841064  6612 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:46.838614  6607 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:23:46.887317 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.865921 31745 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:23:46.840396  6548 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.030s
I20260709 12:23:46.868899 31677 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:46.870955  6588 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:23:46.843799  6421 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:46.866060  6602 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:46.871789  6563 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:46.876468  6516 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:46.880599  6507 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.831975  6303 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.865809  6594 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:46.866807  6487 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:46.901629 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:46.894758 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.895607  6620 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 12:23:46.901265 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.898696 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:46.905581 31824 raft_consensus.cc:2410] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:23:46.900035 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:46.904273 31818 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:46.906205 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:46.906291 31823 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:46.907765 31825 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
W20260709 12:23:46.908512  6597 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.566s	user 0.002s	sys 0.001s
W20260709 12:23:46.910445 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.912788 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:46.913720 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:46.894057  6489 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 9, Time since last communication: 0.020s
I20260709 12:23:46.914755 32374 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 22 because replica has last-logged OpId of term: 21 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 16.
I20260709 12:23:46.914809  6489 consensus_queue.cc:1243] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 19.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:23:46.915592  6489 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:46.916024  6489 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:46.917068  6489 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.919307 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:46.920296 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:46.864316  6251 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:46.919555  6610 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.007s
I20260709 12:23:46.892486  6599 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:46.927946 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:46.933506 31822 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:46.933948 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.934288  6614 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:46.934481 31827 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:23:46.934820  6611 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:46.935284 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.935565  6611 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:46.936498 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:46.928040  5844 consensus_queue.cc:1175] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 20 status { last_received { term: 17 index: 10 } last_committed_idx: 10 error { code: PRECEDING_ENTRY_DIDNT_MATCH status { code: ILLEGAL_STATE message: "Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:23:46.940016  6303 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:46.936333  6611 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:46.893355  6612 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:46.941872 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:46.897101  6516 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:46.900843  6607 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:46.969316 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:46.994371 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:46.948849 31821 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
W20260709 12:23:46.970870 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50514.3us
I20260709 12:23:46.968735  5759 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:46.971678 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:46.971900  6610 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:46.972800  6612 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.976981 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:46.978089  6516 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:46.979123  6498 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:46.927080  6489 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:46.983625  6507 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:46.930401  6497 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:46.943177  6587 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election lost for term 16. Reason: could not achieve majority
I20260709 12:23:46.990965  6599 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:46.992400  6596 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.574s	user 0.003s	sys 0.000s
I20260709 12:23:46.993216  6251 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:46.928498  6579 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:46.993866 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 127495us
I20260709 12:23:46.941102  6575 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.002254 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:47.005196 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:47.064845 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59907.1us
I20260709 12:23:47.006042  6501 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:47.003504  5759 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.022701  6516 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:47.023082  6623 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:23:47.023406  6618 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:47.027215  6620 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:23:47.027767  6621 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
I20260709 12:23:47.027776  6303 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.028620  6610 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:47.031026  6607 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.021997 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:47.042670 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:47.044299 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 72724.6us
I20260709 12:23:47.013633  6611 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:47.044955  6460 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.049962 31823 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:23:47.052399 31748 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: [16.11-16.11]
I20260709 12:23:47.037961 32372 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:47.055210 31744 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:47.056764  6587 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:47.057377 31746 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:23:47.060824  6247 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:47.067636 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 65578.2us
I20260709 12:23:47.068069 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:47.065397  6452 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.080734 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:47.049980  6594 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.070334  6501 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:47.086431 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 240245us
I20260709 12:23:47.054704  6507 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
I20260709 12:23:47.069662  6331 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:47.069837  5759 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:47.089989 31745 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:47.068053  6575 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:47.093309  6587 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.094607  6452 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:47.125581 31745 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:47.088284  6605 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.005s
I20260709 12:23:47.122720  6618 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:47.085063  6292 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:47.127560 31819 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:47.128062 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:47.076189  6497 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 12:23:47.085466  6303 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:47.131829 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 116392us
I20260709 12:23:47.119652  6501 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:47.123575  6620 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:47.110065  6421 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:47.148073 31745 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.087568  6485 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.045s
I20260709 12:23:47.151865 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:47.133569  6594 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:47.157140 31827 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:47.129667  6634 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.059s
I20260709 12:23:47.149147  6292 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:47.198526 31745 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:47.134903  6614 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.025s
I20260709 12:23:47.205686  6218 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:47.211949  6218 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 15 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:47.214164  6587 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:47.219537 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 98418.6us
I20260709 12:23:47.225179  5759 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.226653 31745 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 12:23:47.227200 31745 pending_rounds.cc:85] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 11
I20260709 12:23:47.227718 31745 pending_rounds.cc:107] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 21.12
W20260709 12:23:47.235973 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 84186.5us
I20260709 12:23:47.237147 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:47.196600  6292 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:47.203433  6606 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:47.228107 31745 raft_consensus.cc:2889] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:47.276602  5759 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:47.209124  6611 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.209534  6507 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:47.287336 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:47.287863 31718 connection.cc:256] RPC call timeout handler was delayed by -2.0818579110000428s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1577s for a response
I20260709 12:23:47.288930 32378 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
W20260709 12:23:47.290761 31718 connection.cc:256] RPC call timeout handler was delayed by -1.9144579110002269s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1763s for a response
W20260709 12:23:47.291203 31718 connection.cc:256] RPC call timeout handler was delayed by -1.4986579110000093s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22250000000000003s for a response
I20260709 12:23:47.301292 32374 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 20: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:47.303460 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:47.229269  6599 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:47.207520  6594 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.078s
I20260709 12:23:47.157056  6639 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.132s
I20260709 12:23:47.276789  6421 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:47.282040  6606 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:47.289600  6616 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
W20260709 12:23:47.314749 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 77724.3us
I20260709 12:23:47.288286  6644 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.108s
I20260709 12:23:47.310695  6634 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.313835  6460 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:47.332369 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 204404us
I20260709 12:23:47.333643  6452 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:47.313176  6620 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.335045  6639 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:47.336589  6303 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:47.228732  6610 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.341869 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:23:47.342854 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 200520us
I20260709 12:23:47.342957 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:47.343781 31823 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:47.320472  6606 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:47.301719  6611 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:47.349108 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:47.351998 31747 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:47.352820 31821 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:23:47.354601 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:47.355343 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:47.345683 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 190958us
W20260709 12:23:47.346907 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 226278us
I20260709 12:23:47.355418 31823 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:47.365466 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:47.366442 31753 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:47.368839 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:47.370013 31823 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:47.380908 31820 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 19 because replica has last-logged OpId of term: 17 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:23:47.372819 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:47.346436  6247 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:47.391314 31750 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 12. (term mismatch)
I20260709 12:23:47.391927 31750 pending_rounds.cc:85] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 11
I20260709 12:23:47.392213 31750 pending_rounds.cc:107] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260709 12:23:47.392498 31750 raft_consensus.cc:2889] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:47.385689 31821 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:23:47.387079 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:47.394129 31823 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:47.353694  6303 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:23:47.398926 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 57146.2us
I20260709 12:23:47.399978 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:47.383587 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 68971.8us
I20260709 12:23:47.443017 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:47.374811  6575 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:47.456665 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:47.312793  6618 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:47.356264  6331 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.088s
I20260709 12:23:47.357980  6460 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.050s
I20260709 12:23:47.341442  6634 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:47.457340 31747 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:23:47.397063  6303 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.469262 31823 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:47.470661 31753 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:47.446456  6643 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:47.485371 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:47.491024 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:47.462015  6602 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:47.491360  6012 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:47.475886  6303 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:47.484282  6643 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:47.463083  6247 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.096s
W20260709 12:23:47.623682  6627 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.065s	user 0.000s	sys 0.001s
I20260709 12:23:47.624092  6647 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.266s
I20260709 12:23:47.573943  6614 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.489917  6648 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.491868  6587 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:47.579425  6599 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:23:47.479879  6651 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.139s
I20260709 12:23:47.722788  6614 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:47.638010 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 147063us
I20260709 12:23:47.733196 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:47.735265  6599 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:47.580785 31823 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:23:47.567360  6489 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.060s
W20260709 12:23:47.672535  6650 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.214s	user 0.000s	sys 0.001s
I20260709 12:23:47.764530  6647 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.006s
I20260709 12:23:47.683568  6648 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:47.766167  6647 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:47.766708  6647 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:47.700307  6634 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:47.768585  6489 consensus_queue.cc:1243] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 22.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:23:47.767567  6649 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:47.776590  6649 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:47.777232  6516 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.697s	user 0.012s	sys 0.000s
I20260709 12:23:47.702481  6643 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:47.704562  6641 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.390s	user 0.001s	sys 0.001s
I20260709 12:23:47.705082  6507 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:47.780637  6507 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:47.705760  6646 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.375s	user 0.000s	sys 0.002s
I20260709 12:23:47.707654  6611 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:47.715070  6587 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:47.785385  6303 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.621335  6656 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.103s
W20260709 12:23:47.656273 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 81899.5us
I20260709 12:23:47.767634 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:47.713735  6292 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:47.770732  6648 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:47.770933  5949 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:47.785254  6587 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:47.716707  6654 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.788906  6643 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:47.778390 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:47.791102 31823 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 15 index: 8. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:47.823405 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:47.795020 31827 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.612s	user 0.008s	sys 0.003s
I20260709 12:23:47.795091 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:47.726352  6614 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:47.793973  6579 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.715s	user 0.014s	sys 0.000s
I20260709 12:23:47.790374  6507 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:47.824941 31753 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:23:47.824560 31827 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:47.829236 31822 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:47.829046 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:47.838186 31823 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:47.813655  6460 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:47.829281 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:47.843618 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:47.848747 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:47.849931 31749 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:47.834204  6579 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:47.841351  6276 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:23:47.868518 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 66739.3us
I20260709 12:23:47.869685 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:47.871272 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:47.871446 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:47.786697  6647 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:47.861946 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:47.834867  6611 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:47.877456 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:47.875695 31820 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:47.878410 31676 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 24 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
W20260709 12:23:47.844362  6604 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.608s	user 0.005s	sys 0.012s
I20260709 12:23:47.882498 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:47.883630 31675 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:47.884603 31818 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:23:47.884769 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:47.874152  6614 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:47.888336 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:47.888166  6614 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:47.889041 31671 raft_consensus.cc:2376] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20. Re-sending same reply.
I20260709 12:23:47.890383  6647 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:47.890906 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:47.891598 31677 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:47.877446  6656 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:23:47.896420 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.860745  6507 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:47.897329  6507 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.898141 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:47.899035 32378 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:47.899185 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:47.899695  6507 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:47.854523  6649 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:47.902123 31675 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:23:47.902204 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:47.826401  6654 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.903442 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:47.904698  6654 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:47.886502 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 118211us
I20260709 12:23:47.905557  6218 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 15.9, Last appended by leader: 9, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.906616 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:47.907588  6292 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:47.845554  6460 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.908517  6247 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:47.909991  6460 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:47.910275  6497 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:47.868638  6276 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:47.910800  6497 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:47.911569  6276 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.911782  6497 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.912638  6276 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:47.913206 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:47.913605 31745 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:23:47.916200 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3200) took 1391 ms. Trace:
I20260709 12:23:47.916327  6223 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:47.916699 31819 rpcz_store.cc:276] 0709 12:23:46.525307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.560060 (+ 34753us) service_pool.cc:224] Handling call
0709 12:23:46.577794 (+ 17734us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:47.145240 (+567446us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:47.915980 (+770740us) spinlock_profiling.cc:243] Waited 549 ms on lock 0x7b54000fc654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:47.916137 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1263813376}
I20260709 12:23:47.916908  6223 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:47.916349  6303 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:47.918484 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:47.886482 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:47.873529  6501 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.718s	user 0.012s	sys 0.002s
I20260709 12:23:47.886938 31674 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 24 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:23:47.907896 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:47.925601 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:47.881983  6604 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.728s	user 0.005s	sys 0.012s
I20260709 12:23:47.925128 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:47.936816  6430 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:47.937256  6430 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.937630  6430 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:47.935654 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.938944  6575 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:47.947177 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:47.852639  6606 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:47.931944  6656 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:47.919632 31568 catalog_manager.cc:5697] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 11 to 14. New cstate: current_term: 14 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:47.964269  6640 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:47.964823  6640 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:47.968801 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 1201) took 1067 ms. Trace:
I20260709 12:23:47.969321 32372 rpcz_store.cc:276] 0709 12:23:46.901173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.901338 (+   165us) service_pool.cc:224] Handling call
0709 12:23:47.968566 (+1067228us) spinlock_profiling.cc:243] Waited 136 ms on lock 0x7b54005c30d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:47.968718 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":311657216}
W20260709 12:23:47.970893 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 99556.5us
I20260709 12:23:47.919313 31820 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:23:47.924335  6218 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:47.918619 31819 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:23:47.978117 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:47.979115 31819 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:47.979483 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:47.975819 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:47.975286 31749 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:23:47.926501  6497 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:47.935364  6303 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:47.990298 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:47.992391 31820 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:23:47.965724  6640 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:47.972886  6421 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:23:48.007274 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:47.924566  6632 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.070s
I20260709 12:23:47.919783  6588 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.075s
I20260709 12:23:47.995131  6303 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:47.932307  6501 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.025557 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:48.026335 31820 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:48.006716  6611 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:48.036893 31670 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:23:48.029119  6501 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:48.007427  6421 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:48.042873 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.046955  6012 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.047519  6012 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.049597  6012 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:48.050136  6012 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.026397  6303 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:48.015762  6640 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:48.050724 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.015308  6614 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:48.050945 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:48.057452 31820 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:48.063493 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:48.052608 32373 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:48.009461  6610 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.043s
I20260709 12:23:48.067893 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.068776 31678 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:23:48.070555 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:48.064817 31673 raft_consensus.cc:2376] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21. Re-sending same reply.
I20260709 12:23:48.074352 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.083914 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:48.058707  6645 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.599s	user 0.007s	sys 0.005s
I20260709 12:23:48.087123 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:48.090291 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.056571  6620 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.092654 31678 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:23:48.092929  6645 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.640s	user 0.007s	sys 0.006s
I20260709 12:23:48.071421  6643 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.108388 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:48.110169 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.110414 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:48.042714  6656 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 12:23:48.115417 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:48.137957 31820 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: [21.12-21.12]
I20260709 12:23:48.113034 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:48.099682  5949 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.093361  6594 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.139329 31673 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:23:48.147785 31825 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:48.050578  6012 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.158152 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:48.164274 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:48.135999  6599 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:48.115196  6223 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:48.155869  6614 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:23:48.170584  6660 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:48.189524 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:48.190485 31820 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:48.171236  6609 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:23:48.171931 31567 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 239) took 31043 ms (client timeout 15000 ms). Trace:
I20260709 12:23:48.172251 31818 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:48.173207 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.167191 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:48.196820 31678 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:48.197883 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.198469 31678 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:48.199360 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.174734 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:48.201354 31675 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:48.112804  6661 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.066s
I20260709 12:23:48.050462 31568 catalog_manager.cc:5697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 10 to 18, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 18 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:23:48.177130  6602 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.594s	user 0.006s	sys 0.006s
I20260709 12:23:48.206486 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.188918  6660 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.176390  6606 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:48.207150  6662 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.153s	user 0.001s	sys 0.001s
I20260709 12:23:48.160015  5949 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.214030 31675 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:48.214979 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.187875 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:48.238166 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:48.238788 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51055.9us
I20260709 12:23:48.239972 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:48.210966 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:48.116103  6620 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.204715  6501 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.145714  6652 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.094s
I20260709 12:23:48.162678  6604 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.050s
I20260709 12:23:48.137066  6421 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.241511 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:48.161625  6643 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.245537 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.220330  6602 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:48.276571 31672 raft_consensus.cc:2410] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 16 because replica has last-logged OpId of term: 15 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 8.
W20260709 12:23:48.250232 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 139877us
I20260709 12:23:48.214505  6599 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:48.212235  6609 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:48.169500  6594 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.247865 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:48.277504 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:48.282377 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:48.272506 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.276427 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:48.274050 31675 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:48.285188 31675 pending_rounds.cc:85] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 12
I20260709 12:23:48.285600 31675 pending_rounds.cc:107] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 22.13
I20260709 12:23:48.285955 31675 raft_consensus.cc:2889] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:48.286898  6516 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 22.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:23:48.288489  6670 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
W20260709 12:23:48.290072  6664 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
W20260709 12:23:48.291726  6667 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.084s	user 0.001s	sys 0.000s
W20260709 12:23:48.293609  6676 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:23:48.295617 31676 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
W20260709 12:23:48.296602  6666 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
I20260709 12:23:48.249934  6331 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:48.297540 31825 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:23:48.279156  6599 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:48.298166  6599 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:48.271986  6651 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 20.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.047s
I20260709 12:23:48.299939 31825 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
W20260709 12:23:48.301762 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60320.6us
I20260709 12:23:48.302665 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:48.303155  6620 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:48.303506  6611 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:48.304302  6611 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:48.297559  6331 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:48.305563  6331 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.253224  6659 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.059s
I20260709 12:23:48.307658 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:48.308166  6594 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:48.308650 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:48.308810  6594 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:48.309285  6594 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:48.309535 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:48.310864  6649 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:48.248315  6507 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:48.309688  6594 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:48.243001  6587 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:48.314345 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:48.315659 31669 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:48.235642  6660 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:48.311298  6649 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:48.310424 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:48.326740 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50470.6us
I20260709 12:23:48.327157 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:48.321434 31717 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:48.327574 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:48.250120  6645 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:48.242704  6489 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:48.308059  6659 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:23:48.332207 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3205) took 1433 ms. Trace:
I20260709 12:23:48.332257 31820 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:48.334703  6421 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:48.335388  6682 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:23:48.253924  6501 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.319748  6680 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 20.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.019s
I20260709 12:23:48.339490  6680 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:48.339855  6680 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:48.340590  6501 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:48.340719  6680 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.342133  6680 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:48.320199  6654 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:48.342929  6654 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.343351  6654 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:48.332818 31826 rpcz_store.cc:276] 0709 12:23:46.898594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:46.905253 (+  6659us) service_pool.cc:224] Handling call
0709 12:23:46.905536 (+   283us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:46.905876 (+   340us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:46.905903 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:46.912237 (+  6334us) log.cc:844] Serialized 26 byte log entry
0709 12:23:47.925178 (+1012941us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b18003a47a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:23:48.095988 (+170810us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:48.096044 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.096126 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:48.323939 (+227813us) raft_consensus.cc:1685] finished
0709 12:23:48.323962 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:48.332140 (+  8178us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7496,"raft.queue_time_us":119,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":9393,"spinlock_wait_cycles":23545600,"thread_start_us":162763,"threads_started":1,"wal-append.queue_time_us":1243439}
I20260709 12:23:48.315531  6218 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:48.315006  6507 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.315861  6610 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:48.329123  6649 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:48.345223  6602 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:48.283396  6656 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.067s
I20260709 12:23:48.345369  6659 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:48.356139  6218 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:48.373974 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:48.372014 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44518.6us
I20260709 12:23:48.359434  6620 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:48.411291  6604 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.412242  6604 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.412573  6599 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.359498  6507 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:48.360093  6610 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.384761 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:48.384802 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:48.380508 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 59155.9us
I20260709 12:23:48.419219 31820 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:48.387638  6247 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:48.390945  6674 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:48.392693 31568 catalog_manager.cc:5697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 14 to 15, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 15 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:48.420735  6674 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:48.394819  5949 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:48.399899 31677 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:48.354126  6489 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.403172  6643 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:48.196125 31567 rpcz_store.cc:269] 0709 12:23:17.128077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:17.128419 (+   342us) service_pool.cc:224] Handling call
0709 12:23:48.168060 (+31039641us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:37.046175 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:37.219174 (+172999us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:37.241692 (+ 22518us) write_op.cc:435] Acquired schema lock
0709 12:23:37.248802 (+  7110us) tablet.cc:662] Decoding operations
0709 12:23:37.379624 (+130822us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:37.379720 (+    96us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:37.379752 (+    32us) tablet.cc:685] Acquiring locks for 19 operations
0709 12:23:37.381301 (+  1549us) tablet.cc:701] Row locks acquired
0709 12:23:37.381353 (+    52us) write_op.cc:260] PREPARE: finished
0709 12:23:37.381565 (+   212us) write_op.cc:270] Start()
0709 12:23:37.381753 (+   188us) write_op.cc:276] Timestamp: P: 1783599817381484 usec, L: 0
0709 12:23:37.381800 (+    47us) op_driver.cc:348] REPLICATION: starting
0709 12:23:37.409161 (+ 27361us) log.cc:844] Serialized 7534 byte log entry
0709 12:23:38.837722 (+1428561us) op_driver.cc:464] REPLICATION: finished
0709 12:23:39.227165 (+389443us) write_op.cc:301] APPLY: starting
0709 12:23:39.227286 (+   121us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:39.586584 (+359298us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:39.586755 (+   171us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:39.803547 (+216792us) spinlock_profiling.cc:243] Waited 2.69 ms on lock 0x7b1800405f40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:39.949144 (+145597us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:39.949701 (+   557us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=38,key_file_lookups=38,delta_file_lookups=0,mrs_lookups=0
0709 12:23:39.949775 (+    74us) write_op.cc:312] APPLY: finished
0709 12:23:39.950328 (+   553us) log.cc:844] Serialized 172 byte log entry
0709 12:23:40.098708 (+148380us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:40.098972 (+   264us) write_op.cc:454] Released schema lock
0709 12:23:40.099084 (+   112us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:40.099783 (+   699us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:40.099827 (+    44us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":136,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":353484,"apply.run_cpu_time_us":47434,"apply.run_wall_time_us":872574,"cfile_cache_hit":78,"cfile_cache_hit_bytes":83509,"num_ops":19,"prepare.queue_time_us":278031,"prepare.run_cpu_time_us":29329,"prepare.run_wall_time_us":755935,"raft.queue_time_us":524779,"raft.run_cpu_time_us":24190,"raft.run_wall_time_us":356215,"replication_time_us":1404652,"spinlock_wait_cycles":6181632,"thread_start_us":935505,"threads_started":5,"wal-append.queue_time_us":488002,"wal-append.run_cpu_time_us":49072,"wal-append.run_wall_time_us":2762406}]]}
I20260709 12:23:48.385685  6218 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:48.415678 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:48.414341  6604 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:48.417340 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 86462.8us
W20260709 12:23:48.447985 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.355827  6661 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 12:23:48.386392  6683 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:23:48.383433  6659 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:48.441773 31669 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
W20260709 12:23:48.442878 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58207.9us
I20260709 12:23:48.449478 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:48.421937 31568 catalog_manager.cc:5697] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 14 to 15, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 15 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:48.435516  6610 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:23:48.444715  6276 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.526s	user 0.009s	sys 0.005s
I20260709 12:23:48.439302  6674 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:48.436501  6599 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:48.448910  6643 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:48.469352 31678 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 24 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:23:48.446218  5949 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:23:48.418286  6609 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.475232 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:48.498811  6430 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.561s	user 0.013s	sys 0.001s
I20260709 12:23:48.499377  6430 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.501778  6430 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:48.505481 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3159) took 1463 ms. Trace:
I20260709 12:23:48.506047 31750 rpcz_store.cc:276] 0709 12:23:47.041616 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.041776 (+   160us) service_pool.cc:224] Handling call
0709 12:23:47.062663 (+ 20887us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:47.393078 (+330415us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.505189 (+1112111us) spinlock_profiling.cc:243] Waited 328 ms on lock 0x7b54000b8bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:48.505405 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":754085120}
W20260709 12:23:48.501842 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:48.515236 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.512912  6689 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:48.518584 31676 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: []
I20260709 12:23:48.521024 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:48.522289 31825 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:48.479019  6602 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:48.530722 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55575.7us
I20260709 12:23:48.527261 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.536034 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:48.536993 31825 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:48.539235  6489 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:48.540139 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.540915 31676 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:48.542433 31676 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:23:48.477461  6303 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.476596  6674 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:48.546516 31825 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: []
I20260709 12:23:48.488376  6647 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.044s
I20260709 12:23:48.484619  6683 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.055s
I20260709 12:23:48.510424  6654 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.562016 31825 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:48.531284  6616 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:48.567552 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:48.568499 31676 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:48.569736 31825 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:48.651077 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:48.652040 32374 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:48.652915 31825 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
W20260709 12:23:48.572089 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 122961us
I20260709 12:23:48.558027  6639 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.567901  6489 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:48.571645  6588 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.672739 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:48.672737 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:48.658519  6611 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:48.688045  6674 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:48.666258  6697 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.698734  6303 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:48.699507  6697 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.674927  6602 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.666016  6639 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.706741  6656 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:48.707504  6656 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.707952  6656 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:48.712064 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:48.717878 32372 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:48.723371  6639 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:48.727350  6645 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.002s
W20260709 12:23:48.729456 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 800314us
I20260709 12:23:48.737800 31673 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 9. Preceding OpId from leader: term: 16 index: 9. (term mismatch)
I20260709 12:23:48.739797  6498 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:48.740278  6498 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:48.738821 31673 pending_rounds.cc:85] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 8
I20260709 12:23:48.741417 31673 pending_rounds.cc:107] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 15.9
I20260709 12:23:48.741555  6012 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:48.727731 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.741703 31673 raft_consensus.cc:2889] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:48.743980 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:48.762476 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:48.764189 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:48.765765 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:48.776708 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64736us
I20260709 12:23:48.778065 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:48.780018 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:48.780958 31748 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:23:48.783715 31744 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:23:48.785996 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:48.782303 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:48.785079 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:48.792896 31744 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 16 index: 9. (index mismatch)
I20260709 12:23:48.793833 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:48.790061 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:48.797983 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:48.799746 31750 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: [20.12-20.12]
I20260709 12:23:48.802758 31751 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16. Ops: [15.9-15.9]
I20260709 12:23:48.804366 31748 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:48.806128 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:48.807000 31746 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:23:48.722385  6587 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:23:48.779560  6498 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.813294  6498 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:48.775590  6640 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.814752 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:48.815267  6640 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.815866 31745 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:48.818099  6640 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:48.756033  6689 raft_consensus.cc:2764] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:23:48.798345  6421 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:23:48.791374  6683 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:48.820653  6683 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.791743  6644 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.823240  6644 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.831058  6639 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:48.831790  6639 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.710287  6588 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.834798  6639 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:48.834873  6697 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:48.837013  6588 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:48.797699  6455 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:48.838140  6455 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:48.838603  6455 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:48.775563  6700 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:48.839941  6700 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.844022  6611 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:48.844614  6611 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:48.844707  6697 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.845060  6611 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:48.845896 31821 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:48.845523  6697 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.845597 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 1248) took 1780 ms. Trace:
I20260709 12:23:48.847213 31827 rpcz_store.cc:276] 0709 12:23:47.064800 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.065017 (+   217us) service_pool.cc:224] Handling call
0709 12:23:48.845400 (+1780383us) spinlock_profiling.cc:243] Waited 68.2 ms on lock 0x7b540037a354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:48.845514 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":156892928}
I20260709 12:23:48.849386 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:48.676268  6489 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.852108 32370 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:48.776273  6092 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.720255  6610 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.842466  6700 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:48.834292  6507 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.877569 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:48.879987 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:48.892633 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:48.899088  6563 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:48.899890  6563 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:48.884558  6640 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.901533  6640 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:48.864926  6680 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.522s	user 0.011s	sys 0.006s
I20260709 12:23:48.904337 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:48.908731 32376 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:48.930189 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:48.852120  6649 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.902743  6588 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.935014 31747 raft_consensus.cc:2410] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 22 because replica has last-logged OpId of term: 21 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 10.
I20260709 12:23:48.934971  6640 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:48.905138  6587 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:48.935860 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:48.936317 32378 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:48.957129 32376 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:48.938606 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:48.896409  6657 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:48.942965  5949 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:48.902514  6563 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:48.860816  6092 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.951112 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:48.968927 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:48.936630 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:48.970300 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:48.959810 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:48.965229 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 87770us
I20260709 12:23:48.971131 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:48.896319  6701 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:48.972929 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:49.008140 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:49.009689 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:48.965986  6660 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.631s	user 0.006s	sys 0.006s
I20260709 12:23:49.028307 31751 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:23:49.030189 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:49.033777 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:49.058400 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:49.049832 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 79608.5us
I20260709 12:23:49.109802 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:48.980608 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:23:49.139299 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:49.139945 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.012349 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:49.141310 32376 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:49.033352 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:49.143618 32377 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
W20260709 12:23:49.083175 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 221789us
I20260709 12:23:49.094913 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:49.147200 31748 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 14.
W20260709 12:23:49.138571 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:49.149657 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.108867 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:49.152235 31753 raft_consensus.cc:2410] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 19 because replica has last-logged OpId of term: 17 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 12.
I20260709 12:23:49.153862 31753 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:49.162213 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:48.970046  6594 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.639s	user 0.009s	sys 0.002s
I20260709 12:23:49.167594 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:49.161674 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:49.170069 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:49.171231 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.171360 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:49.172217  6507 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:49.173013 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:49.192960 31672 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:48.988129  6563 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:48.987735  6683 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:49.174659  6659 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.706s	user 0.012s	sys 0.000s
I20260709 12:23:49.175428  6709 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:49.175740 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:49.176606  6610 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:48.970934  5949 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:48.966161  6588 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:49.057936  6611 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.214440  6611 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:49.190325  6460 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.215802  5949 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:49.216144 32375 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:23:49.161433  6660 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.191146  6661 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.016052  6633 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.217402  6661 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.218001  6633 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:49.201687  6706 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:49.219084  6706 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.219599  6661 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:49.219913 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3218) took 1839 ms (client timeout 1928 ms). Trace:
I20260709 12:23:49.220326  6633 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:49.198525  6501 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:49.220695 31819 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: [21.11-21.11]
I20260709 12:23:49.201004  6659 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.222023 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3102) took 1448 ms. Trace:
I20260709 12:23:49.223125  6659 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:49.220345 31822 rpcz_store.cc:269] 0709 12:23:47.380861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.381120 (+   259us) service_pool.cc:224] Handling call
0709 12:23:49.219692 (+1838572us) spinlock_profiling.cc:243] Waited 370 ms on lock 0x7b540030ded4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:49.219852 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":850699520}
W20260709 12:23:49.181545  6218 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.729s	user 0.008s	sys 0.002s
I20260709 12:23:49.226709 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:49.016788  6701 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.231177 31674 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:23:48.864665 31568 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 11 to 12, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 12 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:49.171484  6489 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:49.055305  6643 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:49.108760  6485 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.182837  6697 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:49.017355  6644 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:49.108946  6656 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:49.137169  6455 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:49.184655 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.041285 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:49.251313 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:49.251605 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 210400us
W20260709 12:23:49.254261 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.039837  6640 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:49.055995  6612 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:49.173563  6607 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.193576 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:49.049346  6649 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:48.981091  6654 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:49.195087 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:49.296867 31823 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:49.192138 31675 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:49.050379  6247 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:49.147559  6251 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.161634  6614 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.204056 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:49.200692  6602 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.207146 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:49.190397  6331 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:49.179080 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:49.222433 31819 rpcz_store.cc:276] 0709 12:23:47.773289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.980600 (+207311us) service_pool.cc:224] Handling call
0709 12:23:49.220440 (+1239840us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:49.221172 (+   732us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.221826 (+   654us) spinlock_profiling.cc:243] Waited 1.24 s on lock 0x7b540030ded0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:49.221961 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2850422528}
W20260709 12:23:49.232227 31824 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.523s	user 0.020s	sys 0.025s
I20260709 12:23:49.236672 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.309023 32378 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
W20260709 12:23:49.309343 31824 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.600s	user 0.020s	sys 0.025s
I20260709 12:23:49.218406  6660 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:49.200425  6563 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.312418  6563 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:49.312757  6430 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:23:49.313524 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:49.314023  6633 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:49.314199 31820 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:23:49.314483  6633 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.314960  6633 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:23:49.315783  6711 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:23:49.317513 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:49.318410 31818 raft_consensus.cc:2410] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:23:49.319208 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3264) took 1425 ms. Trace:
I20260709 12:23:49.319833 31670 rpcz_store.cc:276] 0709 12:23:47.893965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.919789 (+ 25824us) service_pool.cc:224] Handling call
0709 12:23:47.920215 (+   426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:48.043705 (+123490us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.318841 (+1275136us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54002e6b54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:49.319121 (+   280us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":253234944}
I20260709 12:23:49.326813  6602 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.329844 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:49.273460  6276 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.330932 31824 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:23:49.331045  6276 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:49.333117  6709 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:23:49.333439  6276 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:49.333559  6709 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:49.240706  5949 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.065s
I20260709 12:23:49.334113 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1255) took 1409 ms. Trace:
I20260709 12:23:49.334777  6092 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:49.335093  6251 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.304665  6607 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:49.336871 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.314821  6331 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:49.339413  6713 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.166s	user 0.003s	sys 0.000s
I20260709 12:23:49.307972  6659 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:49.340044  6659 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.340420  6659 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:49.334952  6614 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.309563 31676 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
W20260709 12:23:49.342710  6704 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.171s	user 0.002s	sys 0.000s
I20260709 12:23:49.342911 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.343408 31826 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:49.343665 31676 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:23:49.344575 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:49.345338  6710 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.169s	user 0.000s	sys 0.002s
I20260709 12:23:49.334583 31821 rpcz_store.cc:276] 0709 12:23:47.924979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.379713 (+454734us) service_pool.cc:224] Handling call
0709 12:23:48.415531 (+ 35818us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.846313 (+430782us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.333824 (+487511us) spinlock_profiling.cc:243] Waited 430 ms on lock 0x7b5400196ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:49.334044 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":989227904}
I20260709 12:23:49.349303  6656 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:49.350266 31821 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:49.351066  6643 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.352867 31827 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:49.240166  6651 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.065s
I20260709 12:23:49.355399  5949 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:23:49.355794  5949 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:49.356052 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.356613  5949 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.357970  5949 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:49.358167 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:49.245862  6485 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.329869 31568 catalog_manager.cc:5697] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 10 to 12. New cstate: current_term: 12 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:23:49.362494 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3267) took 1427 ms. Trace:
I20260709 12:23:49.362998 31674 rpcz_store.cc:276] 0709 12:23:47.934955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.295014 (+360059us) service_pool.cc:224] Handling call
0709 12:23:48.295401 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:49.234307 (+938906us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.362222 (+127915us) spinlock_profiling.cc:243] Waited 931 ms on lock 0x7b5400253ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:49.362428 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2141500800}
I20260709 12:23:49.364476 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:49.365123 31674 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:49.369798 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.370658 31675 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:49.372232 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:49.373428 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 179129us
I20260709 12:23:49.322196 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:49.238462 31672 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:49.237922 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:48.971572  6657 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.226912  6218 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:49.196635  6594 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.233068  6588 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:49.292392  6640 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:49.289739  6697 raft_consensus.cc:2764] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:23:49.293561 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:49.301580 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 108048us
I20260709 12:23:49.345878 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:49.238466  6501 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:49.289745  6612 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:49.215852  6460 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.304877 32369 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:49.306787 31746 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:49.309727 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:49.296586  6421 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:49.303844  6606 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:49.311645 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2359) took 1249 ms. Trace:
I20260709 12:23:49.396835 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:49.397101 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.396564 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:49.397787 31675 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:23:49.396540 31673 rpcz_store.cc:276] 0709 12:23:48.062176 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.301193 (+239017us) service_pool.cc:224] Handling call
0709 12:23:48.736386 (+435193us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:48.756072 (+ 19686us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.311433 (+555361us) spinlock_profiling.cc:243] Waited 433 ms on lock 0x7b540017bbd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:49.311578 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":994833152}
W20260709 12:23:49.380748 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 142896us
I20260709 12:23:49.385993 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:49.389516 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:49.404508 31745 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 12:23:49.404623 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:49.406103 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:49.393038 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 99575.1us
I20260709 12:23:49.406599 31670 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:23:49.393939 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:49.394541 31823 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:49.409641 31669 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:49.410104 31826 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:49.409106 31751 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:23:49.411414 31822 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:49.405718  6640 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:49.416728 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:49.418068 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:49.416929 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:49.391248  6651 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:49.386691  6455 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:49.426403 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:49.388054  5844 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:49.431087 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:49.383342  6655 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.032s
W20260709 12:23:49.441843 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:49.443172 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:49.443840 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.433900 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:49.408195  6689 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:49.428041  6247 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.021s
I20260709 12:23:49.407474  6614 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:49.415889  6649 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:49.458662  6724 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:49.428333  6497 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:23:49.450132 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 271128us
I20260709 12:23:49.472940 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2352) took 1413 ms. Trace:
I20260709 12:23:49.334523  6709 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:49.458878 32369 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:49.459470 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:49.477813 32378 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:23:49.460100 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:49.459848  6612 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.460454  6501 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.428815  6660 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:49.423125  6654 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:49.463071 31672 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:23:49.449971 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:49.410975  6421 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:49.463778 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:49.485834 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:49.435777  6678 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:49.481825 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:49.448818  6611 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:23:49.488111 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:49.459771  6594 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:49.421523  6092 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:23:49.500525 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:49.502238 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:49.503054 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:49.503450 31823 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:49.462198  6716 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.145s	user 0.001s	sys 0.001s
I20260709 12:23:49.484755 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:49.411211  6606 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.450018  6689 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.455636 31822 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: [19.12-21.13]
I20260709 12:23:49.485692  5759 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:49.474697 31678 rpcz_store.cc:276] 0709 12:23:48.060035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.199098 (+139063us) service_pool.cc:224] Handling call
0709 12:23:49.462521 (+1263423us) spinlock_profiling.cc:243] Waited 265 ms on lock 0x7b540036f1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:49.472859 (+ 10338us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":609952000}
I20260709 12:23:49.512115 31819 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:49.449461  6655 consensus_queue.cc:1243] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 18.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:49.417209  6697 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.055s
I20260709 12:23:49.495720 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:49.508342 31820 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:23:49.511039 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:49.571077 31823 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:49.476934  6497 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.506819 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:49.574918 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 68174.2us
I20260709 12:23:49.514139 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3260) took 1621 ms. Trace:
I20260709 12:23:49.480543  6602 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:49.461566  6616 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 16.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.044s
I20260709 12:23:49.454483  6656 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 12:23:49.479823  6251 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:49.447938  5844 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:49.563944 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52896.8us
I20260709 12:23:49.485035  6660 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.479280  6496 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:49.662357 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:49.487435  6647 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.576284 31671 rpcz_store.cc:276] 0709 12:23:47.892746 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:47.916434 (+ 23688us) service_pool.cc:224] Handling call
0709 12:23:47.916804 (+   370us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:47.917172 (+   368us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:47.917205 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:47.929234 (+ 12029us) log.cc:844] Serialized 26 byte log entry
0709 12:23:48.064004 (+134770us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:23:48.128874 (+ 64870us) log.cc:844] Serialized 14 byte log entry
0709 12:23:48.129224 (+   350us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:48.129335 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:49.194323 (+1064988us) spinlock_profiling.cc:243] Waited 1.44 ms on lock 0x7b180021fbe0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:49.353847 (+159524us) spinlock_profiling.cc:243] Waited 5.99 ms on lock 0x7b180021fbe0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:49.513874 (+160027us) raft_consensus.cc:1685] finished
0709 12:23:49.513906 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:49.514065 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":71725,"prepare.run_cpu_time_us":101,"prepare.run_wall_time_us":6935,"raft.queue_time_us":1802,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":6541,"spinlock_wait_cycles":17091712,"thread_start_us":96897,"threads_started":1,"wal-append.queue_time_us":198672}
I20260709 12:23:49.588667 31819 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
I20260709 12:23:49.672091 31819 pending_rounds.cc:85] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 12
I20260709 12:23:49.589570 32375 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:49.489398  6455 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:23:49.559836 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 75153.1us
I20260709 12:23:49.500025  6612 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:49.482561  6607 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:49.496738  6618 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:49.695433 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:49.651813 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:49.489646  6331 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:49.490192  6678 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:23:49.517184  6587 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.546s	user 0.012s	sys 0.000s
I20260709 12:23:49.698854 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:49.500784  6701 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:49.562223  6498 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.748s	user 0.002s	sys 0.013s
I20260709 12:23:49.661324 31823 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:23:49.715559 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.716938 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.717278  6610 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:49.717633 31671 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
W20260709 12:23:49.716734 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54500.8us
I20260709 12:23:49.516206  6689 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:49.485260  6588 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.075s
I20260709 12:23:49.499033  6738 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:49.404464 31568 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 12 to 14, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 14 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:23:49.593765  6497 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:49.560647  6671 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:49.666953 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:49.668099 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 219952us
I20260709 12:23:49.724217 31820 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:49.561290 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.724371 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:49.724857 31678 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:23:49.672472 31819 pending_rounds.cc:107] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 12:23:49.725728 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:49.726046 31819 raft_consensus.cc:2889] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:49.655207  6661 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:49.651804  6489 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:49.655458  6602 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:23:49.698218 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.666546  6660 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:49.699705 31747 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:23:49.667002  6485 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:49.668201  6496 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.699301  6678 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:23:49.711313  6726 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:23:49.695742  6618 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:49.722049  6700 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:49.715260 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 15 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:49.813328 31752 raft_consensus.cc:2410] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 18 because replica has last-logged OpId of term: 16 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 12.
I20260709 12:23:49.719813 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:49.814144 31748 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:23:49.721441 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:49.731371 31678 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 15. Current term is 16. Ops: []
W20260709 12:23:49.722489 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:49.740027 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 428156us
W20260709 12:23:49.744669 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:49.745116 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:49.797322 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:49.676826  6647 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:49.740743 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 306948us
I20260709 12:23:49.715093  6587 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:23:49.723487  6671 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:49.827237  6729 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:49.827857  6729 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:49.829084  6729 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:49.795784  6247 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:49.737792  6460 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:49.732106  6489 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.737648  6602 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.805974 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:49.798732  6496 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:49.857707 31825 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: []
I20260709 12:23:49.839094 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:49.859870 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:49.840049 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:49.810241  6618 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:49.817067  6633 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.501s	user 0.012s	sys 0.024s
I20260709 12:23:49.850294 31670 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:49.851528 31747 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:23:49.746083  6656 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.091s
I20260709 12:23:49.746243  6331 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:49.893165  6331 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:49.866233 31674 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:23:49.824784  6700 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:49.894503  6614 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:49.896914  6700 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:49.867921  6729 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:49.900779 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.902930 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.904349 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:49.905284 31670 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 25 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:23:49.796595  6683 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 12:23:49.873015  6455 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:23:49.873839  6421 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:49.821697  6610 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.086s
I20260709 12:23:49.858173  6602 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:49.868425  6611 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:49.876125  6587 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:49.841652  6736 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:23:49.878988  6616 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:49.879931  5949 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.521s	user 0.012s	sys 0.000s
I20260709 12:23:49.880157  6671 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 20.16, Last appended by leader: 16, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:49.881451 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:49.883651  6588 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:49.816360  6701 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.052s
I20260709 12:23:49.885274  6496 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:49.887810  6092 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 17.11, Next index: 12, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:49.888979  6647 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:49.888636  6633 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:49.892361  6706 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:49.892285  6680 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:23:49.894598 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54655.1us
W20260709 12:23:49.872179  6730 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
I20260709 12:23:50.085179  6496 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:50.086308 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3268) took 1982 ms. Trace:
I20260709 12:23:50.086728 31677 rpcz_store.cc:276] 0709 12:23:48.104024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.457913 (+353889us) service_pool.cc:224] Handling call
0709 12:23:48.471624 (+ 13711us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:49.050803 (+579179us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:49.303627 (+252824us) log.cc:844] Serialized 14 byte log entry
0709 12:23:49.486413 (+182786us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:49.596332 (+109919us) log.cc:844] Serialized 26 byte log entry
0709 12:23:49.608971 (+ 12639us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:49.609046 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.609162 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:50.086080 (+476918us) raft_consensus.cc:1685] finished
0709 12:23:50.086102 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:50.086245 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":826440,"prepare.queue_time_us":257424,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":611865,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":5024,"thread_start_us":407421,"threads_started":2,"wal-append.queue_time_us":349809}
I20260709 12:23:49.955878  6736 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.914098 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:49.902280 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:49.907289 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:49.910892 31670 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:23:49.911887 32375 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
W20260709 12:23:50.101536 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 187542us
I20260709 12:23:49.889495 31568 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 6 to 14, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 14 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:23:49.915417 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 466936us
I20260709 12:23:49.857820  6651 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:49.898353  6218 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:49.871703  5844 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:49.941120 31820 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: [20.11-20.11]
W20260709 12:23:49.935664 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 209961us
I20260709 12:23:49.947383 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3270) took 1842 ms. Trace:
I20260709 12:23:49.968343 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3170) took 1024 ms. Trace:
I20260709 12:23:49.919034  6455 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:49.894522  6331 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:49.916100  6247 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:23:50.080333  6671 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:50.081061  6633 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:50.006572  6588 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.016366  6680 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:50.081786 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63977.4us
I20260709 12:23:49.907140 31674 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:49.955018  6606 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:49.972401  6700 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:49.852489  6489 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:49.977496  6683 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:23:50.104581 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 197356us
W20260709 12:23:50.099400  6563 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.786s	user 0.012s	sys 0.001s
W20260709 12:23:50.102340 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 233198us
I20260709 12:23:50.191376 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:50.135011 31669 rpcz_store.cc:276] 0709 12:23:48.104816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.461004 (+356188us) service_pool.cc:224] Handling call
0709 12:23:48.461306 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:49.410030 (+948724us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.947043 (+537013us) spinlock_profiling.cc:243] Waited 948 ms on lock 0x7b54002535d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:49.947294 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2180582144}
I20260709 12:23:49.955538  6611 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.135150 31745 rpcz_store.cc:276] 0709 12:23:48.944330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.949570 (+  5240us) service_pool.cc:224] Handling call
0709 12:23:48.968437 (+ 18867us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:49.405226 (+436789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.967908 (+562682us) spinlock_profiling.cc:243] Waited 436 ms on lock 0x7b54000f2dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:49.968240 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1002288000}
I20260709 12:23:50.149379  6738 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:50.205056 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:50.128984 32344 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.535s	user 0.010s	sys 0.012s
I20260709 12:23:50.153355  6247 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:50.206199 32344 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.612s	user 0.010s	sys 0.012s
I20260709 12:23:49.943714  6587 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.209298 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:50.210060 32375 raft_consensus.cc:2376] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20. Re-sending same reply.
W20260709 12:23:50.137873 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 280360us
I20260709 12:23:50.146142 31675 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:50.102710  6736 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:23:50.150966 31670 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:49.971755  6616 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:50.151882 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 292085us
I20260709 12:23:50.128006  6740 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:50.177811  6606 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.141534  6455 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:23:50.178577 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:50.241410 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:50.193712  6683 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:50.242130 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63678.1us
I20260709 12:23:50.243045 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:50.243940 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:50.149389  6507 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:50.137909  6496 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.023s
I20260709 12:23:50.135749 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:50.153401  6588 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:50.150138  6747 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:23:50.151597  6331 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:50.256814 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:50.165311  6741 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.095s
I20260709 12:23:50.160894  6700 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:50.101130  6421 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:50.238993 32374 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:23:50.214694  6647 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:50.304955  6647 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:50.306169  6647 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.298365  6657 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:50.313180 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1132) took 1060 ms. Trace:
I20260709 12:23:50.313719 31673 rpcz_store.cc:276] 0709 12:23:49.252697 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.421129 (+168432us) service_pool.cc:224] Handling call
0709 12:23:49.446440 (+ 25311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:49.500937 (+ 54497us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:49.500972 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:49.501016 (+    44us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:49.501050 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:50.312878 (+811828us) spinlock_profiling.cc:243] Waited 54.2 ms on lock 0x7b54001824d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:50.312931 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:50.313103 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":124748544}
W20260709 12:23:50.316533  6739 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 12:23:50.319600  6618 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.321058  6683 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.320621  6455 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.322697  6683 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:50.323254 32369 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:50.323076  6644 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:50.323940  6644 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:50.325466  6744 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:23:50.325330  6644 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.258106 32370 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 21: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:50.216298 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:50.352070 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 135874us
I20260709 12:23:50.205804  6620 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.043s
I20260709 12:23:50.355353  6620 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:23:50.251847 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 116180us
I20260709 12:23:50.356469 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:50.357170  5844 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:50.357713  5844 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:50.357810 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:50.266814  6743 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:23:50.267726  6697 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:50.250450  6501 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:50.262233  6738 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.275102  6747 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.387230  6747 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.274957  6700 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.275740  6588 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.254055  6607 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:23:50.279390  6733 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.790s	user 0.007s	sys 0.008s
I20260709 12:23:50.251434  6680 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.281647  6741 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:50.254735  6507 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:50.283779  6661 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.551s	user 0.006s	sys 0.005s
I20260709 12:23:50.246104  6606 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.232079  6247 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.440527  6247 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:50.441279  6247 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:50.236567  6651 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.024s
W20260709 12:23:50.442143  6745 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.181s	user 0.002s	sys 0.000s
W20260709 12:23:50.304183  6497 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.567s	user 0.010s	sys 0.000s
I20260709 12:23:50.321506  6682 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:23:50.207577 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:50.327896  6725 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.610s	user 0.005s	sys 0.002s
I20260709 12:23:50.328639  6616 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:50.329512 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:50.501060 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.199039  6701 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 15.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.047s
I20260709 12:23:50.356238  6620 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:23:50.200040  6563 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.268221  6421 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.197512  6678 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.354413 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:50.361696 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:50.285219 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.370821 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:50.373665 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:50.463109  6497 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:50.373827  6659 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.033s	user 0.014s	sys 0.000s
I20260709 12:23:50.531263  6659 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.377724 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:50.358917  5844 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.391602 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1259) took 1090 ms. Trace:
I20260709 12:23:50.533887 31824 rpcz_store.cc:276] 0709 12:23:49.301026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.331761 (+ 30735us) service_pool.cc:224] Handling call
0709 12:23:49.332102 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:49.563890 (+231788us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:49.563936 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:49.563993 (+    57us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:49.564040 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:50.391334 (+827294us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b54002080d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:50.391392 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:50.391538 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":532515712}
I20260709 12:23:50.419612 32342 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:50.345508  6709 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:23:50.422364 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.425582 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:50.436517  6661 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:50.437420  6455 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:50.438140  6700 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:50.442960  5949 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:50.286972  6611 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:50.444314  6759 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:50.446539  6758 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:50.444353  6746 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.179s	user 0.000s	sys 0.002s
I20260709 12:23:50.598804 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:50.450734  6706 consensus_queue.cc:1175] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b95aeeaf4c51442b8092cff1de674f2b" responder_term: 18 status { last_received { term: 18 index: 12 } last_committed_idx: 11 last_received_current_leader { term: 18 index: 12 } }
W20260709 12:23:50.484131 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.561089  6247 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.430321  6644 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:50.424710  6657 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.427925  6501 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:50.613862 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3114) took 1307 ms. Trace:
I20260709 12:23:50.452313  6697 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:50.453461  6587 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:50.543884 31672 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:50.454227  6607 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:50.545284 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:50.620939 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.547231 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:50.534880 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 173260us
I20260709 12:23:50.554257 32374 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:50.544435 32369 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:50.384325  6736 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.432672  6741 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:50.558810 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:50.641297 31674 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
W20260709 12:23:50.559017 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 139497us
W20260709 12:23:50.504997  6660 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.700s	user 0.014s	sys 0.000s
I20260709 12:23:50.499616  6643 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.463850  6682 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:50.593369 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:50.561116  6640 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.848s	user 0.023s	sys 0.041s
I20260709 12:23:50.438514  6606 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:50.601701 31675 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: [17.10-17.10]
I20260709 12:23:50.234380 31568 catalog_manager.cc:5697] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 13 to 14, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 14 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:50.557638  6498 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:50.369580 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:23:50.721964 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 352472us
I20260709 12:23:50.724684  6754 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Stepping down as leader of term 15
I20260709 12:23:50.725198  6754 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 15 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:50.607575 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:50.607651 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 253340us
I20260709 12:23:50.726548 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:50.557744  6755 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:50.614310 31827 rpcz_store.cc:276] 0709 12:23:49.306721 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.353961 (+ 47240us) service_pool.cc:224] Handling call
0709 12:23:49.354393 (+   432us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:49.359697 (+  5304us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:49.359732 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:49.360341 (+   609us) log.cc:844] Serialized 52 byte log entry
0709 12:23:49.450700 (+ 90359us) raft_consensus.cc:1645] Marking committed up to 8
0709 12:23:49.450764 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:49.450857 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:50.278679 (+827822us) spinlock_profiling.cc:243] Waited 8.15 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:50.607151 (+328472us) raft_consensus.cc:1685] finished
0709 12:23:50.607173 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:50.613790 (+  6617us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":509,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"spinlock_wait_cycles":18754304,"thread_start_us":89466,"threads_started":1,"wal-append.queue_time_us":128108}
W20260709 12:23:50.727638  6757 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.162s	user 0.001s	sys 0.000s
I20260709 12:23:50.728315 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:50.686856  6588 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:50.729635 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:50.729817  6421 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:50.598124  5844 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:50.730500  6421 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:50.734122  6421 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.734227  6659 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:50.621886 31678 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
W20260709 12:23:50.715010  6602 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.776s	user 0.013s	sys 0.000s
I20260709 12:23:50.700176 31568 catalog_manager.cc:5697] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 13 to 15. New cstate: current_term: 15 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:23:50.735287  6602 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.600673  6743 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:50.643016 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:50.737946 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 95097.2us
I20260709 12:23:50.652287 31825 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: []
I20260709 12:23:50.669039  6489 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.655354 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:50.656380 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 220354us
W20260709 12:23:50.663632 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.666304 31673 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 21: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:50.742331 31819 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 26: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:50.596725  6610 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.056s
I20260709 12:23:50.740871  6588 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:50.669761 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:50.743098  6588 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:50.743507 31822 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:50.743702  6768 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.744272  6768 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.745154 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:50.745611 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:50.671793 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:50.614768  6247 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.746560 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:50.747443  6768 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:50.747749 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:50.748373  6247 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:50.673005 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:50.748749 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:50.749155 31824 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:50.749682 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:50.750273 31749 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:23:50.679394 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:50.598953  6759 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:50.661862  6640 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.949s	user 0.023s	sys 0.041s
W20260709 12:23:50.675573  6689 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.947s	user 0.014s	sys 0.000s
I20260709 12:23:50.752965  6759 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:50.647621  6587 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:50.752923  6689 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.665766  6747 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:50.753798  6587 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.605151  6758 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.680229  6497 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:50.690186  6678 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:50.690517  6750 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.663952  6643 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.747462 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:50.695597  6741 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.623198  6657 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.650789  6660 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.681164  6761 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:50.716549  6736 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:50.603896  6611 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:50.739517 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:50.653576 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:50.738654 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:50.730278 31827 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:50.740660 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:23:50.830689 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 90105.8us
I20260709 12:23:50.619277  6651 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:50.684576  6498 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.738582  6618 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:50.751976  6680 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:50.685561 31674 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:23:50.748389 31750 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:23:50.875039 31674 raft_consensus.cc:606] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Received request to transfer leadership to TS 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:50.875468 31674 raft_consensus.cc:614] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:50.875765 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:50.876493 31750 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:50.747459 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:50.762006 31745 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 18 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:23:50.763342 32374 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:50.765538 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:50.766659 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:50.767765 31818 raft_consensus.cc:2368] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18.
I20260709 12:23:50.877483 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:50.781523 31822 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:50.796628  6689 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:50.797474 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:50.797827  6758 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:50.728852  6754 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 15, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.798131 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:50.762161 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:50.890002 31745 pending_rounds.cc:85] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 10
I20260709 12:23:50.890277 31819 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:50.890378 31745 pending_rounds.cc:107] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 12:23:50.891099 31745 raft_consensus.cc:2889] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:50.734558  6421 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:50.798171 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:50.799983 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:50.892340 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 92449.1us
I20260709 12:23:50.843784  6678 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:50.844558 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:50.829200 31825 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:23:50.896490 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:50.897050 31825 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:50.751850 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:50.737569  6602 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:50.799118  6706 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.088s
W20260709 12:23:50.862049 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 123490us
I20260709 12:23:50.869813 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:50.871006 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:50.878971 31753 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
W20260709 12:23:50.887306  6671 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.738s	user 0.006s	sys 0.007s
I20260709 12:23:50.920508 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:50.920857 31751 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:50.921324 31753 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:50.882174 31821 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:50.882961 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:50.883733 32369 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:50.798903  6661 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:50.884546 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:50.828079  6761 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.843454  6741 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:50.829941  6092 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:23:50.797825  6657 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:50.849432  6759 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:50.893607 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:50.964078 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 70546us
I20260709 12:23:50.964807 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3115) took 1586 ms. Trace:
I20260709 12:23:50.965221 31818 rpcz_store.cc:276] 0709 12:23:49.378807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.384892 (+  6085us) service_pool.cc:224] Handling call
0709 12:23:50.964587 (+1579695us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54000fa854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:50.964740 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3121926912}
W20260709 12:23:50.808369 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 46125.2us
I20260709 12:23:50.894490 32370 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:50.895148 31749 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:23:50.899765 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55302.9us
I20260709 12:23:50.900506 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:50.799387  6750 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:50.901132 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.968170 31746 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:50.902808  6489 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:50.903071  6762 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.195s	user 0.001s	sys 0.000s
I20260709 12:23:50.906724  6659 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:50.860857  6736 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:23:50.743534  6588 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:50.898358 31820 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:23:50.911350 31825 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:50.913486 31824 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:23:50.705628  6501 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 18.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.709755  6607 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.878950  6563 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:50.884784  6729 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:50.884661  6507 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:50.922585 31822 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:51.009406  6497 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:50.930768  6660 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:51.047153 31824 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:23:51.011981  6734 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:23:50.714902  6682 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.940424  6643 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:50.941766 31826 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:50.942652 31819 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:23:50.943553 31821 pending_rounds.cc:85] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 11
I20260709 12:23:50.849661  6747 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.049s
I20260709 12:23:50.945315 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:51.092006 31827 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:50.945995  6661 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:50.948218  6498 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:50.951946 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:50.955096  6587 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:50.959264  6763 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.271s	user 0.001s	sys 0.000s
I20260709 12:23:50.850564  6611 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:50.843705  6738 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.085s
W20260709 12:23:50.961651 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 90734.2us
W20260709 12:23:50.899279  6331 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.623s	user 0.011s	sys 0.000s
W20260709 12:23:50.967382 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 66969.2us
I20260709 12:23:51.123382 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.124209 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:50.902242  6766 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.134s	user 0.000s	sys 0.000s
W20260709 12:23:51.019999 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53125us
I20260709 12:23:51.132886 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.134155 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.024718 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3291) took 1177 ms. Trace:
I20260709 12:23:51.134778 31672 rpcz_store.cc:276] 0709 12:23:49.847664 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.902013 (+ 54349us) service_pool.cc:224] Handling call
0709 12:23:51.024429 (+1122416us) spinlock_profiling.cc:243] Waited 433 ms on lock 0x7b540017cfd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:51.024634 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":995470208}
I20260709 12:23:51.136107 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:51.136850 31672 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 19: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:51.137655 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.027004 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:51.029873 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:51.038106 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:51.139345 31820 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:51.039863 31749 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:23:51.013609  6092 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:23:51.144220 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 114427us
I20260709 12:23:50.924600 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:50.923194  6689 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:51.050081 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.051291 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:50.914525  6678 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.054749 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.163522 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.165369 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:51.012950  6755 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:51.021240  6659 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.070457 31821 pending_rounds.cc:107] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 20.12
I20260709 12:23:51.071910 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:51.092736 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.169169 31821 raft_consensus.cc:2889] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:51.094525 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:51.169749 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 97980.9us
I20260709 12:23:51.094743 31822 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:51.068894  6758 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:23:51.040410  6759 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:51.099634 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.100656 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.174017 31677 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:51.103977 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:51.174942 31826 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:23:51.046792  6729 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.057971  6734 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.129014 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:51.173414 31675 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:51.185456 31819 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:51.071141  6497 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:51.189153  6657 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:51.189447  6682 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:51.189925  6657 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.189713  6497 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.143543  6498 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.047s
I20260709 12:23:51.191783  6498 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:51.192178  6498 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:51.192574  6657 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:51.192960  6498 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.185281 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:51.197738 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:51.189261 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:51.091408  6602 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.122519 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:51.131374 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.199183 31826 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:51.138796 32374 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:51.149410 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:51.160310 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.093508 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.201411 31678 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:51.169487 31673 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:51.171957 31674 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
W20260709 12:23:51.172593 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 252909us
I20260709 12:23:51.204367 31820 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:51.205118 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:51.205804 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 83371.3us
W20260709 12:23:51.206063 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 74768.6us
I20260709 12:23:51.206847 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:51.142539  6660 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:23:51.215842 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.152376  6643 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:51.218753 31670 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:23:51.218868 31746 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 17: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:51.144878  6092 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:51.172284  6671 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:51.230481 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.167979  6743 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:51.168429  6706 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.173300  6489 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:51.239315 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.223341 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:51.222847 31826 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 12. (term mismatch)
I20260709 12:23:51.299397 31826 pending_rounds.cc:85] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 11
I20260709 12:23:51.301215 31826 pending_rounds.cc:107] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:23:51.302210 31826 raft_consensus.cc:2889] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:51.224659 32369 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:51.228493 32374 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:23:51.178053  6659 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:51.184496  6735 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:51.185843  6729 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:51.197619  6734 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:51.237062 31674 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
I20260709 12:23:51.223359  6434 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:51.221961  6643 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.251161 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:51.252322  6706 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.323385 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:51.325347  6706 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:51.238907  6489 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.258983  6633 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:51.326076  6489 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:51.327620  6434 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:51.328291  6434 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:51.329443  6434 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.268715  6588 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:51.272780  6700 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.682s	user 0.010s	sys 0.000s
I20260709 12:23:51.229369  6092 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:51.331076  6421 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.334044  6643 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:51.275314  6651 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:23:51.279414  6697 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.632s	user 0.009s	sys 0.001s
I20260709 12:23:51.279955  6759 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.281576  6750 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:51.210656  6737 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:51.284571  6678 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:51.286369  6761 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:51.209725  6656 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
I20260709 12:23:51.183534 31827 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:23:51.217065  6496 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:23:51.218376  6660 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:51.344732  6683 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.021s	user 0.015s	sys 0.000s
I20260709 12:23:51.318729 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.432183 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:51.432796 31674 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:51.194568  6498 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:23:51.319468 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.234820  6671 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.322408 31746 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:51.185058  6755 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.268461  6611 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:51.337486  6610 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:51.320631 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.211560  6602 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.294837  6758 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.220901  6736 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.101s
W20260709 12:23:51.315336 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 108336us
W20260709 12:23:51.467942 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 147432us
I20260709 12:23:51.431125  6660 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:23:51.425559 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 216955us
W20260709 12:23:51.473378 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:51.426209 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 219036us
I20260709 12:23:51.318989  6743 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:51.443864 31747 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 20 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 10.
I20260709 12:23:51.365701  6697 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:51.468117 31826 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 12:23:51.424489 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.383566  6496 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.394407 31677 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:51.452622  6752 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:51.481680 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.492936 32369 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:23:51.493670 32374 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:51.487504 31718 leader_election.cc:400] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18.
I20260709 12:23:51.447047  6671 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:51.494639 31718 leader_election.cc:403] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:51.495872 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:51.497948 32369 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:23:51.487782 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.513561 31823 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:51.517616 31674 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:23:51.500139 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.518924 32373 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:51.524623 31746 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:23:51.526463 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:51.527634 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.530292 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.535187 31677 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:51.526484 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.539722 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.477013  6656 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:23:51.542208 32370 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:51.545444 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.477533  6706 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:51.546202 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58880us
I20260709 12:23:51.511014  6737 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.494005  6750 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:51.492671  6752 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:51.573680 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.471870  6758 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.475576  6497 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:51.482108  6496 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:23:51.555446 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 357802us
I20260709 12:23:51.571918 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.554885 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.572821 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:51.526876  6644 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.472263  6610 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:51.588562 32369 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 19: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:51.588840  6644 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.515144  6735 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:51.491729  6247 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:51.575914 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.574458 31746 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
W20260709 12:23:51.580415 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.534341  6661 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:51.594894 32370 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:51.545715  6683 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:51.591367 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.591807 32374 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:51.543987  6501 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:51.544322  6697 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:51.601743 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:51.593312 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:51.602380 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:51.575006  6700 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:51.606696 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.604763 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:51.608319  6421 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:51.608626  6588 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:51.575284  6737 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.592566  6497 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.610915  6737 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:51.611565  6497 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:51.612248  6774 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.141s	user 0.000s	sys 0.000s
I20260709 12:23:51.614027 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.608325  6092 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.617120 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.617971  6092 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:51.619122  6697 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:51.619405  6755 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:51.620664  6218 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.620199  6697 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.621204  6218 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.623413  6218 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:51.623934  6218 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.624423  6218 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:51.624907 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:51.625829 32373 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:51.553433  6706 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:51.629210  6706 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:51.602167  6501 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.630273  6501 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:51.611604  6737 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:23:51.589329  6610 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 19.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.599985  6743 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:51.633502  6680 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.753s	user 0.008s	sys 0.003s
I20260709 12:23:51.634424 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:51.635236 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.596544  6647 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 18
I20260709 12:23:51.634776 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.637004 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:51.637553 32371 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:51.576788  6750 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.635270 32378 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 19: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:51.640933  6750 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:51.643185  6790 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:23:51.643422 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.616111 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.644496 31753 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:51.646019 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.606782 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:51.646663 31751 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:23:51.649194  5949 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:51.586679  6780 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:23:51.649686  6661 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:51.603503  6656 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:51.652107  6771 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.331s	user 0.001s	sys 0.000s
I20260709 12:23:51.652676 31672 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:23:51.653360 31672 raft_consensus.cc:606] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:51.648571 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.658891 31672 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:23:51.659722 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.656324  6643 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.663626 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:51.664479 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:51.665825 31753 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:23:51.661983 32371 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:51.668462 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:51.669636 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 64956us
I20260709 12:23:51.672781 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:51.667630  6460 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:23:51.671540  6421 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:51.662374  6607 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:51.665077  6604 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.010s
I20260709 12:23:51.675726  6783 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.672946 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.649657  5949 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:51.690311 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.676162  6092 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:51.624441  6758 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:51.689687 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:51.703709 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:51.657775  6736 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:51.714546 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.703490  6460 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:51.751487  6460 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 15.10, Last appended by leader: 10, Current term: 16, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.706175  6750 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:51.733640 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3245) took 1063 ms. Trace:
I20260709 12:23:51.735145 31744 raft_consensus.cc:2410] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 23 because replica has last-logged OpId of term: 22 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:23:51.727093  6780 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:51.736737 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:51.678108 31792 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:51.653559  6656 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 14.8, Last appended by leader: 8, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.744552 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.744951 31748 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:51.746390 31753 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:23:51.748081 32370 raft_consensus.cc:2368] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26.
I20260709 12:23:51.682698  6734 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.667889  6729 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.744419  6736 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.705541  5949 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:23:51.755816 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 65621.3us
I20260709 12:23:51.755774 31823 rpcz_store.cc:276] 0709 12:23:50.670498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.670656 (+   158us) service_pool.cc:224] Handling call
0709 12:23:50.670951 (+   295us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:51.514035 (+843084us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.733264 (+219229us) spinlock_profiling.cc:243] Waited 842 ms on lock 0x7b54001969d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:51.733543 (+   279us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1937387264}
W20260709 12:23:51.759781 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 81732.6us
I20260709 12:23:51.745664  6498 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:51.657601  6700 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.665915  6706 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:51.763828 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.747686  6607 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:51.677963  6680 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.767427 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:51.602509  6735 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.602206  6247 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.758277  6780 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:51.769739 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.775914 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.781442 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:51.757390  6683 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:51.760643  6331 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:51.758051  6750 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:51.771719 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:51.783056  6780 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:51.792697  6737 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.004s
W20260709 12:23:51.797616  6775 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.159s	user 0.001s	sys 0.000s
I20260709 12:23:51.765528  6758 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:51.787683 31753 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:51.799037  6758 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:51.800099  6758 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 16.13, Last appended by leader: 13, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.780872 32372 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:23:51.761138  6659 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.802482 31673 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:23:51.803122 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.803567  6659 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:51.804450 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:51.772123  6736 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:51.805586 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.806176 31748 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:51.807240  6489 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.807987  6640 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:23:51.808403  6640 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:51.808683  6682 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.619s	user 0.011s	sys 0.000s
I20260709 12:23:51.809257 31750 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:51.809350  6640 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.794157  6734 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:51.813205  6218 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:51.813478 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:51.814143 31671 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 24
I20260709 12:23:51.794004  6729 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:51.816792  6736 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.819396 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:51.819945  6736 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:51.820078 31744 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:51.823729 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.824074  6700 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:51.824129  6618 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:51.824967 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:51.825091  6618 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:51.825567  6750 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.676272  6783 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.826069  6618 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.827200  6618 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:51.786720  6724 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:51.828670  6783 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:51.829272  6724 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:51.800606 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3184) took 1057 ms. Trace:
I20260709 12:23:51.830343 31747 rpcz_store.cc:276] 0709 12:23:50.743242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.745379 (+  2137us) service_pool.cc:224] Handling call
0709 12:23:51.800344 (+1054965us) spinlock_profiling.cc:243] Waited 697 ms on lock 0x7b54002959d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:51.800537 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1603249408}
I20260709 12:23:51.830884 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.831604 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:51.831801 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44300.5us
W20260709 12:23:51.832574  6796 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:23:51.832854 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.834033 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.834700 31675 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:23:51.835961 31750 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:51.836817 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:51.837471 31820 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:23:51.837348 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:51.837424 31822 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:23:51.830188  6724 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.845394 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:51.838814 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:51.774823  6706 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.844563 31672 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:23:51.848306 31822 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [19.12-21.13]
W20260709 12:23:51.857700 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.848965 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:51.849112 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:51.852954 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:51.796262  6786 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260709 12:23:51.854068 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:51.861907 31826 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:23:51.854278  6507 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:51.866756 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.867252 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3258) took 1101 ms. Trace:
I20260709 12:23:51.867189  6507 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.867727 31821 rpcz_store.cc:276] 0709 12:23:50.765899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.943046 (+177147us) service_pool.cc:224] Handling call
0709 12:23:50.943364 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:51.186113 (+242749us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:51.186148 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:51.186676 (+   528us) log.cc:844] Serialized 26 byte log entry
0709 12:23:51.487965 (+301289us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:51.488004 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:51.488077 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:51.795575 (+307498us) spinlock_profiling.cc:243] Waited 42.2 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:51.866939 (+ 71364us) raft_consensus.cc:1685] finished
0709 12:23:51.866963 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:51.867186 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":96966528,"thread_start_us":240980,"threads_started":1,"wal-append.queue_time_us":357112}
I20260709 12:23:51.854909 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:51.859457 31823 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:51.859958 31675 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:51.865792  6737 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:51.866333  6604 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:23:51.867591  6783 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:51.830888  6644 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:51.869630 31818 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:51.845526  6640 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:51.883667 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.912318 31678 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:51.891623 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:51.889791 31818 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:23:51.871546  6783 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:51.894531  6760 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.236s	user 0.000s	sys 0.018s
I20260709 12:23:51.895625  6737 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:51.876394  6661 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 12:23:51.843070  6331 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:51.896749  6507 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:51.902797  6754 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:23:51.904356 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.843256  6700 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.843554  6659 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.905534 31671 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:23:51.905886  6247 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:51.908280  6787 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.290s	user 0.004s	sys 0.004s
I20260709 12:23:51.909343 31677 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:23:51.950237 31677 raft_consensus.cc:606] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:51.841420  6489 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:51.889539 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.913153 31753 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:51.882819  6779 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.310s	user 0.000s	sys 0.002s
I20260709 12:23:51.862030  6706 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:51.881568 31675 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:23:51.929911 31823 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:23:51.779963  6671 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.782135  6680 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.933653 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.941785 31744 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:23:51.928393  6783 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:51.982607 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.800091  6607 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.950927 31671 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:23:51.951977 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:51.954337 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.932243  6331 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:51.932717  6660 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:51.981671 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 92229.5us
W20260709 12:23:51.982088 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 48564.7us
I20260709 12:23:51.953079  6604 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:51.983785 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.978351  6768 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:51.950603 31677 raft_consensus.cc:614] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:51.986526 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.975688 31718 leader_election.cc:400] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26.
W20260709 12:23:51.987607 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:51.988332  5949 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:51.971336  6247 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:52.042841  6247 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.045207  6247 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:52.045399  6498 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:52.056010  6680 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:52.056866  6647 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 17. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18.
I20260709 12:23:51.991027 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:51.991693 31753 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:52.102067 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:51.984159  6643 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:51.986562 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:52.105406 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 118902us
I20260709 12:23:51.985716  6489 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:52.106511 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:52.039479  6768 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.040629 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:52.041226  6737 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:52.042227 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3292) took 2194 ms. Trace:
I20260709 12:23:52.057228  6671 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:51.950157  6659 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.140244  6783 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:52.141395  6659 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:52.061023 31748 raft_consensus.cc:2410] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 22 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:23:52.061784  6735 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:52.062786  6660 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:52.064081 31751 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:23:52.065387  6651 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.143131  6651 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.145870  6651 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:52.066380  6602 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:52.068204 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.149181 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:52.151371  6602 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.152046  6602 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:51.949980  6700 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:52.154166  6602 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:51.989974 31746 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:52.156301 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3197) took 1386 ms. Trace:
I20260709 12:23:52.050045  6501 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:52.156932 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:52.157692 31746 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:52.101017  6218 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:52.159101  6501 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:51.995304 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:52.083714 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:52.090816 31718 leader_election.cc:403] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
W20260709 12:23:52.162046 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 186495us
I20260709 12:23:52.103420 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:52.162860  6647 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26.
I20260709 12:23:52.156867 31745 rpcz_store.cc:276] 0709 12:23:50.769522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.882376 (+112854us) service_pool.cc:224] Handling call
0709 12:23:50.889757 (+  7381us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:50.891990 (+  2233us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:50.892025 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:50.892534 (+   509us) log.cc:844] Serialized 52 byte log entry
0709 12:23:51.350405 (+457871us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:51.372054 (+ 21649us) log.cc:844] Serialized 14 byte log entry
0709 12:23:51.796982 (+424928us) spinlock_profiling.cc:243] Waited 23.6 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:52.155548 (+358566us) spinlock_profiling.cc:243] Waited 4.51 ms on lock 0x7b18002b8100. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:23:52.155742 (+   194us) log.cc:844] Serialized 14 byte log entry
0709 12:23:52.155968 (+   226us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.156043 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:52.156087 (+    44us) raft_consensus.cc:1685] finished
0709 12:23:52.156099 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.156235 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14452,"prepare.queue_time_us":28006,"raft.queue_time_us":10856,"raft.run_cpu_time_us":454,"raft.run_wall_time_us":68424,"spinlock_wait_cycles":75376000,"thread_start_us":333050,"threads_started":1,"wal-append.queue_time_us":627490}
I20260709 12:23:52.164296 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:52.164628 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:52.165382 31745 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:51.991961 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:52.050347  6496 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.167032 31673 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:52.109263 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:52.113947 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:52.116128 31676 rpcz_store.cc:276] 0709 12:23:49.848005 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:49.975801 (+127796us) service_pool.cc:224] Handling call
0709 12:23:49.976251 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:50.010255 (+ 34004us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:50.010294 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:50.011001 (+   707us) log.cc:844] Serialized 26 byte log entry
0709 12:23:50.391357 (+380356us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:50.391421 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:50.391527 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:51.893867 (+1502340us) spinlock_profiling.cc:243] Waited 13.9 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:51.990314 (+ 96447us) raft_consensus.cc:1685] finished
0709 12:23:51.990335 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.042135 (+ 51800us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33485,"raft.queue_time_us":34163,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":74162,"spinlock_wait_cycles":31865088,"thread_start_us":249999,"threads_started":1,"wal-append.queue_time_us":486084}
I20260709 12:23:52.121719  5949 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:52.125612 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:51.995929  6604 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:52.138345 31749 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:52.177467 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:52.178336 32374 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:52.185724 31676 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:52.104820  6661 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:52.156201  6700 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:52.103140 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 112179us
I20260709 12:23:52.110117  6489 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.191533 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:52.120985  6247 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:51.986584  6497 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:52.207309 31746 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:23:52.124294  6759 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:23:52.171393 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 62274.6us
I20260709 12:23:52.213866 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:52.154917 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:52.113140  6507 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:52.165805 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:52.217783 31746 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 13. (term mismatch)
I20260709 12:23:52.218202 31746 pending_rounds.cc:85] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 12
I20260709 12:23:52.178835  6602 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:52.194589 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:52.222762 32371 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:52.195389 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:52.199952 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:52.201069 31823 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 20 index: 11. (index mismatch)
I20260709 12:23:52.225649 31823 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: []
W20260709 12:23:52.226117 31669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2376) took 1173 ms (client timeout 1559 ms). Trace:
I20260709 12:23:52.221596 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3198) took 1383 ms. Trace:
I20260709 12:23:52.227192 31750 rpcz_store.cc:276] 0709 12:23:50.838716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.877164 (+ 38448us) service_pool.cc:224] Handling call
0709 12:23:52.221418 (+1344254us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b54002f67d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:52.221530 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2135950720}
W20260709 12:23:52.226569 31669 rpcz_store.cc:269] 0709 12:23:51.052877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.053484 (+   607us) service_pool.cc:224] Handling call
0709 12:23:52.225918 (+1172434us) spinlock_profiling.cc:243] Waited 280 ms on lock 0x7b5400046ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:52.226062 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":642851584}
I20260709 12:23:52.206485 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.091588  6697 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:52.210206 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:52.243530 31824 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:52.218485 31746 pending_rounds.cc:107] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 12:23:52.244138 31746 raft_consensus.cc:2889] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:52.215124 31751 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:23:52.245643 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.221624 31753 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:52.202713 31676 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:52.246956 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1142) took 1013 ms. Trace:
I20260709 12:23:52.247385 31676 rpcz_store.cc:276] 0709 12:23:51.233122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.202126 (+969004us) service_pool.cc:224] Handling call
0709 12:23:52.202473 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.246569 (+ 44096us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.246884 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:52.249070 31676 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:23:52.251508 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1140) took 1027 ms. Trace:
I20260709 12:23:52.251914 31677 rpcz_store.cc:276] 0709 12:23:51.223755 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.198664 (+974909us) service_pool.cc:224] Handling call
0709 12:23:52.251038 (+ 52374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.251196 (+   158us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:52.251219 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:52.251260 (+    41us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:52.251288 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.251338 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.251458 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:52.229482 31822 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:23:52.235460 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.227535 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:52.243353 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:51.989107  6683 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.130494  6706 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:52.196137  6735 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:23:52.159121  6218 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:52.261821 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.259893 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3261) took 1363 ms. Trace:
I20260709 12:23:52.202988  6489 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:52.269315 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.260113 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:52.292572 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.293727 31823 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:52.207935  6247 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.271919 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3204) took 1372 ms. Trace:
I20260709 12:23:52.212455  6759 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:52.207470  6497 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.145716  6752 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:52.279733 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 52917.7us
I20260709 12:23:52.286747 31824 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:23:52.195889  6501 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.083s
I20260709 12:23:52.314419 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.314975 31747 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:52.315140 31824 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
W20260709 12:23:52.298141  6434 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.635s	user 0.007s	sys 0.019s
I20260709 12:23:52.296867 31751 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:52.319813 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:52.320047 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.298353 31746 rpcz_store.cc:276] 0709 12:23:50.899674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.208178 (+1308504us) service_pool.cc:224] Handling call
0709 12:23:52.210478 (+  2300us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.271572 (+ 61094us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.271846 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:52.273336 31675 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:23:52.208367  6661 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:52.289819 31825 rpcz_store.cc:276] 0709 12:23:50.896725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.027832 (+131107us) service_pool.cc:224] Handling call
0709 12:23:51.028147 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:51.028357 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:23:51.320206 (+291849us) log.cc:844] Serialized 14 byte log entry
0709 12:23:51.612772 (+292566us) spinlock_profiling.cc:243] Waited 4.9 ms on lock 0x7b18004dfce0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:23:52.227979 (+615207us) spinlock_profiling.cc:243] Waited 4.88 ms on lock 0x7b18004dfce0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:23:52.259342 (+ 31363us) log.cc:844] Serialized 14 byte log entry
0709 12:23:52.259551 (+   209us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:52.259598 (+    47us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:52.259642 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.259706 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.259830 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28133,"prepare.queue_time_us":425990,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"raft.queue_time_us":13842,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":8849,"spinlock_wait_cycles":22543360,"thread_start_us":361618,"threads_started":2,"wal-append.queue_time_us":153559}
W20260709 12:23:52.326933 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.327677 31748 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: [16.9-16.9]
I20260709 12:23:52.328058 31825 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:23:52.328409 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2540) took 1206 ms. Trace:
I20260709 12:23:52.328856 31748 rpcz_store.cc:276] 0709 12:23:51.121630 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.298208 (+1176578us) service_pool.cc:224] Handling call
0709 12:23:52.320611 (+ 22403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:52.328115 (+  7504us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.328350 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:52.329713 31825 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
W20260709 12:23:52.307976  6780 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.511s	user 0.005s	sys 0.005s
I20260709 12:23:52.332538 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3303) took 1173 ms. Trace:
I20260709 12:23:52.333122 31674 rpcz_store.cc:276] 0709 12:23:51.159384 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.823472 (+664088us) service_pool.cc:224] Handling call
0709 12:23:52.229364 (+405892us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.229544 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:52.229570 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:52.229616 (+    46us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:23:52.229653 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.229702 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.332303 (+102601us) spinlock_profiling.cc:243] Waited 406 ms on lock 0x7b54002544d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:52.332457 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":932707712}
I20260709 12:23:52.303359  6247 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:52.342890 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1056) took 1126 ms. Trace:
I20260709 12:23:52.343182 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:52.343422 31753 rpcz_store.cc:276] 0709 12:23:51.216412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.341940 (+1125528us) service_pool.cc:224] Handling call
0709 12:23:52.342310 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.342499 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:52.342529 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:52.342576 (+    47us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:52.342613 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.342702 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.342822 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:52.304421 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.345445 31753 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [19.12-21.13]
I20260709 12:23:52.242920  6697 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:52.312405 31642 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.901s (SENT) [suppressed 9 similar messages]
I20260709 12:23:52.294904 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.267962  6616 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:52.319047 31824 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:52.218003  6507 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.326577 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:52.244665  6689 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 14.8, Next index: 9, Last known committed idx: 7, Time since last communication: 0.089s
W20260709 12:23:52.303995 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 86788.5us
I20260709 12:23:52.311926  6759 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:52.355424  6805 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.206s	user 0.001s	sys 0.001s
I20260709 12:23:52.350099 31749 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 21 index: 17. (index mismatch)
I20260709 12:23:52.407315 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:52.244550  6421 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:23:52.313203  6497 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:52.267586  6736 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.055s
I20260709 12:23:52.310704  6783 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:52.356642 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:52.377209  6507 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:52.359966 31823 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: [22.17-22.17]
I20260709 12:23:52.277879  6604 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.380892 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.347635 31753 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 12:23:52.449752 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.450165 31753 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:23:52.451639 31753 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 11. (term mismatch)
I20260709 12:23:52.376538 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.452518 31825 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:52.400750 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:52.403509 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1054) took 1249 ms. Trace:
I20260709 12:23:52.453126 31677 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
W20260709 12:23:52.435179  6806 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.257s	user 0.001s	sys 0.000s
I20260709 12:23:52.425995  6421 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:52.411993 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:52.464003 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:52.464823 32373 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:52.410444 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:52.465404 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.465641 31750 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:52.465952 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:52.466118 31825 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:52.427024 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:52.372723  6697 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:52.431046 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 118690us
I20260709 12:23:52.435570 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:52.435254 32376 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 12:23:52.319850  6434 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.007s	sys 0.020s
I20260709 12:23:52.442404 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2542) took 1320 ms. Trace:
I20260709 12:23:52.443022 31678 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:23:52.313959  6735 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:52.446558 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:52.480962 31823 raft_consensus.cc:2368] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26.
I20260709 12:23:52.482127 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:52.483007 31823 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:52.447892 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:52.435447 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:52.453079 31748 rpcz_store.cc:276] 0709 12:23:51.154123 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.329664 (+1175541us) service_pool.cc:224] Handling call
0709 12:23:52.402817 (+ 73153us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.403155 (+   338us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:52.403185 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:52.403232 (+    47us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:52.403266 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.403325 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.403442 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:52.337440  6602 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:52.452003 31753 pending_rounds.cc:85] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 10
I20260709 12:23:52.486922 31753 pending_rounds.cc:107] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:23:52.487473 31753 raft_consensus.cc:2889] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:52.425436  6815 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.493409  6815 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:52.494144  6788 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.870s	user 0.000s	sys 0.005s
W20260709 12:23:52.495817  6807 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:23:52.496503  6735 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.346230  6780 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.498425  6218 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:52.500954  6780 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:52.501992 31824 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:52.408392 31675 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:23:52.503907  6808 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:23:52.461687 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:52.414554  6616 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:52.506040  6616 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.11, Last appended by leader: 11, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:52.471029 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.477720 31746 rpcz_store.cc:276] 0709 12:23:51.122066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.322058 (+1199992us) service_pool.cc:224] Handling call
0709 12:23:52.356470 (+ 34412us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.356730 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:23:52.357019 (+   289us) log.cc:844] Serialized 14 byte log entry
0709 12:23:52.441801 (+ 84782us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:52.441904 (+   103us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:52.441972 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.442089 (+   117us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.442295 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33570,"prepare.queue_time_us":514,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":4437,"raft.queue_time_us":705,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":6919,"spinlock_wait_cycles":9312512,"thread_start_us":84309,"threads_started":1}
I20260709 12:23:52.492388 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:52.512497 31676 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:52.503965 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.514957  6815 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:52.516332  6331 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.539315 31673 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:52.454849  6496 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:52.527694 31676 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
I20260709 12:23:52.530303 32370 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:23:52.541325 31672 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:23:52.473047  6421 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:52.478767  6758 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:52.541951 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:52.542907 31676 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:52.543910 31676 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
W20260709 12:23:52.544239 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 217719us
I20260709 12:23:52.545200  5949 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:52.545732  5949 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.546082 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.546751 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:52.547437 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:52.548770 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:52.483836  6604 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:52.566499 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:52.512610  6640 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.583s	user 0.009s	sys 0.002s
I20260709 12:23:52.580713 31753 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:23:52.560563 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:52.512631  6754 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.568s	user 0.009s	sys 0.000s
I20260709 12:23:52.561889 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:52.562567 31669 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
W20260709 12:23:52.563823 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53360.2us
W20260709 12:23:52.583745 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 52721.5us
I20260709 12:23:52.564555 31747 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:23:52.565430 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:52.560359 32372 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:23:52.581492 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:52.583946 32377 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:23:52.584633 31746 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:52.587720 31670 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:52.592224 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:52.597220 31818 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:23:52.593986 31748 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:23:52.597448  5949 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:52.586192 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:52.598471 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:52.598809 31823 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
W20260709 12:23:52.583297  6811 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
W20260709 12:23:52.600615 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.560158  6647 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:52.586537 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:52.595827 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:52.608918 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:52.610829  5949 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:52.609668 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:52.609721 31825 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:52.611575 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:52.612897 31818 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:23:52.611549  5949 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:52.616803 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:52.617265 31823 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:23:52.620942 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:52.622534  6823 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:52.622440  6651 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:52.623107  6736 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:52.603392  6814 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:52.613574  6828 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:52.630429 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:52.628742  5949 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:52.603600  6706 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:52.682737 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 52392.5us
I20260709 12:23:52.684299 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:52.636480 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:52.615109  6218 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:52.583354  6813 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 12:23:52.615767  6758 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Leader pre-election won for term 15
I20260709 12:23:52.655514  6783 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:52.661353 31818 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:23:52.639390  6823 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:52.623171  6700 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.062s
I20260709 12:23:52.662715  6825 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:23:52.713361  6825 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:52.714490  6825 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:52.663386  6501 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:52.660570 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:52.603798  6780 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:52.685812 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:52.687460 31752 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
W20260709 12:23:52.677973  6786 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
W20260709 12:23:52.692008 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 55594.3us
I20260709 12:23:52.614092  6647 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.652477 31825 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
W20260709 12:23:52.699822 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.681190  6810 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.091s
I20260709 12:23:52.675166  6814 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:52.696889  6758 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:52.702911  6823 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:52.703678  6819 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.117s	user 0.000s	sys 0.000s
W20260709 12:23:52.704430  6812 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:23:52.743219 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:52.744593 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:52.658057  6736 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:52.725214  6218 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.137s
I20260709 12:23:52.686668  6816 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:52.781471 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:52.880848 31749 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:52.695003  6607 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
W20260709 12:23:52.823455 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 137746us
I20260709 12:23:52.842801 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:52.653959  6651 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.706264  6759 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:52.719905  6661 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:52.849928  6497 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.719470  6783 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:52.690315  6724 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.081s
I20260709 12:23:52.856737  6758 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:52.864283 31746 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
W20260709 12:23:52.756666 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:52.685386  6828 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.842067  6496 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:23:52.857618  6814 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:52.859001  6780 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:52.869170  6736 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:23:52.919414 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:52.871791  6660 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.015s	sys 0.018s
I20260709 12:23:52.956584 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.877146  6822 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:52.937364 31746 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:23:52.957405 31827 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:52.948585 31749 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:52.869545  6697 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:52.953455 31671 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
W20260709 12:23:52.958750  6660 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.657s	user 0.015s	sys 0.019s
I20260709 12:23:52.972033 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:52.972908 31827 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:52.973158  6651 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:52.973786 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.975029 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:52.975692 31827 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:52.976548 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:52.978039 31827 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:23:52.979096 31827 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
I20260709 12:23:52.979249  6331 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:52.980235 31823 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:23:52.980736 31823 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:52.983134  6822 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:52.987716  6759 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:52.990480 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3201) took 2114 ms. Trace:
I20260709 12:23:52.990981 31745 rpcz_store.cc:276] 0709 12:23:50.876409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.166017 (+1289608us) service_pool.cc:224] Handling call
0709 12:23:52.166351 (+   334us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:52.166745 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:52.166777 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:52.167388 (+   611us) log.cc:844] Serialized 52 byte log entry
0709 12:23:52.592829 (+425441us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:52.989527 (+396698us) log.cc:844] Serialized 14 byte log entry
0709 12:23:52.989923 (+   396us) log.cc:844] Serialized 14 byte log entry
0709 12:23:52.990123 (+   200us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.990206 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:52.990254 (+    48us) raft_consensus.cc:1685] finished
0709 12:23:52.990269 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:52.990414 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":396341,"raft.queue_time_us":3369,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":247,"thread_start_us":425062,"threads_started":1,"wal-append.queue_time_us":452171}
I20260709 12:23:52.995765  6828 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:52.996809  6818 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.315s	user 0.000s	sys 0.001s
I20260709 12:23:53.002487 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2538) took 1881 ms. Trace:
I20260709 12:23:53.002916  6783 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.003206 31750 rpcz_store.cc:276] 0709 12:23:51.121127 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.259478 (+1138351us) service_pool.cc:224] Handling call
0709 12:23:52.465461 (+205983us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.466126 (+   665us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.002116 (+535990us) spinlock_profiling.cc:243] Waited 206 ms on lock 0x7b54001ee1d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:53.002383 (+   267us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":472997248}
W20260709 12:23:53.008422 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 122320us
I20260709 12:23:53.016093 31822 raft_consensus.cc:2368] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20.
I20260709 12:23:53.018371 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:53.019472 31827 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:53.019986 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2541) took 1898 ms. Trace:
I20260709 12:23:53.020797 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:53.022995  6507 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.567s	user 0.004s	sys 0.008s
I20260709 12:23:53.023589  6507 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:53.036500  6697 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:53.037846  6780 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.043939  6647 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:53.020596 31751 rpcz_store.cc:276] 0709 12:23:51.121851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.320510 (+1198659us) service_pool.cc:224] Handling call
0709 12:23:52.320892 (+   382us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:52.321274 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:52.321305 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:52.323731 (+  2426us) log.cc:844] Serialized 52 byte log entry
0709 12:23:52.884067 (+560336us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:52.884130 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:52.884236 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:53.019652 (+135416us) raft_consensus.cc:1685] finished
0709 12:23:53.019686 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.019898 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1633,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":18026,"thread_start_us":450340,"threads_started":1,"wal-append.queue_time_us":621376}
I20260709 12:23:52.981690 31823 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 16.9, Last appended by leader: 9, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.048663 31823 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:23:52.956992  6421 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:53.063578 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 220844us
W20260709 12:23:53.045655  6829 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.091s	user 0.000s	sys 0.001s
I20260709 12:23:53.067735 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:53.067970 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:53.063544  6247 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.662s	user 0.010s	sys 0.003s
I20260709 12:23:53.073598  6840 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:53.075583  6840 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:53.075263  6758 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.076056  6840 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:53.076301  6736 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.074561 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.092407 31820 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:53.092753 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.078266  6758 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:53.096434 31823 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 9. (term mismatch)
I20260709 12:23:53.101112 31823 pending_rounds.cc:85] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 8
I20260709 12:23:53.101414 31823 pending_rounds.cc:107] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 16.9
I20260709 12:23:53.101708 31823 raft_consensus.cc:2889] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:53.072330  6697 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.110013 31820 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:23:53.111266 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:53.078308  6736 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:53.096995  6828 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:23:53.113790  6602 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 17 current term: 17 is higher.
I20260709 12:23:53.124379 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.083606  6247 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.105474  6842 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:53.135185 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.115654  6497 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:53.141693 31824 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:53.131600  6828 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:53.136788  6489 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.822s	user 0.010s	sys 0.003s
I20260709 12:23:53.134265  6507 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:53.133312  6841 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:23:53.140714  6843 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:53.143576  6825 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:53.144371  6689 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:53.147806  6823 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:53.161907  6827 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.278s	user 0.001s	sys 0.001s
I20260709 12:23:53.165647 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3309) took 2004 ms. Trace:
I20260709 12:23:53.124270  6736 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.017s
I20260709 12:23:53.142223 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:53.176496 31671 rpcz_store.cc:276] 0709 12:23:51.161391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.160972 (+999581us) service_pool.cc:224] Handling call
0709 12:23:52.161386 (+   414us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:52.962482 (+801096us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.165283 (+202801us) spinlock_profiling.cc:243] Waited 792 ms on lock 0x7b54001b35d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:53.165557 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1821119872}
I20260709 12:23:53.115101  6697 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.184753 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:53.169991  6501 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.174817  6689 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:53.195724 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 149071us
I20260709 12:23:53.170804  6489 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:53.205256 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.206234 31744 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:23:53.187629  6604 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.609s	user 0.013s	sys 0.000s
I20260709 12:23:53.207930 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.194005  6842 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:53.210104 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.210537 31676 raft_consensus.cc:606] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:53.220038 31676 raft_consensus.cc:614] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:53.207808  6647 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:53.211788  6706 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:53.220703  6421 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:53.212977  6489 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:53.201566  6656 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.201568  6825 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:53.223697 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:53.224776 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.200064  6501 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.219852  6842 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:53.224648  6821 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:23:53.223433  6647 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:23:53.229976  6825 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:53.227387  6823 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:53.231244  6689 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.231446  6842 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:53.249832 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:53.251257 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.252737 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:53.253036 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.254976 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:53.254590 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.232856  6489 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:53.264241  6841 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:53.265440  6247 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:53.225988  6828 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.267660  6697 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:53.274055 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 1.50229e+06us
I20260709 12:23:53.274770  6729 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:23:53.255642 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3311) took 1962 ms. Trace:
I20260709 12:23:53.276906  6752 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:53.262425  6643 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:23:53.255623  6823 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:53.287010 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:53.287473  6823 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:53.287635 31674 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:53.287926  6823 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:53.291061 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:53.255793  6735 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:53.292480 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:53.293593 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:53.295054 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:53.295768  6743 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.296272  6743 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.296693  6640 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.297183  6640 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.297494 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:53.278764 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:53.280545 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:53.303314 31818 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:53.302186 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:53.305040 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:53.298991 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:53.307695 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:53.311880 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:53.285018 31678 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:23:53.313557 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:53.306615 31669 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:23:53.310199 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:53.312948 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:53.314762 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:53.315373 31749 raft_consensus.cc:2368] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18.
I20260709 12:23:53.315692 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:53.316213 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:53.317371 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:53.286113 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3134) took 1017 ms. Trace:
I20260709 12:23:53.319456 31749 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:53.283562 31672 rpcz_store.cc:276] 0709 12:23:51.292973 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.228803 (+935830us) service_pool.cc:224] Handling call
0709 12:23:52.541078 (+312275us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:52.541753 (+   675us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.255428 (+713675us) spinlock_profiling.cc:243] Waited 312 ms on lock 0x7b5400254ed0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:53.255580 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":717282176}
I20260709 12:23:53.298484  6743 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:53.326293 31674 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:53.327592 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:53.255378  6647 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.229501  6656 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.340402 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 16 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:53.346488 31749 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:53.318014 31821 rpcz_store.cc:276] 0709 12:23:52.268410 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.321097 (+ 52687us) service_pool.cc:224] Handling call
0709 12:23:52.370029 (+ 48932us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:52.592651 (+222622us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:52.610950 (+ 18299us) log.cc:844] Serialized 14 byte log entry
0709 12:23:53.149954 (+539004us) spinlock_profiling.cc:243] Waited 8.9 ms on lock 0x7b1800313120. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:23:53.219290 (+ 69336us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:53.223214 (+  3924us) log.cc:844] Serialized 26 byte log entry
0709 12:23:53.223369 (+   155us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:53.223402 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.223547 (+   145us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:53.275395 (+ 51848us) raft_consensus.cc:1685] finished
0709 12:23:53.285796 (+ 10401us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.286025 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33532,"prepare.queue_time_us":16834,"prepare.run_cpu_time_us":61,"prepare.run_wall_time_us":5528,"raft.queue_time_us":79189,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":11317,"spinlock_wait_cycles":20461312,"thread_start_us":241072,"threads_started":2,"wal-append.queue_time_us":593526}
I20260709 12:23:53.348196 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:53.349181 32370 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:53.349337 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:53.349857 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 94954.5us
I20260709 12:23:53.351861 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:53.352079 31718 leader_election.cc:400] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26.
I20260709 12:23:53.294008  6735 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:53.353664 31753 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:23:53.285575  6604 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.355660 31745 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 13. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
W20260709 12:23:53.308243  6827 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:23:53.324396  6689 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:53.300976  6247 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.075s
I20260709 12:23:53.354022  6743 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:53.325297  6697 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.074s
I20260709 12:23:53.354389  6647 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.366420 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.354807 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:53.367909 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:53.368088 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:53.382951 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:53.453845 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:53.454254 31745 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:53.454569 31824 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:53.390797 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3283) took 1192 ms. Trace:
I20260709 12:23:53.455237 32375 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:23:53.360141 31750 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
W20260709 12:23:53.434363 31752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3188) took 2689 ms (client timeout 2038 ms). Trace:
W20260709 12:23:53.450778 31744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3213) took 1616 ms (client timeout 1905 ms). Trace:
I20260709 12:23:53.423247 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3284) took 1223 ms. Trace:
W20260709 12:23:53.515570 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 149253us
I20260709 12:23:53.433673 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3141) took 1092 ms. Trace:
I20260709 12:23:53.455626 31819 rpcz_store.cc:276] 0709 12:23:52.198861 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.199373 (+   512us) service_pool.cc:224] Handling call
0709 12:23:52.199793 (+   420us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:52.200195 (+   402us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:23:52.423613 (+223418us) log.cc:844] Serialized 14 byte log entry
0709 12:23:52.610910 (+187297us) log.cc:844] Serialized 14 byte log entry
0709 12:23:52.857689 (+246779us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:52.861947 (+  4258us) log.cc:844] Serialized 52 byte log entry
0709 12:23:52.862139 (+   192us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:52.862472 (+   333us) log.cc:844] Serialized 14 byte log entry
0709 12:23:53.061911 (+199439us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.061997 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:53.161279 (+ 99282us) spinlock_profiling.cc:243] Waited 4.41 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:53.390429 (+229150us) raft_consensus.cc:1685] finished
0709 12:23:53.390466 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.390702 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":137862,"prepare.queue_time_us":165311,"prepare.run_cpu_time_us":181,"prepare.run_wall_time_us":11859,"raft.queue_time_us":241542,"raft.run_cpu_time_us":379,"raft.run_wall_time_us":10552,"spinlock_wait_cycles":10140416,"thread_start_us":653456,"threads_started":4,"wal-append.queue_time_us":259198}
I20260709 12:23:53.427443 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:53.441164 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:53.442341 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:53.413826 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3317) took 1556 ms. Trace:
W20260709 12:23:53.442790 32377 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 1068) took 1604 ms (client timeout 1638 ms). Trace:
I20260709 12:23:53.384192  6683 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.437754 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.590106 31670 rpcz_store.cc:276] 0709 12:23:51.857350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.904057 (+ 46707us) service_pool.cc:224] Handling call
0709 12:23:53.413572 (+1509515us) spinlock_profiling.cc:243] Waited 649 ms on lock 0x7b5400037354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:53.413741 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1492349696}
I20260709 12:23:53.375829  6489 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:53.439117 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:53.588721 31747 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:53.591928 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 154282us
I20260709 12:23:53.593755 31669 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 22: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:53.562973 31821 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:23:53.588186 31820 rpcz_store.cc:276] 0709 12:23:52.199614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.199756 (+   142us) service_pool.cc:224] Handling call
0709 12:23:53.423029 (+1223273us) spinlock_profiling.cc:243] Waited 868 ms on lock 0x7b54001c93d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:53.423171 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1997213952}
W20260709 12:23:53.516424 31752 rpcz_store.cc:269] 0709 12:23:50.744464 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:50.748507 (+  4043us) service_pool.cc:224] Handling call
0709 12:23:53.434105 (+2685598us) spinlock_profiling.cc:243] Waited 1.84 s on lock 0x7b54002ac6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:53.434229 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4236754816}
I20260709 12:23:53.368326  6729 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:53.559712 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:53.451366 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:53.592222 31744 rpcz_store.cc:269] 0709 12:23:51.834043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.995026 (+160983us) service_pool.cc:224] Handling call
0709 12:23:53.450532 (+1455506us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b5400212fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:53.450710 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2401334144}
I20260709 12:23:53.587920 31822 rpcz_store.cc:276] 0709 12:23:52.341725 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.342846 (+  1121us) service_pool.cc:224] Handling call
0709 12:23:53.433390 (+1090544us) spinlock_profiling.cc:243] Waited 672 ms on lock 0x7b5400166cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:53.433594 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1544554368}
I20260709 12:23:53.594648 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:53.434224 31827 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:53.588806 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.595983 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:53.589815 32377 rpcz_store.cc:269] 0709 12:23:51.838414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:51.838539 (+   125us) service_pool.cc:224] Handling call
0709 12:23:53.442603 (+1604064us) spinlock_profiling.cc:243] Waited 737 ms on lock 0x7b54004c4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:53.442731 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1695762688}
I20260709 12:23:53.423517  6247 consensus_queue.cc:1243] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 20.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:23:53.373061  6768 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:53.604002 31744 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:23:53.604393 31820 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:53.601038 31824 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:53.415776  6851 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
W20260709 12:23:53.403618  6854 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 12:23:53.444952  6843 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:23:53.402434  6849 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.207s	user 0.001s	sys 0.000s
I20260709 12:23:53.416815  6656 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:53.417148  6850 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.209s	user 0.000s	sys 0.001s
I20260709 12:23:53.694591 31824 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:53.630509 31791 leader_election.cc:400] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18.
I20260709 12:23:53.415503  6501 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:53.417716  6682 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.709399  6682 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.440807  6841 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.436753  6647 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:53.415148  6640 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:53.635941 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 184640us
I20260709 12:23:53.559355  6643 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.553835  6759 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:23:53.423079  6697 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:53.738864 31821 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:23:53.425671  6651 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.369012  6709 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.379333  6604 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.399963  6700 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.207s
W20260709 12:23:53.641657 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 199401us
I20260709 12:23:53.744807 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:53.643455 31819 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:23:53.646881 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:53.747205 31823 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:53.748495 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:53.749343 31823 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:53.652443 31820 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:23:53.653582  6783 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:23:53.653888  6842 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.657176 31827 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:23:53.425959  6735 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.16, Last appended by leader: 16, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.639338  6843 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:53.368844  6840 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.664551  6657 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:53.605024  6683 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.415958  6743 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.640820  6752 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 12:23:53.429489  6823 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.647504  6857 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:53.637203  6815 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:53.427933  6814 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.382812  6825 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.608461  6794 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:23:53.600678  6729 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:23:53.687685 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 385594us
I20260709 12:23:53.819973 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:53.699645 31791 leader_election.cc:403] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 election: Cancelling election due to peer responding with higher term
I20260709 12:23:53.711131 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3295) took 1103 ms. Trace:
I20260709 12:23:53.711702 31749 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:23:53.821786 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 191410us
I20260709 12:23:53.823869 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:53.713503 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 359887us
I20260709 12:23:53.735453 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:53.829062 31824 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:23:53.756188 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:53.761597  6737 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:23:53.785607  6497 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.788113  6604 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:53.761086  6709 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.790722  6660 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:53.790866  6659 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:53.753715  6651 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.719348  6841 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.760675  6640 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 12:23:53.738716  6697 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:53.761767  6750 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:53.738374  6784 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.070s
I20260709 12:23:53.783571  6657 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:53.765326  6783 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:53.739145  6643 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.821372 31826 rpcz_store.cc:276] 0709 12:23:52.607463 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.607592 (+   129us) service_pool.cc:224] Handling call
0709 12:23:52.821721 (+214129us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:52.963046 (+141325us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:52.963099 (+    53us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:53.011250 (+ 48151us) log.cc:844] Serialized 52 byte log entry
0709 12:23:53.140661 (+129411us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:53.140708 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.140777 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:53.710817 (+570040us) raft_consensus.cc:1685] finished
0709 12:23:53.710839 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.711046 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17819,"raft.queue_time_us":147863,"raft.run_cpu_time_us":95,"raft.run_wall_time_us":96,"thread_start_us":189308,"threads_started":2,"wal-append.queue_time_us":113161}
I20260709 12:23:53.833767 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 15 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.870087 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:53.827906 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:53.871519 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 43710.3us
I20260709 12:23:53.873621 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:53.840713 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 237271us
I20260709 12:23:53.823096  6729 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:53.819700  6828 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:53.835421  6247 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election lost for term 17. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 17. Current term is 18.
I20260709 12:23:53.840516  6497 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.846803  6859 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:53.851823  6858 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:53.915691  6858 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:53.915937  6737 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:53.916216  6858 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:53.863931  6864 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:53.916608  6737 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.917121  6864 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.861419 32371 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:23:53.919400  6737 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:53.919940  6864 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:53.850711  6660 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:53.915704  6859 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:53.921094  6660 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.921437  6859 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.874708 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:53.922912  6743 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:53.836980  6736 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:53.873883  6841 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:53.928105 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53520.1us
W20260709 12:23:53.927263 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53937.6us
I20260709 12:23:53.930797 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:53.932928  5949 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:23:53.885001  6759 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader election lost for term 15. Reason: could not achieve majority
I20260709 12:23:53.857547  6794 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:53.901497  6247 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:53.847857  6659 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.908862  6497 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:53.846885  6661 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.029s
I20260709 12:23:53.865295  6657 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:53.891930  6689 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.019s
I20260709 12:23:53.878228  6682 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:53.911375  6821 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:23:53.950151 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 130269us
I20260709 12:23:53.951625 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:53.912774  6709 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:53.913267  6697 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.989198  6697 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:53.991461  6743 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:23:53.992403  6743 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:53.992744  6743 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:53.993651  6743 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.865312  6750 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.914709  6828 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:53.925873  6683 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:53.928138  6736 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:53.938146  5949 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
W20260709 12:23:53.879830 31716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.508s	user 0.011s	sys 0.021s
W20260709 12:23:53.997844 31716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.626s	user 0.011s	sys 0.021s
I20260709 12:23:53.943265  6862 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:23:53.927269  6814 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:53.935904  6706 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:53.939421  6823 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:53.937318  6864 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:53.896965 31749 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:23:53.973985  6640 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:53.976336 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1079) took 1021 ms. Trace:
I20260709 12:23:53.976143  6783 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:53.977434  6842 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:53.977883  6784 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:53.947363  6840 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:53.983901  6659 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:53.994825  6455 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:53.937057  6647 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.012s
I20260709 12:23:53.936933  6489 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.012s
I20260709 12:23:53.941336  6794 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.016935 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:54.049548 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 293483us
W20260709 12:23:54.030752 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.032177 32372 rpcz_store.cc:276] 0709 12:23:52.955312 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:52.955460 (+   148us) service_pool.cc:224] Handling call
0709 12:23:52.955798 (+   338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:53.299081 (+343283us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:53.299145 (+    64us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:53.338507 (+ 39362us) log.cc:844] Serialized 26 byte log entry
0709 12:23:53.759932 (+421425us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:53.759994 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:53.760114 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:53.976031 (+215917us) raft_consensus.cc:1685] finished
0709 12:23:53.976064 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:53.976253 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":345103,"raft.queue_time_us":344407,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":31064,"thread_start_us":376145,"threads_started":1,"wal-append.queue_time_us":574729}
W20260709 12:23:54.053239 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 246418us
I20260709 12:23:53.999601  6736 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.014927  6743 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:54.000829  6862 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:54.000609  6821 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:54.055363 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:54.037014  6651 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:54.004770  6858 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.042294  6784 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:54.046011  6455 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:54.074388 31824 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:23:54.031615  6814 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.025s
I20260709 12:23:54.056386  6828 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:54.051668  6640 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:54.103094  6640 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:54.104296  6640 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.104986 31822 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:54.105563  6640 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:54.078089 31821 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:23:54.015025  6729 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.071353 31826 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:23:54.108481  6729 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:54.082998  6784 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:54.084712  6497 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:54.086009 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:54.118680 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:54.119556 31748 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:54.120223 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:23:54.098843 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 43836.3us
I20260709 12:23:54.100087  6868 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:54.101075  6643 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:54.068958  6659 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.033s
I20260709 12:23:54.058220  6501 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:54.078600  6682 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:23:54.102022 31827 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:54.102118  6683 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:54.120829 31747 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:54.121487 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:54.128947 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:54.130285 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:54.132357 31744 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:23:54.052285  6857 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:54.124486 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:54.144392 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:23:54.147651 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 170838us
I20260709 12:23:54.154563 31747 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:23:54.129397  6659 consensus_queue.cc:1243] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 22.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 11
I20260709 12:23:54.155654 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:54.126197  6497 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.158344  6735 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:23:54.158596 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 265154us
I20260709 12:23:54.110004  6843 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.159150 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:54.166255 31751 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:54.187067 31747 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:23:54.191463 31747 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:23:54.193491 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:54.196672 31669 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 12. (term mismatch)
I20260709 12:23:54.197144 31669 pending_rounds.cc:85] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 11
I20260709 12:23:54.197485 31669 pending_rounds.cc:107] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 19.12
I20260709 12:23:54.197806 31669 raft_consensus.cc:2889] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:54.198334 31669 pending_rounds.cc:107] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 12:23:54.205868 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.207257 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:54.207499 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.207280 31670 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:54.203264 31669 raft_consensus.cc:2889] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:54.209889 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:54.210647 31744 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:54.213760 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:54.214538 31744 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:54.218245 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:54.219120 31749 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:54.220072 31676 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:54.225339 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:54.228439 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:54.228752 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.229408 31677 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:54.229799  6497 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:54.200518  6729 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:54.230391  6497 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:54.190230  6700 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:23:54.233134  5949 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:54.233611  5949 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.234858 31746 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:23:54.236084 31744 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:54.166509  6874 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:54.231832 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:54.233556 31747 pending_rounds.cc:85] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 12
I20260709 12:23:54.244459 31747 pending_rounds.cc:107] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260709 12:23:54.244611 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:54.244860 31747 raft_consensus.cc:2889] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:54.185120  6794 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:54.244206 31827 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:54.198000  6843 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:54.238929 31749 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 17. (term mismatch)
I20260709 12:23:54.251466 31749 pending_rounds.cc:85] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 16
I20260709 12:23:54.251806 31749 pending_rounds.cc:107] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 22.17
I20260709 12:23:54.252096 31749 raft_consensus.cc:2889] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:54.252830 31746 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:23:54.239073 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.240209 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:54.254865 31752 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:23:54.184146  6821 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.9, Last appended by leader: 9, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.283913 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:54.241791  6759 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:54.242379 32372 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 15 index: 12. Preceding OpId from leader: term: 18 index: 14. (index mismatch)
I20260709 12:23:54.242654 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:54.188202  6656 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:54.344132 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:54.343642 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.345224 31670 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:23:54.238312 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:54.263646 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:54.231438  6700 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:54.262105 31748 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:23:54.198887  6858 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:54.220098  6660 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.022s
I20260709 12:23:54.342942 31676 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:23:54.355211 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:54.184442  6877 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:54.345293 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.380992  5949 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:54.381551 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.346419 32375 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:23:54.266333 32377 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:23:54.343277 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.382469 31673 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:23:54.386078 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.342753 31674 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:54.343840 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:54.318523 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 376162us
I20260709 12:23:54.347487 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.236498  6706 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.350890 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:54.351580 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:54.245607  6489 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.379297 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.387739 31672 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:54.389379 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.391903 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:54.392239 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:54.396193 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.399883 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.285504  6828 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.406952 31827 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:54.283824  6874 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:54.228473  6657 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.340945  6759 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.424666 31670 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 14 index: 8. Preceding OpId from leader: term: 17 index: 9. (index mismatch)
I20260709 12:23:54.406452 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:54.404726  6706 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.429013 31670 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:23:54.339658  6735 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:54.379477  6840 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.187791  6455 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.420795 31676 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:54.436056 31718 leader_election.cc:400] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20.
I20260709 12:23:54.379850  6842 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.390707  6689 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.344683  6656 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:54.282872  6882 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.172s
I20260709 12:23:54.430186 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.398118  6651 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:54.440584 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 252107us
W20260709 12:23:54.439607 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 244036us
I20260709 12:23:54.428181  6759 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:54.448802 31677 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:54.246313  6857 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.450584 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:54.409247  6841 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:54.442539  6868 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:54.453269 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:54.471104 31827 raft_consensus.cc:2376] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23. Re-sending same reply.
I20260709 12:23:54.404245  6823 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.036s
W20260709 12:23:54.479521  6875 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.196s	user 0.000s	sys 0.000s
I20260709 12:23:54.480418  6862 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.456929 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:54.480674  6497 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:23:54.481755 31818 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:54.460775 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:54.483058 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:54.484449 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:54.410658  6489 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.488484  6489 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:54.489455 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:54.491406 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:54.493040  6858 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:54.493595  6858 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.494055  6858 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:54.451612  6864 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260709 12:23:54.468940 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:54.469938 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:54.496644 31823 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:23:54.419245  6828 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.478029 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:54.497947 31827 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:54.478940 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:54.468756 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:54.499223 31819 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 15: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:54.480747 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:54.499439 31824 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:23:54.508495 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.508739 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:54.516144 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:54.516959 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:54.526849 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.443686  6840 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.521597 31669 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:54.529517 31827 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:54.446779  6842 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.449561  6689 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.454080  6815 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.055s
I20260709 12:23:54.526460  6735 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:54.539397 31820 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:54.540967  6689 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:54.538385 31717 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:54.517462  6868 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.443825  6700 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.526566  6876 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:54.445605 31672 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:23:54.456796  6651 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.514858  6877 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.022s
I20260709 12:23:54.551586 31674 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:54.552984 31669 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:23:54.514560  6859 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.042s
W20260709 12:23:54.547060  6888 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260709 12:23:54.569653  6885 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:23:54.557842 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 107340us
W20260709 12:23:54.547175 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 140294us
I20260709 12:23:54.553591 31717 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:54.515285  6874 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.573424 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.546181  6864 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:23:54.581916 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:54.584941 32373 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:23:54.550294  6889 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:54.587877 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:54.547940  6894 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:54.590257  5949 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:54.591969  5949 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:23:54.590345 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:54.594595  6823 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:54.595201  6823 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.596196 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:54.594655 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.600834  6904 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:54.598084  6823 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:54.601451  6904 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:54.587038  6859 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:23:54.602447  6906 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.017s
I20260709 12:23:54.604506  6842 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:54.604903  6840 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:54.606204  6828 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:54.526674  6656 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.593719  6736 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:54.609385  6750 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:54.586743  6489 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:54.611147  6706 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:54.612126  6657 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:54.588821  6841 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.024s
I20260709 12:23:54.602497  6794 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.014s
I20260709 12:23:54.587960  6881 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:23:54.590700 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:54.619669 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 201934us
W20260709 12:23:54.618363  6743 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.552s	user 0.008s	sys 0.000s
W20260709 12:23:54.621600 31718 connection.cc:256] RPC call timeout handler was delayed by -1.1418632859999889s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1638s for a response
W20260709 12:23:54.622324 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 166559us
W20260709 12:23:54.684238 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.598037 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:54.685263 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 87316us
I20260709 12:23:54.685526 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:54.637893 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50938.5us
I20260709 12:23:54.622754  6877 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:54.687749  6877 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:54.687858 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:54.688566 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.688809  6877 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.689869 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.588850  6689 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:54.690850  6489 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:23:54.691211 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:54.691916 31669 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:54.692378 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:54.693051 31749 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:23:54.693468 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:54.606927  6892 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
W20260709 12:23:54.644120 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 135465us
I20260709 12:23:54.695966 31749 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:23:54.697170 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:54.697810 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:54.698189 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:54.647641 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3236) took 1290 ms. Trace:
I20260709 12:23:54.554723  6868 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.698896  6857 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:54.700551  6857 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:54.699213 31750 rpcz_store.cc:276] 0709 12:23:53.356702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:53.539966 (+183264us) service_pool.cc:224] Handling call
0709 12:23:53.540448 (+   482us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:53.549077 (+  8629us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:23:53.630186 (+ 81109us) log.cc:844] Serialized 14 byte log entry
0709 12:23:54.495342 (+865156us) spinlock_profiling.cc:243] Waited 2.33 ms on lock 0x7b180010d3e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:23:54.641641 (+146299us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:54.645326 (+  3685us) log.cc:844] Serialized 26 byte log entry
0709 12:23:54.645581 (+   255us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:54.645639 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:54.645797 (+   158us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:54.647181 (+  1384us) raft_consensus.cc:1685] finished
0709 12:23:54.647314 (+   133us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:54.647546 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25887,"prepare.queue_time_us":80687,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":7590,"raft.queue_time_us":140763,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":39269,"spinlock_wait_cycles":5362048,"thread_start_us":449060,"threads_started":2,"wal-append.queue_time_us":430175}
I20260709 12:23:54.701090  6868 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:54.690852  6689 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.702243  6689 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:54.635046  6859 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:54.635353  6904 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:54.646242  6752 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:23:54.666842 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.665135  6881 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:54.690474  6877 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:54.727569  6881 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.729118  6881 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:54.731415 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:54.731755  6604 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:54.732033 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:54.733060 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:54.671171 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 80615.2us
I20260709 12:23:54.733915 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 14 index: 8 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:54.734416 31819 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:23:54.734586 31823 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:23:54.735606 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:54.736292 31819 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:23:54.736402 31818 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:23:54.737246 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:54.672056  6884 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:54.748415 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:54.677711  6858 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:54.683405  6898 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 12:23:54.683768  6729 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:54.698999 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:54.759723 32370 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:54.713449  5949 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:23:54.708276 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:54.762821 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 54639.3us
I20260709 12:23:54.718032 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:54.721233 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:54.738093 31819 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 19 because replica has last-logged OpId of term: 17 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:23:54.738166 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:54.735636 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:54.764936 32374 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
W20260709 12:23:54.765339 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44225.2us
I20260709 12:23:54.766142 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:54.739447 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:23:54.691366  6890 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:23:54.753921 31748 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 16. Preceding OpId from leader: term: 22 index: 17. (index mismatch)
I20260709 12:23:54.697855 32369 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:23:54.691332  6489 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:54.774573 31672 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:23:54.674136  6759 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.776889 31822 raft_consensus.cc:2368] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22.
I20260709 12:23:54.770792 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:54.722730  6823 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:23:54.722883  6859 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:54.784312 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:54.784564 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 45197us
I20260709 12:23:54.785110  6903 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:54.789134  6903 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.789387 32369 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:23:54.789515  6903 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:54.784128  6828 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:54.791117  6828 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.791520  6828 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:54.785261  6497 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:54.792402  6497 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:54.792164 31574 catalog_manager.cc:5697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 14 to 18, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 18 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:54.793183  6497 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.661171  6736 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.751024  6901 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:54.797528 32376 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:54.787137 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:54.803984 32374 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:54.787535 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:54.747980  6884 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.794449  6497 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:54.794535 31574 catalog_manager.cc:5697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 17 to 19, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 19 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:23:54.722293  6247 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.076s
I20260709 12:23:54.773345  6895 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:54.752224  6743 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.071s
I20260709 12:23:54.807437  6657 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:54.785565  6489 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:23:54.843538 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 72851.3us
I20260709 12:23:54.823398  6884 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:54.845695 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:54.838248  6876 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:54.847462  6682 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:54.812644  6729 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:54.849591  6729 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.848634 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:54.852443  6729 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:54.840167  6657 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:54.854550  6657 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:54.848670  6876 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 17.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.829515  6901 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:54.881176 31744 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 15 index: 10. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:54.886483 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:54.878607 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 15 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:54.839622  6794 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:23:54.832144  6909 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
W20260709 12:23:54.896775 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.858448  6700 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:23:54.892899  6640 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.787s	user 0.014s	sys 0.001s
I20260709 12:23:54.869525  6884 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:54.913389  6909 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:54.919862  6657 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:54.920321  6909 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:54.837694  6842 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 12:23:54.928102 32369 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 14. (term mismatch)
I20260709 12:23:54.928632 32369 pending_rounds.cc:85] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 13
I20260709 12:23:54.928912 32369 pending_rounds.cc:107] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 20.14
I20260709 12:23:54.929193 32369 raft_consensus.cc:2889] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:54.925670 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:54.922488  6657 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:54.955725  6736 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:54.979686  6657 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:55.098661  6910 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:55.026139  6864 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 14 FOLLOWER]: Advancing to term 15
W20260709 12:23:55.056533  6914 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:23:55.025458  6895 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:55.119835  6920 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:23:55.110726  6910 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.123092  6904 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.125360  6910 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:55.126498  6904 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:55.129299  6909 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.132243  6909 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:55.003994  6901 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.135372  6903 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.138374  6903 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:55.171669 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:55.185711 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 300950us
W20260709 12:23:55.191584 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 406087us
I20260709 12:23:55.195163 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:55.197041 31744 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:55.210583 31827 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 19: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:55.215754 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:55.219463 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:55.225600 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:55.228870 31672 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:55.238523  6657 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:55.239372  6657 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:55.230041  6736 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:55.311230  6903 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:23:55.135293  6884 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:55.347452  6895 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.059178  6842 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:23:55.298408 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 411897us
W20260709 12:23:55.381991 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.314126  6903 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.049s
W20260709 12:23:55.398859 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.016s	sys 0.017s
W20260709 12:23:55.399451 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.602s	user 0.016s	sys 0.017s
I20260709 12:23:55.402663  6907 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.402824  6604 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:55.403755  6604 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.403807  6907 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.406870  6904 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.407755  6904 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.408720  6924 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.408952  6907 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:55.409718  6924 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.410785  6604 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:55.411541 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.101940  6858 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:55.424113 31744 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:23:55.422971  6843 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.446400 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:55.447203 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:55.447739 31824 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:55.450302 31825 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:23:55.451704  6843 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.453890 32374 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:55.458956  6934 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.459831  6934 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.460547 31824 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:23:55.461578  6843 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:55.454042 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 89358.9us
W20260709 12:23:55.465158 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.464915  6842 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:55.466758  6929 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:55.467037  6842 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.467928 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:55.468473  6929 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.471048  6904 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:55.473691  6929 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:55.462589  6934 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:55.488729 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:55.489302 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:55.491313  6934 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:23:55.492050  6934 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:55.495013  6903 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:23:55.495436 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:55.497169  6859 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.718s	user 0.013s	sys 0.009s
I20260709 12:23:55.497326  6903 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:55.497435 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:55.393967  6736 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.499809 31825 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
W20260709 12:23:55.508018  6784 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.397s	user 0.011s	sys 0.013s
I20260709 12:23:55.508411 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:55.508620  6784 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.509332 31825 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:55.493678  6934 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.506364 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:55.515079 32378 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:55.511324 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:55.502189  6736 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:55.526436 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:55.547362  6647 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:23:55.548805  6647 consensus_queue.cc:1243] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 22.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:23:55.549561  6647 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:55.549978  6647 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:55.551589  6647 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.491333 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1020) took 1224 ms. Trace:
I20260709 12:23:55.553297 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:23:55.554927  6926 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
W20260709 12:23:55.528956  5949 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.011s	sys 0.006s
I20260709 12:23:55.514209  6930 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:55.531174  6901 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:55.579751  5949 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.742s	user 0.014s	sys 0.006s
I20260709 12:23:55.575971 31819 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:23:55.586450 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 71973.1us
I20260709 12:23:55.587654 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:55.517071  6859 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.587903 31672 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:55.562582 32376 rpcz_store.cc:276] 0709 12:23:54.267303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.267484 (+   181us) service_pool.cc:224] Handling call
0709 12:23:54.267782 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:54.812176 (+544394us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.490703 (+678527us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b5400646c54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:55.491099 (+   396us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1217852288}
I20260709 12:23:55.566569  6651 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:55.591632  6497 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.760s	user 0.007s	sys 0.012s
W20260709 12:23:55.592223 31748 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.666s	user 0.008s	sys 0.005s
W20260709 12:23:55.600344 31748 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.797s	user 0.010s	sys 0.005s
I20260709 12:23:55.602325 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:55.610355 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:55.611183 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:55.612228 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:55.613554 31745 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:23:55.616547 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:55.625147  6881 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.895s	user 0.011s	sys 0.008s
I20260709 12:23:55.618826  6247 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:55.626291 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:55.627714  6828 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.836s	user 0.013s	sys 0.006s
I20260709 12:23:55.628144 31823 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:23:55.628216  6828 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.629858 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:55.631865 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 78787.6us
I20260709 12:23:55.630952 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:55.633199  6929 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:55.634697  6930 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.637321  6930 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:55.637979 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:55.642937 31744 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:23:55.643200 31823 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 12:23:55.645416  6858 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:55.652297  6689 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.949s	user 0.014s	sys 0.007s
I20260709 12:23:55.652894  6689 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.654996 31820 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:55.657670 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:55.658965  6930 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:23:55.659430  6930 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.659806  6930 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:55.667047  6910 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:55.671177  6939 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:55.672062  6706 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:55.672240  6939 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.673574  6840 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:55.674085  6840 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.674492  6840 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:55.677289  6939 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:55.679728  6924 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:55.634719  6904 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.700532  6904 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.701077  6657 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.703034 32373 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: []
I20260709 12:23:55.606601  6736 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:23:55.705152  6736 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.705608  6736 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:55.706060  6859 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:55.711521  6784 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:55.710888 31574 catalog_manager.cc:5697] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 18 to 21, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 21 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:23:55.713743  6489 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.832s	user 0.012s	sys 0.008s
I20260709 12:23:55.714466  6489 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.600384  6647 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:55.717653 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3331) took 1365 ms. Trace:
I20260709 12:23:55.631055  6876 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:55.718259 31674 rpcz_store.cc:276] 0709 12:23:54.352328 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.423009 (+ 70681us) service_pool.cc:224] Handling call
0709 12:23:54.551410 (+128401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:54.555038 (+  3628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.717160 (+1162122us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b54001810d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:55.717434 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":253966208}
I20260709 12:23:55.719344  6876 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:55.719935  6876 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:55.727260 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 141752us
I20260709 12:23:55.713654 31574 catalog_manager.cc:5697] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 12 to 21, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 21 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:23:55.753798  6700 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.841s	user 0.014s	sys 0.005s
I20260709 12:23:55.630481  6828 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:55.741626 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:55.646217  6858 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.765609 31825 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:23:55.760505  6497 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:55.799610 31823 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:23:55.808143 31819 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:23:55.818245 31751 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
W20260709 12:23:55.818885 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:55.824656  6794 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:23:55.805174  6929 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.023s
I20260709 12:23:55.828197  6651 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.829943  6904 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:55.830200  6651 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.832588  6858 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:55.801364  6881 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:55.827349  6683 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:23:55.835770 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 64297.2us
I20260709 12:23:55.745986  6901 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.836845  6881 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:23:55.841456  6944 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:23:55.841354  6881 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.11, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.836640  6683 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:55.843870  6864 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.731s	user 0.009s	sys 0.005s
I20260709 12:23:55.844136  6683 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:55.844477  6864 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election lost for term 15. Reason: could not achieve majority
W20260709 12:23:55.849466 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 255726us
I20260709 12:23:55.851922 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:55.854169  6924 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:55.856575  6889 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:55.857564  6497 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:23:55.859407 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:55.861034 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3327) took 1109 ms. Trace:
I20260709 12:23:55.865505 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3258) took 1124 ms. Trace:
I20260709 12:23:55.867688  6864 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:23:55.869714  6904 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:55.871832 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:55.874997  6752 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:55.875511  6752 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:55.879040  6840 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.879938  6759 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:55.870410  6904 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.914794 31819 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: []
I20260709 12:23:55.884330  6924 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:55.916951  6904 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:55.883030  6957 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.052s	user 0.000s	sys 0.002s
I20260709 12:23:55.871986  6689 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:55.859486  6938 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:23:55.919042  6840 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:55.918919  6743 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:55.920351  6743 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:23:55.884460 31826 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:55.920807  6743 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:55.922991 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3318) took 1194 ms. Trace:
I20260709 12:23:55.923218  6924 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:23:55.887050 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 307520us
I20260709 12:23:55.888141 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2586) took 1302 ms. Trace:
I20260709 12:23:55.927853 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:55.926707 31746 rpcz_store.cc:276] 0709 12:23:54.585578 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.585745 (+   167us) service_pool.cc:224] Handling call
0709 12:23:54.587424 (+  1679us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:54.702036 (+114612us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:54.702074 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:54.702554 (+   480us) log.cc:844] Serialized 26 byte log entry
0709 12:23:54.883191 (+180637us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:54.883269 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.887431 (+1004162us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b54004033d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:55.887513 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:55.887588 (+    75us) raft_consensus.cc:1685] finished
0709 12:23:55.887607 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:55.887957 (+   350us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1882,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":26394,"spinlock_wait_cycles":262597504,"thread_start_us":177131,"threads_started":1,"wal-append.queue_time_us":250528}
I20260709 12:23:55.886361 31824 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 17 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:23:55.896613  6862 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:55.865952 31749 rpcz_store.cc:276] 0709 12:23:54.740659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.745549 (+  4890us) service_pool.cc:224] Handling call
0709 12:23:54.745901 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:55.864339 (+1118438us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:55.864385 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:55.864426 (+    41us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:55.864471 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.865082 (+   611us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b5400403dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:55.865124 (+    42us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:55.865307 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2571188096}
W20260709 12:23:55.892434 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47862.3us
I20260709 12:23:55.861440 31821 rpcz_store.cc:276] 0709 12:23:54.751337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.770807 (+ 19470us) service_pool.cc:224] Handling call
0709 12:23:55.419764 (+648957us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:55.444364 (+ 24600us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:55.444409 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:55.445178 (+   769us) log.cc:844] Serialized 26 byte log entry
0709 12:23:55.596779 (+151601us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:55.596847 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.604948 (+  8101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:55.859980 (+255032us) raft_consensus.cc:1685] finished
0709 12:23:55.860001 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:55.860647 (+   646us) spinlock_profiling.cc:243] Waited 540 ms on lock 0x7b54003ca5d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:55.860855 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11531,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":122,"spinlock_wait_cycles":1242804096,"thread_start_us":150844,"threads_started":1,"wal-append.queue_time_us":387308}
I20260709 12:23:55.930037 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:55.907217 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:55.907567  6864 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:23:55.907747 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3322) took 1178 ms. Trace:
I20260709 12:23:55.907992 31825 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: []
W20260709 12:23:55.908782 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3168) took 1448 ms (client timeout 1677 ms). Trace:
I20260709 12:23:55.909294  6657 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:55.911238 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1235) took 1172 ms. Trace:
I20260709 12:23:55.937594 31827 rpcz_store.cc:276] 0709 12:23:54.729331 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.732764 (+  3433us) service_pool.cc:224] Handling call
0709 12:23:55.907454 (+1174690us) spinlock_profiling.cc:243] Waited 476 ms on lock 0x7b540043cbd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:55.907656 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1094143232}
I20260709 12:23:55.922103  6743 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.923512 31826 rpcz_store.cc:276] 0709 12:23:54.728104 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.736187 (+  8083us) service_pool.cc:224] Handling call
0709 12:23:55.922716 (+1186529us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54000fc654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:55.922925 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2609610496}
I20260709 12:23:55.924326  6814 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:55.947182  6814 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.949681 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:55.950678 31672 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
W20260709 12:23:55.905403  6911 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.195s	user 0.004s	sys 0.000s
I20260709 12:23:55.898556  6930 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.932499 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:55.926688  6768 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:55.934369 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:55.934746 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:55.934813 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:55.955279 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:55.955559 31678 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: [19.10-19.10]
I20260709 12:23:55.956089 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:55.956806 32377 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:55.935600  6661 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:23:55.958352 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:55.958913  6661 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:55.959085  6930 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:55.959463  6661 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:55.959832 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:55.960695 31674 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:55.961710  6901 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:55.936115 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:23:55.938776 31822 rpcz_store.cc:269] 0709 12:23:54.460116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.460414 (+   298us) service_pool.cc:224] Handling call
0709 12:23:55.908331 (+1447917us) spinlock_profiling.cc:243] Waited 294 ms on lock 0x7b54001973d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:55.908586 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":676044672}
I20260709 12:23:55.942046 32369 rpcz_store.cc:276] 0709 12:23:54.738443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.790322 (+ 51879us) service_pool.cc:224] Handling call
0709 12:23:54.797957 (+  7635us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:54.929746 (+131789us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.910486 (+980740us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b54004c5c54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:55.911010 (+   524us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":298322176}
I20260709 12:23:55.963570 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:55.965473 32374 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:55.943913  6877 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.002s
I20260709 12:23:55.944716  6501 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:55.966114 32376 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:55.959300 31669 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:55.923009 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:55.966487  6768 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:55.927312  6924 consensus_queue.cc:1243] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 20.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:23:55.975225  6768 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.976105  6924 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:55.976444 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:55.977196 31826 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:23:55.977283  6924 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:55.978710  6924 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.16, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:55.980821 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:55.977036  6768 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:56.009723 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:56.021652  6643 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:55.910274  6683 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:55.968274  6840 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.027832 31675 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:56.024621  6814 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:55.975296 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52404.6us
I20260709 12:23:56.024322  6743 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:56.043368  6877 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:23:55.993911 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.054878 32374 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
W20260709 12:23:56.090904 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 161016us
I20260709 12:23:56.091126 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:56.044395  6689 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.099642  6759 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.103377  6689 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:23:56.103602  6759 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.103907 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:56.104625 32374 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:56.105118 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:56.105710  6689 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:23:56.105805  6759 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:56.106490 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:56.106866 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.108479 31718 leader_election.cc:400] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22.
I20260709 12:23:56.108776 31718 leader_election.cc:403] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:23:56.108959  6858 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.109630  6858 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.111006  5949 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:56.111559  5949 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:56.111886  6861 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 22
I20260709 12:23:56.112605  5949 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.115108 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:56.115934 31827 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:56.116074  6683 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:56.119959 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 75125.5us
I20260709 12:23:56.121419 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:56.121867 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2589) took 1432 ms. Trace:
I20260709 12:23:56.122823  6828 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:56.123557  6828 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:56.123912 31750 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:23:56.124603  6828 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.126082  6651 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:56.122443 31751 rpcz_store.cc:276] 0709 12:23:54.689183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.693209 (+  4026us) service_pool.cc:224] Handling call
0709 12:23:56.121421 (+1428212us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b5400295ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:56.121660 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2480568576}
I20260709 12:23:56.128562 31824 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: [20.13-20.13]
I20260709 12:23:56.130033 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 15 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:56.130941 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:56.122740  6904 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:23:56.133100 31827 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:56.105831 31677 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:23:56.131165 31822 raft_consensus.cc:2410] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 19 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 15 index: 10.
I20260709 12:23:56.138921  6876 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.141763  6876 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:56.142726  6489 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:56.145006  6861 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 21. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22.
I20260709 12:23:56.145094 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:56.146198 31819 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:56.141649 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:56.150921 32377 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:56.152220 32375 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:56.152285  6876 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:56.152060 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:56.156296  6840 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.160506  6752 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.161754 31746 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:56.071463  6877 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:56.164078 31675 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:56.164463  6877 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:56.168912  6840 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:56.173346 31745 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 14. (term mismatch)
I20260709 12:23:56.173808 31745 pending_rounds.cc:85] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 13
I20260709 12:23:56.174078 31745 pending_rounds.cc:107] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 23.14
I20260709 12:23:56.174430 31745 raft_consensus.cc:2889] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:56.179215 31819 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:23:56.181262 31792 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:56.190384 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:56.191294  6455 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:56.194170 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:56.195000  6909 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.195557 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:56.195701  6909 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.195557 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:23:56.197261 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:56.197511 31751 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:56.102370 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:56.129304  6656 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:56.208283 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:56.215011 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:56.217093 32374 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:56.042239  6736 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.220631 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:56.221643 31674 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
W20260709 12:23:56.225894 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 123829us
I20260709 12:23:56.198441  6501 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.218611 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:56.231092 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:56.222550 31827 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:23:56.232723 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:56.233042 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:56.232725  6501 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.235076 31824 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
W20260709 12:23:56.225091  6899 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.631s	user 0.006s	sys 0.004s
I20260709 12:23:56.218901  6489 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:56.234614 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 138496us
I20260709 12:23:56.238443  6706 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:56.219189  6840 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:56.240446  6794 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:56.241466  6689 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:56.221673  6643 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:56.210309  6651 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:56.258733  6960 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:23:56.259414  6960 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:56.259241  6651 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.255970 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:56.260982 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2580) took 2057 ms. Trace:
I20260709 12:23:56.261484 31747 rpcz_store.cc:276] 0709 12:23:54.203818 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.233022 (+ 29204us) service_pool.cc:224] Handling call
0709 12:23:54.233356 (+   334us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:54.247324 (+ 13968us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:54.247370 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:54.248009 (+   639us) log.cc:844] Serialized 52 byte log entry
0709 12:23:54.397588 (+149579us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:54.397789 (+   201us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:54.397870 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:56.196461 (+1798591us) spinlock_profiling.cc:243] Waited 45.6 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:56.260420 (+ 63959us) raft_consensus.cc:1685] finished
0709 12:23:56.260448 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:56.260774 (+   326us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40492,"raft.queue_time_us":38482,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":57037,"spinlock_wait_cycles":104855808,"thread_start_us":148018,"threads_started":1,"wal-append.queue_time_us":284830}
I20260709 12:23:56.263849 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:56.253129 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:56.266110  6889 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:56.266211  6929 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:56.266660  6889 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:56.267158  6889 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:56.266944  6814 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:23:56.267392  6947 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:56.268114  6947 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.268815  6814 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:56.269263 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3268) took 1030 ms. Trace:
I20260709 12:23:56.269667 31745 rpcz_store.cc:276] 0709 12:23:55.239257 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.624110 (+384853us) service_pool.cc:224] Handling call
0709 12:23:55.624510 (+   400us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:56.174949 (+550439us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.269006 (+ 94057us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x7b54004047d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:56.269190 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1260921728}
I20260709 12:23:56.271776  6904 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:56.272984  6825 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.274199  6643 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.274771  6842 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:56.275545  6842 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.276886  6643 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:56.277714  6661 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.284248  6689 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:56.285430  6689 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.287484  6647 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:23:56.288007  6647 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:56.288228  6815 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:56.288492  6647 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:56.289113  6815 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.291138 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1082) took 1822 ms. Trace:
I20260709 12:23:56.292033  6823 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:56.291654 32378 rpcz_store.cc:276] 0709 12:23:54.468850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.469230 (+   380us) service_pool.cc:224] Handling call
0709 12:23:54.477784 (+  8554us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:55.515645 (+1037861us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.290836 (+775191us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b540055da54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:56.291075 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2384497920}
I20260709 12:23:56.260572  6960 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.272886  6821 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.272598  6904 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.239559 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.323375 31751 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:56.294677 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.295708 31717 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:56.254412 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:56.308796 31745 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:23:56.250205  6930 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:23:56.319403 31746 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:56.320732  6657 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.256523  6840 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:56.256392  6794 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:23:56.292716  6823 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.270256  6783 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.078s
I20260709 12:23:56.251448  6489 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.291194  6862 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.052s
I20260709 12:23:56.358814 31677 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: []
I20260709 12:23:56.353025 31752 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:56.354434 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.362910 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:56.338487 31751 pending_rounds.cc:85] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 11
I20260709 12:23:56.370617 31751 pending_rounds.cc:107] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:23:56.371119 31751 raft_consensus.cc:2889] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:56.356312  6661 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:56.372995  6752 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:56.374152 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:56.335405  6840 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:56.375368 32370 raft_consensus.cc:2410] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 24 because replica has last-logged OpId of term: 23 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:23:56.343258  6904 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:56.340497  6858 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:56.387130 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:56.267098  6929 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.341512  6909 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:56.398380 31747 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:56.400506  6885 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:23:56.403827 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:56.343000  6901 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.339848  6841 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:56.341100  6501 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:56.408754 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:56.439805 31674 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:23:56.410336  6661 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.410817  6815 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:56.410948 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:56.411664 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.412617  6823 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:56.321997  6825 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.415580  6909 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:56.415498 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:56.374142  6821 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.402020  6876 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.352252  6882 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:56.439035 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.335505  6657 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.460151 31746 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:23:56.462023 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:56.465369 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61668.3us
I20260709 12:23:56.470495 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.472462 32377 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: [19.13-19.13]
I20260709 12:23:56.454617  6489 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:56.494426 31752 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:56.495049 31747 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:56.441671  6783 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:23:56.464298  6862 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:56.457015  6909 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:56.464382  6842 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:56.507087 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:56.507648 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.500023 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:23:56.513665 31749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.009s	sys 0.011s
I20260709 12:23:56.468080  6876 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.548610  6862 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:56.628540 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:56.560563 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 98639.6us
I20260709 12:23:56.546931 31746 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:56.524842  6909 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:56.452271  6901 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.521101  6858 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:56.456945  6841 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.592672 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:56.651309 32377 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:56.499542  6661 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.654755  6905 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:23:56.655249  6905 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:56.655668  6905 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:23:56.656412  6661 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:56.506170  6924 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.524s	user 0.013s	sys 0.027s
I20260709 12:23:56.636485  6455 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.640793 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:56.643514 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:56.590939  6735 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:56.511421  6882 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.688941 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:56.624132 31749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.009s	sys 0.011s
I20260709 12:23:56.638813  6683 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:56.582600  6501 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.061s
I20260709 12:23:56.653565  6821 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:56.683967  6924 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.702s	user 0.014s	sys 0.027s
I20260709 12:23:56.684696  6489 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.685091  6743 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:23:56.697623 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:56.700595 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64673.9us
W20260709 12:23:56.696200 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 239419us
I20260709 12:23:56.692688  6455 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:56.701973  6967 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.148s	user 0.000s	sys 0.001s
I20260709 12:23:56.690388  6878 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:56.700186 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:56.709573 31746 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:56.703431 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:56.687208  6881 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.018s	sys 0.032s
I20260709 12:23:56.653664  6647 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.707669 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:56.690924  6661 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:56.692243  6828 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.014s	sys 0.032s
I20260709 12:23:56.690477  6768 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:56.718888  6896 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:56.721271  6647 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:56.712929  6881 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.837s	user 0.019s	sys 0.033s
W20260709 12:23:56.727447  6968 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:23:56.651031  6794 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.706992  6683 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.715739  6933 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.091s
I20260709 12:23:56.716480 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3340) took 2214 ms. Trace:
I20260709 12:23:56.716261  6661 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:56.717098  6828 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.014s	sys 0.032s
I20260709 12:23:56.691062 31752 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:23:56.759881  6896 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:56.750773  6647 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:56.744304  6929 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:56.744913  6657 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:56.733664  6876 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:56.738473 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1290) took 1244 ms. Trace:
I20260709 12:23:56.732117  6862 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.746304  6455 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:56.737144 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:56.736672  6878 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: INVALID_TERM, Last received: 14.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:23:56.743773 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3274) took 1244 ms. Trace:
W20260709 12:23:56.741547  6656 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.518s	user 0.012s	sys 0.000s
I20260709 12:23:56.737744  6841 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:56.739365  6825 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:56.743229  6901 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:56.735275 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:56.766079  6489 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.756589 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3262) took 2014 ms. Trace:
I20260709 12:23:56.752866  6889 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.800590 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:56.748008  6882 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:56.710814  6877 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.546s	user 0.011s	sys 0.003s
I20260709 12:23:56.835620 32371 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:23:56.803608 31750 rpcz_store.cc:276] 0709 12:23:54.741980 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.765674 (+ 23694us) service_pool.cc:224] Handling call
0709 12:23:56.123325 (+1357651us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:56.124338 (+  1013us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.756198 (+631860us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b5400295ed0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:56.756458 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3121268480}
I20260709 12:23:56.776948 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3269) took 1420 ms. Trace:
I20260709 12:23:56.782085 31823 rpcz_store.cc:276] 0709 12:23:55.493884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.650968 (+157084us) service_pool.cc:224] Handling call
0709 12:23:55.769400 (+118432us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:55.806098 (+ 36698us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.738158 (+932060us) spinlock_profiling.cc:243] Waited 114 ms on lock 0x7b540043eed0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:56.738386 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":263159552}
W20260709 12:23:56.775033  6934 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.925s	user 0.005s	sys 0.019s
I20260709 12:23:56.785316 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:56.837126 31744 rpcz_store.cc:276] 0709 12:23:55.356262 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.644036 (+287774us) service_pool.cc:224] Handling call
0709 12:23:55.644587 (+   551us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:55.738378 (+ 93791us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:55.738430 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:55.758313 (+ 19883us) log.cc:844] Serialized 26 byte log entry
0709 12:23:56.203415 (+445102us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:56.203490 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.203614 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:56.776719 (+573105us) raft_consensus.cc:1685] finished
0709 12:23:56.776746 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:56.776890 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39822,"raft.queue_time_us":40040,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":22777,"thread_start_us":398135,"threads_started":1,"wal-append.queue_time_us":684021}
I20260709 12:23:56.845947 31674 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:56.788393 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:56.841091  6970 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:23:56.755046  6736 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:56.803085 31672 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:56.766430 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:23:56.801316 31753 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 13. (term mismatch)
I20260709 12:23:56.803859 31747 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
W20260709 12:23:56.799819 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 481127us
I20260709 12:23:56.801949 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:56.804834 31819 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.014s	sys 0.020s
I20260709 12:23:56.789076 31749 rpcz_store.cc:276] 0709 12:23:55.499132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.930590 (+431458us) service_pool.cc:224] Handling call
0709 12:23:55.931069 (+   479us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:55.931554 (+   485us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:55.931587 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:55.932281 (+   694us) log.cc:844] Serialized 52 byte log entry
0709 12:23:56.689928 (+757647us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:23:56.689967 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.690030 (+    63us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:56.743515 (+ 53485us) raft_consensus.cc:1685] finished
0709 12:23:56.743537 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:56.743688 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1569,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":3096,"thread_start_us":757199,"threads_started":1,"wal-append.queue_time_us":794860}
I20260709 12:23:56.841632 31677 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:56.846602 31820 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:56.771214 31670 rpcz_store.cc:276] 0709 12:23:54.436555 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.436692 (+   137us) service_pool.cc:224] Handling call
0709 12:23:54.437095 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:54.439173 (+  2078us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:23:54.855376 (+416203us) spinlock_profiling.cc:243] Waited 4.58 ms on lock 0x7b1800499a20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1
0709 12:23:55.714947 (+859571us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b1800499a20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:23:55.793986 (+ 79039us) log.cc:844] Serialized 14 byte log entry
0709 12:23:56.322338 (+528352us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:56.406055 (+ 83717us) log.cc:844] Serialized 26 byte log entry
0709 12:23:56.406226 (+   171us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:56.406264 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.418110 (+ 11846us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:56.647831 (+229721us) raft_consensus.cc:1685] finished
0709 12:23:56.647871 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:56.650848 (+  2977us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":79477,"prepare.queue_time_us":72667,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":49397,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":7092,"spinlock_wait_cycles":47475072,"thread_start_us":466081,"threads_started":2,"wal-append.queue_time_us":593958}
W20260709 12:23:56.860559 31819 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.014s	sys 0.021s
I20260709 12:23:56.856653 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:56.849499 31746 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:23:56.862845 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:56.861814 32376 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:56.857527 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:56.859023 31753 pending_rounds.cc:85] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 12
I20260709 12:23:56.868364 31753 pending_rounds.cc:107] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 20.13
I20260709 12:23:56.868724 31753 raft_consensus.cc:2889] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:56.870483 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.871356 31746 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:56.872133 32373 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:23:56.849246  6923 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.345s	user 0.014s	sys 0.007s
I20260709 12:23:56.868921 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:56.873236 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:23:56.845655  6934 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.006s	user 0.005s	sys 0.020s
I20260709 12:23:56.800735  6683 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:56.875475 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:56.875764 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.849291  6814 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:56.791246  6840 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.878685 31747 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:56.877067 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:56.867312 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.836673  6930 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:56.879699 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:56.881804 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:56.799006  6647 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:56.884231 31750 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:23:56.847430  6901 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:56.882203 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.881989 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:56.888481 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:56.876164  6923 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.523s	user 0.016s	sys 0.008s
I20260709 12:23:56.888473 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:56.890977 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.893893 31752 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:23:56.889385 31820 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:23:56.901978 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:56.887576  6896 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:56.846972  6877 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.925326 31744 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
W20260709 12:23:56.947083 31745 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.544s	user 0.009s	sys 0.041s
I20260709 12:23:56.945528 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:56.949002 31745 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.559s	user 0.009s	sys 0.041s
I20260709 12:23:56.867549  6878 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.051s
I20260709 12:23:56.985706 31673 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:23:56.989815 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:56.992637 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.999217 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:57.000780 31747 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 26 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
W20260709 12:23:56.952445 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 239529us
I20260709 12:23:56.954272 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:56.952822 31749 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:23:56.926834 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:57.012269 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:57.076881 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:57.014246 32376 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
W20260709 12:23:57.015591  6975 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:23:56.924948  6882 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:56.891642  6783 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:23:57.015770  6930 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 17.11, Last appended by leader: 11, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:57.016562  6889 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:57.102576 32375 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:23:57.017248 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:56.942066  6876 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:23:57.019313 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:57.022260  6840 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:57.027669  6974 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.174s	user 0.000s	sys 0.001s
W20260709 12:23:57.047559  6978 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 12:23:57.049103 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3350) took 1806 ms. Trace:
I20260709 12:23:57.051436  6489 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:56.918081  6814 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:57.059947  6661 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:57.061784  6958 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.922s	user 0.000s	sys 0.003s
I20260709 12:23:57.064039  6682 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.064129 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:57.067585 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:57.071517 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:57.072419  6752 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:23:56.919996  6901 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:57.073530  6647 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:57.075863  6861 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.132570  6861 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:57.135066  6861 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:56.885084  6455 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.104s
I20260709 12:23:57.097772 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:57.136323  6455 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:23:57.136781  6455 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:57.100814 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.138020  6455 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:56.951025  6896 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:57.141216  6896 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:57.103498 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:57.055927  6905 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:57.116745 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:57.117261 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:57.196539 31747 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:23:57.097044  6882 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:57.096385  6783 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:57.120471 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:57.199548 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 79167.3us
I20260709 12:23:57.120611 31752 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:57.120640 31823 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:57.108414 31671 rpcz_store.cc:276] 0709 12:23:55.293517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.950531 (+657014us) service_pool.cc:224] Handling call
0709 12:23:55.951037 (+   506us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:55.951748 (+   711us) raft_consensus.cc:1534] Early marking committed up to index 7
0709 12:23:55.951795 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:55.952466 (+   671us) log.cc:844] Serialized 26 byte log entry
0709 12:23:56.250914 (+298448us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:56.250959 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.251044 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.048560 (+797516us) raft_consensus.cc:1685] finished
0709 12:23:57.048598 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.048855 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2049,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":24605,"thread_start_us":297988,"threads_started":1,"wal-append.queue_time_us":545876}
I20260709 12:23:57.072649 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:57.129235  6682 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:57.138444 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.100715 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.138845  6489 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:57.143586  6905 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:57.205343 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.216521 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.217227 31675 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:23:57.143975 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1033) took 1198 ms. Trace:
I20260709 12:23:57.144423 32376 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:23:57.145180  6743 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:23:57.218073 32372 rpcz_store.cc:276] 0709 12:23:55.945456 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.034653 (+ 89197us) service_pool.cc:224] Handling call
0709 12:23:56.045326 (+ 10673us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.045787 (+   461us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.045815 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:56.098848 (+ 53033us) log.cc:844] Serialized 26 byte log entry
0709 12:23:56.672261 (+573413us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:56.674377 (+  2116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.674875 (+   498us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.143541 (+468666us) raft_consensus.cc:1685] finished
0709 12:23:57.143571 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.143752 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1538,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":30341,"thread_start_us":453320,"threads_started":1,"wal-append.queue_time_us":683568}
I20260709 12:23:57.145356  6455 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:57.220273 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.135084 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2592) took 2332 ms. Trace:
I20260709 12:23:57.152729  6901 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:57.118878  6876 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:57.223192  6876 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:57.224004  6876 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:57.222851 31748 rpcz_store.cc:276] 0709 12:23:54.802859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.803024 (+   165us) service_pool.cc:224] Handling call
0709 12:23:54.803393 (+   369us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:55.608815 (+805422us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:55.608873 (+    58us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:55.609839 (+   966us) log.cc:844] Serialized 26 byte log entry
0709 12:23:55.799561 (+189722us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:23:55.799605 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.805528 (+  5923us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.061129 (+1255601us) spinlock_profiling.cc:243] Waited 42.5 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:57.134712 (+ 73583us) raft_consensus.cc:1685] finished
0709 12:23:57.134736 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.134987 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":33486,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":34410,"spinlock_wait_cycles":97792384,"thread_start_us":986183,"threads_started":2,"wal-append.queue_time_us":424974}
I20260709 12:23:57.136986 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:57.154044 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1299) took 1141 ms. Trace:
I20260709 12:23:57.226891 31826 rpcz_store.cc:276] 0709 12:23:56.012676 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.012829 (+   153us) service_pool.cc:224] Handling call
0709 12:23:56.020200 (+  7371us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.040708 (+ 20508us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.040758 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:56.092091 (+ 51333us) log.cc:844] Serialized 26 byte log entry
0709 12:23:56.507360 (+415269us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:56.507420 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.507535 (+   115us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.153788 (+646253us) raft_consensus.cc:1685] finished
0709 12:23:57.153817 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.153974 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9648,"raft.queue_time_us":16664,"raft.run_cpu_time_us":173,"raft.run_wall_time_us":33248,"thread_start_us":388236,"threads_started":1,"wal-append.queue_time_us":663453}
I20260709 12:23:57.228220 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3276) took 1390 ms. Trace:
I20260709 12:23:57.228658 31744 rpcz_store.cc:276] 0709 12:23:55.837965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.227355 (+1389390us) service_pool.cc:224] Handling call
0709 12:23:57.227673 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.227880 (+   207us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.227903 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:57.227947 (+    44us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:57.227984 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.228040 (+    56us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.228138 (+    98us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.156430 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:57.231128 31822 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:57.160782 31677 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:23:57.233021 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:57.158989  6814 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:57.106087  6881 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 17.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260709 12:23:57.234937 31748 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: []
I20260709 12:23:57.235467  6881 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:23:57.235900 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1075) took 1564 ms. Trace:
I20260709 12:23:57.163065 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3190) took 1035 ms. Trace:
W20260709 12:23:57.182329 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.183845 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:57.237607 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53893.9us
I20260709 12:23:57.236768 31821 rpcz_store.cc:276] 0709 12:23:56.127365 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.128433 (+  1068us) service_pool.cc:224] Handling call
0709 12:23:56.128794 (+   361us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.129324 (+   530us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.129356 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:56.129953 (+   597us) log.cc:844] Serialized 26 byte log entry
0709 12:23:56.626179 (+496226us) raft_consensus.cc:1645] Marking committed up to 7
0709 12:23:56.626245 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:56.626344 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.162723 (+536379us) raft_consensus.cc:1685] finished
0709 12:23:57.162758 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.162971 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1704,"raft.run_cpu_time_us":100,"raft.run_wall_time_us":101,"thread_start_us":495662,"threads_started":1,"wal-append.queue_time_us":732914}
I20260709 12:23:57.184986 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2588) took 2496 ms. Trace:
I20260709 12:23:57.239054 31753 rpcz_store.cc:276] 0709 12:23:54.688843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.692731 (+  3888us) service_pool.cc:224] Handling call
0709 12:23:54.693022 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:56.995196 (+2302174us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.184672 (+189476us) spinlock_profiling.cc:243] Waited 2.11 s on lock 0x7b54000b95d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:57.184911 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4847191296}
I20260709 12:23:57.241520 31753 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:23:57.188364  6651 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:57.171118  6877 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:57.205330 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3193) took 1077 ms. Trace:
I20260709 12:23:57.207526 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.243737 31824 rpcz_store.cc:276] 0709 12:23:56.128268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.139902 (+ 11634us) service_pool.cc:224] Handling call
0709 12:23:56.140358 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:56.248329 (+107971us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.205044 (+956715us) spinlock_profiling.cc:243] Waited 94.4 ms on lock 0x7b54004461d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:57.205263 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":217173376}
W20260709 12:23:57.198659 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 81999.3us
I20260709 12:23:57.242381 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1079) took 1291 ms. Trace:
I20260709 12:23:57.235985  6881 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:57.246487 31753 rpcz_store.cc:276] 0709 12:23:55.951472 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.240480 (+1289008us) service_pool.cc:224] Handling call
0709 12:23:57.241017 (+   537us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:57.242037 (+  1020us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.242307 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.247577  6881 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:57.127636  6752 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:57.238505  6735 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
W20260709 12:23:57.198544  6909 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.539s	user 0.010s	sys 0.004s
I20260709 12:23:57.262851 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:57.234417 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.264192 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:57.222347 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.251744 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:57.253394 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:57.244591 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.236690 31748 rpcz_store.cc:276] 0709 12:23:55.671701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.226131 (+1554430us) service_pool.cc:224] Handling call
0709 12:23:57.226612 (+   481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.235599 (+  8987us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.235824 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.200380  6783 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:23:57.199311  6882 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:57.260993 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:57.258371 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3277) took 1309 ms. Trace:
I20260709 12:23:57.323793 31669 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:57.323688 31744 rpcz_store.cc:276] 0709 12:23:55.941121 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.229958 (+1288837us) service_pool.cc:224] Handling call
0709 12:23:57.230351 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.230592 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.250172 (+ 19580us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:57.250244 (+    72us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:57.250300 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.250394 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.250542 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.200237  6861 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:23:57.268218 31747 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 15.
I20260709 12:23:57.263540 31752 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:23:57.313097  6489 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:57.314885 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:57.317091 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52970.4us
W20260709 12:23:57.317457 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 244866us
I20260709 12:23:57.317760  6923 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:23:57.317930  6651 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:57.318989 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:57.319787  6969 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:23:57.320662  6752 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:57.249370  6881 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:23:57.321403  6876 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:57.206377  6896 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:57.325609 31671 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:57.347909 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.327968 31748 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:23:57.331664 31823 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:57.328510  6651 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:57.349622 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2431) took 1067 ms. Trace:
I20260709 12:23:57.349861 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2422) took 1070 ms. Trace:
I20260709 12:23:57.350145 31671 rpcz_store.cc:276] 0709 12:23:56.282027 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.251383 (+969356us) service_pool.cc:224] Handling call
0709 12:23:57.349543 (+ 98160us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.346511 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:57.344856 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 268045us
W20260709 12:23:57.352067 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.352593 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3279) took 1392 ms. Trace:
I20260709 12:23:57.352990 31748 rpcz_store.cc:276] 0709 12:23:55.959740 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.268603 (+1308863us) service_pool.cc:224] Handling call
0709 12:23:57.322467 (+ 53864us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.348511 (+ 26044us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.352536 (+  4025us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.350301 31675 rpcz_store.cc:276] 0709 12:23:56.279681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.343385 (+1063704us) service_pool.cc:224] Handling call
0709 12:23:57.349797 (+  6412us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.351774 31673 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:23:57.356153 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2424) took 1075 ms. Trace:
I20260709 12:23:57.358582 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.359406 31671 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 14.
I20260709 12:23:57.360553 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.361392 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2423) took 1081 ms. Trace:
I20260709 12:23:57.361734 31671 rpcz_store.cc:276] 0709 12:23:56.280029 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.360261 (+1080232us) service_pool.cc:224] Handling call
0709 12:23:57.361331 (+  1070us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:57.363293  6995 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.186s	user 0.000s	sys 0.000s
I20260709 12:23:57.356567 31673 rpcz_store.cc:276] 0709 12:23:56.280300 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.334951 (+1054651us) service_pool.cc:224] Handling call
0709 12:23:57.356073 (+ 21122us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.371785 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.375559 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:57.376380 31823 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:57.325249  6909 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:57.379076 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1159) took 1190 ms. Trace:
I20260709 12:23:57.379967 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2432) took 1097 ms. Trace:
I20260709 12:23:57.382678 31675 rpcz_store.cc:276] 0709 12:23:56.282270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.355192 (+1072922us) service_pool.cc:224] Handling call
0709 12:23:57.379908 (+ 24716us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.384037 31675 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:23:57.384845 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2426) took 1104 ms. Trace:
I20260709 12:23:57.385248 31675 rpcz_store.cc:276] 0709 12:23:56.280771 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.383364 (+1102593us) service_pool.cc:224] Handling call
0709 12:23:57.383712 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:57.384522 (+   810us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.384777 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.349176  6896 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:57.388708  6643 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:23:57.389045  6896 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:57.389235  6643 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:57.390357  6643 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:57.391178  6989 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.257s	user 0.001s	sys 0.000s
I20260709 12:23:57.391502  6643 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:57.393946  6960 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:23:57.394385  6960 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:57.395344  6960 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:57.396744  6960 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:57.352326  6651 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:57.381831 31672 rpcz_store.cc:276] 0709 12:23:56.188561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.188671 (+   110us) service_pool.cc:224] Handling call
0709 12:23:56.802913 (+614242us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:56.857306 (+ 54393us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.378800 (+521494us) spinlock_profiling.cc:243] Waited 608 ms on lock 0x7b54002e7a50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:57.378996 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1397881600}
W20260709 12:23:57.388093  6998 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
W20260709 12:23:57.399605 32371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 1016) took 3161 ms (client timeout 1604 ms). Trace:
I20260709 12:23:57.399907  6489 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:57.371573  6941 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 12:23:57.406556  6947 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:57.416181  6647 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:57.416692 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:57.407171 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1291) took 1894 ms. Trace:
I20260709 12:23:57.416786  6647 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:57.417273 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 89267.2us
I20260709 12:23:57.417431 31825 rpcz_store.cc:276] 0709 12:23:55.513012 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.814092 (+301080us) service_pool.cc:224] Handling call
0709 12:23:55.907802 (+ 93710us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:55.938785 (+ 30983us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.406897 (+1468112us) spinlock_profiling.cc:243] Waited 59.2 ms on lock 0x7b540043cbd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:57.407087 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":136240896}
I20260709 12:23:57.418144 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:57.409591  6992 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.274s	user 0.000s	sys 0.001s
I20260709 12:23:57.411019 31746 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:23:57.412289  6909 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:57.356683  6752 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:57.412637 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3361) took 1068 ms. Trace:
I20260709 12:23:57.420578  6657 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:57.421657  6982 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.279s	user 0.000s	sys 0.001s
W20260709 12:23:57.413084 32371 rpcz_store.cc:269] 0709 12:23:54.237699 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:54.237974 (+   275us) service_pool.cc:224] Handling call
0709 12:23:57.399347 (+3161373us) spinlock_profiling.cc:243] Waited 2.28 s on lock 0x7b54004c4354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:57.399500 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5247057152}
I20260709 12:23:57.422567  6647 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:57.422956 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:57.424240  6923 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:57.424717 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:57.402302 31747 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 12:23:57.427467  6889 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:23:57.428519  6794 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:57.429080  6876 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:57.431700 32370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 1035) took 1485 ms (client timeout 1743 ms). Trace:
I20260709 12:23:57.433780  6858 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:23:57.436065 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:57.438609 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3275) took 1838 ms. Trace:
I20260709 12:23:57.442411 31574 catalog_manager.cc:5697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 14 to 19, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 19 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:23:57.447389  6941 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.448788 31678 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 17 index: 10. (index mismatch)
I20260709 12:23:57.456280 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:57.457415 31822 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:23:57.465227 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:57.466053 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:57.592458 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.592578 31825 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:23:57.593043 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:57.590893 31751 rpcz_store.cc:276] 0709 12:23:55.580854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.326467 (+745613us) service_pool.cc:224] Handling call
0709 12:23:56.327006 (+   539us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:23:56.877818 (+550812us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.877867 (+    49us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:23:56.942017 (+ 64150us) log.cc:844] Serialized 78 byte log entry
0709 12:23:57.217665 (+275648us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:57.262575 (+ 44910us) log.cc:844] Serialized 14 byte log entry
0709 12:23:57.413266 (+150691us) spinlock_profiling.cc:243] Waited 6.39 ms on lock 0x7b1800363760. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:23:57.413748 (+   482us) log.cc:844] Serialized 14 byte log entry
0709 12:23:57.413958 (+   210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.414039 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.414175 (+   136us) raft_consensus.cc:1685] finished
0709 12:23:57.414190 (+    15us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.414340 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":76303,"prepare.queue_time_us":57575,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":2718,"raft.queue_time_us":168267,"raft.run_cpu_time_us":245,"raft.run_wall_time_us":245,"spinlock_wait_cycles":14706432,"thread_start_us":652787,"threads_started":2,"wal-append.queue_time_us":182494}
I20260709 12:23:57.591297 31672 rpcz_store.cc:276] 0709 12:23:56.343942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.411612 (+1067670us) service_pool.cc:224] Handling call
0709 12:23:57.411996 (+   384us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.412240 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.412267 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:57.412313 (+    46us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:23:57.412370 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.412435 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.412546 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:57.472122 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.461995 31670 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 20 index: 10. (index mismatch)
I20260709 12:23:57.598451 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3282) took 1630 ms. Trace:
I20260709 12:23:57.599335 31747 rpcz_store.cc:276] 0709 12:23:55.961059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.401348 (+1440289us) service_pool.cc:224] Handling call
0709 12:23:57.401806 (+   458us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.591088 (+189282us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.591359 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:23:57.386090 32342 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.545s (SENT) [suppressed 7 similar messages]
I20260709 12:23:57.591747 31823 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:23:57.491628 31669 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
W20260709 12:23:57.602249 31669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1164) took 1371 ms (client timeout 1545 ms). Trace:
W20260709 12:23:57.595640 32370 rpcz_store.cc:269] 0709 12:23:55.947167 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.963284 (+ 16117us) service_pool.cc:224] Handling call
0709 12:23:57.431425 (+1468141us) spinlock_profiling.cc:243] Waited 411 ms on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:57.431600 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":944881408}
I20260709 12:23:57.612046 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2437) took 1249 ms. Trace:
I20260709 12:23:57.613938 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:23:57.602758 31669 rpcz_store.cc:269] 0709 12:23:56.230581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.249964 (+1019383us) service_pool.cc:224] Handling call
0709 12:23:57.602185 (+352221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.612847 31678 rpcz_store.cc:276] 0709 12:23:56.345056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.448001 (+1102945us) service_pool.cc:224] Handling call
0709 12:23:57.448406 (+   405us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.592019 (+143613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.594324 (+  2305us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.623137 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2438) took 1254 ms. Trace:
I20260709 12:23:57.623638 31670 rpcz_store.cc:276] 0709 12:23:56.345325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.461008 (+1115683us) service_pool.cc:224] Handling call
0709 12:23:57.461497 (+   489us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.597207 (+135710us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.599450 (+  2243us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.625460  6706 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:23:57.625626 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 160506us
I20260709 12:23:57.626199  6706 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:57.626489 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:57.627266  6706 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:57.628253  6909 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:23:57.628715  6909 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:57.372385 31671 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:23:57.629067  6909 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:57.630862  6643 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:57.631385 31672 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:23:57.631479 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:57.632097  6923 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:57.633044 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:57.636015  6933 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:23:57.630359 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:57.459548  6963 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.341s	user 0.003s	sys 0.000s
I20260709 12:23:57.637470 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:57.637832 31670 raft_consensus.cc:2410] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:23:57.637717 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.630359 31671 raft_consensus.cc:606] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Received request to transfer leadership to TS 039d66a6c85e43c3936bf5017501b344
W20260709 12:23:57.469224  6985 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.334s	user 0.002s	sys 0.012s
I20260709 12:23:57.659274 31675 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 13. (term mismatch)
I20260709 12:23:57.659714 31675 pending_rounds.cc:85] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 12
I20260709 12:23:57.659943 31675 pending_rounds.cc:107] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 19.13
I20260709 12:23:57.660190 31675 raft_consensus.cc:2889] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:57.632151 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1169) took 1225 ms. Trace:
I20260709 12:23:57.660652 31672 rpcz_store.cc:276] 0709 12:23:56.407135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.610843 (+1203708us) service_pool.cc:224] Handling call
0709 12:23:57.631024 (+ 20181us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.631840 (+   816us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.632055 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.661438 31822 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:23:57.657320 31671 raft_consensus.cc:614] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:23:57.593779  6990 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.654120 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:57.664479 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:57.662537 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:57.665217 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:57.666002 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:57.671253 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:23:57.666844 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 427207us
I20260709 12:23:57.458464  6896 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:57.676564 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.638978  6862 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:23:57.676689 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.680832 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:57.681986 31822 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:57.645143  6647 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:23:57.674175 31824 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:57.677328 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:40834 (request call id 24) took 1297 ms. Trace:
I20260709 12:23:57.692018 31671 rpcz_store.cc:276] 0709 12:23:56.378762 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.362467 (+983705us) service_pool.cc:224] Handling call
0709 12:23:57.662615 (+300148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":62464}
I20260709 12:23:57.683265 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.683182 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:57.663322  6859 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Stepping down as leader of term 14
I20260709 12:23:57.598459  6657 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:57.719458 31678 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:57.598554  6941 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:57.664664  6501 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:57.734673  6501 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:57.735164  6501 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:57.736482  6896 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:57.663767  6828 raft_consensus.cc:2764] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:23:57.719121 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:57.721263  6859 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:57.740124 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:57.740798 31822 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:23:57.740948  6859 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 14, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:57.723347 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:57.741665 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:57.743680 31822 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:23:57.733011 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3364) took 1183 ms. Trace:
I20260709 12:23:57.744432 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:57.744679 31678 rpcz_store.cc:276] 0709 12:23:56.549701 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.680044 (+1130343us) service_pool.cc:224] Handling call
0709 12:23:57.719274 (+ 39230us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.732503 (+ 13229us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.732880 (+   377us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.745272 31749 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:23:57.743053 31824 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:23:57.749018 31669 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:23:57.691339  6647 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:57.751122 31824 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260709 12:23:57.751264 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.752630 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:57.753805 31751 raft_consensus.cc:2410] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 21 because replica has last-logged OpId of term: 19 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
W20260709 12:23:57.755693 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 369319us
W20260709 12:23:57.742516  6985 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:23:57.757135 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3335) took 1448 ms. Trace:
I20260709 12:23:57.668764  6990 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:57.762265 31824 rpcz_store.cc:276] 0709 12:23:56.308765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.750459 (+1441694us) service_pool.cc:224] Handling call
0709 12:23:57.750792 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.751535 (+   743us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.757063 (+  5528us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.763634 31822 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
W20260709 12:23:57.756918 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.767558 31749 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:23:57.769467 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:57.805191 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 382280us
I20260709 12:23:57.807801 32371 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: [21.13-21.13]
I20260709 12:23:57.807888 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3333) took 1476 ms. Trace:
I20260709 12:23:57.785684 31824 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: []
I20260709 12:23:57.808481 31822 rpcz_store.cc:276] 0709 12:23:56.290217 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.749297 (+1459080us) service_pool.cc:224] Handling call
0709 12:23:57.763368 (+ 14071us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.766563 (+  3195us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.766788 (+   225us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.769515 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2595) took 1428 ms. Trace:
I20260709 12:23:57.813515 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:57.818667 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.823562 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:57.823948 32377 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:23:57.814373 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3336) took 1481 ms. Trace:
I20260709 12:23:57.827327 31824 rpcz_store.cc:276] 0709 12:23:56.333342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.765958 (+1432616us) service_pool.cc:224] Handling call
0709 12:23:57.766264 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.814091 (+ 47827us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.814315 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.834100 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:57.806488  6794 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:57.835692 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 201806us
I20260709 12:23:57.836935 31824 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 18 index: 12. (index mismatch)
I20260709 12:23:57.764796  6859 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 14 FOLLOWER]: Advancing to term 15
I20260709 12:23:57.816800 31749 rpcz_store.cc:276] 0709 12:23:56.340919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.747250 (+1406331us) service_pool.cc:224] Handling call
0709 12:23:57.767295 (+ 20045us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.769157 (+  1862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.769435 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.837806 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3339) took 1388 ms. Trace:
I20260709 12:23:57.804770  6828 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:57.839726 31824 rpcz_store.cc:276] 0709 12:23:56.449647 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.833105 (+1383458us) service_pool.cc:224] Handling call
0709 12:23:57.833461 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.837452 (+  3991us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.837722 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.841598 31671 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:23:57.837663 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 153458us
W20260709 12:23:57.832695  6948 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.928s	user 0.013s	sys 0.018s
I20260709 12:23:57.755793  6882 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:57.842494 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3340) took 1374 ms. Trace:
W20260709 12:23:57.851408 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.839216  6929 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:23:57.852466 31824 rpcz_store.cc:276] 0709 12:23:56.468356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.841428 (+1373072us) service_pool.cc:224] Handling call
0709 12:23:57.841854 (+   426us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.842086 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.842124 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:57.842183 (+    59us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:57.842242 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.842310 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.842431 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.746567  6651 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:57.859093 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.859992 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.864549 31819 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:23:57.864876 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:57.870432 31824 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 14. Current term is 15. Ops: []
I20260709 12:23:57.872136  6896 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:23:57.874454 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:57.879405  7007 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
I20260709 12:23:57.880008 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:57.848402  6794 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:57.848133  6647 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:57.881822 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:57.883173  7006 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:57.884641 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2421) took 1628 ms. Trace:
I20260709 12:23:57.896708 32377 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:23:57.896907 32373 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:23:57.860177  6929 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:57.920099 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1303) took 1326 ms. Trace:
I20260709 12:23:57.937139 32376 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 17. (term mismatch)
I20260709 12:23:57.937847 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3280) took 1977 ms. Trace:
I20260709 12:23:57.933348 31673 rpcz_store.cc:276] 0709 12:23:56.256431 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.372237 (+1115806us) service_pool.cc:224] Handling call
0709 12:23:57.382136 (+  9899us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:57.382342 (+   206us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.382366 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:57.435209 (+ 52843us) log.cc:844] Serialized 26 byte log entry
0709 12:23:57.715424 (+280215us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:57.715465 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.715541 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.884318 (+168777us) raft_consensus.cc:1685] finished
0709 12:23:57.884350 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.884557 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":279773,"threads_started":1,"wal-append.queue_time_us":373455}
I20260709 12:23:57.938879 31824 rpcz_store.cc:276] 0709 12:23:56.594240 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.866520 (+1272280us) service_pool.cc:224] Handling call
0709 12:23:57.868265 (+  1745us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.918998 (+ 50733us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.920038 (+  1040us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.941205 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:23:57.952868 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 78492us
I20260709 12:23:57.953644 32369 raft_consensus.cc:2410] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 25 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:23:57.850989  6828 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:57.939087 32376 pending_rounds.cc:85] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 16
I20260709 12:23:57.957844 32376 pending_rounds.cc:107] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 21.17
I20260709 12:23:57.958132 32376 raft_consensus.cc:2889] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:57.959336 31824 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 11. Preceding OpId from leader: term: 20 index: 13. (index mismatch)
I20260709 12:23:57.959565 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2606) took 1591 ms. Trace:
I20260709 12:23:57.959921 31750 rpcz_store.cc:276] 0709 12:23:56.367836 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.902619 (+1534783us) service_pool.cc:224] Handling call
0709 12:23:57.938074 (+ 35455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.959169 (+ 21095us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.959199 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:57.959241 (+    42us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:57.959311 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.959374 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.959509 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20624}
I20260709 12:23:57.961715 31750 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 17. Preceding OpId from leader: term: 22 index: 18. (index mismatch)
I20260709 12:23:57.964416 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:23:57.962385 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1080) took 1579 ms. Trace:
I20260709 12:23:57.966063 31750 rpcz_store.cc:276] 0709 12:23:56.382777 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.960931 (+1578154us) service_pool.cc:224] Handling call
0709 12:23:57.961306 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.962077 (+   771us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.962320 (+   243us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.969266 31750 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:57.971145 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2610) took 1506 ms. Trace:
I20260709 12:23:57.972098 31750 rpcz_store.cc:276] 0709 12:23:56.468689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.967944 (+1499255us) service_pool.cc:224] Handling call
0709 12:23:57.968801 (+   857us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.970371 (+  1570us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.971009 (+   638us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:57.976828 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3284) took 1421 ms. Trace:
I20260709 12:23:57.855832  6735 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.082s
I20260709 12:23:57.981521 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:57.982064 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:57.983407 31669 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:23:57.978235  7000 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:57.957321  6960 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:57.985359 32372 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:23:57.995939 31751 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 10. Preceding OpId from leader: term: 20 index: 10. (term mismatch)
W20260709 12:23:58.033469 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:57.938741  6929 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:23:57.917598  7013 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:57.997700 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:58.033100 31673 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:23:57.999866 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:58.003644 31750 rpcz_store.cc:276] 0709 12:23:56.555942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.974490 (+1418548us) service_pool.cc:224] Handling call
0709 12:23:57.975484 (+   994us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.975957 (+   473us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.976029 (+    72us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:57.976153 (+   124us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:57.976234 (+    81us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.976371 (+   137us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.976695 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.005733 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3202) took 1120 ms. Trace:
W20260709 12:23:57.999193 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 332023us
I20260709 12:23:58.037088 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:58.034195 31751 pending_rounds.cc:85] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 9
I20260709 12:23:58.041713 31751 pending_rounds.cc:107] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 19.10
I20260709 12:23:58.042181 31751 raft_consensus.cc:2889] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:23:58.037252 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 203218us
I20260709 12:23:58.041720 31821 rpcz_store.cc:276] 0709 12:23:56.885613 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.900471 (+1014858us) service_pool.cc:224] Handling call
0709 12:23:57.900938 (+   467us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.004186 (+103248us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:58.004251 (+    65us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:58.004357 (+   106us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:23:58.004427 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.004577 (+   150us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.005486 (+   909us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":97611,"threads_started":1}
I20260709 12:23:58.041002 31672 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:58.046605 31672 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:58.043527 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:58.001926  6647 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:23:58.059031  6784 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.013s	sys 0.011s
I20260709 12:23:58.081982 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3287) took 1525 ms. Trace:
I20260709 12:23:58.097450 31750 rpcz_store.cc:276] 0709 12:23:56.556961 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.041954 (+1484993us) service_pool.cc:224] Handling call
0709 12:23:58.080831 (+ 38877us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.081452 (+   621us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:58.081525 (+    73us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:58.081621 (+    96us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:58.081662 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.081729 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.081898 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.035228 31678 raft_consensus.cc:2376] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23. Re-sending same reply.
I20260709 12:23:58.001678  6896 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.060s
W20260709 12:23:58.119206 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 163869us
I20260709 12:23:58.120857 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:58.111104  6784 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.014s	sys 0.012s
I20260709 12:23:58.117053 31672 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:58.131673 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3289) took 1573 ms. Trace:
I20260709 12:23:58.058212  6657 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:58.162222 31750 rpcz_store.cc:276] 0709 12:23:56.557884 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.099003 (+1541119us) service_pool.cc:224] Handling call
0709 12:23:58.131107 (+ 32104us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.131312 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:58.131336 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:58.131398 (+    62us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:58.131439 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.131499 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.131606 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.112262  6941 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:58.165995  6941 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:58.166561  6941 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.167002 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3208) took 1068 ms. Trace:
I20260709 12:23:58.085498  6933 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:58.161639 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3329) took 3079 ms. Trace:
I20260709 12:23:58.050969  6960 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:58.180186 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:58.186535 32376 raft_consensus.cc:2410] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 22 because replica has last-logged OpId of term: 20 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 14.
I20260709 12:23:58.170477 31824 rpcz_store.cc:276] 0709 12:23:57.099056 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.960160 (+861104us) service_pool.cc:224] Handling call
0709 12:23:57.960440 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.000368 (+ 39928us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:58.000399 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:58.000434 (+    35us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:58.000464 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.166555 (+166091us) spinlock_profiling.cc:243] Waited 39.7 ms on lock 0x7b54004475d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:58.166659 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.166907 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":91390208}
I20260709 12:23:58.054155  7013 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:58.189280 31750 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:23:58.112023  6651 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:58.192200 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2611) took 1416 ms. Trace:
I20260709 12:23:58.192766 31750 rpcz_store.cc:276] 0709 12:23:56.775792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.163884 (+1388092us) service_pool.cc:224] Handling call
0709 12:23:58.189023 (+ 25139us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:58.191825 (+  2802us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.192115 (+   290us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.183087 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:58.194756 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:58.196640 32376 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 17. (term mismatch)
I20260709 12:23:58.197163 32376 pending_rounds.cc:85] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 16
I20260709 12:23:58.197460 32376 pending_rounds.cc:107] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 23.17
I20260709 12:23:58.197753 32376 raft_consensus.cc:2889] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:58.202173 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3332) took 1912 ms. Trace:
I20260709 12:23:58.202826 31825 rpcz_store.cc:276] 0709 12:23:56.289925 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.418396 (+1128471us) service_pool.cc:224] Handling call
0709 12:23:57.592350 (+173954us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.594054 (+  1704us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.189511 (+595457us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b540030e8d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:58.202076 (+ 12565us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":391370112}
W20260709 12:23:58.227383 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.167026  6941 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:58.294770 31672 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:58.332683 32378 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 1098) took 2005 ms (client timeout 1933 ms). Trace:
W20260709 12:23:58.333261 32378 rpcz_store.cc:269] 0709 12:23:56.327667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.182795 (+1855128us) service_pool.cc:224] Handling call
0709 12:23:58.332587 (+149792us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.341522 31750 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:23:58.348133 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:58.351969 31750 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: []
I20260709 12:23:58.353276 32376 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: []
I20260709 12:23:58.177698 31818 rpcz_store.cc:276] 0709 12:23:54.788540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:55.207842 (+419302us) service_pool.cc:224] Handling call
0709 12:23:55.208773 (+   931us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:55.221144 (+ 12371us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:55.221182 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:55.222125 (+   943us) log.cc:844] Serialized 26 byte log entry
0709 12:23:55.388554 (+166429us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:55.388600 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:55.392842 (+  4242us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:57.624728 (+2231886us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:57.862814 (+238086us) raft_consensus.cc:1685] finished
0709 12:23:57.862842 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.867692 (+  4850us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":102303,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":273,"spinlock_wait_cycles":2626304,"thread_start_us":165868,"threads_started":1,"wal-append.queue_time_us":674585}
I20260709 12:23:58.376617 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:58.205400  6882 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:58.386363 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:58.401763 32372 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:23:58.410926  6735 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:58.411677 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:58.411707  6735 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:58.413319 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:58.418895 31672 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:23:58.448071 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3338) took 2042 ms. Trace:
I20260709 12:23:58.451665 31822 rpcz_store.cc:276] 0709 12:23:56.406034 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.814262 (+1408228us) service_pool.cc:224] Handling call
0709 12:23:57.818988 (+  4726us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.838154 (+ 19166us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:57.960601 (+122447us) log.cc:844] Serialized 14 byte log entry
0709 12:23:58.447316 (+486715us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:58.447654 (+   338us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:58.447733 (+    79us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.447836 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.447994 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":339748,"prepare.run_cpu_time_us":108,"prepare.run_wall_time_us":110,"thread_start_us":584257,"threads_started":3}
W20260709 12:23:58.471657 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 428184us
W20260709 12:23:58.479846 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 442829us
I20260709 12:23:58.489996 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1167) took 2148 ms. Trace:
I20260709 12:23:58.482817 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2598) took 2141 ms. Trace:
I20260709 12:23:58.493875 31675 rpcz_store.cc:276] 0709 12:23:56.341305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.386032 (+1044727us) service_pool.cc:224] Handling call
0709 12:23:57.386394 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.670700 (+284306us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.489692 (+818992us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b5400046ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:58.489912 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":626654592}
I20260709 12:23:58.494854 31746 rpcz_store.cc:276] 0709 12:23:56.341878 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.011899 (+670021us) service_pool.cc:224] Handling call
0709 12:23:58.482529 (+1470630us) spinlock_profiling.cc:243] Waited 334 ms on lock 0x7b5400404cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:58.482731 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":767446400}
W20260709 12:23:58.488360 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:58.487704 31819 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3199) took 1616 ms (client timeout 1939 ms). Trace:
I20260709 12:23:58.489126 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3281) took 2528 ms. Trace:
I20260709 12:23:58.505285 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:58.505378 31820 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
W20260709 12:23:58.498368 31819 rpcz_store.cc:269] 0709 12:23:56.870897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.953947 (+ 83050us) service_pool.cc:224] Handling call
0709 12:23:58.487422 (+1533475us) spinlock_profiling.cc:243] Waited 861 ms on lock 0x7b54000fb754. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:58.487618 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1980624768}
I20260709 12:23:58.505903 31748 rpcz_store.cc:276] 0709 12:23:55.960707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.374821 (+1414114us) service_pool.cc:224] Handling call
0709 12:23:57.473747 (+ 98926us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.534388 (+ 60641us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.534447 (+    59us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:57.550381 (+ 15934us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:57.550448 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.550543 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.488891 (+938348us) spinlock_profiling.cc:243] Waited 85 ms on lock 0x7b54003e3ad0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:58.489045 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":195424640}
I20260709 12:23:58.509030 32377 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:23:58.510955 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3218) took 1073 ms. Trace:
I20260709 12:23:58.518309 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:58.522317 31819 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:23:58.524103 31819 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:23:58.466002  6821 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:58.517901 31820 rpcz_store.cc:276] 0709 12:23:57.437353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.483654 (+1046301us) service_pool.cc:224] Handling call
0709 12:23:58.484052 (+   398us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.506200 (+ 22148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.510889 (+  4689us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.535420 31820 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:23:58.464090  6896 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:58.478754 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:58.539614 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:58.543232 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3293) took 1070 ms. Trace:
I20260709 12:23:58.543744 31747 rpcz_store.cc:276] 0709 12:23:57.472919 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.505211 (+1032292us) service_pool.cc:224] Handling call
0709 12:23:58.542642 (+ 37431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.542868 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:58.542896 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:58.542939 (+    43us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:58.542970 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.543023 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.543157 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:23:58.549033 31819 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:58.550621 31819 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:58.549791 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:58.551396 31822 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:58.502784  6882 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:58.513465  6794 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:58.538244  6909 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.908s	user 0.010s	sys 0.004s
W20260709 12:23:58.548098  6501 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.812s	user 0.013s	sys 0.001s
I20260709 12:23:58.536593  6828 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:58.551785  6814 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:23:58.562919  6643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.018s	sys 0.049s
I20260709 12:23:58.579330 31819 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:58.594138  6643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.558s	user 0.019s	sys 0.049s
I20260709 12:23:58.595351  6643 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20260709 12:23:58.584111 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.609745 31819 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:23:58.618216 31671 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.685s	user 0.009s	sys 0.003s
I20260709 12:23:58.618786 31671 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:23:58.591533  6930 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:58.610019  6783 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.260s	user 0.011s	sys 0.003s
I20260709 12:23:58.549573  6933 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:58.620582  7026 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.268s	user 0.002s	sys 0.000s
I20260709 12:23:58.583982  6501 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:58.606978  6896 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:23:58.620142  7010 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.530s	user 0.001s	sys 0.001s
I20260709 12:23:58.598533  6828 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 12:23:58.644438 31822 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
W20260709 12:23:58.646436 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.578365  6909 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:58.691876 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:23:58.643464  7019 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.478s	user 0.001s	sys 0.001s
I20260709 12:23:58.614567  7027 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.030s
I20260709 12:23:58.608666  7042 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.084s
W20260709 12:23:58.692978  7022 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.340s	user 0.002s	sys 0.000s
W20260709 12:23:58.711334  7017 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.514s	user 0.000s	sys 0.001s
I20260709 12:23:58.632647  6930 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:58.635311  6783 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:58.723155  7039 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.028s
W20260709 12:23:58.740125  7018 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
W20260709 12:23:58.743552  7025 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260709 12:23:58.746968  6794 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:58.747676  6794 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:58.748456  6980 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.606s	user 0.003s	sys 0.009s
I20260709 12:23:58.750269  6794 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:58.707338  7013 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:58.711889  7030 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:23:58.770109  7030 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:23:58.771541  7030 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:58.774016  6735 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:58.777984 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:58.697091  6896 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:58.781064  6896 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:58.755312  6828 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:23:58.783324  6828 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:58.784672  6828 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:58.712541  7021 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:23:58.764815 31673 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.712s	user 0.012s	sys 0.000s
I20260709 12:23:58.765653  6743 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:58.799127 31671 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:23:58.768409  6682 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:58.769153  6909 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:58.788650 31670 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
W20260709 12:23:58.793740 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.797719 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:58.817451 31678 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
W20260709 12:23:58.817411 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.819461  5949 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:58.811888 31673 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:23:58.798434 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:58.797323 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:58.798043 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:58.823247 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:58.770051  7032 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.317s	user 0.002s	sys 0.000s
I20260709 12:23:58.828861 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:23:58.832765 31671 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:58.830557 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:58.820088  5949 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:58.795396  7042 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:23:58.819504  6752 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.195s	user 0.007s	sys 0.008s
I20260709 12:23:58.786576  6828 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:58.818445  6743 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:58.827992  6794 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:58.855190  6909 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:23:58.794245  7021 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:23:58.857235  6859 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.008s	user 0.013s	sys 0.001s
I20260709 12:23:58.857431  7021 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:58.725770  6960 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:58.789779  6783 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:58.826561  7014 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:23:58.836078  6651 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.014s
I20260709 12:23:58.834740  7042 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:23:58.850826  5949 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:58.859086 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 380449us
I20260709 12:23:58.878458 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:58.879990 31671 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:23:58.859439  6882 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:23:58.885216 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:58.885370  6743 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:58.886613  6882 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:58.891247 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3215) took 1454 ms. Trace:
I20260709 12:23:58.894773  7037 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:23:58.895339  7037 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:58.879599 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:58.896598 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:58.900334 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 334491us
I20260709 12:23:58.833412 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:58.916507 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:58.945055 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:23:58.967360 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2602) took 2624 ms. Trace:
I20260709 12:23:58.967983 31749 rpcz_store.cc:276] 0709 12:23:56.343103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.840247 (+1497144us) service_pool.cc:224] Handling call
0709 12:23:57.921172 (+ 80925us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:57.963911 (+ 42739us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.963954 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:57.964565 (+   611us) log.cc:844] Serialized 26 byte log entry
0709 12:23:58.123455 (+158890us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:23:58.123516 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.123612 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:58.967029 (+843417us) raft_consensus.cc:1685] finished
0709 12:23:58.967064 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.967266 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22898,"raft.queue_time_us":9497,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":17333,"thread_start_us":158030,"threads_started":1,"wal-append.queue_time_us":447582}
I20260709 12:23:58.895757  6736 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:58.892153 31824 rpcz_store.cc:276] 0709 12:23:57.436498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.199436 (+762938us) service_pool.cc:224] Handling call
0709 12:23:58.200421 (+   985us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.200728 (+   307us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:58.602252 (+401524us) log.cc:844] Serialized 14 byte log entry
0709 12:23:58.890041 (+287789us) log.cc:844] Serialized 14 byte log entry
0709 12:23:58.890433 (+   392us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:58.890517 (+    84us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:23:58.890667 (+   150us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.890815 (+   148us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:58.891119 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":407,"prepare.queue_time_us":352491,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":104,"raft.queue_time_us":10557,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":166,"thread_start_us":546052,"threads_started":2}
I20260709 12:23:58.995728 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:58.900808  7037 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:23:58.970517  7031 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:23:59.017642 31824 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:23:59.020043 31749 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 23 index: 12. (index mismatch)
I20260709 12:23:59.021762 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:23:58.901221  7049 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:58.922785  7053 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:58.922490  6783 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:58.943215  6743 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:58.976258 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:23:59.037189 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.033828 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:58.963415  6643 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.060s
I20260709 12:23:59.049800 31818 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 11. Preceding OpId from leader: term: 24 index: 12. (index mismatch)
I20260709 12:23:59.027873  7041 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:23:59.035171 31824 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:59.022023  7045 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:23:59.027637  7049 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:59.040495 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3220) took 1602 ms. Trace:
I20260709 12:23:59.027153  7053 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:59.017694  6736 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:59.053578 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.931s	user 0.005s	sys 0.012s
I20260709 12:23:59.058248 31827 rpcz_store.cc:276] 0709 12:23:57.437881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.486162 (+1048281us) service_pool.cc:224] Handling call
0709 12:23:58.486661 (+   499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.486941 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 9
0709 12:23:58.543371 (+ 56430us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.040035 (+496664us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.040107 (+    72us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:23:59.040176 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.040266 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.040422 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":103878,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":57,"thread_start_us":496217,"threads_started":1}
I20260709 12:23:59.044713  6885 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:23:59.076141 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.953s	user 0.005s	sys 0.012s
I20260709 12:23:59.038338  6783 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:59.079605 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 958902us
I20260709 12:23:59.072816  7049 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:59.059193  7006 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.115s	user 0.015s	sys 0.000s
W20260709 12:23:59.058161  6858 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.942s	user 0.006s	sys 0.011s
I20260709 12:23:59.050130  6864 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:59.053367  7041 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:59.017197  7042 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.042081  6743 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:59.128283 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 46672.6us
I20260709 12:23:59.075227  7053 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:23:59.144032 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:23:59.146322 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 293103us
I20260709 12:23:59.045423  6828 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.031s
I20260709 12:23:59.079025  6930 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:23:59.136884  6858 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.041s	user 0.006s	sys 0.011s
I20260709 12:23:59.078912  6885 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:59.165899 31821 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.007s	sys 0.010s
W20260709 12:23:59.166438 31821 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.007s	sys 0.010s
I20260709 12:23:59.136554  7006 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:23:59.136081  7045 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:23:59.144476  7047 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.273s	user 0.000s	sys 0.001s
W20260709 12:23:59.144994  7057 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:23:59.170794 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:59.171779 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:59.174738  6960 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:23:59.175040  5949 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:23:59.175309  7048 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:23:59.175923 31669 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:23:59.174095  6885 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.178448 31674 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 22. Current term is 23. Ops: [22.18-22.18]
I20260709 12:23:59.179260 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:59.180423  6909 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
W20260709 12:23:59.181704  7020 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.061s	user 0.001s	sys 0.001s
I20260709 12:23:59.180464 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1242) took 2692 ms. Trace:
W20260709 12:23:59.178510  7024 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.785s	user 0.001s	sys 0.000s
I20260709 12:23:59.184074  6821 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:59.271337  6885 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:23:59.271932  6821 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:59.185002  6743 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:57.939707 31744 rpcz_store.cc:276] 0709 12:23:55.960320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.344963 (+1384643us) service_pool.cc:224] Handling call
0709 12:23:57.354040 (+  9077us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:57.354253 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:57.680565 (+326312us) log.cc:844] Serialized 14 byte log entry
0709 12:23:57.937488 (+256923us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:57.937548 (+    60us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:57.937605 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.937677 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:57.937790 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3946,"prepare.queue_time_us":46109,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":25717,"thread_start_us":486635,"threads_started":2}
I20260709 12:23:59.139400  7051 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.098s
I20260709 12:23:59.275059 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:59.275983 31744 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:23:59.277521 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1048) took 1466 ms. Trace:
I20260709 12:23:59.277462 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:59.278877 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3278) took 3324 ms. Trace:
I20260709 12:23:59.278046 32371 rpcz_store.cc:276] 0709 12:23:57.811024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.811151 (+   127us) service_pool.cc:224] Handling call
0709 12:23:57.822212 (+ 11061us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:57.838541 (+ 16329us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.838590 (+    49us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:57.855795 (+ 17205us) log.cc:844] Serialized 52 byte log entry
0709 12:23:58.031594 (+175799us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:58.031669 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:58.031791 (+   122us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.177846 (+1146055us) spinlock_profiling.cc:243] Waited 15.8 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:59.277194 (+ 99348us) raft_consensus.cc:1685] finished
0709 12:23:59.277232 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.277459 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":600,"raft.queue_time_us":7713,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":6552,"spinlock_wait_cycles":36433664,"thread_start_us":135368,"threads_started":1,"wal-append.queue_time_us":268218}
I20260709 12:23:59.279392 31753 rpcz_store.cc:276] 0709 12:23:55.954288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.247913 (+1293625us) service_pool.cc:224] Handling call
0709 12:23:57.248490 (+   577us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:57.312667 (+ 64177us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:57.662202 (+349535us) log.cc:844] Serialized 14 byte log entry
0709 12:23:57.863712 (+201510us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:57.931667 (+ 67955us) log.cc:844] Serialized 26 byte log entry
0709 12:23:57.931861 (+   194us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:57.931900 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:57.931986 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.181106 (+1249120us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:59.278473 (+ 97367us) raft_consensus.cc:1685] finished
0709 12:23:59.278616 (+   143us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.278814 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24430,"prepare.queue_time_us":65442,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":65,"raft.queue_time_us":67263,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":5397,"spinlock_wait_cycles":39497472,"thread_start_us":428369,"threads_started":2,"wal-append.queue_time_us":285299}
W20260709 12:23:59.186796 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:59.282791 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.268275 32374 rpcz_store.cc:276] 0709 12:23:56.488478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.488674 (+   196us) service_pool.cc:224] Handling call
0709 12:23:56.513678 (+ 25004us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:57.430964 (+917286us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:57.430999 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:57.431577 (+   578us) log.cc:844] Serialized 26 byte log entry
0709 12:23:57.752096 (+320519us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:57.752133 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.180013 (+1427880us) spinlock_profiling.cc:243] Waited 917 ms on lock 0x7b540031d654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:59.180069 (+    56us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.180117 (+    48us) raft_consensus.cc:1685] finished
0709 12:23:59.180135 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.180373 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2108776832,"thread_start_us":320008,"threads_started":1,"wal-append.queue_time_us":440090,"wal-append.run_cpu_time_us":2105,"wal-append.run_wall_time_us":1002133}
W20260709 12:23:59.283352 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.142091  6864 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:59.289423 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 286467us
I20260709 12:23:59.200235  6783 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:59.291586 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:59.203847  7052 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:23:59.204783 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2596) took 2863 ms. Trace:
I20260709 12:23:59.228425  6862 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:59.304237 31751 rpcz_store.cc:276] 0709 12:23:56.341259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.762728 (+1421469us) service_pool.cc:224] Handling call
0709 12:23:57.809380 (+ 46652us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.052487 (+243107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.204413 (+1151926us) spinlock_profiling.cc:243] Waited 173 ms on lock 0x7b54001a4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:59.204706 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":397349632}
I20260709 12:23:59.208122  6735 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Stepping down as leader of term 19
W20260709 12:23:59.208388  6647 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.078s	user 0.012s	sys 0.000s
I20260709 12:23:59.222846 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:23:59.226825 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.142786  7041 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:23:59.231473 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1308) took 1807 ms. Trace:
W20260709 12:23:59.231134 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 61550.5us
W20260709 12:23:59.318823 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 96083.4us
W20260709 12:23:59.234750 31677 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.004s	sys 0.047s
I20260709 12:23:59.248798 31669 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:59.249796  5949 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:59.257647 31673 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: []
W20260709 12:23:59.259325 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.263283  6784 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.287132 31744 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 23 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:23:59.196978  6501 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:59.226209  7045 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:59.200235  7001 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:23:59.321038 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:59.147607  7056 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.139s
I20260709 12:23:59.257462  7042 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:23:59.347018 31677 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.008s	sys 0.060s
I20260709 12:23:59.218518  7021 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:59.229560  6881 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.080s
I20260709 12:23:59.297144  6643 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:23:59.319707  7047 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:23:59.351752 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:59.302186  7048 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:59.320533  6941 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.944s	user 0.014s	sys 0.000s
I20260709 12:23:59.345137  7036 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:59.349581  6647 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:23:59.350229 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:23:59.310278  6783 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:23:59.354472 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:59.319226 31821 rpcz_store.cc:276] 0709 12:23:57.423711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.051961 (+628250us) service_pool.cc:224] Handling call
0709 12:23:58.087727 (+ 35766us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.088009 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:58.600908 (+512899us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.231067 (+630159us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.231131 (+    64us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:59.231186 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.231270 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.231411 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":96156,"prepare.queue_time_us":325439,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":64616,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":23338,"thread_start_us":1031361,"threads_started":2,"wal-append.queue_time_us":699931}
I20260709 12:23:59.349771  7045 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:59.307575  6735 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 19 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:59.344981  6909 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.289722  7006 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.061s
I20260709 12:23:59.361255 31751 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:23:59.348331  6876 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.344870  6700 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.354187 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:59.407480 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53076.5us
I20260709 12:23:59.355080  5949 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.349617  7001 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:23:59.376001  7038 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.601s	user 0.045s	sys 0.095s
W20260709 12:23:59.412940  7038 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.045s	sys 0.096s
I20260709 12:23:59.358445  6651 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:23:59.413647  6651 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:59.414112  6651 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:59.376655 31820 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:23:59.359246  7060 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.310832  6862 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:59.488013  6862 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:59.370720  7056 consensus_queue.cc:1243] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 20.12 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:23:59.399461  6990 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:59.412129 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58048.9us
I20260709 12:23:59.499708 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:59.414968  6960 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:59.500988 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:59.416041  6647 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:59.376664  6941 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:59.364272  6784 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.503994  6743 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:59.503896  7038 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:23:59.506300  5949 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:23:59.503367 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 91882.1us
W20260709 12:23:59.410276  6814 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.808s	user 0.006s	sys 0.007s
I20260709 12:23:59.359409  6682 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.036s
W20260709 12:23:59.376641  6929 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.287s	user 0.015s	sys 0.000s
I20260709 12:23:59.487390  7001 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.361024  6643 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:23:59.488951  7030 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:59.490989  7005 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.776s	user 0.010s	sys 0.002s
I20260709 12:23:59.381467  7027 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:59.415702  7071 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
W20260709 12:23:59.540046 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.509680 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3353) took 1132 ms. Trace:
I20260709 12:23:59.504880  6743 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:59.409008  6909 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:59.551442 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.015s	sys 0.051s
I20260709 12:23:59.381697  7036 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:59.415251  6700 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.508344  6859 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.411120  6876 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.504908  7039 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
I20260709 12:23:59.498874  6783 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.024s
W20260709 12:23:59.553718 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.541234  6682 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.518697  6814 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:23:59.559597 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50694.1us
I20260709 12:23:59.580319 31749 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 18. Current term is 19. Ops: []
I20260709 12:23:59.571360  6859 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:23:59.583207  7072 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
I20260709 12:23:59.583616  6859 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:59.584120  6700 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:59.584429  6859 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:23:59.584666  7050 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.204s	user 0.000s	sys 0.000s
W20260709 12:23:59.552771 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.505s	user 0.015s	sys 0.051s
I20260709 12:23:59.570895  6876 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:59.552199  7001 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
W20260709 12:23:59.591043 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.545305  6743 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:59.513965  7049 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:59.519127  6929 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:59.542701 31822 rpcz_store.cc:276] 0709 12:23:58.377058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.690456 (+313398us) service_pool.cc:224] Handling call
0709 12:23:58.707167 (+ 16711us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.719036 (+ 11869us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:23:58.731034 (+ 11998us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.507807 (+776773us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.507893 (+    86us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:59.507967 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.509394 (+  1427us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.509581 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1666,"prepare.queue_time_us":25054,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":109421,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":126,"thread_start_us":721741,"threads_started":2,"wal-append.queue_time_us":605134}
I20260709 12:23:59.512202  7060 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.557160  7071 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:23:59.640766 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60649us
W20260709 12:23:59.630390 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:59.599427 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 565647us
I20260709 12:23:59.550115  6858 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:23:59.642799 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:23:59.590742  6828 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:23:59.543352  7030 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.578824  6784 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:23:59.594848  7066 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
I20260709 12:23:59.584987  6859 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.592610  6864 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:23:59.656457 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 435765us
I20260709 12:23:59.644567  6828 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.632359  6743 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:23:59.643438  7037 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.620s	user 0.007s	sys 0.006s
W20260709 12:23:59.650058 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 147837us
W20260709 12:23:59.647287  7059 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.286s	user 0.001s	sys 0.000s
I20260709 12:23:59.594799  6682 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:59.664512 31825 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:23:59.650116  7065 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.234s	user 0.001s	sys 0.000s
W20260709 12:23:59.657963 31718 connection.cc:256] RPC call timeout handler was delayed by -1.0936732590002975s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2141s for a response
I20260709 12:23:59.645826  6923 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:59.653581  6700 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.539067  7048 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.695071 32369 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 17. (term mismatch)
I20260709 12:23:59.695472 32369 pending_rounds.cc:85] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 16
I20260709 12:23:59.695695 32369 pending_rounds.cc:107] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 24.17
I20260709 12:23:59.695945 32369 raft_consensus.cc:2889] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:23:59.684681 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:59.686002 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:59.593020  6643 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.693014 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:59.688496 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:59.700953  7070 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.696913 32370 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:23:59.678071 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:59.674741  7021 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:59.707954 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:59.711551 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:59.682469  7063 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:23:59.679385  7075 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:23:59.711400 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:59.715548 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:59.682152  7039 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.730266 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:23:59.730909 31744 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:23:59.719753 31748 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:23:59.653374  6647 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:23:59.727643 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49650.8us
I20260709 12:23:59.693531  6864 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.729599 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:23:59.736971 31752 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.004s	sys 0.001s
I20260709 12:23:59.736858 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:23:59.737411 31752 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.516s	user 0.004s	sys 0.001s
I20260709 12:23:59.723286 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:59.674443  6501 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.032s
I20260709 12:23:59.676868  7037 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:23:59.765822 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.776311 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:59.725324  6990 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:23:59.780354 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:23:59.779835 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:23:59.725772  6752 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:59.781236 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:59.725373  7075 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:23:59.789688 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:23:59.787302 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:59.698606  6700 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.774976  7027 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.697893  6923 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:59.841548 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3292) took 2369 ms. Trace:
I20260709 12:23:59.778084  7021 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:23:59.803428 31744 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:23:59.844425 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 1313) took 1119 ms. Trace:
I20260709 12:23:59.820231 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3344) took 1617 ms. Trace:
I20260709 12:23:59.862751 31825 rpcz_store.cc:276] 0709 12:23:58.202765 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.480442 (+277677us) service_pool.cc:224] Handling call
0709 12:23:59.820027 (+1339585us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54003c8cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:59.820168 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2624555904}
I20260709 12:23:59.839151 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3356) took 1439 ms. Trace:
W20260709 12:23:59.867412 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:23:59.858914 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 69296.1us
I20260709 12:23:59.861020 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:23:59.781639  6941 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:59.864360 31820 rpcz_store.cc:276] 0709 12:23:58.399816 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.855134 (+455318us) service_pool.cc:224] Handling call
0709 12:23:58.855449 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.449168 (+593719us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.838873 (+389705us) spinlock_profiling.cc:243] Waited 521 ms on lock 0x7b540043dfd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:23:59.839078 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1198156032}
I20260709 12:23:59.745996  6930 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:23:59.857909 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 157325us
I20260709 12:23:59.862509 31823 rpcz_store.cc:276] 0709 12:23:58.724943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.777545 (+ 52602us) service_pool.cc:224] Handling call
0709 12:23:59.844201 (+1066656us) spinlock_profiling.cc:243] Waited 524 ms on lock 0x7b5400447ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:23:59.844356 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1205754624}
I20260709 12:23:59.795002 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:23:59.828936  7067 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.202s	user 0.001s	sys 0.000s
I20260709 12:23:59.814144  7058 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:23:59.812093  7073 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.166s	user 0.000s	sys 0.000s
W20260709 12:23:59.811442  7080 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:23:59.915129 32378 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 16 index: 10. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:23:59.734776  7077 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.131s
I20260709 12:23:59.721894  7070 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.856869 31750 rpcz_store.cc:276] 0709 12:23:57.472617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.371251 (+898634us) service_pool.cc:224] Handling call
0709 12:23:58.403577 (+ 32326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:58.404196 (+   619us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:23:59.210952 (+806756us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b18002bbbe0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:23:59.513523 (+302571us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.714076 (+200553us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:59.714590 (+   514us) log.cc:844] Serialized 26 byte log entry
0709 12:23:59.714960 (+   370us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:23:59.715001 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.715082 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.841204 (+126122us) raft_consensus.cc:1685] finished
0709 12:23:59.841318 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.841465 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3235,"prepare.queue_time_us":732948,"prepare.run_cpu_time_us":90,"prepare.run_wall_time_us":6796,"raft.queue_time_us":5624,"raft.run_cpu_time_us":390,"raft.run_wall_time_us":62841,"spinlock_wait_cycles":27199360,"thread_start_us":454274,"threads_started":2,"wal-append.queue_time_us":249618}
W20260709 12:23:59.810201  7079 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.105s	user 0.000s	sys 0.001s
I20260709 12:23:59.923525 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:23:59.960719 32369 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:23:59.923926 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:23:59.823331  7062 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.372s	user 0.003s	sys 0.000s
I20260709 12:23:59.792279  7084 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:23:59.964418 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1074) took 4293 ms. Trace:
I20260709 12:23:59.738705  7039 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.927413 32376 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:23:59.968856  7039 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:59.969594  7039 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.794325  6862 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:23:59.971915  6862 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:23:59.964896 31745 rpcz_store.cc:276] 0709 12:23:55.671314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:56.327442 (+656128us) service_pool.cc:224] Handling call
0709 12:23:56.351737 (+ 24295us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:23:56.984187 (+632450us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:56.984231 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:23:58.752925 (+1768694us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b2c00001158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:23:58.753613 (+   688us) log.cc:844] Serialized 52 byte log entry
0709 12:23:59.049977 (+296364us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:23:59.050016 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:23:59.051361 (+  1345us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:23:59.822553 (+771192us) spinlock_profiling.cc:243] Waited 22.6 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:23:59.964103 (+141550us) raft_consensus.cc:1685] finished
0709 12:23:59.964134 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:23:59.964351 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31495,"raft.queue_time_us":50391,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":13275,"spinlock_wait_cycles":60301568,"thread_start_us":856218,"threads_started":2,"wal-append.queue_time_us":602235}
I20260709 12:23:59.931774 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:59.832499  7014 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:23:59.813659  7083 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:23:59.812424  7093 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:23:59.839306  6909 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:59.813447  6682 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:23:59.940127 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:23:59.810359  6700 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:23:59.816609  7037 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:23:59.880656  7053 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.672s	user 0.013s	sys 0.000s
W20260709 12:23:59.943976 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 149050us
I20260709 12:23:59.944494  7051 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.732021  7063 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:59.945017  7091 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:23:59.791500  7071 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 17.10, Last appended by leader: 10, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:59.803761  7075 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:23:59.862959  7027 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.874234  6784 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:23:59.879674  6752 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:23:59.924571  6930 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:23:59.946988  7041 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.591s	user 0.011s	sys 0.000s
I20260709 12:23:59.947384  6647 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.948313  7064 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.094s
I20260709 12:23:59.951026 31746 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:23:59.772796  6864 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:59.915390  6876 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:23:59.959915 31751 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 22. Current term is 23. Ops: [22.18-22.18]
I20260709 12:23:59.974802 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:23:59.976120 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:23:59.981956 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 258779us
I20260709 12:23:59.985217 31744 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:23:59.935527  6941 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:23:59.810503  6986 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.184s
I20260709 12:23:59.821003  6651 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:59.985298 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:23:59.923394  6858 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.056s
I20260709 12:23:59.970189  7039 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:23:59.993525 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:23:59.958364  7087 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.003388 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3298) took 1491 ms. Trace:
I20260709 12:23:59.816443  6735 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:23:59.975378  6929 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:23:59.978682  7083 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:23:59.986501  7049 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:00.024701 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 148443us
W20260709 12:24:00.032194  5949 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.525s	user 0.012s	sys 0.000s
W20260709 12:24:00.031267 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 251487us
I20260709 12:24:00.034010 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:00.043479 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 48377.2us
I20260709 12:24:00.017098  7100 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:24:00.047134 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 61901.1us
I20260709 12:24:00.024492  7030 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.030031  6784 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.051553 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.040606  6930 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:00.052251  6783 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:00.008009  7053 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.050663 31752 rpcz_store.cc:276] 0709 12:23:58.512116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.751285 (+239169us) service_pool.cc:224] Handling call
0709 12:23:58.751716 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.752019 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:23:59.167229 (+415210us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.982600 (+815371us) log.cc:844] Serialized 14 byte log entry
0709 12:24:00.002924 (+ 20324us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:00.002994 (+    70us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:00.003059 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.003147 (+    88us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.003309 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19610,"prepare.queue_time_us":449375,"prepare.run_cpu_time_us":112,"prepare.run_wall_time_us":111,"raft.queue_time_us":88298,"raft.run_cpu_time_us":374,"raft.run_wall_time_us":82799,"thread_start_us":1083056,"threads_started":3,"wal-append.queue_time_us":641945}
I20260709 12:24:00.044884  6881 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:23:59.994928  7014 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.075786 31675 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:00.046293  6828 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:00.033394  6752 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:00.079615 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:00.080969 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:00.083057  7091 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:00.078619  6784 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:00.082335  6923 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.090966 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.116516 31678 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:00.092792 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.088904  6783 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:00.063591  6876 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.017328  7051 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.058481  7083 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:00.041391  7041 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.049304  6647 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.059430  7100 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:00.087471  7030 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:00.105932 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:00.144941 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:00.106596  5949 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:00.107971  6743 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.105629 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:00.148810  7094 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:00.149359 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:00.149434  7094 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:00.095126  6735 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:00.150137  7094 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:00.150367  6743 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.151495 32371 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:00.152393  7104 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:00.154721 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:00.104233  6752 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:00.156075 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3296) took 1658 ms. Trace:
I20260709 12:24:00.156484 31746 rpcz_store.cc:276] 0709 12:23:58.497562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.531407 (+ 33845us) service_pool.cc:224] Handling call
0709 12:23:58.559764 (+ 28357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.042095 (+1482331us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.155833 (+113738us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400295ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:00.156014 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3198846336}
W20260709 12:24:00.158874  6885 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.887s	user 0.009s	sys 0.005s
I20260709 12:24:00.159332  7100 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:00.139694 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.162720 31669 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.164139 31669 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:00.164222  7096 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:00.164853  7096 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:00.165220  7096 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:00.156627  7051 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:00.121798  6783 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:00.127854  6876 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.063148  7087 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.098071  6881 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.078243  7060 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.155531  7086 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:00.111150  6828 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:00.154937  7013 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:24:00.171109 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.177045 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:00.137975  6923 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.096596  7093 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.053s
W20260709 12:24:00.166589  6821 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.894s	user 0.010s	sys 0.004s
I20260709 12:24:00.334681 31827 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:24:00.222046 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:00.340548 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 43789.4us
I20260709 12:24:00.325001 31825 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:24:00.360658 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:00.361012 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 139047us
I20260709 12:24:00.285701 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1178) took 2330 ms. Trace:
I20260709 12:24:00.291960 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.215502 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1180) took 1044 ms. Trace:
I20260709 12:24:00.363446 31673 rpcz_store.cc:276] 0709 12:23:59.170912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.176539 (+  5627us) service_pool.cc:224] Handling call
0709 12:23:59.176858 (+   319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:59.346985 (+170127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.215253 (+868268us) spinlock_profiling.cc:243] Waited 80.5 ms on lock 0x7b54001b2bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:00.215437 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":185226624}
I20260709 12:24:00.283360 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3373) took 2590 ms. Trace:
I20260709 12:24:00.327270 31818 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
W20260709 12:24:00.188519 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:00.284996 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:00.367551 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 190644us
I20260709 12:24:00.341132 31676 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:24:00.369079 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:00.369309 31670 rpcz_store.cc:276] 0709 12:23:57.692573 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:57.725551 (+ 32978us) service_pool.cc:224] Handling call
0709 12:23:57.726006 (+   455us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.818005 (+1091999us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.283069 (+1465064us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54002b0a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:00.283285 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2443285376}
W20260709 12:24:00.191051 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 137378us
I20260709 12:24:00.371331 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:00.177749  7104 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:24:00.386607 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 290838us
I20260709 12:24:00.375865 31672 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:00.362452 31677 rpcz_store.cc:276] 0709 12:23:57.955229 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:58.038466 (+ 83237us) service_pool.cc:224] Handling call
0709 12:23:58.083133 (+ 44667us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:23:58.375456 (+292323us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:23:58.425711 (+ 50255us) log.cc:844] Serialized 14 byte log entry
0709 12:23:58.710596 (+284885us) log.cc:844] Serialized 14 byte log entry
0709 12:23:59.352506 (+641910us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:23:59.352581 (+    75us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:23:59.352649 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.285346 (+932697us) spinlock_profiling.cc:243] Waited 292 ms on lock 0x7b54002b0f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:00.285417 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.285617 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13531,"prepare.queue_time_us":36351,"prepare.run_cpu_time_us":98,"prepare.run_wall_time_us":2607,"raft.queue_time_us":190999,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":170,"spinlock_wait_cycles":671375232,"thread_start_us":908343,"threads_started":2,"wal-append.queue_time_us":389165}
I20260709 12:24:00.380666 31826 pending_rounds.cc:85] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 11
I20260709 12:24:00.387847 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3392) took 1218 ms. Trace:
I20260709 12:24:00.388051 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:00.388304 31826 pending_rounds.cc:107] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 22.12
I20260709 12:24:00.167758  6990 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:00.388727 31826 raft_consensus.cc:2889] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:00.387907 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:00.388381 31669 rpcz_store.cc:276] 0709 12:23:59.169046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.175449 (+  6403us) service_pool.cc:224] Handling call
0709 12:23:59.175801 (+   352us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.162297 (+986496us) spinlock_profiling.cc:243] Waited 360 ms on lock 0x7b540035f320. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd5306f 00007fe8fcdcbf60 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62
0709 12:24:00.387514 (+225217us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:00.387545 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:00.387583 (+    38us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:00.387616 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.387675 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.387788 (+   113us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9614,"spinlock_wait_cycles":827681664}
I20260709 12:24:00.391604 31823 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: [22.14-22.14]
W20260709 12:24:00.332835  7045 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.921s	user 0.010s	sys 0.004s
I20260709 12:24:00.399456 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:00.401094 32370 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:24:00.396658 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:00.405406 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:00.203666  6736 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:00.397397 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:00.208599  6876 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.325433  6859 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.207355  7098 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:00.330242  6643 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:00.209029  7053 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:00.430917 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:00.408989 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.431960 31825 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:00.184363  7112 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:00.211992  7070 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:00.198329  7086 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:00.213191  6651 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.321692  7013 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:00.413164 31753 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:00.213867  7027 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:00.215600  7041 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:00.414954  5949 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:00.425459  7037 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:00.421837 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.374905  6794 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.047s
I20260709 12:24:00.398870  7063 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.399437  6930 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.459698  7096 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.374605  6682 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:00.462620  6881 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:00.424932 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.430209 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:00.484421 31824 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:00.375732  6864 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.430475 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:00.485558 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78797.2us
I20260709 12:24:00.182027  6909 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.217s
I20260709 12:24:00.216485  7039 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.406934 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:00.488623 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.491449 31824 raft_consensus.cc:2410] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:24:00.211638  7014 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:00.417050 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:00.436336 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.494078 31669 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:00.396358 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.437844 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:00.439142 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.209129  6752 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.436422 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:00.396834  7049 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.399544  7077 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:00.407557  7060 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:00.413203  7076 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:00.190085  7095 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.243s
I20260709 12:24:00.368139  6821 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.450735 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.511344 31825 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:00.203030  6862 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.212381  7106 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.341s
I20260709 12:24:00.428293  6882 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:00.461568  5949 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.419224 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:00.444404  6960 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.938s	user 0.010s	sys 0.003s
I20260709 12:24:00.327872  7094 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.381788  6923 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.536866  6923 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:00.451993  7086 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:00.444998  7058 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:00.470980 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.539780 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:00.540439 31673 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:00.465628  7013 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.543434  7013 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:00.545218 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.423846 31818 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:24:00.546830 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.470378 31672 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: []
I20260709 12:24:00.548098  6700 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.548619 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.396348  7045 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.548694  6700 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.551049  7106 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:00.551244  7045 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:00.486086 31823 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:24:00.552640 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.529963  6882 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.553646 31821 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:24:00.487025 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:00.554656 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 67696.3us
I20260709 12:24:00.554842  6905 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:00.555285  6905 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:00.555634  6905 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:00.488325 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.558246 31671 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:00.559669 31671 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:00.474390  6930 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:00.399690  6990 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:00.495955 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.510402  7041 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Leader pre-election won for term 16
W20260709 12:24:00.498658 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 91824.6us
I20260709 12:24:00.572818 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:00.501080 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.573050 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:00.503603 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 67271.5us
I20260709 12:24:00.574086 31825 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:00.505204 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.474150  6783 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.484105  7096 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.584219 31675 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:00.576519 31827 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:00.586287 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.587168  6700 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:00.587505 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:00.591243 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.590824  6990 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.539072  7068 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:00.435473  7098 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.487561  6794 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:00.507293  6864 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:00.470042  7037 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:00.529421 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 110271us
I20260709 12:24:00.469470  7101 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:24:00.615370 31753 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:00.530305  5949 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:00.513773  7049 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.534044  7071 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:00.551568  7106 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:00.637588 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:00.572403  7041 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.642922 31747 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:00.523396  7014 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:00.578985 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:00.656819 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:00.583379  6783 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:00.594336 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:00.671406 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.453413  7112 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.497236  7074 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.074s
W20260709 12:24:00.509774 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 92808.5us
I20260709 12:24:00.677867 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:00.679949 31748 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:00.600731 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:00.680150 31642 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:00.683195 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:00.546468  7094 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.540453  7087 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.686002 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:00.532737 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:00.615434 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:00.619488  5949 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:00.533540  6960 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:00.572002  6862 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:00.636300 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.507262  7060 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.641237  7071 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:00.736320 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 134241us
I20260709 12:24:00.478067  6784 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:00.611716  7042 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.253s	user 0.012s	sys 0.005s
I20260709 12:24:00.567354 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.749979 31669 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 14.
I20260709 12:24:00.659636 31749 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:00.669557  6736 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.567384  6858 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 21.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.056s
I20260709 12:24:00.755574 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:00.676843 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 16 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.619091  6864 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:00.618714 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.692947 31745 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:00.707746 31574 catalog_manager.cc:5697] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 16 to 17, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 17 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:00.709471 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:00.717006 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:00.727981 31747 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:00.657958  7041 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 15 FOLLOWER]: Advancing to term 16
W20260709 12:24:00.728823 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 196239us
I20260709 12:24:00.692019  5949 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:24:00.731791 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48727.3us
I20260709 12:24:00.733284 31825 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:24:00.738090  6783 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:00.616308  7029 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.070s
I20260709 12:24:00.745052  7071 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:00.641779 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:00.776633  6876 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.777464  6876 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.780102  6876 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.786960  6783 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:00.787359  6735 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.637s	user 0.013s	sys 0.004s
I20260709 12:24:00.788831 31677 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 10.
I20260709 12:24:00.788317  6783 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 21.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.790823  6682 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:00.791057  6752 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.679581  6905 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.793259 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.794931  6905 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:00.764183  6864 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:00.795686 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.797627  5949 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:00.798018  5949 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:00.798767  5949 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.620265  6930 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.070s
I20260709 12:24:00.670747  7104 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:00.747874 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:00.637846  6794 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.807029  5949 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:00.686884  7039 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:00.760725 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:00.566951  7115 consensus_queue.cc:1175] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5206fd0087404d2da964a5d985a04ff4" responder_term: 22 status { last_received { term: 18 index: 13 } last_committed_idx: 13 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [21.14-21.14]" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:24:00.816547 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:00.763104  7114 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:00.751413 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 94299.1us
I20260709 12:24:00.694497  6990 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.764689 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:00.733235 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:00.768888 32377 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:24:00.732673  6882 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:00.737542  7084 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:00.736532  7060 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:00.737035 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:00.787176  7117 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.192s	user 0.001s	sys 0.000s
I20260709 12:24:00.799149 31746 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:00.799558 31748 raft_consensus.cc:2410] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 23 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:24:00.741761 31673 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:24:00.801345 31822 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:00.738111 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:00.751389  6501 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.805011 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:00.795578  7049 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:00.767544  7042 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.847993 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:00.811594 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:00.825042 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 308061us
I20260709 12:24:00.830276 31744 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:00.849296 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:00.850653 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 112665us
I20260709 12:24:00.834283 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:00.854223 31752 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:00.836935 31747 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:00.768262  6960 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.841286 31748 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260709 12:24:00.861490 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:00.847249 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1108) took 1627 ms. Trace:
I20260709 12:24:00.847852 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:00.862311 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50852.7us
I20260709 12:24:00.812659 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:00.855512 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50575.7us
W20260709 12:24:00.864554 31792 leader_election.cc:336] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.743s (SENT)
I20260709 12:24:00.866206 31675 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:00.866940 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:00.763926  6736 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.870079 31744 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:00.863045 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:00.800912  7015 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.073s
W20260709 12:24:00.814747  7021 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.691s	user 0.016s	sys 0.006s
I20260709 12:24:00.829855  7084 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:00.877569 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:00.872427 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:00.873864 31822 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:00.906246 31676 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:00.872283 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:00.867214 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:00.805034  7091 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.056s
I20260709 12:24:00.887358 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:00.862195 32373 rpcz_store.cc:276] 0709 12:23:59.219751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.219945 (+   194us) service_pool.cc:224] Handling call
0709 12:23:59.220279 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:00.182974 (+962695us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.183026 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:00.183092 (+    66us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:00.183142 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.328287 (+145145us) spinlock_profiling.cc:243] Waited 8.45 ms on lock 0x7b1800378a60. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419
0709 12:24:00.846916 (+518629us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b54004312d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:00.846997 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.847176 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":106,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":29628,"spinlock_wait_cycles":2206756096}
I20260709 12:24:00.894126  7058 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.001s
I20260709 12:24:00.894410 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:00.914062 32378 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:00.896147  7015 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:24:00.899044  6941 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.851s	user 0.004s	sys 0.008s
I20260709 12:24:00.929255  6941 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:00.930343  6661 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:00.930806  6661 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:00.931504  6784 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:00.931757  6661 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.868412  7042 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.934296 31574 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 19 to 20, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 20 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:00.936003 31825 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:00.935935 31574 catalog_manager.cc:5697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 19 to 22, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 22 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:00.937561 31574 catalog_manager.cc:5697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 11 to 19, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 19 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:00.873689  6876 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:00.865147  6501 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.871942  6960 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.861914  7006 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.908968 32369 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:00.900444  7077 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:00.926311  7096 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:00.810849  7104 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.926673 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1252) took 1255 ms. Trace:
I20260709 12:24:00.927445 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:00.868112  7087 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.920795  7015 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:00.911137  7098 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.815394  7109 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.076s
I20260709 12:24:00.832358  7114 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.938563 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.940263 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:00.929055 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:00.960216 31748 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:00.960692 31825 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:00.940482 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:00.961416 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:00.921934 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:00.907388  7021 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.789s	user 0.017s	sys 0.006s
I20260709 12:24:00.940654 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:00.888448  7132 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:00.965615 31747 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:00.891710  6821 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:00.911927  7084 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:00.963446 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:00.978694 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:00.910164  6647 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.982685 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:00.958935  6643 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.523s	user 0.014s	sys 0.000s
I20260709 12:24:00.946339 32374 rpcz_store.cc:276] 0709 12:23:59.671139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.671811 (+   672us) service_pool.cc:224] Handling call
0709 12:23:59.689368 (+ 17557us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:23:59.709363 (+ 19995us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:23:59.709396 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:23:59.713019 (+  3623us) log.cc:844] Serialized 26 byte log entry
0709 12:24:00.204545 (+491526us) spinlock_profiling.cc:243] Waited 4.89 ms on lock 0x7b18003a6f60. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:24:00.449455 (+244910us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:00.829191 (+379736us) log.cc:844] Serialized 14 byte log entry
0709 12:24:00.831389 (+  2198us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.831887 (+   498us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:00.926329 (+ 94442us) raft_consensus.cc:1685] finished
0709 12:24:00.926449 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:00.926588 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20655,"prepare.queue_time_us":412892,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":36368,"raft.run_cpu_time_us":325,"raft.run_wall_time_us":58032,"spinlock_wait_cycles":11258112,"thread_start_us":585362,"threads_started":2,"wal-append.queue_time_us":757233}
I20260709 12:24:00.933308 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:00.963214  5949 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:00.939877  6736 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:00.964956 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.002265 31745 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:00.982684 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:00.894433  7029 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.943766  6960 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.972225  6923 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:00.942982  6501 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:00.941030  6876 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:00.953701 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:00.999476 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:00.943949  7076 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:00.985877 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.000559 31747 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:01.005846 31676 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:00.998929 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:00.944525 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:00.947705  7104 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:00.943289  7006 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:00.947034  7087 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:00.957460  7015 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.259518 31825 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:01.260527 32377 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:01.258508 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:00.922771  7131 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:01.069257 31744 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:24:01.000528 31821 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:01.125592 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:01.013090 31672 raft_consensus.cc:2410] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:24:00.958801  7070 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:00.955469  7096 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:01.010859 31826 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.011s	sys 0.018s
I20260709 12:24:01.188202 31752 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:01.264631 32374 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 15.
I20260709 12:24:01.015108 32378 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:01.262204 32369 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:01.265888 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 280060us
I20260709 12:24:01.078061 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:01.183593 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:01.185397  7083 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.042s	user 0.014s	sys 0.010s
W20260709 12:24:01.288060  7141 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.122s	user 0.000s	sys 0.001s
W20260709 12:24:01.194468  7075 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.153s	user 0.007s	sys 0.011s
W20260709 12:24:01.176967  7127 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 12:24:01.261283 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:01.263332  6923 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:01.288044  7015 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:01.289392 31826 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.826s	user 0.011s	sys 0.020s
I20260709 12:24:01.291144 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:01.263136  7027 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:01.319478 32377 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:01.263192  6876 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:01.332453 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:01.333514 31825 raft_consensus.cc:2368] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26.
I20260709 12:24:01.245298  7109 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.071s
I20260709 12:24:01.290161  7013 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.046s
I20260709 12:24:01.335160  7109 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:01.322482  7107 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:01.115643  6864 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.336113  7107 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.120471  6814 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:01.337218  6814 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:01.337824  6864 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:01.230515  7145 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.338109  6814 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:01.339247  6814 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:01.338950  7145 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.243850  6643 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:01.341961  6643 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.335152 32369 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:01.343279  7036 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:01.343847 32369 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:01.332495 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:01.344288  6643 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:01.345259 31827 raft_consensus.cc:2410] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:24:01.345439 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:01.346069  7136 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:24:01.322790  7027 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:01.347239  7027 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:01.348070 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:01.109478  7129 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.349187  7129 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.350812 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:01.351637  7129 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:01.334503 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:01.352739 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:01.354025 31677 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:24:01.145190  7111 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.208191 31574 catalog_manager.cc:5697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 15 to 21. New cstate: current_term: 21 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:01.292328 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.183933  6862 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:00.990264  6794 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:00.994867  7132 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:01.174180  7086 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.636s	user 0.013s	sys 0.002s
I20260709 12:24:01.365790 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:01.317624 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.263656  7131 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:01.196568  7001 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.555s	user 0.002s	sys 0.013s
I20260709 12:24:01.319726 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3366) took 1013 ms. Trace:
I20260709 12:24:01.289759  6933 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:01.305073  7091 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.058s
I20260709 12:24:01.386605 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:00.984431  6700 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:00.997543  7100 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:01.322183  6923 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:01.327620 32374 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:01.304728  7070 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:01.290323  7096 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.331525  7006 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:01.288339  7083 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.404765 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:01.397724  7014 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.718s	user 0.009s	sys 0.003s
I20260709 12:24:01.341477  7145 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:00.972589  6930 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.319s
I20260709 12:24:01.379209  6794 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:01.231499  7101 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:01.360637 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 68365.9us
I20260709 12:24:01.360852 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:00.984103  6784 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:01.368172 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50691.3us
I20260709 12:24:01.368716 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:00.987975  7021 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.336s
I20260709 12:24:01.370118  7086 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:01.371749 31820 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.572s	user 0.028s	sys 0.013s
W20260709 12:24:01.372565 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:01.455078 31820 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.680s	user 0.029s	sys 0.013s
I20260709 12:24:01.130350  7071 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.379768 31674 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:01.383350 31826 rpcz_store.cc:276] 0709 12:24:00.306507 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.306707 (+   200us) service_pool.cc:224] Handling call
0709 12:24:00.380359 (+ 73652us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.422334 (+ 41975us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.422376 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:00.424631 (+  2255us) log.cc:844] Serialized 26 byte log entry
0709 12:24:01.319345 (+894714us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:01.319398 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.319484 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:01.319530 (+    46us) raft_consensus.cc:1685] finished
0709 12:24:01.319539 (+     9us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.319673 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":855883,"threads_started":1,"wal-append.queue_time_us":461052}
I20260709 12:24:01.289565  7029 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.292919  7104 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:01.204406  7037 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:01.462816  7013 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:01.463266  7013 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:01.464243  7013 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:01.465277  7144 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:24:01.465672  7013 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:01.467929  7106 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.18, Last appended by leader: 18, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:01.365419  7133 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:24:01.297164  7075 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.470918  7115 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.471495  7115 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.472128  7075 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:01.473135  7042 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:01.473815 31819 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:01.407984  7074 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:01.409533  7014 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:01.393942 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:01.359531  6862 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:01.486251 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:01.487269  6862 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.488577  6862 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:01.490206  6986 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:01.364941  7036 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:01.486089 31748 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 12:24:01.401831 32370 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.007s	sys 0.040s
W20260709 12:24:01.385356  7041 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.566s	user 0.006s	sys 0.006s
W20260709 12:24:01.492296 32370 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.007s	sys 0.041s
I20260709 12:24:01.492480  7041 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.380851  7131 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:01.335561  7109 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:01.495100  7037 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260709 12:24:01.495540  7109 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.496405  7037 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:01.496822  7037 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:01.496935  7109 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:01.394239  7113 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.409201 32372 raft_consensus.cc:2368] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25.
I20260709 12:24:01.070191  7112 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:01.509469  7083 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:01.421221 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1184) took 1037 ms. Trace:
I20260709 12:24:01.454133  6930 consensus_queue.cc:1243] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 19.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260709 12:24:01.451737 31827 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:01.452189 31752 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 15.
I20260709 12:24:01.473729  7115 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:01.363524  7111 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.452832 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.402644  6700 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:01.392894  6876 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.461938  7076 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.393746 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:01.497774  7037 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 20.10, Last appended by leader: 10, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.486089 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:01.558293 31671 raft_consensus.cc:2410] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:24:01.483562 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3367) took 1176 ms. Trace:
I20260709 12:24:01.558655 31825 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:01.398766 31749 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 19: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:01.503103 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:01.558799 31819 rpcz_store.cc:276] 0709 12:24:00.306933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.307081 (+   148us) service_pool.cc:224] Handling call
0709 12:24:00.327672 (+ 20591us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:01.474309 (+1146637us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.483179 (+  8870us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540043d5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:01.483475 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2635366656}
I20260709 12:24:01.491822 31822 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:24:01.507578  7113 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.508296 31818 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:01.449841 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:01.511737 31753 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:01.514089 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.560384 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:01.561208 32369 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:01.511399 31670 rpcz_store.cc:276] 0709 12:24:00.383962 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.384187 (+   225us) service_pool.cc:224] Handling call
0709 12:24:00.384559 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.422916 (+ 38357us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.422951 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:00.423452 (+   501us) log.cc:844] Serialized 26 byte log entry
0709 12:24:00.560038 (+136586us) spinlock_profiling.cc:243] Waited 2.18 ms on lock 0x7b18003cbfe0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:24:00.755798 (+195760us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:00.755839 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.755903 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:01.412733 (+656830us) raft_consensus.cc:1685] finished
0709 12:24:01.412759 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.421105 (+  8346us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":75912,"raft.queue_time_us":31607,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":7718,"spinlock_wait_cycles":5007744,"thread_start_us":195544,"threads_started":1,"wal-append.queue_time_us":498381}
I20260709 12:24:01.561530 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:01.369031 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:01.479981  7074 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:01.562568 31747 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:01.562855 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:01.554159 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 101411us
I20260709 12:24:01.564020 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.554874 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:01.555831 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.522529 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:01.558030 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.559511 31826 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:01.563040  7145 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:01.570108 32378 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:01.570165 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 17 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:01.516741 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:01.480991 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:01.561820 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:01.612509  7037 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:01.617211 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:01.419168  7077 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.091s
I20260709 12:24:01.511128  7132 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.560582 31746 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:01.567401 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:01.621910 31749 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:01.480687  6736 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.073s
W20260709 12:24:01.607473 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 89949.1us
I20260709 12:24:01.611377 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.453639  7101 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:01.606321  7140 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:24:01.626077 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:01.621912 31827 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:24:01.625249 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:01.627718  7115 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:01.628811  7115 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:01.566599  7001 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Leader pre-election won for term 16
I20260709 12:24:01.630462 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:01.630455 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:01.612161 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48235.9us
I20260709 12:24:01.632059 31827 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:01.632124 31819 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:01.633468 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:01.622228 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 141329us
I20260709 12:24:01.634413 31827 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:24:01.634793 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.630501 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.615485 31670 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 21 because replica has last-logged OpId of term: 20 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 11.
I20260709 12:24:01.638598 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.624316 31750 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:01.610761  6864 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:01.512209  7084 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.639772 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:01.629176  7115 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:01.640218 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.557332  7083 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.063s
W20260709 12:24:01.648046 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.650179 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.652081 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:01.653539 31746 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:01.654726 31825 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:01.656546 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:01.657470  7074 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.658718  7074 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:01.632977  7001 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:01.659294  7001 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:01.661624  7086 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:01.663688  6784 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:01.664765 31747 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:01.667083 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.633237  7077 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:01.670328 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:01.670948  7071 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:01.671829  7096 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:01.672768  7146 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.161s	user 0.001s	sys 0.000s
I20260709 12:24:01.674288  7087 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:01.683890  7101 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:01.637826  7014 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:01.684878  7014 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:01.685292  7014 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:01.687494 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:01.688529 31827 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:01.689457  6876 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:01.690667  6929 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:01.691150  7070 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:01.667806  6700 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.9, Last appended by leader: 10, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.693238 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:01.694296 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.695236  7021 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:01.648121  7027 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.697671  7027 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:01.633117  6736 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.699548  6736 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.665671  7111 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:01.695848  6828 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:01.704970 32377 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:24:01.705681 31819 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:24:01.669546 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:01.689435 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:01.695628 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.702309  6736 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:01.598186  7076 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.714696 31746 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:01.668365  7036 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:01.720120 31823 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:24:01.733418 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:01.670365  7077 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:01.734912 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:01.721159  7060 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.857s	user 0.008s	sys 0.004s
I20260709 12:24:01.735628 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.737838 31746 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 15.
I20260709 12:24:01.738708 31749 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:01.725697  7021 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:01.725063  6930 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.744617 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:01.746479 31678 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 14.
I20260709 12:24:01.733040  6828 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:01.748379 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:01.751526 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.751784 31678 raft_consensus.cc:2410] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 23 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:24:01.752533 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2455) took 1154 ms. Trace:
I20260709 12:24:01.753000 31678 rpcz_store.cc:276] 0709 12:24:00.598566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.748116 (+1149550us) service_pool.cc:224] Handling call
0709 12:24:01.752444 (+  4328us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:01.754026 31746 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:01.754441 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:01.755383 31678 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:01.759630 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:01.760449 31825 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:01.743455  7148 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
W20260709 12:24:01.775509  7151 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:24:01.741932  7060 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.784246 32373 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:01.785153  7060 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:01.788959 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:01.790861  7013 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:01.808264  7076 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:01.809888 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.812458 31748 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
W20260709 12:24:01.813102 32372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 1053) took 1890 ms (client timeout 1743 ms). Trace:
W20260709 12:24:01.813571 32372 rpcz_store.cc:269] 0709 12:23:59.923153 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:59.923664 (+   511us) service_pool.cc:224] Handling call
0709 12:24:01.812850 (+1889186us) spinlock_profiling.cc:243] Waited 1.45 s on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:01.813024 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3328097024}
I20260709 12:24:01.814877 32375 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:01.816406  7130 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.679s	user 0.012s	sys 0.000s
I20260709 12:24:01.818161 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.818665  6930 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.820052 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.820509 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:01.821768 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:01.822393 31819 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:24:01.790333 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:01.825671 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:01.826156 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:01.827301 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:01.822861 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2627) took 1221 ms. Trace:
I20260709 12:24:01.831000 31751 rpcz_store.cc:276] 0709 12:24:00.601471 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.601958 (+   487us) service_pool.cc:224] Handling call
0709 12:24:00.602250 (+   292us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.621409 (+ 19159us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:00.621444 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:00.642459 (+ 21015us) log.cc:844] Serialized 26 byte log entry
0709 12:24:00.745051 (+102592us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:00.745092 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:00.745166 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:01.815712 (+1070546us) spinlock_profiling.cc:243] Waited 34.3 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:01.822489 (+  6777us) raft_consensus.cc:1685] finished
0709 12:24:01.822514 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.822783 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18777,"raft.queue_time_us":35618,"raft.run_cpu_time_us":202,"raft.run_wall_time_us":27805,"spinlock_wait_cycles":80781440,"thread_start_us":101199,"threads_started":1,"wal-append.queue_time_us":466600}
I20260709 12:24:01.833724 31751 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:01.834903 31751 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:01.837416 31825 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:24:01.841740 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:01.842841 31827 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:24:01.853233 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 18 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:01.852818  7077 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.854852  7036 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:01.856171  7036 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:01.856587  7036 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:01.856993  7036 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:01.858894 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1325) took 1393 ms. Trace:
I20260709 12:24:01.859848 31746 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:01.859392 31819 rpcz_store.cc:276] 0709 12:24:00.468175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.854507 (+1386332us) service_pool.cc:224] Handling call
0709 12:24:01.858260 (+  3753us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:01.858469 (+   209us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:01.858501 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:01.858552 (+    51us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:01.858589 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.858688 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:01.858827 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:01.860774  6923 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.861162 31819 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:01.880203 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3374) took 1399 ms. Trace:
I20260709 12:24:01.880831 31819 rpcz_store.cc:276] 0709 12:24:00.480400 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.860668 (+1380268us) service_pool.cc:224] Handling call
0709 12:24:01.860962 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:01.879757 (+ 18795us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.880105 (+   348us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:01.829440 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:01.716029  7115 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:01.890997 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61601.9us
I20260709 12:24:01.892318 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.893308 31676 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:01.863823  7014 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.866290  7131 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.867631  7021 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.889883  7145 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:01.861174 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:01.852911 31751 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:01.910959  6736 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:01.911557  6736 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:01.911855 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.910961 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.912454  6736 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.913321 31746 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:01.914160 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.915196 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:01.915697 31746 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 15 FOLLOWER]: Advancing to term 16
I20260709 12:24:01.915970  7058 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:01.916365  7058 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:01.917160  7058 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 18.11, Last appended by leader: 11, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.918093  7058 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:01.919953 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:01.881556 31678 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:01.927623 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2463) took 1319 ms. Trace:
I20260709 12:24:01.928004 31678 rpcz_store.cc:276] 0709 12:24:00.608641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.754108 (+1145467us) service_pool.cc:224] Handling call
0709 12:24:01.927556 (+173448us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:01.928925 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:01.882874 31752 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:01.813887 31748 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:01.886397 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:01.887776 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60555.6us
I20260709 12:24:01.894027 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:01.939531 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45620.9us
I20260709 12:24:01.939759 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:01.896598 31827 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:01.897374 31745 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:01.882230  7074 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:01.889631  7109 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.903194  7115 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:01.880506  7076 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:01.878968  7015 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:01.904367  7084 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:01.879101  7100 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.030s
I20260709 12:24:01.890796  7013 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:01.920460  6941 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:01.897353  6814 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:01.920109  6929 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:01.912480 31751 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:24:01.822386  6930 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:01.935840 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:01.965967 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:01.937414 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 1326) took 1155 ms. Trace:
I20260709 12:24:01.966912 31821 rpcz_store.cc:276] 0709 12:24:00.781656 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.792958 (+ 11302us) service_pool.cc:224] Handling call
0709 12:24:01.937143 (+1144185us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b54003ca0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:01.937322 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":475065600}
I20260709 12:24:01.954402  7111 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:01.949880  7074 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:01.971077  7074 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:01.963653  7087 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.010s
I20260709 12:24:01.952052  7109 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.977502  7109 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:01.978253  7076 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:01.979408  7076 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:01.979863  7076 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:01.952356  7013 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:01.980849  7013 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:01.984714  7112 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:01.980875  7076 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:01.948936 31748 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:01.998807 31748 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:24:01.957243  7027 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:24:02.000222 31748 raft_consensus.cc:1217] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Deduplicated request from leader. Original: 15.8->[16.9-18.11]   Dedup: 17.10->[18.11-18.11]
I20260709 12:24:02.000717 31748 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [16.9-18.11]
I20260709 12:24:02.000782  7027 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:24:01.946998 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 70761.4us
I20260709 12:24:02.001170  7027 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:01.963889  6814 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.003576  7076 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:02.004926  6814 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:01.964157  6941 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:02.005980 31818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 1319) took 1593 ms (client timeout 1788 ms). Trace:
I20260709 12:24:02.006351  6941 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:02.006384 31818 rpcz_store.cc:269] 0709 12:24:00.412323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.546513 (+134190us) service_pool.cc:224] Handling call
0709 12:24:02.005720 (+1459207us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x7b540030edd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:02.005880 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2186425600}
I20260709 12:24:01.950621 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:01.992735  7113 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:01.996780  7112 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:01.919420  7001 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:02.016463 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:01.950204  7075 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:01.941007 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3372) took 1498 ms. Trace:
I20260709 12:24:02.033490 31827 rpcz_store.cc:276] 0709 12:24:00.442197 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.843878 (+1401681us) service_pool.cc:224] Handling call
0709 12:24:01.896355 (+ 52477us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:01.940702 (+ 44347us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.940936 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:02.029991 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 79427.5us
I20260709 12:24:02.002017  7027 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.035195 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:02.031800 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:02.033597 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 15 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:02.038137 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:02.041023 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:02.022605  7112 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:02.043144 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:02.053668 31818 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:02.022281  7084 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:01.953585  7015 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.051260 32369 raft_consensus.cc:2368] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16.
I20260709 12:24:02.037241  6929 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:02.033430  6930 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:02.068078 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:02.090520 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.092015 31677 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:02.048969 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:02.055346  7042 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:02.056535  7014 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:02.057318  7131 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:02.057711 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:02.103879 31827 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:24:02.069474 31673 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:02.072652 31824 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
W20260709 12:24:02.094605 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45724.5us
I20260709 12:24:02.095598 31746 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:24:02.045109  7077 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:02.109313 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:02.110603 31673 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:02.112525 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:02.108166 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:02.114262 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3380) took 1223 ms. Trace:
I20260709 12:24:02.126368 31824 rpcz_store.cc:276] 0709 12:24:00.891080 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.068772 (+1177692us) service_pool.cc:224] Handling call
0709 12:24:02.072391 (+  3619us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.113855 (+ 41464us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.114179 (+   324us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.128453 31824 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:02.117905  6859 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.011s
I20260709 12:24:02.046244  7075 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:02.093948  7042 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:02.130892  7042 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:02.114241  7037 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:02.131696  7037 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.130967  7075 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:02.132105  7037 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:02.133846 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3381) took 1242 ms. Trace:
I20260709 12:24:02.134298 31824 rpcz_store.cc:276] 0709 12:24:00.891396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.127797 (+1236401us) service_pool.cc:224] Handling call
0709 12:24:02.128212 (+   415us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.133519 (+  5307us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.133769 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:02.136220  7153 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.513s	user 0.003s	sys 0.000s
I20260709 12:24:02.136786 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3382) took 1245 ms. Trace:
I20260709 12:24:02.137100  6828 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:02.137665  6828 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:02.137253 31824 rpcz_store.cc:276] 0709 12:24:00.891747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.135779 (+1244032us) service_pool.cc:224] Handling call
0709 12:24:02.136175 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.136391 (+   216us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.136420 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:02.136473 (+    53us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:02.136511 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.136572 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.136717 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.112967 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:02.041221  7109 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.065s
I20260709 12:24:02.138770  6828 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:02.140152  7109 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:02.140360  6828 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:02.140652  7109 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:02.141819  7109 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.103742  7113 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:02.122462  6736 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 21.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.022s
I20260709 12:24:02.122890 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:02.153712 31746 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
W20260709 12:24:02.124336 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.106932  7027 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:02.174254  7060 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:02.175266  7060 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:02.176540  7060 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:02.150336 31818 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:02.145522 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:02.179795 31818 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:02.180501 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3237) took 1227 ms. Trace:
I20260709 12:24:02.180712 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:02.180821 31818 rpcz_store.cc:276] 0709 12:24:00.953105 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.179247 (+1226142us) service_pool.cc:224] Handling call
0709 12:24:02.179594 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.180218 (+   624us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.180424 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.181929  7083 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:02.182396  7083 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.182821  7083 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:02.183468  7109 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:02.187071  7048 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:02.187587  7048 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.117015  7068 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:02.188099  7048 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:02.188424  7068 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:02.189471  7068 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.193522 31824 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:24:02.202795 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3242) took 1183 ms. Trace:
I20260709 12:24:02.203583 31818 rpcz_store.cc:276] 0709 12:24:00.999236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.181829 (+1182593us) service_pool.cc:224] Handling call
0709 12:24:02.182139 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.182336 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.182364 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:02.182409 (+    45us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:02.182441 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.182493 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.182601 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.187454 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:02.088073  7084 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:02.097249  6930 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.221730  6859 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:02.222049  6736 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:02.224671  7068 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:02.237668  7015 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:02.241204 31821 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 20 because replica has last-logged OpId of term: 18 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 9.
I20260709 12:24:02.243420  7013 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:02.245747  7084 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:02.248605 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1321) took 1823 ms. Trace:
I20260709 12:24:02.162076  7112 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.249040 31825 rpcz_store.cc:276] 0709 12:24:00.427881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.838313 (+1410432us) service_pool.cc:224] Handling call
0709 12:24:01.838702 (+   389us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:01.890842 (+ 52140us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:01.890894 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:01.935321 (+ 44427us) log.cc:844] Serialized 26 byte log entry
0709 12:24:02.114697 (+179376us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:02.114745 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.114814 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.248362 (+133548us) raft_consensus.cc:1685] finished
0709 12:24:02.248383 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.248533 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":43164,"raft.queue_time_us":54306,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":15661,"thread_start_us":157686,"threads_started":1,"wal-append.queue_time_us":277500}
I20260709 12:24:02.251605  6930 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:02.232774  7036 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.231508  6923 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:02.237826  7001 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:02.255770  7112 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:02.265069  7042 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.254422  7036 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:02.273857 32373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 1113) took 1380 ms (client timeout 1612 ms). Trace:
I20260709 12:24:02.275466  7037 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:02.274423 32373 rpcz_store.cc:269] 0709 12:24:00.893244 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.927135 (+ 33891us) service_pool.cc:224] Handling call
0709 12:24:02.273606 (+1346471us) spinlock_profiling.cc:243] Waited 837 ms on lock 0x7b54005947d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:02.273754 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1925318016}
I20260709 12:24:02.276928  7048 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:02.278118 31678 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:02.234396  6859 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.287292  6859 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:02.290166 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:02.291363 31678 raft_consensus.cc:2410] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:24:02.291885 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1194) took 1339 ms. Trace:
I20260709 12:24:02.292271 31678 rpcz_store.cc:276] 0709 12:24:00.952404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.289871 (+1337467us) service_pool.cc:224] Handling call
0709 12:24:02.291820 (+  1949us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.234618  6736 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:02.294116 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:02.282155  7156 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.157s	user 0.000s	sys 0.000s
I20260709 12:24:02.279770  7013 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:02.303157 31678 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:02.296409 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:02.294263  6736 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:02.316293 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 160588us
I20260709 12:24:02.314960 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3377) took 1425 ms. Trace:
I20260709 12:24:02.315527  7042 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:02.314330 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:02.324661 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:02.315651 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:02.326715 31671 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:02.323310 31825 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:02.324425 31819 rpcz_store.cc:276] 0709 12:24:00.889641 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.882080 (+992439us) service_pool.cc:224] Handling call
0709 12:24:01.889264 (+  7184us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:01.897789 (+  8525us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:01.897823 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:01.898364 (+   541us) log.cc:844] Serialized 26 byte log entry
0709 12:24:02.219421 (+321057us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:02.219466 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.219546 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.314717 (+ 95171us) raft_consensus.cc:1685] finished
0709 12:24:02.314738 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.314897 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8009,"raft.queue_time_us":15534,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":122,"thread_start_us":240319,"threads_started":1,"wal-append.queue_time_us":331925}
I20260709 12:24:02.327472 31673 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:02.325230 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:02.328665 31678 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:02.329851 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:02.330588 31678 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 14 FOLLOWER]: Advancing to term 16
I20260709 12:24:02.333595  6986 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:02.340646  6986 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.343075  6986 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:02.351742 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:02.352535 31825 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 23: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:02.352324 31671 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:02.353717 31574 catalog_manager.cc:5697] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 15 to 20, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 20 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:02.355679  7048 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:02.356117 31574 catalog_manager.cc:5697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 15 to 18, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 18 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:24:02.357488 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.358610  5949 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:02.358332 31574 catalog_manager.cc:5697] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } }
I20260709 12:24:02.359026  5949 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:02.359172 31826 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:02.359458  5949 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:02.359912 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3386) took 1108 ms. Trace:
I20260709 12:24:02.360236 31826 rpcz_store.cc:276] 0709 12:24:01.251434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.340118 (+1088684us) service_pool.cc:224] Handling call
0709 12:24:02.358990 (+ 18872us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.359617 (+   627us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.359835 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.359721 31574 catalog_manager.cc:5697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 16 to 18, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 18 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:02.361313 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:02.362097 31826 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:02.363446 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:02.362109 31574 catalog_manager.cc:5697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 19 to 20, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 20 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:02.364835 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:02.365270 31574 catalog_manager.cc:5697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 8 to 15, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 15 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:24:02.366401 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 734061us
I20260709 12:24:02.367242 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:02.368367 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:02.368727 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:02.369138 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.369534 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:02.375871 31819 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: [22.14-22.14]
I20260709 12:24:02.376585 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3387) took 1124 ms. Trace:
I20260709 12:24:02.376902 31819 rpcz_store.cc:276] 0709 12:24:01.251876 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.342173 (+1090297us) service_pool.cc:224] Handling call
0709 12:24:02.375646 (+ 33473us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.376308 (+   662us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.376527 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.380508  7041 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:02.380384  7083 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:02.382177 32370 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:02.341696  6859 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:02.383919  7049 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:02.384316  7049 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:02.385293  7049 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.386363  7049 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:02.387045  7029 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:02.374419  7027 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:02.388283 31823 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 1324) took 1933 ms (client timeout 1951 ms). Trace:
I20260709 12:24:02.389231 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1264) took 1523 ms. Trace:
I20260709 12:24:02.408236 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:02.408458 32378 rpcz_store.cc:276] 0709 12:24:00.865756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.305236 (+439480us) service_pool.cc:224] Handling call
0709 12:24:01.305609 (+   373us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:01.570660 (+265051us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.388883 (+818223us) spinlock_profiling.cc:243] Waited 264 ms on lock 0x7b540031d654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:02.389086 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":607751040}
W20260709 12:24:02.390913 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 282856us
I20260709 12:24:02.410032 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:02.412137 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:02.391057  6700 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:02.373344  7112 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.394397 31751 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:02.372428 31826 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:02.395040  6923 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.416700 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:02.417220 31819 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:02.397040  6794 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:02.398168 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:02.372357  6661 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:02.400744 31568 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42070 (request call id 225) took 65533 ms (client timeout 15000 ms). Trace:
I20260709 12:24:02.400882  7037 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:02.402191 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.387068 32373 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:02.347119  6736 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:02.403980 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:02.403793 31574 catalog_manager.cc:5697] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 17 to 20. New cstate: current_term: 20 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
W20260709 12:24:02.405793 32343 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.951s (SENT) [suppressed 7 similar messages]
I20260709 12:24:02.436656 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:02.436990 31678 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:24:02.415695 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:02.395750 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:02.437889 31826 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:02.439028 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:02.422716  7095 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:02.420194  5949 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.466526  7095 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.424008 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:02.467167  7095 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:02.421267  6700 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:02.471527 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.472376  7106 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:02.416384  7027 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:02.478130 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.482532 31670 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:24:02.427470 31669 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:02.427603 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:02.424783 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:02.428809 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:02.429423 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:02.483702 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:02.484488 32374 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:02.484731 31670 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:24:02.430104 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:02.485503 31748 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 23
W20260709 12:24:02.431160 31672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1188) took 1954 ms (client timeout 2320 ms). Trace:
W20260709 12:24:02.431160 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.437302 31819 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:02.407720 31823 rpcz_store.cc:269] 0709 12:24:00.455006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.552348 (+ 97342us) service_pool.cc:224] Handling call
0709 12:24:02.388092 (+1835744us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b54000fa854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:02.388217 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2683076352}
I20260709 12:24:02.419330  7112 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:02.387072  6859 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:02.418134 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:02.421504  6923 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.485356 31826 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
W20260709 12:24:02.511297 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:02.645694 31669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2449) took 2050 ms (client timeout 1690 ms). Trace:
I20260709 12:24:02.638458 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2469) took 1178 ms. Trace:
I20260709 12:24:02.642119 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1114) took 1737 ms. Trace:
W20260709 12:24:02.653225 31669 rpcz_store.cc:269] 0709 12:24:00.595368 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.771917 (+176549us) service_pool.cc:224] Handling call
0709 12:24:02.645401 (+1873484us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54002b0a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:02.645594 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3596708736}
I20260709 12:24:02.654369  7083 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:02.655151 31669 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 20 index: 12. (index mismatch)
I20260709 12:24:02.655779 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2451) took 2059 ms. Trace:
I20260709 12:24:02.655700  6700 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.656256 31669 rpcz_store.cc:276] 0709 12:24:00.596708 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.654450 (+2057742us) service_pool.cc:224] Handling call
0709 12:24:02.654813 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.655527 (+   714us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.655720 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.657626  7131 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:02.653559 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:02.659245 31826 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:02.659391  7114 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:02.659859 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:02.660173 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:02.487834 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 63123.5us
I20260709 12:24:02.661173 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:02.661659 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:02.508060 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:02.575461 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.663130 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2473) took 1201 ms. Trace:
W20260709 12:24:02.663316 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.663527 31673 rpcz_store.cc:276] 0709 12:24:01.461212 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.507784 (+1046572us) service_pool.cc:224] Handling call
0709 12:24:02.663063 (+155279us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.509501 31818 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 11. Preceding OpId from leader: term: 19 index: 11. (term mismatch)
W20260709 12:24:02.664501 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3379) took 1774 ms (client timeout 1751 ms). Trace:
I20260709 12:24:02.664736 31818 pending_rounds.cc:85] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 10
I20260709 12:24:02.665143 31818 pending_rounds.cc:107] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 18.11
I20260709 12:24:02.534251 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3320) took 1049 ms. Trace:
W20260709 12:24:02.651669 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 69208.1us
I20260709 12:24:02.665709 31818 raft_consensus.cc:2889] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:02.487133 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58416.1us
I20260709 12:24:02.508296 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3234) took 1605 ms. Trace:
I20260709 12:24:02.485312 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2472) took 1024 ms. Trace:
I20260709 12:24:02.667955 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:02.668812 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.621443 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:02.621354 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1055) took 2184 ms. Trace:
I20260709 12:24:02.669000 32370 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:02.667558 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:02.533419 31674 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2457) took 1930 ms (client timeout 1619 ms). Trace:
W20260709 12:24:02.667953 31822 rpcz_store.cc:269] 0709 12:24:00.890318 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.983680 (+ 93362us) service_pool.cc:224] Handling call
0709 12:24:02.664318 (+1680638us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b54001c89d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:02.664440 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":463899008}
W20260709 12:24:02.669984 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:02.671178 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:02.507172 31753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2630) took 1833 ms (client timeout 1791 ms). Trace:
I20260709 12:24:02.670863 32375 rpcz_store.cc:276] 0709 12:24:00.437088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.437226 (+   138us) service_pool.cc:224] Handling call
0709 12:24:01.814692 (+1377466us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:01.815318 (+   626us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.621044 (+805726us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b5400593650. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:02.621268 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3167099136}
W20260709 12:24:02.581198 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 142249us
W20260709 12:24:02.664117 31672 rpcz_store.cc:269] 0709 12:24:00.469720 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.548313 (+ 78593us) service_pool.cc:224] Handling call
0709 12:24:02.424057 (+1875744us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b5400181ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:02.424201 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1065655168}
I20260709 12:24:02.665820 31752 rpcz_store.cc:276] 0709 12:24:01.484483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.561285 (+ 76802us) service_pool.cc:224] Handling call
0709 12:24:02.534043 (+972758us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b54002945d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:02.534176 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":710652672}
I20260709 12:24:02.674134 31672 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:02.535655 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3318) took 1059 ms. Trace:
I20260709 12:24:02.583211 31671 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:02.674846 31745 rpcz_store.cc:276] 0709 12:24:01.475885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.559139 (+ 83254us) service_pool.cc:224] Handling call
0709 12:24:02.535397 (+976258us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b54002959d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:02.535578 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":764782336}
I20260709 12:24:02.667750 31824 rpcz_store.cc:276] 0709 12:24:00.903395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.146230 (+1242835us) service_pool.cc:224] Handling call
0709 12:24:02.190874 (+ 44644us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.199231 (+  8357us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.508101 (+308870us) spinlock_profiling.cc:243] Waited 44.1 ms on lock 0x7b540043eed0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:02.508229 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":101397376}
I20260709 12:24:02.501454 31746 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:24:02.507916  5949 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:02.674592 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 256546us
I20260709 12:24:02.677906 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:02.496016 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:02.671255 31674 rpcz_store.cc:269] 0709 12:24:00.602806 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.736888 (+134082us) service_pool.cc:224] Handling call
0709 12:24:02.533175 (+1796287us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b540036f1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:02.533310 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1238909696}
I20260709 12:24:02.667943 31670 rpcz_store.cc:276] 0709 12:24:01.461006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.483399 (+1022393us) service_pool.cc:224] Handling call
0709 12:24:02.485252 (+  1853us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.680028 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1191) took 1876 ms. Trace:
I20260709 12:24:02.492892 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3383) took 1600 ms. Trace:
I20260709 12:24:02.681380 31827 rpcz_store.cc:276] 0709 12:24:00.892051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.143844 (+1251793us) service_pool.cc:224] Handling call
0709 12:24:02.214024 (+ 70180us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.233719 (+ 19695us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.233758 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:02.233807 (+    49us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:02.233841 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.492589 (+258748us) spinlock_profiling.cc:243] Waited 19.5 ms on lock 0x7b54000fc154. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:02.492664 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.492811 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":44772096}
I20260709 12:24:02.653957 32377 rpcz_store.cc:276] 0709 12:24:00.905114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.323342 (+418228us) service_pool.cc:224] Handling call
0709 12:24:01.323980 (+   638us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:01.705603 (+381623us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.641857 (+936254us) spinlock_profiling.cc:243] Waited 381 ms on lock 0x7b54006473d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:02.642056 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":875327744}
I20260709 12:24:02.653575 31676 rpcz_store.cc:276] 0709 12:24:01.460334 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.616946 (+156612us) service_pool.cc:224] Handling call
0709 12:24:02.638283 (+1021337us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x7b54002e6b54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:02.638394 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":284352896}
I20260709 12:24:02.528576 31574 catalog_manager.cc:5697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 18 to 21, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 21 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:02.679040 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:02.662041 31823 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
W20260709 12:24:02.635200  7152 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.072s	user 0.007s	sys 0.000s
I20260709 12:24:02.684895 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:02.634608  7164 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:24:02.702454 31718 leader_election.cc:400] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26.
I20260709 12:24:02.721062 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:02.709889 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:02.718327 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:02.721062 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 19 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:02.721974 31718 leader_election.cc:403] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Cancelling election due to peer responding with higher term
I20260709 12:24:02.722678 31822 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:02.686650 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1192) took 1737 ms. Trace:
I20260709 12:24:02.633104  7115 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.686303 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:02.655812  7107 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:02.750033 31677 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:24:02.506757  6752 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:02.731139 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:02.680650 31672 rpcz_store.cc:276] 0709 12:24:00.803359 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.673471 (+1870112us) service_pool.cc:224] Handling call
0709 12:24:02.673949 (+   478us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.679759 (+  5810us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.679971 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.725930 31826 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:24:02.756332 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3392) took 1273 ms. Trace:
I20260709 12:24:02.749186 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:02.757011 32376 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:02.487304  7039 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:02.768239 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:02.684919 31753 rpcz_store.cc:269] 0709 12:24:00.673811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.679597 (+  5786us) service_pool.cc:224] Handling call
0709 12:24:02.506894 (+1827297us) spinlock_profiling.cc:243] Waited 687 ms on lock 0x7b54002968d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:02.507085 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1580000000}
I20260709 12:24:02.751665 31819 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:02.509117  7165 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:02.748049 31824 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:02.727043 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59750.8us
I20260709 12:24:02.723848 32370 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 20: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:02.709987 32377 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 16: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:24:02.740954 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 79367.8us
I20260709 12:24:02.752965 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:02.727335  5949 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election lost for term 25. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26.
I20260709 12:24:02.756808 31826 rpcz_store.cc:276] 0709 12:24:01.482525 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.681124 (+1198599us) service_pool.cc:224] Handling call
0709 12:24:02.725609 (+ 44485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.756087 (+ 30478us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.756278 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.756508 32374 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:24:02.757335 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:02.758069 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:02.749476 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:02.690937 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:02.686335 31574 catalog_manager.cc:5697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 3 to 21, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 21 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:02.723445 31674 rpcz_store.cc:276] 0709 12:24:00.949098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.685794 (+1736696us) service_pool.cc:224] Handling call
0709 12:24:02.686127 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.686291 (+   164us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.686318 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:02.686362 (+    44us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:02.686405 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.686468 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.686577 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.611034  6858 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:02.750658 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:02.686076  6794 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:02.717934  7145 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:24:02.799813 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:02.782702 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:02.816207 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 269165us
I20260709 12:24:02.832969 32378 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:02.601068  7068 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:02.833755 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:02.835047  7083 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.878912 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:02.877593 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 16 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:02.856031  7152 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.092s	user 0.001s	sys 0.000s
I20260709 12:24:02.618644  6986 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:02.840204 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 87297.3us
I20260709 12:24:02.900347 31643 leader_election.cc:400] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16.
I20260709 12:24:02.900861 31643 leader_election.cc:403] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 15 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:02.852895 32376 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:02.685285  6651 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:02.906594  6651 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:02.854395  7076 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:02.718197  6933 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:02.907562 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:02.908776  6933 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:02.910367  6909 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:02.863126 31751 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:02.833459 31678 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:24:02.767180  7109 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.128s
I20260709 12:24:02.859671 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:02.873523 31672 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:24:02.528381  7060 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:02.881417 31748 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:02.894765  7107 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:02.749738  7021 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:02.897588 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:02.830385  7039 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:02.899821  7068 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:02.856969  7014 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:02.902683 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3315) took 1440 ms. Trace:
I20260709 12:24:02.903308  7145 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:02.908345  7076 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.903650  6661 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 15. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 15. Current term is 16.
I20260709 12:24:02.904186  6647 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:02.905501 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:24:02.968454 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63053.6us
I20260709 12:24:02.969587 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:02.857692 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:02.970990 31674 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:02.971156  7013 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:02.971769  7013 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:02.972221  7013 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:02.972940 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2462) took 2364 ms. Trace:
I20260709 12:24:02.975821  7109 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:02.867416  6794 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:02.976889  6794 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:02.913962 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3418) took 1879 ms. Trace:
I20260709 12:24:02.978960 31678 rpcz_store.cc:276] 0709 12:24:01.034670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.760612 (+1725942us) service_pool.cc:224] Handling call
0709 12:24:02.833210 (+ 72598us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.913638 (+ 80428us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.913873 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:02.980100 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:02.981110 31678 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:02.720633  6941 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:02.754783  7115 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:02.836102 31574 catalog_manager.cc:5697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 12 to 14, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 14 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:02.931648 31821 raft_consensus.cc:2368] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 19. Current term is 20.
I20260709 12:24:02.900295  7048 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:02.912595 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:02.934800 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:02.902922  7165 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader election won for term 16
I20260709 12:24:02.938376 31749 rpcz_store.cc:276] 0709 12:24:01.462169 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.740378 (+278209us) service_pool.cc:224] Handling call
0709 12:24:01.740726 (+   348us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.395636 (+654910us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.395669 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:02.396220 (+   551us) log.cc:844] Serialized 26 byte log entry
0709 12:24:02.619667 (+223447us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:02.619735 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.619834 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.902447 (+282613us) raft_consensus.cc:1685] finished
0709 12:24:02.902468 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.902602 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":649958,"raft.queue_time_us":650974,"raft.run_cpu_time_us":72,"raft.run_wall_time_us":72,"thread_start_us":222974,"threads_started":1,"wal-append.queue_time_us":442319}
I20260709 12:24:02.859162  6858 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:02.895627  7083 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:02.935215  7149 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:02.984931  7107 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:03.047384 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3417) took 2057 ms. Trace:
W20260709 12:24:03.047981 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.048391 31674 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:24:02.943264  7036 raft_consensus.cc:2764] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:24:02.982331 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 16 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:02.974665 31669 rpcz_store.cc:276] 0709 12:24:00.608273 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.657325 (+2049052us) service_pool.cc:224] Handling call
0709 12:24:02.657675 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.657894 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:02.658204 (+   310us) log.cc:844] Serialized 14 byte log entry
0709 12:24:02.971304 (+313100us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:02.971620 (+   316us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:02.971766 (+   146us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.972019 (+   253us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:02.972522 (+   503us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19252,"prepare.queue_time_us":2147,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":25063,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"thread_start_us":292951,"threads_started":1}
W20260709 12:24:03.064333 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:03.052930 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 118247us
I20260709 12:24:02.969058  7076 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:03.066679 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:03.071696 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:02.985723 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:03.056883 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 144351us
I20260709 12:24:03.073834 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:03.073835  7041 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.003s
I20260709 12:24:03.077521  7041 consensus_queue.cc:1243] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 19.11 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 10
I20260709 12:24:03.061159  7145 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.083493  7107 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:03.086894  7039 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.090927  7068 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.092273  6941 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:24:03.094720  7169 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.179s	user 0.002s	sys 0.000s
I20260709 12:24:03.097363 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3313) took 1635 ms. Trace:
I20260709 12:24:03.051290  6933 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:03.097756 31747 rpcz_store.cc:276] 0709 12:24:01.461650 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:01.665596 (+203946us) service_pool.cc:224] Handling call
0709 12:24:01.665893 (+   297us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.363185 (+697292us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.363219 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:02.363750 (+   531us) log.cc:844] Serialized 26 byte log entry
0709 12:24:02.527457 (+163707us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:02.527519 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.527623 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:03.094130 (+566507us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:03.097063 (+  2933us) raft_consensus.cc:1685] finished
0709 12:24:03.097089 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:03.097302 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":696894,"raft.queue_time_us":698168,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"spinlock_wait_cycles":4764416,"thread_start_us":163217,"threads_started":1,"wal-append.queue_time_us":465688}
I20260709 12:24:03.100638 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:02.933972  7060 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.102063 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:02.986173  7014 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:03.055395 31574 catalog_manager.cc:5697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 14 to 17, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 17 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:24:03.057677 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 88022.8us
I20260709 12:24:03.059413  7165 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:03.058365 31676 rpcz_store.cc:276] 0709 12:24:00.989553 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.704165 (+1714612us) service_pool.cc:224] Handling call
0709 12:24:02.751487 (+ 47322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.751676 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:02.819712 (+ 68036us) log.cc:844] Serialized 14 byte log entry
0709 12:24:03.046891 (+227179us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:03.046970 (+    79us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:03.047030 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.047124 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:03.047302 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":77386,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":58184,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":2003,"thread_start_us":168571,"threads_started":1}
I20260709 12:24:03.060873  6862 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:24:03.058151 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 300121us
I20260709 12:24:02.907931  6651 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.107604  7114 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:03.118623  7114 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:03.119033 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1190) took 2465 ms. Trace:
I20260709 12:24:03.119653 31673 rpcz_store.cc:276] 0709 12:24:00.653356 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.664433 (+2011077us) service_pool.cc:224] Handling call
0709 12:24:02.664751 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.685342 (+ 20591us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.685379 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:02.754792 (+ 69413us) log.cc:844] Serialized 26 byte log entry
0709 12:24:03.026302 (+271510us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:03.026367 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.026471 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:03.118721 (+ 92250us) raft_consensus.cc:1685] finished
0709 12:24:03.118751 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:03.118951 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2964,"raft.queue_time_us":37224,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":29643,"thread_start_us":144839,"threads_started":1,"wal-append.queue_time_us":207609}
I20260709 12:24:03.121511 31675 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:03.116981 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:03.107246  7036 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:03.107360 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:03.058516  7149 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:03.056947  7021 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:03.124516 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:03.072862  7027 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:03.147075 31574 catalog_manager.cc:5697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 12 to 15, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 15 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:03.125680 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:03.126068 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:03.136422 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:03.064385 31678 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 16: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:03.149842 31674 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:03.151346 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:03.151671  5949 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:03.151968 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:03.151997 31669 raft_consensus.cc:2376] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26. Re-sending same reply.
I20260709 12:24:03.114398  7094 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:03.173323 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:03.133064  7142 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.513s	user 0.005s	sys 0.000s
I20260709 12:24:03.119339  6923 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:03.155560 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.126874  6647 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:03.116410  6862 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:03.119055  7114 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:03.144685  7048 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:03.075213  6909 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.077s
W20260709 12:24:03.184904 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.173586 31574 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 16 to 19, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 19 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:03.133790  7178 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:03.213846 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:03.197223 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.119446  7013 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:03.175477  7150 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:03.116355  7165 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 14.10, Last appended by leader: 10, Current term: 16, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.424312  7107 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:03.254794 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:03.427006 32374 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:03.426712  7107 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:03.425285 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.006s	sys 0.042s
W20260709 12:24:03.428987 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.561s	user 0.006s	sys 0.042s
I20260709 12:24:03.306234 31574 catalog_manager.cc:5697] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 2 to 21, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 21 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:03.172755  6661 raft_consensus.cc:2764] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election decision vote started in defunct term 19: won
I20260709 12:24:03.431754 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:03.442351 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3243) took 2350 ms. Trace:
I20260709 12:24:03.217846  7083 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:03.218685 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:03.194341  7094 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:03.219501 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:03.220825 31819 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:03.457276 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:03.221683 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:03.401531  6933 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.193598  6862 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:03.228672 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3414) took 2243 ms. Trace:
I20260709 12:24:03.152372  7021 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:03.237825 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:03.464680 31823 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:03.239552  7178 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:03.240655 32376 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:03.241544 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:03.283185  6909 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:03.228756  6700 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.014s
I20260709 12:24:03.247373 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:03.444645  7013 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:03.251617 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:03.253211 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:03.253749 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3246) took 1933 ms. Trace:
W20260709 12:24:03.425314 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 211589us
I20260709 12:24:03.477856 31825 rpcz_store.cc:276] 0709 12:24:01.320177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.367413 (+1047236us) service_pool.cc:224] Handling call
0709 12:24:02.383295 (+ 15882us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.415298 (+ 32003us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.415326 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:02.430186 (+ 14860us) log.cc:844] Serialized 26 byte log entry
0709 12:24:02.938523 (+508337us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:02.938568 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:02.938661 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:03.253458 (+314797us) raft_consensus.cc:1685] finished
0709 12:24:03.253488 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:03.253673 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37654,"raft.queue_time_us":34907,"raft.run_cpu_time_us":232,"raft.run_wall_time_us":16827,"thread_start_us":319335,"threads_started":1,"wal-append.queue_time_us":573891}
W20260709 12:24:03.423264 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 226396us
I20260709 12:24:03.299580 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:03.483354 31669 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:03.396601  6794 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:03.425794 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:03.497514 31672 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:03.467370  7048 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:03.437602 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:03.439620 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.443184 31818 rpcz_store.cc:276] 0709 12:24:01.092564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.205009 (+1112445us) service_pool.cc:224] Handling call
0709 12:24:02.493992 (+288983us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.666597 (+172605us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.442073 (+775476us) spinlock_profiling.cc:243] Waited 288 ms on lock 0x7b5400379e50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:03.442274 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":663324928}
I20260709 12:24:03.510293 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:03.473019  6909 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:03.446143 31574 catalog_manager.cc:5697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 15 to 20, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 20 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:03.450584 31827 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:03.218918  7188 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:03.451051 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:24:03.453013 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 306593us
I20260709 12:24:03.457401 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:03.458526 31822 raft_consensus.cc:2410] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 25 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:24:03.460608 31826 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 27: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:03.460278 31670 rpcz_store.cc:276] 0709 12:24:00.985692 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.717830 (+1732138us) service_pool.cc:224] Handling call
0709 12:24:02.759861 (+ 42031us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:02.760036 (+   175us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.760051 (+    15us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:02.823579 (+ 63528us) log.cc:844] Serialized 26 byte log entry
0709 12:24:03.133252 (+309673us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:03.133292 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.133361 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:03.228274 (+ 94913us) raft_consensus.cc:1685] finished
0709 12:24:03.228310 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:03.228567 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":231536,"threads_started":1,"wal-append.queue_time_us":345899}
I20260709 12:24:03.515901 31670 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: [20.10-20.10]
I20260709 12:24:03.516088 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:03.516654 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2471) took 2055 ms. Trace:
I20260709 12:24:03.516791 31672 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:03.517089 31670 rpcz_store.cc:276] 0709 12:24:01.460759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.515280 (+2054521us) service_pool.cc:224] Handling call
0709 12:24:03.515645 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.516343 (+   698us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.516593 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:03.276743  7174 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
W20260709 12:24:03.315272  7095 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.847s	user 0.012s	sys 0.000s
W20260709 12:24:03.262534  7106 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.790s	user 0.008s	sys 0.001s
I20260709 12:24:03.192924  7039 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:24:03.293413  7142 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.077s	user 0.000s	sys 0.000s
I20260709 12:24:03.340209  6923 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.470085 32377 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:03.241966  7112 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:03.268270  7060 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:03.472406 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:03.471336 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:03.667690 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.264833  6858 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:03.250957  6986 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:03.216100 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:03.436203 31824 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.580s	user 0.010s	sys 0.000s
I20260709 12:24:03.678057 31824 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:24:03.283445  6752 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.192s
I20260709 12:24:03.488150 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:03.469036  7083 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.467729  7145 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.465888  6933 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:03.519965 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:03.519323 31574 catalog_manager.cc:5697] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 10 to 19, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 19 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
W20260709 12:24:03.520144 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 88557.4us
I20260709 12:24:03.523295 31672 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:03.524869 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2448) took 2929 ms. Trace:
I20260709 12:24:03.526487 31669 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [17.10-18.11]
I20260709 12:24:03.526841 31670 pending_rounds.cc:85] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 14
I20260709 12:24:03.258855  7149 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.665247 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:03.474387 31674 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 26: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:03.521972  7027 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:03.669793 32370 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:03.672560 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:03.684696 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.810721 31826 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:03.513227  7068 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.162s
I20260709 12:24:03.678601 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 1338) took 1017 ms. Trace:
W20260709 12:24:03.683588 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:03.815433 31824 rpcz_store.cc:276] 0709 12:24:02.660820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.686040 (+ 25220us) service_pool.cc:224] Handling call
0709 12:24:03.678539 (+992499us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:03.803454 31677 rpcz_store.cc:276] 0709 12:24:00.594972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.576103 (+1981131us) service_pool.cc:224] Handling call
0709 12:24:02.576509 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:02.750438 (+173929us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.524604 (+774166us) spinlock_profiling.cc:243] Waited 171 ms on lock 0x7b54003921d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:03.524804 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":394397440}
W20260709 12:24:03.693243 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 205139us
I20260709 12:24:03.812610 31826 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:03.817621 31824 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 10. Preceding OpId from leader: term: 21 index: 10. (term mismatch)
I20260709 12:24:03.818153 31824 pending_rounds.cc:85] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 9
I20260709 12:24:03.818434 31824 pending_rounds.cc:107] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 20.10
I20260709 12:24:03.819015 31824 raft_consensus.cc:2889] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:03.819008 31826 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:03.674968  6647 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election lost for term 19. Reason: could not achieve majority
I20260709 12:24:03.809602 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2466) took 2350 ms. Trace:
I20260709 12:24:03.736980 32377 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:03.811909 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1197) took 1100 ms. Trace:
I20260709 12:24:03.825094 31674 rpcz_store.cc:276] 0709 12:24:02.711425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.246988 (+535563us) service_pool.cc:224] Handling call
0709 12:24:03.811845 (+564857us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:03.827201 31674 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:03.827961 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3421) took 1943 ms. Trace:
I20260709 12:24:03.563767  7131 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:03.512630  6700 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:03.810437 31670 pending_rounds.cc:107] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260709 12:24:03.808324 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2477) took 2084 ms. Trace:
I20260709 12:24:03.837221 31670 raft_consensus.cc:2889] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:03.837366 31672 rpcz_store.cc:276] 0709 12:24:01.723744 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.517978 (+1794234us) service_pool.cc:224] Handling call
0709 12:24:03.518487 (+   509us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:03.807945 (+289458us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.808242 (+   297us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":75008}
I20260709 12:24:03.819625 31574 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 17 to 20, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 20 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:03.665684  6923 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:03.821497 31818 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:03.738996  7037 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:03.738785 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 1340) took 1077 ms. Trace:
I20260709 12:24:03.674255  7106 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:03.737174 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:03.828349 31674 rpcz_store.cc:276] 0709 12:24:01.884151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.826443 (+1942292us) service_pool.cc:224] Handling call
0709 12:24:03.826928 (+   485us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:03.827687 (+   759us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.827889 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:03.843663 31674 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21. Ops: []
W20260709 12:24:03.838953 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 313506us
I20260709 12:24:03.844404 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3424) took 1959 ms. Trace:
I20260709 12:24:03.663326  7001 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.824880 31669 rpcz_store.cc:276] 0709 12:24:01.459575 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.492675 (+2033100us) service_pool.cc:224] Handling call
0709 12:24:03.526205 (+ 33530us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:03.809228 (+283023us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.809528 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:03.845029 32343 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 16 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:03.844790 31674 rpcz_store.cc:276] 0709 12:24:01.885009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.843106 (+1958097us) service_pool.cc:224] Handling call
0709 12:24:03.843472 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:03.844100 (+   628us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.844348 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:03.847443 31674 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:03.848457 31827 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:03.850839 31674 pending_rounds.cc:85] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 10
I20260709 12:24:03.851339 31674 pending_rounds.cc:107] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 19.11
I20260709 12:24:03.851691 31674 raft_consensus.cc:2889] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:03.563431  7039 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:03.740012 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:03.810122  7027 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:03.841982 31821 rpcz_store.cc:276] 0709 12:24:02.661353 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.669713 (+  8360us) service_pool.cc:224] Handling call
0709 12:24:03.675400 (+1005687us) spinlock_profiling.cc:243] Waited 209 ms on lock 0x7b54000fa854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:03.738682 (+ 63282us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":480273280}
I20260709 12:24:03.669337  7041 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.738199  7112 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:03.872614 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3362) took 3582 ms. Trace:
W20260709 12:24:03.872934  7191 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:24:03.873914  7112 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.874195 31673 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:24:03.804762  7021 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:03.874532  7112 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:03.838982  7131 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:03.665071  7094 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:03.806901  7083 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.839583 31574 catalog_manager.cc:5697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 17 to 19. New cstate: current_term: 19 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:03.674324  7178 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.879782 31574 catalog_manager.cc:5697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 8 to 17, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 17 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:03.815629  7145 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.854612 31819 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:03.882568  7041 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:03.883275 31675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2456) took 3284 ms (client timeout 2243 ms). Trace:
I20260709 12:24:03.553090  7076 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:03.843701 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 327659us
I20260709 12:24:03.563819  7095 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:03.814404  6986 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.884759  6986 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.885572  6941 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.853806  7001 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.888003 32371 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 23 because replica has last-logged OpId of term: 22 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 17.
I20260709 12:24:03.860325  7150 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.888868  7150 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:03.891409 31673 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:24:03.854243  6858 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.893558  6858 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.895097 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2481) took 1553 ms. Trace:
I20260709 12:24:03.870033 31677 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:03.895644 31671 rpcz_store.cc:276] 0709 12:24:02.341556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.417653 (+ 76097us) service_pool.cc:224] Handling call
0709 12:24:03.886454 (+1468801us) spinlock_profiling.cc:243] Waited 98.9 ms on lock 0x7b540017bbd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:03.895019 (+  8565us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":227507584}
I20260709 12:24:03.896178 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3422) took 2011 ms. Trace:
I20260709 12:24:03.896554 31677 rpcz_store.cc:276] 0709 12:24:01.884448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.840555 (+1956107us) service_pool.cc:224] Handling call
0709 12:24:03.869820 (+ 29265us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:03.895889 (+ 26069us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.896120 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:03.870886  7183 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.379s	user 0.001s	sys 0.000s
I20260709 12:24:03.899499 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:03.871261  7114 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.656s	user 0.006s	sys 0.006s
W20260709 12:24:03.871816  7192 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:24:03.872314 31669 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 22. Current term is 23. Ops: [22.18-22.18]
I20260709 12:24:03.912209 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1198) took 1188 ms. Trace:
I20260709 12:24:03.912573 31669 rpcz_store.cc:276] 0709 12:24:02.723742 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.845593 (+1121851us) service_pool.cc:224] Handling call
0709 12:24:03.872028 (+ 26435us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.911854 (+ 39826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.912131 (+   277us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:03.883802 31675 rpcz_store.cc:269] 0709 12:24:00.598868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.953413 (+354545us) service_pool.cc:224] Handling call
0709 12:24:03.883034 (+2929621us) spinlock_profiling.cc:243] Waited 1.85 s on lock 0x7b5400253fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:03.883165 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4261992832}
I20260709 12:24:03.886140  7076 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:03.902079 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:03.814153  6661 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.051s
I20260709 12:24:03.831100  7075 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:03.921775  7107 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:03.922186  7115 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:03.922353  7075 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:03.924832  7075 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:03.925288  6651 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:03.926250  6986 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:03.910229 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:02.426198 31568 rpcz_store.cc:269] 0709 12:22:56.867376 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:56.867692 (+   316us) service_pool.cc:224] Handling call
0709 12:22:56.948768 (+ 81076us) spinlock_profiling.cc:243] Waited 2.43 ms on lock 0x7b54002828d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3f090 00007fe8fcdddf3a 00007fe902b93e9c 00007fe902c9204e 00007fe9011532a4 00007fe901153232 00007fe901153161 00007fe9011530dc 00007fe901152392 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:02.400444 (+65451676us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:23:51.652846 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:23:51.653531 (+   685us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:23:51.653591 (+    60us) write_op.cc:435] Acquired schema lock
0709 12:23:51.653611 (+    20us) tablet.cc:662] Decoding operations
0709 12:23:51.654675 (+  1064us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:23:51.654772 (+    97us) write_op.cc:641] Partition lock acquired for write op
0709 12:23:51.654800 (+    28us) tablet.cc:685] Acquiring locks for 10 operations
0709 12:23:51.655765 (+   965us) tablet.cc:701] Row locks acquired
0709 12:23:51.655812 (+    47us) write_op.cc:260] PREPARE: finished
0709 12:23:51.655990 (+   178us) write_op.cc:270] Start()
0709 12:23:51.664122 (+  8132us) write_op.cc:276] Timestamp: P: 1783599831655926 usec, L: 0
0709 12:23:51.671337 (+  7215us) op_driver.cc:348] REPLICATION: starting
0709 12:23:51.864065 (+192728us) log.cc:844] Serialized 4039 byte log entry
0709 12:23:53.007136 (+1143071us) op_driver.cc:464] REPLICATION: finished
0709 12:23:53.225081 (+217945us) write_op.cc:301] APPLY: starting
0709 12:23:53.230320 (+  5239us) tablet.cc:1367] starting BulkCheckPresence
0709 12:23:53.426188 (+195868us) tablet.cc:1370] finished BulkCheckPresence
0709 12:23:53.426493 (+   305us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:23:53.684139 (+257646us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:23:53.684484 (+   345us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=20,key_file_lookups=20,delta_file_lookups=0,mrs_lookups=0
0709 12:23:53.684566 (+    82us) write_op.cc:312] APPLY: finished
0709 12:23:53.684988 (+   422us) log.cc:844] Serialized 97 byte log entry
0709 12:23:53.685197 (+   209us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:53.685337 (+   140us) write_op.cc:454] Released schema lock
0709 12:23:53.685410 (+    73us) write_op.cc:341] FINISH: Updating metrics
0709 12:23:53.685892 (+   482us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:23:53.685921 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":930671,"reported_tablets":50,"spinlock_wait_cycles":5594368,"child_traces":[["op",{"apply.queue_time_us":192739,"apply.run_cpu_time_us":17843,"apply.run_wall_time_us":475679,"cfile_cache_hit":42,"cfile_cache_hit_bytes":45907,"num_ops":10,"prepare.queue_time_us":353643,"prepare.run_cpu_time_us":46184,"prepare.run_wall_time_us":350143,"raft.queue_time_us":373748,"raft.run_cpu_time_us":31446,"raft.run_wall_time_us":125260,"replication_time_us":1335587,"thread_start_us":700814,"threads_started":4,"wal-append.queue_time_us":241149,"wal-append.run_cpu_time_us":32218,"wal-append.run_wall_time_us":2572220}]]}
I20260709 12:24:03.926800  6651 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:03.926005  7077 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:03.927778  7077 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:03.930078  7077 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:03.838001  7048 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.921057  7095 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:03.857725  7039 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:03.937750 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:03.873104 31820 rpcz_store.cc:276] 0709 12:24:00.290536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.290691 (+   155us) service_pool.cc:224] Handling call
0709 12:24:00.291098 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:00.391731 (+100633us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:00.741836 (+350105us) log.cc:844] Serialized 14 byte log entry
0709 12:24:01.480453 (+738617us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:01.480965 (+   512us) log.cc:844] Serialized 26 byte log entry
0709 12:24:01.481098 (+   133us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:01.481327 (+   229us) log.cc:844] Serialized 14 byte log entry
0709 12:24:01.481531 (+   204us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:01.481610 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:02.859812 (+1378202us) spinlock_profiling.cc:243] Waited 7.22 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:03.872254 (+1012442us) raft_consensus.cc:1685] finished
0709 12:24:03.872286 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:03.872512 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":100453,"prepare.queue_time_us":106337,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":88,"raft.queue_time_us":88748,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":15010,"spinlock_wait_cycles":16604032,"thread_start_us":969060,"threads_started":2,"wal-append.queue_time_us":759275}
I20260709 12:24:03.862056  7027 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:03.906729 31818 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 17.
W20260709 12:24:03.942165 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 97214.9us
I20260709 12:24:03.959156 32343 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:03.940623 32371 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 1109) took 3081 ms (client timeout 1968 ms). Trace:
W20260709 12:24:03.961486 32371 rpcz_store.cc:269] 0709 12:24:00.859154 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:00.872708 (+ 13554us) service_pool.cc:224] Handling call
0709 12:24:03.940344 (+3067636us) spinlock_profiling.cc:243] Waited 2.98 s on lock 0x7b5400544f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:03.940546 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6856647552}
I20260709 12:24:03.914390 31819 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
W20260709 12:24:03.970949 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 513572us
W20260709 12:24:04.011505 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 166487us
I20260709 12:24:04.015137 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:04.015887 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.017338 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:04.019883 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 82551.7us
W20260709 12:24:04.022812 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:04.030297 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.031721 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:04.036379 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:04.026585 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:03.843093  6700 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:04.063714 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.009773  7115 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:04.083827 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:04.084946 31677 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:24:04.085579 31671 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:24:04.075367 31673 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 9. Preceding OpId from leader: term: 19 index: 11. (index mismatch)
I20260709 12:24:03.953910  7095 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:04.042979 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:04.093024 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 134007us
W20260709 12:24:04.093647 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 50770.3us
I20260709 12:24:04.093906 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:03.959827  7106 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:04.019779  7060 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:04.089870 31669 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 22. Current term is 23. Ops: []
W20260709 12:24:04.089916 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 72679us
I20260709 12:24:04.091084 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:04.013732  7190 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader election lost for term 16. Reason: could not achieve majority
I20260709 12:24:03.932289  7114 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:03.902480  6941 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:04.097656 31673 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:24:04.064285  7075 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.037895  7077 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.178372 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:04.111148 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:04.106433 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:04.179879 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:04.047350  7076 raft_consensus.cc:2764] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:24:04.190066 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1067) took 1494 ms. Trace:
I20260709 12:24:04.084843  7045 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:04.083813  7115 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:04.083395  7041 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:04.251461 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:04.233297 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.235836 31747 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 22 index: 13. (index mismatch)
I20260709 12:24:04.254489 31672 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:04.082727  7200 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.196s
I20260709 12:24:04.300943 31746 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:24:04.341511 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:04.261245 31745 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:04.260730 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:04.344184 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:04.345252 31745 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:04.346903 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:04.191058  7145 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:04.108970  7042 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:04.257131 32375 rpcz_store.cc:276] 0709 12:24:02.695922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.696103 (+   181us) service_pool.cc:224] Handling call
0709 12:24:02.696519 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:03.123146 (+426627us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.123177 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:03.123223 (+    46us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:03.123253 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.189666 (+1066413us) spinlock_profiling.cc:243] Waited 426 ms on lock 0x7b540053a554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:04.189745 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:04.189968 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":980613376}
W20260709 12:24:04.259267 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 246922us
I20260709 12:24:04.114880  7095 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:04.351243 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:04.379207 31745 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:04.110136  7201 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.186717  6794 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:04.347667 31747 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:04.256588  7204 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:04.349383 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:04.351190 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:04.192291  7150 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:04.358793 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 252474us
I20260709 12:24:04.178514  7112 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:04.404753 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1107) took 1684 ms. Trace:
I20260709 12:24:04.405333 31744 rpcz_store.cc:276] 0709 12:24:02.720674 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.720815 (+   141us) service_pool.cc:224] Handling call
0709 12:24:04.404456 (+1683641us) spinlock_profiling.cc:243] Waited 832 ms on lock 0x7b5400064d54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:04.404665 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1914016000}
I20260709 12:24:04.223867  7216 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:04.354885 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:04.241551 31671 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
W20260709 12:24:04.370733 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 292557us
I20260709 12:24:04.403716 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:04.410341 31744 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
W20260709 12:24:04.374378 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.414577 31744 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:04.417165 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:04.419344 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:04.420823 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:04.421753 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.424329 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:04.425315 31824 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:04.427392 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:04.355329  7045 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:04.369203  7194 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:04.440402 32371 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:04.382517 31672 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:04.441696 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:04.442530 31672 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:24:04.287427  7196 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:04.348389  7041 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:04.400466 31677 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:24:04.384385  7145 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.352452  7198 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:04.513300 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:04.514768 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.403345 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:04.511961 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 111313us
I20260709 12:24:04.363467 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:04.285012  7077 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:04.438647 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:04.223613  7075 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:04.365454  7076 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.256645  7190 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:04.437703 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:04.377321  6986 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:04.400660 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:04.531829 31669 raft_consensus.cc:2376] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22. Re-sending same reply.
I20260709 12:24:04.496662 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:04.376799  7219 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:04.501538 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:04.538853 31821 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:04.503571 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:04.540095 31746 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:04.513295 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:04.395525 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:04.514691 31672 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:04.395251 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:04.519688 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:04.542140 31822 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:04.401590  7042 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:04.405611  7083 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:04.401765  7204 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:04.446089  6794 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.526072 31671 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:24:04.533392 32377 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:04.438776  7112 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.524906 31673 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:04.637092 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:04.634310 31746 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:24:04.367444  7106 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:04.634878 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:04.454327  6933 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:04.529927  7013 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.637804 31827 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:04.649328 32373 raft_consensus.cc:2410] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:24:04.528271  7197 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:04.535158  6858 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:04.535712  7216 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:04.653049 32377 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:04.621553  6651 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:04.527349  7145 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:04.653167  6862 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.182s	user 0.016s	sys 0.005s
I20260709 12:24:04.630299  7204 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:04.711890 32343 leader_election.cc:336] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.951s (SENT)
I20260709 12:24:04.512202  7041 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:04.720770 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:04.556125  7076 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:04.564141  6986 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:04.658829  7112 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:04.638662  6794 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:04.709892  6700 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:04.894294 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:04.401257  7201 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:04.743700  7114 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:04.664165  7013 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:04.501216  7196 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:04.566066  7219 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:04.714845  7205 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.725775  7208 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:04.621426  7045 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:04.714568  6933 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.860039 31671 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260709 12:24:04.860841 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 149271us
I20260709 12:24:04.651731  7224 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.715327  7231 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.677106  7197 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:04.743286  6862 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:04.902153 31744 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:04.813810  7014 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:04.909713 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:04.910413  7209 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.517s	user 0.008s	sys 0.002s
I20260709 12:24:04.894025  7083 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.941421 31749 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:04.949631  7075 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:04.950505  7075 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.951179  7075 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:04.867336  7225 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:04.876922  7216 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.920588  6700 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:04.952416 31752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1104) took 2249 ms (client timeout 1849 ms). Trace:
W20260709 12:24:05.037617 31752 rpcz_store.cc:269] 0709 12:24:02.690369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.690548 (+   179us) service_pool.cc:224] Handling call
0709 12:24:04.939477 (+2248929us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b54004042d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:04.939706 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1390900608}
W20260709 12:24:04.926761  7217 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.288s	user 0.001s	sys 0.000s
I20260709 12:24:04.901677  6651 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.799535  7042 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:04.931308  7205 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:04.955178 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:04.904942  7235 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.933583  7218 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:04.989749 31751 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 26: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:04.927222  7220 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:04.993789 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:05.088675 31753 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:04.940256  6941 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:04.944814  7231 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:04.931964 31745 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.552s	user 0.012s	sys 0.000s
I20260709 12:24:05.137677 31745 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
W20260709 12:24:05.004980 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 609716us
I20260709 12:24:04.921638  7114 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:04.958272  7095 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.003124 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1268) took 2985 ms. Trace:
I20260709 12:24:05.158735 32372 rpcz_store.cc:276] 0709 12:24:02.017594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.051291 (+ 33697us) service_pool.cc:224] Handling call
0709 12:24:03.941123 (+1889832us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:03.941562 (+   439us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:03.941604 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:03.941667 (+    63us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:03.941725 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:03.941807 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.002884 (+1061077us) spinlock_profiling.cc:243] Waited 1.89 s on lock 0x7b5400544f50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:05.003045 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31311,"raft.queue_time_us":925,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":67902,"spinlock_wait_cycles":4345193344}
W20260709 12:24:04.991155  7068 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.024s	user 0.014s	sys 0.012s
I20260709 12:24:04.939828  6933 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:05.033293 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.023105 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3420) took 3139 ms. Trace:
I20260709 12:24:05.167294 31670 rpcz_store.cc:276] 0709 12:24:01.883843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.518074 (+1634231us) service_pool.cc:224] Handling call
0709 12:24:03.518433 (+   359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:03.837932 (+319499us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.838165 (+   233us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:03.840436 (+  2271us) log.cc:844] Serialized 26 byte log entry
0709 12:24:04.035638 (+195202us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:04.035676 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.035741 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.022751 (+987010us) raft_consensus.cc:1685] finished
0709 12:24:05.022788 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.023018 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":135773,"threads_started":1,"wal-append.queue_time_us":1102372}
I20260709 12:24:05.168851 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:05.169400 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:04.999280  7210 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.478s	user 0.009s	sys 0.001s
W20260709 12:24:05.036597 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.178905 31673 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:24:05.050303 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:05.180080 31752 raft_consensus.cc:2376] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23. Re-sending same reply.
I20260709 12:24:05.083155  7218 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:05.068111  6700 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:05.133795  7198 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:05.071143 31678 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.989s	user 0.010s	sys 0.020s
W20260709 12:24:05.227010 31678 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.160s	user 0.010s	sys 0.020s
I20260709 12:24:05.227866 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2467) took 3768 ms. Trace:
I20260709 12:24:05.075578  6651 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:04.963280  7083 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.007972  7001 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:05.081630 31674 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.789s	user 0.013s	sys 0.018s
W20260709 12:24:05.259756 31674 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.967s	user 0.013s	sys 0.019s
I20260709 12:24:05.005672  7194 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:04.931090  7208 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:05.000065 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.669s	user 0.013s	sys 0.016s
I20260709 12:24:04.942019  7224 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.061367  7232 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:05.153282 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:05.076455  7077 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:05.217249  5949 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:24:05.158133  7229 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.414s	user 0.001s	sys 0.000s
W20260709 12:24:05.189608  7178 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.996s	user 0.021s	sys 0.034s
I20260709 12:24:05.222226 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:05.050442  7225 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:05.166762  7068 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.214s	user 0.015s	sys 0.012s
I20260709 12:24:04.995965  7190 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:05.180632 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:05.078157 31821 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.539s	user 0.012s	sys 0.000s
I20260709 12:24:05.228433 31678 rpcz_store.cc:276] 0709 12:24:01.459864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.510229 (+2050365us) service_pool.cc:224] Handling call
0709 12:24:03.553883 (+ 43654us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:03.554113 (+   230us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:03.904916 (+350803us) log.cc:844] Serialized 14 byte log entry
0709 12:24:05.227484 (+1322568us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:05.227548 (+    64us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:05.227590 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.227656 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.227789 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":107458,"prepare.queue_time_us":509233,"prepare.run_cpu_time_us":93,"prepare.run_wall_time_us":9561,"raft.queue_time_us":295166,"spinlock_wait_cycles":26290304,"thread_start_us":1555360,"threads_started":3,"wal-append.queue_time_us":387415}
I20260709 12:24:05.291850 31821 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:05.088042  7235 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:05.254269 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.026s	sys 0.080s
W20260709 12:24:05.297874 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.027s	sys 0.080s
W20260709 12:24:05.260564 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.047s	sys 0.083s
I20260709 12:24:05.213601  7218 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:05.267093 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.936s	user 0.013s	sys 0.016s
I20260709 12:24:05.051683  7216 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:05.235958  7048 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.859s	user 0.003s	sys 0.003s
I20260709 12:24:05.272368 31752 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:05.273708  5949 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:05.307293  5949 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 15, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.308420  5949 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:05.277973 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2486) took 2150 ms. Trace:
W20260709 12:24:05.266572  7200 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.753s	user 0.000s	sys 0.011s
I20260709 12:24:05.227869  6941 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.326721  7243 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.327039  6941 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:05.327392  7243 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:05.329622  6941 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:05.288417  7041 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:05.167052  7114 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.281985 31671 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:05.310514 31674 rpcz_store.cc:276] 0709 12:24:03.127653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.850036 (+722383us) service_pool.cc:224] Handling call
0709 12:24:03.850529 (+   493us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:03.908154 (+ 57625us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.908208 (+    54us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:03.908975 (+   767us) log.cc:844] Serialized 52 byte log entry
0709 12:24:04.177993 (+269018us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:05.260222 (+1082229us) log.cc:844] Serialized 14 byte log entry
0709 12:24:05.271138 (+ 10916us) log.cc:844] Serialized 14 byte log entry
0709 12:24:05.277599 (+  6461us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.277689 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.277740 (+    51us) raft_consensus.cc:1685] finished
0709 12:24:05.277759 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.277909 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":220115,"prepare.queue_time_us":218143,"prepare.run_cpu_time_us":133,"prepare.run_wall_time_us":3552,"raft.queue_time_us":135802,"raft.run_cpu_time_us":369,"raft.run_wall_time_us":52850,"spinlock_wait_cycles":25184256,"thread_start_us":1204795,"threads_started":2,"wal-append.queue_time_us":470674}
I20260709 12:24:05.288393 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:05.290248 31747 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:05.336757 31749 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
W20260709 12:24:05.165261  7209 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
W20260709 12:24:05.298560 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.672s	user 0.048s	sys 0.083s
W20260709 12:24:05.278519  7178 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.100s	user 0.021s	sys 0.034s
I20260709 12:24:05.303943  7112 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:05.324321  7239 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.158s	user 0.001s	sys 0.000s
I20260709 12:24:05.268100  7084 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.345993  7197 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.278344  7014 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.187538  6909 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:05.346290  7084 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.346920  7014 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.348526  7084 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.349794  7083 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:05.278093  7246 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:05.349988  7190 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.280088  7076 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.350960  7246 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.355969  7114 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:05.356240  6909 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:05.357252  7216 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:05.358333  7039 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.349s	user 0.009s	sys 0.002s
W20260709 12:24:05.324918  7214 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.327s	user 0.001s	sys 0.000s
W20260709 12:24:05.306670  7048 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.949s	user 0.003s	sys 0.004s
W20260709 12:24:05.311267  7230 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.348s	user 0.001s	sys 0.001s
I20260709 12:24:05.331200  6700 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.303539  6794 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:05.259097  7145 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.304543  6986 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:05.301476 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 1.12162e+06us
I20260709 12:24:05.341913 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:05.342809 31674 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:24:05.423101 31674 raft_consensus.cc:606] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Received request to transfer leadership to TS 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:05.423569 31674 raft_consensus.cc:614] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:05.311575  7150 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:05.423828 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 81007.1us
I20260709 12:24:05.361737 31821 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:05.362097 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1065) took 2693 ms. Trace:
I20260709 12:24:05.362344 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:05.425527 32369 rpcz_store.cc:276] 0709 12:24:02.668263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:02.668378 (+   115us) service_pool.cc:224] Handling call
0709 12:24:02.718477 (+ 50099us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:02.748189 (+ 29712us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:02.748244 (+    55us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:03.870622 (+1122378us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b2c000eba68. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:24:03.871322 (+   700us) log.cc:844] Serialized 52 byte log entry
0709 12:24:04.015816 (+144494us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:04.042450 (+ 26634us) log.cc:844] Serialized 14 byte log entry
0709 12:24:04.076852 (+ 34402us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.076972 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.361800 (+1284828us) raft_consensus.cc:1685] finished
0709 12:24:05.361900 (+   100us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.362043 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6745,"prepare.queue_time_us":26378,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":150826,"raft.run_cpu_time_us":308,"raft.run_wall_time_us":35479,"spinlock_wait_cycles":4837120,"thread_start_us":143994,"threads_started":1,"wal-append.queue_time_us":360189}
I20260709 12:24:05.373651  7013 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:05.365777  7027 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.352s	user 0.006s	sys 0.004s
I20260709 12:24:05.366400 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:05.434078 31671 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:05.368183 31678 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:24:05.368392 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3337) took 1036 ms. Trace:
W20260709 12:24:05.320833  7200 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.808s	user 0.001s	sys 0.011s
I20260709 12:24:05.369606 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:05.435859 31744 rpcz_store.cc:276] 0709 12:24:04.331592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.415625 (+ 84033us) service_pool.cc:224] Handling call
0709 12:24:04.415983 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:04.944695 (+528712us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:05.368128 (+423433us) spinlock_profiling.cc:243] Waited 486 ms on lock 0x7b54000b8bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:05.368313 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1117584128}
I20260709 12:24:05.369822 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:05.437927 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:05.372097 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.329654  7243 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:05.378655  7131 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.502s	user 0.009s	sys 0.005s
I20260709 12:24:05.418594  7083 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:05.338501  7245 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.392436 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:05.361738  7149 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:05.400118 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.422577 31752 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:05.424782 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 16 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:05.359089  6651 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.431658 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:05.362466 31827 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 23 index: 16. (term mismatch)
I20260709 12:24:05.432865 32372 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 12:24:05.438109 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 149794us
I20260709 12:24:05.371505 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:05.438397 31670 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
W20260709 12:24:05.439064 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.378919 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:05.444377 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:05.496639 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:05.544056 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.545989 31744 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:05.546306 31669 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:24:05.446569 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:05.547673 31674 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:05.544147 31827 pending_rounds.cc:85] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 15
I20260709 12:24:05.548408 32369 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:05.548924 31827 pending_rounds.cc:107] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:24:05.549268 31827 raft_consensus.cc:2889] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:05.552062 31744 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:05.552490 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:05.544484 32377 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:05.555724 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:05.557195 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.433138  7201 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:05.544824 31753 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:05.385141  7039 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.556468 31678 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:24:05.388302  7095 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:05.572835 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:05.573522 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:05.457304  7131 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:05.592782 32376 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:05.343751  7194 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.574441 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:05.394776  7075 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:05.576413 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:05.576516 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:05.440011  7150 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:05.573949 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:05.581648 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:05.339181  7198 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:05.444475  7048 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:05.550469  7219 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:05.492637  7204 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.706s	user 0.011s	sys 0.001s
I20260709 12:24:05.543859  7083 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:05.569520  7208 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.444468  7027 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:05.594220  7037 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.740s	user 0.013s	sys 0.002s
I20260709 12:24:05.603029  7201 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:05.573998  6859 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.031s
I20260709 12:24:05.607069  6923 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.630949  7204 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:05.608315  7131 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.607959  7218 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.631573  7224 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:05.561203  7245 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:05.677325  7245 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.614746  7263 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:05.678417  7263 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:05.679641  7263 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.587426  7227 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:05.617223  7232 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:05.593468  7222 raft_consensus.cc:2764] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election decision vote started in defunct term 15: won
I20260709 12:24:05.610296  7246 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:05.595420  7095 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:05.684021 32371 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:05.640408  7218 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.634696  7201 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:05.642189  7027 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:05.588388  7255 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.082s
I20260709 12:24:05.703086  7255 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:05.703541  7255 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:05.704708  7220 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:05.704515  7255 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.705451  7220 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:05.706087  7255 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:05.636693  7037 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:05.645779  7247 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:05.647361  7254 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:24:05.650017  7083 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.653420  7115 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:05.655967  6862 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:05.671785  7235 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.673059  7205 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:05.674008  6933 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:05.632789  7212 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
W20260709 12:24:05.702435  7256 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.113s	user 0.000s	sys 0.000s
I20260709 12:24:05.666392  7039 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:05.698001  7201 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:05.709491  7247 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.725337  7204 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.719846  7150 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.719177  7083 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:05.728935  7196 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.730954  7231 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.718482  7115 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:05.733530  7257 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.379s	user 0.000s	sys 0.001s
I20260709 12:24:05.753885  7220 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.756500  7220 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:05.749621  7201 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:05.759518  7201 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.590803 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:05.457340 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 122299us
I20260709 12:24:05.799053 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:05.799829 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.597098 31671 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:05.601548 31752 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:05.602355 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:05.597700 31750 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:24:05.580395 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 155712us
W20260709 12:24:05.693746 32378 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 1121) took 2442 ms (client timeout 1769 ms). Trace:
I20260709 12:24:05.802137 31676 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:24:05.799899 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 356741us
W20260709 12:24:05.809057 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.814430 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 240175us
W20260709 12:24:05.813529 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:05.871338  7106 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.000s	sys 0.008s
W20260709 12:24:05.872012  7106 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s	user 0.000s	sys 0.009s
I20260709 12:24:05.816365 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:05.816910 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:05.811866 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3256) took 2580 ms. Trace:
I20260709 12:24:05.880699  7095 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:05.879017 31823 rpcz_store.cc:276] 0709 12:24:03.230837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.674245 (+443408us) service_pool.cc:224] Handling call
0709 12:24:03.674653 (+   408us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:03.808162 (+133509us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:03.808230 (+    68us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:03.831532 (+ 23302us) log.cc:844] Serialized 52 byte log entry
0709 12:24:04.038751 (+207219us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:04.038793 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:04.038860 (+    67us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:05.810984 (+1772124us) raft_consensus.cc:1685] finished
0709 12:24:05.811047 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:05.811603 (+   556us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17306,"raft.queue_time_us":17621,"raft.run_cpu_time_us":209,"raft.run_wall_time_us":72841,"thread_start_us":140039,"threads_started":1,"wal-append.queue_time_us":1567603}
W20260709 12:24:05.884719 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.824647 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:05.887511 31678 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:05.801978 31675 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:05.810178 31825 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:05.817718 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:05.916360 31745 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:05.919057 31753 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:05.923130 31750 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:05.923610 32378 rpcz_store.cc:269] 0709 12:24:03.251098 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.251254 (+   156us) service_pool.cc:224] Handling call
0709 12:24:05.693402 (+2442148us) spinlock_profiling.cc:243] Waited 1.92 s on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:05.693633 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4408222080}
I20260709 12:24:05.937196 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:05.962437 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.962615 31678 raft_consensus.cc:2410] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 27 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:24:05.963899 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:05.966025  7225 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:05.965931 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:05.940017  7095 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:05.963994 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:05.971163 31669 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
W20260709 12:24:05.959932 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 320730us
I20260709 12:24:05.960841 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:05.986172 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1202) took 1962 ms. Trace:
I20260709 12:24:05.986683 31676 rpcz_store.cc:276] 0709 12:24:04.023593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.045631 (+ 22038us) service_pool.cc:224] Handling call
0709 12:24:05.985827 (+1940196us) spinlock_profiling.cc:243] Waited 1.62 s on lock 0x7b54001d5e54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:05.986077 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3725396864}
I20260709 12:24:05.987740 31750 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:05.992235 31671 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:24:05.990257 31676 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:05.990350 31750 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:06.005141 31825 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3261) took 2548 ms (client timeout 2159 ms). Trace:
W20260709 12:24:06.005801 31825 rpcz_store.cc:269] 0709 12:24:03.456498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.514578 (+ 58080us) service_pool.cc:224] Handling call
0709 12:24:06.004717 (+2490139us) spinlock_profiling.cc:243] Waited 2.01 s on lock 0x7b54004475d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:06.004978 (+   261us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4612431616}
I20260709 12:24:06.008548 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1203) took 1984 ms. Trace:
I20260709 12:24:06.009071 31675 rpcz_store.cc:276] 0709 12:24:04.023985 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.024110 (+   125us) service_pool.cc:224] Handling call
0709 12:24:04.024468 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:05.899153 (+1874685us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.008140 (+108987us) spinlock_profiling.cc:243] Waited 1.78 s on lock 0x7b54002b1454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:06.008449 (+   309us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4085078400}
I20260709 12:24:06.009822 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:06.010540 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:06.010859 32375 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:24:06.012957 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.013067 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:06.014906 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.015275 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:06.016310 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:05.991770  6933 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.019099  6862 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:06.019838 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:06.019939  6933 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.020962 32377 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:06.020661  7201 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:06.021533 31671 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:06.021591  7201 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:06.023442 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3405) took 1606 ms. Trace:
I20260709 12:24:06.023943 31818 rpcz_store.cc:276] 0709 12:24:04.417148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.418859 (+  1711us) service_pool.cc:224] Handling call
0709 12:24:06.022905 (+1604046us) spinlock_profiling.cc:243] Waited 1.4 s on lock 0x7b54002675d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:06.023183 (+   278us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3216016128}
I20260709 12:24:06.023576  7201 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.020802  7243 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:06.026643  7201 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:06.028141  7243 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:06.028651  7243 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:06.030342  7243 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.032068  7243 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:06.034473  6933 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:06.048266 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:06.050056  7014 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:06.053300 31642 leader_election.cc:336] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.751s (SENT)
I20260709 12:24:05.992820  7095 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:06.056246  7095 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.035595 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:06.036374  7220 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:06.059376 32378 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
W20260709 12:24:06.061053 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 127801us
I20260709 12:24:06.037606 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:06.038341  7225 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.038862 31677 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:06.039403  6862 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.001s
I20260709 12:24:06.038709 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.079025 31671 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:06.091122 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:06.090986  7225 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:06.091156 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:06.094480  7037 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.090144  7255 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:24:06.094862 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56418us
I20260709 12:24:06.096865  7255 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.104947  7220 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:06.105154  7188 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:06.105587  7220 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.106097  7220 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:06.104072 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:06.106189  7188 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.105199  6858 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.006s
I20260709 12:24:06.111400  7045 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:06.112282 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3412) took 1683 ms. Trace:
I20260709 12:24:06.112223  7045 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.112838 31819 rpcz_store.cc:276] 0709 12:24:04.429006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.502389 (+ 73383us) service_pool.cc:224] Handling call
0709 12:24:04.535657 (+ 33268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:04.535883 (+   226us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:05.881884 (+1346001us) spinlock_profiling.cc:243] Waited 5.69 ms on lock 0x7b1800249e80. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf
0709 12:24:05.882316 (+   432us) log.cc:844] Serialized 14 byte log entry
0709 12:24:06.084163 (+201847us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:06.097117 (+ 12954us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:06.102785 (+  5668us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.108261 (+  5476us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:06.112085 (+  3824us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8737,"prepare.queue_time_us":478392,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":85145,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":290,"spinlock_wait_cycles":31964800,"thread_start_us":516189,"threads_started":2,"wal-append.queue_time_us":188001}
I20260709 12:24:06.115929 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2667) took 1857 ms. Trace:
I20260709 12:24:06.116408 31748 rpcz_store.cc:276] 0709 12:24:04.258835 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.258994 (+   159us) service_pool.cc:224] Handling call
0709 12:24:04.259350 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:04.441055 (+181705us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:04.441095 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:04.441133 (+    38us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:04.441163 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.115364 (+1674201us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b54004056d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:06.115434 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:06.115703 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":417412992}
I20260709 12:24:06.117142  7188 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:06.086352  7014 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.123838 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.107244  7255 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:06.129716 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 2.05313e+06us
I20260709 12:24:06.132367  7045 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:06.133431  7247 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:06.096890  7037 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:06.134974  7114 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:06.135645  7258 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.757s	user 0.000s	sys 0.004s
I20260709 12:24:06.135988  7114 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.137295  7037 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:06.138980 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.140527  7014 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:06.146831 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:06.125093  7042 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.002s	sys 0.014s
W20260709 12:24:06.160641 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58025.9us
I20260709 12:24:06.155973 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:06.162354 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 58105.6us
I20260709 12:24:06.163156 32376 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
I20260709 12:24:06.163491  6862 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:06.163846 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.164579  6933 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:06.165045  6933 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.165424  7267 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:06.165308  7013 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.003s
I20260709 12:24:06.162370 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:06.169795 31751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1112) took 2271 ms (client timeout 2577 ms). Trace:
W20260709 12:24:06.161583  7042 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.002s	sys 0.015s
W20260709 12:24:06.171855 31751 rpcz_store.cc:269] 0709 12:24:03.898068 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.909817 (+ 11749us) service_pool.cc:224] Handling call
0709 12:24:06.168063 (+2258246us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b54002f5dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:06.169548 (+  1485us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2443088256}
I20260709 12:24:06.165472  6933 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:06.167073  7267 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.175464  6862 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:06.177700 31750 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 16. (index mismatch)
W20260709 12:24:06.192557 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 139395us
I20260709 12:24:06.170511  7037 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.190742 32371 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:24:06.197365 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.194741 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.200405 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:06.199702 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 133429us
I20260709 12:24:06.174325  7225 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:06.191623  7267 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:06.209036 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.193365  7114 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:06.217417 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.220952 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.221944 31827 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:06.229512 31819 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:06.230837  6752 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:24:06.232672 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.233417  7242 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:06.234126  7242 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.234619  7242 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:06.236852 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.237834 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.239297 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:06.240204 31750 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 24 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:24:06.241050 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.245843 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:06.246117 31670 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:24:06.248037 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:06.249090 31677 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:24:06.217439 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.250368 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:06.251089 31744 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:06.230918  7224 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 12:24:06.256811 31751 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:06.258158 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3347) took 1244 ms. Trace:
I20260709 12:24:06.258685 31749 rpcz_store.cc:276] 0709 12:24:04.953077 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.954870 (+  1793us) service_pool.cc:224] Handling call
0709 12:24:06.179905 (+1225035us) spinlock_profiling.cc:243] Waited 267 ms on lock 0x7b54000b81d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:06.197020 (+ 17115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":614104320}
I20260709 12:24:06.251256 31669 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:06.228262 31747 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:06.228643 31745 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:06.263054 31753 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:06.265379 31752 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
W20260709 12:24:06.274317 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 68921us
I20260709 12:24:06.272347  7014 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:06.276973 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.278020 31752 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260709 12:24:06.276985 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.275601 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.285840 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:06.290372 31642 leader_election.cc:336] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.883s (SENT)
I20260709 12:24:06.267158  7037 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.293624 31826 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:24:06.296008 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:06.312021 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:06.318972  7286 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.121s	user 0.000s	sys 0.001s
I20260709 12:24:06.327008 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:06.328249 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.328516 31820 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:06.329850  7037 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:06.331745 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.330981 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.349160 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 16 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.330596  6858 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:06.346979 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:06.351933 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 128374us
I20260709 12:24:06.357787 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:06.351836  6858 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:06.366068  6858 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:06.364409 32342 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.370237 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.364758 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.331187  7106 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.031s
I20260709 12:24:06.362709  7014 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:06.331467  7083 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:24:06.381321 31820 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
W20260709 12:24:06.383311 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 50838.7us
W20260709 12:24:06.384040 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.374257  7201 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:06.347898  7255 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:06.390232 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 114936us
I20260709 12:24:06.391065 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.357225  6862 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:06.380983  7014 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:06.399201 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:06.400271 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:06.401849 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:06.399540  7014 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:06.409930 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:06.412173 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.410039 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:06.414616 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.415012 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.415376 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.415683 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.409833 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:06.416498 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 109047us
I20260709 12:24:06.417137 31820 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:06.417912 31669 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:06.418444 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.419458 31751 raft_consensus.cc:2410] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 26 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 16.
I20260709 12:24:06.416254 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:06.418188 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:06.415683 31749 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:06.420338 31677 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:06.423852 31745 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260709 12:24:06.424533 32371 pending_rounds.cc:85] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 17
I20260709 12:24:06.429023 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.424341 31745 pending_rounds.cc:85] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 15
I20260709 12:24:06.430140 31750 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:06.431762  7037 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.432039 31747 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:24:06.432215 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:06.432591  7037 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.433091  7255 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:06.424919 32371 pending_rounds.cc:107] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 22.18
I20260709 12:24:06.431891 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.435904 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:06.436220 32371 raft_consensus.cc:2889] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:06.430114 31745 pending_rounds.cc:107] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:24:06.437201 31745 raft_consensus.cc:2889] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:06.437793 31826 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:06.439805 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:06.440529 31670 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:06.439697 31749 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:24:06.443583 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:06.445833 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.443171 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.448820 31744 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:06.442938 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:06.443883 31791 connection.cc:256] RPC call timeout handler was delayed by -1.9405501560004268s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2121s for a response
I20260709 12:24:06.454015 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:06.440516  7201 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:06.466034 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.471230 31820 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:06.471757 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.445689  7042 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:06.479110 31791 connection.cc:256] RPC call timeout handler was delayed by -1.4893274640003256s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1758s for a response
W20260709 12:24:06.484144 31791 connection.cc:256] RPC call timeout handler was delayed by -1.4326274640002339s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1821s for a response
W20260709 12:24:06.485517 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 87196.7us
I20260709 12:24:06.468209  7255 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:24:06.488365 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 48540.8us
I20260709 12:24:06.482427 31673 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 18 index: 10. (index mismatch)
I20260709 12:24:06.464824  7201 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:06.490345 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0861903270001676s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1791s for a response
I20260709 12:24:06.491637 31748 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:06.494805 31677 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 28 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:24:06.499034 31750 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:06.497084 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.451426  7243 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:24:06.504777 31820 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:06.506910 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:06.508656 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.501238 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:06.498972  7201 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:06.512789 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 170243us
I20260709 12:24:06.514967 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:06.514175 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:06.511690 32378 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
W20260709 12:24:06.522444  7288 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.156s	user 0.001s	sys 0.000s
I20260709 12:24:06.522742  7194 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.007s
I20260709 12:24:06.524302 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3345) took 1798 ms. Trace:
I20260709 12:24:06.526345 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.535058 31820 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:06.524741 31746 rpcz_store.cc:276] 0709 12:24:04.725859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:04.865778 (+139919us) service_pool.cc:224] Handling call
0709 12:24:04.866313 (+   535us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:04.866623 (+   310us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:04.866689 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:06.037740 (+1171051us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b2c00001158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:24:06.038649 (+   909us) log.cc:844] Serialized 26 byte log entry
0709 12:24:06.099764 (+ 61115us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:06.101567 (+  1803us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:06.104422 (+  2855us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:06.523879 (+419457us) raft_consensus.cc:1685] finished
0709 12:24:06.523901 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:06.524129 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24298368,"thread_start_us":60607,"threads_started":1,"wal-append.queue_time_us":151647}
I20260709 12:24:06.513635  7289 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:06.531566 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.540033  7045 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:06.541572  6651 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:06.542150  6651 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.542551  6651 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:06.551301  6933 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:06.551251 31747 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:06.555589 31677 pending_rounds.cc:85] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 16
I20260709 12:24:06.555958 31677 pending_rounds.cc:107] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260709 12:24:06.556124 31677 raft_consensus.cc:2889] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:06.561311 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 97765.3us
I20260709 12:24:06.562664 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:06.563607  7037 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:06.571238 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 105067us
I20260709 12:24:06.572638 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.573909 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.574790  7297 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 12:24:06.516462  7042 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:06.576999  7083 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:06.577483  7083 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:06.594792 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 89384.6us
W20260709 12:24:06.589248 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50739.1us
I20260709 12:24:06.588308  7243 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:06.592320  7247 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:24:06.601490 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53287.3us
I20260709 12:24:06.532297  7224 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:06.592555  7013 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:06.605057 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.605820 31825 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:06.613286  7068 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.614075  7068 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.616736  7068 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:06.617415  7068 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.617985  7068 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.623960  7220 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.625656 31753 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:06.628252  7068 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:06.627380  7083 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:06.634893  7295 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:06.635165 31819 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:24:06.605212  7013 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.635303 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:06.538524  7289 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:06.648514 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45532.5us
I20260709 12:24:06.649259 31818 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:06.591998  7042 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:06.600598  7247 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:06.657723 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.658243  7247 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:06.658962 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.659698 31823 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:06.661085 31826 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:24:06.666815 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.666994 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.671872  7220 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:06.672302 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.673228  7045 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:06.673674 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:06.674305 31822 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:06.674463 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.675086  7045 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:06.652803 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.657450  7042 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:06.696738  6933 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:06.682099 31818 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:06.645171  7295 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.652314  7013 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:06.682842  7037 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:06.685249 32378 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:24:06.686092  7114 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:06.653074  7289 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:06.688879 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:06.690474 31821 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:24:06.691354 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.691752 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:06.692122 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.692664 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:06.693264  7188 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:06.694908 31827 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:06.656606  7083 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:06.697882  6752 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:06.698166 31822 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:06.656531 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:06.699541  7291 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
I20260709 12:24:06.623960  7267 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:06.706893 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:06.706979 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:06.710603 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.699549 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:06.719098 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 197974us
I20260709 12:24:06.675555  7045 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:06.721926 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.721127 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:06.764602 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:06.743567 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.650030  7255 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:06.750145 31819 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:06.770005  6933 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:06.741007  7242 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.506s	user 0.008s	sys 0.006s
I20260709 12:24:06.765687 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.771597 32372 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:06.757601 31820 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:06.774904 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:06.780865 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 124554us
I20260709 12:24:06.775533 31750 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:24:06.782661 31825 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:06.776608 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.746007  7114 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.745462  7225 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.786002 31821 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:06.743875  7295 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:06.788125 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.784102 31824 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:06.789749 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.763298  6752 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:06.773257  7045 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:06.804245  6752 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:06.805502  6752 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:06.809162 31717 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.809785  7255 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:06.811143  7095 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:06.811637  7095 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.764966  7068 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.794519 31822 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
W20260709 12:24:06.814260 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:06.815280 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 148640us
W20260709 12:24:06.820148 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.791595  7114 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:06.823994 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:06.791337  7220 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:06.777813  6933 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.826959 32372 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:06.794955  7224 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:06.838948 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:06.841154 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.846971 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:06.850374 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.773999  6858 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:06.822871  7095 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:06.853179 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.788918  7242 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:06.857008 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.858881 31822 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:06.860251 31826 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:24:06.796387  7014 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:06.863329 31827 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:06.866916 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:06.867398  7037 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:06.867611 31823 raft_consensus.cc:2376] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23. Re-sending same reply.
I20260709 12:24:06.867920  7037 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.868378  7037 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:06.859529  7188 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:06.869127  7188 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:06.870131  7188 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.870757 31753 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:24:06.872316 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.875983 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:06.831214  7224 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.855674  7220 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:06.895424 32372 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
W20260709 12:24:06.886767 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56118.9us
I20260709 12:24:06.903635 32370 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 28: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:06.903836 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:06.831470  7225 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:06.884132  7290 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.276s	user 0.003s	sys 0.001s
I20260709 12:24:06.881667  7224 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:06.856551  7201 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.924364  7045 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.925665  7302 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:24:06.934808 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 223226us
I20260709 12:24:06.935681  6651 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.936004 31792 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:06.927058 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:06.938396 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.937402 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.891420  7068 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:06.949040 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:06.951161 31818 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:06.936369  7045 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:06.959163 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 87439.5us
I20260709 12:24:06.963173 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:06.970765  7296 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:06.971375  7296 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:06.971726  7296 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:06.970765 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:06.948514  7083 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:06.975651  7083 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:06.976307 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:06.993902 31818 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:06.995023 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:06.998662 31818 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:06.997911 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53060.6us
I20260709 12:24:07.006598 32372 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:07.006510 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 70613.4us
W20260709 12:24:07.008298 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 52235us
W20260709 12:24:06.980700  7298 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.101s	user 0.000s	sys 0.001s
I20260709 12:24:07.013816  7243 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:07.014312  7243 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:07.014716  7243 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:07.015425 32373 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 23 index: 18. (index mismatch)
I20260709 12:24:06.982414  7045 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:07.016835 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:07.022464  6858 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.024473  7013 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.016734  7220 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:06.980733  7114 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.035538  7242 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.014317  6651 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.039043  7083 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.039561  6752 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.040863  7037 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.013654  6933 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.044885 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:07.046276 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:07.047149 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:07.050827 31673 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:07.050945 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:07.076356 31677 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:07.052035 31678 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:07.056098  7068 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:07.061858 32372 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
W20260709 12:24:07.022368 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 77870us
I20260709 12:24:07.013629 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:07.016829 31718 connection.cc:256] RPC call timeout handler was delayed by -1.9625454319998425s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16210000000000002s for a response
I20260709 12:24:07.064708 31821 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
W20260709 12:24:07.065171 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 127877us
W20260709 12:24:07.075552 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 65279.7us
W20260709 12:24:07.094348 31718 connection.cc:256] RPC call timeout handler was delayed by -1.9607454319998396s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1623s for a response
W20260709 12:24:07.097442 31718 connection.cc:256] RPC call timeout handler was delayed by -1.8554454320001241s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17400000000000002s for a response
W20260709 12:24:07.097826 31718 connection.cc:256] RPC call timeout handler was delayed by -1.8293454320000819s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1769s for a response
W20260709 12:24:07.098127 31718 connection.cc:256] RPC call timeout handler was delayed by -1.4873454319999837s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2149s for a response
W20260709 12:24:07.098421 31718 connection.cc:256] RPC call timeout handler was delayed by -1.4783454319999692s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21589999999999998s for a response
W20260709 12:24:07.098775 31718 connection.cc:256] RPC call timeout handler was delayed by -1.4252454319998833s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2218s for a response
W20260709 12:24:07.099102 31718 connection.cc:256] RPC call timeout handler was delayed by -1.4072454319998542s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2238s for a response
W20260709 12:24:07.099398 31718 connection.cc:256] RPC call timeout handler was delayed by -1.1786454319999393s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2492s for a response
W20260709 12:24:07.099685 31718 connection.cc:256] RPC call timeout handler was delayed by -1.1633454319999146s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2509s for a response
I20260709 12:24:07.107439 31670 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:07.107506 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:07.108693 31671 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:07.110975 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:07.111822 32373 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:07.113797 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:07.124166 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.100828  7267 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:07.125627 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:07.138778 31678 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:07.139880 31671 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:24:07.141750 32376 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:24:07.143803 31818 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
W20260709 12:24:07.144482 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 83219us
I20260709 12:24:07.154281 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:07.101338  6752 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:07.155110 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:07.169389  7318 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.170394  7318 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:07.176637  7224 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:07.179574 31753 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:07.145889  7320 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:07.180927  7320 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.181305  7320 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:07.182770  7304 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:07.183830  7304 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.185974 31672 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:07.187665 31672 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 21 index: 10. (index mismatch)
I20260709 12:24:07.188870 31674 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: [23.18-23.18]
I20260709 12:24:07.111907  7194 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
W20260709 12:24:07.200835 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 65738.5us
I20260709 12:24:07.121007  6651 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.146713  7313 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:07.237587 31821 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
W20260709 12:24:07.236797 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 83661.3us
I20260709 12:24:07.247682 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:07.247682 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:07.248495 31753 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:24:07.205484  7201 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.241218 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:07.256364 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 148942us
I20260709 12:24:07.229694  7225 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.287585 31674 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:07.289012 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 275475us
I20260709 12:24:07.291735 31750 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:24:07.294934 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:07.297436 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:07.261085  6862 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:07.300798 31672 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:07.251380  7313 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:07.248782  7289 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.299163  7317 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:24:07.280162  6651 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.330438 31753 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:07.331032 31642 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:07.338985 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:07.287055  6933 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.026s
I20260709 12:24:07.340678 31670 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:07.352183 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:07.360620 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.363502 31753 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 17. (term mismatch)
I20260709 12:24:07.363937 31753 pending_rounds.cc:85] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 16
I20260709 12:24:07.364176 31753 pending_rounds.cc:107] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 24.17
I20260709 12:24:07.364432 31753 raft_consensus.cc:2889] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:07.361080 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 215621us
I20260709 12:24:07.345321  7037 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:07.345501  7013 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.382570  7224 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:07.389317  7316 raft_consensus.cc:2764] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
W20260709 12:24:07.402736 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55183.9us
W20260709 12:24:07.404168 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 73216.2us
I20260709 12:24:07.358289  7308 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.200s
I20260709 12:24:07.429975 31674 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:07.392459  7324 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.036s
I20260709 12:24:07.423600  6752 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:07.423674 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:07.424171 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:07.424552  7201 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:07.424736  7295 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.600s	user 0.011s	sys 0.001s
I20260709 12:24:07.486753  7304 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:07.486912  7295 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.487931 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:07.488878 31672 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:07.489401  7295 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.425114 31792 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:07.490995 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:07.492281 31753 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 24 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 18.
I20260709 12:24:07.425472 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:07.425220  7321 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:07.415436  7313 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:07.341208  7194 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.032s
I20260709 12:24:07.425243 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:07.428202 31749 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 16. (term mismatch)
W20260709 12:24:07.422101  7247 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.763s	user 0.009s	sys 0.004s
I20260709 12:24:07.433239  6651 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.499413 31749 pending_rounds.cc:85] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 15
I20260709 12:24:07.503436 31749 pending_rounds.cc:107] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 25.16
I20260709 12:24:07.434005 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:07.503695 31749 raft_consensus.cc:2889] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:07.439705  7322 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
W20260709 12:24:07.441642 31643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.914s (SENT) [suppressed 27 similar messages]
I20260709 12:24:07.444077 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:07.444674 31677 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
W20260709 12:24:07.445561  7325 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.189s	user 0.002s	sys 0.000s
W20260709 12:24:07.505276 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61268.9us
I20260709 12:24:07.446851  7289 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.448760 32376 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17"
dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:58522
I20260709 12:24:07.449918 31678 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:07.507244 31746 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:07.507362 32376 raft_consensus.cc:606] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:07.507946 32376 raft_consensus.cc:614] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:07.430972  7255 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:07.451639  7243 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.452291 32373 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:07.454617  7296 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.455595  7045 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:07.458660  7315 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.199s	user 0.003s	sys 0.000s
W20260709 12:24:07.459626  7095 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.567s	user 0.008s	sys 0.004s
W20260709 12:24:07.520210 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78568.1us
I20260709 12:24:07.466063 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:07.478137 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.477995 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:07.484750  7337 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:07.485471 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:07.563112 31674 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:07.491058 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 66078us
I20260709 12:24:07.493795 31748 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
W20260709 12:24:07.496942 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 68593us
W20260709 12:24:07.498288 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.500327 31750 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 26: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:24:07.502177 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 119296us
I20260709 12:24:07.439059 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:07.501026  7304 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:07.517509  7255 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:07.524912  7243 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.525386  7014 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:07.525804  7310 raft_consensus.cc:2764] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:24:07.528465  7095 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.529597  7225 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:07.536119  7296 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.542737  7242 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:07.543147 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:07.543684  6651 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:07.605872  6651 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:07.606366  6651 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.606755  6651 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:07.544139  7220 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:07.611239  7220 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:07.611708  7220 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:07.612617  7220 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:07.613627  7243 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:07.614064  7220 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:07.545847  7318 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:07.594677  7114 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:07.614084  7243 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:07.548386 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:07.505867  7247 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:07.615831 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 176921us
I20260709 12:24:07.616240 32376 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:07.502928  7201 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:07.507470  7308 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:07.553596  7320 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:07.521870 31744 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:24:07.556842  7312 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:07.557770 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:07.558571 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.560997 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:07.561547  7316 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 12:24:07.560230 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58899us
W20260709 12:24:07.576448  7301 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.012s	sys 0.008s
I20260709 12:24:07.588160  7326 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.022s
I20260709 12:24:07.588836  7304 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:07.631984 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 74291.2us
I20260709 12:24:07.592592 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:07.593811 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:07.602097  7303 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.269s	user 0.000s	sys 0.003s
I20260709 12:24:07.623699 31746 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
W20260709 12:24:07.626484 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 83397.1us
I20260709 12:24:07.636623 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:07.637784 31819 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:24:07.638748  7289 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:07.660535 31825 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:07.660684  7289 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.661152  7289 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:07.661599 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:07.662192 31825 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:07.662885 31673 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b416bf5304744b14b72501e697b8c951"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "039d66a6c85e43c3936bf5017501b344"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:24:07.663497 31673 raft_consensus.cc:606] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Received request to transfer leadership to TS 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:07.663882 31673 raft_consensus.cc:614] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:07.666983 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:07.639357 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:07.640955  7243 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:07.599104  7225 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:07.643417  7095 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.562479  7337 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:07.645623  7247 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.646271 31671 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:24:07.647084 31818 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:07.648099  7320 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:07.648362 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 87449.7us
I20260709 12:24:07.649320 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:07.649801  7201 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:07.625154  7308 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.650969  7255 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 18.13, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:07.651671 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:07.652096  7316 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:07.653086  7304 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:07.702185 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50595.9us
I20260709 12:24:07.703091 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:07.657907 31674 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:07.659112 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:07.650974  7310 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:07.659821 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:07.638929 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:07.632042  7301 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.013s	sys 0.008s
W20260709 12:24:07.668759 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 78677.8us
I20260709 12:24:07.669605  7321 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:07.663213 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:07.681671 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:07.714973 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:07.696583 32376 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
W20260709 12:24:07.699872 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:07.700299 31822 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:07.697073 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:07.704098 31820 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:07.704070  7313 raft_consensus.cc:2764] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:24:07.708436  7243 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:07.710716 31824 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:07.688874  7322 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.005s
I20260709 12:24:07.718047  7334 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.036s
I20260709 12:24:07.718789  7201 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:07.719239  7095 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:07.716413 31819 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:07.688959  7296 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:24:07.688055  7318 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.028s
I20260709 12:24:07.703976  7308 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:07.733135 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:07.727676 31821 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:07.735311 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53757.9us
I20260709 12:24:07.738601 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:07.739392 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:07.791924 31822 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:07.747159 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 80245.3us
I20260709 12:24:07.723539  7310 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:07.755263 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:07.755321 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 116077us
I20260709 12:24:07.725975  7326 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:07.785821 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:07.738595  7225 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.787426 31825 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:07.807327 31825 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:07.794360  7295 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:07.795233  6862 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:07.796188  7310 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 22.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:07.797462 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:07.794222 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:07.799309 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:07.800112  7301 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:07.800242 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 67174.7us
I20260709 12:24:07.802650 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:07.808288 31822 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:07.810020 31825 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:07.815301 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:07.822342  7296 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:07.822746  7247 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:07.814808  7324 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:07.823868  7324 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:07.824297  7324 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:07.824434 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:07.827050 31825 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:07.826858  6858 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:07.804924  7326 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.848984  7320 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:07.811452  6862 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:07.813894  7301 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:07.852911 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 117263us
I20260709 12:24:07.814910  7295 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:07.808589  6752 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.041s
I20260709 12:24:07.833791  7326 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:07.855507  7334 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader election won for term 17
I20260709 12:24:07.863243  7334 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:07.864347  7334 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 17, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.866962  6862 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.868268  6862 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:07.869342  7289 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.857014  7301 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:07.882339 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57984.9us
I20260709 12:24:07.857918  7225 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:07.871690 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:07.883340 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:07.816766  6651 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.852679  7296 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:07.898319 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:07.899312 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 96808.6us
I20260709 12:24:07.870882  7201 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:07.909734 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 94510.3us
W20260709 12:24:07.896924  7332 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.050s	user 0.000s	sys 0.007s
I20260709 12:24:07.917971 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:07.920307 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:07.923364 31821 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:07.918733 31825 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:07.932823 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:07.932725 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:07.917546  7320 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:07.884096  7247 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:24:07.918128  7201 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.907269  7337 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:07.939788 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:07.921752  7045 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:07.948509  7320 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.956050 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:07.955425 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:07.956934 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46179.2us
I20260709 12:24:07.923557  7188 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.032s
I20260709 12:24:07.948959  7224 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:07.954228  7201 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:07.965631 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:07.965914 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:07.966584 31671 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:07.960712  7114 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:07.917728 31569 catalog_manager.cc:5697] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 9 to 11. New cstate: current_term: 11 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:07.967705  6858 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:07.956107  7045 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:07.975266 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50738.1us
I20260709 12:24:07.975658  7045 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:07.977205 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:07.968246 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:07.978554  7114 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:07.958868  7289 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:07.959107  7320 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:07.982115  6933 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:07.982537  6933 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:07.982910  6933 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:07.952016  7295 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.978924 31673 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:07.973855 31670 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:24:07.972640 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:07.964118  7316 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:07.996153 31674 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:07.963076  7220 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:07.963297  7224 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:08.002946  7336 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.003430  7114 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:08.014238 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:07.990175  7318 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:08.005332  7320 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.013089 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:07.965628  7306 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20260709 12:24:07.967753  6752 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 12:24:07.999718  7316 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.008215 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:08.015264 31674 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:08.027253 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:07.971938  7310 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:08.021518 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:08.023314  7352 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:08.017758 31671 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
W20260709 12:24:08.040846  7350 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.145s	user 0.000s	sys 0.001s
W20260709 12:24:08.042055  7344 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.204s	user 0.001s	sys 0.000s
I20260709 12:24:08.042382  7310 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.042819  7304 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:08.042843  7310 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:08.010020  7224 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:08.009248  7014 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:08.052301 31669 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:08.053864 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:07.964011  7296 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:07.994701  7106 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 12:24:08.021171  7318 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:08.080149 31673 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:08.048753  7316 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:08.086342 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 203081us
I20260709 12:24:08.087846 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:08.082592  7267 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:08.055498  7243 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.054s
W20260709 12:24:08.090464 31574 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42072 (request call id 244) took 31428 ms (client timeout 15000 ms). Trace:
W20260709 12:24:08.091495 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 76459.7us
I20260709 12:24:08.098716 31674 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:08.110607 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 102450us
I20260709 12:24:08.119367 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:08.022022  7336 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.110391  7045 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:08.171559 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.081089  7242 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.192659  7352 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:08.085561  7313 raft_consensus.cc:2764] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election decision vote started in defunct term 23: won
I20260709 12:24:08.186467  7308 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.198662  7352 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.221915  7224 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.224130  7308 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:08.200507  7359 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:08.106554  6933 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.225520  7312 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:08.226042  7312 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:08.226974  6933 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:08.088688  7339 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:08.115720 31574 rpcz_store.cc:269] 0709 12:23:36.662135 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.662568 (+   433us) service_pool.cc:224] Handling call
0709 12:24:08.090220 (+31427652us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:05.366370 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:05.367110 (+   740us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:05.367167 (+    57us) write_op.cc:435] Acquired schema lock
0709 12:24:05.367185 (+    18us) tablet.cc:662] Decoding operations
0709 12:24:05.380050 (+ 12865us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:05.380167 (+   117us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:05.380197 (+    30us) tablet.cc:685] Acquiring locks for 29 operations
0709 12:24:05.669173 (+288976us) tablet.cc:701] Row locks acquired
0709 12:24:05.669232 (+    59us) write_op.cc:260] PREPARE: finished
0709 12:24:05.669471 (+   239us) write_op.cc:270] Start()
0709 12:24:05.669678 (+   207us) write_op.cc:276] Timestamp: P: 1783599845669384 usec, L: 0
0709 12:24:05.669721 (+    43us) op_driver.cc:348] REPLICATION: starting
0709 12:24:05.737387 (+ 67666us) log.cc:844] Serialized 11449 byte log entry
0709 12:24:06.036004 (+298617us) op_driver.cc:464] REPLICATION: finished
0709 12:24:06.178151 (+142147us) write_op.cc:301] APPLY: starting
0709 12:24:06.196632 (+ 18481us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:06.545625 (+348993us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:06.545745 (+   120us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:06.576091 (+ 30346us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:06.576573 (+   482us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=58,key_file_lookups=58,delta_file_lookups=0,mrs_lookups=0
0709 12:24:06.576636 (+    63us) write_op.cc:312] APPLY: finished
0709 12:24:06.577293 (+   657us) log.cc:844] Serialized 252 byte log entry
0709 12:24:06.578687 (+  1394us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:06.611191 (+ 32504us) write_op.cc:454] Released schema lock
0709 12:24:06.611292 (+   101us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:06.611866 (+   574us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:06.611895 (+    29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":7714822,"reported_tablets":48,"spinlock_wait_cycles":79744,"child_traces":[["op",{"apply.queue_time_us":89484,"apply.run_cpu_time_us":45171,"apply.run_wall_time_us":444794,"cfile_cache_hit":118,"cfile_cache_hit_bytes":125289,"num_ops":29,"prepare.queue_time_us":831455,"prepare.run_cpu_time_us":15452,"prepare.run_wall_time_us":521206,"raft.queue_time_us":178451,"raft.run_cpu_time_us":7960,"raft.run_wall_time_us":126360,"replication_time_us":365989,"thread_start_us":491214,"threads_started":4,"wal-append.queue_time_us":23906,"wal-append.run_cpu_time_us":33233,"wal-append.run_wall_time_us":1817207}]]}
W20260709 12:24:08.228809 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 216379us
W20260709 12:24:08.229327 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 208711us
I20260709 12:24:08.231814 31674 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:08.207476 31569 catalog_manager.cc:5697] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 6 to 15, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 15 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:08.216168  7320 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:08.229736 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 172873us
I20260709 12:24:08.230931 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:08.216905  7336 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:08.217023 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 91115.9us
I20260709 12:24:08.239096 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:08.107813  7188 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.112s
I20260709 12:24:08.220433  7242 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.181874  7247 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:08.224180  7224 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:08.224912  7356 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.242s	user 0.001s	sys 0.001s
W20260709 12:24:08.199872 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 112117us
I20260709 12:24:08.114313  6651 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:08.236773 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 117571us
I20260709 12:24:08.246672 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:08.247900 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:08.234999 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:08.240108 31822 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:24:08.244616 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:08.239740 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:08.254647 31821 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:08.257529 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:08.259377 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:08.230686  7220 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:08.109170  7324 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.105880  7326 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:08.284421 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44777.2us
I20260709 12:24:08.285535  7326 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:08.286523  7369 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
I20260709 12:24:08.226971  7312 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.288529  7045 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:08.289156  7312 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:08.288887  7201 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.261350  7289 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:08.290210  7289 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:08.290822  7224 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:08.242290  7301 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.291321  7224 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:08.291234  7289 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 17.9, Last appended by leader: 9, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.271302  7220 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.293354  7301 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:08.245359  7247 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:08.274274  7304 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:08.276463  7242 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:08.269937  7297 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:08.283406  7324 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:08.283279 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:08.286873  7095 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:08.291924  7224 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:08.265445 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:08.241904  7364 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:08.260885  7225 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:08.269863 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:08.297078 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:08.253736  7319 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:08.260820  6933 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:08.321976 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:08.322357 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52620.4us
I20260709 12:24:08.301034  7247 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:08.318832 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:08.317356  7045 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:08.323974  7045 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:08.324362  7045 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:08.325076 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:08.317524 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60101.4us
W20260709 12:24:08.320299 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.290441  6651 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.029s
I20260709 12:24:08.262374  7310 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.328519 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:08.315402  7242 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:08.301092  7083 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.033s
I20260709 12:24:08.335289 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:08.366587 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:08.261961  7318 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.368855  7242 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.369827  7318 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:08.372089  7369 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:08.372900  7369 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.373413  7369 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:08.373459  7225 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:08.375283  7225 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:08.375654  7225 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:08.375982  7225 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:08.376844  7316 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.378073  7295 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:08.372198  7242 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:08.327716  7324 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:08.344324  7320 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:08.325642  6933 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.328334  7188 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:08.330869  7306 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.020s
I20260709 12:24:08.352471  7310 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:08.352339  7201 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:08.353756  6752 raft_consensus.cc:2764] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
W20260709 12:24:08.354364 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 89054.8us
I20260709 12:24:08.354945  7095 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:08.355279 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:08.360131 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:08.343716 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:08.334133  7339 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:08.391299 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:08.368850 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:08.327769  7364 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:08.332284  7359 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.029s
W20260709 12:24:08.390694 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 56616.1us
I20260709 12:24:08.412308 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:08.417764 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:08.418620 31674 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:08.386922 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:08.388109 31824 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
W20260709 12:24:08.394469 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50836.8us
I20260709 12:24:08.420709 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:08.417124 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:08.424196 31671 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: [22.13-22.13]
I20260709 12:24:08.443143 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:08.446249 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 91056.9us
I20260709 12:24:08.459532 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:08.387308  6933 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:08.379040  7324 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:08.386947  7319 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:08.391662  7095 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:08.378597  7295 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:08.463385 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:08.465072 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:08.471431 31669 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:08.471899 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 51306.8us
I20260709 12:24:08.472963 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:08.442552  7201 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:08.422991  7320 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:08.441439  7083 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:08.423660  6651 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:24:08.465874 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.466607  7319 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:08.469676 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 109631us
I20260709 12:24:08.483323 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:08.466656  7324 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:08.442024  7313 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:08.423548  7301 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:08.465378  7267 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:08.480479 31671 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
W20260709 12:24:08.480213 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.456296  7339 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:08.470222  7306 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:08.470620  7243 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:08.506307 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 48870.9us
I20260709 12:24:08.503270  7339 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.461447  7336 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.025s
I20260709 12:24:08.478497  7320 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:08.487286  7319 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:08.512598 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:08.526103 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:08.481331  7083 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:08.481411  7316 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:08.466029  7224 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.529284 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:08.529552 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 56725.4us
I20260709 12:24:08.496150  6651 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:08.530870 31751 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:08.532127 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:08.534339 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:08.501564  7308 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:08.536906 32370 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:24:08.537806 32371 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:08.537513 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:08.537717 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:08.538579 31753 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:08.478526  7045 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.507607  7297 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:08.506325  7267 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:08.546325 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 81321.5us
I20260709 12:24:08.482240  7201 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.527446  7320 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:08.551424 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:08.574213 32375 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:08.528388  6862 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:08.577831 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.527748  7359 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:08.588392 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:08.591254 31671 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:08.591945 31670 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:24:08.528162  7296 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:08.532503  7083 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:08.597810 32371 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:08.594036 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:08.595505 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44166.9us
I20260709 12:24:08.607980 31676 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:08.608460 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:08.608034 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:08.614352 31750 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:08.614660 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:08.608819 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:08.541056  7308 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:08.617632 31675 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:08.552434  7304 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:08.522666  7243 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.647511 31675 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: [21.14-23.15]
I20260709 12:24:08.565465  7316 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:08.641126 31746 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:24:08.648059 32372 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:08.627854 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:08.649966  7313 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:08.645246 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:08.589709  7306 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:08.600495  7315 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:24:08.579094  7297 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:08.651557 31752 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:08.651942 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:08.545006  7114 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.073s
I20260709 12:24:08.505268  7295 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.624147  7365 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:24:08.662639 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 93561.4us
I20260709 12:24:08.749927 31642 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:08.630321  7308 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:08.616808  7296 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.754518  7296 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:08.755240  7296 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:08.755818  7296 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:08.756783  7296 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.565150  7095 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.759677  7295 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:08.761961 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:08.762828 31746 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 28: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:08.763756 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:08.765552 31676 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 14. (term mismatch)
I20260709 12:24:08.765942 31676 pending_rounds.cc:85] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 13
I20260709 12:24:08.766232 31676 pending_rounds.cc:107] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20260709 12:24:08.766961 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:08.766752  6752 consensus_queue.cc:1175] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b95aeeaf4c51442b8092cff1de674f2b" responder_term: 23 status { last_received { term: 21 index: 12 } last_committed_idx: 12 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: [22.13-22.13]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:08.768903  7368 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.204s
I20260709 12:24:08.770886  7267 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.771703  7354 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.193s
I20260709 12:24:08.776605  7224 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:08.600366  6862 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.779287  7316 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:08.613160  7359 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.782232  7359 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:08.554615  7364 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.660595 31670 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:24:08.624363  7242 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:08.766507 31676 raft_consensus.cc:2889] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:08.704953 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:08.743716 31753 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:08.747210 31678 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 14 index: 10. Preceding OpId from leader: term: 16 index: 11. (index mismatch)
I20260709 12:24:08.747061  7313 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:08.783936 31749 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:24:08.786784 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:08.810426 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.821861 31670 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 12:24:08.820408 31669 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:08.771936  7308 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:08.827701 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 319050us
W20260709 12:24:08.797623 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 186535us
I20260709 12:24:08.850530 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:08.850374 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 269862us
I20260709 12:24:08.788371  7312 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:08.853299 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:08.858842 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:08.861795 31670 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:24:08.859439 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 346891us
I20260709 12:24:08.859498 32372 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:08.878607 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:08.872567 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:08.849673  7334 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.872s	user 0.013s	sys 0.000s
I20260709 12:24:08.888041 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:08.889160 31676 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:24:08.889561 31752 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:08.821972  7242 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:08.892954 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:08.907095 32378 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
W20260709 12:24:08.890254  7334 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.923s	user 0.014s	sys 0.000s
I20260709 12:24:08.836709  7368 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.059s
I20260709 12:24:08.922330 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:08.922829 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:08.897698  6862 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:08.932408 31670 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:24:08.939307 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:08.904606  7312 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:08.836509  6651 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:08.879022  7326 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.042s
I20260709 12:24:08.971580 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:08.974315 31677 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
W20260709 12:24:08.978173 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 105797us
I20260709 12:24:08.978605 31677 raft_consensus.cc:606] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:08.979081 31677 raft_consensus.cc:614] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:08.980690 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:08.977465 31752 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:08.933400  7336 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.043s
I20260709 12:24:08.945039  7354 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:08.997923  7353 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:08.941028  7389 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.074s
I20260709 12:24:08.998168  7243 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:08.999166 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 117160us
I20260709 12:24:09.006523 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:09.007083 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 62756.7us
I20260709 12:24:08.973212  7201 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:09.016235  7243 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.016820  7243 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:09.004242  7242 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:09.019236  7243 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:09.019222  7312 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:09.004768  7354 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.023075  7354 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:08.932955  7352 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.067s
I20260709 12:24:09.026675 31676 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: [24.14-26.15]
I20260709 12:24:09.027366 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:09.001145 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:09.032016 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:08.998198  7386 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.119s	user 0.001s	sys 0.000s
I20260709 12:24:09.040969 32371 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
W20260709 12:24:09.050518 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44042.5us
W20260709 12:24:09.022462  7318 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.652s	user 0.010s	sys 0.000s
W20260709 12:24:09.062119 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61062.5us
I20260709 12:24:09.049007  7336 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:09.068017  7336 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:09.069644 32371 raft_consensus.cc:1217] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Deduplicated request from leader. Original: 24.15->[25.16-25.16]   Dedup: 25.16->[]
I20260709 12:24:09.080459 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:09.081977 31752 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
W20260709 12:24:09.086540 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.087406 32371 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:24:08.922645  7297 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:09.096458 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:08.981320  7396 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.108s
W20260709 12:24:09.104332 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.106910 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:09.107931 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:09.112596 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:09.059307  7353 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:09.031173  7359 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.057s
I20260709 12:24:09.122174 31752 raft_consensus.cc:2410] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 26 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
W20260709 12:24:09.122357  7247 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.790s	user 0.002s	sys 0.009s
I20260709 12:24:09.122820  7247 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:09.124698 31672 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:09.087930  7389 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:09.128851 31751 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.577s	user 0.012s	sys 0.000s
W20260709 12:24:09.111191 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.129460 31751 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:24:09.043941  7378 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.069s
I20260709 12:24:09.099318  7301 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:09.144666 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:09.113358  7304 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.156973 31677 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:09.161999  7242 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:09.171514  7242 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:09.223829  6651 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.247290 31676 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:24:09.251114 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:09.251955 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:09.254769 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:09.144925  7308 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:09.363925 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:09.345588 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.317488 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.334918 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:09.335588 31677 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:09.258888 31676 raft_consensus.cc:606] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:09.353847 31678 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260709 12:24:09.426599 31676 raft_consensus.cc:614] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:09.157790  7301 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:09.422943 32374 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:09.423945 31745 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:24:09.435402 31672 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:09.360082  7247 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:09.311571  7386 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:24:09.359654  7394 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.322s	user 0.001s	sys 0.001s
W20260709 12:24:09.378264  7295 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.617s	user 0.010s	sys 0.000s
W20260709 12:24:09.373703  7319 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.835s	user 0.008s	sys 0.002s
W20260709 12:24:09.379837  7369 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.006s	user 0.011s	sys 0.003s
I20260709 12:24:09.461480 31678 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: []
W20260709 12:24:09.407680  6933 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.931s	user 0.006s	sys 0.006s
W20260709 12:24:09.412597  7225 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.036s	user 0.015s	sys 0.000s
I20260709 12:24:09.284102  7399 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:24:09.475049 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 140243us
I20260709 12:24:09.259929  7194 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:09.304579  7326 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.321977  7368 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.322734  7114 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:09.478974 31670 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:09.489436 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:09.465803  7392 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:09.466516  7308 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.492496 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:09.480381 31678 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [18.10-18.10]
I20260709 12:24:09.253255  7304 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:09.496106  7045 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:09.496531  7375 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.538s	user 0.000s	sys 0.001s
W20260709 12:24:09.480647  7324 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.981s	user 0.013s	sys 0.000s
I20260709 12:24:09.312906  7403 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.397s
I20260709 12:24:09.466938  7412 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:09.436551  6651 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:09.486114  7194 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:09.434353  7354 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:09.508245 31719 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.013s	sys 0.004s
W20260709 12:24:09.504554  7397 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:24:09.207078  7242 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:09.515564 31719 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.591s	user 0.013s	sys 0.005s
I20260709 12:24:09.506367 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:09.492864  7392 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:09.521584 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1084) took 1006 ms. Trace:
W20260709 12:24:09.521651 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 489693us
I20260709 12:24:09.494390  7399 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:09.481832  7225 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:09.322926  7353 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:09.486276  6933 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:09.522042 32373 rpcz_store.cc:276] 0709 12:24:08.514885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.515051 (+   166us) service_pool.cc:224] Handling call
0709 12:24:08.515416 (+   365us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.515685 (+   269us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.515729 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:08.528633 (+ 12904us) log.cc:844] Serialized 26 byte log entry
0709 12:24:08.832277 (+303644us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:08.832317 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.832399 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:09.521355 (+688956us) raft_consensus.cc:1685] finished
0709 12:24:09.521377 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:09.521524 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":245269,"threads_started":1,"wal-append.queue_time_us":340291}
I20260709 12:24:09.477851  7369 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:09.497756  7114 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:09.525477 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:09.527210 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:09.477613  7319 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:09.492708  7326 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:09.574725 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:09.528925 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:09.576421 31822 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:09.529443 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:09.576504 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:09.577093 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1088) took 1043 ms. Trace:
I20260709 12:24:09.577276 31674 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:09.577876 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:09.501583  7410 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:09.501601  7308 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:09.527334 31678 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:09.503902  7324 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:09.494445  7368 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:09.579555 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:09.579080 32370 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:09.577818 32375 rpcz_store.cc:276] 0709 12:24:08.533481 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.533614 (+   133us) service_pool.cc:224] Handling call
0709 12:24:08.564883 (+ 31269us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:08.579266 (+ 14383us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.576737 (+997471us) spinlock_profiling.cc:243] Waited 61.2 ms on lock 0x7b540053a820. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd8a02 00007fe8fcdd4dd6 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:24:09.577008 (+   271us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":140753664}
I20260709 12:24:09.585599 31671 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:24:09.586513 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 149535us
I20260709 12:24:09.513823  7045 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.032s
W20260709 12:24:09.727759 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 202349us
I20260709 12:24:09.581864  7289 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.718573 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:09.596768  7392 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:09.595746 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:24:09.719728 32376 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:09.718854 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:09.721127 31819 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
W20260709 12:24:09.728014 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.592603  7320 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.993s	user 0.012s	sys 0.002s
I20260709 12:24:09.722421 31824 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:09.729456 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:09.596294 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:09.738291 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:09.742697 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:09.787130 31670 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:09.744683 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:09.744858  6933 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:09.745980  7402 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.159s	user 0.000s	sys 0.001s
I20260709 12:24:09.746415  7289 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:09.748529  7369 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:09.750140  7225 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:09.751437 31820 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
W20260709 12:24:09.785457  7416 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.273s	user 0.000s	sys 0.010s
I20260709 12:24:09.752174 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:09.753369  7114 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:09.755376  7308 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:09.755936  6651 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:09.756242  7389 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:09.758129  7324 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:09.587998  7194 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:09.763063  7354 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:09.722209  7432 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
W20260709 12:24:09.764300  7413 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.272s	user 0.006s	sys 0.007s
I20260709 12:24:09.779512  7297 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:09.780429  7430 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.788841 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:09.789727 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:09.768644  7301 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:09.794270 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:09.731594  7083 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.103s	user 0.005s	sys 0.017s
I20260709 12:24:09.750861  7399 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:09.733028  7334 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:09.808063 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:09.732558  7410 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.808745 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:09.820434 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 91043.1us
I20260709 12:24:09.800972  7403 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.840268 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:09.862432 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 282936us
I20260709 12:24:09.797612  7320 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:09.804010  7392 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:09.847150 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:09.948539 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:09.929072 31825 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:24:10.029915 31675 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:10.030232 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:09.895097 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 174388us
I20260709 12:24:09.925894 31818 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: [21.10-21.10]
I20260709 12:24:10.033062 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:10.033883 31819 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:09.895002 31672 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:09.965063 31827 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:24:09.898429 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:09.896531 31826 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
W20260709 12:24:10.027369 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:09.920356 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:09.944779 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:09.928951 31677 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:24:10.025988 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:09.851722  7225 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.083613 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:10.042958 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.087488 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.085165 31678 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
W20260709 12:24:09.986812  7411 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.224s	user 0.004s	sys 0.008s
I20260709 12:24:10.089056 31826 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:10.094316 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:09.890290  7297 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:09.887717  7432 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:09.828047  7114 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:09.978967  7045 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:09.914973  6933 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.094471 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:10.098264 31825 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:10.098974 31819 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:09.893668  7389 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:10.106474 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:10.115608 31678 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:10.086269  7420 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.290s	user 0.001s	sys 0.009s
I20260709 12:24:10.124078 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2516) took 1293 ms. Trace:
I20260709 12:24:10.117707  7114 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.124776 31669 rpcz_store.cc:276] 0709 12:24:08.831071 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.857649 (+ 26578us) service_pool.cc:224] Handling call
0709 12:24:08.888937 (+ 31288us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.971189 (+ 82252us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.971250 (+    61us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:09.006371 (+ 35121us) log.cc:844] Serialized 26 byte log entry
0709 12:24:09.332763 (+326392us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:09.332823 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.332925 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.123798 (+790873us) raft_consensus.cc:1685] finished
0709 12:24:10.123824 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.124001 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37735,"raft.queue_time_us":39293,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":26987,"thread_start_us":244780,"threads_started":1,"wal-append.queue_time_us":360477}
I20260709 12:24:10.128147 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:10.128731  7114 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:10.107878 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 82013.2us
I20260709 12:24:09.985669  7304 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:10.112406  7378 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.129690  7114 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:10.106858  7432 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:09.870678  7407 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:09.845800  7308 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.259s
I20260709 12:24:10.133271  7320 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:10.108860 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:10.087670  7334 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:10.036715  7224 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:10.036192  7435 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:09.897096  7194 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:10.107188  7339 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:10.118830 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1135) took 1623 ms. Trace:
I20260709 12:24:10.107179  7297 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:10.087833  7319 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:10.108498 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.136533 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:10.137565 31678 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:24:10.184288 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:10.096300  7324 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.097714  7225 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:10.028709  7083 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.161118 31824 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:10.163869 31672 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:10.101157  7326 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:10.164803 31744 rpcz_store.cc:276] 0709 12:24:08.495270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.504654 (+  9384us) service_pool.cc:224] Handling call
0709 12:24:08.533298 (+ 28644us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:08.820974 (+287676us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.821024 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:08.868855 (+ 47831us) log.cc:844] Serialized 52 byte log entry
0709 12:24:09.130462 (+261607us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:09.130509 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.130585 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.118490 (+987905us) raft_consensus.cc:1685] finished
0709 12:24:10.118512 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.118761 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19903,"raft.queue_time_us":351922,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":8084,"thread_start_us":469654,"threads_started":2,"wal-append.queue_time_us":604210}
I20260709 12:24:10.158728  7368 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:10.192189 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:10.195494 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:10.198684 31672 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:10.135380  7045 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.149225  6933 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:10.188195  7289 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:10.210347  7426 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
W20260709 12:24:10.211793  7429 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
I20260709 12:24:10.215365 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:10.216176  7428 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:10.216284 31826 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:10.216910  7428 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:10.217417  7428 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:10.206274  7319 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:10.218500  7319 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:10.218789  7427 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
W20260709 12:24:10.220052  7422 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:24:10.215018  7334 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.222786 31819 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:24:10.165763  7224 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:10.219533  7319 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.226393 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:10.229247 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:10.169582  7339 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.232313 31819 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:10.236977 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:10.233754 31826 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:10.240113 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:10.241014 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 157852us
I20260709 12:24:10.160928  7435 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:09.928867  7430 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.247025 31819 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:10.036284  7410 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.237550 31678 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:24:10.194536  7324 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.248027 31826 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:10.248675 31825 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:10.255898 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:10.201414  7440 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.103s
I20260709 12:24:10.031246  7403 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.235402  7320 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.264374 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:10.271525 31821 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:10.131217  7378 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:10.235858  7441 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:10.264545 31819 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:10.263983  7424 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.278s	user 0.000s	sys 0.001s
I20260709 12:24:10.273497 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:10.274223  7224 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:10.274987  7083 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:10.276228  7045 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:10.277495 31676 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:24:10.265535 31826 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:10.280721 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:10.119691  7389 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.283600 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:10.291754 31819 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:10.291741 31672 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:10.293915 31827 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:24:10.296306 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 112121us
I20260709 12:24:10.244654  7326 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.039s
I20260709 12:24:10.304658 31669 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:10.305919 31678 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 9. Preceding OpId from leader: term: 19 index: 10. (index mismatch)
I20260709 12:24:10.307713  7353 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.308327  7366 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.313987  7318 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.314381  7289 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.314604  7405 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.331398 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:10.331648 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:10.313987 31825 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:24:10.315968 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:10.319861 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:10.323449 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:10.296454  7437 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:10.330338  7289 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:10.336941 31821 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:24:10.282413  7320 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:10.341243 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:10.331197  7318 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.344274  6651 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:10.296265  7441 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.349538  7289 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:10.322279  7359 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.020s
I20260709 12:24:10.321372  7045 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.014s
I20260709 12:24:10.342944 31825 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:10.305601  7368 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.036s
I20260709 12:24:10.332145  7326 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:10.323172  7440 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.019s
I20260709 12:24:10.346352  7318 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:10.377961 31821 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:10.355195 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:10.338621  7437 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:10.380424 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 109808us
I20260709 12:24:10.384750 31825 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:10.386349 31821 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:10.386574 31825 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:24:10.386926 31821 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:10.388322 31818 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:10.388728 31818 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:10.373416  7438 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:24:10.397904 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:10.398959  7318 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:10.399921  7365 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:10.400693  7453 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:10.358505  7326 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:10.335243  7353 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:10.383134  7437 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:10.401230  7399 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.014s
I20260709 12:24:10.416281  6651 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.001s
I20260709 12:24:10.416428  7326 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.418205  7326 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:10.407809  7289 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.012s
I20260709 12:24:10.420364  7353 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:10.423108  7398 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:10.334700  7366 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:10.414527 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:10.335100  7405 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:10.435551 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 119413us
I20260709 12:24:10.389633  7460 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.015s
I20260709 12:24:10.408746  7225 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:10.448447 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1143) took 1451 ms. Trace:
I20260709 12:24:10.424489  6933 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:10.461269 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:10.465701 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:10.438356  7398 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:10.457664 31752 rpcz_store.cc:276] 0709 12:24:09.000316 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.123008 (+122692us) service_pool.cc:224] Handling call
0709 12:24:09.123385 (+   377us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:09.123767 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:09.124097 (+   330us) log.cc:844] Serialized 14 byte log entry
0709 12:24:09.509001 (+384904us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:09.510466 (+  1465us) log.cc:844] Serialized 26 byte log entry
0709 12:24:09.510660 (+   194us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:09.521827 (+ 11167us) log.cc:844] Serialized 14 byte log entry
0709 12:24:09.530040 (+  8213us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:09.530179 (+   139us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.448118 (+917939us) raft_consensus.cc:1685] finished
0709 12:24:10.448151 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:10.448353 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":164221,"prepare.queue_time_us":14569,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":64831,"raft.run_cpu_time_us":435,"raft.run_wall_time_us":21016,"spinlock_wait_cycles":8476544,"thread_start_us":384303,"threads_started":1,"wal-append.queue_time_us":763358}
W20260709 12:24:10.457293  7444 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:24:10.454262  7224 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.456801  7289 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:10.507059 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:10.501345 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.479450  7450 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:24:10.467798  7453 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:10.474195  7365 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:10.486507  7304 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:10.459061  7353 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 17.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.079s
W20260709 12:24:10.554090 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58488us
I20260709 12:24:10.481503  7368 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:10.495913  7440 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:10.468528  7399 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.080s
I20260709 12:24:10.589906  7465 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.131s
I20260709 12:24:10.594805  7462 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:10.595414  7462 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:10.596426  7462 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:10.597903  7462 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:10.598961 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:10.480118  7430 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:10.544107  7224 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.604180  7378 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:10.604275  7432 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.605495  7435 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:10.484762  7437 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.543179  7243 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:24:10.602388  7392 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.555s	user 0.011s	sys 0.000s
I20260709 12:24:10.609550 32375 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:24:10.552296  7441 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:10.620487 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 80971us
I20260709 12:24:10.553644  7304 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:10.584507  7368 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:10.558722  7410 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:10.627676 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 70080.5us
I20260709 12:24:10.606523  7224 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:10.558343  7324 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:10.589401  7440 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:10.485610  7412 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.125s
I20260709 12:24:10.630952 31819 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 14. (term mismatch)
I20260709 12:24:10.608949  7432 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:10.613373  7243 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:10.587621  7403 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:10.639814 31744 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:10.650826 31819 pending_rounds.cc:85] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 13
I20260709 12:24:10.651415 31819 pending_rounds.cc:107] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 23.14
I20260709 12:24:10.640210  7320 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:10.641103  7465 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:10.643493 31746 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 20 index: 15. (index mismatch)
I20260709 12:24:10.552642  7289 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.074s
I20260709 12:24:10.623190  7295 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.651731 31819 raft_consensus.cc:2889] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:10.952358  7470 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.007s
I20260709 12:24:10.952721  7295 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:10.953298  7336 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.954948  7295 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:10.653290  6752 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:10.954389  7336 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:10.653177  7243 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:10.765861 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:10.787036  7368 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:10.762331  7304 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.642004  7297 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:10.760116  7463 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.137s
I20260709 12:24:11.015419 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1236) took 1178 ms. Trace:
I20260709 12:24:10.624228  7398 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.10, Last appended by leader: 10, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:10.872459 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:10.694875 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:10.946599  7466 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 17.9, Next index: 10, Last known committed idx: 7, Time since last communication: 0.001s
W20260709 12:24:10.947932 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 294092us
I20260709 12:24:11.023626 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:10.764081  7452 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:10.760056  7447 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.617074  7461 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.042s
I20260709 12:24:11.031101 32373 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:10.780658 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:10.950033  7432 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.994984  7366 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:10.997395 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:10.987854  6752 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:10.999001  7435 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:10.999130  7470 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:24:11.000494  7469 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.240s	user 0.002s	sys 0.010s
I20260709 12:24:11.001272 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3490) took 2453 ms. Trace:
I20260709 12:24:11.003165  7440 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.003988 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1083) took 2490 ms. Trace:
I20260709 12:24:11.041179 31674 rpcz_store.cc:276] 0709 12:24:08.548023 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.548593 (+   570us) service_pool.cc:224] Handling call
0709 12:24:08.548944 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.578341 (+1029397us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.000912 (+1422571us) spinlock_profiling.cc:243] Waited 1.03 s on lock 0x7b54002549d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:11.001164 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2364518144}
I20260709 12:24:11.005584 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3497) took 1138 ms. Trace:
I20260709 12:24:11.006004 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:11.007129  7475 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:24:11.043555 32376 rpcz_store.cc:276] 0709 12:24:08.514378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.527660 (+ 13282us) service_pool.cc:224] Handling call
0709 12:24:08.527986 (+   326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:09.738258 (+1210272us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.003689 (+1265431us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b5400546354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:11.003895 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2445821824}
I20260709 12:24:11.043936 31675 rpcz_store.cc:276] 0709 12:24:09.866989 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.986004 (+1119015us) service_pool.cc:224] Handling call
0709 12:24:10.987751 (+  1747us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.005177 (+ 17426us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:11.005213 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:11.005268 (+    55us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:11.005306 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.005384 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.005517 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17010,"raft.queue_time_us":28243,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":10941}
W20260709 12:24:11.008232  7468 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:24:11.010844  7437 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:11.013481  7194 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.848s	user 0.012s	sys 0.020s
I20260709 12:24:11.013739 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.058135  6752 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:11.059192  7391 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.274s	user 0.004s	sys 0.000s
I20260709 12:24:11.015667 31752 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:24:11.060846  7384 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:11.061591  7384 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:10.676589 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.063894  7384 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:11.064837  7445 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:11.066639 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:11.068464 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:11.069361 31819 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:11.070942  7440 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:11.071121 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:11.072018 31822 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:11.075189 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3496) took 1226 ms. Trace:
I20260709 12:24:11.075779 31676 rpcz_store.cc:276] 0709 12:24:09.849228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.328076 (+478848us) service_pool.cc:224] Handling call
0709 12:24:10.335715 (+  7639us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:10.336086 (+   371us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:10.336114 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:10.352325 (+ 16211us) log.cc:844] Serialized 26 byte log entry
0709 12:24:10.463941 (+111616us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:10.463995 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.464081 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:11.007449 (+543368us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:11.074736 (+ 67287us) raft_consensus.cc:1685] finished
0709 12:24:11.074772 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.075101 (+   329us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15488,"raft.queue_time_us":107,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":3797,"spinlock_wait_cycles":23168256,"thread_start_us":60627,"threads_started":1,"wal-append.queue_time_us":252401}
W20260709 12:24:11.009534  7428 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.792s	user 0.017s	sys 0.003s
I20260709 12:24:11.035197 31744 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:11.078965 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.079573 31744 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:11.026870 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.041106  7435 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.078424  7428 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.041216  7470 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:11.083458  7470 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.084168  7428 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:11.084800 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.031400  7425 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:11.075205  7437 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:11.048933 31749 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:11.085619  7425 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.087791 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.089341 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.090344 31749 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 27 because replica has last-logged OpId of term: 26 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
I20260709 12:24:11.048844 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.046087 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:11.090991 31746 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
W20260709 12:24:11.091197 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 45147.7us
I20260709 12:24:11.092093 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.033114  7465 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.093154 31746 raft_consensus.cc:2410] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:24:11.048940 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:11.033468  7304 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:11.094062 32376 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:11.095503 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.096478 31746 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:11.096673 31674 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:11.097069 31674 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:11.097460 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.060937 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1137) took 2558 ms. Trace:
I20260709 12:24:11.099390 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:11.099506  7435 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:11.098403 31748 rpcz_store.cc:276] 0709 12:24:08.502560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.510934 (+  8374us) service_pool.cc:224] Handling call
0709 12:24:08.536950 (+ 26016us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.537182 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.537207 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:08.578681 (+ 41474us) log.cc:844] Serialized 26 byte log entry
0709 12:24:08.863579 (+284898us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:08.903953 (+ 40374us) log.cc:844] Serialized 14 byte log entry
0709 12:24:08.989448 (+ 85495us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.994598 (+  5150us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:11.030013 (+2035415us) spinlock_profiling.cc:243] Waited 5.39 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:11.060596 (+ 30583us) raft_consensus.cc:1685] finished
0709 12:24:11.060695 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.060859 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27725,"prepare.queue_time_us":44602,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":9647,"raft.queue_time_us":112330,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":107,"spinlock_wait_cycles":12387200,"thread_start_us":264047,"threads_started":2,"wal-append.queue_time_us":905309}
I20260709 12:24:11.100333 32376 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:24:11.031666  7452 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.100935 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.101462 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:11.102093 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.102476  7452 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:11.092933  7465 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.104832  7465 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:11.051508 31675 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
W20260709 12:24:11.095763  7326 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.657s	user 0.009s	sys 0.013s
I20260709 12:24:11.105192 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:11.105677 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1244) took 1001 ms. Trace:
I20260709 12:24:11.106122 31675 rpcz_store.cc:276] 0709 12:24:10.103868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.051137 (+947269us) service_pool.cc:224] Handling call
0709 12:24:11.051387 (+   250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.105423 (+ 54036us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.105624 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.084689  7470 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:11.112362 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.116663  7297 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:11.120644 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2510) took 2514 ms. Trace:
I20260709 12:24:11.121266 31671 rpcz_store.cc:276] 0709 12:24:08.606021 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.607342 (+  1321us) service_pool.cc:224] Handling call
0709 12:24:11.120337 (+2512995us) spinlock_profiling.cc:243] Waited 970 ms on lock 0x7b54001b35d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:11.120525 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2230704384}
I20260709 12:24:11.100461  7435 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:11.122905  7435 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:11.123370  7435 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
W20260709 12:24:11.124385  7415 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.393s	user 0.001s	sys 0.001s
I20260709 12:24:11.125757 31748 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 24 because replica has last-logged OpId of term: 23 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 17.
I20260709 12:24:11.127213 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.127334 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3485) took 2709 ms. Trace:
I20260709 12:24:11.128234  7433 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:11.128531 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.128688  7433 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:11.129702  7433 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.097904 31674 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.130790  7433 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:11.130982 31674 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:11.042766  7368 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:11.020052 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.134382 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.127796 31673 rpcz_store.cc:276] 0709 12:24:08.418296 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.418802 (+   506us) service_pool.cc:224] Handling call
0709 12:24:08.419187 (+   385us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.595881 (+176694us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:08.595912 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:08.640723 (+ 44811us) log.cc:844] Serialized 26 byte log entry
0709 12:24:09.099119 (+458396us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:09.099177 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.113906 (+2014729us) spinlock_profiling.cc:243] Waited 65.7 ms on lock 0x7b54000465d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:11.113975 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:11.123506 (+  9531us) spinlock_profiling.cc:243] Waited 22.1 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:11.127093 (+  3587us) raft_consensus.cc:1685] finished
0709 12:24:11.127116 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.127270 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9979,"raft.queue_time_us":132870,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":21229,"spinlock_wait_cycles":202078976,"thread_start_us":408241,"threads_started":2,"wal-append.queue_time_us":790421}
I20260709 12:24:11.135466 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.137059 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.137672  7390 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:11.139585 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.036526  7432 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.141927 31673 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:11.142087  7390 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.143059 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:11.070334 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:11.156090 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.157660  7301 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:11.158748 32370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 1139) took 1889 ms (client timeout 1548 ms). Trace:
W20260709 12:24:11.159302 32370 rpcz_store.cc:269] 0709 12:24:09.255087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.478182 (+223095us) service_pool.cc:224] Handling call
0709 12:24:11.144493 (+1666311us) spinlock_profiling.cc:243] Waited 39.7 ms on lock 0x7b540055da54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:11.144733 (+   240us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":91362176}
I20260709 12:24:11.160225 31824 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:11.160724 31824 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:11.161149 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:11.161609 31824 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.158380 31673 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: [23.13-23.13]
W20260709 12:24:11.166807 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0509674830000222s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1597s for a response
I20260709 12:24:11.102527 32376 raft_consensus.cc:2410] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:24:11.124444 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.174033 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:11.146476 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:11.138306 31751 raft_consensus.cc:2410] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 23 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:24:11.175143 31819 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 20: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:11.062114 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:11.171291  7431 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.392s	user 0.000s	sys 0.003s
I20260709 12:24:11.085875  7437 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.151405 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1311) took 1753 ms. Trace:
I20260709 12:24:11.176533 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.177608 32374 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:11.178606 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.016269 31670 rpcz_store.cc:276] 0709 12:24:09.837385 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.838751 (+  1366us) service_pool.cc:224] Handling call
0709 12:24:09.915840 (+ 77089us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:09.916358 (+   518us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:09.916400 (+    42us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:09.966804 (+ 50404us) log.cc:844] Serialized 78 byte log entry
0709 12:24:10.230126 (+263322us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:10.337579 (+107453us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.341137 (+  3558us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.355916 (+ 14779us) log.cc:844] Serialized 14 byte log entry
0709 12:24:10.356093 (+   177us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.356168 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.962206 (+606038us) raft_consensus.cc:1685] finished
0709 12:24:10.962240 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.015240 (+ 53000us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30580,"prepare.queue_time_us":525707,"prepare.run_cpu_time_us":84,"prepare.run_wall_time_us":89,"raft.queue_time_us":19297,"raft.run_cpu_time_us":472,"raft.run_wall_time_us":23366,"thread_start_us":283627,"threads_started":2,"wal-append.queue_time_us":318744}
I20260709 12:24:11.034793  7447 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:11.183744 32377 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 1087) took 2656 ms (client timeout 1729 ms). Trace:
I20260709 12:24:11.191570 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:11.150978 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.186767 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:11.192921 32377 rpcz_store.cc:269] 0709 12:24:08.515851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.515985 (+   134us) service_pool.cc:224] Handling call
0709 12:24:08.627675 (+111690us) spinlock_profiling.cc:243] Waited 1.01 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:24:11.172215 (+2544540us) spinlock_profiling.cc:243] Waited 789 ms on lock 0x7b54000127d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:11.172383 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1817537536}
I20260709 12:24:11.194926 32370 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:11.151842  7392 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.151986  7378 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.200141 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:11.204193 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.206172 31674 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:24:11.178984 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:11.186908 31822 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:11.187984 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:11.207926 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.193714 31819 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:11.210055 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1312) took 1712 ms. Trace:
I20260709 12:24:11.210456 32370 rpcz_store.cc:276] 0709 12:24:09.498047 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.169410 (+1671363us) service_pool.cc:224] Handling call
0709 12:24:11.194707 (+ 25297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.209797 (+ 15090us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.209998 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.211362 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.213660 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:11.167213  7384 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:11.214895  7378 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.289863  7378 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:11.194658  7462 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.104s
I20260709 12:24:11.291920 31819 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:11.189853 32369 rpcz_store.cc:276] 0709 12:24:09.380024 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.839094 (+459070us) service_pool.cc:224] Handling call
0709 12:24:09.839554 (+   460us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:09.843913 (+  4359us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:09.843960 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:09.844607 (+   647us) log.cc:844] Serialized 26 byte log entry
0709 12:24:10.987317 (+1142710us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b1800004ba0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:24:11.031199 (+ 43882us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:11.031255 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.031533 (+   278us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:11.133414 (+101881us) raft_consensus.cc:1685] finished
0709 12:24:11.133438 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.133646 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3873,"raft.queue_time_us":74699,"raft.run_cpu_time_us":172,"raft.run_wall_time_us":19806,"spinlock_wait_cycles":37729536,"thread_start_us":43354,"threads_started":1,"wal-append.queue_time_us":1136725}
I20260709 12:24:11.296535 32369 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
W20260709 12:24:11.298430 32375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 1141) took 2012 ms (client timeout 1776 ms). Trace:
I20260709 12:24:11.298547 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:11.286504 32372 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: []
W20260709 12:24:11.300966 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.250335 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.302428 31746 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:11.302980 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 91697.4us
I20260709 12:24:11.234546 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:11.303826 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:11.255882 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 77034.4us
W20260709 12:24:11.299027 32375 rpcz_store.cc:269] 0709 12:24:09.286188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.616766 (+330578us) service_pool.cc:224] Handling call
0709 12:24:11.298224 (+1681458us) spinlock_profiling.cc:243] Waited 802 ms on lock 0x7b54004312d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:11.298353 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1844991744}
I20260709 12:24:11.305440 32377 raft_consensus.cc:2410] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 17 because replica has last-logged OpId of term: 16 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 14 index: 10.
I20260709 12:24:11.305848 32372 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:11.184306  7301 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.215492 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.318568 31751 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:11.304795 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:11.233465 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.312049 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.315264 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:11.307266 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 355570us
I20260709 12:24:11.320815 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.322899 31670 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:11.325726 31744 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:11.325840 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.328734 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.326099 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.331655 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3459) took 1071 ms. Trace:
I20260709 12:24:11.334363 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.336257 31674 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:11.340906 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.346336 31674 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
W20260709 12:24:11.300566  7385 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260709 12:24:11.305459  7461 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:11.380784 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:11.362684  7417 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.834s	user 0.001s	sys 0.001s
I20260709 12:24:11.306770  7481 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:11.355314 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.407465 31676 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:11.301791  7392 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.356428 31670 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:24:11.215876  7445 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:11.217360  7403 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:11.187141  7465 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.157s
I20260709 12:24:11.349578  7464 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:11.187140  7353 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.157s
I20260709 12:24:11.343133  7412 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:11.328325  7384 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.356907  7432 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:11.342999  7114 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:11.331492  7301 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.359557  7390 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:11.359908  7339 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:11.361945 31821 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.362666  6752 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.013s
I20260709 12:24:11.343400  7378 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:11.332208 31827 rpcz_store.cc:276] 0709 12:24:10.259773 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.340945 (+ 81172us) service_pool.cc:224] Handling call
0709 12:24:10.342011 (+  1066us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:10.382985 (+ 40974us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:10.383390 (+   405us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:10.389742 (+  6352us) log.cc:844] Serialized 26 byte log entry
0709 12:24:10.485278 (+ 95536us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:10.485407 (+   129us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.485632 (+   225us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.999570 (+513938us) spinlock_profiling.cc:243] Waited 1.2 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:11.303228 (+303658us) raft_consensus.cc:1685] finished
0709 12:24:11.303249 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.331564 (+ 28315us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47472,"raft.queue_time_us":125217,"raft.run_cpu_time_us":226,"raft.run_wall_time_us":62966,"spinlock_wait_cycles":2769792,"thread_start_us":125350,"threads_started":2,"wal-append.queue_time_us":232288}
I20260709 12:24:11.367375 32378 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:11.368415 31746 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:11.441664  7412 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.456813  7412 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:11.457207 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.457978 31746 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:11.368727 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.355314 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:11.463055 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1309) took 2452 ms. Trace:
I20260709 12:24:11.353657 31674 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:24:11.306422  7304 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:11.463521 32378 rpcz_store.cc:276] 0709 12:24:09.010200 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.010526 (+   326us) service_pool.cc:224] Handling call
0709 12:24:11.366822 (+2356296us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:24:11.367115 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.462765 (+ 95650us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.462985 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5852928}
I20260709 12:24:11.465360 31821 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:11.467661  7412 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.354166 31748 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:11.325048  7462 consensus_queue.cc:1243] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 19.10 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 9
I20260709 12:24:11.388336  7437 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:11.392284  7461 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.371243  7289 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 10, Time since last communication: 0.024s
I20260709 12:24:11.370769  7441 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:11.356031  7243 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:11.382043  7388 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.625s	user 0.042s	sys 0.088s
I20260709 12:24:11.374358  7336 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.377544  7320 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:11.428311 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.435995 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.431114 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:11.351994  7453 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.064s
I20260709 12:24:11.531296 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.531944 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:11.441995 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:11.443151 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 324971us
I20260709 12:24:11.534516 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:11.444546 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.383122  7477 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.056s
I20260709 12:24:11.424736  7445 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:11.433004  7365 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.428284  7447 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:11.453745 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.472086  7384 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:11.453507 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 1.16282e+06us
I20260709 12:24:11.549425 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:11.552628 31642 leader_election.cc:336] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.534s (SENT)
I20260709 12:24:11.552973 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:11.456159 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.480783 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:11.468482 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:11.472729 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:11.470788 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.553923 31749 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:11.555059 32371 raft_consensus.cc:2410] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 27 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 12:24:11.469345 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:11.451509  7353 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.054s
I20260709 12:24:11.528555 31821 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 13. (term mismatch)
I20260709 12:24:11.529678 31751 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:11.476014  7464 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
W20260709 12:24:11.523386  7388 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.770s	user 0.042s	sys 0.089s
I20260709 12:24:11.530478 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:11.561698 31822 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:11.561075 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:11.406936  7481 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:11.544623  7445 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:11.479859  7471 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.842s	user 0.000s	sys 0.001s
I20260709 12:24:11.537146 31746 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
W20260709 12:24:11.540856 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 99930.3us
I20260709 12:24:11.483137  7318 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:24:11.454834 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 81312.3us
I20260709 12:24:11.547484 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.518594  7441 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:11.553684 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.554778 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 127623us
W20260709 12:24:11.577474 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.555732 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:11.556418 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 342823us
I20260709 12:24:11.557512 31674 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:24:11.558917 31642 leader_election.cc:336] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.543s (SENT)
I20260709 12:24:11.532379  7364 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:11.561075 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:11.561836 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:11.579806 31827 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:11.579989 31642 connection.cc:256] RPC call timeout handler was delayed by -1.2134116230004111s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1728s for a response
I20260709 12:24:11.580564 31676 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:11.563280 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.542757  7462 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:11.474416  7393 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 2.493s	user 0.003s	sys 0.001s
I20260709 12:24:11.537846  7432 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:11.566965 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:11.593472  7432 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.595983 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.597306 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.597707  7432 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:11.549052  7319 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:11.598495 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:11.599123  7488 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260709 12:24:11.599383 31753 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:11.599706 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:11.600298 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.601619  7378 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:11.602061 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.603029  7378 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:11.603636 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.604830 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.604941 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.605451  7243 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:11.570788 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.571429 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.607008 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:11.607626 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.576122  7318 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:11.608311  7491 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:11.608865 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:11.609093  7430 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:11.609174  7318 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.606212 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.609668  7430 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:11.609706  7447 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:11.610371 31745 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:11.611218 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.611510 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.612133  7447 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:11.612612  7425 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:11.612744 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 19 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.613929 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.612664  7447 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:11.527956  7461 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.618561 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:11.554733  7354 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
I20260709 12:24:11.626168  7301 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:11.554160  7365 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.627857  7398 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:11.628394  7365 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:11.607625  7464 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:11.629072  7339 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:11.629559  7405 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:11.630051  7396 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:11.561347 31821 pending_rounds.cc:85] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 12
I20260709 12:24:11.583777 32375 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:24:11.631886 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 1140) took 2370 ms. Trace:
I20260709 12:24:11.632141 31821 pending_rounds.cc:107] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 25.13
W20260709 12:24:11.632556 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.582490  7289 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:11.584833 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.583510  7441 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:11.592458 31751 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
W20260709 12:24:11.580513 31642 connection.cc:256] RPC call timeout handler was delayed by -1.1918116230003761s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17520000000000002s for a response
I20260709 12:24:11.536787  7336 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.615581 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:11.583508  7364 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:11.615299  7462 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.625149 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1141) took 3113 ms. Trace:
I20260709 12:24:11.599152  7319 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:11.632584 31821 raft_consensus.cc:2889] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:11.656471 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.656951 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.660694 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.663228 31822 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:11.661566 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3311) took 1352 ms. Trace:
I20260709 12:24:11.570251 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:11.632355 32373 rpcz_store.cc:276] 0709 12:24:09.261931 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:09.788554 (+526623us) service_pool.cc:224] Handling call
0709 12:24:11.631654 (+1843100us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b5400427f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:11.631822 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2619773824}
I20260709 12:24:11.566967  7481 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:11.609586  7320 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:11.603767  7378 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:11.637239 32372 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:24:11.634193 31749 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:11.671897 31678 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:11.672302 32375 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:11.672477 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.664429 31821 rpcz_store.cc:276] 0709 12:24:10.309645 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.384676 (+ 75031us) service_pool.cc:224] Handling call
0709 12:24:10.386216 (+  1540us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.361498 (+975282us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x7b4c0003cd00. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb0a1a20 00007fe8fb0b2a29 00007fe8fb078644 00007fe8fb09ff2d 00007fe8fb09fb93 00007fe8fcdd018b 00007fe8fcdd0cb5 00007fe8fcdcbf30 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416
0709 12:24:11.661187 (+299689us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.661497 (+   310us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39453568}
W20260709 12:24:11.640257 31642 connection.cc:256] RPC call timeout handler was delayed by -1.1765116230003514s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1769s for a response
I20260709 12:24:11.643262 31676 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
W20260709 12:24:11.673381 31642 connection.cc:256] RPC call timeout handler was delayed by -1.1567116230003194s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1791s for a response
W20260709 12:24:11.675024 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.675379 31642 connection.cc:256] RPC call timeout handler was delayed by -1.0820116230001986s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1874s for a response
W20260709 12:24:11.675787 31642 connection.cc:256] RPC call timeout handler was delayed by -1.0640116230001695s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1894s for a response
I20260709 12:24:11.582074 31748 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:11.629640  7339 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:11.646782 31747 rpcz_store.cc:276] 0709 12:24:08.511651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.511787 (+   136us) service_pool.cc:224] Handling call
0709 12:24:08.512126 (+   339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:08.528862 (+ 16736us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:08.551685 (+ 22823us) log.cc:844] Serialized 14 byte log entry
0709 12:24:08.904925 (+353240us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:08.905904 (+   979us) log.cc:844] Serialized 26 byte log entry
0709 12:24:08.906102 (+   198us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:08.920835 (+ 14733us) log.cc:844] Serialized 14 byte log entry
0709 12:24:08.923552 (+  2717us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:08.923663 (+   111us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:09.371295 (+447632us) spinlock_profiling.cc:243] Waited 9.27 ms on lock 0x7b18002d95e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:11.473811 (+2102516us) spinlock_profiling.cc:243] Waited 8.19 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:11.624938 (+151127us) raft_consensus.cc:1685] finished
0709 12:24:11.624966 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.625088 (+   122us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27936,"prepare.queue_time_us":50259,"prepare.run_cpu_time_us":81,"prepare.run_wall_time_us":83,"raft.queue_time_us":281157,"raft.run_cpu_time_us":454,"raft.run_wall_time_us":33738,"spinlock_wait_cycles":41771520,"thread_start_us":241190,"threads_started":2,"wal-append.queue_time_us":299557}
I20260709 12:24:11.677677 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:11.679988 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.680931 31748 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:24:11.656198 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:11.665601 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.683171 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:11.685981 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 101222us
I20260709 12:24:11.687065 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3456) took 1428 ms. Trace:
W20260709 12:24:11.655843  7435 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.531s	user 0.008s	sys 0.004s
I20260709 12:24:11.621652 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.685861 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:11.691282 31819 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:11.692237 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:11.673602 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:11.692855 31819 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:11.688935 31823 rpcz_store.cc:276] 0709 12:24:10.258860 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.336279 (+ 77419us) service_pool.cc:224] Handling call
0709 12:24:10.353193 (+ 16914us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:10.398536 (+ 45343us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:10.398574 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:10.399174 (+   600us) log.cc:844] Serialized 26 byte log entry
0709 12:24:10.463526 (+ 64352us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:10.463581 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:10.463678 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:10.770547 (+306869us) spinlock_profiling.cc:243] Waited 16.7 ms on lock 0x7b4800130158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:24:11.479197 (+708650us) spinlock_profiling.cc:243] Waited 1.39 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:11.686783 (+207586us) raft_consensus.cc:1685] finished
0709 12:24:11.686811 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:11.686991 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":176992,"raft.queue_time_us":120414,"raft.run_cpu_time_us":2588,"raft.run_wall_time_us":290444,"spinlock_wait_cycles":41548800,"thread_start_us":91553,"threads_started":2,"wal-append.queue_time_us":157948}
I20260709 12:24:11.689698 31751 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:24:11.694046 31753 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:11.694562 31821 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:11.652007  7477 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:11.697822 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.686043 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.688232 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.700306 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.701086 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:11.677443  7463 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260709 12:24:11.673919 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.684110 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:11.694818 31748 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: [20.15-22.16]
I20260709 12:24:11.631578  7447 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.707132 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:11.707659 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.707888 31823 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:11.659914 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.703209 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:11.703812 31676 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 13. (index mismatch)
I20260709 12:24:11.710944 31753 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:11.712189 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2711) took 1079 ms. Trace:
W20260709 12:24:11.713165 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 53345.4us
I20260709 12:24:11.706643 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2709) took 1074 ms. Trace:
I20260709 12:24:11.713852 31748 rpcz_store.cc:276] 0709 12:24:10.632070 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.681830 (+1049760us) service_pool.cc:224] Handling call
0709 12:24:11.682322 (+   492us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:11.706232 (+ 23910us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.706547 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.713217  7301 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:11.715034 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:11.678819  7378 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:11.694479  7399 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:11.712921 31753 rpcz_store.cc:276] 0709 12:24:10.632594 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.694830 (+1062236us) service_pool.cc:224] Handling call
0709 12:24:11.707598 (+ 12768us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.711855 (+  4257us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.712119 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.721289 31675 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:11.717123 31819 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:11.726552 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:11.716174 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:11.727451 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:11.728279 31822 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: [22.14-24.15]
I20260709 12:24:11.728386 31825 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:11.704185 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2710) took 1061 ms. Trace:
I20260709 12:24:11.729445 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.729619 31751 rpcz_store.cc:276] 0709 12:24:10.632348 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.683446 (+1051098us) service_pool.cc:224] Handling call
0709 12:24:11.689299 (+  5853us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:11.693974 (+  4675us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.694195 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.729499 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:11.707291 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.734002  7354 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.734576  7354 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:11.735975 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52005.3us
I20260709 12:24:11.703048  7477 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:11.776489 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 47074.9us
I20260709 12:24:11.775643 31819 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: [22.12-22.12]
I20260709 12:24:11.778362 31819 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:24:11.777936 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:11.740314 31747 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260709 12:24:11.779721 31748 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: []
W20260709 12:24:11.780155 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 43731us
I20260709 12:24:11.780378 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2713) took 1147 ms. Trace:
I20260709 12:24:11.780953 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.654111  7430 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.791230 31819 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:11.778918 31821 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:11.775406  7320 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:11.807763 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.694255  7435 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.704728  7428 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.080s
I20260709 12:24:11.727427  7480 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:11.810110 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:11.720269  7464 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:11.804603  7433 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:11.780735 31748 rpcz_store.cc:276] 0709 12:24:10.632952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.714646 (+1081694us) service_pool.cc:224] Handling call
0709 12:24:11.779530 (+ 64884us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.780143 (+   613us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.780324 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.826822 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:11.707123  7243 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.105s
I20260709 12:24:11.779765 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2712) took 1147 ms. Trace:
I20260709 12:24:11.874759 31747 rpcz_store.cc:276] 0709 12:24:10.632789 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.708628 (+1075839us) service_pool.cc:224] Handling call
0709 12:24:11.739937 (+ 31309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:11.779509 (+ 39572us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.779712 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.727808  7365 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.104s
I20260709 12:24:11.824256  7440 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:11.782815  7304 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:11.783483  7477 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:11.705422  7364 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.889417 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.831300  7378 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.774974  7494 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.147s
W20260709 12:24:11.877446 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 140568us
I20260709 12:24:11.914525 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:11.915302 31747 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:11.878429 31825 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 17. Current term is 18. Ops: [17.12-17.12]
I20260709 12:24:11.834776 31821 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:11.879714 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:11.812350 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.831601  7412 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:11.889417  7492 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.196s	user 0.001s	sys 0.000s
I20260709 12:24:11.845942  7480 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:11.890734  7354 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:11.898059 31748 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:11.837113  7465 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:11.842619  7430 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:11.921226 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 186300us
I20260709 12:24:11.958230 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:11.923146  7433 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:11.925066 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.927685 31675 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:24:11.934247 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 104630us
I20260709 12:24:11.945067 31821 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 16. (term mismatch)
I20260709 12:24:11.962082 31821 pending_rounds.cc:85] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 15
I20260709 12:24:11.962410 31821 pending_rounds.cc:107] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 22.16
I20260709 12:24:11.921800  7336 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:11.962677 31821 raft_consensus.cc:2889] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:11.876749  7114 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.054s
I20260709 12:24:11.963577 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:11.992161 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.965106 31826 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:11.980038  7407 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:24:11.994798 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:11.946444  7412 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:11.993718 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.999708 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 14 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:12.003206 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.003183 31675 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:12.006197 31747 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:24:12.007427 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:12.008278 31747 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:11.836810  7319 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.008816 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:12.006080 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:12.013206 31675 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:24:12.014191 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:12.014732 31675 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:12.006680 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:11.832898  7339 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:11.954109  7461 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:11.991501  7354 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:11.958961  7378 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:12.017369 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:12.011543 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48926.1us
I20260709 12:24:11.962774  7462 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:12.026310 31752 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 28 because replica has last-logged OpId of term: 26 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
I20260709 12:24:12.027015 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:12.026213 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.028824 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:12.030575 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:11.998965  7301 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:12.007735  7465 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.043478 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:12.044342 31747 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:12.012697  7392 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:12.056182 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.057446 31752 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:12.058080 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2717) took 1424 ms. Trace:
I20260709 12:24:12.058386 31752 rpcz_store.cc:276] 0709 12:24:10.633734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.056914 (+1423180us) service_pool.cc:224] Handling call
0709 12:24:12.057260 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.057834 (+   574us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.058027 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:11.996196  7403 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.044s
I20260709 12:24:11.995927  7433 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.017976  7114 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:12.051982 31827 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:12.019164  7313 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.034s
I20260709 12:24:12.020085  7365 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.022s
I20260709 12:24:12.023383  7504 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:12.051134  7389 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:12.004418  7437 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.039s
I20260709 12:24:12.059492 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:12.008683  7412 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.088574 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:12.093436  7498 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.165s	user 0.001s	sys 0.000s
I20260709 12:24:12.072305 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:12.106408 31747 raft_consensus.cc:2410] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 24 because replica has last-logged OpId of term: 22 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:24:12.050653  7425 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:24:12.097852 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 136449us
I20260709 12:24:12.067337  7462 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:12.104877 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:12.112396 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:24:12.117147 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.116784 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:12.037361  7354 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.087733  7493 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:12.097573  7114 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:12.128388 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 101413us
I20260709 12:24:12.055536  7465 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.130281 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:12.114256  7461 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.114890  7452 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:12.166127 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:12.167098 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:12.119207  7389 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:12.206190 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 150065us
I20260709 12:24:12.129215  7501 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:12.128427  7425 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:12.132442  7354 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:12.128867  7435 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:12.219444 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.169174  7452 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:12.226112  7502 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.227s	user 0.001s	sys 0.000s
I20260709 12:24:12.210567  7461 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.227895  7465 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:12.228956  7433 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:12.221055  7496 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.282s	user 0.001s	sys 0.000s
I20260709 12:24:12.212395  7304 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.232303 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3320) took 1165 ms. Trace:
I20260709 12:24:12.212442  7511 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:12.231598  7452 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:12.254957  7313 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:12.257036  7403 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:12.257443 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1401) took 1107 ms. Trace:
I20260709 12:24:12.258855 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:12.215752  7425 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.269345 31823 rpcz_store.cc:276] 0709 12:24:11.066484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.775206 (+708722us) service_pool.cc:224] Handling call
0709 12:24:11.823545 (+ 48339us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:11.836032 (+ 12487us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:11.836072 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:11.846466 (+ 10394us) log.cc:844] Serialized 26 byte log entry
0709 12:24:12.006532 (+160066us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:12.006574 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.006667 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:12.232068 (+225401us) raft_consensus.cc:1685] finished
0709 12:24:12.232105 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:12.232244 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":6397,"raft.queue_time_us":5274,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":9597,"thread_start_us":84994,"threads_started":1,"wal-append.queue_time_us":236770}
I20260709 12:24:12.225276  7504 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:12.235760  7480 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:12.298553 31826 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:24:12.280200 31819 rpcz_store.cc:276] 0709 12:24:11.149922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.807159 (+657237us) service_pool.cc:224] Handling call
0709 12:24:11.807414 (+   255us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.807713 (+   299us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:11.880717 (+ 73004us) log.cc:844] Serialized 14 byte log entry
0709 12:24:12.256810 (+376093us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:12.256872 (+    62us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:12.256925 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.256996 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:12.257378 (+   382us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":73320,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":17016,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":9525,"thread_start_us":295555,"threads_started":1}
I20260709 12:24:12.281108 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:12.235666 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:12.268985  7396 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:12.314695 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:12.315896 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3308) took 2085 ms. Trace:
I20260709 12:24:12.317838 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:12.282689 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47976.1us
I20260709 12:24:12.316570 31826 rpcz_store.cc:276] 0709 12:24:10.230577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.233226 (+  2649us) service_pool.cc:224] Handling call
0709 12:24:10.233601 (+   375us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.964621 (+1731020us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b58007fd9d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd310aa 00007fe8fcdb3447 00007fe8fcdcbfe7 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4
0709 12:24:12.315556 (+350935us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.315815 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":268300160}
I20260709 12:24:12.319173 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:12.319797 31678 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
W20260709 12:24:12.320122 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 153114us
I20260709 12:24:12.279628  7313 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:12.282610  7481 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.591s	user 0.012s	sys 0.000s
I20260709 12:24:12.284641  7403 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:12.323854 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:12.284754  7425 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:12.321228 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:12.322232 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 205582us
I20260709 12:24:12.297595  7114 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.283491  7514 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:12.311806  7339 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:12.339912 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:12.341673  7504 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:24:12.344622  7515 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:24:12.344846  7403 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 22.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.345623 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2544) took 1220 ms. Trace:
I20260709 12:24:12.362159  7433 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:12.362053 31673 rpcz_store.cc:276] 0709 12:24:11.125189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.165594 (+ 40405us) service_pool.cc:224] Handling call
0709 12:24:11.166044 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.166321 (+   277us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:11.184948 (+ 18627us) log.cc:844] Serialized 14 byte log entry
0709 12:24:12.282421 (+1097473us) spinlock_profiling.cc:243] Waited 2.15 ms on lock 0x7b1800341320. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:24:12.345210 (+ 62789us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:12.345273 (+    63us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:12.345325 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.345401 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:12.345540 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14021,"prepare.queue_time_us":18396,"prepare.run_cpu_time_us":57,"prepare.run_wall_time_us":5417,"raft.queue_time_us":14353,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"spinlock_wait_cycles":4950272,"thread_start_us":48463,"threads_started":1}
I20260709 12:24:12.346498 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:12.368947  7368 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.236s	user 0.005s	sys 0.009s
I20260709 12:24:12.316650  7396 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:12.370898  7396 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:12.371842 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1385) took 2380 ms. Trace:
I20260709 12:24:12.372282 31820 rpcz_store.cc:276] 0709 12:24:09.991048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.086507 (+ 95459us) service_pool.cc:224] Handling call
0709 12:24:10.086791 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:10.126859 (+ 40068us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:10.126900 (+    41us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:10.126948 (+    48us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:10.126991 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.371572 (+2244581us) spinlock_profiling.cc:243] Waited 39.8 ms on lock 0x7b54003ca0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:12.371623 (+    51us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:12.371775 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":91630336}
I20260709 12:24:12.278452  7304 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.376282  7304 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:12.350260 31747 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:12.377513 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:12.378296  7445 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.792s	user 0.005s	sys 0.005s
I20260709 12:24:12.378849 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:12.384608 31675 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 28: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:12.363579  7433 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:12.379717  7461 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:12.380708 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 59611.7us
I20260709 12:24:12.317210  7440 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.383512 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:12.389494  7512 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:12.349572  7384 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:12.393133  7412 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:12.394230  7433 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:12.403836  7501 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:24:12.404196 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57809.2us
I20260709 12:24:12.454756 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:12.341161  7481 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.407285  7334 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:12.499643  7334 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:12.500617  7334 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.501940  7334 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:12.449833  7440 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.330750  7389 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.390457  7313 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.358722  7504 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:12.357824  7507 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.253s	user 0.000s	sys 0.001s
I20260709 12:24:12.409582 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:12.412606 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:12.415331 32375 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
W20260709 12:24:12.515841 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 103340us
I20260709 12:24:12.424283 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:12.517141 31820 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 24 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 12.
I20260709 12:24:12.443962 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:12.426734 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:12.432706 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.518415 31819 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:12.426280 31826 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:12.428020 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:12.457561 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:12.407339  7398 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.779s	user 0.011s	sys 0.003s
W20260709 12:24:12.451411  7466 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.387s	user 0.004s	sys 0.026s
W20260709 12:24:12.480400 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.515235 31670 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
W20260709 12:24:12.526932 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 214195us
W20260709 12:24:12.527992 31718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.852s (SENT) [suppressed 11 similar messages]
W20260709 12:24:12.454447  7441 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.801s	user 0.007s	sys 0.006s
I20260709 12:24:12.488953  7470 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:12.489799 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:12.491673 31669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2535) took 2451 ms (client timeout 1796 ms). Trace:
I20260709 12:24:12.492815 31672 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:24:12.495512 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:12.535935 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:12.496279  7514 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:12.536742  7378 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:12.536744 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:12.537437  7378 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.537942  7378 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:12.538148 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.538414  7385 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:12.443554  7445 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.416220 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:12.519779 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 91902.4us
I20260709 12:24:12.550058 31673 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:12.550432 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:12.551719 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:12.552286 31819 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:12.530226 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:12.553043 31820 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:24:12.534988 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:12.538148 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:12.534827 31669 rpcz_store.cc:269] 0709 12:24:10.040281 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.130402 (+ 90121us) service_pool.cc:224] Handling call
0709 12:24:12.491411 (+2361009us) spinlock_profiling.cc:243] Waited 122 ms on lock 0x7b54001815d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:12.491587 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":279917952}
I20260709 12:24:12.540402 31745 raft_consensus.cc:2410] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 22 because replica has last-logged OpId of term: 21 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 12:24:12.541431  7313 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:12.516996  7480 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:12.516712  7504 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:12.533471  7326 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:12.543395 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:12.543655  7441 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:12.529357  7466 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.465s	user 0.004s	sys 0.026s
I20260709 12:24:12.539108  7385 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.006s
I20260709 12:24:12.545326  7433 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.15, Last appended by leader: 15, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.454046  7501 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:12.548816 31744 raft_consensus.cc:2368] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
I20260709 12:24:12.548868 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2715) took 1915 ms. Trace:
I20260709 12:24:12.552189 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:12.522740 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.560691 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:12.564402 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:12.537258  7514 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:12.568249 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:12.571620 31820 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:12.580883 31747 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:12.534677  7499 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:12.575450 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:12.572876 32374 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 1143) took 1950 ms (client timeout 1922 ms). Trace:
I20260709 12:24:12.478448  7452 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.550863  7470 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:12.563970  7242 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:12.586067 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:12.571034 31753 rpcz_store.cc:276] 0709 12:24:10.633329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.721453 (+1088124us) service_pool.cc:224] Handling call
0709 12:24:11.781338 (+ 59885us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:11.894897 (+113559us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:11.894928 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:11.895618 (+   690us) log.cc:844] Serialized 26 byte log entry
0709 12:24:11.960400 (+ 64782us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:11.960445 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:11.960515 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:12.548617 (+588102us) raft_consensus.cc:1685] finished
0709 12:24:12.548648 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:12.548795 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":94378,"raft.queue_time_us":82595,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":26774,"thread_start_us":64310,"threads_started":1,"wal-append.queue_time_us":124883}
I20260709 12:24:12.593215 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.595429 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:12.601408 31822 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
W20260709 12:24:12.589779 32374 rpcz_store.cc:269] 0709 12:24:10.622741 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.106518 (+483777us) service_pool.cc:224] Handling call
0709 12:24:12.572641 (+1466123us) spinlock_profiling.cc:243] Waited 25.1 ms on lock 0x7b54005947d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:12.572792 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":57643264}
I20260709 12:24:12.562788  7491 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:12.590665 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:12.597959 31821 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:12.604796 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.608153 31820 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:12.609063 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.559930  7531 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:12.581125  7304 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:12.620898  7304 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.620664  7470 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.621338  7304 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:12.622059  7470 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:12.621604  7396 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.570238  7520 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.053s
I20260709 12:24:12.623083  7365 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:12.612486  7453 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:12.613543  7440 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:12.580444  7339 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:24:12.597682  7398 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:12.608726  7430 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:12.616036 31753 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:24:12.587744  7476 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.045s
I20260709 12:24:12.640765 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3387) took 1985 ms. Trace:
I20260709 12:24:12.611569  7405 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:12.641188 31753 rpcz_store.cc:276] 0709 12:24:10.655409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.600279 (+1944870us) service_pool.cc:224] Handling call
0709 12:24:12.615603 (+ 15324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.640443 (+ 24840us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.640685 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:12.616137  7441 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:12.648841 31750 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: [23.17-23.17]
I20260709 12:24:12.597147  7514 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:12.649619 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3398) took 1571 ms. Trace:
I20260709 12:24:12.599406  7289 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 12:24:12.575103  7480 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.651304  7504 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.652410  7480 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:12.599416  7474 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 12:24:12.652958  6651 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:12.653419  6651 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:12.599103  7493 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:12.654042  7493 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:12.599452  7412 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.040s
I20260709 12:24:12.654495  7493 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:12.654248  6651 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.649973 31750 rpcz_store.cc:276] 0709 12:24:11.078391 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.648194 (+1569803us) service_pool.cc:224] Handling call
0709 12:24:12.648588 (+   394us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.649324 (+   736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.649538 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:12.655568  6651 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:12.613548  7445 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:12.658780 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.630710  7242 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:12.619177  7519 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:12.535670  7425 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.660337  7365 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:12.624257 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:12.661350  7425 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:12.661782  7452 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:12.652443 31823 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:12.650207  7514 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:12.619241 31747 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: [21.13-23.14]
I20260709 12:24:12.664640 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2718) took 2030 ms. Trace:
I20260709 12:24:12.665050 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3506) took 1481 ms. Trace:
I20260709 12:24:12.665494 31672 rpcz_store.cc:276] 0709 12:24:11.183969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.547577 (+363608us) service_pool.cc:224] Handling call
0709 12:24:12.492424 (+944847us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.535160 (+ 42736us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.664847 (+129687us) spinlock_profiling.cc:243] Waited 929 ms on lock 0x7b54001815d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:12.664986 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2137699200}
I20260709 12:24:12.623587 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.666745 31750 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:12.667273 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 43781.4us
I20260709 12:24:12.667958  7365 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:12.668094 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:12.668754 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.669080  7312 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:12.669445 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.669534  7312 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:12.670490  7289 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:12.670749  7430 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.671315  7289 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:12.671715  7289 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.625653 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3319) took 1559 ms. Trace:
I20260709 12:24:12.672437 31825 rpcz_store.cc:276] 0709 12:24:11.066189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.778899 (+712710us) service_pool.cc:224] Handling call
0709 12:24:11.809223 (+ 30324us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:11.916367 (+107144us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.625400 (+709033us) spinlock_profiling.cc:243] Waited 68.8 ms on lock 0x7b54000fc154. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:12.625595 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":158293888}
I20260709 12:24:12.673297  7430 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:12.673722  7365 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:12.674152  7365 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:12.629588 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:12.675290  7396 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:12.674495 31825 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 16. (term mismatch)
I20260709 12:24:12.674990  7365 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:12.676492 31825 pending_rounds.cc:85] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 15
I20260709 12:24:12.629339 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:12.676818 31825 pending_rounds.cc:107] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 25.16
I20260709 12:24:12.677142 31825 raft_consensus.cc:2889] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:12.677510 31822 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 29 because replica has last-logged OpId of term: 28 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 18.
I20260709 12:24:12.580493  7461 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.060s
I20260709 12:24:12.599231  7499 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:12.691208 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:12.648442 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:12.650398  7481 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:12.659906  7242 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.663923 31751 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:24:12.609444 31819 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:12.697578 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2716) took 2064 ms. Trace:
I20260709 12:24:12.698021 31751 rpcz_store.cc:276] 0709 12:24:10.633564 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.730387 (+1096823us) service_pool.cc:224] Handling call
0709 12:24:11.805088 (+ 74701us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.663674 (+858586us) spinlock_profiling.cc:243] Waited 75.1 ms on lock 0x7b54003ba9a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:24:12.697321 (+ 33647us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.697528 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":172832000}
I20260709 12:24:12.698596 31819 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:24:12.698827 31825 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:12.672102 31818 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 18.10, Last appended by leader: 10, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.699900 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:12.700587 31672 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:12.700659 31818 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:12.607272  7491 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.672072  7289 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:12.641603 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:12.694697  7312 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.708765 31752 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: [22.16-22.16]
I20260709 12:24:12.709386 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3403) took 1602 ms. Trace:
I20260709 12:24:12.709690 31752 rpcz_store.cc:276] 0709 12:24:11.107158 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.708184 (+1601026us) service_pool.cc:224] Handling call
0709 12:24:12.708536 (+   352us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.709126 (+   590us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.709334 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:12.606045  7477 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.675s	user 0.012s	sys 0.000s
I20260709 12:24:12.647104 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.679008  7445 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:12.691875 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62397.3us
I20260709 12:24:12.705933  7491 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:12.678542 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:12.695995 31753 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:12.701722 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:12.724196 31670 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:12.724227 31822 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:12.697181  7396 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:12.718125 31752 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
I20260709 12:24:12.691063  7476 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.020s
I20260709 12:24:12.728899 31752 pending_rounds.cc:85] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 16
I20260709 12:24:12.729439 31752 pending_rounds.cc:107] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260709 12:24:12.729785 31752 raft_consensus.cc:2889] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:12.722688 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 75711.6us
I20260709 12:24:12.730688 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.665017 31747 rpcz_store.cc:276] 0709 12:24:10.633890 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.591613 (+1957723us) service_pool.cc:224] Handling call
0709 12:24:12.618900 (+ 27287us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:12.664342 (+ 45442us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.664563 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:12.724833 31825 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:12.734887 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.737381 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.705528  7481 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.024s
I20260709 12:24:12.721051 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:12.758265 31818 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 10. Preceding OpId from leader: term: 19 index: 10. (term mismatch)
I20260709 12:24:12.758733 31818 pending_rounds.cc:85] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 9
I20260709 12:24:12.758967 31818 pending_rounds.cc:107] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 18.10
I20260709 12:24:12.763003 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:12.738595  7430 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:12.767560 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 229456us
I20260709 12:24:12.702982 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.741228 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:12.723721  7312 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:12.761904 31818 raft_consensus.cc:2889] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:12.770879 31747 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
W20260709 12:24:12.791168 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 88301.2us
I20260709 12:24:12.755403  7440 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:12.791975 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3408) took 1683 ms. Trace:
I20260709 12:24:12.722358  7477 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.754923  7441 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:12.792518 31747 rpcz_store.cc:276] 0709 12:24:11.108671 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.758308 (+1649637us) service_pool.cc:224] Handling call
0709 12:24:12.770676 (+ 12368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.791590 (+ 20914us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.791892 (+   302us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:12.792884 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.794119 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3314) took 2481 ms. Trace:
I20260709 12:24:12.728545  7445 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.801803 31818 rpcz_store.cc:276] 0709 12:24:10.312260 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.387459 (+ 75199us) service_pool.cc:224] Handling call
0709 12:24:10.387730 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.470534 (+1082804us) spinlock_profiling.cc:243] Waited 35.1 ms on lock 0x7b4c0003cd00. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb0a1a20 00007fe8fb0b2a29 00007fe8fb078644 00007fe8fb09ff2d 00007fe8fb09fb93 00007fe8fcdd018b 00007fe8fcdd0cb5 00007fe8fcdcbf30 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416
0709 12:24:12.671665 (+1201131us) spinlock_profiling.cc:243] Waited 49 ms on lock 0x7b54000dc3a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd5306f 00007fe8fcdcbf60 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62
0709 12:24:12.793242 (+121577us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.794037 (+   795us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":193268736}
I20260709 12:24:12.739305 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3404) took 1631 ms. Trace:
I20260709 12:24:12.764693  7301 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:12.811434 32373 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:12.767222  7339 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:12.812945 31752 rpcz_store.cc:276] 0709 12:24:11.107577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.717247 (+1609670us) service_pool.cc:224] Handling call
0709 12:24:12.717677 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.734748 (+ 17071us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.739232 (+  4484us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:12.814054 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:12.825871 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.820420 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:12.793546  7447 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:24:12.804296  7460 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 19 current term: 19 is higher.
I20260709 12:24:12.832388 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:12.800872  7441 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:12.853423 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:12.765811  7519 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.047s
I20260709 12:24:12.858385  7304 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.870756  7304 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:12.801091  7476 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:12.805373  7521 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:24:12.838984 31642 connection.cc:256] RPC call timeout handler was delayed by -1.1157925359998444s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.32360000000000005s for a response
I20260709 12:24:12.861341 31753 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:12.861510 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:12.875345 31747 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:12.864560  7425 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:12.881992  7365 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:12.883778 31747 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:12.884155  7378 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:12.885802 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:12.886564  7433 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:12.889573 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3411) took 1780 ms. Trace:
I20260709 12:24:12.890735 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1313) took 1759 ms. Trace:
I20260709 12:24:12.800742  7398 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20260709 12:24:12.814761  7518 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 18. Reason: could not achieve majority
I20260709 12:24:12.910787  7477 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:12.912099 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:12.912910 32373 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:12.866029 31822 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:12.866724  7295 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:12.820137  7301 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:12.863999 32377 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:24:12.818908  7512 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:12.889972 31747 rpcz_store.cc:276] 0709 12:24:11.109468 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.888703 (+1779235us) service_pool.cc:224] Handling call
0709 12:24:12.889036 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.889209 (+   173us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:12.889234 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:12.889282 (+    48us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:12.889313 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.889377 (+    64us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:12.889477 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:12.891165 32370 rpcz_store.cc:276] 0709 12:24:11.131031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.296460 (+165429us) service_pool.cc:224] Handling call
0709 12:24:11.296859 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.306586 (+  9727us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:11.306645 (+    59us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:11.306699 (+    54us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:11.306737 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.890460 (+1583723us) spinlock_profiling.cc:243] Waited 9.53 ms on lock 0x7b5400544f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:12.890505 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:12.890674 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":21930368}
I20260709 12:24:12.903697 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:12.865036 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 61011.6us
I20260709 12:24:12.897845  7504 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:12.907414 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:12.761858  7499 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:12.920782 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 319748us
I20260709 12:24:12.940208  6752 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:12.940011 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:12.942382 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:12.942868 31822 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:12.943004 31670 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:12.905735 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:12.916942 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:12.918860 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:12.946471 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:12.946971 32373 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:12.947384  6752 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:12.947824 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:12.948635  6752 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:12.949054  6752 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:12.949409  6752 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:12.919646 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:12.951073 32376 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:12.952731  7501 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.953915  7501 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:12.864928  7447 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:12.954917  7447 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:12.957511 31744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3400) took 1851 ms (client timeout 2085 ms). Trace:
W20260709 12:24:12.957942 31744 rpcz_store.cc:269] 0709 12:24:11.106254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.403019 (+296765us) service_pool.cc:224] Handling call
0709 12:24:12.957284 (+1554265us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b5400064854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:12.957449 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2346429312}
I20260709 12:24:12.965725  6752 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.968273  6752 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:12.969142  7447 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:12.971295  7480 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Stepping down as leader of term 17
I20260709 12:24:12.971493  7447 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:12.974083 31751 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:12.975975 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:12.978310  7521 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:12.978820  7521 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:12.984824 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:12.985662 31826 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:12.989444  7399 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:12.989895  7399 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:12.991394 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3482) took 1348 ms. Trace:
I20260709 12:24:12.991781 31821 rpcz_store.cc:276] 0709 12:24:11.642562 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.998768 (+356206us) service_pool.cc:224] Handling call
0709 12:24:12.011425 (+ 12657us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.608089 (+596664us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.991113 (+383024us) spinlock_profiling.cc:243] Waited 586 ms on lock 0x7b540030ded4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:12.991321 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1348337920}
I20260709 12:24:12.992959 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:12.993546 31821 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:13.004946  7536 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.114s	user 0.000s	sys 0.001s
W20260709 12:24:13.007998 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.009111 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:13.009342 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:13.009958 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.011656 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3401) took 1905 ms. Trace:
I20260709 12:24:13.012073 31751 rpcz_store.cc:276] 0709 12:24:11.106572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.699146 (+1592574us) service_pool.cc:224] Handling call
0709 12:24:12.973766 (+274620us) spinlock_profiling.cc:243] Waited 17.1 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:24:12.973916 (+   150us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.974521 (+   605us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.011593 (+ 37072us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":39290112}
I20260709 12:24:12.906771  7476 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:13.014055  7476 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:13.016566  7534 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.201s	user 0.000s	sys 0.000s
I20260709 12:24:12.923394  7301 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:12.909564  7481 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.109s
W20260709 12:24:13.018491  7533 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 12:24:13.019971 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:13.021806 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1145) took 1600 ms. Trace:
I20260709 12:24:13.022336 31747 rpcz_store.cc:276] 0709 12:24:11.421396 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.020660 (+1599264us) service_pool.cc:224] Handling call
0709 12:24:13.021110 (+   450us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.021354 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.021396 (+    42us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:13.021455 (+    59us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:13.021503 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.021576 (+    73us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.021728 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:13.024596 31747 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:24:13.024644 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:13.025305 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1148) took 1602 ms. Trace:
I20260709 12:24:13.025648 31747 rpcz_store.cc:276] 0709 12:24:11.422499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.023457 (+1600958us) service_pool.cc:224] Handling call
0709 12:24:13.024116 (+   659us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.025060 (+   944us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.025252 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:12.905746  7378 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:13.030500  7365 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:13.031023  7445 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:13.031857  7535 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.230s	user 0.000s	sys 0.000s
I20260709 12:24:13.031773 31747 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:24:13.032760  7432 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:13.035481 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1150) took 1612 ms. Trace:
I20260709 12:24:12.944286  7412 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:13.035835 31747 rpcz_store.cc:276] 0709 12:24:11.423162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.026370 (+1603208us) service_pool.cc:224] Handling call
0709 12:24:13.031382 (+  5012us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.035205 (+  3823us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.035423 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:13.036791  7447 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.037595 31747 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:13.037318  7447 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.032037 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:13.039368  7447 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:13.040251 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1153) took 1603 ms. Trace:
I20260709 12:24:13.040591 31747 rpcz_store.cc:276] 0709 12:24:11.436723 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.036993 (+1600270us) service_pool.cc:224] Handling call
0709 12:24:13.037287 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.039914 (+  2627us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.040167 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:13.043496 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:13.045861 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:12.969154 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:13.049010 31825 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:12.541937 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:12.895344  7441 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:13.058027 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.058609 32373 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
W20260709 12:24:13.058808 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 113889us
I20260709 12:24:13.062829 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:13.063755  7412 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:13.059065 32372 raft_consensus.cc:2368] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28.
I20260709 12:24:13.042567 31747 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:13.064378  7412 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:13.065527 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3423) took 1513 ms. Trace:
I20260709 12:24:13.065873 31747 rpcz_store.cc:276] 0709 12:24:11.552517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.041421 (+1488904us) service_pool.cc:224] Handling call
0709 12:24:13.041917 (+   496us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.065176 (+ 23259us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.065452 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:13.066773  7412 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:13.067322 31747 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 24 index: 18. (term mismatch)
I20260709 12:24:13.067783 31747 pending_rounds.cc:85] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 17
I20260709 12:24:13.068069 31747 pending_rounds.cc:107] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 23.18
I20260709 12:24:13.068811  7412 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.069397  7412 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:13.072989  7412 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:13.075532 31747 raft_consensus.cc:2889] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:13.076571 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3426) took 1523 ms. Trace:
I20260709 12:24:13.078778  7365 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:13.081218 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:13.081863 31669 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:13.076977 31747 rpcz_store.cc:276] 0709 12:24:11.553490 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.066653 (+1513163us) service_pool.cc:224] Handling call
0709 12:24:13.066964 (+   311us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.076276 (+  9312us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.076511 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:12.971760  7480 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 17 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:13.006953  7114 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:13.105351 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:13.106428 32343 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:13.112206 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:13.119709 31821 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:13.121630 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:13.093238  7520 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.092866  7519 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:13.100247  7493 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.134889  7433 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.135619 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3526) took 1308 ms. Trace:
I20260709 12:24:13.138070 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:13.139473 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.140789 32377 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:13.146432 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:13.156720 31669 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:13.151966 31678 rpcz_store.cc:276] 0709 12:24:11.827301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.325852 (+498551us) service_pool.cc:224] Handling call
0709 12:24:12.326256 (+   404us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.326541 (+   285us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:12.326585 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:12.335275 (+  8690us) log.cc:844] Serialized 26 byte log entry
0709 12:24:12.811174 (+475899us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:12.811208 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.811282 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.135341 (+324059us) raft_consensus.cc:1685] finished
0709 12:24:13.135363 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.135549 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":358508,"threads_started":1,"wal-append.queue_time_us":722287}
W20260709 12:24:13.169584 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.171187 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:13.172298 31670 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:24:13.178375 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:13.125367  7432 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:13.208297 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:13.212443 31821 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:13.221020 31825 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:24:13.224452  7364 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:13.229112 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:13.229946 31752 raft_consensus.cc:2376] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24. Re-sending same reply.
I20260709 12:24:13.234431 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:13.236171 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:13.237959 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:13.244753 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:13.245189 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:13.239414 31824 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:13.251546 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3391) took 2212 ms. Trace:
I20260709 12:24:13.253360 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:13.254256 31752 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:13.251976 31745 rpcz_store.cc:276] 0709 12:24:11.039309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.613686 (+574377us) service_pool.cc:224] Handling call
0709 12:24:13.251337 (+1637651us) spinlock_profiling.cc:243] Waited 919 ms on lock 0x7b54001a4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:13.251484 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2113374080}
I20260709 12:24:13.165081  7430 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:13.258982 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.264422  7543 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:13.265010  7463 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.265262 31745 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: [20.15-20.15]
I20260709 12:24:13.265199 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:13.125335  7531 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.126s
I20260709 12:24:13.173636  7512 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.175429  7425 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:13.282020 32377 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:24:13.284026 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2731) took 1327 ms. Trace:
I20260709 12:24:13.293284 31745 rpcz_store.cc:276] 0709 12:24:11.956173 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.264480 (+1308307us) service_pool.cc:224] Handling call
0709 12:24:13.265020 (+   540us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.283588 (+ 18568us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.283940 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:13.175262  7312 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:13.152727  7520 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.338335  7312 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:13.339500  7312 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.12, Last appended by leader: 12, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.340283  7520 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:13.174921  7519 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:13.316268 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.013s	sys 0.017s
W20260709 12:24:13.319116 31818 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.018s	sys 0.046s
W20260709 12:24:13.344347 31818 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.534s	user 0.019s	sys 0.046s
I20260709 12:24:13.320092 31745 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:13.282091  7389 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:13.281631  7432 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:13.283129  7543 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:13.329629 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 456006us
W20260709 12:24:13.333851 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.174423  7433 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:13.344449 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.013s	sys 0.017s
I20260709 12:24:13.311899  7384 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:13.258217  7511 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.070s
I20260709 12:24:13.334928  7392 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:13.357506 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 563754us
W20260709 12:24:13.369486  7289 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.655s	user 0.011s	sys 0.000s
I20260709 12:24:13.370059  7289 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.372457 31748 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
W20260709 12:24:13.355108 31819 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.595s	user 0.022s	sys 0.049s
W20260709 12:24:13.379439 31819 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.022s	sys 0.049s
I20260709 12:24:13.311905  7512 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:13.383235 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:13.384626  7512 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:13.385978  7511 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:13.386396  7511 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:13.387173  7511 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.387856 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:13.388491 31670 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:13.388643 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:13.250502  7480 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 17, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.389556 31669 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:13.389605 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:13.391831 31675 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:13.396179 31678 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:13.308435  7463 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:13.410140 31673 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:13.343048  7519 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:13.388345  7334 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.886s	user 0.008s	sys 0.004s
I20260709 12:24:13.444851 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:13.403755 31821 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:13.358479  7384 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:13.411943  7542 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.086s	user 0.001s	sys 0.001s
I20260709 12:24:13.413421 31824 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 21 index: 13. (index mismatch)
I20260709 12:24:13.401584  7430 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:13.388396  7511 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:13.394513  7313 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.831s	user 0.012s	sys 0.000s
I20260709 12:24:13.367683  7432 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:13.459565 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:13.412823 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:13.408712  7547 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.050s	user 0.000s	sys 0.001s
I20260709 12:24:13.462399 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:13.475654 31670 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:13.358652  7543 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:13.462888 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:13.462960 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:13.463172 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1390) took 3263 ms. Trace:
I20260709 12:24:13.463176 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:13.466734 31821 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:24:13.480744 31745 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:13.468336  7308 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 21 current term: 21 is higher.
I20260709 12:24:13.477391 31678 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:13.465286  7304 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:13.409842  7477 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.479768 31824 rpcz_store.cc:276] 0709 12:24:10.200052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:10.200501 (+   449us) service_pool.cc:224] Handling call
0709 12:24:10.223135 (+ 22634us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:11.160099 (+936964us) spinlock_profiling.cc:243] Waited 48.4 ms on lock 0x7b5400277a20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:24:11.874991 (+714892us) spinlock_profiling.cc:243] Waited 137 ms on lock 0x7b5800770cd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd310aa 00007fe8fcdb3447 00007fe8fcdcbfe7 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4
0709 12:24:13.238619 (+1363628us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b5400277a20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4aa1d 00007fe8fcd311a4 00007fe8fcdb3447 00007fe8fcdcbfe7 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419
0709 12:24:13.462443 (+223824us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.463110 (+   667us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":719372032}
I20260709 12:24:13.483946 31819 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:13.462266  7460 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.409729  7301 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:13.474957  7318 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:13.474654  7522 raft_consensus.cc:2764] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election decision vote started in defunct term 27: won
I20260709 12:24:13.468802  7339 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 23.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:24:13.479884  7430 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:13.501407 32375 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:24:13.491765  7304 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.480683 31669 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
W20260709 12:24:13.506544 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 335457us
I20260709 12:24:13.506057 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:13.468520  7520 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.029s
I20260709 12:24:13.488670  7461 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:13.504792  7384 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.527948  7447 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.537240 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3328) took 1103 ms. Trace:
I20260709 12:24:13.577047 31820 rpcz_store.cc:276] 0709 12:24:12.433427 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.662445 (+229018us) service_pool.cc:224] Handling call
0709 12:24:12.662761 (+   316us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.663085 (+   324us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:12.663113 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:12.663663 (+   550us) log.cc:844] Serialized 26 byte log entry
0709 12:24:12.949559 (+285896us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:12.949602 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.949680 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.536983 (+587303us) raft_consensus.cc:1685] finished
0709 12:24:13.537011 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.537168 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3356,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":87,"thread_start_us":285438,"threads_started":1,"wal-append.queue_time_us":629246}
W20260709 12:24:13.578549 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 72647us
I20260709 12:24:13.537582  7301 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:13.548481 31749 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:24:13.582763 31820 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:13.548856  7304 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:13.554606 31819 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:13.555606 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:13.605861 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50336.2us
W20260709 12:24:13.558983 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.567544 31745 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:13.607410 31752 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:24:13.586688  7430 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.591099  7447 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.591802  7308 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.592214  7432 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:13.594276  7470 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:13.596668  7384 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:13.596571  7425 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.598708  7339 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:13.620244  7441 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.515s	user 0.011s	sys 0.000s
W20260709 12:24:13.609328 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.610661 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:13.615226  7334 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.637496  7334 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.619040  7242 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.021s
I20260709 12:24:13.627992  7470 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:13.629621  7425 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:13.643337  7470 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:13.635845  7441 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.645097  7396 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:13.645644  7396 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.646127  7441 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:13.498936  7460 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.629887  7339 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.647145  7339 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:13.648484  7460 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:13.648799  7501 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.694s	user 0.009s	sys 0.002s
I20260709 12:24:13.649204 31821 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: [19.11-19.11]
I20260709 12:24:13.651377 31821 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:24:13.652405 31822 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:13.499423  7477 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.654670  7477 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:13.654892 31746 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:13.655447 31824 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:13.656498 31827 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:13.500340  7550 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.103s
I20260709 12:24:13.658708  7481 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.658946 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3412) took 2549 ms. Trace:
I20260709 12:24:13.578042  7461 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:13.622545 31820 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: [26.13-26.13]
I20260709 12:24:13.659288  7481 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.659920  7461 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:13.622061  7405 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.661624  7447 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:13.623499  7308 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:13.662987 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:13.621716  7319 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.007s
I20260709 12:24:13.599539  7433 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:13.664147 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:13.665338 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:13.667050 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.667451 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:13.667094 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:13.626948  7463 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:13.611677  7452 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:13.629868  7437 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s	user 0.027s	sys 0.029s
I20260709 12:24:13.635538 31745 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:13.555629  7514 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:24:13.640004  7539 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.749s	user 0.000s	sys 0.003s
I20260709 12:24:13.661971  7405 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.664234  7308 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:13.630108 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.553996  7551 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.109s
W20260709 12:24:13.630142  7491 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.896s	user 0.008s	sys 0.005s
I20260709 12:24:13.659304 31748 rpcz_store.cc:276] 0709 12:24:11.109726 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.889490 (+1779764us) service_pool.cc:224] Handling call
0709 12:24:12.889812 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.372946 (+483134us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.658713 (+285767us) spinlock_profiling.cc:243] Waited 482 ms on lock 0x7b5400403dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:13.658884 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1109454080}
I20260709 12:24:13.658435 31824 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:13.665979 31744 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:13.617173 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:13.676026 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:13.640012  7334 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:13.715246 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 98147.6us
I20260709 12:24:13.728093 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:13.679559 31747 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.011s	sys 0.022s
I20260709 12:24:13.694972 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:13.730151 31747 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.579s	user 0.012s	sys 0.022s
I20260709 12:24:13.717187 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:13.719589 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:13.691712  7437 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.692s	user 0.027s	sys 0.029s
I20260709 12:24:13.740473 31752 raft_consensus.cc:2368] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
I20260709 12:24:13.741565 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:13.692610  7514 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:13.697285  7550 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.636148  7541 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.099s
I20260709 12:24:13.849860 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:13.851897 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:13.758701 31745 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:13.763000 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:13.861796 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:13.756800 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:13.743180  7463 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:13.768872  7550 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:13.769464  7514 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:13.770920  7512 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:13.772356 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:13.773561 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:13.713760  7364 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.803599 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3520) took 2084 ms. Trace:
I20260709 12:24:13.804620 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3399) took 2726 ms. Trace:
I20260709 12:24:13.886891 31750 rpcz_store.cc:276] 0709 12:24:11.105829 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.655461 (+1549632us) service_pool.cc:224] Handling call
0709 12:24:12.655799 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:12.667173 (+ 11374us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.804368 (+1137195us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b54002963d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:13.804557 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":24699776}
I20260709 12:24:13.805802 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:13.806248 31749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2727) took 2151 ms (client timeout 1920 ms). Trace:
I20260709 12:24:13.806947  7334 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.810774 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:13.892829 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:13.893594 31676 raft_consensus.cc:2368] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28.
I20260709 12:24:13.811482 31670 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:13.894095 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.894747 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:13.885161 31677 rpcz_store.cc:276] 0709 12:24:11.718953 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.970985 (+252032us) service_pool.cc:224] Handling call
0709 12:24:11.971393 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:12.360909 (+389516us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:12.360956 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:12.361593 (+   637us) log.cc:844] Serialized 26 byte log entry
0709 12:24:12.800105 (+438512us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:12.800161 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:12.800251 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.639361 (+839110us) spinlock_profiling.cc:243] Waited 14.1 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:13.803126 (+163765us) raft_consensus.cc:1685] finished
0709 12:24:13.803166 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:13.803502 (+   336us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37778,"raft.queue_time_us":137036,"raft.run_cpu_time_us":133,"raft.run_wall_time_us":9305,"spinlock_wait_cycles":32468352,"thread_start_us":789280,"threads_started":2,"wal-append.queue_time_us":529386}
I20260709 12:24:13.896389  7364 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.898109 31751 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:13.812567 31675 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:13.732859  7491 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.815713 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:13.819566  7396 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:13.820977  7493 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:13.720691  7318 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:13.824863  7521 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.846s	user 0.012s	sys 0.001s
I20260709 12:24:13.829115  7389 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:13.831295  7405 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:13.834708 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:13.842561 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:13.942950 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.845122  7289 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:13.867229 31752 raft_consensus.cc:2410] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 22 because replica has last-logged OpId of term: 21 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
W20260709 12:24:13.881831 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 401366us
I20260709 12:24:13.735090  7354 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:24:13.889175 31749 rpcz_store.cc:269] 0709 12:24:11.654759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.672227 (+ 17468us) service_pool.cc:224] Handling call
0709 12:24:13.806041 (+2133814us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b54002959d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:13.806183 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4312030976}
I20260709 12:24:13.899178 31671 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:13.896194 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:13.953625 31745 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 26 because replica has last-logged OpId of term: 23 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:24:13.910734 31750 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:24:13.879405  7412 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:13.921377 31744 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:13.923000 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:13.892865  7522 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:13.866048  7543 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:13.927011 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:13.894116  7463 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:13.974139 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:13.892191  7518 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:13.944228 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 109655us
I20260709 12:24:13.915757  7474 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:13.991974 31676 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:13.995107 31745 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
I20260709 12:24:13.937439  7493 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:14.001384 31674 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: [26.13-26.13]
I20260709 12:24:14.001443 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.002318 31676 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:24:14.003605 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:14.010800 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:13.950765  7313 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.947711  7403 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:13.968596  7389 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:13.927309  7318 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:13.965329  7304 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:13.962123 31677 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:13.907888  7334 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:13.910341  7336 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.013s	sys 0.014s
I20260709 12:24:14.021288 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.022305 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2730) took 2066 ms. Trace:
W20260709 12:24:14.022936 31746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3427) took 2469 ms (client timeout 1619 ms). Trace:
W20260709 12:24:14.079649 31746 rpcz_store.cc:269] 0709 12:24:11.553748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.570354 (+ 16606us) service_pool.cc:224] Handling call
0709 12:24:14.022689 (+2452335us) spinlock_profiling.cc:243] Waited 2.04 s on lock 0x7b5400064354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:14.022836 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4687205760}
I20260709 12:24:14.081171 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:14.025031 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3486) took 1442 ms. Trace:
I20260709 12:24:14.030735 31753 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:13.990769  7541 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:14.040259  7543 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:14.000723  7354 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:14.043509 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3489) took 1365 ms. Trace:
I20260709 12:24:13.927187  7521 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.100382 31752 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
W20260709 12:24:14.077867  7336 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.014s	sys 0.014s
I20260709 12:24:14.082845 31827 rpcz_store.cc:276] 0709 12:24:12.582292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.041567 (+459275us) service_pool.cc:224] Handling call
0709 12:24:13.656328 (+614761us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.656887 (+   559us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.024739 (+367852us) spinlock_profiling.cc:243] Waited 614 ms on lock 0x7b54003c9bd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:14.024946 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1413145344}
I20260709 12:24:14.115561 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.116691 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:14.086613 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.035152  7412 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:14.109751 31823 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
W20260709 12:24:14.111740 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.093819 31825 rpcz_store.cc:276] 0709 12:24:12.677952 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.262916 (+584964us) service_pool.cc:224] Handling call
0709 12:24:13.263399 (+   483us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:13.290811 (+ 27412us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.290869 (+    58us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:13.304911 (+ 14042us) log.cc:844] Serialized 52 byte log entry
0709 12:24:13.722495 (+417584us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:13.722554 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.723602 (+  1048us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:14.043173 (+319571us) raft_consensus.cc:1685] finished
0709 12:24:14.043203 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.043432 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":171,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":102,"thread_start_us":376720,"threads_started":1,"wal-append.queue_time_us":648323}
I20260709 12:24:14.122879 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.123602 31825 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:14.123977  7334 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.126008  7521 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:13.991016  7522 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.127869  7318 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:14.128070 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.129112  7550 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:14.129637 31825 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:14.017726  7493 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:13.961181 31818 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.008s	sys 0.021s
W20260709 12:24:14.133481 31818 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.753s	user 0.008s	sys 0.021s
I20260709 12:24:14.000695  7518 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.098526  7389 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:14.140933 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.160027 31673 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 22 because replica has last-logged OpId of term: 21 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:24:14.161159 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:14.141499 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 184673us
W20260709 12:24:14.143097 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 291295us
I20260709 12:24:14.049976 31671 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:24:14.146415 31825 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: [23.16-23.16]
W20260709 12:24:14.154371 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.044479  7538 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.099s
I20260709 12:24:14.029971  7313 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.174975 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1413) took 1047 ms. Trace:
I20260709 12:24:14.029753  7403 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.118717  7541 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:14.181478 31825 rpcz_store.cc:276] 0709 12:24:13.127382 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.145589 (+1018207us) service_pool.cc:224] Handling call
0709 12:24:14.146081 (+   492us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.174539 (+ 28458us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.174892 (+   353us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:14.181751 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.098785  7339 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.191226 31673 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:24:14.183893 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.128404  7522 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:14.191522 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:14.191707 31718 leader_election.cc:400] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28.
I20260709 12:24:14.192063 31676 raft_consensus.cc:2410] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 26 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:24:14.195338 31718 leader_election.cc:403] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:14.197889  7474 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28.
I20260709 12:24:14.193493  7521 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.186053  7460 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.204150 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:14.186494  7465 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:14.207376 31669 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:24:14.207376 31825 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260709 12:24:14.196236  7512 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:14.193971  7466 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.138670  7412 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.210421 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3352) took 1082 ms. Trace:
I20260709 12:24:14.219532 31825 rpcz_store.cc:276] 0709 12:24:13.128142 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.206791 (+1078649us) service_pool.cc:224] Handling call
0709 12:24:14.207163 (+   372us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.209005 (+  1842us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.210308 (+  1303us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:14.215127 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:14.199126  7519 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:14.225296 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.206964  7447 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:14.217829 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:14.155405  7463 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.224254 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.206758  7491 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.236862 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:14.236584 31671 raft_consensus.cc:2410] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 23 because replica has last-logged OpId of term: 21 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
I20260709 12:24:14.201951 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:14.224546  7461 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.564s	user 0.009s	sys 0.007s
I20260709 12:24:14.202649  7354 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.240619 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:14.251120  7399 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.252305  7399 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:14.252862  7412 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:14.253499  7412 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:14.253494  7522 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 23.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20260709 12:24:14.239291  7565 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
W20260709 12:24:14.237674  7425 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.595s	user 0.010s	sys 0.002s
I20260709 12:24:14.223527  7573 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:14.244016  7477 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:14.244343  7474 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:14.244814  7493 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:14.245018 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.272938 31671 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
W20260709 12:24:14.230891 31821 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.014s	sys 0.017s
I20260709 12:24:14.205354  7521 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.216586  7465 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.254137  7412 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:14.208654  7460 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.219965  7437 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 20.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.028s
W20260709 12:24:14.260535 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 313461us
I20260709 12:24:14.237614  7550 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:14.279242  7477 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:14.284772  7477 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.287340  7518 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:14.290014  7566 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:24:14.288295 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:14.291281 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3485) took 1724 ms. Trace:
I20260709 12:24:14.281968  7474 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:14.291705 31818 rpcz_store.cc:276] 0709 12:24:12.567315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.808663 (+241348us) service_pool.cc:224] Handling call
0709 12:24:12.809639 (+   976us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.345756 (+536117us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.345786 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.357164 (+ 11378us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.134033 (+776869us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:14.134089 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.134176 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:14.291028 (+156852us) raft_consensus.cc:1685] finished
0709 12:24:14.291049 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.291204 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":61332,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":93,"thread_start_us":1289259,"threads_started":2,"wal-append.queue_time_us":756686}
I20260709 12:24:14.292704  7474 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 19.15, Last appended by leader: 15, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.293783 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:14.295673 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:14.299398  7574 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
W20260709 12:24:14.301157  7572 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:24:14.259891 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1410) took 1137 ms. Trace:
I20260709 12:24:14.234544  7308 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:14.265121  7573 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.303566  7573 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:14.219892  7512 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.224087  7466 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:14.307488  7580 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:24:14.304280 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:14.274892 31821 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.554s	user 0.014s	sys 0.017s
I20260709 12:24:14.242877 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:14.246099  7430 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:14.311040  7430 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.215054  7389 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.319880 31671 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:24:14.320559  7550 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:14.323189  7543 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:14.323632  7313 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.323819  7543 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:14.325531  7543 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.327528  7543 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:14.327698 31672 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:24:14.266037  7425 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.330225 31820 rpcz_store.cc:276] 0709 12:24:13.122983 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.660465 (+537482us) service_pool.cc:224] Handling call
0709 12:24:13.660819 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.661047 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:14.045684 (+384637us) log.cc:844] Serialized 14 byte log entry
0709 12:24:14.259436 (+213752us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:14.259498 (+    62us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:14.259550 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.259627 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.259812 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":605695,"prepare.run_cpu_time_us":70,"prepare.run_wall_time_us":2037,"raft.queue_time_us":1133,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":7063,"thread_start_us":595244,"threads_started":2}
W20260709 12:24:14.333027 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 90281.9us
I20260709 12:24:14.334792 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:14.337292 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:14.338572 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.340523 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:14.342353 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:14.343043 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.236564  7519 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.345460 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1411) took 1219 ms. Trace:
I20260709 12:24:14.355440 31824 rpcz_store.cc:276] 0709 12:24:13.126542 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.767284 (+640742us) service_pool.cc:224] Handling call
0709 12:24:13.823328 (+ 56044us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.823656 (+   328us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:24:14.150257 (+326601us) log.cc:844] Serialized 14 byte log entry
0709 12:24:14.344962 (+194705us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:14.345041 (+    79us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:14.345108 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.345204 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.345386 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":489680,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":436,"thread_start_us":519829,"threads_started":2}
I20260709 12:24:14.291980 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:14.321792  7550 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:14.289233  7577 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
I20260709 12:24:14.360316  7512 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:14.255934  7461 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:14.288643  7576 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:24:14.330665  7339 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.362188  7392 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:14.357961 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:14.364095 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3350) took 1318 ms. Trace:
W20260709 12:24:14.370306 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.376592 32371 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:14.241966  7541 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.400058 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.400591 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:14.380926 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 89142.8us
I20260709 12:24:14.302958  7308 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.387782 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:14.395958 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:14.406833 31746 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:14.390789 31822 rpcz_store.cc:276] 0709 12:24:13.045747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.654334 (+608587us) service_pool.cc:224] Handling call
0709 12:24:13.669041 (+ 14707us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.699411 (+ 30370us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.699462 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.713996 (+ 14534us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.163946 (+449950us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:14.164008 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.164105 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:14.363849 (+199744us) raft_consensus.cc:1685] finished
0709 12:24:14.363870 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.364027 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30359,"raft.queue_time_us":30871,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":19373,"thread_start_us":392686,"threads_started":1,"wal-append.queue_time_us":532685}
W20260709 12:24:14.403096 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.409415 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.411813  7447 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:14.420014  7447 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.420503  7447 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:14.412340 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.422757 31677 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:14.416214  7403 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:14.417443 31670 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:24:14.387260  7590 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.365139  7567 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:14.393666  7392 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:14.389715  7584 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.429538 31717 leader_election.cc:400] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28.
I20260709 12:24:14.430549 31670 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 12:24:14.436039 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:14.450834 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46444.7us
I20260709 12:24:14.451664 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:14.454088 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:14.426856  7425 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.462191 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3346) took 1559 ms. Trace:
I20260709 12:24:14.470466 31819 rpcz_store.cc:276] 0709 12:24:12.902987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.605093 (+702106us) service_pool.cc:224] Handling call
0709 12:24:13.610879 (+  5786us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.614989 (+  4110us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.615024 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.615528 (+   504us) log.cc:844] Serialized 26 byte log entry
0709 12:24:13.967224 (+351696us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:13.967277 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.967349 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:14.461902 (+494553us) raft_consensus.cc:1685] finished
0709 12:24:14.461934 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.462101 (+   167us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":30311,"raft.run_cpu_time_us":55,"raft.run_wall_time_us":56,"thread_start_us":341499,"threads_started":1,"wal-append.queue_time_us":529935}
I20260709 12:24:14.453704  7318 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.014s
I20260709 12:24:14.467790  7491 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:14.468852  7339 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:14.482327 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:14.482995  7339 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.484721 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3349) took 1453 ms. Trace:
I20260709 12:24:14.485884  7339 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:14.469659 31677 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:14.472133  7466 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:14.480846 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 17 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:14.489866 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:14.485142 31821 rpcz_store.cc:276] 0709 12:24:13.031239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.652176 (+620937us) service_pool.cc:224] Handling call
0709 12:24:13.652508 (+   332us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.674968 (+ 22460us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.675007 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.675516 (+   509us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.309267 (+633751us) raft_consensus.cc:1645] Marking committed up to 9
0709 12:24:14.309313 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.309390 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:14.484512 (+175122us) raft_consensus.cc:1685] finished
0709 12:24:14.484533 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.484659 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24209,"raft.queue_time_us":40204,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":4442,"thread_start_us":588902,"threads_started":1,"wal-append.queue_time_us":788210}
I20260709 12:24:14.489194 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.476233  7519 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:14.490967 31819 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:14.463460  7521 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.500085 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.500519  7295 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.501012 31819 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:24:14.501143  7295 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:14.502137 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.503032 31819 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:14.504124 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.505517 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.506584 31819 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:14.507280 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3342) took 1605 ms. Trace:
I20260709 12:24:14.451328  7432 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:24:14.430078  7590 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.507793 31826 rpcz_store.cc:276] 0709 12:24:12.901665 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.588753 (+687088us) service_pool.cc:224] Handling call
0709 12:24:13.589280 (+   527us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.947825 (+358545us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.947859 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.948344 (+   485us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.233581 (+285237us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:14.233712 (+   131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.234014 (+   302us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:14.507063 (+273049us) raft_consensus.cc:1685] finished
0709 12:24:14.507084 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.507221 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":51551,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":86,"thread_start_us":562184,"threads_started":2,"wal-append.queue_time_us":437836}
I20260709 12:24:14.508160 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:14.509536 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.515839 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1314) took 3384 ms. Trace:
I20260709 12:24:14.516630 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:14.518823 31746 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:14.520103  7432 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:14.451545  7584 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.445817  7412 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.516261 32369 rpcz_store.cc:276] 0709 12:24:11.131341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:11.297700 (+166359us) service_pool.cc:224] Handling call
0709 12:24:12.573988 (+1276288us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.044088 (+470100us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:13.044136 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:13.047929 (+  3793us) log.cc:844] Serialized 26 byte log entry
0709 12:24:13.506925 (+458996us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:13.506986 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:13.507087 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:13.817059 (+309972us) raft_consensus.cc:1685] finished
0709 12:24:13.817082 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.513920 (+696838us) spinlock_profiling.cc:243] Waited 1.28 s on lock 0x7b54005947d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:14.515763 (+  1843us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":42730,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":89,"spinlock_wait_cycles":2934142208,"thread_start_us":863305,"threads_started":2,"wal-append.queue_time_us":719572}
I20260709 12:24:14.526142 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:14.525977 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1416) took 1200 ms. Trace:
I20260709 12:24:14.533772 31821 rpcz_store.cc:276] 0709 12:24:13.325307 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.523056 (+1197749us) service_pool.cc:224] Handling call
0709 12:24:14.525326 (+  2270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:14.525527 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.525565 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:14.525620 (+    55us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:14.525665 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.525741 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.525873 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:14.515058  7493 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:14.543004  7465 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:14.536537 31826 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:14.538231  7584 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.539217 31749 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 21 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:24:14.531158  7432 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:14.539825  7481 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.543395  7573 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.547863 31818 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:14.548143  7590 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:14.548347  7364 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.549253 31826 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:24:14.552069 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.554282 31818 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:14.528956 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 18 index: 9 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:14.556695 31746 raft_consensus.cc:2410] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 23 because replica has last-logged OpId of term: 21 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 9.
W20260709 12:24:14.561821 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.591012 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:14.592092 31746 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:14.593663 31746 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:14.601089 31746 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
W20260709 12:24:14.621104 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.532497  7295 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:14.534987  6752 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:14.650194 31818 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:24:14.650741 31821 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
W20260709 12:24:14.648864  7569 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.113s	user 0.000s	sys 0.001s
I20260709 12:24:14.656461  6752 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.668176  6752 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:14.670956 31821 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:14.658864  7461 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:14.698954 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 281253us
I20260709 12:24:14.659276  7295 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.707985 31818 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:24:14.720342 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:14.723949 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 275810us
I20260709 12:24:14.730314 31821 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:14.684973  7476 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:14.732511 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.689107  7590 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.733570 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:14.735018 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:14.738148 31744 raft_consensus.cc:2410] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
W20260709 12:24:14.738901 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 289188us
I20260709 12:24:14.712239  7567 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.738660 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:14.740595 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 288994us
I20260709 12:24:14.740962 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:14.741920 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:14.742486 31748 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 15. (term mismatch)
W20260709 12:24:14.742481 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.743171 31748 pending_rounds.cc:85] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 14
W20260709 12:24:14.743460 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.743557 31748 pending_rounds.cc:107] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 26.15
I20260709 12:24:14.744059 31748 raft_consensus.cc:2889] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:14.744776 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:14.745540 31675 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:14.746536 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:14.725345  7465 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.725579  7295 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.690943  7447 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.756717 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:14.745483  7477 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:14.762153 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:14.768045 31744 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:14.773628  7592 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:14.734457  7461 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.026s
I20260709 12:24:14.711796  7432 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.766472  7389 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:14.821343  7389 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.796351  7591 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.026s
I20260709 12:24:14.794066  7474 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 12, Time since last communication: 0.005s
I20260709 12:24:14.747335  7567 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.799404  7318 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:14.785364  7550 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:14.823477  7318 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.823954  7318 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:14.825168  7567 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:14.800094  7592 consensus_queue.cc:1243] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 26.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:14.802007 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.828594 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:14.807392  7608 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:24:14.810477  7573 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:14.756453  7541 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.040s
I20260709 12:24:14.819675  7295 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.742754  7590 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:14.837757 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 84847.7us
I20260709 12:24:14.757661  7465 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.784322  7613 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:14.836437 31744 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:14.777930  7466 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:14.843500 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:14.844980 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.822922  7474 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:14.772615  7452 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.071s
I20260709 12:24:14.846660 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:14.852372 31746 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 23 because replica has last-logged OpId of term: 22 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 11.
I20260709 12:24:14.854622 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2747) took 1449 ms. Trace:
W20260709 12:24:14.854494 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 112630us
I20260709 12:24:14.855214 31751 rpcz_store.cc:276] 0709 12:24:13.405606 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.668884 (+263278us) service_pool.cc:224] Handling call
0709 12:24:14.854387 (+1185503us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b54004056d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:14.854540 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":385576704}
I20260709 12:24:14.855986 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:14.856621 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:14.868316 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 111712us
I20260709 12:24:14.869199 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:14.867726 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:14.870189 31744 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:24:14.870545 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:14.851078  7474 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:14.872920 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:14.875464 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:14.859191  7466 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:14.880137 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:14.880918 31751 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:14.865743  7447 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:14.877617  7590 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.885020 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:14.880187 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:14.879635  7504 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:14.874008  7445 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:14.886495  7620 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:14.899276  7620 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.899780  7620 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:14.902062 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:14.903023  7613 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:14.903813  7613 consensus_queue.cc:1243] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 26.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:14.903988 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1167) took 1096 ms. Trace:
I20260709 12:24:14.904237  7334 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:14.904390 31752 rpcz_store.cc:276] 0709 12:24:13.807503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.951674 (+144171us) service_pool.cc:224] Handling call
0709 12:24:13.952017 (+   343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:14.100901 (+148884us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.903740 (+802839us) spinlock_profiling.cc:243] Waited 148 ms on lock 0x7b5400213ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:14.903925 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":340174592}
I20260709 12:24:14.905792 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:14.905102 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:14.906867 31744 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:14.907589 31751 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:24:14.908601 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:14.909773 31744 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:14.879123  7295 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:14.911886 32371 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:14.911901 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:14.908705 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:14.913957 31751 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:14.914300 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1158) took 1964 ms. Trace:
I20260709 12:24:14.914491  7466 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.15, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.890772  7378 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.916772  7389 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.917431  7378 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.915108 32370 rpcz_store.cc:276] 0709 12:24:12.949579 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.026113 (+ 76534us) service_pool.cc:224] Handling call
0709 12:24:13.026451 (+   338us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.059348 (+ 32897us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:14.044387 (+985039us) spinlock_profiling.cc:243] Waited 7.5 ms on lock 0x7b1800408c40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:24:14.527310 (+482923us) log.cc:844] Serialized 14 byte log entry
0709 12:24:14.762427 (+235117us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:14.763001 (+   574us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.763144 (+   143us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:14.763179 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.763250 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:14.913993 (+150743us) raft_consensus.cc:1685] finished
0709 12:24:14.914112 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.914246 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37351,"prepare.queue_time_us":43319,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":84313,"raft.run_cpu_time_us":568,"raft.run_wall_time_us":34258,"spinlock_wait_cycles":17241344,"thread_start_us":703602,"threads_started":2,"wal-append.queue_time_us":314707}
I20260709 12:24:14.894143 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3530) took 1085 ms. Trace:
I20260709 12:24:14.919523 31669 rpcz_store.cc:276] 0709 12:24:13.808859 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.812967 (+  4108us) service_pool.cc:224] Handling call
0709 12:24:13.813383 (+   416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.208652 (+395269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.893887 (+685235us) spinlock_profiling.cc:243] Waited 394 ms on lock 0x7b5400391cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:14.894083 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":905499008}
I20260709 12:24:14.920415 31670 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260709 12:24:14.919806  7378 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.895159 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1321) took 2330 ms. Trace:
I20260709 12:24:14.921169 32375 rpcz_store.cc:276] 0709 12:24:12.565087 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.565287 (+   200us) service_pool.cc:224] Handling call
0709 12:24:12.565701 (+   414us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:13.511713 (+946012us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.894890 (+1383177us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b54004984d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:14.895104 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2151031680}
I20260709 12:24:14.921744 31744 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:14.884018  7567 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.011s
I20260709 12:24:14.922485 31744 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:14.896925  7614 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.014s
W20260709 12:24:14.914784 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.917423 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:14.925359 31745 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:14.886703  7618 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:14.910014 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3343) took 2008 ms. Trace:
I20260709 12:24:14.934764 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.936065 31676 raft_consensus.cc:2410] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 24 because replica has last-logged OpId of term: 22 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 15.
I20260709 12:24:14.935547 31823 rpcz_store.cc:276] 0709 12:24:12.902026 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.602826 (+700800us) service_pool.cc:224] Handling call
0709 12:24:13.603344 (+   518us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.121514 (+518170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.909762 (+788248us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b540037a854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:14.909954 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1164028288}
I20260709 12:24:14.886611  7474 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.873206  7573 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:14.920815 31670 pending_rounds.cc:85] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 15
I20260709 12:24:14.940718 31670 pending_rounds.cc:107] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:24:14.941053 31670 raft_consensus.cc:2889] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:14.893374  7465 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:14.909308 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3416) took 3698 ms. Trace:
I20260709 12:24:14.941982 31753 rpcz_store.cc:276] 0709 12:24:11.210524 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:12.893261 (+1682737us) service_pool.cc:224] Handling call
0709 12:24:12.893571 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:14.083060 (+1189489us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.909088 (+826028us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54001edcd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:14.909252 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2614667136}
I20260709 12:24:14.925757 31745 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:14.926349 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.906484 31752 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:14.863027  7392 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:14.926895 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46869.6us
I20260709 12:24:14.930248 31675 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:14.925628  7584 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.009s
I20260709 12:24:14.943521 31745 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 26.13, Last appended by leader: 13, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.934621 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.937988 31819 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:24:14.940127 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.927502 31744 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.947677 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.948760 31676 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:24:14.949272 31744 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:14.949702 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.945259 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:14.945351 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:14.945772 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.947644 31745 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:14.951292 31675 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:14.952472 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.952615 31675 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 14. (term mismatch)
I20260709 12:24:14.953065 31675 pending_rounds.cc:85] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 13
I20260709 12:24:14.950321 31676 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:24:14.953298 31675 pending_rounds.cc:107] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20260709 12:24:14.953965 31675 raft_consensus.cc:2889] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:14.954321 31676 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:14.953282 31818 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:14.954954 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:14.952768 31753 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: [20.15-23.16]
I20260709 12:24:14.955535 31819 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:14.957628 31753 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:14.935377  7618 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:14.959177 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.959215  7618 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:14.960813 31677 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: [24.14-27.15]
I20260709 12:24:14.962246 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.963079 31823 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:14.964211 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.964983 31826 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:14.967056 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:14.970037  7573 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:14.970881 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.972594 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:14.973204 31669 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:24:14.974398 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.975626 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.976852 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.972728 31826 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 20 index: 15. Preceding OpId from leader: term: 21 index: 15. (term mismatch)
I20260709 12:24:14.979980 31826 pending_rounds.cc:85] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 14
I20260709 12:24:14.980270 31826 pending_rounds.cc:107] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 20.15
I20260709 12:24:14.980295 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:14.980639 31826 raft_consensus.cc:2889] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:14.980481 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:14.981040 31821 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:24:14.981117 31669 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
W20260709 12:24:14.978742 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.982605 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:14.983413 31676 raft_consensus.cc:2410] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:24:14.984201  7378 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 12:24:14.984212  7616 consensus_queue.cc:1175] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b95aeeaf4c51442b8092cff1de674f2b" responder_term: 24 status { last_received { term: 21 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: [23.15-23.15]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:14.986315  7480 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:14.986660  7308 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:14.934588  7445 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:14.987860  7445 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.940722  7398 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:14.990787 32369 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:14.991110  7398 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:14.992241 31752 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:14.994465  7474 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:14.983498 31669 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:15.001821 31752 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:15.005021  7318 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.008194 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:15.009058 32370 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:14.985245  6752 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:14.927603  7412 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:14.993427  7398 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:15.014147 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:15.016199 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:15.036304 31677 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:15.036875  7614 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 27 current term: 27 is higher.
I20260709 12:24:15.037446  7480 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.038187  7318 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:14.942554  7447 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:15.038983  7447 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:15.039901  7447 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.012521  7334 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:15.031442 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.037962  7480 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:15.042476 31819 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:15.032341 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:15.045658 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:15.047788 31675 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 23
I20260709 12:24:15.053544 31744 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:24:15.012558  7304 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:15.030906  7308 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:15.061285 31675 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:15.032981  7378 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:15.064805 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:15.059525 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:15.067435 31826 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:15.069352 31677 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:15.069092 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:15.048353  7518 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:15.033279  7339 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
I20260709 12:24:15.078804 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:15.034801  7584 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:24:15.075373  7466 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:15.052235  7334 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.091461 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:15.065636  7590 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.018s
I20260709 12:24:15.094280 31744 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:15.082159  7304 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:15.096107 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:15.096791  7618 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.084522  7412 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:15.074764  7378 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.104616 31826 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:15.106019  7480 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:15.098166  7304 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.107709  7304 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:15.111254  7412 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.112222 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3533) took 1302 ms. Trace:
I20260709 12:24:15.112603 31674 rpcz_store.cc:276] 0709 12:24:13.809839 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.814866 (+  5027us) service_pool.cc:224] Handling call
0709 12:24:13.815223 (+   357us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.001953 (+186730us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.111946 (+1109993us) spinlock_profiling.cc:243] Waited 186 ms on lock 0x7b5400046fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:15.112160 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":427393280}
I20260709 12:24:15.114439  7412 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:15.116200  7522 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:15.116631  7522 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:15.117005  7618 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:15.117429  7522 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:15.121230  7599 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.395s	user 0.003s	sys 0.000s
I20260709 12:24:15.122203 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:15.123000 31675 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:15.127446 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1414) took 1999 ms. Trace:
I20260709 12:24:15.127753 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:15.128561 32375 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:15.128661 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:15.130319 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:15.130618 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:15.131016 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:15.136507 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:15.132761  7522 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:15.133386  7632 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:24:15.127936 31827 rpcz_store.cc:276] 0709 12:24:13.127703 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.138567 (+1010864us) service_pool.cc:224] Handling call
0709 12:24:14.138983 (+   416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.139222 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.139247 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:14.178767 (+ 39520us) log.cc:844] Serialized 26 byte log entry
0709 12:24:14.492897 (+314130us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:14.493201 (+   304us) log.cc:844] Serialized 14 byte log entry
0709 12:24:14.511395 (+ 18194us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.511511 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:15.120660 (+609149us) spinlock_profiling.cc:243] Waited 3.8 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:15.127098 (+  6438us) raft_consensus.cc:1685] finished
0709 12:24:15.127218 (+   120us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:15.127383 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":626,"prepare.run_cpu_time_us":2679,"prepare.run_wall_time_us":11872,"raft.queue_time_us":11860,"raft.run_cpu_time_us":127,"raft.run_wall_time_us":3079,"spinlock_wait_cycles":8732544,"thread_start_us":309747,"threads_started":1,"wal-append.queue_time_us":479763}
I20260709 12:24:15.091701  7561 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:15.095856  7308 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:15.093123  7392 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:15.153931 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:15.154932 31746 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:15.156167  7514 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:15.159008 32375 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:15.159328  7364 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:15.159767  7364 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:15.160642  7364 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.161660  7364 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:15.165789 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.099803  7590 consensus_queue.cc:1243] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 22.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:15.167768  7633 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:15.168280  7633 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.168617  7633 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:15.170451 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:15.171226 31819 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:15.171532  7378 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:15.173668 31746 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:24:15.173563  7620 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.174768  7561 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:15.175825 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:15.176600 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:15.177034  7561 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:15.177765 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:15.178386 31818 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:24:15.178653  7308 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:15.179622  7308 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.182965 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:15.165839 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 18 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:15.190140 32375 raft_consensus.cc:2410] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 13.
I20260709 12:24:15.191895 31676 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:24:15.101300  6752 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:15.235813 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 43753.1us
I20260709 12:24:15.231266 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:15.236230 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 192607us
I20260709 12:24:15.237053 31642 leader_election.cc:400] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
I20260709 12:24:15.237360 31642 leader_election.cc:403] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Cancelling election due to peer responding with higher term
I20260709 12:24:15.235450 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:15.239886 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:15.104333  7466 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:15.241451 31818 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 13. (term mismatch)
I20260709 12:24:15.245843 31818 pending_rounds.cc:85] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 12
I20260709 12:24:15.246228 31818 pending_rounds.cc:107] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 21.13
I20260709 12:24:15.246565 31818 raft_consensus.cc:2889] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:15.247018 31818 pending_rounds.cc:107] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 23.14
I20260709 12:24:15.247339 31818 raft_consensus.cc:2889] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:15.243461 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:15.244196 31676 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:15.179649  7501 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:15.265427 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:15.194283  7570 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:15.266816 31826 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:15.265542 31642 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:15.250304  7447 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:15.281250 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:15.253993  7378 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election lost for term 20. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
I20260709 12:24:15.293953 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:15.296298 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:15.296557 31744 raft_consensus.cc:2410] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 19 because replica has last-logged OpId of term: 17 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:24:15.296965 31745 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:15.239320  7405 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:24:15.261339  6752 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:15.260190  7501 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:15.301041 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:15.238593  7561 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.033s
W20260709 12:24:15.346087 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44104.2us
I20260709 12:24:15.347323 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:15.348281 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:15.348538  7365 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:15.350476  7405 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:15.350967  7405 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.349076  7628 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:15.352065  7628 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.352500  7628 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:15.355355 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:15.363981 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.325364 31745 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:15.351352  7405 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:15.414885 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:15.268280  7541 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.060s
I20260709 12:24:15.268460  7573 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.058s
W20260709 12:24:15.452677 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:15.465304 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50488.7us
I20260709 12:24:15.475150 31826 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:15.305506  7511 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:15.477572  7511 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:15.470769 31751 raft_consensus.cc:2410] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:24:15.273440  7425 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.096s
I20260709 12:24:15.455255  7295 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:15.484440  7295 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.484829  7295 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:15.456236  7612 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:24:15.493106  7640 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.188s	user 0.000s	sys 0.001s
I20260709 12:24:15.295591  7447 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.480782  7584 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:15.272378  7570 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.356719  7630 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:15.294646  7354 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.175s
I20260709 12:24:15.304823  7480 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.164s
I20260709 12:24:15.371743  7339 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:24:15.504344 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:15.509553 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:15.507511 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:15.508726 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 227757us
I20260709 12:24:15.512004 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:15.359163  7543 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.144s
I20260709 12:24:15.540014 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:15.563314 31677 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:15.478242  7511 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.565263 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:15.411567  7616 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:15.506961 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 205983us
W20260709 12:24:15.555631 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.569024 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:15.623579 31744 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:15.470371  7304 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.571020 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:15.568364 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:15.543771 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 130217us
W20260709 12:24:15.609140 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:15.505628  7612 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:15.496727  7570 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:15.571805 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:15.635093  7647 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
W20260709 12:24:15.656625 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 84918us
I20260709 12:24:15.657572 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:15.669976 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:15.672071 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:15.544345  7648 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:15.566439  7339 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:15.562454  7630 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:15.693631  7432 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:15.695312 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:15.466749  7466 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.701956 31818 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: [22.14-22.14]
W20260709 12:24:15.703325 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 428781us
I20260709 12:24:15.656644  7620 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:15.561457  7446 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.142s
I20260709 12:24:15.675169  7561 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:15.682928  7378 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:15.682009  7308 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:15.709466  7648 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.713910  7412 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.714013  7591 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.714414  7425 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.709036  7570 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:15.715878 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:15.717329 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45382.8us
I20260709 12:24:15.718254  7612 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:15.718999 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:15.726300  7463 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:15.725880 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:15.790146 31827 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:15.625721  7501 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.12, Last appended by leader: 12, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.704027  7339 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:15.792125 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 73210.9us
I20260709 12:24:15.793906 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:15.795490 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 79702.2us
I20260709 12:24:15.641197  7616 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.787645 31674 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:15.811549 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:15.743196  7620 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:15.666235  7584 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:15.814369 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:15.817528 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:15.822912  7649 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
I20260709 12:24:15.816464  7308 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:15.808966  7550 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.021s
I20260709 12:24:15.802480  7463 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.786996  7570 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:15.830186  7655 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:15.794849  7474 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:24:15.836088  7616 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:15.788926  7425 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:15.845484 31827 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:15.868256 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:15.870138 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:15.707152  7652 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.125s
I20260709 12:24:15.853776 31674 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:15.802721  7445 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:15.880699 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:15.881919 32374 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 1113) took 2133 ms (client timeout 1784 ms). Trace:
I20260709 12:24:15.858495  7501 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:15.780133  7412 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:15.879550 31675 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.696s	user 0.082s	sys 0.093s
I20260709 12:24:15.864741  7570 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:15.810524  7541 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:15.892746 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:15.882468 32374 rpcz_store.cc:269] 0709 12:24:13.748543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.748706 (+   163us) service_pool.cc:224] Handling call
0709 12:24:15.880580 (+2131874us) spinlock_profiling.cc:243] Waited 1.24 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:24:15.881758 (+  1178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2851584}
I20260709 12:24:15.862807  7463 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:15.895720 31675 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.712s	user 0.083s	sys 0.093s
I20260709 12:24:15.786724  7591 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:15.815739  7378 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.060s
I20260709 12:24:15.814204  7447 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:15.857903  7403 consensus_queue.cc:1175] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "039d66a6c85e43c3936bf5017501b344" responder_term: 24 status { last_received { term: 23 index: 12 } last_committed_idx: 12 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:15.854743  7620 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:15.888087  7616 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:15.918016  7518 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:24:15.929016  7522 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.780s	user 0.013s	sys 0.000s
I20260709 12:24:15.924562  7511 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:15.880095  7339 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:16.099862 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:14.079522 31747 rpcz_store.cc:276] 0709 12:24:11.955832 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.093785 (+1137953us) service_pool.cc:224] Handling call
0709 12:24:13.094223 (+   438us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:13.094524 (+   301us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:13.730749 (+636225us) log.cc:844] Serialized 14 byte log entry
0709 12:24:14.021914 (+291165us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:14.021980 (+    66us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:14.022037 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:14.022103 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:14.022239 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":162991,"prepare.run_cpu_time_us":27,"prepare.run_wall_time_us":27,"raft.queue_time_us":14633,"raft.run_cpu_time_us":276,"raft.run_wall_time_us":6176,"thread_start_us":870361,"threads_started":2}
I20260709 12:24:16.070502  7616 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:16.108857  7616 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:16.129220 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 308488us
I20260709 12:24:15.931293  7304 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:16.103116  7642 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.379s	user 0.001s	sys 0.000s
I20260709 12:24:16.070112  7630 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.099937  7541 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.103258  7663 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:16.056070  6752 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:15.928963  7584 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 21.10, Last appended by leader: 10, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.132422  7339 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:16.069864  7651 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.239s
I20260709 12:24:16.162858 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1166) took 1155 ms. Trace:
I20260709 12:24:16.164192  7412 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:16.165467  7648 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:16.170122  7633 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.001s	user 0.012s	sys 0.000s
I20260709 12:24:16.152721  7663 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:16.156483  7304 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:16.156100  7541 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.215130  7591 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:16.220083  7620 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.221774  7663 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.223707  7663 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:16.200426  7425 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:16.229343  7405 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.749s	user 0.013s	sys 0.001s
I20260709 12:24:16.229928  7405 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:16.200371  7304 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.192939  7633 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.233677  7633 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.204818  7541 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:16.251586  7628 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.899s	user 0.014s	sys 0.000s
W20260709 12:24:16.290275 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.296567 31674 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260709 12:24:16.299937 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 482248us
I20260709 12:24:16.300034 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:16.301875 31674 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:16.140926 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:16.144578 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:16.318001 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:16.323086 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 420010us
I20260709 12:24:16.178123 32369 rpcz_store.cc:276] 0709 12:24:15.007059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.010024 (+  2965us) service_pool.cc:224] Handling call
0709 12:24:15.010519 (+   495us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:15.011568 (+  1049us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:15.012348 (+   780us) log.cc:844] Serialized 14 byte log entry
0709 12:24:15.252951 (+240603us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:15.253426 (+   475us) log.cc:844] Serialized 26 byte log entry
0709 12:24:15.253559 (+   133us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:15.253589 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.253657 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.162513 (+908856us) raft_consensus.cc:1685] finished
0709 12:24:16.162538 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.162790 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46876,"prepare.queue_time_us":194,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":35,"raft.queue_time_us":47285,"raft.run_cpu_time_us":186,"raft.run_wall_time_us":187,"thread_start_us":193190,"threads_started":1,"wal-append.queue_time_us":306778}
I20260709 12:24:16.325945 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:16.326617 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:16.327442 31826 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 28 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 14.
I20260709 12:24:16.328177  7684 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:16.330981 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:16.331894 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:16.155710  7445 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:16.342566  7405 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:16.343894  7648 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:24:16.234912  7295 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.734s	user 0.014s	sys 0.000s
I20260709 12:24:16.345255  7612 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.345791 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:16.253261  7628 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.351274 32375 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:16.353897  7480 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:16.250432  7570 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.363900  7480 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.359779  7648 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.366900  7648 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:16.173730  6752 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.367338  7445 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.361786  7466 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:16.370467  6752 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.370570  7445 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.378190  7684 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:16.370939 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:16.391495 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.001s	sys 0.009s
I20260709 12:24:16.392243  7295 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.395537 31826 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:16.408572 32370 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:16.411746 32374 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
W20260709 12:24:16.410036 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.002s	sys 0.010s
I20260709 12:24:16.363955  7630 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:16.406077  7684 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:16.371857  7466 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.372514  7612 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.452520 31674 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:16.443037  7684 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:16.459964  7425 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:16.461050  7425 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.463029  7612 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:16.471832  7304 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.472522 31827 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:16.472911 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:16.473016  7570 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.471962  7466 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:16.474900  7405 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:16.475529  7405 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:16.476140  7405 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:16.478729 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:16.475975 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:16.480831  7620 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.482798  7620 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.478830  6752 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:16.457204  7480 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.498257 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:16.503713  7679 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.504587  7679 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.507308  7679 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:16.509140  7570 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:16.509935  7570 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.511905  7432 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:16.513468  7570 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:16.514886 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 66526.8us
I20260709 12:24:16.475186  7630 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.516481 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:16.517870  7679 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:16.521399  7378 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:16.521515 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:16.522078  7378 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:16.522344 32378 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:16.522536  7378 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:16.484627  6752 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.532255  6752 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.525885  7522 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:16.558933 31826 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:16.560828 32370 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:16.518501  7630 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.547639 31746 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:24:16.562563 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:16.552172 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:16.554579  7620 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:16.555598  7628 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:16.561246  7295 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:16.583180 31749 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:16.581208 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:16.590811  7541 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.016s
I20260709 12:24:16.592890  7425 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:16.599399 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:16.600523 31753 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:24:16.600670 31744 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 12:24:16.602048 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:16.602363 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:16.603276  7691 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:16.605033 31744 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: [23.13-23.13]
I20260709 12:24:16.604205  7691 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.576021  7630 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.608641 31744 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:24:16.609338  7691 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:16.609376  7463 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.689s	user 0.015s	sys 0.013s
I20260709 12:24:16.610742 31744 pending_rounds.cc:85] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 12
I20260709 12:24:16.611196 31744 pending_rounds.cc:107] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 26.13
I20260709 12:24:16.611349  7480 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:16.611626 31744 raft_consensus.cc:2889] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:16.612030  7480 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:16.615383 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:16.615903  7480 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.613444  7628 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.619197  7628 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.582777  6752 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:16.622855  6752 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:16.622953  7630 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.623543  6752 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:16.610114  7463 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.624969  7378 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.612790 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:16.625564  7630 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:16.631290 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.627143  7463 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.629065  7628 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:16.637007  7378 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:16.640049 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 123846us
I20260709 12:24:16.583334  7684 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.640897  7378 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:16.642947  7378 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:16.641242 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:16.643453  7378 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:16.656415 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3562) took 1535 ms. Trace:
I20260709 12:24:16.656971 31677 rpcz_store.cc:276] 0709 12:24:15.120769 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.151674 (+ 30905us) service_pool.cc:224] Handling call
0709 12:24:16.656017 (+1504343us) spinlock_profiling.cc:243] Waited 323 ms on lock 0x7b54001810d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:16.656214 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":742253440}
W20260709 12:24:16.661094 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.663995  7648 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:16.662242  7501 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.757s	user 0.003s	sys 0.020s
I20260709 12:24:16.666870  7648 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:16.643726  7684 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:16.657831  7405 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.669853 32378 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:16.672439 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:16.667405 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3436) took 3167 ms. Trace:
I20260709 12:24:16.673368 31826 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:16.674157  7522 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:16.679235 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3448) took 1776 ms. Trace:
I20260709 12:24:16.676301 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3442) took 2836 ms. Trace:
I20260709 12:24:16.676575  7511 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:16.675222  7396 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:16.686744 31751 rpcz_store.cc:276] 0709 12:24:14.901683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.914541 (+ 12858us) service_pool.cc:224] Handling call
0709 12:24:16.677309 (+1762768us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b5400294fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:16.677572 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1250256128}
I20260709 12:24:16.680713 31748 rpcz_store.cc:276] 0709 12:24:13.500388 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.934920 (+434532us) service_pool.cc:224] Handling call
0709 12:24:13.938203 (+  3283us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:14.744680 (+806477us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.666868 (+1922188us) spinlock_profiling.cc:243] Waited 802 ms on lock 0x7b54003e30d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:16.667201 (+   333us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1843502464}
W20260709 12:24:16.695961  7339 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.514s	user 0.003s	sys 0.019s
I20260709 12:24:16.696931  7679 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:24:16.698777 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.687928 31750 rpcz_store.cc:276] 0709 12:24:13.839862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:13.964846 (+124984us) service_pool.cc:224] Handling call
0709 12:24:13.994151 (+ 29305us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:14.101414 (+107263us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.101461 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:14.102061 (+   600us) log.cc:844] Serialized 26 byte log entry
0709 12:24:15.134618 (+1032557us) spinlock_profiling.cc:243] Waited 5.82 ms on lock 0x7b18001e88c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:24:15.239369 (+104751us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:15.239407 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.675682 (+1436275us) spinlock_profiling.cc:243] Waited 94.5 ms on lock 0x7b5400064d54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:16.675777 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.675854 (+    77us) raft_consensus.cc:1685] finished
0709 12:24:16.675873 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.676112 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9173,"raft.queue_time_us":13161,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":6821,"spinlock_wait_cycles":230803584,"thread_start_us":104537,"threads_started":1,"wal-append.queue_time_us":1370435,"wal-append.run_cpu_time_us":1757,"wal-append.run_wall_time_us":1063955}
I20260709 12:24:16.699817  7630 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:16.703148 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:16.704885 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 63884.7us
I20260709 12:24:16.705045  6752 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.705631 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:16.707425 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:16.708352 31827 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:16.709363 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3372) took 1235 ms. Trace:
I20260709 12:24:16.710909  7628 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:16.711238  7295 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.711759  7628 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.712101  7295 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.713091  7584 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.713455  7378 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.713820  7584 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.714907 31748 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:16.715099  7522 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:16.715704  7522 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:16.717132  7522 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.718170 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:16.719040  7511 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:16.719550  7511 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:16.720141 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:16.704815 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:16.706274 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:16.709801 31818 rpcz_store.cc:276] 0709 12:24:15.474315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.476267 (+  1952us) service_pool.cc:224] Handling call
0709 12:24:15.476670 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:15.714876 (+238206us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.708738 (+993862us) spinlock_profiling.cc:243] Waited 225 ms on lock 0x7b54004470d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:16.709144 (+   406us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":517490560}
W20260709 12:24:16.718251 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.724433  7396 consensus_queue.cc:1243] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 24.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:16.725015 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3370) took 1594 ms. Trace:
I20260709 12:24:16.725260 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:16.725553  7616 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:16.725517 31821 rpcz_store.cc:276] 0709 12:24:15.130587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.130753 (+   166us) service_pool.cc:224] Handling call
0709 12:24:16.724502 (+1593749us) spinlock_profiling.cc:243] Waited 497 ms on lock 0x7b5400447fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:16.724822 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1143115648}
I20260709 12:24:16.725277  7396 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:16.726922  7396 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:16.726984  7378 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:16.726136 31818 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:16.727770 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:16.735117 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:16.714425  7628 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.738227 31749 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:16.735203  7396 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.744715 31822 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 27: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:16.755678 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:16.755980 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:16.756759 31821 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
W20260709 12:24:16.757776 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.758003 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:16.761059  7396 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:16.761333 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:16.765820 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:16.720912  7511 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:16.767632  7511 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:16.726424  7616 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:16.768466 31820 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:16.774814  7684 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:16.775650  7684 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.776716 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2768) took 1209 ms. Trace:
I20260709 12:24:16.778117  7684 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:16.782572 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:16.780704 31818 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:16.781059 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:16.781895  7378 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:16.784268  7378 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.787592  7378 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.788905  7616 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:16.790176  7334 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:16.779099  7648 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:16.770465  7630 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:24:16.787627 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 81594.6us
I20260709 12:24:16.771497  7522 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:16.770428  7405 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:16.792569 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.793305  7648 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.794996  7339 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:16.795300 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:16.795842  7501 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:16.797999 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:16.798192  7430 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.799474  7460 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.802313  7481 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:16.804353 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 20 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:16.806618  7518 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:16.808698 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:16.812505 31748 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:16.791903 31827 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:24:16.818779 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:16.779827 31745 rpcz_store.cc:276] 0709 12:24:15.567103 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.574680 (+  7577us) service_pool.cc:224] Handling call
0709 12:24:15.575103 (+   423us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:15.861283 (+286180us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:15.861321 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:15.867209 (+  5888us) log.cc:844] Serialized 26 byte log entry
0709 12:24:16.665588 (+798379us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b18003f2ce0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:24:16.706933 (+ 41345us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:16.707069 (+   136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.708080 (+  1011us) spinlock_profiling.cc:243] Waited 261 ms on lock 0x7b54002963d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:16.708140 (+    60us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.733206 (+ 25066us) raft_consensus.cc:1685] finished
0709 12:24:16.733230 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.776511 (+ 43281us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":50252,"raft.queue_time_us":24453,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":10338,"spinlock_wait_cycles":625275520,"thread_start_us":25402,"threads_started":1,"wal-append.queue_time_us":806784}
I20260709 12:24:16.830706 31822 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:16.831491 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:16.826836  7518 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:16.823938  7481 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:16.845847 32371 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:16.847388 31826 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:16.847970  7501 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:16.841164  6752 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:16.832425  7339 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.855020  7501 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:16.844959  7463 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:16.862329 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:16.866851  7481 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:16.868844 31826 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:16.870098  7481 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:16.864210  7501 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:16.875569 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:16.878129 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59557.5us
I20260709 12:24:16.871160  7648 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:16.887441 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:16.880714  7691 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:16.899258 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1281) took 2108 ms. Trace:
I20260709 12:24:16.899842  7403 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:16.912827 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:16.885237  7463 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:16.916684 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:16.912922 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:16.923837 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.918929  7403 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:16.924114  7697 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.098s	user 0.000s	sys 0.002s
I20260709 12:24:16.909534 31671 rpcz_store.cc:276] 0709 12:24:14.725246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.831493 (+106247us) service_pool.cc:224] Handling call
0709 12:24:14.833574 (+  2081us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:14.833793 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:14.853263 (+ 19470us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.642408 (+1789145us) spinlock_profiling.cc:243] Waited 4.17 ms on lock 0x7b1800210fa0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:24:16.814024 (+171616us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:16.817273 (+  3249us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:16.817330 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.826491 (+  9161us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.833450 (+  6959us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8128,"prepare.queue_time_us":18359,"prepare.run_cpu_time_us":39,"prepare.run_wall_time_us":39,"raft.queue_time_us":18302,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":130,"spinlock_wait_cycles":9596672,"thread_start_us":128097,"threads_started":1,"wal-append.queue_time_us":1911951}
I20260709 12:24:16.932204 31746 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:16.932756 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:17.017640 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:17.018244 31746 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
W20260709 12:24:16.924518 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:17.019400 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:16.933496  7339 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:16.937026  7643 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.375s	user 0.001s	sys 0.000s
W20260709 12:24:16.938294  7644 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.372s	user 0.002s	sys 0.000s
I20260709 12:24:16.938774  7403 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:17.024765 31752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1173) took 2081 ms (client timeout 1847 ms). Trace:
W20260709 12:24:17.025290 31752 rpcz_store.cc:269] 0709 12:24:14.943297 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.966809 (+ 23512us) service_pool.cc:224] Handling call
0709 12:24:17.024412 (+2057603us) spinlock_profiling.cc:243] Waited 24.6 ms on lock 0x7b54004042d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:17.024601 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":56535424}
I20260709 12:24:17.020547 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:16.943125  7699 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.074s	user 0.000s	sys 0.001s
W20260709 12:24:16.948311  7658 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.783s	user 0.000s	sys 0.001s
I20260709 12:24:17.029706 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1287) took 1848 ms. Trace:
I20260709 12:24:17.030408 31747 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.18-24.18]
I20260709 12:24:16.955631 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3366) took 1983 ms. Trace:
I20260709 12:24:17.031369 31823 rpcz_store.cc:276] 0709 12:24:14.971914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.975414 (+  3500us) service_pool.cc:224] Handling call
0709 12:24:14.975710 (+   296us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:14.976081 (+   371us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.976114 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:14.976795 (+   681us) log.cc:844] Serialized 52 byte log entry
0709 12:24:15.044245 (+ 67450us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:15.070525 (+ 26280us) log.cc:844] Serialized 14 byte log entry
0709 12:24:15.071084 (+   559us) log.cc:844] Serialized 14 byte log entry
0709 12:24:15.071296 (+   212us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.071414 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.929596 (+1858182us) spinlock_profiling.cc:243] Waited 3.59 ms on lock 0x7b1800458620. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1
0709 12:24:16.930265 (+   669us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b1800458620. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:16.955262 (+ 24997us) raft_consensus.cc:1685] finished
0709 12:24:16.955291 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.955474 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27032,"prepare.queue_time_us":41109,"prepare.run_cpu_time_us":86,"prepare.run_wall_time_us":89,"raft.queue_time_us":3678,"raft.run_cpu_time_us":213,"raft.run_wall_time_us":4308,"spinlock_wait_cycles":16791552,"thread_start_us":67011,"threads_started":1,"wal-append.queue_time_us":107452}
I20260709 12:24:16.963748 31674 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:17.036044 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.036417  7295 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:17.037423 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:16.977174  7242 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:17.039101  7242 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:17.039403 31824 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:24:17.040350  7242 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.041437  7398 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:17.041749  7242 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:17.041916  7398 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:16.966519 31672 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:24:17.043802 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.044786 31672 raft_consensus.cc:2410] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 20 because replica has last-logged OpId of term: 19 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 18 index: 12.
I20260709 12:24:16.955179  7684 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:17.048731  7684 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:17.049134  7684 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:16.952013  7671 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.752s	user 0.000s	sys 0.001s
I20260709 12:24:17.051159  7551 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:16.981650  7463 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:17.046298 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.057040 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.058255 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.058938  7501 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.060796  7398 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:17.062250  7398 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:16.982239 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1278) took 2761 ms. Trace:
I20260709 12:24:17.063660 31670 rpcz_store.cc:276] 0709 12:24:14.220850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.450407 (+229557us) service_pool.cc:224] Handling call
0709 12:24:14.920042 (+469635us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:14.941500 (+ 21458us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.981756 (+2040256us) spinlock_profiling.cc:243] Waited 469 ms on lock 0x7b540022b850. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:16.982018 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1079245184}
I20260709 12:24:17.065303 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:16.979844 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1285) took 1798 ms. Trace:
I20260709 12:24:17.068027 31669 rpcz_store.cc:276] 0709 12:24:15.180968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.184342 (+  3374us) service_pool.cc:224] Handling call
0709 12:24:15.184638 (+   296us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:15.194347 (+  9709us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:15.231106 (+ 36759us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:15.231698 (+   592us) log.cc:844] Serialized 52 byte log entry
0709 12:24:15.345077 (+113379us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:16.862178 (+1517101us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b18003cde40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:16.978433 (+116255us) spinlock_profiling.cc:243] Waited 46.1 ms on lock 0x7b18003cde40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf
0709 12:24:16.978726 (+   293us) log.cc:844] Serialized 14 byte log entry
0709 12:24:16.979187 (+   461us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.979341 (+   154us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.979418 (+    77us) raft_consensus.cc:1685] finished
0709 12:24:16.979437 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.979629 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7,"prepare.queue_time_us":146652,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":76666,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":47434,"spinlock_wait_cycles":164756736,"thread_start_us":191181,"threads_started":2,"wal-append.queue_time_us":203349}
I20260709 12:24:16.991003  7514 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:16.994882 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.071674 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:17.072871 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:17.073589 31676 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:17.073653  7463 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:17.074324  7463 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:16.995719 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1415) took 3728 ms. Trace:
I20260709 12:24:17.074803  7679 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:17.075146 31825 rpcz_store.cc:276] 0709 12:24:13.267476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:14.231262 (+963786us) service_pool.cc:224] Handling call
0709 12:24:14.231661 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:14.249429 (+ 17768us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:14.249461 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:14.249506 (+    45us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:14.249543 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.995159 (+2745616us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x7b5400196ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:16.995262 (+   103us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:16.995504 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":40345472}
I20260709 12:24:17.076333  7501 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:17.076411  7679 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:17.082413 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:17.083137 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:17.009764  7613 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260709 12:24:17.030071 31675 rpcz_store.cc:276] 0709 12:24:15.181478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.182352 (+   874us) service_pool.cc:224] Handling call
0709 12:24:15.182678 (+   326us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:15.182986 (+   308us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:15.183012 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:15.183427 (+   415us) log.cc:844] Serialized 26 byte log entry
0709 12:24:15.911873 (+728446us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:15.911929 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:15.912031 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:16.947250 (+1035219us) spinlock_profiling.cc:243] Waited 55.9 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:17.029320 (+ 82070us) raft_consensus.cc:1685] finished
0709 12:24:17.029340 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:17.029540 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2122,"raft.run_cpu_time_us":146,"raft.run_wall_time_us":43184,"spinlock_wait_cycles":128573824,"thread_start_us":728087,"threads_started":1,"wal-append.queue_time_us":981300}
I20260709 12:24:17.086508 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:17.086596 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.087236 31822 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:24:17.088306 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:17.088408 31675 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 15. Preceding OpId from leader: term: 21 index: 16. (index mismatch)
I20260709 12:24:17.089627 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:16.994009 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.084060 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:16.999279  7584 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:17.086549 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:17.084080 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1171) took 1268 ms. Trace:
I20260709 12:24:17.093218 32375 rpcz_store.cc:276] 0709 12:24:15.815987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:15.816178 (+   191us) service_pool.cc:224] Handling call
0709 12:24:15.816519 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:16.389312 (+572793us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.083782 (+694470us) spinlock_profiling.cc:243] Waited 523 ms on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:17.084005 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1203855104}
I20260709 12:24:17.090497 31678 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
W20260709 12:24:16.950621  7668 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.803s	user 0.001s	sys 0.000s
I20260709 12:24:17.096637 31823 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:24:17.084029 31825 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:16.998333 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.098832 31671 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
W20260709 12:24:16.992646 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 101475us
I20260709 12:24:17.099963 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:17.100958  7339 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:17.101375  7339 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:17.101719  7339 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:16.997316 31673 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:17.111896  7590 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.112540  7590 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:17.113595  7396 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:17.115928  7396 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:17.116339  7396 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:17.117339  7396 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.119668 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:17.119714 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 93079.2us
I20260709 12:24:17.121564 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:17.121973  7392 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:16.987221 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:17.123545 31745 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:17.123654  7561 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:17.125257 31750 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:17.126780  7396 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:17.122449 31747 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:17.128032 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:17.128976 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1428) took 1329 ms. Trace:
I20260709 12:24:17.129441 31819 rpcz_store.cc:276] 0709 12:24:15.799939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.332365 (+532426us) service_pool.cc:224] Handling call
0709 12:24:16.332991 (+   626us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:16.534381 (+201390us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:16.534436 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:16.555518 (+ 21082us) log.cc:844] Serialized 26 byte log entry
0709 12:24:16.691971 (+136453us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:16.692234 (+   263us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:16.692353 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:17.128286 (+435933us) spinlock_profiling.cc:243] Waited 7.14 ms on lock 0x7b54001f3c20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4adbe 00007fe8fcd55930 00007fe8fcd558d8 00007fe8fcd55858 00007fe8fcd55813 00007fe8fcd54a9c 00007fe902c402f6 00007fe902c3aaa8 00007fe8fcd917cb 00007fe8fcd942c0
0709 12:24:17.128676 (+   390us) raft_consensus.cc:1685] finished
0709 12:24:17.128695 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:17.128910 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":31831,"raft.run_cpu_time_us":264,"raft.run_wall_time_us":264,"spinlock_wait_cycles":16430848,"thread_start_us":279447,"threads_started":2,"wal-append.queue_time_us":126822}
I20260709 12:24:17.075580  7463 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:17.141444 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 18 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:17.126699 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:17.120537 31752 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:17.143599 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:17.146507 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46738.7us
I20260709 12:24:17.147720 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.126518 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:17.149772 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.149982 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:17.066490 31670 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 24 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
W20260709 12:24:17.152208 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.158581 32371 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:17.173418 31820 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
W20260709 12:24:17.144840  7701 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.315s	user 0.000s	sys 0.004s
I20260709 12:24:17.172390  7551 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:17.189141 31669 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:17.183856 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.204463 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:17.206844  7673 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:17.208483 31678 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
W20260709 12:24:17.220543 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:17.221042 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.182114  7501 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:17.183096  7551 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:17.227217 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.230834  7551 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:17.235136  7663 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:17.236006  7663 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.240329 31674 raft_consensus.cc:2410] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 26 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:24:17.244393 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:17.245393 31672 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:17.246776  7531 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.211037  7590 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:17.247390  7531 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:17.248402 31748 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 18.
I20260709 12:24:17.249675  7531 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:17.259126 31673 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:17.252233  7295 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.193099  7684 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.265113 31670 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:24:17.273846 31669 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:17.276082 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.258155  7522 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:17.274968  7691 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:17.267727  7663 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:17.283004 31820 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:24:17.283815 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.261828  7551 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.190992 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:17.287075  7354 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.003s
I20260709 12:24:17.268800  7620 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.251870  7518 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.289058 31747 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:17.290766 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:17.290906 31820 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 9. Preceding OpId from leader: term: 23 index: 11. (index mismatch)
I20260709 12:24:17.294821  7684 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:17.261873  7295 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:17.296452 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:17.297372 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 121617us
I20260709 12:24:17.300786 31673 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:17.298455 31670 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
W20260709 12:24:17.306318 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 115424us
I20260709 12:24:17.308205 31819 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:17.264933  7501 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:17.317487 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.311939 31676 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:17.307200 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:17.317028 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:17.323323 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.325574 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.324597 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.328581 31673 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:17.326450 31670 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:17.335659 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:17.336501 31676 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:17.337859 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.339840 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:17.309147  7511 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.334542 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.337651 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:17.344552 31674 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 26 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:24:17.345018 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:17.341686 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51150.7us
I20260709 12:24:17.347743 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:17.349638 31750 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 29 because replica has last-logged OpId of term: 27 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:24:17.337994  7531 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election won for term 18
I20260709 12:24:17.400442 31750 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
I20260709 12:24:17.350281 31751 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:17.333734  7430 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:17.315403  7620 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.403551 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.352465 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:17.352593  7691 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:17.411226 31744 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:17.364850 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:17.348927  7570 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:17.420001 31753 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 14. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:17.420694 31670 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 26 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
W20260709 12:24:17.425408 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 73022us
I20260709 12:24:17.415987  7365 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 11, Time since last communication: 0.111s
I20260709 12:24:17.407274  7460 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.024s
W20260709 12:24:17.444417 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 125330us
I20260709 12:24:17.427407 31753 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
W20260709 12:24:17.426697 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.447966 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 18 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:17.431763 32371 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 16. (term mismatch)
W20260709 12:24:17.462222 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 97500.1us
I20260709 12:24:17.454051 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:17.447319 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:17.462306 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.465042 31745 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:17.349159  7673 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.075s
I20260709 12:24:17.409946  7430 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:17.446807 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:17.424062 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:17.463492 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:17.462983 32371 pending_rounds.cc:85] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 15
I20260709 12:24:17.446169 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:17.508498 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 89408.2us
I20260709 12:24:17.453445 31677 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 10. Preceding OpId from leader: term: 23 index: 11. (index mismatch)
I20260709 12:24:17.511960 31672 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
W20260709 12:24:17.486284 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.517123  7378 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.517803  7378 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.403467  7446 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.252s
I20260709 12:24:17.522501  7465 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:17.523056  7465 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:17.439524  7531 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:17.523458  7465 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:17.523563  7501 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:17.524353  7531 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 18, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:17.524978  7445 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.013s
I20260709 12:24:17.526767 32371 pending_rounds.cc:107] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 23.16
I20260709 12:24:17.527109 32371 raft_consensus.cc:2889] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:17.527397 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:17.511586  7430 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:17.449051 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:17.352746  7613 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.155s
I20260709 12:24:17.537405 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:17.538180  7432 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:17.538713  7432 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:17.539754  7432 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.541211  7432 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:17.542402 32371 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 15. (index mismatch)
I20260709 12:24:17.544996 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:17.548681 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.549461 31672 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:24:17.546332 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:17.436470  7620 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:17.551260 32375 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:17.553807 31669 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:17.446789  7295 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:17.350911  7511 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:17.444629  7654 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.520s	user 0.028s	sys 0.017s
I20260709 12:24:17.422983  7570 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:17.436764  7648 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.580734 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:17.596396 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 133885us
I20260709 12:24:17.597992 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:17.594692  7570 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:17.426929  7584 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:17.579341 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:17.440759  7551 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:17.427667  7518 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:17.581601 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:17.583030  7592 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:17.586968 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:17.589450  7339 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:17.575067  7654 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.731s	user 0.028s	sys 0.023s
I20260709 12:24:17.590837  7543 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 18.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:17.591997 31744 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:24:17.580574 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:17.629606 31745 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:17.578346  7501 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:17.423847  7691 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:17.610699 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:17.621006 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.574061  7460 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.062s
I20260709 12:24:17.610773  7584 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:17.636101 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:17.600278  7365 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:17.623803 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.634605 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:17.580098  7392 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.045s
W20260709 12:24:17.623235  7493 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s	user 0.011s	sys 0.044s
W20260709 12:24:17.632306 31746 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.010s	sys 0.015s
W20260709 12:24:17.650568 31746 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.010s	sys 0.015s
I20260709 12:24:17.649367 31670 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:24:17.625582  7620 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election won for term 21
W20260709 12:24:17.628057  7403 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.015s	sys 0.009s
I20260709 12:24:17.635934 32342 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:17.624217  7592 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:17.599910  7445 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.048s
I20260709 12:24:17.626053  7630 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.631191  7673 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:17.707832 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.709139 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:17.607873  7465 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:17.633443  7295 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:24:17.648460  7493 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.011s	sys 0.045s
I20260709 12:24:17.715359 31745 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:17.637887  7691 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:17.731902 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 56970.5us
W20260709 12:24:17.678761  7403 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.016s	sys 0.009s
I20260709 12:24:17.647926  7590 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:17.647675  7551 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:17.623885  7648 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.753883  7648 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:17.754606  7551 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:17.755352  7551 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:17.652599  7518 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:17.764045  7465 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:17.766863  7339 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:17.771601  7378 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:17.772195  7365 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:17.764134 31674 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:17.776628  7551 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.750248  7630 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:17.785547  7630 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:17.759889  7590 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.648144  7584 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:17.787168 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 176655us
I20260709 12:24:17.787985  7590 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:17.788748 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:17.791703  7446 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:17.792245  7446 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:17.792670  7446 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:17.798452 31745 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:24:17.719600  7543 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:17.816797 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:17.823526 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 410229us
I20260709 12:24:17.734251  7392 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:17.650117  7501 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:17.732044  7620 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:17.861819 31753 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:17.877501 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:17.879464 31671 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.018s	sys 0.028s
W20260709 12:24:17.879865 31671 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.604s	user 0.018s	sys 0.028s
I20260709 12:24:17.863785 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:17.824685  7543 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:17.840579  7673 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:17.847575  7398 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:17.898293 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:17.861522  7493 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:24:17.919627 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 130991us
I20260709 12:24:17.920302 31669 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:24:17.921692 31792 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:17.923446  7465 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:17.879043  7663 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.061s
I20260709 12:24:17.907708  6752 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 8, Time since last communication: 0.025s
I20260709 12:24:17.927251  7543 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.928556  7543 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:17.895780  7481 raft_consensus.cc:2764] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:24:17.940017 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:17.952023  7446 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.955204  7430 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.956537 31819 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:17.957656  7430 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:17.904151  7398 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:17.916667  7318 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.589s	user 0.010s	sys 0.013s
I20260709 12:24:17.967285 31753 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
W20260709 12:24:17.970188 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:17.973811 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:17.987737 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:17.977766  7318 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.011s	sys 0.014s
I20260709 12:24:17.989588 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:17.990718 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 50813.5us
I20260709 12:24:17.991060 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:17.847942  7392 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:17.954514  7446 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:17.978906  7398 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:18.002555 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:17.994040  7430 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:18.040194  7430 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:18.040791  7430 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:18.040902  7392 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:18.041174  7430 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:18.044158  7511 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:18.044788 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:18.047181 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 44687.3us
W20260709 12:24:18.053640  7705 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.347s	user 0.001s	sys 0.000s
W20260709 12:24:18.056600  7432 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.515s	user 0.008s	sys 0.004s
I20260709 12:24:18.057612 32377 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 25 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
W20260709 12:24:18.058686 31793 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.756s	user 0.012s	sys 0.024s
W20260709 12:24:18.059046 31793 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.012s	sys 0.025s
I20260709 12:24:17.972746  7673 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 12:24:18.068131 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1435) took 1477 ms. Trace:
I20260709 12:24:17.996862  7481 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:18.001128  6752 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.069613 31827 rpcz_store.cc:276] 0709 12:24:16.590363 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.827253 (+236890us) service_pool.cc:224] Handling call
0709 12:24:16.827606 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:16.838391 (+ 10785us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:16.838412 (+    21us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:16.858257 (+ 19845us) log.cc:844] Serialized 26 byte log entry
0709 12:24:17.317981 (+459724us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:17.318034 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.318132 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:18.052841 (+734709us) spinlock_profiling.cc:243] Waited 7.93 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:18.067764 (+ 14923us) raft_consensus.cc:1685] finished
0709 12:24:18.067790 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.068048 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3827,"raft.queue_time_us":1958,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":788,"spinlock_wait_cycles":19559680,"thread_start_us":426977,"threads_started":1,"wal-append.queue_time_us":720354}
I20260709 12:24:17.931267  7620 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 19.11, Last appended by leader: 11, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:18.075510 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.060767  7551 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:18.063968  7514 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.945s	user 0.019s	sys 0.015s
I20260709 12:24:17.931612  7592 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:18.085695 31675 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.788s	user 0.006s	sys 0.041s
W20260709 12:24:18.099318 31675 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.809s	user 0.006s	sys 0.041s
I20260709 12:24:18.028642  7445 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
W20260709 12:24:18.091634  7514 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.973s	user 0.020s	sys 0.016s
I20260709 12:24:18.077309  6752 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:18.078871  7481 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:18.112895 31673 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.013s	sys 0.008s
W20260709 12:24:18.115751 31673 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.013s	sys 0.009s
W20260709 12:24:18.126996 31752 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.615s	user 0.006s	sys 0.017s
W20260709 12:24:18.128952 31752 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.006s	sys 0.018s
W20260709 12:24:18.116822 31676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.677s	user 0.010s	sys 0.021s
W20260709 12:24:18.134622 31676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.011s	sys 0.021s
W20260709 12:24:18.127581 31747 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.708s	user 0.014s	sys 0.013s
W20260709 12:24:18.136409 31747 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.717s	user 0.015s	sys 0.013s
W20260709 12:24:18.127964 31748 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.789s	user 0.009s	sys 0.018s
W20260709 12:24:18.137043 31748 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.798s	user 0.009s	sys 0.018s
I20260709 12:24:18.106874  7465 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:18.116585  7242 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.683s	user 0.011s	sys 0.019s
W20260709 12:24:18.122936  7463 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.812s	user 0.004s	sys 0.023s
W20260709 12:24:18.147980 31677 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.005s	sys 0.021s
W20260709 12:24:18.148445 31677 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.005s	sys 0.021s
I20260709 12:24:18.112924  7501 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:18.167629 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:18.147802  7463 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.005s	sys 0.023s
I20260709 12:24:18.145423  7590 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.151747  7354 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:18.196875  7592 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:18.220409 31751 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.684s	user 0.019s	sys 0.019s
W20260709 12:24:18.223699 31751 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.688s	user 0.019s	sys 0.020s
I20260709 12:24:18.210860  7551 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:18.149880  7242 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.716s	user 0.012s	sys 0.019s
W20260709 12:24:18.241999 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.016s	sys 0.014s
W20260709 12:24:18.242558 31672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.015s	sys 0.025s
W20260709 12:24:18.243036 31672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.015s	sys 0.025s
I20260709 12:24:18.212394  7354 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:18.190989  7465 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:18.254693 31750 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.737s	user 0.009s	sys 0.026s
W20260709 12:24:18.255739 31750 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.737s	user 0.009s	sys 0.026s
W20260709 12:24:18.254911 31749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.013s	sys 0.007s
W20260709 12:24:18.258512 31749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.609s	user 0.013s	sys 0.007s
I20260709 12:24:18.259577 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2786) took 1209 ms. Trace:
W20260709 12:24:18.242597 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.611s	user 0.016s	sys 0.014s
W20260709 12:24:18.255820 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.015s	sys 0.006s
W20260709 12:24:18.270704 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.618s	user 0.015s	sys 0.006s
I20260709 12:24:18.245136  7432 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:18.271556 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 635468us
I20260709 12:24:18.214346  7590 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:18.151672  7430 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:18.278678 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:18.285207 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.681s	user 0.003s	sys 0.016s
W20260709 12:24:18.285737 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.681s	user 0.003s	sys 0.016s
W20260709 12:24:18.284358 31674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.006s	sys 0.031s
W20260709 12:24:18.290139 31674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.007s	sys 0.031s
W20260709 12:24:18.320392 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.287132 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:18.268656  7445 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.268342  7365 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:24:18.332067 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53518.5us
I20260709 12:24:18.260028 31749 rpcz_store.cc:276] 0709 12:24:17.050315 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.126527 (+ 76212us) service_pool.cc:224] Handling call
0709 12:24:17.129256 (+  2729us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.129637 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:17.600307 (+470670us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.259116 (+658809us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:18.259173 (+    57us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:18.259217 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.259358 (+   141us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.259518 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":620003,"prepare.run_cpu_time_us":89,"prepare.run_wall_time_us":13636,"thread_start_us":1080157,"threads_started":2}
W20260709 12:24:18.335193 31790 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.010s	sys 0.009s
W20260709 12:24:18.336342 31790 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.010s	sys 0.010s
I20260709 12:24:18.249670  7551 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:18.345566 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:18.323973  7612 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.558s	user 0.012s	sys 0.019s
W20260709 12:24:18.340727 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53723.8us
I20260709 12:24:18.246248  7481 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:18.330955  7701 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
W20260709 12:24:18.340477  7531 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.786s	user 0.006s	sys 0.027s
I20260709 12:24:18.344410  7630 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:18.343833  7501 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:18.351890 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:18.350486  7570 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.719s	user 0.010s	sys 0.002s
I20260709 12:24:18.282907  7432 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:18.357658  6752 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:18.359519 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:18.355562  7612 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.586s	user 0.012s	sys 0.019s
W20260709 12:24:18.363583  7531 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.810s	user 0.006s	sys 0.028s
I20260709 12:24:18.343683  7460 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.020s
W20260709 12:24:18.376518  7711 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
W20260709 12:24:18.380573 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:18.379029  7709 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.241s	user 0.000s	sys 0.001s
I20260709 12:24:18.396723 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:18.342404  7445 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:18.259189  7522 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.811s	user 0.026s	sys 0.018s
I20260709 12:24:18.344105  7463 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.027s
W20260709 12:24:18.366150  7573 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.011s	sys 0.023s
W20260709 12:24:18.395475 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.408756  7735 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:18.409890  7590 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:18.381405 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:18.411023  7735 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:18.370469  7501 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:18.382316  7514 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:24:18.393050 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 831256us
W20260709 12:24:18.417186 31746 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.711s	user 0.000s	sys 0.019s
W20260709 12:24:18.417640 31746 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.744s	user 0.000s	sys 0.019s
W20260709 12:24:18.404716  7522 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.962s	user 0.026s	sys 0.019s
W20260709 12:24:18.428170 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46905.4us
I20260709 12:24:18.356432  7354 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:18.393239  7389 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.085s
I20260709 12:24:18.411604  7735 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:18.394579 31569 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 238) took 89515 ms (client timeout 15000 ms). Trace:
W20260709 12:24:18.417071  7654 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.639s	user 0.017s	sys 0.009s
W20260709 12:24:18.424736  7684 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.972s	user 0.005s	sys 0.023s
I20260709 12:24:18.431515 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2787) took 1356 ms. Trace:
I20260709 12:24:18.432545 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:18.375821  7570 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:18.407344  7573 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.815s	user 0.012s	sys 0.024s
W20260709 12:24:18.418129 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.587s	user 0.004s	sys 0.015s
W20260709 12:24:18.505311 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.675s	user 0.004s	sys 0.016s
I20260709 12:24:18.407368  7613 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.412039  7501 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:18.461860  7392 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:18.499123  7354 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:18.499234  7721 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:24:18.498533 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:18.417023  7514 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:18.472086  7654 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.694s	user 0.017s	sys 0.009s
W20260709 12:24:18.515959 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.475085 31746 rpcz_store.cc:276] 0709 12:24:17.074863 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.129235 (+ 54372us) service_pool.cc:224] Handling call
0709 12:24:17.129573 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.129777 (+   204us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:17.673445 (+543668us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.430952 (+757507us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.431121 (+   169us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:18.431170 (+    49us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:18.431229 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.431303 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.431452 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":728818,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":98,"thread_start_us":1265280,"threads_started":2}
W20260709 12:24:18.472494  7684 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.020s	user 0.005s	sys 0.024s
W20260709 12:24:18.519971 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 657080us
I20260709 12:24:18.507659 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3478) took 1187 ms. Trace:
I20260709 12:24:18.510381  7392 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:18.526170 31748 rpcz_store.cc:276] 0709 12:24:17.320313 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.320460 (+   147us) service_pool.cc:224] Handling call
0709 12:24:17.320878 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.321153 (+   275us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:18.145344 (+824191us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.507154 (+361810us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:18.507220 (+    66us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:18.507295 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.507400 (+   105us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.507579 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":576751,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":43,"thread_start_us":1158299,"threads_started":2}
I20260709 12:24:18.528049 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2624) took 1329 ms. Trace:
I20260709 12:24:18.528609 31676 rpcz_store.cc:276] 0709 12:24:17.198325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.345212 (+146887us) service_pool.cc:224] Handling call
0709 12:24:17.401029 (+ 55817us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:17.438914 (+ 37885us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:17.438951 (+    37us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:17.439750 (+   799us) log.cc:844] Serialized 78 byte log entry
0709 12:24:18.140679 (+700929us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:18.140755 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.140860 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:18.527755 (+386895us) raft_consensus.cc:1685] finished
0709 12:24:18.527785 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.527969 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31257,"raft.queue_time_us":31591,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":159,"thread_start_us":694801,"threads_started":1,"wal-append.queue_time_us":928293}
W20260709 12:24:18.530756 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:18.543340  7719 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.168s	user 0.001s	sys 0.000s
W20260709 12:24:18.514308  7709 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:24:18.545289 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2620) took 1369 ms. Trace:
W20260709 12:24:18.552099 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 916166us
I20260709 12:24:18.551278 31675 rpcz_store.cc:276] 0709 12:24:17.175435 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.175587 (+   152us) service_pool.cc:224] Handling call
0709 12:24:17.187193 (+ 11606us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:17.217828 (+ 30635us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:17.217877 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:17.260355 (+ 42478us) log.cc:844] Serialized 26 byte log entry
0709 12:24:18.099929 (+839574us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:18.099986 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.100137 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:18.544955 (+444818us) raft_consensus.cc:1685] finished
0709 12:24:18.544988 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.545207 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3339,"raft.queue_time_us":14881,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":7185,"thread_start_us":809109,"threads_started":1,"wal-append.queue_time_us":1011069}
I20260709 12:24:18.556764 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:18.557479 31676 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:18.510583  7514 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:18.510861  7531 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.517311  7378 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.560576 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2789) took 1415 ms. Trace:
W20260709 12:24:18.586108 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.518488  7522 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.520882  7465 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:18.591969 31752 rpcz_store.cc:276] 0709 12:24:17.145117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.314938 (+169821us) service_pool.cc:224] Handling call
0709 12:24:17.404039 (+ 89101us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.423169 (+ 19130us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:18.135879 (+712710us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.560234 (+424355us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:18.560286 (+    52us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:18.560329 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.560399 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.560519 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":475209,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"thread_start_us":1036679,"threads_started":2}
W20260709 12:24:18.562218  7561 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.766s	user 0.004s	sys 0.030s
I20260709 12:24:18.614571 31675 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: []
W20260709 12:24:18.626134 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.612978 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3483) took 1097 ms. Trace:
I20260709 12:24:18.638546 31750 rpcz_store.cc:276] 0709 12:24:17.515821 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.517202 (+  1381us) service_pool.cc:224] Handling call
0709 12:24:17.517540 (+   338us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.517758 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:18.256385 (+738627us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.612557 (+356172us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:18.612618 (+    61us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:18.612674 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.612759 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.612911 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":380876,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":28,"raft.queue_time_us":122611,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":6663,"thread_start_us":1075413,"threads_started":2}
I20260709 12:24:18.514497  7613 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:18.595779  7573 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:18.596097  7430 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:18.599598  7654 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:24:18.610474  7318 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.031s	sys 0.051s
I20260709 12:24:18.657289 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1439) took 1795 ms. Trace:
I20260709 12:24:18.524346  7736 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.089s
W20260709 12:24:18.639276  7561 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.843s	user 0.004s	sys 0.031s
I20260709 12:24:18.657946 31826 rpcz_store.cc:276] 0709 12:24:16.861955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:16.881294 (+ 19339us) service_pool.cc:224] Handling call
0709 12:24:16.895874 (+ 14580us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:16.897047 (+  1173us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:16.897079 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:16.918114 (+ 21035us) log.cc:844] Serialized 26 byte log entry
0709 12:24:17.510134 (+592020us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:17.568683 (+ 58549us) log.cc:844] Serialized 14 byte log entry
0709 12:24:17.580263 (+ 11580us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.580383 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:18.657028 (+1076645us) raft_consensus.cc:1685] finished
0709 12:24:18.657054 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.657216 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":27976,"prepare.queue_time_us":64756,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"raft.queue_time_us":25773,"raft.run_cpu_time_us":195,"raft.run_wall_time_us":6710,"thread_start_us":474753,"threads_started":1,"wal-append.queue_time_us":943736}
I20260709 12:24:18.613745  7463 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.643584  7745 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.599108  7531 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:18.676486  7318 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.031s	sys 0.051s
W20260709 12:24:18.700289 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 143802us
W20260709 12:24:18.695111 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 335887us
I20260709 12:24:18.601651  7378 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:18.613548  7522 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:18.700446  7432 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:18.754150 31676 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:24:18.667480  7573 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:18.835378 31819 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:18.835992  7573 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:18.678298  7445 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:18.735713  7691 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.918s	user 0.009s	sys 0.005s
I20260709 12:24:18.838955 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:18.839439  7691 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:18.840613 31819 raft_consensus.cc:2410] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 25 because replica has last-logged OpId of term: 23 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:24:18.842362  7691 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:18.855618 31823 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:18.858320 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:18.854394  7750 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:18.862039  7750 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:18.862764  7749 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.862454 31818 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:18.863426  7749 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:18.864408 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:18.760059  7745 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:18.866283 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:18.677560  7392 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:18.741123  7463 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:18.867365  7745 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:18.867662 31818 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:18.868471  7463 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:18.857682  7295 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:18.869441 31818 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:18.869418  7295 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:18.654510  7735 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 18.14, Last appended by leader: 15, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:18.873919 31821 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:18.876915 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3481) took 1361 ms. Trace:
W20260709 12:24:18.878681 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.880955 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3482) took 1365 ms. Trace:
I20260709 12:24:18.764601  7430 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:18.869978  6752 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:18.877542 31753 rpcz_store.cc:276] 0709 12:24:17.515290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.516557 (+  1267us) service_pool.cc:224] Handling call
0709 12:24:18.876665 (+1360108us) spinlock_profiling.cc:243] Waited 227 ms on lock 0x7b5400213ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:18.876826 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":522224384}
I20260709 12:24:18.904062 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3476) took 1585 ms. Trace:
W20260709 12:24:18.876614  7718 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.282s	user 0.000s	sys 0.002s
W20260709 12:24:18.908672 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.864540  7750 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:18.867408  7392 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:18.877724  7445 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.876753  7714 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:18.903313  7722 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.422s	user 0.000s	sys 0.001s
I20260709 12:24:18.883527 31751 rpcz_store.cc:276] 0709 12:24:17.515572 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.516011 (+   439us) service_pool.cc:224] Handling call
0709 12:24:17.535436 (+ 19425us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.535656 (+   220us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:18.240344 (+704688us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.880602 (+640258us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:18.880653 (+    51us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:18.880706 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.880775 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.880895 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":897,"prepare.queue_time_us":363194,"prepare.run_cpu_time_us":25,"prepare.run_wall_time_us":26,"raft.queue_time_us":240056,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":45319,"thread_start_us":1324349,"threads_started":3,"wal-append.queue_time_us":488282}
I20260709 12:24:18.918620 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:18.942806 31677 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:18.907238 31747 rpcz_store.cc:276] 0709 12:24:17.319330 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.350339 (+ 31009us) service_pool.cc:224] Handling call
0709 12:24:17.350916 (+   577us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.351456 (+   540us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:18.144679 (+793223us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.903617 (+758938us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:18.903682 (+    65us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:18.903733 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.903835 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:18.903988 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":546726,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":268899,"spinlock_wait_cycles":7753600,"thread_start_us":1416590,"threads_started":3,"wal-append.queue_time_us":708534}
W20260709 12:24:18.924288 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.922437 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:18.946074 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:18.929136 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:18.948213 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.940608 31672 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: [20.14-21.15]
I20260709 12:24:18.911582  7511 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:18.922113  7654 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.892011  6752 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:18.901796  7430 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:18.952373 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:18.953123 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:18.946034  7734 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.388s	user 0.001s	sys 0.001s
I20260709 12:24:18.958962 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:18.963040 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.966462 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:18.964174 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:18.967559 31753 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:18.967800 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:18.948906  7584 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.162s	user 0.016s	sys 0.000s
I20260709 12:24:18.974996 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:18.941001  7570 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:18.981693 31823 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:18.982550 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:18.965325 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:18.984181 31672 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:18.968469 31669 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:18.975212 32375 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:24:18.975597 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:18.976788 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:18.985486 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:18.987571 31822 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:24:18.981689 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:18.988891 31751 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:18.931025  7620 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 26.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:24:18.991339 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:18.997059 32369 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:18.991356 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:18.950695  7463 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.951283  7759 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.993495 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:18.999054 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:18.958953  7612 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.002399 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:18.951828  7758 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:19.015578 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:18.995553 32374 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:19.014241 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2618) took 1857 ms. Trace:
I20260709 12:24:19.008864 31822 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:18.941623  7445 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:18.964062  7736 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:19.012301 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 579851us
I20260709 12:24:19.012431 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:19.013942 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54203.8us
I20260709 12:24:19.016840 31671 rpcz_store.cc:276] 0709 12:24:17.157122 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.157269 (+   147us) service_pool.cc:224] Handling call
0709 12:24:17.180010 (+ 22741us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:17.202340 (+ 22330us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:17.202390 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:17.254748 (+ 52358us) log.cc:844] Serialized 26 byte log entry
0709 12:24:17.896253 (+641505us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:17.896320 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:17.896420 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:19.013996 (+1117576us) raft_consensus.cc:1685] finished
0709 12:24:19.014020 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:19.014173 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5390,"raft.queue_time_us":8057,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":22925,"thread_start_us":604546,"threads_started":1,"wal-append.queue_time_us":948683}
W20260709 12:24:19.032583 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48609.4us
I20260709 12:24:19.016251 31747 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:24:18.991314  7739 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:19.038170 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:19.084113  7739 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.084872 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:19.038367 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:19.086912 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48696.9us
I20260709 12:24:19.087442  7531 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:19.039023  7769 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:19.088106  7769 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.088539  7769 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:19.089164  7430 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:19.089577  7747 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.386s	user 0.001s	sys 0.001s
I20260709 12:24:19.039134 31672 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:18.959477  7511 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.093206  7511 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:19.013005  7463 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.093873  7511 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:19.040930  7432 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:19.041441  7242 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:19.016088  7758 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.095077  7242 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:19.095285  7463 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:19.095659  7242 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:19.095957 31672 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:19.042279  7452 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 20.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:19.042614 31818 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:24:19.042944 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:19.046859  7365 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:19.097820 31671 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:19.048302 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:19.050124  7732 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.437s	user 0.001s	sys 0.001s
I20260709 12:24:19.049978 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:19.101503 31752 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:19.048914 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:19.048252  7766 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:19.052420  7445 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:19.053529  7753 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
W20260709 12:24:19.054441  7744 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.173s	user 0.000s	sys 0.000s
I20260709 12:24:19.058908 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:19.059319  7741 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 12:24:19.060686  7736 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:19.064285 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.170238 31748 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260709 12:24:19.065448  7748 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:24:18.957898  7714 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.067915 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:19.090621  7749 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:19.071230  7738 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.311s	user 0.001s	sys 0.000s
W20260709 12:24:19.072260  7398 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.002s	user 0.008s	sys 0.005s
I20260709 12:24:19.072842  7767 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:18.966460  7654 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:18.463198 31569 rpcz_store.cc:269] 0709 12:22:48.681475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:48.681817 (+   342us) service_pool.cc:224] Handling call
0709 12:24:18.185946 (+89504129us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:09.494904 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:09.495690 (+   786us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:09.495751 (+    61us) write_op.cc:435] Acquired schema lock
0709 12:24:09.495773 (+    22us) tablet.cc:662] Decoding operations
0709 12:24:09.496278 (+   505us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:09.496385 (+   107us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:09.496433 (+    48us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:24:09.496912 (+   479us) tablet.cc:701] Row locks acquired
0709 12:24:09.496960 (+    48us) write_op.cc:260] PREPARE: finished
0709 12:24:09.497152 (+   192us) write_op.cc:270] Start()
0709 12:24:09.497316 (+   164us) write_op.cc:276] Timestamp: P: 1783599849497081 usec, L: 0
0709 12:24:09.497335 (+    19us) op_driver.cc:348] REPLICATION: starting
0709 12:24:09.599585 (+102250us) log.cc:844] Serialized 930 byte log entry
0709 12:24:11.142912 (+1543327us) spinlock_profiling.cc:243] Waited 8.38 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8f9ccb0ac 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b
0709 12:24:11.887245 (+744333us) op_driver.cc:464] REPLICATION: finished
0709 12:24:11.979830 (+ 92585us) write_op.cc:301] APPLY: starting
0709 12:24:11.979969 (+   139us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:12.133800 (+153831us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:12.136478 (+  2678us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:12.394453 (+257975us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:12.394846 (+   393us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:24:12.394910 (+    64us) write_op.cc:312] APPLY: finished
0709 12:24:12.395198 (+   288us) log.cc:844] Serialized 33 byte log entry
0709 12:24:12.395437 (+   239us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:12.395562 (+   125us) write_op.cc:454] Released schema lock
0709 12:24:12.395623 (+    61us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:12.486750 (+ 91127us) spinlock_profiling.cc:243] Waited 30.4 ms on lock 0x7b54002828d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd0e28 00007fe90080b581 00007fe90080ad58 00007fe90080f9f1 00007fe90080f9a9 00007fe90080f939 00007fe90080f901 00007fe90080eb2d 0000000000388634 0000000000388479
0709 12:24:12.487292 (+   542us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:12.487340 (+    48us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":1139490,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":80470,"apply.run_cpu_time_us":13989,"apply.run_wall_time_us":416288,"cfile_cache_hit":10,"cfile_cache_hit_bytes":12483,"num_ops":2,"prepare.queue_time_us":962714,"prepare.run_cpu_time_us":159101,"prepare.run_wall_time_us":3000616,"raft.queue_time_us":204840,"raft.run_cpu_time_us":16309,"raft.run_wall_time_us":167837,"replication_time_us":2340667,"spinlock_wait_cycles":89192704,"thread_start_us":2275433,"threads_started":4,"wal-append.queue_time_us":1717632,"wal-append.run_cpu_time_us":28497,"wal-append.run_wall_time_us":1957985}]]}
I20260709 12:24:19.078377  7522 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:19.080042 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:19.007874  7772 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.085s
I20260709 12:24:19.095014  7758 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:19.050416 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 172904us
I20260709 12:24:19.110368  7613 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:19.111321 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:19.120050 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:19.118196 31677 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:19.120240 31753 raft_consensus.cc:2410] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 27 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 12:24:19.123246  7365 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.035831  7759 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:19.120782  7740 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.427s	user 0.000s	sys 0.001s
W20260709 12:24:19.125187  7751 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
I20260709 12:24:19.060132 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:19.060753  7757 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:24:19.166643 31751 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:18.992587  7584 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.068231  7295 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:19.110489  7242 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.029371  7612 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.182057 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:19.185472 31823 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
W20260709 12:24:19.218528 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.221390 32369 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:19.198659 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1299) took 1585 ms. Trace:
I20260709 12:24:19.218886 32373 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:19.212098 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1300) took 1595 ms. Trace:
I20260709 12:24:19.222486 31674 rpcz_store.cc:276] 0709 12:24:17.613131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.776930 (+163799us) service_pool.cc:224] Handling call
0709 12:24:17.777374 (+   444us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:17.777836 (+   462us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:17.777877 (+    41us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:17.778502 (+   625us) log.cc:844] Serialized 26 byte log entry
0709 12:24:18.291040 (+512538us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:18.291109 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.291221 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:19.198330 (+907109us) raft_consensus.cc:1685] finished
0709 12:24:19.198362 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:19.198551 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":10815,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":128,"thread_start_us":509986,"threads_started":1,"wal-append.queue_time_us":982725}
I20260709 12:24:19.233116 31752 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:19.228582 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:19.242321 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:19.169623  7684 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:19.245821 31748 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:19.239475 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:19.248030 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:19.220641  7398 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.251303 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:19.251931 31745 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:19.252503  7365 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:19.252888 31749 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:19.245771 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:19.249296  7745 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:19.254391  7763 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:19.223853 31670 rpcz_store.cc:276] 0709 12:24:17.617199 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.846419 (+229220us) service_pool.cc:224] Handling call
0709 12:24:17.879174 (+ 32755us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:17.903435 (+ 24261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:17.903482 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:17.930938 (+ 27456us) log.cc:844] Serialized 26 byte log entry
0709 12:24:18.367514 (+436576us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:18.367576 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:18.367666 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:19.211770 (+844104us) raft_consensus.cc:1685] finished
0709 12:24:19.211803 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:19.212006 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7939,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":24513,"thread_start_us":436137,"threads_started":1,"wal-append.queue_time_us":759813}
I20260709 12:24:19.253206  7398 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:19.254886  7763 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:19.255194 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:19.228713  7445 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:19.251032 31752 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:19.251613  7570 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:24:19.287755 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 200117us
W20260709 12:24:19.289499 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.169416  7766 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.258520  7590 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:19.234223  7761 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:19.292387 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:19.266650  7543 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.338s	user 0.017s	sys 0.000s
I20260709 12:24:19.307531 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:19.307647 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 274767us
I20260709 12:24:19.316927 31752 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
W20260709 12:24:19.269209  7630 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.875s	user 0.010s	sys 0.011s
W20260709 12:24:19.285902  7501 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.768s	user 0.013s	sys 0.000s
I20260709 12:24:19.309458 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:19.187737  7760 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 18.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.083s
I20260709 12:24:19.266189  7755 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:19.339383 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:19.314476 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:19.263095  7763 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:19.268227  7445 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:19.284893  7242 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:19.317091 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:19.287742  7570 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.329192 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:19.229519  7767 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:24:19.323036  7514 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.683s	user 0.008s	sys 0.004s
W20260709 12:24:19.333045  7592 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.091s	user 0.012s	sys 0.000s
I20260709 12:24:19.334048  7736 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:19.312211  7522 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:19.320988  7630 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.932s	user 0.011s	sys 0.011s
I20260709 12:24:19.284489  7684 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.351281 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:19.335976  7755 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.327848  7745 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.355888 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:19.348629 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:19.389147 31748 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
W20260709 12:24:19.360090  7770 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.234s	user 0.000s	sys 0.001s
W20260709 12:24:19.361709 31678 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3583) took 2081 ms (client timeout 2202 ms). Trace:
W20260709 12:24:19.398720 31678 rpcz_store.cc:269] 0709 12:24:17.280808 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.312161 (+ 31353us) service_pool.cc:224] Handling call
0709 12:24:19.361487 (+2049326us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540017d9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:19.361648 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2439609216}
I20260709 12:24:19.364778  7378 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:19.366703  7551 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:19.358152 31823 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:19.342919  7772 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:19.378015  7543 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:19.391773  7522 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.385448  7630 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:19.336968  7501 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.394551  7684 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:19.395780 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 136550us
I20260709 12:24:19.333045  7749 raft_consensus.cc:2764] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:24:19.400987 31745 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:19.348268  7445 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:19.379560  7570 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:19.397325  7755 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:19.424651 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.432135 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:19.433013 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 79202.9us
W20260709 12:24:19.416574 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 102173us
W20260709 12:24:19.379048 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 338507us
I20260709 12:24:19.448524 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:19.449563 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:19.432052 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 177386us
I20260709 12:24:19.454404 31823 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:19.453927 31824 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:19.455988 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:19.322063  7761 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.404665  7739 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:19.466614  7398 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:19.467120  7398 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:19.467515  7398 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:19.468159  7522 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:19.469906  7630 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 20.12, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:19.471566  7778 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.119s	user 0.000s	sys 0.000s
I20260709 12:24:19.403282  7365 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.429559  7543 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:19.400640  7354 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.075s
I20260709 12:24:19.387521  7514 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.504212 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:19.480924  7772 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:19.515453 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:19.516233 31823 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:19.466710  7780 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:19.404451  7745 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.487072  7543 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:19.489831  7392 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.561s	user 0.010s	sys 0.002s
I20260709 12:24:19.524861 31675 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:19.526331 31824 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:24:19.493929  7584 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:19.481317  7365 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.554706 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:19.566651 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:19.567454 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:19.560969 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 106189us
I20260709 12:24:19.521795  7772 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:19.585032 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60702us
W20260709 12:24:19.590548 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 188426us
W20260709 12:24:19.552721  7754 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.666s	user 0.004s	sys 0.001s
W20260709 12:24:19.599268 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 140178us
I20260709 12:24:19.540311  7612 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:19.601943 32342 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:19.558591  7654 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:19.584889  7392 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:24:19.599766  7778 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.057s	user 0.000s	sys 0.000s
I20260709 12:24:19.559130  7759 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:19.588838  7663 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:19.557029  7714 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:19.589215  7772 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:19.565001  7739 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:19.734346  7739 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:19.735044  7739 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:19.739362  7663 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:19.740589  7750 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:19.740532  7663 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 17.12, Last appended by leader: 12, Current term: 19, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:19.644124 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 140003us
I20260709 12:24:19.607304  7378 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.742604  7378 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.745216  7378 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:19.651355  7735 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.747149  7735 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.749670  7735 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:19.746480 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:19.751431  7785 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.235s	user 0.000s	sys 0.001s
W20260709 12:24:19.752265  7803 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:24:19.753839  7781 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.271s
I20260709 12:24:19.754213  7761 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:19.653115  7769 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.564s	user 0.014s	sys 0.005s
I20260709 12:24:19.751471  7452 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.023s
I20260709 12:24:19.774618  7769 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.777129  7739 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.777762  7769 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:19.778329  7783 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 12:24:19.778995  7786 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.294s
I20260709 12:24:19.780284  7739 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:19.781994  7511 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:19.782423 32374 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 28: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:24:19.761595 31671 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.642s	user 0.009s	sys 0.003s
W20260709 12:24:19.756224 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 128278us
W20260709 12:24:19.764981  7762 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.822s	user 0.004s	sys 0.000s
I20260709 12:24:19.773870 31824 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:19.773751 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:19.793113 32377 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:19.785934 31671 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:24:19.650800  7392 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:19.804774  7780 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:19.782529  7511 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:19.639967  7733 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.383s
I20260709 12:24:19.754992  7781 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.056s
I20260709 12:24:19.819723 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:19.785382  7769 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:24:19.824164 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 296476us
I20260709 12:24:19.816402  7378 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:19.821616  7511 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:19.811097  7786 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:19.823562  7389 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:24:19.841804 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 272630us
W20260709 12:24:19.846804 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 244964us
W20260709 12:24:19.829007  7758 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.583s	user 0.007s	sys 0.005s
I20260709 12:24:19.851953 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:19.860389  7758 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.826215  7501 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:19.837591  7769 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:19.842765  7739 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.872974 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:19.827512  7378 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.843152  7389 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:19.862823  7813 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:19.863576  7514 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:19.825242  7780 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.840785  7786 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:19.874152 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:19.887982 31674 raft_consensus.cc:2410] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 24 because replica has last-logged OpId of term: 22 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 13.
I20260709 12:24:19.888809 31677 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:19.878695 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:19.834764  7767 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:19.868141  7758 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:19.878762  7739 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.902434  7613 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:19.904410  7613 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:19.904795  7613 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:19.904971  7739 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:19.905745  7613 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.891983  7808 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:19.912833  7808 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:19.913250  7756 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:19.913753  7756 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:19.892305  7810 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:19.914609  7756 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.914914  7755 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.914024  7808 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.915791  7756 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:19.916312  7808 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:19.917273  7755 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:19.917977 31670 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:19.918293  7613 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:19.919502 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 110644us
I20260709 12:24:19.892743  7812 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:19.892849 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:19.893494  7813 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:19.893842  7788 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
W20260709 12:24:19.894363 31792 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.845s (SENT) [suppressed 9 similar messages]
I20260709 12:24:19.853070  7392 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.901476  7378 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:19.901680  7786 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:19.902220  7769 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 20.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.891692  7765 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:19.927292 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 322552us
I20260709 12:24:19.890867 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3401) took 1048 ms. Trace:
W20260709 12:24:19.936072 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:19.946506 31823 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:19.948930 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:19.940151  7763 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.584s	user 0.005s	sys 0.006s
I20260709 12:24:19.951236 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:19.964766 31674 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:19.949802 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:19.954213 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3488) took 1001 ms. Trace:
I20260709 12:24:19.906371  7398 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.966322 31749 rpcz_store.cc:276] 0709 12:24:18.952592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.952701 (+   109us) service_pool.cc:224] Handling call
0709 12:24:18.966002 (+ 13301us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:19.253414 (+287412us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.953947 (+700533us) spinlock_profiling.cc:243] Waited 287 ms on lock 0x7b54002ac6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:19.954149 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":659321728}
I20260709 12:24:19.963449 31819 rpcz_store.cc:276] 0709 12:24:18.842138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.843322 (+  1184us) service_pool.cc:224] Handling call
0709 12:24:18.843951 (+   629us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:18.887808 (+ 43857us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:18.908769 (+ 20961us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:18.929342 (+ 20573us) log.cc:844] Serialized 52 byte log entry
0709 12:24:19.132027 (+202685us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:19.509663 (+377636us) log.cc:844] Serialized 14 byte log entry
0709 12:24:19.889948 (+380285us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.890099 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:19.890246 (+   147us) raft_consensus.cc:1685] finished
0709 12:24:19.890317 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:19.890734 (+   417us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":78150,"prepare.queue_time_us":336828,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":30,"raft.queue_time_us":293841,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":10526,"thread_start_us":866598,"threads_started":3,"wal-append.queue_time_us":347864}
I20260709 12:24:19.981040 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:19.981887 31674 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:24:19.982920 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:19.984365 31674 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:19.943198  7812 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.986783 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:19.988680 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:19.988998  7812 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:19.989672  7812 raft_consensus.cc:2764] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election decision vote started in defunct term 20: won
I20260709 12:24:19.990581 31670 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:19.991361  7389 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.992210  7378 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:19.953384  7755 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:19.997444  7772 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:19.990347  7812 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:19.999809 31674 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:24:20.007300 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:19.999310 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:19.967284  7242 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.609s	user 0.010s	sys 0.001s
I20260709 12:24:20.010735 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:20.008019 31674 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:20.014684 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:24:20.032743 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 139999us
W20260709 12:24:20.033380 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.027520  7445 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.554s	user 0.008s	sys 0.003s
I20260709 12:24:20.043190 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:20.047484 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1137) took 1095 ms. Trace:
I20260709 12:24:20.027321  7389 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:20.049563 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 85311.3us
I20260709 12:24:20.028043  7739 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:19.966753  7765 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:19.979041  7763 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.031705  7767 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:19.998871  7543 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.059500 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:20.031695  7780 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:20.067569 32370 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:24:20.048004 32371 rpcz_store.cc:276] 0709 12:24:18.951893 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.959864 (+  7971us) service_pool.cc:224] Handling call
0709 12:24:18.981647 (+ 21783us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:19.031048 (+ 49401us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:19.031101 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:19.115353 (+ 84252us) log.cc:844] Serialized 26 byte log entry
0709 12:24:19.492546 (+377193us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:19.492611 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.492705 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:20.047235 (+554530us) raft_consensus.cc:1685] finished
0709 12:24:20.047260 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.047401 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7298,"raft.queue_time_us":23816,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":94,"thread_start_us":317420,"threads_started":1,"wal-append.queue_time_us":498787}
I20260709 12:24:20.038983  7817 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.039248  7612 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.133822 32372 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:20.135082  7817 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:20.133363 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.133493 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 239148us
I20260709 12:24:20.137439  7817 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:20.065551 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.117205 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:20.048311  7392 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:20.138854 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 79486.5us
I20260709 12:24:20.141220  7354 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 21.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:20.068425  7813 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.142158  7354 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:20.142604  7354 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.142967  7354 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:20.159395 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:20.163306 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.164634 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.166112 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:20.167222 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:20.168331 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.169411 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:20.170799 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 115202us
I20260709 12:24:20.171032 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:20.173602 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:20.173681 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:20.048038  7445 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.176229 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.131945  7782 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:20.134501  7767 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:20.157413 31643 connection.cc:256] RPC call timeout handler was delayed by -1.3060264120003922s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21509999999999999s for a response
I20260709 12:24:20.183333 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.174871 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:20.159238  7760 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.798s	user 0.012s	sys 0.017s
W20260709 12:24:20.185926 32342 leader_election.cc:336] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.847s (SENT)
I20260709 12:24:20.190104 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:20.190397 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:20.186935 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:20.188964 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.186697  7805 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:20.191140 31676 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:20.155354  7755 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:20.192869 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.194715 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.197162 31678 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:20.133816  7242 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.048s
I20260709 12:24:20.178844  7792 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:20.201907 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:20.170591  7801 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.023s
I20260709 12:24:20.206902 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:20.179874  7398 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:20.195618  7760 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.012s	sys 0.018s
W20260709 12:24:20.201390 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.199934 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:20.204273 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.156543  7612 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.211113 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:20.188872  7460 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:20.216118  7800 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:20.197662  7761 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:20.215570  7767 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.220520 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:20.220741 31819 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:20.193853  7805 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:20.232901 32373 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:20.223671 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.239569 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:20.244817 31824 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 21: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:20.250082 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.251233 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.213716  7514 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:20.254490 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:20.232646  7493 raft_consensus.cc:2764] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:24:20.208185  7430 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.043s
I20260709 12:24:20.201452  7755 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.193009  7782 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.264123  7818 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.093s
W20260709 12:24:20.277150 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 91244.3us
I20260709 12:24:20.277298 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:20.277884  7430 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:20.278292  7430 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:20.278360 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:20.278776 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:20.279490 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:20.279213  7430 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.279635 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:20.280426  7430 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:20.280947 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:20.281605 32375 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:20.282193  7832 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:20.282734  7832 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.282883 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:20.283089  7832 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:20.283567  7739 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:20.284121 31822 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:20.284654  7378 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:20.284801  7739 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:20.285208  7739 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:20.285532  7831 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:24:20.264353 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.287384 31826 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:20.266569 32369 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 15. (term mismatch)
I20260709 12:24:20.288264 32369 pending_rounds.cc:85] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 14
I20260709 12:24:20.288524 32369 pending_rounds.cc:107] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 21.15
I20260709 12:24:20.288757 32369 raft_consensus.cc:2889] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:20.266767 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:20.267590 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:20.236495  7761 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.255635  7800 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:20.239914  7758 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:20.303362 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:20.304333 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.240252  7769 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:20.275002  7798 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:20.276119  7612 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:20.276952  7782 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:20.286816  7392 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:20.283679 32378 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:20.282449 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:20.316012 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.293514 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:20.259585  7570 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.785s	user 0.013s	sys 0.000s
I20260709 12:24:20.247273  7805 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:20.286110  7739 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.277462  7818 consensus_queue.cc:1243] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 27.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:24:20.301268 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:20.302220 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:20.256932  7514 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.215121  7792 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.308866 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.267462  7759 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:20.267995  7772 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:20.302709  7761 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:20.322543 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 118876us
I20260709 12:24:20.328189 31825 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:20.310758  7798 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.351833  7798 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:20.354032  7452 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:20.354645  7452 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:20.355578  7452 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.356647  7452 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:20.358078 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:20.361123  7834 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:24:20.315779  7392 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:20.362514  7392 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.363636  7392 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:20.345588 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2628) took 1428 ms. Trace:
I20260709 12:24:20.364969 31675 rpcz_store.cc:276] 0709 12:24:18.917253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.927760 (+ 10507us) service_pool.cc:224] Handling call
0709 12:24:18.942918 (+ 15158us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:19.539641 (+596723us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.345329 (+805688us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b540022ae54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:20.345521 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1337720064}
I20260709 12:24:20.366356 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:20.367142 31675 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:20.368063 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:20.368237  7823 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:24:20.371893 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:20.339087  7739 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:20.344589 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 1135) took 1395 ms. Trace:
I20260709 12:24:20.345507 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.330415  7759 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:20.346457  7832 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.347055  7800 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.348349  7590 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:20.386186 31672 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:20.388218 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.391402 31822 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:20.322088  7531 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:20.390281 31826 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:20.389503 32377 rpcz_store.cc:276] 0709 12:24:18.949099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.955571 (+  6472us) service_pool.cc:224] Handling call
0709 12:24:20.344325 (+1388754us) spinlock_profiling.cc:243] Waited 557 ms on lock 0x7b54004c4354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:20.344469 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1281467264}
I20260709 12:24:20.327823  7514 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:20.437238 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.467121 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.436064 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64269.2us
I20260709 12:24:20.454684 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.470283 31825 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:20.389973  7543 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:20.324077  7570 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.489964 32370 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:20.392261  7763 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:20.452243  7531 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.453553  7590 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:20.489078 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:20.394665  7759 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:20.433985  7772 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:20.497092 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:20.510215 31823 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:20.481369  7817 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:20.515717 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.520674 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.520506 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:20.521251 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.436033  7354 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.492692  7430 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:20.499578  7531 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:20.530193 31792 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:20.528765 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:20.540808 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:20.541476 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 73063.1us
W20260709 12:24:20.524266  7786 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.560s	user 0.011s	sys 0.000s
W20260709 12:24:20.538805  7822 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.346s	user 0.000s	sys 0.008s
I20260709 12:24:20.512423  7765 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:20.544622 31672 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:20.524062  7613 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:20.552848 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:20.554236 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.554919 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:20.559005 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3405) took 1675 ms. Trace:
I20260709 12:24:20.530303  7817 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.547626  7818 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:20.561230  7818 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:20.562007  7817 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:20.562178  7818 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.563203 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1454) took 1692 ms. Trace:
I20260709 12:24:20.563582 31818 rpcz_store.cc:276] 0709 12:24:18.870933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.873547 (+  2614us) service_pool.cc:224] Handling call
0709 12:24:18.873911 (+   364us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:19.097081 (+223170us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.562959 (+1465878us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b54000fa854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:20.563150 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":387287936}
I20260709 12:24:20.491726  7761 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.559867 31820 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:20.565876  7354 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:20.566180  7761 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:20.566762  7761 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:20.551193  7801 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.017s
W20260709 12:24:20.568761 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.564529 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.540182  7799 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.020s
I20260709 12:24:20.564565  7432 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:20.569687  7432 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:20.570608  7432 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.572304  7832 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:20.559607 31827 rpcz_store.cc:276] 0709 12:24:18.902840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.915511 (+ 12671us) service_pool.cc:224] Handling call
0709 12:24:18.931862 (+ 16351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:18.995595 (+ 63733us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:18.995631 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:18.995678 (+    47us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:18.995724 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.558642 (+1562918us) spinlock_profiling.cc:243] Waited 63.5 ms on lock 0x7b5400379454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:20.558720 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.558924 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":146026752}
I20260709 12:24:20.567260  7761 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.577788  7761 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:20.579921 31717 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:20.581146  7772 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:20.581892 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:20.586328 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.591423 31675 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:20.597584  7763 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:20.598673  7801 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:20.599056  7801 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:20.600013  7801 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.601800  7801 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:20.602950 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.615638  7792 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:20.620019  7570 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:20.541386  7430 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.623728  7430 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:20.626816 31675 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:20.630165 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:20.631596  7430 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:20.632202  7430 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:20.633473  7430 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.635046  7755 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:20.639581 31819 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
W20260709 12:24:20.640689 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 81440.4us
I20260709 12:24:20.543363  7551 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:20.650104 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:20.656634 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:20.659773 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:20.602087  7756 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.686s	user 0.005s	sys 0.004s
I20260709 12:24:20.559248  7378 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:24:20.667650 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 85886.5us
I20260709 12:24:20.668633 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:20.552186  7786 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.643698  7765 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:24:20.675980  7779 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.059s	user 0.068s	sys 0.090s
I20260709 12:24:20.677268 31672 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:20.680223  7831 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:20.681136  7378 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:20.681238  7832 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:20.705837 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:20.718067 31669 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:20.666419  7767 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:20.682770 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:20.663247  7613 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:24:20.678758  7389 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.615s	user 0.009s	sys 0.003s
W20260709 12:24:20.683547  7511 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.825s	user 0.013s	sys 0.000s
I20260709 12:24:20.739199  7511 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.740029 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:20.740734 31678 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:20.741829  7745 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:20.742846  7543 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:20.743444  7543 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:20.743873 32371 raft_consensus.cc:2410] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 26 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:24:20.744253 31676 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:20.744367  7543 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:20.705399  7779 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.108s	user 0.068s	sys 0.090s
I20260709 12:24:20.720830  7831 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.725059  7813 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.725615  7832 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.674489  7755 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:20.731495  6752 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.734957  7786 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:20.762919 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:20.766757 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:20.742411  7511 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:20.768731 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:20.765609 31751 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
W20260709 12:24:20.771037 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56620.4us
I20260709 12:24:20.771790 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:20.773375 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:20.769568 31749 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:20.797340 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:20.801553 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.802337 31747 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:20.802809 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:20.760591  7810 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:20.765050  6752 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:20.806939 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:20.767282  7772 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.720983  7756 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.076s
I20260709 12:24:20.810849 31746 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:20.768273  7295 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:20.821364  7832 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:20.795291  7592 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:20.818768 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:20.832950 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:20.833796 31747 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:20.768759  7813 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.826154 31751 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 18 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
W20260709 12:24:20.805837  7808 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.889s	user 0.014s	sys 0.000s
W20260709 12:24:20.847761 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74979.1us
I20260709 12:24:20.865718 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.867470 31820 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:20.869180 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.876434 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3406) took 1973 ms. Trace:
I20260709 12:24:20.876910 31821 rpcz_store.cc:276] 0709 12:24:18.903345 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:18.911659 (+  8314us) service_pool.cc:224] Handling call
0709 12:24:18.912045 (+   386us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:19.250248 (+338203us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:19.250345 (+    97us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:19.250393 (+    48us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:19.250430 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:20.876212 (+1625782us) spinlock_profiling.cc:243] Waited 338 ms on lock 0x7b54002675d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:20.876257 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.876382 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":777279744}
I20260709 12:24:20.857211  7755 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:20.880445 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:20.884876 31821 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:20.824115  7772 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.887627 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:20.888175 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:20.885269 31821 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:20.862680  7460 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:20.894084 31751 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:24:20.899394  7831 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:20.905409  7739 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.510s	user 0.008s	sys 0.000s
W20260709 12:24:20.910414  7846 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:24:20.840373  7295 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.942104 31678 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:20.942378 32372 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:20.847291  7592 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.883180  7679 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:24:20.911839  7798 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.559s	user 0.002s	sys 0.010s
I20260709 12:24:20.952919 31818 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: []
I20260709 12:24:20.961400  7798 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.963754  7798 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:20.964579 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:20.965652 32374 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 1190) took 1856 ms (client timeout 2072 ms). Trace:
I20260709 12:24:20.913213  7561 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:20.967005  7561 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:20.968122  7561 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.969192 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:20.969678  7561 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:20.969849 32378 raft_consensus.cc:2376] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24. Re-sending same reply.
I20260709 12:24:20.885409  7493 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:20.971067  7493 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:20.973119  7365 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:20.973636  7493 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:20.974676 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:20.976598  7452 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.620s	user 0.012s	sys 0.001s
I20260709 12:24:20.978989  7295 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:20.979749  7295 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:20.980204  7295 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:20.980618  7295 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:20.983901  7813 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:20.984262 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:20.984440  7592 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:20.984926 31678 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:24:20.985563 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:20.986266 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1306) took 1057 ms. Trace:
I20260709 12:24:20.986408 31748 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:20.986713 31670 rpcz_store.cc:276] 0709 12:24:19.929177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.930180 (+  1003us) service_pool.cc:224] Handling call
0709 12:24:20.986059 (+1055879us) spinlock_profiling.cc:243] Waited 25.6 ms on lock 0x7b54001d6354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:20.986198 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":58765568}
W20260709 12:24:20.988374 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.990137 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:20.964797 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2623) took 3778 ms. Trace:
I20260709 12:24:20.993255 32377 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:24:20.992354 31673 rpcz_store.cc:276] 0709 12:24:17.186002 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:17.347183 (+161181us) service_pool.cc:224] Handling call
0709 12:24:17.454591 (+107408us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:17.512611 (+ 58020us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:17.527145 (+ 14534us) log.cc:844] Serialized 14 byte log entry
0709 12:24:18.807717 (+1280572us) spinlock_profiling.cc:243] Waited 5.17 ms on lock 0x7b180004e3a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:24:18.957560 (+149843us) log.cc:844] Serialized 14 byte log entry
0709 12:24:19.255209 (+297649us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:19.255260 (+    51us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:19.255325 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:19.255402 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:20.964611 (+1709209us) spinlock_profiling.cc:243] Waited 52.4 ms on lock 0x7b54002553d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:20.964734 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11857,"prepare.queue_time_us":729743,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":71,"raft.queue_time_us":142311,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":9916,"spinlock_wait_cycles":132408064,"thread_start_us":1037384,"threads_started":3,"wal-append.queue_time_us":814714,"wal-append.run_cpu_time_us":2713,"wal-append.run_wall_time_us":1687912}
I20260709 12:24:20.995450 31669 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
W20260709 12:24:20.997534 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 340990us
I20260709 12:24:20.997634 32375 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:20.966264 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:20.998672 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:20.944367 31821 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.15, Last appended by leader: 15, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:20.914055 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:20.966192 32374 rpcz_store.cc:269] 0709 12:24:19.109317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.109493 (+   176us) service_pool.cc:224] Handling call
0709 12:24:20.965442 (+1855949us) spinlock_profiling.cc:243] Waited 565 ms on lock 0x7b54006c0554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:20.965591 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1299174784}
I20260709 12:24:21.001569 31746 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:24:20.951272  7679 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:20.942025  7460 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.013111 31822 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:21.026391 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:21.001323 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.014513 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:21.003427  7452 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:21.021209  7779 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Stepping down as leader of term 18
W20260709 12:24:21.023867 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.002277 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:21.025604  7780 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.891s	user 0.009s	sys 0.004s
W20260709 12:24:21.027437 31671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2641) took 1979 ms (client timeout 1751 ms). Trace:
I20260709 12:24:21.043735  7452 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:21.043640 31671 rpcz_store.cc:269] 0709 12:24:19.047566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.084614 (+ 37048us) service_pool.cc:224] Handling call
0709 12:24:21.027204 (+1942590us) spinlock_profiling.cc:243] Waited 12.2 ms on lock 0x7b5400046ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:21.027338 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":28104448}
I20260709 12:24:21.041229  7779 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:21.030939 31677 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 20. Current term is 21. Ops: []
W20260709 12:24:21.045176  7819 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.019s	user 0.000s	sys 0.011s
I20260709 12:24:21.045351  7779 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.13, Last appended by leader: 13, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:21.046145 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1307) took 1116 ms. Trace:
I20260709 12:24:21.047067 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:21.047472 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:21.046620  7779 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:21.008189  7570 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.048s
I20260709 12:24:21.056222 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:21.056787 31750 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:21.056948  7679 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:21.058017  7679 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:21.035555  7772 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:21.048205  7801 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:24:21.037186  7514 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.549s	user 0.011s	sys 0.000s
I20260709 12:24:21.060730 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:21.060827  7514 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.062307 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:21.063107  7514 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:21.063747 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:21.064563  7856 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:21.065092  7856 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:21.066324 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:21.066236  7856 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:21.067688 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:21.068339 31744 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:21.069578 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:21.064924 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:21.072038 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:21.072453 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:20.881356  7818 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:21.041702 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.042388 32378 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 28: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:21.047685 31745 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:21.019227  7854 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:21.048017 31747 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:21.040340  7857 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:21.033697 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:21.087680 31748 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:21.046514 31677 rpcz_store.cc:276] 0709 12:24:19.929536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.929709 (+   173us) service_pool.cc:224] Handling call
0709 12:24:21.030739 (+1101030us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.045149 (+ 14410us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.045931 (+   782us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400046ad0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:21.046090 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2531212160}
I20260709 12:24:21.046990 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:21.049832 32372 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:21.047801  7452 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 23.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.055464 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:21.089762 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:21.077142 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.099268 32378 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:21.099475 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:21.073711 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 318136us
I20260709 12:24:21.003078  7759 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:21.049645  7805 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.713s	user 0.007s	sys 0.007s
I20260709 12:24:21.136965 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:21.026286  7590 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.508s	user 0.013s	sys 0.002s
I20260709 12:24:21.095788 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:21.074131 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.106480 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:21.137741 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:21.137779 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:21.143028 31744 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:21.063772  7514 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:21.140444 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:21.145884 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:21.082083  7684 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:21.083436  7801 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:21.083177  7501 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:21.152770 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:21.153597 31747 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:21.151517 31751 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:21.087440  7857 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.102900  7780 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:21.173115 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 77472.1us
I20260709 12:24:21.168720 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:21.204233 31748 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:21.172020 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 65691.2us
I20260709 12:24:21.204723 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:21.205905 32372 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:24:21.205734 31642 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:21.175948 31752 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:21.181173 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:21.182016  7813 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:21.150795  7514 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:21.182580  7801 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.185036  7460 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:21.186546  7854 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:21.187968  7766 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:21.191390  7759 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:21.159973  7684 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:21.192178 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:21.172730  7857 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:21.195343  7465 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:21.196434 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:21.197962 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:21.164604 31745 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:21.201717 31750 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:21.210608 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:21.203874  7501 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:21.210088  7813 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:21.186756  7853 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:21.212829  7514 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:21.123071  7818 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.217578 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:21.239801 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:21.226080  7684 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:21.155357  7590 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:21.223888 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:21.236884  7460 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 19. Reason: could not achieve majority
I20260709 12:24:21.150157  7805 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.215595  7733 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:21.206951  7663 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.234139 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:21.236158 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:21.219825  7430 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.582s	user 0.016s	sys 0.055s
I20260709 12:24:21.215963  7736 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:21.231096  7465 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:21.224332 32376 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 28: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:21.223798  7780 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.020s
I20260709 12:24:21.238488  7813 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:21.240036  7853 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:21.258257 31824 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.500s	user 0.008s	sys 0.019s
W20260709 12:24:21.265830 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 377734us
W20260709 12:24:21.292850 31824 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.009s	sys 0.019s
I20260709 12:24:21.293373 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:21.266583 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:21.261763 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:21.268666 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:21.263788  7854 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:24:21.300266 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.262578 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:21.263922 31827 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:21.261145  7430 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.016s	sys 0.056s
I20260709 12:24:21.289484 31748 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
W20260709 12:24:21.270539  7531 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.720s	user 0.009s	sys 0.002s
W20260709 12:24:21.312497 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50822.2us
I20260709 12:24:21.313521 31642 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:21.316932 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:21.334079 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:21.335938 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:21.337280 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.324199  7592 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:21.341954  7592 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:21.342367  7592 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:21.344348  7590 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:21.333380  7522 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:21.266274  7501 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.264729  7378 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:24:21.346553 31821 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:21.261422  7663 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:21.264695  7295 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:21.301014  7854 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:21.289691  7853 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:21.351897 32378 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:21.324507  7792 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:21.359434 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 90825.6us
I20260709 12:24:21.335343  7531 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.363029 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 1349) took 1087 ms. Trace:
I20260709 12:24:21.353235  7766 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:21.360955  7590 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:24:21.355640 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 93251.6us
W20260709 12:24:21.387840  7848 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:24:21.406848 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:21.407660 31826 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.514s	user 0.010s	sys 0.020s
I20260709 12:24:21.333171  7786 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:21.363816 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 1341) took 1104 ms. Trace:
I20260709 12:24:21.394199 32370 rpcz_store.cc:276] 0709 12:24:20.275707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.300928 (+ 25221us) service_pool.cc:224] Handling call
0709 12:24:21.362809 (+1061881us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b5400544f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:21.362962 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":372047616}
I20260709 12:24:21.403900 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:21.266711  7760 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.073s
I20260709 12:24:21.403757 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:21.407603 31752 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 24 index: 18. (index mismatch)
I20260709 12:24:21.387997 31827 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
W20260709 12:24:21.400409 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64606.5us
I20260709 12:24:21.477599 32371 rpcz_store.cc:276] 0709 12:24:20.259194 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.266401 (+  7207us) service_pool.cc:224] Handling call
0709 12:24:21.363626 (+1097225us) spinlock_profiling.cc:243] Waited 465 ms on lock 0x7b54004984d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:21.363756 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1070279040}
I20260709 12:24:21.467862 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:21.481189 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 77380.1us
I20260709 12:24:21.481407 31744 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:21.478547 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3551) took 1023 ms. Trace:
I20260709 12:24:21.397943  7613 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:21.488157 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:21.482513 31826 rpcz_store.cc:276] 0709 12:24:20.455514 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.455666 (+   152us) service_pool.cc:224] Handling call
0709 12:24:20.484355 (+ 28689us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:20.484732 (+   377us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:20.484776 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:20.485339 (+   563us) log.cc:844] Serialized 52 byte log entry
0709 12:24:20.663831 (+178492us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:20.871477 (+207646us) log.cc:844] Serialized 14 byte log entry
0709 12:24:20.892354 (+ 20877us) log.cc:844] Serialized 14 byte log entry
0709 12:24:21.193460 (+301106us) spinlock_profiling.cc:243] Waited 3.11 ms on lock 0x7b180048c6a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:21.408343 (+214883us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.408432 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:21.478257 (+ 69825us) raft_consensus.cc:1685] finished
0709 12:24:21.478289 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:21.478478 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21297,"prepare.queue_time_us":511879,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":126030,"raft.run_cpu_time_us":385,"raft.run_wall_time_us":12459,"spinlock_wait_cycles":7144192,"thread_start_us":884842,"threads_started":3,"wal-append.queue_time_us":238090}
I20260709 12:24:21.391582  7854 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:21.493446 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 89790.2us
I20260709 12:24:21.494514 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:21.442724  7782 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.549s	user 0.006s	sys 0.024s
I20260709 12:24:21.425836 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:21.401894  6752 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:21.442423  7755 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:21.408999  7780 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:21.499373 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:21.469103  7590 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:21.485152  7432 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:21.486282  7522 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
I20260709 12:24:21.531777 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:21.402593  7792 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.443109  7430 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.095s
I20260709 12:24:21.442704  7858 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:21.442974  7856 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.065s
I20260709 12:24:21.536199  7378 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:21.428021  7763 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.516s	user 0.004s	sys 0.020s
I20260709 12:24:21.537815 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:21.495931  7511 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:21.508801  7733 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:24:21.542858 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 117170us
W20260709 12:24:21.545855 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46617.9us
I20260709 12:24:21.541199  7570 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:21.537840  7805 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:21.551128 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:21.557570 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:21.546510  7772 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:24:21.563498 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.548180  7799 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:21.490804  7786 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:21.556432  7763 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.008s	sys 0.024s
I20260709 12:24:21.570356 31745 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:21.571447 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:21.573889 31752 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:21.568632  7733 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:21.556193  7511 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:21.591918 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.549887  7780 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:21.564575  7866 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:21.572611  7522 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:21.626739  7465 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:21.630831 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1145) took 1605 ms. Trace:
I20260709 12:24:21.630437  7733 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:21.635489  7859 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
W20260709 12:24:21.701165 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.691002 32369 rpcz_store.cc:276] 0709 12:24:20.050601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.062191 (+ 11590us) service_pool.cc:224] Handling call
0709 12:24:20.062561 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:20.294787 (+232226us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.630493 (+1335706us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b5400427f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:21.630747 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":443511680}
I20260709 12:24:21.636699  7590 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.638018  7295 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:21.623960  7511 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:21.639585  7792 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:21.647534 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:21.557590  7858 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.557883  7861 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.059s
I20260709 12:24:21.671916 31748 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 28. Current term is 29. Ops: [26.15-28.16]
I20260709 12:24:21.672946 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:21.734872 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:21.735733 31751 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:21.673856 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:21.736663 31747 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:21.685943 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:21.739220 31677 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:21.574572  7865 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:24:21.714424 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.746939 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:21.724375 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:21.747973 31676 raft_consensus.cc:2376] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21. Re-sending same reply.
I20260709 12:24:21.559391  7378 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:21.741109 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.751367 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:21.751754 31746 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:24:21.752135 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:21.754016 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:21.766988 31669 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:21.767231 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:21.790055 31819 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:21.795285  7543 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:21.796137  7543 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:21.798062  7684 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.540s	user 0.007s	sys 0.002s
I20260709 12:24:21.768492  7445 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:21.801626  7765 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:24:21.801857 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.802264  7765 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:21.803056  7663 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:21.803242  7765 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.805639 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:21.807359  7543 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:21.809123 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:21.810040 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3548) took 1490 ms. Trace:
I20260709 12:24:21.668100  7755 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:21.820218 31824 rpcz_store.cc:276] 0709 12:24:20.319849 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.325679 (+  5830us) service_pool.cc:224] Handling call
0709 12:24:20.326065 (+   386us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:20.393312 (+ 67247us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:20.393361 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:20.401529 (+  8168us) log.cc:844] Serialized 26 byte log entry
0709 12:24:20.670288 (+268759us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:21.293551 (+623263us) log.cc:844] Serialized 14 byte log entry
0709 12:24:21.316502 (+ 22951us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.773008 (+456506us) spinlock_profiling.cc:243] Waited 66.9 ms on lock 0x7b540043d5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:21.773086 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:21.773160 (+    74us) raft_consensus.cc:1685] finished
0709 12:24:21.809783 (+ 36623us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:21.809968 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29067,"prepare.queue_time_us":363605,"prepare.run_cpu_time_us":24,"prepare.run_wall_time_us":26,"raft.queue_time_us":85753,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":7006,"spinlock_wait_cycles":153970944,"thread_start_us":713355,"threads_started":2,"wal-append.queue_time_us":242702}
I20260709 12:24:21.825078  7592 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:21.777568  7531 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:21.827696  7679 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.769s	user 0.008s	sys 0.004s
I20260709 12:24:21.834064 31819 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:21.834973 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:21.836330 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:21.837417  7592 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:21.837477 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:21.810976  7865 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:21.839691 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:21.840516 31746 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:21.840600  7865 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.855806 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:21.858668 31567 catalog_manager.cc:5697] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 11 to 21, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 21 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:21.861140 31567 catalog_manager.cc:5697] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 17 to 18, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 18 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:21.864562 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1152) took 1159 ms. Trace:
I20260709 12:24:21.610289  7786 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:21.848284 31821 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 15. Preceding OpId from leader: term: 22 index: 15. (term mismatch)
W20260709 12:24:21.877189 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.891956 31821 pending_rounds.cc:85] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 14
W20260709 12:24:21.892298 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.892334 31821 pending_rounds.cc:107] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 21.15
W20260709 12:24:21.893018 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:21.892970 31821 raft_consensus.cc:2889] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:21.863991 31567 catalog_manager.cc:5697] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 16 to 19, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 19 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:24:21.898736 32342 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.897s	user 0.009s	sys 0.004s
W20260709 12:24:21.899224 32342 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.898s	user 0.009s	sys 0.004s
W20260709 12:24:21.899549 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 898353us
I20260709 12:24:21.798543  7684 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:21.865127 32374 rpcz_store.cc:276] 0709 12:24:20.705182 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.011311 (+306129us) service_pool.cc:224] Handling call
0709 12:24:21.015738 (+  4427us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:21.016200 (+   462us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.016238 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:21.016860 (+   622us) log.cc:844] Serialized 26 byte log entry
0709 12:24:21.490470 (+473610us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:21.490534 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.490694 (+   160us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:21.864237 (+373543us) raft_consensus.cc:1685] finished
0709 12:24:21.864272 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:21.864478 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1806,"raft.run_cpu_time_us":90,"raft.run_wall_time_us":91,"thread_start_us":473130,"threads_started":1,"wal-append.queue_time_us":790976}
I20260709 12:24:21.920702 31826 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:21.924935 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1466) took 1046 ms. Trace:
I20260709 12:24:21.925168 31746 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:21.925468 31821 rpcz_store.cc:276] 0709 12:24:20.878124 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.878509 (+   385us) service_pool.cc:224] Handling call
0709 12:24:20.878785 (+   276us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.346127 (+467342us) spinlock_profiling.cc:243] Waited 139 ms on lock 0x7b58007a2dd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd310aa 00007fe8fcdb3447 00007fe8fcdcbfe7 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4
0709 12:24:21.924691 (+578564us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.924875 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":319677824}
I20260709 12:24:21.869529  7780 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:21.930480 31819 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 17 index: 10. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
W20260709 12:24:21.932109  7691 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 22 current term: 22 is higher.
I20260709 12:24:21.932545  7684 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:21.934576 31567 catalog_manager.cc:5697] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 14 to 15, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 15 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:21.900295 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:21.921133 31745 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:21.889124 31747 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:21.839080  7854 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:21.977628 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:21.972044 31567 catalog_manager.cc:5697] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 19 to 20, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 20 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:21.979666 31669 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:21.981462 32377 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.003s	sys 0.036s
W20260709 12:24:21.981869 32377 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.552s	user 0.003s	sys 0.036s
I20260709 12:24:21.983207 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:21.984323 31747 raft_consensus.cc:2410] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 25 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
W20260709 12:24:21.990214 31749 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.023s	sys 0.019s
W20260709 12:24:21.991963 31749 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.023s	sys 0.019s
I20260709 12:24:22.012639 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:21.975945  7452 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:22.014410 31747 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:24:22.024436 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.025394 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:22.026757 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.035369 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:21.978093  7378 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:22.037137 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.052966 31675 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:21.976460  7543 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.044s
I20260709 12:24:22.038023 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:22.057464 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:22.005889  7691 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:22.061772 31744 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.570s	user 0.011s	sys 0.000s
W20260709 12:24:22.046321  7853 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.648s	user 0.011s	sys 0.001s
W20260709 12:24:22.050683 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.058769 31669 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:22.020546  7452 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:22.074146 31644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.571s	user 0.005s	sys 0.027s
W20260709 12:24:22.082548 31644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.007s	sys 0.027s
I20260709 12:24:22.062359 31744 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:22.085696 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:22.086699 31744 raft_consensus.cc:2368] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26.
W20260709 12:24:22.057528  7736 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.790s	user 0.008s	sys 0.004s
W20260709 12:24:22.053717  7514 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.802s	user 0.011s	sys 0.002s
I20260709 12:24:22.093691 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.094352 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 17 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.113180 31675 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 22 because replica has last-logged OpId of term: 19 index: 11, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 10.
I20260709 12:24:22.036352 31567 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 20 to 21, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 21 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:22.047552  7460 raft_consensus.cc:2764] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election decision vote started in defunct term 21: won
I20260709 12:24:22.037065  7445 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.072s
I20260709 12:24:22.116466 31671 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:22.117280  7818 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:22.094492  7452 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:22.179661 31675 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:22.120802 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:22.121239 31670 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: [20.14-21.15]
I20260709 12:24:22.122040 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.123343  7858 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:22.124059 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.215957 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 95234.8us
I20260709 12:24:22.216694 31678 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:24:22.127545  7880 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:24:22.128583  7767 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:22.130779 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.132192  7799 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:22.136325 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.138921 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 1462) took 1274 ms. Trace:
I20260709 12:24:22.199110  7854 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:22.150029 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:22.043392  7815 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.037s
W20260709 12:24:22.073181  7857 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.836s	user 0.013s	sys 0.000s
W20260709 12:24:22.062903  7801 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.010s	sys 0.021s
I20260709 12:24:22.159767 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.251150 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:22.165344 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.551s	user 0.004s	sys 0.020s
I20260709 12:24:22.044579  7761 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.065s
I20260709 12:24:22.255293 31819 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
W20260709 12:24:22.165747 31752 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.004s	sys 0.021s
I20260709 12:24:22.089839  7782 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:22.115619 31673 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:22.115056 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.054759  7750 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.108s
W20260709 12:24:22.157814  7878 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.099s	user 0.001s	sys 0.000s
I20260709 12:24:22.059270 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:22.268148 31747 raft_consensus.cc:2368] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28.
I20260709 12:24:22.090179  7873 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:22.153051  7736 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:22.094054  7378 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.185783 31745 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:22.179121  7432 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.625s	user 0.006s	sys 0.006s
I20260709 12:24:22.090343  7691 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.092370  7853 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.202019 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:22.292428 31671 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:22.182839  7445 consensus_queue.cc:1243] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 27.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:24:22.152247  7514 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:24:22.197266  7813 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.905s	user 0.006s	sys 0.007s
W20260709 12:24:22.204995 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.150727  7501 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.764s	user 0.004s	sys 0.059s
I20260709 12:24:22.218524 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:22.140067  7570 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:22.226402 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:22.225941 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:22.164259 31567 catalog_manager.cc:5697] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 15 to 18. New cstate: current_term: 18 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:22.224268 31827 rpcz_store.cc:276] 0709 12:24:20.864266 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.866709 (+  2443us) service_pool.cc:224] Handling call
0709 12:24:22.138726 (+1272017us) spinlock_profiling.cc:243] Waited 379 ms on lock 0x7b54004461d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:22.138850 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":872178048}
W20260709 12:24:22.229835 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.225461 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:22.320117 31822 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:22.214272 31751 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
W20260709 12:24:22.240399 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 90468.9us
I20260709 12:24:22.241537 31748 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 23 index: 14. (term mismatch)
I20260709 12:24:22.324383 31748 pending_rounds.cc:85] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 13
I20260709 12:24:22.324747 31748 pending_rounds.cc:107] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 22.14
I20260709 12:24:22.325059 31748 raft_consensus.cc:2889] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:22.252969 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.639s	user 0.005s	sys 0.021s
W20260709 12:24:22.258435 31752 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.620s	user 0.004s	sys 0.022s
I20260709 12:24:22.220748  7767 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:22.332562  7767 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.334147  7767 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:22.259621 31676 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:22.226691  7799 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:22.226984  7430 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.339011 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:22.338900  7430 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:22.253891 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.685s	user 0.000s	sys 0.036s
W20260709 12:24:22.340085 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.771s	user 0.000s	sys 0.036s
W20260709 12:24:22.340430 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 782884us
I20260709 12:24:22.242619  7543 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.341593  7430 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:22.341325  7543 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.341984 31676 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:22.343605  7543 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:22.256377  7801 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.709s	user 0.010s	sys 0.021s
I20260709 12:24:22.248726  7766 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:22.286482 31673 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:22.287998 31745 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:22.344429 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:22.295352 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.345871 31673 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
W20260709 12:24:22.315136 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.271935  7782 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:22.322079 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.347376 31673 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:22.347524 31751 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:22.347535  7782 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.266196 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3417) took 1072 ms. Trace:
I20260709 12:24:22.350818 31824 rpcz_store.cc:276] 0709 12:24:21.194198 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.836413 (+642215us) service_pool.cc:224] Handling call
0709 12:24:21.836784 (+   371us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:21.837095 (+   311us) raft_consensus.cc:1534] Early marking committed up to index 11
0709 12:24:21.842252 (+  5157us) log.cc:844] Serialized 14 byte log entry
0709 12:24:22.265844 (+423592us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:22.265901 (+    57us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:22.265947 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.266033 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.266141 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5295,"prepare.queue_time_us":4577,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":26026,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":3789,"thread_start_us":397200,"threads_started":1}
I20260709 12:24:22.230350  7522 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:22.353878  7522 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.354698 31822 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:22.357388 31748 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 23. Current term is 24. Ops: [22.16-23.17]
I20260709 12:24:22.360044 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:22.362417  7799 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.366326 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:22.369490  7799 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:22.375229 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51325.2us
I20260709 12:24:22.376086 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:22.283416  7460 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:22.265092  7815 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.046s
I20260709 12:24:22.254436  7857 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.392509 31674 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:22.394165 31748 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:22.294518  7831 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.393056 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:22.402352 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:22.317282  7501 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.930s	user 0.004s	sys 0.060s
I20260709 12:24:22.312760  7432 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:22.448921 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62789.3us
I20260709 12:24:22.447453 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:22.411196 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 192766us
W20260709 12:24:22.456218  7882 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.069s	user 0.000s	sys 0.000s
I20260709 12:24:22.446821  7894 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.306522  7873 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:22.458009 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2825) took 1585 ms. Trace:
I20260709 12:24:22.459887  7514 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:22.458499 31749 rpcz_store.cc:276] 0709 12:24:20.872280 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.872480 (+   200us) service_pool.cc:224] Handling call
0709 12:24:20.877779 (+  5299us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:21.306376 (+428597us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.306435 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:21.338436 (+ 32001us) log.cc:844] Serialized 26 byte log entry
0709 12:24:21.993359 (+654923us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:22.014371 (+ 21012us) log.cc:844] Serialized 14 byte log entry
0709 12:24:22.014614 (+   243us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.014714 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.126090 (+111376us) spinlock_profiling.cc:243] Waited 15.3 ms on lock 0x7b1800424660. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:22.457636 (+331546us) raft_consensus.cc:1685] finished
0709 12:24:22.457771 (+   135us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.457932 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":25347,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":56,"raft.queue_time_us":95667,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":17983,"spinlock_wait_cycles":35195136,"thread_start_us":1021656,"threads_started":2,"wal-append.queue_time_us":878095}
I20260709 12:24:22.461897  7691 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:22.458526  7873 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.463685  7853 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:22.462241 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:22.466943 31749 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:22.467063 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:22.469425 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.457154  7894 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.470611  6752 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 19.10, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260709 12:24:22.470923  7831 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.472815  7365 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:22.473201  7365 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:22.473526  7857 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:22.473845  7881 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.191s	user 0.000s	sys 0.000s
I20260709 12:24:22.474435 31672 raft_consensus.cc:2368] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28.
I20260709 12:24:22.480571 31567 catalog_manager.cc:5697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 12 to 18, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 18 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:22.474675  7365 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.473178  7894 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:22.409989  7543 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:22.391762  7755 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:22.372362  7880 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:24:22.446615 31823 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:22.410331  7861 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:24:22.479306  7888 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.167s	user 0.000s	sys 0.000s
I20260709 12:24:22.481846 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:22.288865  7378 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.492717 31671 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
I20260709 12:24:22.495663 31567 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 14 to 17. New cstate: current_term: 17 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:22.447739  7756 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:22.497359 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:22.497936 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:22.375135  7570 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:22.446787  7592 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 8, Time since last communication: 0.053s
I20260709 12:24:22.479964  7801 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:22.504698 32374 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:22.373835  7813 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.507887 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:22.508582 32369 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:24:22.508481 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:22.510823 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:22.444926 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1149) took 1761 ms. Trace:
I20260709 12:24:22.517392 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:22.512665 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:22.511644 31749 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:22.497193  7818 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:22.518810 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:22.409601  7766 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:22.519882 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:22.519671 31567 catalog_manager.cc:5697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 12 to 14. New cstate: current_term: 14 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:22.505229  7894 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:22.524442 32378 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:22.551098 31567 catalog_manager.cc:5697] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 18 to 24, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 24 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:22.542490 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:22.527390 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.557749 31673 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:22.558104 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:22.504393  7756 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:22.536485 32370 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:22.391513  7295 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.119s
I20260709 12:24:22.562422 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:22.563246 32373 raft_consensus.cc:2368] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28.
I20260709 12:24:22.519631  7853 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:22.568264 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:22.549579 32369 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
W20260709 12:24:22.512599  7511 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.744s	user 0.011s	sys 0.000s
W20260709 12:24:22.525434 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 274174us
I20260709 12:24:22.560521 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.496204  7543 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:22.490796  7857 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.029s
I20260709 12:24:22.513700  7799 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:24:22.540261  7884 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.337s	user 0.001s	sys 0.000s
I20260709 12:24:22.563565 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.535216  7430 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:22.495828  7432 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.518158 32377 rpcz_store.cc:276] 0709 12:24:20.683051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.995199 (+312148us) service_pool.cc:224] Handling call
0709 12:24:20.995557 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:20.995810 (+   253us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:21.408661 (+412851us) log.cc:844] Serialized 14 byte log entry
0709 12:24:21.982565 (+573904us) log.cc:844] Serialized 14 byte log entry
0709 12:24:22.444447 (+461882us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:22.444506 (+    59us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:22.444558 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.444715 (+   157us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.444861 (+   146us) inbound_call.cc:177] Queueing success response
0709 12:24:22.469406 (+ 24545us) spinlock_profiling.cc:243] Waited 5.83 ms on lock 0x7b180007cf60. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
Metrics: {"prepare.queue_time_us":639897,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":94,"raft.queue_time_us":119934,"raft.run_cpu_time_us":255,"raft.run_wall_time_us":23459,"spinlock_wait_cycles":13409536,"thread_start_us":1406585,"threads_started":3,"wal-append.queue_time_us":845463}
I20260709 12:24:22.535123  7514 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:22.577312 32376 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:22.563993  7756 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:22.580078 32342 leader_election.cc:336] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.695s (SENT)
I20260709 12:24:22.600894 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3519) took 1192 ms. Trace:
I20260709 12:24:22.601086 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:22.518780  7890 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:22.585836 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:22.592103 31671 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:22.562398  7818 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:22.592958 31567 catalog_manager.cc:5697] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 17 to 18, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 18 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:22.577983 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:22.598984  7883 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.302s	user 0.001s	sys 0.000s
I20260709 12:24:22.509220  7755 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.610363 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:22.611367 31827 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:22.612777 32374 raft_consensus.cc:2393] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:22.601354 31750 rpcz_store.cc:276] 0709 12:24:21.408236 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.408365 (+   129us) service_pool.cc:224] Handling call
0709 12:24:21.471747 (+ 63382us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:21.484877 (+ 13130us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.484913 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:21.496877 (+ 11964us) log.cc:844] Serialized 26 byte log entry
0709 12:24:22.005138 (+508261us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:22.005197 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.005297 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.600612 (+595315us) raft_consensus.cc:1685] finished
0709 12:24:22.600643 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.600829 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5299,"raft.queue_time_us":67038,"raft.run_cpu_time_us":179,"raft.run_wall_time_us":11437,"thread_start_us":455973,"threads_started":1,"wal-append.queue_time_us":713305}
I20260709 12:24:22.560468  7750 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
W20260709 12:24:22.610002 32342 leader_election.cc:336] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.758s (SENT)
I20260709 12:24:22.615536 32371 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:22.585925  7902 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.586812  7853 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:22.596396  7295 consensus_queue.cc:1243] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 25.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:24:22.591318  7430 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:22.602537 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.599242  7514 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:22.611155  7903 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.587838  7511 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:22.632081 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.563104  7894 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.645205 31567 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 14 to 19. New cstate: current_term: 19 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:22.632750 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:22.631721 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1465) took 1760 ms. Trace:
W20260709 12:24:22.644114 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.644063 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 273136us
W20260709 12:24:22.651388 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:22.642865  7887 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.346s	user 0.000s	sys 0.001s
I20260709 12:24:22.635432  7430 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:22.610419  7890 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.661912 31567 catalog_manager.cc:5697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 15 to 21, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 21 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:22.659543  7493 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.665297  7493 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:22.666679  7901 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.056s	user 0.000s	sys 0.001s
I20260709 12:24:22.647611 31820 rpcz_store.cc:276] 0709 12:24:20.871298 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:20.871496 (+   198us) service_pool.cc:224] Handling call
0709 12:24:20.873869 (+  2373us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:21.322934 (+449065us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:21.322986 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:21.338439 (+ 15453us) log.cc:844] Serialized 26 byte log entry
0709 12:24:21.920062 (+581623us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:21.920105 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:21.920178 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.631475 (+711297us) raft_consensus.cc:1685] finished
0709 12:24:22.631501 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.631652 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10330,"raft.queue_time_us":60888,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":12420,"thread_start_us":851208,"threads_started":2,"wal-append.queue_time_us":747564}
I20260709 12:24:22.659790 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.620738  7909 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:22.676460 31567 catalog_manager.cc:5697] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 21 to 22, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 22 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:22.667147 31669 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:22.678179 31567 catalog_manager.cc:5697] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 15 to 20, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 20 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:22.650491 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:22.665300  7900 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:24:22.661476  7893 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.686291 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:22.670552  7493 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:22.692001  7895 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.092s	user 0.000s	sys 0.001s
W20260709 12:24:22.693565  7898 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.114s	user 0.001s	sys 0.000s
I20260709 12:24:22.693764  7815 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:22.694249  7815 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:22.688336 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.688336 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.624804  7902 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.695631 31750 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:22.695978 31674 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:22.694643  7815 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:22.691279  7896 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:24:22.700227 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:22.690128  7786 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.753s	user 0.011s	sys 0.002s
I20260709 12:24:22.702809 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.705153 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:22.705668  7452 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.514s	user 0.017s	sys 0.001s
I20260709 12:24:22.702401  7514 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.724179 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.727324 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:22.728132 31750 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:22.725862 31748 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:24:22.729357 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.729856 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:22.649627  7903 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.732031 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:22.721388  7501 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.012s
I20260709 12:24:22.658957  7543 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.738327 31567 catalog_manager.cc:5697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 15 to 19. New cstate: current_term: 19 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:22.734423  7755 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:22.734419  7885 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.006s
I20260709 12:24:22.734055  7765 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:24:22.756953 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.760205  7792 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:22.760768  7792 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:22.761693  7792 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.730753  7452 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:22.757109 31753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3503) took 1712 ms (client timeout 2019 ms). Trace:
W20260709 12:24:22.765990 31753 rpcz_store.cc:269] 0709 12:24:21.044550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.055112 (+ 10562us) service_pool.cc:224] Handling call
0709 12:24:22.756921 (+1701809us) spinlock_profiling.cc:243] Waited 1.17 s on lock 0x7b5400294fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:22.757049 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2701527808}
W20260709 12:24:22.769583  7886 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.398s	user 0.000s	sys 0.001s
I20260709 12:24:22.684968  7818 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.743098 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:22.853781 31750 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:22.743861 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1182) took 1118 ms. Trace:
I20260709 12:24:22.745205 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:22.754213 31823 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3421) took 1552 ms (client timeout 1700 ms). Trace:
I20260709 12:24:22.756701 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:22.856812 31748 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
W20260709 12:24:22.856756 31823 rpcz_store.cc:269] 0709 12:24:21.201551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.333727 (+132176us) service_pool.cc:224] Handling call
0709 12:24:22.754005 (+1420278us) spinlock_profiling.cc:243] Waited 955 ms on lock 0x7b540037a854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:22.754137 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2197579136}
I20260709 12:24:22.742826  7902 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:22.741101  7894 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:22.729238  7813 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:22.781998  7460 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:22.698967  7893 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.790496  7792 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:22.791862  7452 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:22.757153  7800 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.010s
I20260709 12:24:22.791944  7903 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:22.758044  7750 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.025s
I20260709 12:24:22.792867 32375 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:22.792948  7543 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:22.793337  7766 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:22.685642  7853 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.757272  7493 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.010s
I20260709 12:24:22.793744  7909 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:22.794275 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.791167 31567 catalog_manager.cc:5697] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 15 to 17, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 17 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:22.896070 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.896857 31753 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:22.896502 31567 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 12 to 16. New cstate: current_term: 16 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:22.897774 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:22.795552  7570 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.881410  7782 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election won for term 20
I20260709 12:24:22.799605  7890 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:22.899013  7452 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:22.899951  7452 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:22.900290  7570 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:22.898119 31567 catalog_manager.cc:5697] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 10 to 14, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 14 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:22.800297  7432 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:22.723420  7786 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.902051  7365 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:22.807703  7831 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:22.800295  7378 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:22.807785 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2662) took 1123 ms. Trace:
I20260709 12:24:22.902904 31672 rpcz_store.cc:276] 0709 12:24:21.684869 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.685483 (+   614us) service_pool.cc:224] Handling call
0709 12:24:22.807494 (+1122011us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b540022b354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:22.807678 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1649134976}
I20260709 12:24:22.903888  7786 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:22.802788 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.904608 31677 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:22.757615  7905 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.014s
I20260709 12:24:22.819574 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.907723 31670 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:22.908232  7893 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:22.821471 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.828249 31745 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 21. Current term is 22. Ops: [21.15-21.15]
I20260709 12:24:22.822342 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.909791 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1184) took 1064 ms. Trace:
I20260709 12:24:22.838398 31673 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:22.818187 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.910095 31676 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:22.857398 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:22.871330  7460 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:22.868775 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.905838 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2669) took 1160 ms. Trace:
I20260709 12:24:22.914561 31748 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:22.914645 31677 rpcz_store.cc:276] 0709 12:24:21.745882 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.746072 (+   190us) service_pool.cc:224] Handling call
0709 12:24:21.746442 (+   370us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.905029 (+1158587us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.905595 (+   566us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54001ff9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:22.905780 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2663079680}
I20260709 12:24:22.854288 31752 rpcz_store.cc:276] 0709 12:24:21.625162 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.625372 (+   210us) service_pool.cc:224] Handling call
0709 12:24:21.625862 (+   490us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:21.638081 (+ 12219us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:22.331071 (+692990us) log.cc:844] Serialized 14 byte log entry
0709 12:24:22.616874 (+285803us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:22.633943 (+ 17069us) log.cc:844] Serialized 26 byte log entry
0709 12:24:22.634327 (+   384us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:22.634390 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.634511 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:22.743617 (+109106us) raft_consensus.cc:1685] finished
0709 12:24:22.743638 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:22.743792 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11627,"prepare.queue_time_us":29876,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":13299,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":10053,"thread_start_us":976864,"threads_started":2,"wal-append.queue_time_us":327080}
I20260709 12:24:22.916405 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.917562 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:22.871460 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 114863us
I20260709 12:24:22.875344 31744 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:22.919018 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1185) took 1050 ms. Trace:
I20260709 12:24:22.873170  7759 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:22.925256 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 274818us
I20260709 12:24:22.919447 31744 rpcz_store.cc:276] 0709 12:24:21.868349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.346266 (+477917us) service_pool.cc:224] Handling call
0709 12:24:22.874854 (+528588us) spinlock_profiling.cc:243] Waited 16.4 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:24:22.875100 (+   246us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.918697 (+ 43597us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.918950 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":37828480}
I20260709 12:24:22.930128 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:22.931774 31567 catalog_manager.cc:5697] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 15 to 16, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 16 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:22.926684 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:22.934592 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:22.852399  7679 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:22.898224  7465 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:22.900873  7452 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.901989 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:22.955566 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.904006  7511 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:22.905795 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.818008 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.956955 31674 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:22.912806  7800 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:22.911006 31747 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:22.902390  7430 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.840758 31792 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:22.912770 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 55453.1us
I20260709 12:24:22.978111 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.869633 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:22.979189 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.910197 31745 rpcz_store.cc:276] 0709 12:24:21.845733 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.345179 (+499446us) service_pool.cc:224] Handling call
0709 12:24:22.827635 (+482456us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:24:22.827892 (+   257us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:22.909541 (+ 81649us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.909736 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":40037248}
I20260709 12:24:22.933032 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.979633 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:22.980192 31677 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:22.980244 31748 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:22.868134  7756 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:22.885954  7766 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:22.985456 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:22.881093 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 425647us
I20260709 12:24:22.951589 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.951767 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.988536 31669 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:22.952486 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:22.989377 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:22.941210 31752 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:22.955024 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:22.820962 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.916791  7750 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:22.958221 31749 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:22.959774 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:22.991554 31753 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:22.990145 31670 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:22.929023  7893 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:22.917018  7782 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:22.975464 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:22.838779  7861 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:22.932289  7831 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:22.930240 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:22.981406 31677 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
W20260709 12:24:22.981391 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 140717us
I20260709 12:24:22.978413 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:22.980394 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:23.019179 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:23.019496 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:23.020246 31745 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:22.939628  7432 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:23.003554 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:23.003937 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:23.004626 31669 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:23.004832 31673 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:23.005208 31675 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 28. Ops: [26.16-26.16]
I20260709 12:24:22.939407  7493 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.047s
I20260709 12:24:23.009856 31671 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 11. Preceding OpId from leader: term: 21 index: 12. (index mismatch)
I20260709 12:24:22.974743  7800 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:23.011410 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:23.029430 31676 raft_consensus.cc:2410] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 19 because replica has last-logged OpId of term: 18 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 17 index: 12.
I20260709 12:24:22.992131 31672 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:22.936576  7759 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.016659 31747 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:23.018241 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:23.021000 31750 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:23.021472 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:22.956683  7815 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:22.955637  7679 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.023705 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:23.041127 31745 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:23.041441 31675 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:22.992105  7813 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:23.055405  7813 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:23.055886  7813 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:23.026744 31673 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:23.056777  7800 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.058048  7800 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:23.058146 31745 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:22.991601  7750 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:23.027433  7798 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.902s	user 0.007s	sys 0.014s
I20260709 12:24:23.031468 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:23.059247  7798 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.938s	user 0.008s	sys 0.014s
W20260709 12:24:23.022722 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 92567.1us
I20260709 12:24:23.061172 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:23.061930 31673 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:23.060194  7514 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:23.062413 31791 leader_election.cc:400] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28.
I20260709 12:24:23.064321 31791 leader_election.cc:403] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
I20260709 12:24:23.070080  7514 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.071043  7514 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:23.074802 31792 leader_election.cc:400] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28.
I20260709 12:24:23.075350 31792 leader_election.cc:403] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Cancelling election due to peer responding with higher term
W20260709 12:24:23.076058 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 49920.3us
I20260709 12:24:23.076222  7798 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28.
I20260709 12:24:23.025581 31744 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:23.091233 31744 raft_consensus.cc:606] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:23.091778 31744 raft_consensus.cc:614] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:23.109691  7766 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.111691  7766 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:23.026734  7861 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.028193  7378 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:23.124011  7378 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:23.124526  7378 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:23.023509  7511 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:23.126332  7511 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.126719  7511 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:23.026135  7432 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.128750  7432 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:23.131008 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.133965 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2833) took 1445 ms. Trace:
I20260709 12:24:23.042762 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:23.136597 31669 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:23.043110 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.142397 31746 rpcz_store.cc:276] 0709 12:24:21.688064 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:21.978156 (+290092us) service_pool.cc:224] Handling call
0709 12:24:22.009946 (+ 31790us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.010157 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:22.050466 (+ 40309us) log.cc:844] Serialized 14 byte log entry
0709 12:24:22.484065 (+433599us) log.cc:844] Serialized 14 byte log entry
0709 12:24:22.756392 (+272327us) spinlock_profiling.cc:243] Waited 3.91 ms on lock 0x7b1800411520. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:22.842758 (+ 86366us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b1800411520. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:24:23.132870 (+290112us) log.cc:844] Serialized 14 byte log entry
0709 12:24:23.133414 (+   544us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:23.133479 (+    65us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:23.133544 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.133705 (+   161us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.133886 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11872,"prepare.queue_time_us":116362,"prepare.run_cpu_time_us":158,"prepare.run_wall_time_us":23666,"raft.queue_time_us":168312,"raft.run_cpu_time_us":366,"raft.run_wall_time_us":9719,"spinlock_wait_cycles":69904896,"thread_start_us":722346,"threads_started":2,"wal-append.queue_time_us":541133}
I20260709 12:24:23.048389 31747 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:23.148926 31747 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:23.149371 31747 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:23.093608 31744 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:24:23.179468 31744 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:23.179998 31744 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:23.052902 31750 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 28. Ops: []
I20260709 12:24:23.054942  7679 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:23.021759 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:23.222010 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 429588us
W20260709 12:24:23.222313 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 200647us
W20260709 12:24:23.225131 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.059634  7750 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.150349 31747 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:23.235185 31677 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:23.235610 31747 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:23.228041 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.233385 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 201990us
W20260709 12:24:23.236208 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 159244us
W20260709 12:24:23.241933 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.247612 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:23.237159 31744 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:23.265856 31744 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:23.266685 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:23.035146  7820 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:23.047308  7893 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:24:23.267561 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 205348us
I20260709 12:24:23.275190 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:23.279874 31823 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:23.283254 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:23.277155 31748 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:23.285171 31748 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 28. Current term is 29. Ops: [26.15-28.16]
I20260709 12:24:23.286965 32375 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
W20260709 12:24:23.290334 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.299752 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:23.291311 31825 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:23.285192 31669 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:23.291131 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:23.300577 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:23.307924 32371 raft_consensus.cc:2410] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 26 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:24:23.247612  7295 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:24:23.311789 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.332573 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.332954 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.302619 31823 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:23.323931 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:23.035660  7805 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.243s
I20260709 12:24:23.312144 31821 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:23.280699  7798 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:23.036514  7831 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:23.300954 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:23.272354  7820 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:23.337662 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:23.279527  7679 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:23.342041 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:23.053885  7782 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 19.10, Last appended by leader: 10, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.342866 31744 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 26 index: 19. (term mismatch)
I20260709 12:24:23.372125 31744 pending_rounds.cc:85] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 18
I20260709 12:24:23.372400 31744 pending_rounds.cc:107] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 25.19
I20260709 12:24:23.372656 31744 raft_consensus.cc:2889] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:23.344269 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:23.345415 31747 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 24 index: 17. (term mismatch)
I20260709 12:24:23.375497 31747 pending_rounds.cc:85] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 16
I20260709 12:24:23.375351 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:23.288537  7430 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:23.374465 31744 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:23.359503 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:23.376863 31821 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:23.370496  7820 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:23.358969  7779 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.383504  7779 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:23.388916 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 365099us
I20260709 12:24:23.391449  7514 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:23.370937  7295 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:23.393143  7295 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:23.396255 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 52064.9us
I20260709 12:24:23.346751 31823 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:23.371150 31820 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:23.375980 31747 pending_rounds.cc:107] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 23.17
W20260709 12:24:23.325204  7904 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:24:23.368947 31746 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:23.399035 31747 raft_consensus.cc:2889] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:23.391459 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:23.403101  7590 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:23.334901  7756 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:23.335598  7893 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:23.414888 32377 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:24:23.411449 31677 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 13. Preceding OpId from leader: term: 19 index: 13. (term mismatch)
I20260709 12:24:23.417443 31677 pending_rounds.cc:85] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 12
I20260709 12:24:23.417771 31677 pending_rounds.cc:107] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 18.13
I20260709 12:24:23.418066 31677 raft_consensus.cc:2889] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:23.412390 31748 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:23.318676  7493 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.053s
I20260709 12:24:23.421725 31746 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:24:23.396780  7763 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:23.450492 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.451188 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.452967 32371 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
W20260709 12:24:23.453238 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.455559 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:23.456535 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:23.457926 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2851) took 1239 ms. Trace:
I20260709 12:24:23.458453 31745 rpcz_store.cc:276] 0709 12:24:22.219032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.058986 (+839954us) service_pool.cc:224] Handling call
0709 12:24:23.059284 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:23.122859 (+ 63575us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:23.122888 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:23.122929 (+    41us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:23.122960 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.457636 (+334676us) spinlock_profiling.cc:243] Waited 63.4 ms on lock 0x7b5400294ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:23.457701 (+    65us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.457852 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":145802496}
I20260709 12:24:23.465080 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:23.465670 31820 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:24:23.407085  7861 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:23.390340 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:23.475528 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:23.483690 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 93439.4us
I20260709 12:24:23.485247 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:23.486472 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.486843 31677 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:23.492333 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:23.498347 31827 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:23.502578 32373 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:23.396654  7786 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.070s
I20260709 12:24:23.504458 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:23.512621 31669 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:23.386461  7779 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:23.509392  7452 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:23.524931  7792 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.644s	user 0.011s	sys 0.002s
I20260709 12:24:23.518013 31824 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:23.519251 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:23.448390  7763 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:23.529249 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 208423us
I20260709 12:24:23.527643 31642 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:23.499594  7853 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:23.533411 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:23.490238  7590 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:23.529145 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:23.532672 32342 leader_election.cc:336] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 2.224s (SENT)
I20260709 12:24:23.421088  7430 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.096s
I20260709 12:24:23.539783 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:23.490592  7915 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 21.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.047s
I20260709 12:24:23.552707 31677 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 13. (term mismatch)
W20260709 12:24:23.548362 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.554569 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:23.533774  7763 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:23.558918 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.560948 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.507463  7756 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 12, Time since last communication: 0.045s
I20260709 12:24:23.543303  7779 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:24:23.563787 31670 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.571s	user 0.011s	sys 0.003s
I20260709 12:24:23.570501 31670 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:23.556753 31677 pending_rounds.cc:85] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 12
I20260709 12:24:23.571494 31677 pending_rounds.cc:107] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 23.13
I20260709 12:24:23.572304 31677 raft_consensus.cc:2889] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:23.549329  7912 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.817s	user 0.005s	sys 0.001s
I20260709 12:24:23.577154 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:23.548751  7889 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:23.579939 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:23.547938  7927 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election won for term 24
W20260709 12:24:23.583876 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.536868  7755 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:24:23.584522 31670 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 20: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:23.585873 32369 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:24:23.587896 32373 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 13. (term mismatch)
I20260709 12:24:23.558779  7733 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.571532  7861 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:23.574013  7514 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:23.555766  7920 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.038s
I20260709 12:24:23.594583 32373 pending_rounds.cc:85] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 12
W20260709 12:24:23.595280  7926 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.066s	user 0.001s	sys 0.000s
I20260709 12:24:23.571900  7759 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:23.570108  7815 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:23.570252  7779 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.615603  7779 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:23.620438 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:23.597507 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.621883 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:23.601694  7915 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:23.601714  7831 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:23.603396 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2847) took 1635 ms. Trace:
I20260709 12:24:23.642352 31749 rpcz_store.cc:276] 0709 12:24:21.968258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.011120 (+1042862us) service_pool.cc:224] Handling call
0709 12:24:23.025742 (+ 14622us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:23.026102 (+   360us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:23.026137 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:23.026602 (+   465us) log.cc:844] Serialized 26 byte log entry
0709 12:24:23.409272 (+382670us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:23.409335 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.409440 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.603101 (+193661us) raft_consensus.cc:1685] finished
0709 12:24:23.603131 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.603331 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":521,"raft.run_cpu_time_us":70,"raft.run_wall_time_us":70,"thread_start_us":314973,"threads_started":1,"wal-append.queue_time_us":473175}
I20260709 12:24:23.603989 31752 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:23.599964  7927 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:23.598045  7861 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:23.606014  7813 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.550s	user 0.019s	sys 0.030s
I20260709 12:24:23.599751  7733 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.666196  7813 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:23.667557 31820 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:23.668432  7733 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:23.609678  7924 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.249s	user 0.001s	sys 0.000s
I20260709 12:24:23.624464 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:23.675107 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:23.625880  7929 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.259s	user 0.000s	sys 0.001s
I20260709 12:24:23.645756 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:23.715570 31747 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:23.716187 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:23.716744 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:23.717751 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:23.718134 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:23.718997 31747 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:23.720031 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:23.649663 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.721853 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.643687 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:23.723457 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:23.724398 31745 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:24:23.663244 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:23.663275 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.724521 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:23.725243 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.584205  7889 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:23.548496  7893 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:23.669860 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:23.675261 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:23.737705 31749 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:23.641678  7915 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:23.625638  7511 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:23.726222 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:23.744490 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:23.726752 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62539.2us
I20260709 12:24:23.744683 31748 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8725620974f7465eb0371f48761a1625"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:23.727787 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:23.745529 31748 raft_consensus.cc:606] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:23.746119 31746 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:23.738269 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:23.729771 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:23.746881 31824 raft_consensus.cc:2376] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28. Re-sending same reply.
I20260709 12:24:23.600771 32373 pending_rounds.cc:107] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 24.13
W20260709 12:24:23.697945 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.749684 32373 raft_consensus.cc:2889] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:23.668797  7813 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:23.639410 31677 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:23.740741 32343 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:23.746070 31748 raft_consensus.cc:614] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:23.756618 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:23.758559 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:23.752411 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 451510us
I20260709 12:24:23.756670 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:23.763298 32377 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:23.738080  7889 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:23.726137  7590 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:23.741521  7733 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.769765 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:23.774248 31676 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:24:23.771431  7853 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:23.750703  7925 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:23.782243 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.796917 31747 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:23.783614 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:23.784194 31672 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 23 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:24:23.784801 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:23.787250 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1161) took 1224 ms. Trace:
I20260709 12:24:23.791007 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:23.801388 32378 rpcz_store.cc:276] 0709 12:24:22.562780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.563855 (+  1075us) service_pool.cc:224] Handling call
0709 12:24:22.908743 (+344888us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:22.909055 (+   312us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:22.909082 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:22.909127 (+    45us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:22.909157 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:22.909219 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.786986 (+877767us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b54004c4350. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:23.787160 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2639,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"spinlock_wait_cycles":761649024}
I20260709 12:24:23.798750  7786 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.008s
I20260709 12:24:23.808171  7786 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:24:23.792060 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:23.811013 31791 leader_election.cc:400] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28.
I20260709 12:24:23.812278 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:23.813500  7792 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:23.814371  7792 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 24.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:23.815397  7792 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:23.815936  7792 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:23.817013  7792 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.792773  7927 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.799954 31674 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
I20260709 12:24:23.829649  7944 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:23.829943  7930 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
W20260709 12:24:23.831375  7460 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.899s	user 0.013s	sys 0.005s
I20260709 12:24:23.767015  7815 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:23.831828  7460 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.772719  7894 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:23.772492  7756 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:23.788625  7916 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.123s	user 0.001s	sys 0.004s
W20260709 12:24:23.876052 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.792090  7679 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:23.788938  7522 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:23.789642  7511 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:23.891094 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.798756  7813 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:23.963673  7813 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.791467  7853 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:23.800403  7733 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:23.969210 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 63585.5us
I20260709 12:24:23.832937  7915 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:23.984714  7915 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:23.984944  7853 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.903375  7756 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:23.987932  7932 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.381s	user 0.000s	sys 0.000s
I20260709 12:24:23.911557  7936 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:23.989041  7936 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:23.911890  7432 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.783s	user 0.011s	sys 0.004s
I20260709 12:24:23.990731  7432 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:23.993489  7432 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:23.852928  7926 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:24:23.889478  7570 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.023s
I20260709 12:24:23.781762  7940 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.191s
W20260709 12:24:23.912671  7942 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:24:23.997431 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3531) took 1042 ms. Trace:
I20260709 12:24:23.998281 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1327) took 1181 ms. Trace:
I20260709 12:24:23.888172  7799 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.116s
I20260709 12:24:23.997812 31744 rpcz_store.cc:276] 0709 12:24:22.955130 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.379410 (+424280us) service_pool.cc:224] Handling call
0709 12:24:23.379719 (+   309us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:23.380081 (+   362us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:23.380108 (+    27us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:23.380759 (+   651us) log.cc:844] Serialized 78 byte log entry
0709 12:24:23.555922 (+175163us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:23.833405 (+277483us) log.cc:844] Serialized 14 byte log entry
0709 12:24:23.834300 (+   895us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.834599 (+   299us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.997160 (+162561us) raft_consensus.cc:1685] finished
0709 12:24:23.997185 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.997362 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":348304,"prepare.run_cpu_time_us":28,"prepare.run_wall_time_us":29,"raft.queue_time_us":36994,"raft.run_cpu_time_us":339,"raft.run_wall_time_us":84726,"thread_start_us":429307,"threads_started":2,"wal-append.queue_time_us":371315}
I20260709 12:24:23.998973 31678 rpcz_store.cc:276] 0709 12:24:22.817091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.823331 (+  6240us) service_pool.cc:224] Handling call
0709 12:24:22.823799 (+   468us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:22.824302 (+   503us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:22.824347 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:22.825326 (+   979us) log.cc:844] Serialized 52 byte log entry
0709 12:24:23.027512 (+202186us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:23.032618 (+  5106us) log.cc:844] Serialized 14 byte log entry
0709 12:24:23.032874 (+   256us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.032945 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:23.998000 (+965055us) raft_consensus.cc:1685] finished
0709 12:24:23.998031 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:23.998203 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4295,"prepare.queue_time_us":4616,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":1850,"raft.run_cpu_time_us":207,"raft.run_wall_time_us":209,"thread_start_us":201732,"threads_started":1,"wal-append.queue_time_us":595949}
I20260709 12:24:23.795143  7590 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.000605 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:23.904167  7818 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.029s
W20260709 12:24:23.919731  7951 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:24:23.929381 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:23.929724  7955 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:24.055974  7800 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.997s	user 0.006s	sys 0.007s
I20260709 12:24:24.055738  7955 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:24.056659  7543 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.170s	user 0.016s	sys 0.003s
I20260709 12:24:24.058033  7955 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:23.932955 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:23.971606  7465 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.984s	user 0.009s	sys 0.005s
I20260709 12:24:23.941476  7905 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:23.941118  7779 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:23.943405  7798 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.593s	user 0.010s	sys 0.000s
W20260709 12:24:23.945860  7378 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.821s	user 0.002s	sys 0.010s
I20260709 12:24:23.946046  7889 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:23.946549  7755 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:23.958644  7750 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:23.969502 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:23.970780 31827 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:23.909582  7937 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:24:23.993110  7894 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:24.003482 32376 raft_consensus.cc:2368] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26.
W20260709 12:24:23.914672  7820 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.536s	user 0.008s	sys 0.000s
I20260709 12:24:24.011080 31822 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:24.014453  7799 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:24:24.012162 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.059676 32374 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
W20260709 12:24:24.076272 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 289462us
W20260709 12:24:24.104116  7921 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.048s	user 0.014s	sys 0.010s
I20260709 12:24:24.085587  7522 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.086210  7940 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:24.096006 31819 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
W20260709 12:24:24.017230  7365 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.091s	user 0.006s	sys 0.011s
W20260709 12:24:24.012574  7766 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.900s	user 0.007s	sys 0.006s
I20260709 12:24:24.103374 31821 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
W20260709 12:24:24.143810 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 385280us
I20260709 12:24:24.144649 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:24.104838 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.058708  7452 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:24.011596 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:24.170532 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1189) took 2096 ms. Trace:
I20260709 12:24:24.205462 31752 rpcz_store.cc:276] 0709 12:24:22.073840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.037799 (+963959us) service_pool.cc:224] Handling call
0709 12:24:23.603791 (+565992us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:23.661893 (+ 58102us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:24.170328 (+508435us) spinlock_profiling.cc:243] Waited 566 ms on lock 0x7b54000b90d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:24.170472 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1300907392}
I20260709 12:24:24.207034 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:24.158494 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:24.078233 31820 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 13. (index mismatch)
I20260709 12:24:24.208212 31747 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:24:24.146937  6752 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:24.172672 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.209738 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:24.210196 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.173492 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:24.175333 31826 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 23 index: 14. (index mismatch)
I20260709 12:24:24.176371 31745 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:24.177299 31670 raft_consensus.cc:2368] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25.
I20260709 12:24:24.177309 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:24.177947 31674 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:24:24.180023 32378 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29. Ops: [28.18-28.18]
I20260709 12:24:24.180421 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:24.214191 31744 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:24.182236 31669 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:24:24.195003 32371 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:24.195480 31748 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 17. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:24:24.195992 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:24.212102 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:24.187214 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:24.166432  7933 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.583s	user 0.007s	sys 0.001s
I20260709 12:24:24.060505  7905 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:24.200878 31746 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:24:24.143353  7755 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:24.177021  7894 consensus_queue.cc:1243] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 26.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:24:24.146106  7798 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 27. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28.
I20260709 12:24:24.209128  6752 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:24.252229  7779 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:24.253312 31818 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:24.253569 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2849) took 2283 ms. Trace:
I20260709 12:24:24.255026  7894 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:24.255589  7894 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.235937  7766 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.020s
I20260709 12:24:24.256054  7894 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:24.199563  7940 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:24.259974  7940 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:24.262080  7763 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.683s	user 0.007s	sys 0.003s
I20260709 12:24:24.262672  7763 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.176777  7820 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:24.267431  7756 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:24.268079  7820 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:24.268775 31825 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:24.270913  7815 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.221316 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:24.271394  7815 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:24.271634 31678 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:24.272518 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:24.273276  6752 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:24.148878  7813 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:24.179271  7921 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.123s	user 0.014s	sys 0.010s
I20260709 12:24:24.275308 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2854) took 1433 ms. Trace:
I20260709 12:24:24.276098  7957 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:24.224437 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 494749us
I20260709 12:24:24.224701 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:24.275698 31750 rpcz_store.cc:276] 0709 12:24:22.841602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.184269 (+342667us) service_pool.cc:224] Handling call
0709 12:24:23.229203 (+ 44934us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:23.237732 (+  8529us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:23.237765 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:23.282963 (+ 45198us) log.cc:844] Serialized 26 byte log entry
0709 12:24:23.569662 (+286699us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:23.569715 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.569794 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:24.275089 (+705295us) raft_consensus.cc:1685] finished
0709 12:24:24.275111 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:24.275248 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7883,"raft.queue_time_us":34957,"raft.run_cpu_time_us":216,"raft.run_wall_time_us":33752,"thread_start_us":197801,"threads_started":1,"wal-append.queue_time_us":411223}
I20260709 12:24:24.203080  7430 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.227021 31673 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:24.193006 31827 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:24.235558 31749 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
W20260709 12:24:24.236827 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 225357us
I20260709 12:24:24.279320 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:24.239130 31748 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:24:24.213833 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:24.181993  7945 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:24.146332  7378 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.254109 31753 rpcz_store.cc:276] 0709 12:24:21.969854 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.028359 (+1058505us) service_pool.cc:224] Handling call
0709 12:24:23.028766 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:23.120561 (+ 91795us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:23.120906 (+   345us) log.cc:844] Serialized 14 byte log entry
0709 12:24:23.666234 (+545328us) log.cc:844] Serialized 14 byte log entry
0709 12:24:23.666433 (+   199us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:23.944858 (+278425us) spinlock_profiling.cc:243] Waited 4.6 ms on lock 0x7b2c00001158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:24:23.945535 (+   677us) log.cc:844] Serialized 26 byte log entry
0709 12:24:23.945739 (+   204us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:23.945787 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:23.945887 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:24.084704 (+138817us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:24.253280 (+168576us) raft_consensus.cc:1685] finished
0709 12:24:24.253312 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:24.253493 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":388378,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":76,"raft.queue_time_us":131397,"raft.run_cpu_time_us":450,"raft.run_wall_time_us":26972,"spinlock_wait_cycles":33944192,"thread_start_us":521524,"threads_started":3,"wal-append.queue_time_us":422862}
I20260709 12:24:24.281853 31673 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:24:23.990042  7936 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:24.235276  7590 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:24.285261 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:24.283969  7452 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:24.193961  7522 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.291656  7831 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:24.292961  7522 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:24.293161  7831 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:24.290952  6752 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:24.283908  7813 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.295208  7813 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.295895  7452 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:24.297520 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.240823  7759 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:24.298030 31673 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:24.300359 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:24.303359 31820 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:24.311390 31676 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:24.289754  7465 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:24.276764  7957 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:24.323622 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1365) took 1051 ms. Trace:
I20260709 12:24:24.324250 32374 rpcz_store.cc:276] 0709 12:24:23.271972 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.272653 (+   681us) service_pool.cc:224] Handling call
0709 12:24:23.289943 (+ 17290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:24.169086 (+879143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:24.320388 (+151302us) spinlock_profiling.cc:243] Waited 769 ms on lock 0x7b5400499154. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:24.323525 (+  3137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1769404288}
W20260709 12:24:24.326937 32375 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.017s	sys 0.016s
W20260709 12:24:24.327389 32375 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.017s	sys 0.016s
W20260709 12:24:24.321077 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.009s	sys 0.022s
W20260709 12:24:24.329048 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.536s	user 0.010s	sys 0.022s
W20260709 12:24:24.329520 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 536269us
I20260709 12:24:24.307145  7759 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:24.330796 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:24.282845  7430 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.333940  6752 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:24.315603  7733 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:24.338670 31717 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:24.348162 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:24.271342  7905 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.349102 31744 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:24.352502 31673 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:24:24.301754  7945 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.334156  7759 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:24.360559 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:24.392943 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.023s	sys 0.033s
W20260709 12:24:24.393836 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.023s	sys 0.033s
I20260709 12:24:24.332417  7465 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:24.361867 31748 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:24.395417 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:24.354645 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:24.361419 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:24.391700 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:24.396690 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:24.341619  7460 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:24.359238  7927 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.014s	sys 0.024s
I20260709 12:24:24.401355  6752 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:24.333163  7766 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.038s
I20260709 12:24:24.414599 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:24.403535 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:24.406312 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:24.416293 31750 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:24.416599 31673 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:24.408080 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:24.409235 31671 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:24.417908 31745 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:24.410248 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:24.402333 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 617591us
I20260709 12:24:24.358579  7813 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:24.334324  7831 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.042s
I20260709 12:24:24.424377 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:24.433079 31744 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:24.332173  7955 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.031s
I20260709 12:24:24.359514  7365 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.440611 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:24.459581 31824 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: [22.14-26.16]
W20260709 12:24:24.424255  7295 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.003s	sys 0.031s
I20260709 12:24:24.388093  7763 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:24.354923  7820 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.082s
I20260709 12:24:24.367014  7756 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:24.409479  7905 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:24.469993 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:24.483716 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:24.413316  7927 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.590s	user 0.014s	sys 0.024s
I20260709 12:24:24.401687  7890 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:24.477056 31673 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:24.437554  7889 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:24.470930 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:24.479543  7295 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.004s	sys 0.031s
I20260709 12:24:24.488969 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:24.411720  7779 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:24.458523  7522 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.484473 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:24.495827 31673 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:24.435175  7767 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.057s
I20260709 12:24:24.411331  7511 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.074s
W20260709 12:24:24.502130 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 268043us
I20260709 12:24:24.411521  7940 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.075s
I20260709 12:24:24.468127  7893 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:24.510932 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:24.513922 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:24.508848 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:24.519462 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2841) took 2553 ms. Trace:
I20260709 12:24:24.524577 31751 rpcz_store.cc:276] 0709 12:24:21.966489 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:22.348491 (+382002us) service_pool.cc:224] Handling call
0709 12:24:22.794062 (+445571us) spinlock_profiling.cc:243] Waited 14 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:24:22.794785 (+   723us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:24.200337 (+1405552us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:24.200368 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:24.200408 (+    40us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:24.200437 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:24.519130 (+318693us) spinlock_profiling.cc:243] Waited 1.41 s on lock 0x7b54001a59d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:24.519208 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:24.519381 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3145,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":24120,"spinlock_wait_cycles":3264909824}
I20260709 12:24:24.525916 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:24.517077 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.492777  7782 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:24:24.528615 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:24.498358  7763 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.488147  7890 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.477708  7733 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 12:24:24.501807  7779 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.468070  7365 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:24.485356  7947 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 17, Time since last communication: 0.061s
I20260709 12:24:24.467615  7959 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.062s
I20260709 12:24:24.432401  7465 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.464568  7861 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:24.573616 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47764.3us
I20260709 12:24:24.486443  7756 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.522048  7570 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:24.531082  7962 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.540791  7378 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:24.571913  7893 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:24.586372  7763 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.514811  7522 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:24.709580  7763 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:24.586673  7890 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:24.711282  7970 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
W20260709 12:24:24.655268 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.715655  7889 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:24.614244  7902 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:24.716125  7889 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:24.716967  7889 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.718142  7889 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:24.719257 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3614) took 1092 ms. Trace:
I20260709 12:24:24.719919  7927 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 23.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:24.720450 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1484) took 1246 ms. Trace:
I20260709 12:24:24.720811  7927 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:24.720860 31825 rpcz_store.cc:276] 0709 12:24:23.473817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.054990 (+581173us) service_pool.cc:224] Handling call
0709 12:24:24.055348 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:24.269169 (+213821us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:24.720125 (+450956us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x7b540043cbd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:24.720385 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":490393344}
I20260709 12:24:24.722785  7960 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:24.723187  7960 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:24.724069  7960 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.725246  7960 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:24.726230 32376 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 1363) took 1460 ms (client timeout 1598 ms). Trace:
I20260709 12:24:24.726296  7894 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:24.726816 32376 rpcz_store.cc:269] 0709 12:24:23.265713 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.266380 (+   667us) service_pool.cc:224] Handling call
0709 12:24:24.726035 (+1459655us) spinlock_profiling.cc:243] Waited 736 ms on lock 0x7b540031d654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:24.726172 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1692257536}
I20260709 12:24:24.728016  7522 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:24.728672  7915 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.743s	user 0.009s	sys 0.003s
I20260709 12:24:24.728724  7894 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:24.730122  7966 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.146s	user 0.000s	sys 0.000s
I20260709 12:24:24.706522  7570 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:24.586221  7956 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:24.735967 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:24.756095 31824 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:24.719697 31670 rpcz_store.cc:276] 0709 12:24:23.626608 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.626896 (+   288us) service_pool.cc:224] Handling call
0709 12:24:24.719077 (+1092181us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b540017c0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:24.719195 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1152902400}
I20260709 12:24:24.759146 31825 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
W20260709 12:24:24.761272 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46418.7us
I20260709 12:24:24.737618 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:24.741576 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.615466  7779 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:24.764927 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:24.764227 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:24.767287 31824 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:24.773422  7969 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 12:24:24.735095  7957 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:24.778888  7927 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:24.779340  7945 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:24.779752 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:24.780404  7972 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
I20260709 12:24:24.781235  7365 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:24.763923  7831 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.019s
I20260709 12:24:24.783341 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 1485) took 1217 ms. Trace:
I20260709 12:24:24.783870 31827 rpcz_store.cc:276] 0709 12:24:23.566454 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.576839 (+ 10385us) service_pool.cc:224] Handling call
0709 12:24:24.783144 (+1206305us) spinlock_profiling.cc:243] Waited 393 ms on lock 0x7b5400197dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:24.783270 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":903091968}
I20260709 12:24:24.755389  7915 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.787272  7915 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:24.789630  7915 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:24.762143  7956 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.790524  7956 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:24.792042 31818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3570) took 1501 ms (client timeout 1753 ms). Trace:
W20260709 12:24:24.792603 31818 rpcz_store.cc:269] 0709 12:24:23.290667 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.290834 (+   167us) service_pool.cc:224] Handling call
0709 12:24:24.791805 (+1500971us) spinlock_profiling.cc:243] Waited 940 ms on lock 0x7b540030e8d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:24.791958 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2161076096}
I20260709 12:24:24.779423  7927 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:24.797401 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:24.768498 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:24.801146 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:24.801370 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:24.801640 31825 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:24.809216 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 71681.1us
I20260709 12:24:24.818352 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:24.820640 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:24.820672 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 415209us
I20260709 12:24:24.762863  7861 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:24.824344 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 485748us
I20260709 12:24:24.816155 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:24.827708 31676 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:24:24.760458  7936 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.830206 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:24.831082 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:24.835983 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:24.805918  7927 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:24.838376 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:24.826800  7894 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:24.848829 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:24.849473 31670 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:24.840359 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:24.834961  7966 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
I20260709 12:24:24.858698 31824 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:24.867893 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:24.868253 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:24.873122 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:24.852324 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:24.878576 31676 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
W20260709 12:24:24.847461  7750 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.681s	user 0.010s	sys 0.002s
I20260709 12:24:24.881646 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:24.821223  7902 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:24:24.886010 31670 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:24.821488  7890 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:24.763514  7962 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:24.823467  7915 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.033s
I20260709 12:24:24.882393 32375 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:24.847863  7894 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.887096 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:24.882923  7782 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:24.888335 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:24.841265  7936 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:24.901607 32371 raft_consensus.cc:2368] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27.
I20260709 12:24:24.884981 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:24.853343  7831 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.853274  7965 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:24.847476  7365 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:24.867516  7805 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:24.891752 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:24.892227 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:24.892762 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3435) took 1438 ms. Trace:
I20260709 12:24:24.885532  7570 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:24.893337 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:24.896320 31672 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:24.882815  7909 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:24.899536 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:24.887523 31818 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:24.904093 31673 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:24:24.905660 32342 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:24.891750  7945 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:24:24.974260 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 68687.3us
I20260709 12:24:24.981815 31669 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:24.901314  7782 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:24.900731  7894 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:24.979373  7759 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.538s	user 0.010s	sys 0.000s
I20260709 12:24:24.979113  7763 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:24.907378  7902 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:24.900228  7295 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.100s
W20260709 12:24:25.003532 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 104083us
I20260709 12:24:24.989401  7805 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.033628 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:24.999946  7452 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:24.994971  7365 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:24.912539  7800 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.116s
I20260709 12:24:24.898912  7799 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.095s
W20260709 12:24:25.051030 32377 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.736s	user 0.009s	sys 0.009s
I20260709 12:24:24.908618  7831 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.043613 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:25.027542  7944 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:24.986104  7570 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.053s
I20260709 12:24:24.969154 31823 rpcz_store.cc:276] 0709 12:24:23.454811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.462733 (+  7922us) service_pool.cc:224] Handling call
0709 12:24:23.463203 (+   470us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:23.497220 (+ 34017us) raft_consensus.cc:1534] Early marking committed up to index 10
0709 12:24:23.754854 (+257634us) log.cc:844] Serialized 14 byte log entry
0709 12:24:24.221564 (+466710us) log.cc:844] Serialized 14 byte log entry
0709 12:24:24.221818 (+   254us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:24.240375 (+ 18557us) log.cc:844] Serialized 52 byte log entry
0709 12:24:24.240564 (+   189us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:24.243119 (+  2555us) log.cc:844] Serialized 14 byte log entry
0709 12:24:24.243388 (+   269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:24.243471 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:24.708220 (+464749us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b18004386a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:24.891097 (+182877us) spinlock_profiling.cc:243] Waited 5.67 ms on lock 0x7b18004386a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:24.892560 (+  1463us) raft_consensus.cc:1685] finished
0709 12:24:24.892581 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:24.892705 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17192,"prepare.queue_time_us":317284,"prepare.run_cpu_time_us":121,"prepare.run_wall_time_us":124,"raft.queue_time_us":33518,"raft.run_cpu_time_us":516,"raft.run_wall_time_us":19006,"spinlock_wait_cycles":39164160,"thread_start_us":629017,"threads_started":2,"wal-append.queue_time_us":726012}
W20260709 12:24:25.055452 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 163303us
W20260709 12:24:25.052035 32377 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.745s	user 0.009s	sys 0.009s
I20260709 12:24:25.027455  7782 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:25.039915  7805 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:25.084430 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.007s	sys 0.013s
I20260709 12:24:25.080868 31717 leader_election.cc:400] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25.
W20260709 12:24:25.051419  7979 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.169s	user 0.001s	sys 0.000s
I20260709 12:24:25.056494 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:25.045761  7915 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:25.035755  7759 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.046464  7522 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.092128 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:25.104573 31672 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:25.096885 31670 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:25.100042 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:25.071975  7957 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:25.112093 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.008s	sys 0.013s
I20260709 12:24:25.114280 31825 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:25.052006  7452 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:25.036037  7763 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.080371  7984 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:25.139473 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:25.155637 31670 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:25.106851  7990 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:25.080293  7813 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:25.050980  7983 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.067s
I20260709 12:24:25.142146 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:25.103421  7750 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:24:25.111493  7798 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.649s	user 0.007s	sys 0.013s
W20260709 12:24:25.111634  7853 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.835s	user 0.005s	sys 0.017s
I20260709 12:24:25.085832  7945 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.053s
W20260709 12:24:25.159063 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 358675us
W20260709 12:24:25.140370 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 669492us
I20260709 12:24:25.180274 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3438) took 1724 ms. Trace:
W20260709 12:24:25.150254  7976 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.260s	user 0.000s	sys 0.001s
W20260709 12:24:25.179551 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 211696us
I20260709 12:24:25.188011 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2702) took 1074 ms. Trace:
I20260709 12:24:25.180820 31822 rpcz_store.cc:276] 0709 12:24:23.455730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.455861 (+   131us) service_pool.cc:224] Handling call
0709 12:24:23.497227 (+ 41366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:24.167222 (+669995us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:25.179926 (+1012704us) spinlock_profiling.cc:243] Waited 507 ms on lock 0x7b54001c89d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:25.180188 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1164973824}
I20260709 12:24:25.188479 31678 rpcz_store.cc:276] 0709 12:24:24.113491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.272479 (+158988us) service_pool.cc:224] Handling call
0709 12:24:24.272801 (+   322us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:24.273006 (+   205us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:24.273031 (+    25us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:24.273532 (+   501us) log.cc:844] Serialized 26 byte log entry
0709 12:24:24.584391 (+310859us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:24.584430 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:24.584496 (+    66us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:25.187773 (+603277us) raft_consensus.cc:1685] finished
0709 12:24:25.187796 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:25.187945 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":309069,"threads_started":1,"wal-append.queue_time_us":558308}
W20260709 12:24:25.169277  7815 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.897s	user 0.012s	sys 0.001s
I20260709 12:24:25.111982  7522 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.153370  7944 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:25.151851  7992 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:25.180056  7798 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.734s	user 0.008s	sys 0.018s
I20260709 12:24:25.167270  7995 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:24:25.179682  7853 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.903s	user 0.006s	sys 0.017s
I20260709 12:24:25.179239 31565 maintenance_manager.cc:419] P ac4fa664fb214bda87cd96abb2116a4b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033396
I20260709 12:24:25.194716 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:25.192108  7978 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.302s	user 0.001s	sys 0.000s
I20260709 12:24:25.110459  7365 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.182804  7890 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:25.157972  7570 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.051s
I20260709 12:24:25.149158  7957 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.192303  7962 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:25.231513 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1369) took 1464 ms. Trace:
W20260709 12:24:25.245904  7987 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260709 12:24:25.239054 31821 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3436) took 1783 ms (client timeout 1566 ms). Trace:
I20260709 12:24:25.240947 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3617) took 1484 ms. Trace:
I20260709 12:24:25.214246  7936 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:25.214733  7981 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.244s	user 0.001s	sys 0.001s
I20260709 12:24:25.238022  7894 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.237093  7947 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
I20260709 12:24:25.203359  7984 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:25.235668  7982 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.249s	user 0.002s	sys 0.001s
W20260709 12:24:25.250037 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.174654  7990 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.281111 31671 rpcz_store.cc:276] 0709 12:24:23.756872 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.759791 (+  2919us) service_pool.cc:224] Handling call
0709 12:24:23.781179 (+ 21388us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:24.417635 (+636456us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:25.240718 (+823083us) spinlock_profiling.cc:243] Waited 628 ms on lock 0x7b54001d6354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:25.240895 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1443803520}
W20260709 12:24:25.259826 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 65213us
I20260709 12:24:25.295212 31792 leader_election.cc:400] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27.
I20260709 12:24:25.295589 31792 leader_election.cc:403] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Cancelling election due to peer responding with higher term
I20260709 12:24:25.207911  7995 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:25.231837  7991 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
I20260709 12:24:25.278767 32377 rpcz_store.cc:276] 0709 12:24:23.766982 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.767159 (+   177us) service_pool.cc:224] Handling call
0709 12:24:23.767502 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:24.202414 (+434912us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:24.202446 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:24.203383 (+   937us) log.cc:844] Serialized 26 byte log entry
0709 12:24:25.079725 (+876342us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:25.079792 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:25.079898 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:25.231185 (+151287us) raft_consensus.cc:1685] finished
0709 12:24:25.231208 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:25.231425 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":224933,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":109,"thread_start_us":1207332,"threads_started":2,"wal-append.queue_time_us":886721}
W20260709 12:24:25.280910 31821 rpcz_store.cc:269] 0709 12:24:23.455151 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:23.455279 (+   128us) service_pool.cc:224] Handling call
0709 12:24:25.238883 (+1783604us) spinlock_profiling.cc:243] Waited 647 ms on lock 0x7b54001c93d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:25.238988 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1488146304}
I20260709 12:24:25.208803  7944 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:25.315901  7996 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.113s	user 0.000s	sys 0.002s
I20260709 12:24:25.263020  8000 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.061s
I20260709 12:24:25.300115  7766 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:25.206034  7815 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.260741  7543 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:25.226619  7831 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:25.266829  7960 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.541s	user 0.011s	sys 0.005s
W20260709 12:24:25.284734  7779 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.509s	user 0.011s	sys 0.001s
I20260709 12:24:25.265765  7986 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:24:25.281157  7985 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.223s	user 0.001s	sys 0.000s
I20260709 12:24:25.280880  7813 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:25.209548  7992 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.327343  7965 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:25.283424 32342 leader_election.cc:336] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.501s (SENT)
I20260709 12:24:25.308426  7800 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27.
I20260709 12:24:25.263223  7452 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:25.339499 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 109943us
W20260709 12:24:25.343556 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 228974us
I20260709 12:24:25.346083 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:25.307715  7805 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.333114  7543 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:25.353585 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 70233.7us
I20260709 12:24:25.338889  7759 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:25.358368 32342 leader_election.cc:336] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.524s (SENT)
I20260709 12:24:25.300544  7984 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.342404  7986 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:25.364096 32342 leader_election.cc:336] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.587s (SENT)
I20260709 12:24:25.364625 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:25.365860 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 189128us
W20260709 12:24:25.357797  7905 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.868s	user 0.004s	sys 0.011s
I20260709 12:24:25.371157 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:25.345516  7813 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:25.356755  7956 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.566s	user 0.009s	sys 0.005s
I20260709 12:24:25.360204  8010 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:25.357959  7799 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.379428 32377 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 26 index: 19. (index mismatch)
I20260709 12:24:25.363128  8012 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.342875  7779 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.376855 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2865) took 1035 ms. Trace:
I20260709 12:24:25.387413 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:25.379089 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:25.383699 32342 leader_election.cc:336] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.635s (SENT)
I20260709 12:24:25.385011 31822 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:25.380347 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2866) took 1038 ms. Trace:
I20260709 12:24:25.367560  7986 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:25.390753 32377 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: [25.13-25.13]
I20260709 12:24:25.387797 31753 rpcz_store.cc:276] 0709 12:24:24.341903 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.342372 (+   469us) service_pool.cc:224] Handling call
0709 12:24:24.342738 (+   366us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:24.355173 (+ 12435us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:24.355206 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:24.355644 (+   438us) log.cc:844] Serialized 26 byte log entry
0709 12:24:24.754314 (+398670us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:24.754368 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:24.754453 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:25.376550 (+622097us) raft_consensus.cc:1685] finished
0709 12:24:25.376582 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:25.376773 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13370,"raft.queue_time_us":12147,"raft.run_cpu_time_us":225,"raft.run_wall_time_us":31554,"thread_start_us":342060,"threads_started":1,"wal-append.queue_time_us":617848}
I20260709 12:24:25.389310 31747 rpcz_store.cc:276] 0709 12:24:24.342255 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.343056 (+   801us) service_pool.cc:224] Handling call
0709 12:24:24.343348 (+   292us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:24.364278 (+ 20930us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:24.364305 (+    27us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:24.367629 (+  3324us) log.cc:844] Serialized 26 byte log entry
0709 12:24:24.839164 (+471535us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:25.076221 (+237057us) log.cc:844] Serialized 14 byte log entry
0709 12:24:25.295458 (+219237us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:25.295572 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:25.380140 (+ 84568us) raft_consensus.cc:1685] finished
0709 12:24:25.380161 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:25.380289 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20615,"prepare.queue_time_us":363246,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":90192,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":19924,"thread_start_us":766413,"threads_started":3,"wal-append.queue_time_us":639752}
I20260709 12:24:25.398854 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:25.391374 31823 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:25.368618  7921 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:25.384649  7889 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.666s	user 0.013s	sys 0.003s
I20260709 12:24:25.452906 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:25.444186 32376 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:25.447517 31671 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:25.448185 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:25.455636 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:25.448423 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:25.449892 32342 leader_election.cc:336] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.731s (SENT)
I20260709 12:24:25.453701 32375 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 16.
I20260709 12:24:25.456966 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:25.465474 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:25.466446 32373 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:25.468595 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:25.460500 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:25.469549 31678 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:25.461620 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.475065 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:25.388120  7522 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:25.477839 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:25.479722 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:25.385774  7759 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.087s
I20260709 12:24:25.347146  7995 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.395016  8011 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:25.487432 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:25.492731 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 116728us
I20260709 12:24:25.494231 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:25.395823  7799 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:25.458417  7543 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.496531 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:25.500306 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:25.384532  7956 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:25.487115  8005 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.133s	user 0.000s	sys 0.001s
I20260709 12:24:25.486291  7962 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:25.506352  7767 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.009s
I20260709 12:24:25.516494  7767 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:25.489552  7853 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:25.497496  7511 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.518898  7853 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.519143  7990 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:25.519186  7511 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.520040  7990 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:25.520465  7990 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:25.521828  7853 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:25.522328  7511 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:25.481284  7813 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.443425  7965 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.523191  7992 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:25.523412  7813 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:25.528312  7799 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:25.529755  7815 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:25.531050  7905 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:25.492139  7925 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:25.532330  7905 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:25.533375  7905 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:25.533479  7805 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:25.521421  7990 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.544240 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:25.545611 31672 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:25.546270 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:25.513767  7944 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:25.537384  8013 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:25.494699  7522 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.547791  7522 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.548864  7963 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:25.549329  7963 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:25.550706  7543 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:25.560956 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2868) took 1216 ms. Trace:
I20260709 12:24:25.561426 31749 rpcz_store.cc:276] 0709 12:24:24.344241 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.344407 (+   166us) service_pool.cc:224] Handling call
0709 12:24:24.356712 (+ 12305us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:24.766336 (+409624us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:24.766376 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:24.806267 (+ 39891us) log.cc:844] Serialized 26 byte log entry
0709 12:24:25.195430 (+389163us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:25.195471 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:25.195556 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:25.560689 (+365133us) raft_consensus.cc:1685] finished
0709 12:24:25.560712 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:25.560891 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2885,"raft.queue_time_us":39125,"raft.run_cpu_time_us":45,"raft.run_wall_time_us":45,"thread_start_us":704609,"threads_started":2,"wal-append.queue_time_us":495054}
I20260709 12:24:25.563259 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:25.564149 32371 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
W20260709 12:24:25.538877  8014 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.094s	user 0.001s	sys 0.000s
W20260709 12:24:25.539806  7975 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.648s	user 0.002s	sys 0.002s
I20260709 12:24:25.540573 32374 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:24:25.541874  7956 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:25.492954  7902 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:25.522392  7679 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:25.545508  8022 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.536574  7984 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:25.538206  8006 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.199s	user 0.000s	sys 0.001s
I20260709 12:24:25.536721 31678 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 22.
I20260709 12:24:25.556105 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:25.570477  7927 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.704s	user 0.008s	sys 0.005s
I20260709 12:24:25.579453 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:25.558007 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:25.424233  8012 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:25.509047  8009 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:25.513607  7962 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:25.502864  7798 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:25.566362 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:25.534348  7905 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.582125  7756 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:25.582386  7956 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:25.591084  7756 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:25.591226  7956 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:25.587260 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3581) took 1821 ms. Trace:
I20260709 12:24:25.592430 31826 rpcz_store.cc:276] 0709 12:24:23.766177 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.226461 (+460284us) service_pool.cc:224] Handling call
0709 12:24:24.226901 (+   440us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:24.227130 (+   229us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:24.227159 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:24.227649 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 12:24:24.622871 (+395222us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:24.622939 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:24.623046 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:25.510025 (+886979us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:25.586852 (+ 76827us) raft_consensus.cc:1685] finished
0709 12:24:25.586876 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:25.587180 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3986304,"thread_start_us":393807,"threads_started":1,"wal-append.queue_time_us":647036}
I20260709 12:24:25.550323  7522 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:25.570951 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 73176.2us
I20260709 12:24:25.514407  7995 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:25.459998  7921 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.577749 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:25.550230  7963 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.570081  8010 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:25.557670  8002 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.045s
I20260709 12:24:25.591605  7756 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:25.594782  8009 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:25.584374  7779 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:25.514122  7831 consensus_queue.cc:1175] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b95aeeaf4c51442b8092cff1de674f2b" responder_term: 26 status { last_received { term: 25 index: 18 } last_committed_idx: 18 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:25.594967  7956 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.13, Last appended by leader: 13, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:25.596397 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.594696  7962 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:25.572244  7818 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:25.575137  7805 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:25.630734  7805 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:25.633762 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:25.612989  7861 consensus_meta.cc:287] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.783s	user 0.013s	sys 0.002s
I20260709 12:24:25.604964  7522 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:25.580672  8022 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.586131  8012 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:25.612583 32375 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:25.611824  7905 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:25.622990 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.624114  7921 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:25.612515  7963 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:25.610746  7756 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:25.626010  8017 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
I20260709 12:24:25.601446 31643 leader_election.cc:400] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26.
I20260709 12:24:25.637132 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:25.637457 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:25.672150 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:25.638978 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 17 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:25.639256 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:25.643405 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:25.674196 31826 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:25.646359  6752 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:25.674675 31824 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:25.674968  6752 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.647166  8009 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.633841  7805 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:25.650669  8022 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:25.593679  7927 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.659732 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:25.677520 31643 leader_election.cc:403] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:25.681254 31819 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:24:25.682955 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:25.693171 31827 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
W20260709 12:24:25.700627 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 99307.3us
I20260709 12:24:25.703493 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:25.705374 31827 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:25.711324 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:25.713093 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:25.718129 32375 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:25.714685 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:25.722298 31826 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:25.739090 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:25.644330  7818 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.707681  8012 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:25.704531  7957 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:25.707142  7893 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 25. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26.
I20260709 12:24:25.707254  7861 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:25.735291  7992 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:25.705020  7800 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 20.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.049s
I20260709 12:24:25.750586  7889 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.004s
I20260709 12:24:25.752420  7940 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:25.750828  7959 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:25.715315  7805 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:24:25.753141  8012 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:25.708238  8025 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.079s
I20260709 12:24:25.792088  8002 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:25.790273  7940 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:25.790519  7959 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:25.819060  8012 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:25.815258  7522 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:25.855233  8016 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:25.867362  7959 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:25.852044  7940 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:25.882836 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.774061  7861 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:25.913985 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3449) took 1114 ms. Trace:
I20260709 12:24:25.821044  7805 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Stepping down as leader of term 21
W20260709 12:24:25.903527 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 196864us
I20260709 12:24:25.902554  8016 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:25.914471 31818 rpcz_store.cc:276] 0709 12:24:24.799793 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.820316 (+ 20523us) service_pool.cc:224] Handling call
0709 12:24:25.913805 (+1093489us) spinlock_profiling.cc:243] Waited 65.9 ms on lock 0x7b5400207bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:25.913923 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":151653120}
I20260709 12:24:25.847400  8002 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:25.991030 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:25.903252  7983 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:25.986362  8016 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:26.032927  7590 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:26.058568  8030 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.209s	user 0.000s	sys 0.000s
I20260709 12:24:25.920822  7944 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:25.920529  7805 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:25.906047  7522 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.048867  7959 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:26.065244  7782 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.965s	user 0.006s	sys 0.008s
I20260709 12:24:26.037807  7962 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:26.085355 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.068086  7590 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:26.111371  7995 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.509s	user 0.008s	sys 0.003s
I20260709 12:24:26.111793  7818 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:26.140105 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:26.111835 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 196605us
I20260709 12:24:26.063012  8018 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:26.179155  8018 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.181921  8018 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:26.176419 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:26.185817 32376 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:26.187249 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.188344 31825 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: [23.14-23.14]
I20260709 12:24:26.177137 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:26.190873 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.193931 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:26.182451  8018 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.060830  7983 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.200614 31825 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:26.201525 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.205931 31818 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:26.202306 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.206578 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:26.207111 31826 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:26.204320 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:26.066648  7944 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:26.192507  7905 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.539s	user 0.010s	sys 0.000s
I20260709 12:24:26.217561 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:26.218919 31827 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:26.215811 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.185396  7927 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:26.194978  7909 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.175s	user 0.003s	sys 0.022s
I20260709 12:24:26.199275  7936 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:26.227695 31792 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:26.084050  8002 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.208305  8025 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.022s
W20260709 12:24:26.237733  7909 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.224s	user 0.004s	sys 0.022s
I20260709 12:24:26.205307  7779 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:26.252832 31823 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:26.241585  7909 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:26.317600  7570 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:26.320580  7570 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:26.321023  7570 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:26.320905  7909 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.196801  7805 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.257432  7522 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:26.257503 31564 tablet_replica.cc:1406] Time spent applying in-flights took a long time: real 0.403s	user 0.000s	sys 0.000s
I20260709 12:24:26.264257  7590 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:26.266449  7944 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:26.152938  7782 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.211951  7963 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 13, Time since last communication: 0.067s
W20260709 12:24:26.311806 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 84197.1us
W20260709 12:24:26.291119 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:26.343788 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.317070  6752 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:26.319214 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.213693  8018 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.333549  7927 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:26.338835 32377 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.016s	sys 0.026s
W20260709 12:24:26.373728 32377 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.016s	sys 0.026s
W20260709 12:24:26.325033  7986 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.872s	user 0.012s	sys 0.003s
I20260709 12:24:26.307585  7936 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:24:26.277006  7813 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.010s	sys 0.024s
W20260709 12:24:26.372269 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.375603 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:26.331125  8009 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.579s	user 0.053s	sys 0.069s
W20260709 12:24:26.398500  8009 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s	user 0.054s	sys 0.070s
I20260709 12:24:26.270615  7905 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:26.380877 31827 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:24:26.393728 31825 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: [26.15-26.15]
W20260709 12:24:26.400159  7813 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.645s	user 0.010s	sys 0.025s
I20260709 12:24:26.348924  7590 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.379593  7805 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:26.371593  8036 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:26.372427  7861 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:26.401531  7905 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.438845  8036 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.441593  8036 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:26.442176 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1343) took 1619 ms. Trace:
I20260709 12:24:26.442564 31674 rpcz_store.cc:276] 0709 12:24:24.822955 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.850183 (+ 27228us) service_pool.cc:224] Handling call
0709 12:24:24.879799 (+ 29616us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:24.880160 (+   361us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:24.880200 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:24.880690 (+   490us) log.cc:844] Serialized 26 byte log entry
0709 12:24:25.915550 (+1034860us) spinlock_profiling.cc:243] Waited 2.12 ms on lock 0x7b180001a3e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:24:26.339298 (+423748us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:26.339343 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.339422 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:26.441887 (+102465us) raft_consensus.cc:1685] finished
0709 12:24:26.441907 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:26.442115 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19416,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":5459,"spinlock_wait_cycles":4884224,"thread_start_us":423534,"threads_started":1,"wal-append.queue_time_us":1548736}
I20260709 12:24:26.443717 31672 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
W20260709 12:24:26.451022 31642 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.513s (SENT) [suppressed 23 similar messages]
I20260709 12:24:26.420274  7955 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:26.423025  7936 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:26.394443  7986 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.461611  7986 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:26.461812 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:26.463598  7936 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.431541  7430 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:26.467301  7894 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:26.454998  7962 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:26.472513  7430 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.473300  7430 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:26.476209 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.478801  7820 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.479473  7820 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.482179  7820 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:26.483068 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.484406 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.485167 31827 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:26.486181 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.487208 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.487407 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:26.487951 32370 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:26.488266 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.382961  7927 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.513571 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.514575 31823 raft_consensus.cc:2368] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21.
I20260709 12:24:26.515609 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 22 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:26.516664 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:26.517370 31818 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:26.275234  7755 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:26.487017  7756 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.805s	user 0.013s	sys 0.000s
I20260709 12:24:26.529958  7733 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:26.539331  7905 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:26.540017  7766 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.540869  7766 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.542604  7990 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:26.543545  8000 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.546411  8000 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.519973 31819 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:26.552732  7983 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:26.556401  7994 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.365s	user 0.000s	sys 0.019s
I20260709 12:24:26.559615 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3443) took 2387 ms. Trace:
I20260709 12:24:26.523490  8045 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:26.560626  8045 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.561549 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:26.522513 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.486621  7820 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.489989  7799 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:26.563537 31827 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:26.560375 31820 rpcz_store.cc:276] 0709 12:24:24.171992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.305009 (+133017us) service_pool.cc:224] Handling call
0709 12:24:24.305418 (+   409us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:24.309681 (+  4263us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:24.328922 (+ 19241us) log.cc:844] Serialized 14 byte log entry
0709 12:24:26.165176 (+1836254us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x7b1800462220. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:24:26.391411 (+226235us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:26.392101 (+   690us) log.cc:844] Serialized 26 byte log entry
0709 12:24:26.392298 (+   197us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:26.392349 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.392450 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:26.559159 (+166709us) raft_consensus.cc:1685] finished
0709 12:24:26.559311 (+   152us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:26.559520 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":10975,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":5077,"raft.queue_time_us":763,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":12513,"spinlock_wait_cycles":3291904,"thread_start_us":225785,"threads_started":1,"wal-append.queue_time_us":2168540}
I20260709 12:24:26.571326 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:26.572219 31827 raft_consensus.cc:2410] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 25 because replica has last-logged OpId of term: 24 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 15.
I20260709 12:24:26.564499 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:26.564774 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:26.573948 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.577802 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:26.578320 31819 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:26.447031  7590 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.509192  7861 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:26.510180  7962 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:26.584996  8041 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:26.570251  7733 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.577013  7756 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.600277  7756 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:26.603139  7460 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.603987  7460 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.604434 31818 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:24:26.608100 31827 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:26.524137  7944 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.591063  7766 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:26.589059  7782 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:26.583659  7820 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.585141  7799 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.611024  7861 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.614836  7962 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.619765  7944 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.649173  7927 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:26.648957  8000 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:26.659576  8018 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:26.681887  8012 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.775s	user 0.013s	sys 0.000s
I20260709 12:24:26.726480 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2698) took 2614 ms. Trace:
I20260709 12:24:26.727420 31675 rpcz_store.cc:276] 0709 12:24:24.111944 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.112064 (+   120us) service_pool.cc:224] Handling call
0709 12:24:24.168685 (+ 56621us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:24.294086 (+125401us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:24.508730 (+214644us) log.cc:844] Serialized 14 byte log entry
0709 12:24:24.977316 (+468586us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:26.459736 (+1482420us) spinlock_profiling.cc:243] Waited 4.59 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:24:26.460211 (+   475us) log.cc:844] Serialized 26 byte log entry
0709 12:24:26.460355 (+   144us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:26.460391 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.461482 (+  1091us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54002b0554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:26.461534 (+    52us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:26.555802 (+ 94268us) spinlock_profiling.cc:243] Waited 3.69 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:26.725456 (+169654us) raft_consensus.cc:1685] finished
0709 12:24:26.725547 (+    91us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:26.726079 (+   532us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":39088,"prepare.queue_time_us":459087,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":53,"raft.queue_time_us":98239,"raft.run_cpu_time_us":319,"raft.run_wall_time_us":14994,"spinlock_wait_cycles":306538496,"thread_start_us":612571,"threads_started":2,"wal-append.queue_time_us":610210}
W20260709 12:24:26.730942 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.018s	sys 0.025s
W20260709 12:24:26.731566 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.521s	user 0.018s	sys 0.025s
W20260709 12:24:26.733985 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 527421us
W20260709 12:24:26.745870 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.751092 31824 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:26.760048 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.770115 32377 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:26.772305 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.775815  7430 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:26.776953  7430 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:26.777166 32377 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:24:26.634362  7460 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:26.780335  7465 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.585s	user 0.016s	sys 0.012s
W20260709 12:24:26.780958  7465 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.601s	user 0.016s	sys 0.013s
I20260709 12:24:26.788452  7590 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:26.818809 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.820168 31818 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:26.823055  7799 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:26.825361  7861 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:26.825114  8054 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.778121  7430 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.823237  8053 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:26.861920  8000 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:26.868340  7959 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.869405  7959 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:26.873669  8016 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.804s	user 0.012s	sys 0.000s
I20260709 12:24:26.874812  7893 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.018s
W20260709 12:24:26.879076 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 427902us
I20260709 12:24:26.879380  7679 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.880553  7679 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.882831  7782 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.883581  8053 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.888041  7944 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:26.874444  8016 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.889331  8018 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:26.817780  8012 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.892710  8012 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:26.895781 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:26.883809  7782 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.890081  8018 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:26.916376  7940 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.872s	user 0.013s	sys 0.000s
I20260709 12:24:26.917058  7959 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:26.919521  7820 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:26.920949 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:26.922109  8000 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:26.922713  7959 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:26.923151  7893 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 22.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:26.923594  7959 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.924294  7893 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:26.924721  7893 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:26.927217  7959 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:26.927405  7679 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:26.922672  8000 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:26.950870  8000 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:26.946045  8053 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:26.917127  7940 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:26.922465  7570 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.601s	user 0.009s	sys 0.008s
I20260709 12:24:26.974556  7570 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.978168  7570 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:26.978912  8059 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.979429  7962 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:26.979645  8054 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:26.984635  7465 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:26.987190 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:26.988822 31564 maintenance_manager.cc:643] P ac4fa664fb214bda87cd96abb2116a4b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 1.663s	user 0.033s	sys 0.027s Metrics: {"bytes_written":4759,"delete_count":0,"lbm_write_time_us":871,"lbm_writes_lt_1ms":5,"reinsert_count":0,"update_count":178}
I20260709 12:24:26.926158  7893 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:26.991904 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45530.4us
W20260709 12:24:26.991991 31669 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1340) took 2174 ms (client timeout 1944 ms). Trace:
I20260709 12:24:26.932593  8016 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:26.994177 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1341) took 2172 ms. Trace:
I20260709 12:24:26.995483  7992 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.003443  7514 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:27.004169  7940 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.019742 31673 rpcz_store.cc:276] 0709 12:24:24.821220 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.821608 (+   388us) service_pool.cc:224] Handling call
0709 12:24:24.822011 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:24.973772 (+151761us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.993604 (+2019832us) spinlock_profiling.cc:243] Waited 81.7 ms on lock 0x7b54002b0f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:26.993986 (+   382us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":187983616}
I20260709 12:24:26.971367  8012 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.026463  7756 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:27.027966  7756 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.028873 31565 maintenance_manager.cc:419] P ac4fa664fb214bda87cd96abb2116a4b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.205377
I20260709 12:24:27.030171  7763 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:27.031273 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:26.996235  7963 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.032387  7430 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:27.032908  7963 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:26.994207  8018 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:27.019539 31669 rpcz_store.cc:269] 0709 12:24:24.817807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.818011 (+   204us) service_pool.cc:224] Handling call
0709 12:24:26.991558 (+2173547us) spinlock_profiling.cc:243] Waited 162 ms on lock 0x7b54001d5e54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:26.991703 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":373570432}
I20260709 12:24:27.005504  7782 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:27.039703  7766 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.041679  7983 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.041234  8059 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.042366  7927 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:27.042375  7756 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.042693  7295 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:27.050796 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 66869.7us
I20260709 12:24:27.043066  7944 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.051703  7944 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:27.052788 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.054198  7944 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:27.050398  7295 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.042820  7861 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:27.054560  7767 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:27.054752  8016 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:27.055476 31675 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 20 index: 13. Preceding OpId from leader: term: 22 index: 14. (index mismatch)
I20260709 12:24:27.055614  7465 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.057204  8054 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.059425  7756 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.059455  7992 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.060693  8012 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.062901 31748 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:27.063527  7927 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.064522  7983 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.023882  7514 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:27.067587 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.053864 31819 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:27.048501  7766 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.068676  8059 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.073673 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:27.075405 31752 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:27.080035  7861 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.080675 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1169) took 2231 ms. Trace:
I20260709 12:24:27.082218 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1179) took 1446 ms. Trace:
W20260709 12:24:27.092190 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:27.087479 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 76415.7us
I20260709 12:24:27.095225 32374 rpcz_store.cc:276] 0709 12:24:24.848883 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:24.849179 (+   296us) service_pool.cc:224] Handling call
0709 12:24:24.849516 (+   337us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:25.571738 (+722222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.080014 (+1508276us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b540071d4d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:27.080449 (+   435us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1588456192}
I20260709 12:24:27.086719  7756 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:27.116618 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.106070 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:27.121999 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48546.4us
I20260709 12:24:27.125308 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:27.132995  7944 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.133767  7944 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:27.135115 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.146965  8059 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:27.147670  8059 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:27.147898 31825 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:24:27.149030  8059 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.149858  7983 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:27.126883 32342 leader_election.cc:336] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.970s (SENT)
I20260709 12:24:27.095616 32378 rpcz_store.cc:276] 0709 12:24:25.634421 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:25.634622 (+   201us) service_pool.cc:224] Handling call
0709 12:24:25.660253 (+ 25631us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:25.660450 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:26.150176 (+489726us) log.cc:844] Serialized 14 byte log entry
0709 12:24:26.754025 (+603849us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:26.754158 (+   133us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:26.754285 (+   127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:26.754420 (+   135us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.080795 (+326375us) spinlock_profiling.cc:243] Waited 17.8 ms on lock 0x7b5400427f50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:27.080913 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":713499,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":24784,"raft.queue_time_us":52158,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97,"spinlock_wait_cycles":40894208,"thread_start_us":1061448,"threads_started":3,"wal-append.queue_time_us":393968}
I20260709 12:24:27.122668 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.156420 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:27.156594  7927 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.153903 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:27.165371 31752 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:27.166754  7992 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.166879  7543 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.108668  8016 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.172922  8053 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:27.184684  7465 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:27.185846 31669 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:24:27.185732 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:27.184850  8069 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.100s	user 0.000s	sys 0.000s
I20260709 12:24:27.178448 31747 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:27.189218 31748 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:27.195976  7514 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.197368  7514 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:27.199010  8075 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.053s	user 0.001s	sys 0.000s
I20260709 12:24:27.201805 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.202589 31745 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:27.203792 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:27.204417 31745 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 21
W20260709 12:24:27.210387 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 104713us
I20260709 12:24:27.196450 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.192723  8053 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:27.217868 31745 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:24:27.218432  7756 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.218195  8053 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:27.219590 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63378.5us
I20260709 12:24:27.220669  7432 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:27.221630  7432 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.221917  8025 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:27.222506  7964 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:27.223134  8045 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:27.193512  7295 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:27.224928  8073 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.142s	user 0.001s	sys 0.000s
I20260709 12:24:27.225064  7432 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:27.224205  8074 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.080s	user 0.001s	sys 0.000s
I20260709 12:24:27.219724 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.168743  7543 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.241490  8012 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.241669 31745 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:27.243091 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.244498 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.245275 31745 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:27.246352 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.246958 31669 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:24:27.247453  7733 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.246965 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:27.223155  8025 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:27.248384 31748 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:27.248528  8025 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:27.249351 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.250095 31745 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:27.249689  8025 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.259271 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.259752 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:27.260618 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.257918 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:27.264760 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.269670  7733 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:27.270269  7733 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:27.270668  7733 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:27.271104 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:27.271863 32374 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:27.272387  8025 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:27.273021 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:27.273650 31819 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:27.269670 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.284456  7766 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.286110 31819 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:27.286386  7543 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:27.287940 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 19 index: 10 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:27.289073 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:27.289775 32370 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:27.289834 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:27.293192 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:27.297528 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:27.298810 31747 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:27.299319 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:27.299938 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.300061 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.300835 31747 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:27.301430 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.301684 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:27.302368 31671 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:27.303191 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.303462  7944 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:27.304385 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.305073 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:27.305450 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:27.299358 32377 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:27.300105 32375 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:27.306857 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.302433 31747 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:27.310004 31748 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:27.310617 32378 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:27.316269 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:27.317016 31748 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:27.319190  7805 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.876s	user 0.016s	sys 0.008s
I20260709 12:24:27.320307 31823 raft_consensus.cc:2410] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:24:27.320860 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.322535 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.324047 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.328065 31752 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:27.329064 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.330041 31678 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:24:27.333128 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.333822 31671 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:27.334807 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.335664 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.336050 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.336607 31678 raft_consensus.cc:2410] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 14.
I20260709 12:24:27.336975 31671 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:27.340004 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:27.342264  7295 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:27.343137  7295 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.344125  8012 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:27.344384 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:27.345350  7295 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.345829 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:27.348718  8000 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.352357  8000 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.353055  7767 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:27.325663  8059 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:27.355962  7990 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.811s	user 0.013s	sys 0.013s
I20260709 12:24:27.356352 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.381628 31748 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:27.359019 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.360093 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:27.360903 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.361148 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.362103  7963 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:27.363006 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:27.364313  8016 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:27.369975 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:27.370306 32342 leader_election.cc:336] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 2.189s (SENT)
I20260709 12:24:27.371500 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.371505 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.346545 31822 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:24:27.372189 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:27.372833  7779 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:27.373158 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:27.373884  8045 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:27.374828 31669 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:27.380371 31819 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
W20260709 12:24:27.357380  8072 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.368s	user 0.000s	sys 0.004s
I20260709 12:24:27.382922  8059 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.383044 31678 raft_consensus.cc:2368] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25.
I20260709 12:24:27.376008 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:27.382588 31750 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:27.383697 31745 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 30: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:27.383778 31670 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 17. (term mismatch)
I20260709 12:24:27.384835 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.402458 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:27.382831 31818 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 23: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:27.392103 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:27.390108 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:27.383237 31676 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:27.395315 31819 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:27.402048 31670 pending_rounds.cc:85] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 16
I20260709 12:24:27.403343 31750 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:27.405654 31825 raft_consensus.cc:2410] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 28 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 15.
I20260709 12:24:27.429937 31825 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 14. (term mismatch)
I20260709 12:24:27.426856 31670 pending_rounds.cc:107] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260709 12:24:27.386756  7990 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:27.432806 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62942.1us
I20260709 12:24:27.451079 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.444816 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.510216 31670 raft_consensus.cc:2889] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:27.432359 31825 pending_rounds.cc:85] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 13
I20260709 12:24:27.512405 31825 pending_rounds.cc:107] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 21.14
I20260709 12:24:27.512699 31825 raft_consensus.cc:2889] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:27.448316 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.444056 31822 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
W20260709 12:24:27.514962 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:27.515656 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.474844 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.387176  8054 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.514035 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:27.476039 31823 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:27.527895 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:27.447598 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:27.392659  8045 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:27.391083  7779 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:27.540501 31745 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:27.430998  7959 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:27.541266  8045 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:27.541051  7779 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.442870  7522 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:27.543591  7522 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:27.542435 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 167442us
I20260709 12:24:27.544095  7522 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:27.544499  8010 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:24:27.545292  8080 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.113s	user 0.001s	sys 0.000s
I20260709 12:24:27.545773 31751 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:27.546345  8054 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.546880  8018 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:27.547819  8018 consensus_queue.cc:1243] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 29.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:24:27.548418  8018 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
I20260709 12:24:27.548969  8054 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.549455  8018 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:27.542848  7944 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:24:27.549949 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 111454us
W20260709 12:24:27.468708  8068 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.573s	user 0.003s	sys 0.000s
W20260709 12:24:27.544152 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 173875us
I20260709 12:24:27.552132 32375 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:27.559062 31747 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 26 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
I20260709 12:24:27.553534 31676 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:27.389130  8002 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:27.562609 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.449445  8012 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:27.549763 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:27.562570 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.568535 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.550060  7465 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:27.549906  8018 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:27.439589  7733 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.541122  7959 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:27.574961  7465 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:27.567857  7590 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:27.575373 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48048.4us
I20260709 12:24:27.575650 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:27.567618 31819 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:24:27.576946 31673 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:27.580384 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:27.582333 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.585109 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.554095  7990 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.587703 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:27.579331  7295 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:27.593211 31672 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 27 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 14.
I20260709 12:24:27.582619  7959 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:27.595947 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.584368  7465 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:27.596781 31673 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:24:27.542598  8045 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.595922 31675 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:27.598714 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.601665 31752 raft_consensus.cc:2393] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:27.599805 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.577140  8010 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.018s
I20260709 12:24:27.606604 31748 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
W20260709 12:24:27.610600 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.611104  7733 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.612890 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.614986 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.604030 31642 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:27.615787 31745 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 21: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:27.616410 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.617516 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:27.611831 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.588347  7590 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:27.619779 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.620523 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:27.621376  8053 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:24:27.621724 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.617713 31671 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:27.622170 31749 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:27.623224 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:27.628185 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:27.628854 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.616639 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.621899 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:27.631975 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.632656 31745 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:27.635584 31676 raft_consensus.cc:2368] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25.
W20260709 12:24:27.622364 31791 connection.cc:256] RPC call timeout handler was delayed by -1.1382322139997996s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15880000000000002s for a response
I20260709 12:24:27.594105  7295 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:27.638399 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.639070 31673 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:27.643939  7798 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:27.644546 31676 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:24:27.644438  7798 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:27.644999 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:27.646123 31676 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:24:27.644416  7947 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:27.646979  7964 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:27.646867  7962 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:27.645888  7798 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.648383  7798 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:27.649659 31676 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:24:27.655342 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:27.643185  7295 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:27.635732  7514 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:24:27.660775 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 271844us
I20260709 12:24:27.667671  7995 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:27.667481  7543 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:27.669377 31675 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:27.669699  7955 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:27.621816  8053 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:27.691550  6752 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:27.676707 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.636214  8054 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:27.697312 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.678352  8059 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.648516  7962 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:27.684219 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.685599 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:27.687772 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44711.9us
I20260709 12:24:27.700747 31745 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:27.701572 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.688273 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.674540  8018 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.692027  6752 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:27.705641  6752 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:27.692623 31673 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:27.685129  7514 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:27.695808 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:27.703634 31748 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:27.704555 31819 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:27.706944 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:27.697954 31671 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:24:27.697140  8053 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:27.699137  7955 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:27.720191 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:27.699414  8054 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:27.704784  7944 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:27.703042  7962 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:27.730443 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:27.728076 31750 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:27.728401 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.731289 31752 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:27.731540 31745 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:27.714365  7514 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:27.735409  7543 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:27.736414 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.737286 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.740094 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.739434 31675 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:27.741611 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.727696  8016 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:24:27.742760 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 77032.9us
I20260709 12:24:27.743243 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:27.745157 31819 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:27.745316 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.747109  8022 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:27.729770  7944 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:27.743651 31750 raft_consensus.cc:2376] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26. Re-sending same reply.
I20260709 12:24:27.750734 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.730995  8054 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:27.745952 31671 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
W20260709 12:24:27.743496 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60208.5us
I20260709 12:24:27.749930 31823 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:27.749960 31748 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:27.739504  8059 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:27.752295 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:27.755791 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:27.759385 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:27.760538 31823 raft_consensus.cc:2410] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 25 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:24:27.765287  8012 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:27.765789  8012 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:27.766209  8012 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:27.744067  7861 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.758127 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.772408 31750 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:24:27.753216  7944 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:27.777532  8091 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:27.770972 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.752135  8016 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:27.762523 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:27.780686 31752 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:27.783610  7962 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:27.785998 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.792480 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.747691  8022 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.799333  7962 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:27.800876  8022 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:27.803996  6752 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.805446  7889 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:27.805850  7889 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 22 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:27.806960  7889 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.807544  8018 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:27.808701  7889 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:27.810510 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:27.811235 31824 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:27.812247  7590 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:27.813764  8088 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.072s	user 0.001s	sys 0.001s
I20260709 12:24:27.815733  8053 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.818928  8053 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.819691  7465 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.819579  8016 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.806099  6752 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:27.822926 31676 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:27.823215 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3460) took 1345 ms. Trace:
I20260709 12:24:27.823769 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:27.825011 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.825551  8010 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:27.826395  8090 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:24:27.826458 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.797531  7893 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:27.827498 31751 raft_consensus.cc:2410] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 13.
I20260709 12:24:27.828349 31744 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:27.829102 31744 raft_consensus.cc:606] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:27.829435 31744 raft_consensus.cc:614] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:27.798892  7964 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:27.823920  7465 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:27.833539 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:27.793347 31750 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:27.823637 31820 rpcz_store.cc:276] 0709 12:24:26.477547 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.571775 (+ 94228us) service_pool.cc:224] Handling call
0709 12:24:26.572052 (+   277us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.740005 (+167953us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:26.740107 (+   102us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:26.740189 (+    82us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:26.740255 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.822796 (+1082541us) spinlock_profiling.cc:243] Waited 167 ms on lock 0x7b54001973d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:27.822851 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:27.823041 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":384177024}
W20260709 12:24:27.831877 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64551.4us
I20260709 12:24:27.841038 31819 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:27.841398 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:27.842254 31744 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: [25.19-25.19]
W20260709 12:24:27.839174 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 54064.6us
I20260709 12:24:27.844085 31748 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
W20260709 12:24:27.845854  8089 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 12:24:27.778863  8092 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.079s
W20260709 12:24:27.822945 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.855998 31749 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:24:27.856180 31748 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: []
W20260709 12:24:27.856835 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.857445 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:27.853915 31744 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:24:27.785262  7522 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.856582 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.884294 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:27.882689  7940 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:27.886068 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.886786 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.886907 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.887568  7522 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:27.888155 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.888638 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.889506 31671 raft_consensus.cc:2410] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 26 because replica has last-logged OpId of term: 25 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 18.
I20260709 12:24:27.891062 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:27.887585 31675 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:24:27.890972  7959 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:27.892515  7590 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:27.891965 31749 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:24:27.893447  8025 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:27.893972  8025 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:27.894381  8025 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:27.894225 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:27.895043 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1491) took 1416 ms. Trace:
I20260709 12:24:27.895150 31749 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:27.895457 31827 rpcz_store.cc:276] 0709 12:24:26.478389 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.572823 (+ 94434us) service_pool.cc:224] Handling call
0709 12:24:26.607897 (+ 35074us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.608602 (+   705us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.894673 (+1286071us) spinlock_profiling.cc:243] Waited 34.8 ms on lock 0x7b54004461d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:27.894869 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":79969536}
I20260709 12:24:27.896152  8059 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:27.896566  8059 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:27.896971  8059 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:27.883668  7964 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:27.899720  7964 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.902714 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:27.903290 31678 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:27.903960  7940 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:27.905236 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:27.849079  7990 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:27.908264  8091 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:27.909583  8091 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:27.910033  8091 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:27.893764 31791 leader_election.cc:400] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25.
W20260709 12:24:27.914567 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.915518 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:27.919628 31744 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:27.921677 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53730.8us
I20260709 12:24:27.927649 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:27.932770 31752 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:27.955304 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:27.951776 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:27.964452 31820 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:24:27.955258 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:27.968714 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 213244us
I20260709 12:24:27.923707  7959 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:27.978108 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:27.979715 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 86106.2us
I20260709 12:24:27.969765  8053 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:24:28.009725 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:27.970608  7756 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:28.021708 31824 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:28.032322 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:27.964039  8062 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.055s
I20260709 12:24:28.011219  8053 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.016743  7959 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:24:28.072477  8079 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.526s	user 0.003s	sys 0.000s
I20260709 12:24:28.018144  7756 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:27.994424  7955 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:27.973203  8091 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:28.067003  8053 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:28.097080  7756 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:28.121219 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 143170us
I20260709 12:24:28.093817  7465 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:28.128237 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:28.141491 31826 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 26 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
W20260709 12:24:28.146341 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:28.148039 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:28.148116  8093 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:24:28.096304  7963 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.032s
W20260709 12:24:28.153964 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.165153 31676 raft_consensus.cc:2410] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 24 because replica has last-logged OpId of term: 23 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 13.
I20260709 12:24:28.133769  7465 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.168171  8022 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:28.146425  8033 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.195245  7295 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:28.195623  8033 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:28.196233 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:28.196915 32376 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:28.197742  7295 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:28.168613  7959 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:28.198490 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 55038.9us
W20260709 12:24:28.198815  8103 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:24:28.198977  7959 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:28.200906 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:28.201294  7959 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:28.202032  7959 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:28.169445 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.203088 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:28.171193  7955 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:28.204398 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:28.205190 32376 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:28.205335  8033 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:28.205510 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1351) took 2015 ms. Trace:
I20260709 12:24:28.205885  7959 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:28.205925 31670 rpcz_store.cc:276] 0709 12:24:26.189997 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.190165 (+   168us) service_pool.cc:224] Handling call
0709 12:24:26.190447 (+   282us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:27.561740 (+1371293us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.205254 (+643514us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b54003921d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:28.205443 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2742854016}
I20260709 12:24:28.207286  7766 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:28.207695  7766 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:28.194900  7465 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:28.211491  7959 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:28.211951  7514 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:28.212615 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:28.213064 31675 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:28.213217 31746 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
W20260709 12:24:28.213754 31746 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2880) took 1264 ms (client timeout 1588 ms). Trace:
W20260709 12:24:28.214090 31746 rpcz_store.cc:269] 0709 12:24:26.949790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.006271 (+ 56481us) service_pool.cc:224] Handling call
0709 12:24:28.212335 (+1206064us) spinlock_profiling.cc:243] Waited 3.01 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:24:28.213675 (+  1340us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":6926720}
W20260709 12:24:28.179006  8097 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.106s	user 0.000s	sys 0.001s
I20260709 12:24:28.198513 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2878) took 1249 ms. Trace:
I20260709 12:24:28.184156 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:28.184281  7755 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:28.185037 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 22 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:28.193303 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:28.194075 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:28.194886 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:28.215276 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3453) took 1742 ms. Trace:
I20260709 12:24:28.217072  8045 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:28.202368 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:28.174775  8094 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.044s
W20260709 12:24:28.195713  8095 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:24:28.195698 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:28.183275 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.236416 31744 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:28.297596 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:28.297806 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:28.237129 31675 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:28.299126 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:28.299875 31744 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:28.235494 31752 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:28.300652 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3565) took 1170 ms. Trace:
I20260709 12:24:28.257161 32369 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:28.230237 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.259166 32378 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:28.262128 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.207322  8091 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:28.304837 32370 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:28.232136 31753 rpcz_store.cc:276] 0709 12:24:26.948979 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.009688 (+ 60709us) service_pool.cc:224] Handling call
0709 12:24:27.078569 (+ 68881us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:27.093815 (+ 15246us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:27.093852 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:27.114469 (+ 20617us) log.cc:844] Serialized 26 byte log entry
0709 12:24:27.255650 (+141181us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:27.255804 (+   154us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.255869 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:28.071665 (+815796us) spinlock_profiling.cc:243] Waited 89.6 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:28.198007 (+126342us) raft_consensus.cc:1685] finished
0709 12:24:28.198030 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.198304 (+   274us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4688,"raft.queue_time_us":9159,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":4122,"spinlock_wait_cycles":206010624,"thread_start_us":89595,"threads_started":1,"wal-append.queue_time_us":312968}
I20260709 12:24:28.208683  7766 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:28.292192 32374 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:28.255347  8022 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.261001 32377 raft_consensus.cc:2410] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 29 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:24:28.249667  8062 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.303443 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:28.304778 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:28.300967 31752 rpcz_store.cc:276] 0709 12:24:27.129939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.786564 (+656625us) service_pool.cc:224] Handling call
0709 12:24:28.300599 (+514035us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.139828  8012 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:28.239609 31820 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:28.250106  7963 consensus_queue.cc:1243] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 26.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:24:28.251559 31821 rpcz_store.cc:276] 0709 12:24:26.472314 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.474844 (+  2530us) service_pool.cc:224] Handling call
0709 12:24:26.475240 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:26.508843 (+ 33603us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:26.509231 (+   388us) log.cc:844] Serialized 14 byte log entry
0709 12:24:27.608112 (+1098881us) spinlock_profiling.cc:243] Waited 6.74 ms on lock 0x7b180028e2e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:24:27.708804 (+100692us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:27.723600 (+ 14796us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:27.729736 (+  6136us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.825606 (+ 95870us) spinlock_profiling.cc:243] Waited 1.11 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:28.214923 (+389317us) spinlock_profiling.cc:243] Waited 33.2 ms on lock 0x7b5400379e54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:28.214971 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.215110 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":390,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":64,"raft.queue_time_us":7723,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":15423,"spinlock_wait_cycles":94476032,"thread_start_us":85474,"threads_started":1,"wal-append.queue_time_us":1237128}
I20260709 12:24:28.317449 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:28.317755 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:28.317277 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.251995  7944 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:28.314783 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 219738us
I20260709 12:24:28.308112  7889 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election won for term 26
W20260709 12:24:28.331101 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.328994 31746 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:28.324687 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:28.326750 31718 connection.cc:256] RPC call timeout handler was delayed by -1.6230209899999863s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18620000000000003s for a response
I20260709 12:24:28.332399 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.326608 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:28.327513 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:28.334388 31745 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:28.333878 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:28.317497  8000 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:24:28.337200 31718 connection.cc:256] RPC call timeout handler was delayed by -1.4142209900001035s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2094s for a response
W20260709 12:24:28.340468 31718 connection.cc:256] RPC call timeout handler was delayed by -1.3362145199998849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15660000000000002s for a response
W20260709 12:24:28.340806 31718 connection.cc:256] RPC call timeout handler was delayed by -1.2014557219999915s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1602s for a response
W20260709 12:24:28.341183 31718 connection.cc:256] RPC call timeout handler was delayed by -1.0921475810000629s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1572s for a response
I20260709 12:24:28.337427 31749 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:28.339690 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:28.345577 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 137962us
I20260709 12:24:28.343994 31822 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:28.353042 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:28.355957 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:28.354362 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:28.357400 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:28.367638 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:28.368427 31753 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:28.371378  7955 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:28.373829 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.375007 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:28.375999 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:28.335759  7944 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:28.374245  7955 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.343966  8000 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.379937 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.338454  7889 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:28.380510 31791 leader_election.cc:400] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25.
I20260709 12:24:28.386061 31791 leader_election.cc:403] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:24:28.379825 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:28.382179  7766 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:28.384697 31820 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:28.385236 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.388149  8013 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:28.388049 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.389516 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.390275 31821 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 27: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:28.391036  7514 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:28.389518  8112 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:24:28.385701  8000 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:28.392568  8059 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:28.395139 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.391067 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.397161 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:28.380276  8108 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:24:28.395078 31822 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 30 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:24:28.331673  8062 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:28.334875  8022 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:28.402244 31820 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:28.385862  7955 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:28.401388  8025 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.506s	user 0.012s	sys 0.003s
I20260709 12:24:28.401738  7514 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:28.433475  7514 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:28.398067  7861 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:28.433271  8010 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.607s	user 0.013s	sys 0.001s
I20260709 12:24:28.434523  7514 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 22.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:28.435122 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3559) took 1373 ms. Trace:
I20260709 12:24:28.428578 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.404075 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:28.430912 31746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:28.386276  8033 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:28.432924  8025 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:28.441823  7295 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:28.441985  7759 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:28.443831  8025 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:28.428881 32342 leader_election.cc:336] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 2.603s (SENT)
I20260709 12:24:28.436676 31746 raft_consensus.cc:606] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:28.401911  8013 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25.
I20260709 12:24:28.443722 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:28.451947 31818 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:28.447337 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:28.434378  7861 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:28.445010 31746 raft_consensus.cc:614] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:28.435671 31750 rpcz_store.cc:276] 0709 12:24:27.061231 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.860120 (+798889us) service_pool.cc:224] Handling call
0709 12:24:27.894886 (+ 34766us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:27.952279 (+ 57393us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:27.952312 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:27.966295 (+ 13983us) log.cc:844] Serialized 52 byte log entry
0709 12:24:28.127663 (+161368us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:28.127736 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.127863 (+   127us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:28.434863 (+307000us) raft_consensus.cc:1685] finished
0709 12:24:28.434891 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.435054 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26267,"raft.queue_time_us":70005,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":5548,"thread_start_us":110463,"threads_started":1,"wal-append.queue_time_us":305271}
I20260709 12:24:28.435079  8010 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
W20260709 12:24:28.449424 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:28.455433 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 120853us
I20260709 12:24:28.457386 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:28.459794 31822 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:28.464000 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:28.465354 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:28.466414 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.468101 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.428846  8059 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:28.468859 31821 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
W20260709 12:24:28.453881  8054 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.686s	user 0.014s	sys 0.000s
I20260709 12:24:28.441047  8033 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:28.480698 31750 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: [24.17-24.17]
I20260709 12:24:28.455076  7861 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:28.486503 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:28.487978 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.473358  7750 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:28.491976 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1233) took 1424 ms. Trace:
I20260709 12:24:28.485034 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:28.472939  7983 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:28.472756  7944 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:28.495157  6752 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:28.499481 31750 rpcz_store.cc:276] 0709 12:24:27.063694 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.459154 (+1395460us) service_pool.cc:224] Handling call
0709 12:24:28.480479 (+ 21325us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:28.487517 (+  7038us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.487777 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.469980  8114 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:28.506238  8010 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:24:28.506414 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49105.5us
I20260709 12:24:28.507716  6752 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:28.508728 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:28.520545 31749 raft_consensus.cc:2368] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23.
I20260709 12:24:28.509263 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2739) took 1056 ms. Trace:
W20260709 12:24:28.510876  8113 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
W20260709 12:24:28.512072  8115 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
I20260709 12:24:28.485512  8033 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:28.529227 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.529088  8053 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:28.490022  8018 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:28.521701 31672 rpcz_store.cc:276] 0709 12:24:27.452677 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.565148 (+112471us) service_pool.cc:224] Handling call
0709 12:24:28.508981 (+943833us) spinlock_profiling.cc:243] Waited 6.59 ms on lock 0x7b54002553d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:28.509159 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":15155072}
I20260709 12:24:28.495107  7295 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:28.531435 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1358) took 1152 ms. Trace:
I20260709 12:24:28.531970  8053 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:28.531625  7295 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:28.531889 31678 rpcz_store.cc:276] 0709 12:24:27.378894 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.444542 (+ 65648us) service_pool.cc:224] Handling call
0709 12:24:28.531186 (+1086644us) spinlock_profiling.cc:243] Waited 391 ms on lock 0x7b54002544d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:28.531342 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":899064064}
I20260709 12:24:28.491992  7963 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:28.451633  7759 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:28.499140  7750 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.479812  8054 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:28.541082 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 127883us
I20260709 12:24:28.541992 32369 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:24:28.524770  8012 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:28.524034  7962 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.724s	user 0.000s	sys 0.012s
I20260709 12:24:28.539245 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:28.543637 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:28.553118 31749 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:28.547945  6752 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:28.517903  8010 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.560293 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:28.498673  7989 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.034s
I20260709 12:24:28.481603  7889 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:28.568236 32375 raft_consensus.cc:2410] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 25 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 16.
I20260709 12:24:28.571549 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:28.539144  8059 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.575815 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:28.576624 31744 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 26
I20260709 12:24:28.551674  7750 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:28.565281  8010 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:28.630812 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 68768.1us
W20260709 12:24:28.630753 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51522.6us
I20260709 12:24:28.648119  7759 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:28.648947  8085 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.044s	user 0.004s	sys 0.000s
I20260709 12:24:28.650177  7295 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:28.650799 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1232) took 1587 ms. Trace:
W20260709 12:24:28.652652  8129 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:24:28.651290 31748 rpcz_store.cc:276] 0709 12:24:27.063324 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.864169 (+800845us) service_pool.cc:224] Handling call
0709 12:24:27.864511 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:27.920970 (+ 56459us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:24:28.035990 (+115020us) log.cc:844] Serialized 14 byte log entry
0709 12:24:28.445926 (+409936us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:28.467120 (+ 21194us) log.cc:844] Serialized 26 byte log entry
0709 12:24:28.467285 (+   165us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:28.467335 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.467422 (+    87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:28.650518 (+183096us) raft_consensus.cc:1685] finished
0709 12:24:28.650542 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.650715 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41467,"prepare.queue_time_us":185694,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":59,"raft.queue_time_us":81586,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":7821,"thread_start_us":454224,"threads_started":2,"wal-append.queue_time_us":466756}
W20260709 12:24:28.654146 31564 tablet_replica.cc:1411] Time spent flushing the Log queue took a long time: real 0.263s	user 0.010s	sys 0.016s
I20260709 12:24:28.659134 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2726) took 1307 ms. Trace:
I20260709 12:24:28.659956  8054 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:28.661208  8062 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:28.662194 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 119835us
I20260709 12:24:28.663897 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:28.666145 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:28.666553  8091 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:28.658365 32378 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:28.650084 32370 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
W20260709 12:24:28.674841 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 114684us
I20260709 12:24:28.676834 31749 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:28.597535  7779 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:28.685331 31744 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
W20260709 12:24:28.683486 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 197069us
I20260709 12:24:28.659600 31677 rpcz_store.cc:276] 0709 12:24:27.351503 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.353687 (+  2184us) service_pool.cc:224] Handling call
0709 12:24:27.354100 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:27.354398 (+   298us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:27.354426 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:27.355054 (+   628us) log.cc:844] Serialized 26 byte log entry
0709 12:24:27.393223 (+ 38169us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:27.393276 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:27.393368 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:28.648310 (+1254942us) spinlock_profiling.cc:243] Waited 1.05 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:28.658776 (+ 10466us) raft_consensus.cc:1685] finished
0709 12:24:28.658802 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.659057 (+   255us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2413824,"thread_start_us":37674,"threads_started":1,"wal-append.queue_time_us":212460}
I20260709 12:24:28.686750 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:28.689018 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:28.692332 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:28.634155  8059 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:28.700728 31744 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:28.701927 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1237) took 1636 ms. Trace:
I20260709 12:24:28.702497 31744 rpcz_store.cc:276] 0709 12:24:27.065143 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.686176 (+1621033us) service_pool.cc:224] Handling call
0709 12:24:28.687241 (+  1065us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.701549 (+ 14308us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.701836 (+   287us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.687911  7944 raft_consensus.cc:2764] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:24:28.709640 31744 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:28.686041  8114 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:28.716620 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3562) took 1611 ms. Trace:
I20260709 12:24:28.699293  7295 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:28.717430 31744 rpcz_store.cc:276] 0709 12:24:27.103841 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.706733 (+1602892us) service_pool.cc:224] Handling call
0709 12:24:28.707084 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.710016 (+  2932us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.710229 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:28.729552  8127 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.145s	user 0.001s	sys 0.000s
W20260709 12:24:28.731675  8104 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.584s	user 0.000s	sys 0.004s
W20260709 12:24:28.734540 31826 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 1494) took 2215 ms (client timeout 2116 ms). Trace:
I20260709 12:24:28.699857  8022 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:28.741384 31642 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:28.740825  8054 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.748760  8025 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.749001  8132 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:28.749436 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:28.750557 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.762863 31670 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:28.744958 31826 rpcz_store.cc:269] 0709 12:24:26.519019 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:26.561269 (+ 42250us) service_pool.cc:224] Handling call
0709 12:24:28.734281 (+2173012us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b540043e9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:28.734450 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3204973312}
I20260709 12:24:28.765287 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:28.772620 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:28.778828 31753 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:28.779064 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3564) took 1649 ms. Trace:
I20260709 12:24:28.779587 31744 rpcz_store.cc:276] 0709 12:24:27.129561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.739109 (+1609548us) service_pool.cc:224] Handling call
0709 12:24:28.739406 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.764755 (+ 25349us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.764790 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:28.778723 (+ 13933us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:28.778776 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.778848 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.778993 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.780524 31753 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:28.781665 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3570) took 1623 ms. Trace:
I20260709 12:24:28.782096 31753 rpcz_store.cc:276] 0709 12:24:27.158046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.779966 (+1621920us) service_pool.cc:224] Handling call
0709 12:24:28.780263 (+   297us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.781430 (+  1167us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.781614 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.774302 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:28.784494 31749 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:28.786124 31748 raft_consensus.cc:1217] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Deduplicated request from leader. Original: 21.15->[23.16-27.18]   Dedup: 25.17->[27.18-27.18]
W20260709 12:24:28.786342 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 99699.6us
I20260709 12:24:28.787611 31748 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 29. Ops: [23.16-27.18]
I20260709 12:24:28.793008 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3571) took 1634 ms. Trace:
I20260709 12:24:28.793406 31744 rpcz_store.cc:276] 0709 12:24:27.158443 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.780710 (+1622267us) service_pool.cc:224] Handling call
0709 12:24:28.781033 (+   323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.792700 (+ 11667us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:28.792752 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:28.792789 (+    37us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:28.792813 (+    24us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.792861 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.792957 (+    96us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:28.792317 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:28.781821  7805 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:28.752920  8053 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:28.786154  7767 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
W20260709 12:24:28.801375 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 121041us
I20260709 12:24:28.769953  8054 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:28.811486 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3563) took 1686 ms. Trace:
I20260709 12:24:28.788532  8132 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:28.811633 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:28.766245  8025 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:28.812562 31752 rpcz_store.cc:276] 0709 12:24:27.127556 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.726166 (+1598610us) service_pool.cc:224] Handling call
0709 12:24:28.726562 (+   396us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.726844 (+   282us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:24:28.727171 (+   327us) log.cc:844] Serialized 14 byte log entry
0709 12:24:28.810619 (+ 83448us) log.cc:844] Serialized 14 byte log entry
0709 12:24:28.811038 (+   419us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:28.811105 (+    67us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:28.811169 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.811287 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:28.811416 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5847,"prepare.queue_time_us":3055,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":10642,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":3822,"thread_start_us":64254,"threads_started":1}
I20260709 12:24:28.831450  8059 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:28.832000 31718 leader_election.cc:400] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21.
I20260709 12:24:28.832441 31718 leader_election.cc:403] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:28.807893  7861 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:28.834594  7861 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:28.806074  8002 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:28.835239  8002 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.835672  8002 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:28.838419  8054 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:28.840623  7465 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.629s	user 0.011s	sys 0.000s
I20260709 12:24:28.841082  7465 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:28.847612  8132 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:28.806324  7805 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:28.803668  8012 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:24:28.867620 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:28.894788 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:28.920078 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.879634 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:28.887182 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 147211us
I20260709 12:24:28.887606 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:28.953939 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:28.928992 31674 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:24:28.927536 31642 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:28.880878  8131 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.102s	user 0.000s	sys 0.000s
I20260709 12:24:28.830252  7955 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:28.945343  7944 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 20. Current term is 21.
W20260709 12:24:28.876637 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53570.4us
I20260709 12:24:28.866457  7805 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:29.011586 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:29.004119 31642 leader_election.cc:400] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23.
I20260709 12:24:28.867851  8135 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:29.012737 31642 leader_election.cc:403] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Cancelling election due to peer responding with higher term
I20260709 12:24:28.820015  7759 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:28.880023  8022 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:28.946275  7889 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:28.814587  8053 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.014710 31825 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 26 because replica has last-logged OpId of term: 25 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 20 index: 12.
W20260709 12:24:28.993597  7756 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.842s	user 0.013s	sys 0.000s
I20260709 12:24:28.886404  8010 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:28.815533  8114 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:29.011955  8000 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.619s	user 0.010s	sys 0.003s
W20260709 12:24:29.043215  8045 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.783s	user 0.013s	sys 0.000s
I20260709 12:24:29.043366  7465 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:28.890760  8130 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.125s
I20260709 12:24:29.045246  8025 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:29.051025  8135 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:29.054179 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 222352us
I20260709 12:24:28.967195  7514 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:29.082408 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:29.060403 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:29.074060  8010 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:29.074502 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 1.21481e+06us
W20260709 12:24:29.064380  8033 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.536s	user 0.006s	sys 0.005s
I20260709 12:24:29.079464 31747 raft_consensus.cc:2410] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:24:29.058475  7955 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:29.016216  7759 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:29.069082  7755 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.806s	user 0.002s	sys 0.011s
I20260709 12:24:29.061144  7733 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election lost for term 22. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23.
I20260709 12:24:29.064810 31670 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:29.063645  7889 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.070156  7756 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.095257  8045 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.097792  7767 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:29.105772  8130 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:29.106310  7779 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:29.077342  8000 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.112372 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:29.120002 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:29.120363 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60142.4us
I20260709 12:24:29.134847  8045 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.133044 31564 maintenance_manager.cc:643] P ac4fa664fb214bda87cd96abb2116a4b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 2.098s	user 0.108s	sys 0.036s Metrics: {"cfile_cache_miss":6,"cfile_cache_miss_bytes":25373,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"lbm_read_time_us":4162,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":21,"lbm_write_time_us":30941,"lbm_writes_1-10_ms":1,"lbm_writes_10-100_ms":1,"lbm_writes_lt_1ms":8,"peak_mem_usage":262624,"reinsert_count":0,"thread_start_us":231937,"threads_started":3,"update_count":178,"wal-append.queue_time_us":221817}
I20260709 12:24:29.143261 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:29.094223  7514 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:29.103617  7759 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:29.155579 31750 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:24:29.157480 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96647us
W20260709 12:24:29.161995 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63078.7us
I20260709 12:24:29.162407  7984 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.162983  7984 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.163396 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.160310  7963 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:29.160491  8025 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.005s
I20260709 12:24:29.128918  8092 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:29.137458  8130 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:29.147079  7750 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.515s	user 0.011s	sys 0.000s
I20260709 12:24:29.103163  8033 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.178953  8130 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:29.180402  8109 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.751s	user 0.017s	sys 0.007s
I20260709 12:24:29.180524  8130 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:29.167183  8059 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:29.182782 31753 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 18. (term mismatch)
I20260709 12:24:29.191504 31753 pending_rounds.cc:85] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 17
I20260709 12:24:29.183779 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:29.208868 31749 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:29.210510 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3653) took 1285 ms. Trace:
I20260709 12:24:29.211031 31675 rpcz_store.cc:276] 0709 12:24:27.925301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.927300 (+  1999us) service_pool.cc:224] Handling call
0709 12:24:29.210254 (+1282954us) spinlock_profiling.cc:243] Waited 284 ms on lock 0x7b54002e7a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:29.210425 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":654318336}
W20260709 12:24:29.219878 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.220513 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.232769 31753 pending_rounds.cc:107] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 28.18
I20260709 12:24:29.233332 31753 raft_consensus.cc:2889] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:29.205742 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3556) took 2145 ms. Trace:
I20260709 12:24:29.239099 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:29.240294 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45113.9us
I20260709 12:24:29.241379  8033 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:29.242038 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:29.243242  8092 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:29.243707  8092 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:29.244100  8092 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:29.244553  7989 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:29.245179  7989 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.183223  7750 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.177073  7295 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:29.247320 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:29.248082 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3643) took 1393 ms. Trace:
W20260709 12:24:29.255595 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 95307.5us
I20260709 12:24:29.261006 31676 rpcz_store.cc:276] 0709 12:24:27.855107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.856300 (+  1193us) service_pool.cc:224] Handling call
0709 12:24:29.217105 (+1360805us) spinlock_profiling.cc:243] Waited 281 ms on lock 0x7b540022ae54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:29.248000 (+ 30895us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":647011072}
I20260709 12:24:29.263623 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3586) took 1883 ms. Trace:
I20260709 12:24:29.270191 31749 raft_consensus.cc:606] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Received request to transfer leadership to TS b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.270756 31749 raft_consensus.cc:614] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:29.271967 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:29.273342  6752 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:29.196563  8053 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:29.280757 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3467) took 1091 ms. Trace:
I20260709 12:24:29.281194 31824 rpcz_store.cc:276] 0709 12:24:28.189814 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.247089 (+1057275us) service_pool.cc:224] Handling call
0709 12:24:29.280689 (+ 33600us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.285053  6752 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:29.255590  7944 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:29.257422  7295 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.255151  8010 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:29.269644 31746 rpcz_store.cc:276] 0709 12:24:27.370057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.848814 (+1478757us) service_pool.cc:224] Handling call
0709 12:24:28.849265 (+   451us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:28.950842 (+101577us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:29.093287 (+142445us) log.cc:844] Serialized 14 byte log entry
0709 12:24:29.251735 (+158448us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:29.251816 (+    81us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:29.251900 (+    84us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.251979 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.253497 (+  1518us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":113117,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":30,"raft.queue_time_us":1011,"raft.run_cpu_time_us":98,"raft.run_wall_time_us":99,"thread_start_us":183077,"threads_started":2}
I20260709 12:24:29.274991  7959 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:24:29.308194 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 124499us
I20260709 12:24:29.236106  8002 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.319998 31746 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:24:29.315613 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:29.263530  7465 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.042s
I20260709 12:24:29.320763 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3594) took 1445 ms. Trace:
W20260709 12:24:29.321305 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 160320us
I20260709 12:24:29.274477  8013 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:24:29.307484  7295 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:29.302412  7514 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.307299  8010 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.338092  8010 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:29.339710  7766 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.340272  7766 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.346709  8013 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:29.347283  8013 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:29.348308  8013 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.348930  7989 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:29.349720  8013 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:29.351799  7750 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:29.302866  7805 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.354442  7984 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:29.326014 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:29.323680 31746 rpcz_store.cc:276] 0709 12:24:27.875325 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.306051 (+1430726us) service_pool.cc:224] Handling call
0709 12:24:29.316023 (+  9972us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.320497 (+  4474us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.320703 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:29.327874  8145 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.229s	user 0.001s	sys 0.000s
I20260709 12:24:29.379717 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1234) took 2315 ms. Trace:
W20260709 12:24:29.417364 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 91487.2us
I20260709 12:24:29.417532 31748 rpcz_store.cc:276] 0709 12:24:27.064050 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.653507 (+1589457us) service_pool.cc:224] Handling call
0709 12:24:28.779511 (+126004us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:28.794253 (+ 14742us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.379404 (+585151us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54001ee6d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:29.379623 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":288751488}
W20260709 12:24:29.418115 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.328348  8135 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:29.328397  8000 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:29.388700 31745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3572) took 2229 ms (client timeout 2496 ms). Trace:
I20260709 12:24:29.419205 31748 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:24:29.419484  8000 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:24:29.419400 31745 rpcz_store.cc:269] 0709 12:24:27.158838 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.739896 (+581058us) service_pool.cc:224] Handling call
0709 12:24:29.388506 (+1648610us) spinlock_profiling.cc:243] Waited 587 ms on lock 0x7b54001ed2d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:29.388641 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1350866304}
I20260709 12:24:29.420248  8000 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:29.421232  8000 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.421931 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:29.422402  7779 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:29.422994  7779 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.423368 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:29.329689  8098 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.811s	user 0.000s	sys 0.012s
I20260709 12:24:29.332422  7465 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:29.387151 31752 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 14. (term mismatch)
I20260709 12:24:29.372984  8045 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:29.329264  7889 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:29.431538  8054 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:29.425022 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.431773  7465 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:29.436795  8054 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.437642  7465 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.438753  7465 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:29.424683 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.438791 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:29.388013 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:29.440037 31821 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:29.397310 31749 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:29.440547 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:29.441188 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3596) took 1565 ms. Trace:
I20260709 12:24:29.441360 32375 raft_consensus.cc:2368] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23.
W20260709 12:24:29.386499 31747 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2892) took 2036 ms (client timeout 1860 ms). Trace:
W20260709 12:24:29.401603 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.426818 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.376713 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61187.7us
W20260709 12:24:29.445227 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 124755us
I20260709 12:24:29.432003 31752 pending_rounds.cc:85] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 13
I20260709 12:24:29.446027 31752 pending_rounds.cc:107] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 12:24:29.446336 31752 raft_consensus.cc:2889] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:29.436295 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:29.387043 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3574) took 2227 ms. Trace:
I20260709 12:24:29.448195 31753 rpcz_store.cc:276] 0709 12:24:27.159527 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.796544 (+1637017us) service_pool.cc:224] Handling call
0709 12:24:28.812345 (+ 15801us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.269920 (+457575us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.386806 (+116886us) spinlock_profiling.cc:243] Waited 355 ms on lock 0x7b5400064d54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:29.386979 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":817573120}
I20260709 12:24:29.321830  6752 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.403573  8022 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:29.388193 31744 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:29.382335  8126 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.703s	user 0.005s	sys 0.006s
I20260709 12:24:29.451224 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:29.441664 31749 rpcz_store.cc:276] 0709 12:24:27.875891 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.396750 (+1520859us) service_pool.cc:224] Handling call
0709 12:24:29.397101 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.440884 (+ 43783us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.441113 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.367489 32343 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:29.453038 31671 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:24:29.456475 31671 raft_consensus.cc:606] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:29.456954 31671 raft_consensus.cc:614] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:24:29.442921 31747 rpcz_store.cc:269] 0709 12:24:27.349749 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.568283 (+218534us) service_pool.cc:224] Handling call
0709 12:24:29.386210 (+1817927us) spinlock_profiling.cc:243] Waited 1.51 s on lock 0x7b54001ed7d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:29.386413 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3463308160}
I20260709 12:24:29.237118 31751 rpcz_store.cc:276] 0709 12:24:27.060251 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:27.829266 (+769015us) service_pool.cc:224] Handling call
0709 12:24:27.856157 (+ 26891us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:27.856603 (+   446us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:27.856631 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:27.930340 (+ 73709us) log.cc:844] Serialized 52 byte log entry
0709 12:24:28.030186 (+ 99846us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:28.142333 (+112147us) log.cc:844] Serialized 14 byte log entry
0709 12:24:28.142579 (+   246us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:28.149501 (+  6922us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:28.730479 (+580978us) spinlock_profiling.cc:243] Waited 33.8 ms on lock 0x7b1800376c00. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db
0709 12:24:28.731016 (+   537us) spinlock_profiling.cc:243] Waited 2.19 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:28.820262 (+ 89246us) raft_consensus.cc:1685] finished
0709 12:24:28.820283 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.205452 (+385169us) spinlock_profiling.cc:243] Waited 26.5 ms on lock 0x7b54004042d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:29.205648 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":67272,"prepare.queue_time_us":96707,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":735,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":31945,"spinlock_wait_cycles":143716096,"thread_start_us":164595,"threads_started":2,"wal-append.queue_time_us":165660}
I20260709 12:24:29.452397 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.465605  7779 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:29.358883  8002 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:29.359441  7756 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:29.376760  8059 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.355378  7766 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:29.468598 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:29.475119 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:29.489979 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:29.519682 31744 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
I20260709 12:24:29.492013 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:29.489085 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:29.377048  8162 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:29.429001  7984 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:29.508724  8135 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:24:29.510327  8153 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.157s	user 0.002s	sys 0.000s
I20260709 12:24:29.458199  7893 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.458024  8053 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.492609  7779 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:29.491160 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:29.520551 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60189.8us
I20260709 12:24:29.493212 31750 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:24:29.440176  7805 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:29.522662 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3587) took 2152 ms. Trace:
W20260709 12:24:29.537709 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 170320us
I20260709 12:24:29.391227  7295 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:29.541534 31669 raft_consensus.cc:2368] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31.
I20260709 12:24:29.496924  8128 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:29.526115 31821 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:29.552150  8161 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:29.553866  8161 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:29.529420  7944 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election won for term 23
W20260709 12:24:29.530719 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78437.8us
I20260709 12:24:29.486488  8012 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:29.537214 31752 rpcz_store.cc:276] 0709 12:24:27.370361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.851520 (+1481159us) service_pool.cc:224] Handling call
0709 12:24:29.386695 (+535175us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.446931 (+ 60236us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.522409 (+ 75478us) spinlock_profiling.cc:243] Waited 424 ms on lock 0x7b54001ed7d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:29.522571 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":976280832}
I20260709 12:24:29.376359  7759 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.547592 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:29.559669 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:29.452500  8022 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:29.550522 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:29.561472 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 315567us
I20260709 12:24:29.569608 31746 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: [25.16-27.17]
I20260709 12:24:29.554569  8161 consensus_queue.cc:1243] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 27.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:24:29.530695 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3595) took 1655 ms. Trace:
I20260709 12:24:29.583073 32378 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:29.557060  8054 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:29.589156 32378 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:29.585520 31717 leader_election.cc:400] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31.
I20260709 12:24:29.522081  8059 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.590325 31717 leader_election.cc:403] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:24:29.591228  8045 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31.
I20260709 12:24:29.586699 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.535895  8010 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.531213  7984 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.593092 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:29.542258  8053 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.539283  7893 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.582015 31744 rpcz_store.cc:276] 0709 12:24:27.875604 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.387745 (+1512141us) service_pool.cc:224] Handling call
0709 12:24:29.388060 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.530298 (+142238us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.530601 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:29.575487  8091 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:29.597734 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:29.602428 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52015.1us
I20260709 12:24:29.557677  8092 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.622179  7756 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.622763  7756 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.623991  8092 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:29.624871  7756 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:29.627404 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:29.615265  8091 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:29.630025  8091 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.633361  7295 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:29.584622  7766 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:29.605290  7759 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:29.607123  8010 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:29.612129  7984 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:29.578974  7963 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:29.612727 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:29.613909  8160 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.266s	user 0.000s	sys 0.002s
I20260709 12:24:29.614053  8053 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:29.614441  7514 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:29.615244  8156 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.289s	user 0.001s	sys 0.000s
I20260709 12:24:29.616673  7893 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:29.617230 32373 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 27: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:29.617940 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:29.662497 32370 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
W20260709 12:24:29.619637 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60057.6us
I20260709 12:24:29.621397 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.621631 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:29.663697 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:29.580123  7944 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:29.624040 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:29.604506  8059 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:29.634498 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3602) took 1176 ms. Trace:
I20260709 12:24:29.585295  7779 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:29.584375  8012 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:29.635459 31672 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:29.642354 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:29.644377 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:29.678385 31747 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:29.647365 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:29.679352 31753 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:29.680104  7963 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.680606  7944 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 21.16, Last appended by leader: 16, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.649703 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:29.683197  8158 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.324s	user 0.000s	sys 0.002s
I20260709 12:24:29.683416 31749 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:24:29.650432 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:29.684774 31751 raft_consensus.cc:2410] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 30 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 18.
I20260709 12:24:29.603806  7805 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:29.685552  7805 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:29.686000  7805 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:29.654553 32369 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:29.690841  7759 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:29.657140 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3601) took 1199 ms. Trace:
I20260709 12:24:29.691500  7759 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:29.691587 31750 rpcz_store.cc:276] 0709 12:24:28.457311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.520591 (+ 63280us) service_pool.cc:224] Handling call
0709 12:24:29.656904 (+1136313us) spinlock_profiling.cc:243] Waited 602 ms on lock 0x7b54002134d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:29.657057 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1385360640}
I20260709 12:24:29.595338  7767 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:29.692510  7759 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 22.13, Last appended by leader: 13, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.693255  7767 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:29.695259 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 73963.9us
I20260709 12:24:29.695679  8135 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.655002  7995 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:29.696194 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.696599  7995 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.697592 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.698793  8130 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:29.699317  8130 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:29.700168  8140 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:29.700310  8130 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.655231  8155 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.046s
I20260709 12:24:29.664517  7514 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.696213  8135 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.705467  7514 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:29.706099  8135 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:29.707155  8059 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:29.657629 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:29.664011 31669 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:24:29.709034  8012 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.709559  7893 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20260709 12:24:29.711791  8138 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.321s	user 0.001s	sys 0.002s
I20260709 12:24:29.666445 31678 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:29.671065 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:29.712546 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:29.713459 31750 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:29.713871 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:29.714671 31747 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:29.715672 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.672587 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.674083 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:29.675017 31746 rpcz_store.cc:276] 0709 12:24:28.457566 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.417511 (+959945us) service_pool.cc:224] Handling call
0709 12:24:29.417910 (+   399us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:29.575711 (+157801us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.634186 (+ 58475us) spinlock_profiling.cc:243] Waited 149 ms on lock 0x7b5400212fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:29.634431 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":342410112}
I20260709 12:24:29.724097  8163 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:29.724253 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:29.724261 31746 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:24:29.724896  8163 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:29.725560 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:29.725878 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:29.726186 31750 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:24:29.727442 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:29.737991 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:29.728636 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:29.739742 31744 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:29.729516 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:29.727315 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54826.8us
I20260709 12:24:29.741766 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:29.741847 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:29.745335 32369 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:29.750068 31753 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:29.726619 31746 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:29.754345 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:29.759234 31744 pending_rounds.cc:85] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 14
I20260709 12:24:29.758870 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.759764 31744 pending_rounds.cc:107] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 27.15
I20260709 12:24:29.760085 31744 raft_consensus.cc:2889] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:29.760708 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.759341 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:29.764675 31748 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 26 because replica has last-logged OpId of term: 24 index: 13, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:24:29.770591 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:29.736393  8132 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.855s	user 0.011s	sys 0.015s
W20260709 12:24:29.774376 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58766us
I20260709 12:24:29.774686 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:29.775489 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.727519  7984 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:29.782992 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:29.728262  8163 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:29.759430  7514 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:29.758805  7465 raft_consensus.cc:2764] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:24:29.760174  8059 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:24:29.759955  7295 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:29.727065  8167 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:29.763021  8053 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:24:29.766839  8138 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:24:29.780954  8000 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:29.767768  7756 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:29.732016  8013 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.053s
I20260709 12:24:29.774736  8045 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:24:29.780539  8132 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.899s	user 0.011s	sys 0.015s
I20260709 12:24:29.813386 31750 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:29.816751 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:29.763242  7893 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.040s
I20260709 12:24:29.759740  8054 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260709 12:24:29.800988  8155 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:29.865043  8155 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.865607  8155 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:29.787127  7514 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.786374  7984 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:29.800985  8163 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:29.860906 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.820899  8000 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:29.864377 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:29.866852 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 411638us
I20260709 12:24:29.893649 31643 leader_election.cc:400] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 22. Current term is 23.
I20260709 12:24:29.804880  8053 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.894863  8053 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:29.823480  8013 consensus_queue.cc:1243] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 22.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:29.820443  8059 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:29.900782  8059 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:29.906533 31567 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 240) took 44246 ms (client timeout 15000 ms). Trace:
I20260709 12:24:29.910744  7995 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:29.911330 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:24:29.871507  7514 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:29.913897  8053 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:29.920723  8166 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
W20260709 12:24:29.922459 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.869953  8054 consensus_queue.cc:1243] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 26.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:24:29.869653  7295 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:29.890849  7984 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:29.938199 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 77386us
I20260709 12:24:29.940727 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:29.922713  8000 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:29.824134  8167 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.948774 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:29.949740 31752 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:29.930769  8013 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:29.946678 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
W20260709 12:24:29.955478 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.953902 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:29.933511  7295 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:29.930280  7805 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:29.970082 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1359) took 1225 ms. Trace:
W20260709 12:24:29.907238 31567 rpcz_store.cc:269] 0709 12:23:45.619663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:48.507775 (+2888112us) service_pool.cc:224] Handling call
0709 12:24:29.852686 (+41344911us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:23.923245 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:23.924203 (+   958us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:23.924265 (+    62us) write_op.cc:435] Acquired schema lock
0709 12:24:23.924298 (+    33us) tablet.cc:662] Decoding operations
0709 12:24:23.963169 (+ 38871us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:23.963298 (+   129us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:23.963341 (+    43us) tablet.cc:685] Acquiring locks for 21 operations
0709 12:24:24.294808 (+331467us) tablet.cc:701] Row locks acquired
0709 12:24:24.294880 (+    72us) write_op.cc:260] PREPARE: finished
0709 12:24:24.295058 (+   178us) write_op.cc:270] Start()
0709 12:24:24.295270 (+   212us) write_op.cc:276] Timestamp: P: 1783599864294984 usec, L: 0
0709 12:24:24.295314 (+    44us) op_driver.cc:348] REPLICATION: starting
0709 12:24:24.296310 (+   996us) log.cc:844] Serialized 8365 byte log entry
0709 12:24:25.527270 (+1230960us) op_driver.cc:464] REPLICATION: finished
0709 12:24:25.622476 (+ 95206us) write_op.cc:301] APPLY: starting
0709 12:24:25.622586 (+   110us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:25.744764 (+122178us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:25.744899 (+   135us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:26.238085 (+493186us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:26.238526 (+   441us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=42,key_file_lookups=42,delta_file_lookups=42,mrs_lookups=0
0709 12:24:26.238594 (+    68us) write_op.cc:312] APPLY: finished
0709 12:24:26.254548 (+ 15954us) log.cc:844] Serialized 188 byte log entry
0709 12:24:26.255880 (+  1332us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:26.256089 (+   209us) write_op.cc:454] Released schema lock
0709 12:24:26.256174 (+    85us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:26.256665 (+   491us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:26.256704 (+    39us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":12495,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":80256,"apply.run_cpu_time_us":34261,"apply.run_wall_time_us":634198,"cfile_cache_hit":86,"cfile_cache_hit_bytes":91865,"num_ops":21,"prepare.queue_time_us":582519,"prepare.run_cpu_time_us":24745,"prepare.run_wall_time_us":828855,"raft.queue_time_us":190144,"raft.run_cpu_time_us":14426,"raft.run_wall_time_us":50850,"replication_time_us":1231747,"thread_start_us":769235,"threads_started":4,"wal-append.queue_time_us":579040,"wal-append.run_cpu_time_us":73229,"wal-append.run_wall_time_us":2672113}]]}
I20260709 12:24:29.936964  8053 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:29.955744  8013 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:29.961184  8163 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:29.960184  7295 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:29.961592  8161 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:24:29.977159 31792 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.607s	user 0.009s	sys 0.010s
W20260709 12:24:30.046455 31792 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.677s	user 0.009s	sys 0.010s
W20260709 12:24:30.047040 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 677757us
I20260709 12:24:30.049052  7465 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.049832  7465 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:30.052884  7465 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:30.053785 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:30.063740 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:29.972378 31677 rpcz_store.cc:276] 0709 12:24:28.745048 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:28.745260 (+   212us) service_pool.cc:224] Handling call
0709 12:24:29.214462 (+469202us) spinlock_profiling.cc:243] Waited 1.23 ms on lock 0x7b5800105998. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:24:29.258506 (+ 44044us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.269340 (+ 10834us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:24:29.572100 (+302760us) log.cc:844] Serialized 14 byte log entry
0709 12:24:29.969272 (+397172us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:29.969352 (+    80us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:29.969410 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:29.969766 (+   356us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:29.970000 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30355,"prepare.queue_time_us":32015,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":2044,"spinlock_wait_cycles":2826112,"thread_start_us":631646,"threads_started":2}
I20260709 12:24:29.969563  8025 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:29.970819  8162 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:30.071558 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 418363us
I20260709 12:24:30.083140 31752 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:30.110522 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:30.017241  8013 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:30.112167 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:30.141129 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:30.061769  8163 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:30.144330 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:30.148105 31825 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:24:30.149600 31819 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 24 index: 14. (index mismatch)
W20260709 12:24:30.150799 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 380254us
I20260709 12:24:30.161537 31745 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:24:30.152038 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:30.162578 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:30.142596 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:30.107815  8161 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:30.174305 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:30.174978 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:30.174360 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:30.135042  8114 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.124984  8151 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:30.184044 32378 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:30.184219 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:30.185402 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.013s	sys 0.007s
W20260709 12:24:30.200246 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.547s	user 0.013s	sys 0.008s
I20260709 12:24:30.180892 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:30.171934 31752 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:24:30.175405 31819 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 31: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:30.150997  7959 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.199080 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:30.180935  7759 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:30.213938 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 236914us
I20260709 12:24:30.191807  8045 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:30.132004  8025 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:30.223058 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50297us
W20260709 12:24:30.213109  8012 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.015s	sys 0.012s
I20260709 12:24:30.233724 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:30.230446 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:30.238529 31826 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:30.240687 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:30.245561 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:30.246201 31752 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 15. (term mismatch)
I20260709 12:24:30.246670 31819 raft_consensus.cc:2368] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25.
I20260709 12:24:30.268791 31752 pending_rounds.cc:85] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 14
I20260709 12:24:30.269233 31752 pending_rounds.cc:107] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 27.15
I20260709 12:24:30.269585 31752 raft_consensus.cc:2889] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:30.211427  8054 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:24:30.234195  8012 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.522s	user 0.016s	sys 0.012s
W20260709 12:24:30.292244 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53215.4us
W20260709 12:24:30.289170 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47408.9us
I20260709 12:24:30.327320 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:30.331687 31676 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
W20260709 12:24:30.287189  8168 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.339s	user 0.001s	sys 0.000s
I20260709 12:24:30.205410  8114 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:30.237993  7756 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:30.339653 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:30.343868 31826 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:30.344167 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:30.232496  7759 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:30.348204 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:30.220676  7959 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:30.318908  8190 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:30.331735  8054 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:30.370234 31676 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:30.370903  8172 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:30.372035 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 221424us
I20260709 12:24:30.375182 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:30.382321 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:30.269476  8135 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.082s
I20260709 12:24:30.387106 32369 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:30.383116 32377 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:30.399293 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:30.399371 32373 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 17. (term mismatch)
I20260709 12:24:30.400225 32373 pending_rounds.cc:85] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 16
I20260709 12:24:30.400502 32373 pending_rounds.cc:107] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 24.17
I20260709 12:24:30.400785 32373 raft_consensus.cc:2889] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:30.402894 32378 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:30.405998 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:30.406240 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:30.407321 31752 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:30.408753 31752 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:30.415740 31818 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:24:30.417064 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:30.417865 31818 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:30.398993  8172 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:30.420804  8172 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:30.420917 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:30.421743 31825 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:24:30.422693 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:30.424249 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:30.425899 31825 raft_consensus.cc:2410] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:24:30.424973 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:30.427131 31818 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:30.428785 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:30.429593 31818 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:30.430744 31818 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:30.427716 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:30.434038  8022 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.854s	user 0.006s	sys 0.005s
W20260709 12:24:30.435458  8180 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
I20260709 12:24:30.267958  8162 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:30.366840  7805 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:30.443431 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:30.411056  8169 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.438s	user 0.000s	sys 0.001s
I20260709 12:24:30.386195  7756 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.339709  8161 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:30.451973 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:30.452145 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:30.467374 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:30.461153 31825 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:30.500927 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:30.501858  8000 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.549s	user 0.010s	sys 0.000s
I20260709 12:24:30.502370  8000 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:30.505165  8000 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:30.339763  8163 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:30.543337  8183 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.215s	user 0.001s	sys 0.000s
W20260709 12:24:30.553009  8155 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.687s	user 0.009s	sys 0.004s
I20260709 12:24:30.555166  8155 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:30.444789  8182 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:30.434567  8022 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:30.666453 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 228934us
I20260709 12:24:30.510215  8184 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:24:30.671212 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 430598us
I20260709 12:24:30.671818 31821 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:24:30.678370  8165 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:30.685292 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:30.719545 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:30.687281  8025 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:30.719707 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:30.615084  8002 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:24:30.575882  8130 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.654s	user 0.005s	sys 0.027s
I20260709 12:24:30.707660 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:30.716707 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:30.662501  7944 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.964s	user 0.013s	sys 0.003s
W20260709 12:24:30.691399  8175 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.376s	user 0.004s	sys 0.018s
I20260709 12:24:30.615474  8182 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:30.719087  8165 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:30.574470  8054 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:30.725400 31718 connection.cc:256] RPC call timeout handler was delayed by -1.027138455000113s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1761s for a response
I20260709 12:24:30.512171  7756 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:30.739473 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 293491us
I20260709 12:24:30.735031 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:30.748004 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:30.729863 31567 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 241) took 21132 ms (client timeout 15000 ms). Trace:
I20260709 12:24:30.747435 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:30.758252 31567 rpcz_store.cc:269] 0709 12:24:09.718611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.726152 (+21007541us) service_pool.cc:211] Skipping call since client already timed out
0709 12:24:30.729757 (+  3605us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:24:30.743022 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3611) took 1024 ms. Trace:
I20260709 12:24:30.760350 31747 rpcz_store.cc:276] 0709 12:24:29.719049 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.719271 (+   222us) service_pool.cc:224] Handling call
0709 12:24:29.719633 (+   362us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:29.720019 (+   386us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:29.720052 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:29.762917 (+ 42865us) log.cc:844] Serialized 52 byte log entry
0709 12:24:30.272471 (+509554us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:30.662963 (+390492us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.715451 (+ 52488us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.718466 (+  3015us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.718548 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:30.742774 (+ 24226us) raft_consensus.cc:1685] finished
0709 12:24:30.742799 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.742957 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2631,"prepare.queue_time_us":448612,"raft.queue_time_us":37574,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":26546,"thread_start_us":803430,"threads_started":2,"wal-append.queue_time_us":881069}
W20260709 12:24:30.764704 31827 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.563s	user 0.011s	sys 0.000s
I20260709 12:24:30.765419 31827 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:30.766737 31825 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:24:30.768644 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:30.730997  8195 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.220s	user 0.001s	sys 0.000s
I20260709 12:24:30.771063  7995 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:30.785532 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:30.787621 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:30.790266 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:30.792095 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3622) took 1052 ms. Trace:
I20260709 12:24:30.792618 31746 rpcz_store.cc:276] 0709 12:24:29.739670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.803782 (+ 64112us) service_pool.cc:224] Handling call
0709 12:24:29.823510 (+ 19728us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:29.823886 (+   376us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:29.823922 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:29.824415 (+   493us) log.cc:844] Serialized 26 byte log entry
0709 12:24:30.219139 (+394724us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:30.219186 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.219259 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:30.791806 (+572547us) raft_consensus.cc:1685] finished
0709 12:24:30.791836 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.792020 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3956,"raft.queue_time_us":126,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":10882,"thread_start_us":353777,"threads_started":1,"wal-append.queue_time_us":517925}
W20260709 12:24:30.721506  7984 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.768s	user 0.014s	sys 0.004s
I20260709 12:24:30.797319 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:30.733821  8130 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.812s	user 0.005s	sys 0.027s
W20260709 12:24:30.738052  7944 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.040s	user 0.013s	sys 0.004s
I20260709 12:24:30.801681 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:30.738937  8176 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.285s	user 0.001s	sys 0.000s
W20260709 12:24:30.745250  8013 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.601s	user 0.009s	sys 0.001s
I20260709 12:24:30.726855  8143 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:30.735356  8184 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:30.731324  8114 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:30.730417  8151 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:30.811546  6752 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:30.732012  7759 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:30.751714  7756 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:30.813026 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:30.813527 32377 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 19. (term mismatch)
I20260709 12:24:30.739128  8201 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:30.727721  8002 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:30.743446  7959 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:30.743664  8182 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:30.807487  8198 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.430s	user 0.000s	sys 0.001s
I20260709 12:24:30.815232  8143 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:30.802814  8012 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:30.807678  8189 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:30.806726  7889 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:30.827224 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:30.814867  8025 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:30.858935 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:30.865350 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:30.824020 32377 pending_rounds.cc:85] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 18
I20260709 12:24:30.869885 31824 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
W20260709 12:24:30.820164  8178 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.533s	user 0.001s	sys 0.001s
I20260709 12:24:30.819407  8091 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:30.876912 31821 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:30.832520  8151 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:30.804077  7984 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:30.881160 31748 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.503s	user 0.032s	sys 0.027s
W20260709 12:24:30.906934 31748 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.033s	sys 0.027s
I20260709 12:24:30.746183  8165 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:30.877681 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:30.875001 32377 pending_rounds.cc:107] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 28.19
I20260709 12:24:30.909873 32377 raft_consensus.cc:2889] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:30.854374  8002 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:30.831949  7779 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:30.832056  7759 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:30.815083  8013 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:30.920341 31749 raft_consensus.cc:2410] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:24:30.860124  8143 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:30.873957  8206 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:30.868755  7733 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.057s
I20260709 12:24:30.920572  8165 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:30.929440  7514 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.014s	user 0.011s	sys 0.001s
I20260709 12:24:30.929870  7514 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:30.882982  7889 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:30.932137  8022 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:30.920706 31827 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:30.933547 31827 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:30.900897  8189 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:30.856503  8172 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 12:24:30.908856  7766 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:30.923786  8155 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:30.923976  8177 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260709 12:24:30.901938  7963 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:30.905248  8025 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:30.878569  7756 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:30.884732  8197 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.067s
I20260709 12:24:30.919498  7779 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:30.941879 31818 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
W20260709 12:24:30.933235  7295 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.872s	user 0.012s	sys 0.000s
W20260709 12:24:30.943105 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 73972.9us
I20260709 12:24:30.939472 31827 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:30.974571 31827 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:30.882740  8000 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:24:30.854897  8201 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:30.947654  8025 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:30.919704  8161 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:30.950908  7963 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:30.932988  7889 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:30.940871  7733 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.014s
I20260709 12:24:30.941680  7893 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 12:24:30.960184  7514 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:30.960426 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3627) took 1040 ms. Trace:
I20260709 12:24:31.019505 31745 rpcz_store.cc:276] 0709 12:24:29.919751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.177345 (+257594us) service_pool.cc:224] Handling call
0709 12:24:30.177771 (+   426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:30.219060 (+ 41289us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.219201 (+   141us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:30.286473 (+ 67272us) log.cc:844] Serialized 26 byte log entry
0709 12:24:30.811300 (+524827us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:30.811360 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.813278 (+  1918us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:30.960150 (+146872us) raft_consensus.cc:1685] finished
0709 12:24:30.960173 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.960359 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8307,"raft.queue_time_us":43764,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":164,"thread_start_us":446115,"threads_started":1,"wal-append.queue_time_us":596829}
I20260709 12:24:30.975596 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3623) took 1207 ms. Trace:
I20260709 12:24:31.018342 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3619) took 1279 ms. Trace:
I20260709 12:24:31.021674 31748 rpcz_store.cc:276] 0709 12:24:29.768107 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.852353 (+ 84246us) service_pool.cc:224] Handling call
0709 12:24:29.852819 (+   466us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:29.862623 (+  9804us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:29.862678 (+    55us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:29.863194 (+   516us) log.cc:844] Serialized 26 byte log entry
0709 12:24:30.372197 (+509003us) spinlock_profiling.cc:243] Waited 28.1 ms on lock 0x7b1800292ea0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:24:30.907455 (+535258us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:30.907517 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:30.907634 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:30.964829 (+ 57195us) raft_consensus.cc:1685] finished
0709 12:24:30.964860 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:30.975502 (+ 10642us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":15321,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":18636,"spinlock_wait_cycles":64715520,"thread_start_us":535011,"threads_started":1,"wal-append.queue_time_us":1073404}
W20260709 12:24:30.985642 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:30.960278  7984 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:30.973517  7779 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:31.022751 31744 rpcz_store.cc:276] 0709 12:24:29.738970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.758580 (+ 19610us) service_pool.cc:224] Handling call
0709 12:24:29.758989 (+   409us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:30.123065 (+364076us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:30.123106 (+    41us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:30.123864 (+   758us) log.cc:844] Serialized 78 byte log entry
0709 12:24:30.515013 (+391149us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:30.900845 (+385832us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.935832 (+ 34987us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.017804 (+ 81972us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.017988 (+   184us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.018052 (+    64us) raft_consensus.cc:1685] finished
0709 12:24:31.018070 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.018261 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38786,"prepare.queue_time_us":312242,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":78585,"raft.run_cpu_time_us":287,"raft.run_wall_time_us":16008,"spinlock_wait_cycles":4948736,"thread_start_us":1046512,"threads_started":4,"wal-append.queue_time_us":587381}
I20260709 12:24:31.003786  8128 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:31.004067  8012 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:31.020733 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 144261us
I20260709 12:24:30.983315  7295 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:31.044059 31827 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:31.045821 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:31.042187 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 251996us
I20260709 12:24:31.019166  7755 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:31.017274  8045 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:31.009979  7756 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.034s
I20260709 12:24:31.010187  8161 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:31.066289 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3624) took 1297 ms. Trace:
I20260709 12:24:31.066578 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:31.068849 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 309078us
I20260709 12:24:31.042315  7995 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:31.071842 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:31.017395  8151 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.075892 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:31.080994  8216 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.070s	user 0.001s	sys 0.000s
I20260709 12:24:31.076754  8054 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:31.081928 31673 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:31.083613  7295 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:31.085191  8012 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.042088  8128 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:31.088014 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:31.088835  8128 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:31.089471 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.089802  8013 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:31.090302 31670 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:31.092123 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1363) took 1472 ms. Trace:
I20260709 12:24:31.093712 31748 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:24:31.092545 31671 rpcz_store.cc:276] 0709 12:24:29.619543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.625438 (+  5895us) service_pool.cc:224] Handling call
0709 12:24:29.625801 (+   363us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:29.626264 (+   463us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:29.626594 (+   330us) log.cc:844] Serialized 14 byte log entry
0709 12:24:29.865579 (+238985us) spinlock_profiling.cc:243] Waited 7.26 ms on lock 0x7b18004408c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:30.055541 (+189962us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:30.110338 (+ 54797us) log.cc:844] Serialized 52 byte log entry
0709 12:24:30.110516 (+   178us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:30.110809 (+   293us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.655597 (+544788us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b18004408c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:24:31.039512 (+383915us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.055673 (+ 16161us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.055777 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:31.091918 (+ 36141us) raft_consensus.cc:1685] finished
0709 12:24:31.091939 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.092066 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26746,"prepare.queue_time_us":162042,"prepare.run_cpu_time_us":129,"prepare.run_wall_time_us":2605,"raft.queue_time_us":136294,"raft.run_cpu_time_us":1491,"raft.run_wall_time_us":57132,"spinlock_wait_cycles":24303744,"thread_start_us":798583,"threads_started":3,"wal-append.queue_time_us":931307}
I20260709 12:24:31.094990 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:31.095326 31745 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 18. (term mismatch)
I20260709 12:24:31.095647 31745 pending_rounds.cc:85] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 17
I20260709 12:24:31.095871 31745 pending_rounds.cc:107] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 26.18
I20260709 12:24:31.095945 32377 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:31.096122 31745 raft_consensus.cc:2889] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:31.097373 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:31.097892  7295 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:31.102596 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:31.103262 31643 leader_election.cc:400] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25.
I20260709 12:24:31.103444 31746 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:24:31.107436 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:31.107465  8228 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:31.109210  8228 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:31.109560  8167 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:31.111897  7963 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:31.061411  8045 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:31.113173  8025 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:31.115109  8226 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.115674  8226 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.117900  8226 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:31.045794  8189 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.120639 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:31.124722 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.131532 31671 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:31.074771  8197 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:31.146763  8002 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:31.144603 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:31.130694  8211 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:31.153903 31669 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:31.167824 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:31.169358 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:31.172169 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67435.5us
I20260709 12:24:31.172969 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:31.109593  8228 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:31.174778 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.175957 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:31.176851 31673 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:31.177474 31819 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:31.177503 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:31.178802 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:31.179474 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.149380  8135 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:31.178692  8172 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.008s
I20260709 12:24:31.187857 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:31.191025 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:31.191277 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:31.194039 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.131776  8143 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.190397 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:31.197909  7959 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.003s
I20260709 12:24:31.200378 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.202999  8130 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:31.197782 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.205510 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:31.208993 31669 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:31.190796  8135 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.209589 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.209803  8223 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:31.210930 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:31.211040  8135 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:31.205741 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.205952 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:31.206499 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:31.212586 32369 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:31.148948  7995 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.213212 31823 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:31.214495 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.180737  7889 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:31.201280  8234 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.219000 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:31.196702  8201 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:31.171641  8211 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.217330 31823 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:31.185746  8025 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.034s
I20260709 12:24:31.215597  8010 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:31.210891 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:31.240384  8135 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:31.234349 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.234431 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:31.241752 31822 raft_consensus.cc:2368] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31.
I20260709 12:24:31.242129 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.242833 31823 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:31.242929 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.243657 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:31.244494 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.246121 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:31.235222 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.238478 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:31.244616 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:31.247824 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:31.247890 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.249058 31821 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:31.249136 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:31.249876 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.250432 31821 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:31.251396 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.252892 31819 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:31.252997 31821 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:31.252347 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:31.250295 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:31.254367 31819 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:24:31.270442 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.254410 31822 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:31.314580 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3616) took 1592 ms. Trace:
I20260709 12:24:31.336655 31749 rpcz_store.cc:276] 0709 12:24:29.721797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.752508 (+ 30711us) service_pool.cc:224] Handling call
0709 12:24:31.314312 (+1561804us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54001a5ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:31.314511 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2606681472}
I20260709 12:24:31.337198 31669 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
W20260709 12:24:31.315958 32376 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 1195) took 1738 ms (client timeout 2023 ms). Trace:
I20260709 12:24:31.305939 31824 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:31.262267 31826 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
W20260709 12:24:31.339335 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 89113.5us
I20260709 12:24:31.256527 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:31.364702 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:31.365207 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.365903 31676 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:31.366329 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:31.343554 31671 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:31.347594 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:31.347846  7766 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:31.369407 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:31.363291 32376 rpcz_store.cc:269] 0709 12:24:29.577752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.582722 (+  4970us) service_pool.cc:224] Handling call
0709 12:24:31.315707 (+1732985us) spinlock_profiling.cc:243] Waited 582 ms on lock 0x7b5400426dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:31.315858 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1337619840}
W20260709 12:24:31.349274 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 96991.4us
I20260709 12:24:31.370985 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.335896 31822 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:31.266196  8091 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:31.268090  8143 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:31.382206 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.255193  7984 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:31.352952 31819 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:31.280205  8211 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:31.348452  8010 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Leader pre-election won for term 17
I20260709 12:24:31.222770  8234 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.355463  8228 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.067875 31750 rpcz_store.cc:276] 0709 12:24:29.768419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:29.951934 (+183515us) service_pool.cc:224] Handling call
0709 12:24:29.966402 (+ 14468us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:29.966740 (+   338us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:24:29.967669 (+   929us) log.cc:844] Serialized 14 byte log entry
0709 12:24:30.959849 (+992180us) spinlock_profiling.cc:243] Waited 3.8 ms on lock 0x7b18003a1aa0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:24:31.065852 (+106003us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:31.065948 (+    96us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:31.066022 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.066091 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:31.066216 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":645,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":45,"raft.queue_time_us":1863,"spinlock_wait_cycles":8749824,"thread_start_us":105743,"threads_started":1}
W20260709 12:24:31.288734  8199 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.386s	user 0.000s	sys 0.002s
I20260709 12:24:31.239082  8012 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:31.255008  7295 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:31.402369  8012 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.403285 31676 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:31.404664 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.405373  8012 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:31.406070 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.377171 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.407550 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:31.407826 31718 leader_election.cc:400] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31.
I20260709 12:24:31.254808  7756 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:31.408504  8234 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:31.375918 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:31.309273  8045 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:31.351593 31821 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:24:31.358845  8182 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.384519 31822 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:31.381318  8091 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:31.374159  8230 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:31.388708  7766 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:31.396126  8010 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:31.319201  7779 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.389391  8132 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:24:31.275275  8161 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:31.378166  7944 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:24:31.390771  8128 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:31.391817 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:31.393148  6752 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.396310 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:31.397953 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.398095 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:31.402347  8206 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.004s
I20260709 12:24:31.271872  8002 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:31.374953 31792 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:31.265865 31823 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:31.409714 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:31.437742 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:31.448246 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:31.440335 31749 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:31.408815 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:31.419557 31672 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:24:31.450489 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 75624.2us
I20260709 12:24:31.410113 31822 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:31.441622 31669 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:31.477911 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:31.480669 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 71958.3us
I20260709 12:24:31.481572 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:31.484400 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:31.485885 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:31.457100 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.489574 31669 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:31.490386 31671 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: [24.13-25.14]
I20260709 12:24:31.440099  6752 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:31.494879 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:31.415211  8010 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:31.412454  7766 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:31.501806 31676 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:31.439863  8143 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.409771  8184 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:31.534549 31747 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:31.534863  8184 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:31.535305  8184 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:31.499106  7295 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:31.535755 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:31.536201  7295 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:31.536710  7295 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.537178 31747 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:31.538365  7963 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:31.483424  7805 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
W20260709 12:24:31.539263 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.505535  6752 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:31.539902 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:31.506706 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.542554 31747 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21. Ops: []
W20260709 12:24:31.512643 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 335183us
I20260709 12:24:31.543850 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:31.513597  7944 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:31.513949  8054 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:31.517195 31677 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 16. Preceding OpId from leader: term: 23 index: 17. (index mismatch)
I20260709 12:24:31.518069  7779 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:31.491067  7759 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.529875  8143 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:31.491710  8230 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:31.495682  7959 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.035s
I20260709 12:24:31.576031  7759 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:31.496177  8206 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:31.579167  8206 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:31.579732  8206 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:31.482898  8135 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.066s
I20260709 12:24:31.531169  7984 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:31.582978  7984 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:31.584148  7984 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:31.585777  7984 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:31.532011 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:31.587935 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:31.588985 31750 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:31.590085 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:31.491093  8223 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:31.593374  8223 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:31.594686  8223 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.20, Last appended by leader: 20, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.532766 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:31.495756  8045 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:31.533509 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:31.611630 31748 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:31.611781 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:31.612681 31751 raft_consensus.cc:2410] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 25 because replica has last-logged OpId of term: 24 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:24:31.512892 31670 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:31.411285  8182 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:31.548873 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54097.3us
I20260709 12:24:31.555660 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.552659 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:31.445102  8092 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.061s
I20260709 12:24:31.526465  7755 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:31.549409  7805 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:31.571215 31749 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:31.507745  7766 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:31.452996  8128 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:31.597556 31747 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:31.612607 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:31.539040 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:31.641256 31753 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:31.614046 31676 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:31.538954  7963 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:31.651082 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 98486.3us
I20260709 12:24:31.652758 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:31.655375 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 111659us
I20260709 12:24:31.616197  7295 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:31.633893 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:31.662925 31748 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 29: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:31.665359 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:31.662659 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:31.671118 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:31.672236 31749 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 31: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:31.681170 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:31.696571  6752 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:31.624997  8045 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:31.698414 31750 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:31.639442  7805 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:31.701121 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:31.697237  6752 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.646986  8228 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:31.722263 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57023.9us
I20260709 12:24:31.664768  8223 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:31.728524 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:31.702415 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:31.625387  8181 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:31.730784  6752 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:31.730557  8181 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.679884  8230 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:31.684398  8151 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:31.747310 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:31.755024 31753 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:31.757346 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:31.710417  8201 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:31.750190 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:31.649057  7755 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:31.750356 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.722584  7759 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:31.723824  8232 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:31.790454 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:31.757804  8243 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:24:31.748821  8222 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:24:31.802493 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52368us
I20260709 12:24:31.753898  8025 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:24:31.803947 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.762774  8201 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:31.885681 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 174048us
I20260709 12:24:31.725543  8045 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:31.710811  8250 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 21.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.141s
I20260709 12:24:31.918567 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:31.809777  7295 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:31.922127 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.853662  8161 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:24:31.905900 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:31.791991  8230 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:31.909698 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:31.759593  8228 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.116s
I20260709 12:24:31.791718  8151 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:31.914140 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:31.981063 31749 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:31.891561 31753 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:31.983248 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:31.984052 31753 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:31.932988 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:31.960145 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:31.960573 31751 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:31.912725  8201 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:31.963280 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:31.966255 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.969211 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:31.961799  8243 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.063s	user 0.001s	sys 0.000s
I20260709 12:24:31.999984 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:32.001029 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.978123 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:31.988152 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:31.920477  8091 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.510s	user 0.010s	sys 0.001s
I20260709 12:24:31.968179 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:32.013697 31818 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:31.995867 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:31.914275  8211 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.077s
I20260709 12:24:32.007938 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:32.012452 31669 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:31.913818  8232 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:32.042775 31674 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:32.045091 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:32.048875 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:32.049180 31818 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:31.933090  8172 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.079s
I20260709 12:24:31.886827  7766 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:31.975554  7295 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.051448 31669 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:24:32.053025 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:32.066025 31753 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:32.069471 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:32.071376 31753 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:32.072809 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:32.073132 31818 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:32.078101 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:31.970147  8000 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.078s
I20260709 12:24:31.981958  8167 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.076s
I20260709 12:24:31.960235  8163 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.090s
I20260709 12:24:32.089890 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:31.995281  8053 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.079s
I20260709 12:24:32.100468  7295 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:32.052839  8218 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:32.102243 31818 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:24:32.013181  8091 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:32.109445 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:32.110159 32376 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:32.112212  8254 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.178s	user 0.000s	sys 0.000s
I20260709 12:24:32.114097 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1374) took 1038 ms. Trace:
I20260709 12:24:32.114526 31678 rpcz_store.cc:276] 0709 12:24:31.075820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.076078 (+   258us) service_pool.cc:224] Handling call
0709 12:24:31.081590 (+  5512us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:31.511973 (+430383us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:31.512004 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:31.512477 (+   473us) log.cc:844] Serialized 26 byte log entry
0709 12:24:31.680279 (+167802us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:31.693573 (+ 13294us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.694470 (+   897us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.093886 (+399416us) spinlock_profiling.cc:243] Waited 93.8 ms on lock 0x7b54003921d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:32.093929 (+    43us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.113771 (+ 19842us) raft_consensus.cc:1685] finished
0709 12:24:32.113885 (+   114us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.114030 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":336114,"prepare.queue_time_us":16128,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":5344,"raft.queue_time_us":336445,"raft.run_cpu_time_us":187,"raft.run_wall_time_us":185,"spinlock_wait_cycles":215834496,"thread_start_us":108615,"threads_started":1,"wal-append.queue_time_us":229552}
I20260709 12:24:32.119553  8045 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:32.120640  7755 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:32.043485  7889 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.058s
W20260709 12:24:32.126245  8184 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.590s	user 0.004s	sys 0.004s
I20260709 12:24:32.087302  8143 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:32.132031  8128 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:32.135265  8246 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.213s	user 0.000s	sys 0.000s
I20260709 12:24:32.060379  7733 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.037s
W20260709 12:24:32.140406  8255 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.221s	user 0.000s	sys 0.000s
W20260709 12:24:32.141321  8247 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.246s	user 0.001s	sys 0.000s
I20260709 12:24:32.141656  8210 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.170s
W20260709 12:24:32.143105  8253 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
W20260709 12:24:32.143581  8251 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:24:32.143771  8174 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 17, Time since last communication: 0.184s
I20260709 12:24:32.144107  8059 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:32.015046  8197 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.083s
W20260709 12:24:32.144891  8244 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.343s	user 0.000s	sys 0.001s
W20260709 12:24:32.145747  8258 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.218s	user 0.000s	sys 0.000s
W20260709 12:24:32.146489  8249 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.370s	user 0.001s	sys 0.000s
I20260709 12:24:32.110939  8091 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:32.147825  8224 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 27.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.390s
I20260709 12:24:32.148104  7963 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:32.108382  8012 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.087s
I20260709 12:24:32.154330 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:32.100142  8262 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:32.177949 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 72516us
I20260709 12:24:32.097900  8189 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.191316 31749 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 30 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
I20260709 12:24:32.221632 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:32.097532  8264 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:32.186085 31751 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:32.196563 32376 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:32.147083  8232 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:32.201915 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 21 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:32.205044 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 122242us
I20260709 12:24:32.192823  8143 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:32.179517  8184 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:32.195091  8059 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:32.222455 31749 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:32.232079  7984 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.646s	user 0.012s	sys 0.002s
I20260709 12:24:32.097311  8230 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.13, Last appended by leader: 13, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.232895 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:32.252604 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.237157  8143 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:32.176885  8151 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.261655 31749 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:32.267212 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:32.243940  8182 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:32.268991 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 21 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:32.270028 31749 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:32.273978 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:32.258877  8210 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:32.269884  7944 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.707s	user 0.012s	sys 0.001s
I20260709 12:24:32.227198  8189 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.223294  8262 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:32.236750  8264 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.284173 31749 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: [23.13-25.14]
I20260709 12:24:32.300705 31749 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:32.293736 31744 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:32.319859 31746 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:32.321758 31746 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:32.337172 31746 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:32.340456 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1235) took 1213 ms. Trace:
I20260709 12:24:32.345404 31746 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:32.341058 32370 rpcz_store.cc:276] 0709 12:24:31.127116 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.127273 (+   157us) service_pool.cc:224] Handling call
0709 12:24:31.154094 (+ 26821us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:31.173497 (+ 19403us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:24:31.180971 (+  7474us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.728790 (+547819us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:31.729276 (+   486us) log.cc:844] Serialized 26 byte log entry
0709 12:24:31.729438 (+   162us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:31.729471 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.729541 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.340172 (+610631us) raft_consensus.cc:1685] finished
0709 12:24:32.340201 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.340373 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":66151,"prepare.queue_time_us":7560,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":39,"raft.queue_time_us":78716,"raft.run_cpu_time_us":273,"raft.run_wall_time_us":4007,"thread_start_us":473735,"threads_started":2,"wal-append.queue_time_us":504708}
W20260709 12:24:32.352090  8248 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.818s	user 0.003s	sys 0.000s
I20260709 12:24:32.354979  8143 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.356556  8143 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:32.357271 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:32.361877 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.364897  8189 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:32.292766  8059 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.366484  8059 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:32.367578 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:32.368994 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:32.370218  8135 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:32.370712  8135 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:32.372151  8264 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:32.373586  8010 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.814s	user 0.012s	sys 0.000s
I20260709 12:24:32.355540 31750 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: [25.16-27.17]
W20260709 12:24:32.375612 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 735840us
I20260709 12:24:32.375990 31750 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:24:32.377060 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:32.377447  7766 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:32.291890  8210 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:32.378837 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:32.379464 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:32.380937  8210 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:32.381548  8210 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:32.381676  8000 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:32.382304  8000 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:32.382503 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:32.316576  7944 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:32.383246  8000 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.395255 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:32.397383  6752 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:32.397151  8012 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:32.277235  8182 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:32.398595 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:32.401799 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:32.405807 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:32.428054 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:32.370502  8223 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.620s	user 0.010s	sys 0.000s
I20260709 12:24:32.401746 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:32.422927 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:32.424500 32375 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:32.445251 31752 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:32.442576 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:32.439482 32378 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:32.374068  8010 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:32.447469 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:32.448015 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:32.450769 31827 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:24:32.453845  8272 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.170s	user 0.000s	sys 0.000s
W20260709 12:24:32.454221  8266 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
W20260709 12:24:32.455044  8252 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.481s	user 0.000s	sys 0.002s
I20260709 12:24:32.455401  8210 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:32.456003  8275 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:32.457831  6752 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:32.463936 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.464223 32371 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 29: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:32.466946 32377 raft_consensus.cc:2410] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 27 because replica has last-logged OpId of term: 26 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
I20260709 12:24:32.424171  8163 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.442905  8218 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:32.527973 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:32.529292 32373 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:24:32.398861  8273 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.121s
W20260709 12:24:32.519306  8206 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.939s	user 0.011s	sys 0.007s
W20260709 12:24:32.523084  7805 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.775s	user 0.016s	sys 0.001s
I20260709 12:24:32.518330  8262 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:32.546002 32374 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: []
W20260709 12:24:32.526888  8249 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:24:32.587126  8059 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
W20260709 12:24:32.589293 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 141913us
I20260709 12:24:32.597555 31818 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:32.610534 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:32.605180  6752 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.620137 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:32.597234 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:32.595912  8155 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:32.551633  8184 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:32.551599 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 103711us
I20260709 12:24:32.628278 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:32.628264 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:32.595902  8218 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:32.545128  8201 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.530s	user 0.011s	sys 0.000s
I20260709 12:24:32.598871  8210 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:32.625901  7733 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:32.630074 31747 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:32.640806 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:32.688686 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:32.689414 31827 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:24:32.659653 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1260) took 1287 ms. Trace:
I20260709 12:24:32.657330 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:32.606993  8025 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:32.695924 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 55192.2us
W20260709 12:24:32.657312  8276 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.136s	user 0.000s	sys 0.001s
I20260709 12:24:32.604517  8163 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.690761 31745 rpcz_store.cc:276] 0709 12:24:31.372185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.372334 (+   149us) service_pool.cc:224] Handling call
0709 12:24:31.372810 (+   476us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:31.387450 (+ 14640us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:31.387792 (+   342us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.486732 (+ 98940us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.491807 (+  5075us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:31.492483 (+   676us) log.cc:844] Serialized 52 byte log entry
0709 12:24:31.492666 (+   183us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:31.493089 (+   423us) log.cc:844] Serialized 14 byte log entry
0709 12:24:31.493339 (+   250us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:31.493434 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.089171 (+595737us) spinlock_profiling.cc:243] Waited 3.47 ms on lock 0x7b18004e51a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:32.380167 (+290996us) spinlock_profiling.cc:243] Waited 4.31 ms on lock 0x7b18004e51a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8557 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:32.659349 (+279182us) raft_consensus.cc:1685] finished
0709 12:24:32.659384 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.659575 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40227,"prepare.queue_time_us":22738,"prepare.run_cpu_time_us":124,"prepare.run_wall_time_us":5755,"raft.queue_time_us":38643,"raft.run_cpu_time_us":346,"raft.run_wall_time_us":20328,"spinlock_wait_cycles":25395072,"thread_start_us":79631,"threads_started":1,"wal-append.queue_time_us":295622}
I20260709 12:24:32.603938  8206 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.657086  7963 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:32.654994  8012 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.722584 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:32.727324 31827 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 27: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:32.692937  8184 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.632350  8218 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:32.607663  7805 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:32.731057 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:32.736891 31747 raft_consensus.cc:2410] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 22 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:24:32.631065  8174 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.094s
I20260709 12:24:32.824101  8174 consensus_queue.cc:1243] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 22.13 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 12
I20260709 12:24:32.778563  8184 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:32.776613 31822 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:32.656690  8201 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.775619  8274 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:32.787084 31748 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 29: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:32.791725 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:32.698987  8167 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.077s
I20260709 12:24:32.814111 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:32.846866 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.697489  7779 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:32.822921  8012 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.705291  8025 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:32.631810  8197 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.097s
I20260709 12:24:32.844151 31826 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:32.813561  8182 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:32.859440 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:32.865617 31827 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
W20260709 12:24:32.863914  8278 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
W20260709 12:24:32.860695  8280 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
W20260709 12:24:32.887416 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:32.869417 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 1.49105e+06us
I20260709 12:24:32.894793 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:32.895231 32378 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:24:32.895522 32375 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:32.892211 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:32.902029 32377 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 16. Current term is 17. Ops: []
I20260709 12:24:32.902408 32371 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:24:32.909639 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:32.911901 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:32.913900 31675 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:32.916288 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:32.917734 32371 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:32.918879 31671 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
W20260709 12:24:32.922412 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 221426us
I20260709 12:24:32.806993  8210 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.923995 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:32.926672 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:32.928615 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:32.929173 32369 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:32.929708  7963 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:32.930299  7963 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:32.931229  7963 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:32.935305  8010 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:32.938225  8012 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:32.939459  8287 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:24:32.939947 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:32.940654 31822 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
W20260709 12:24:32.943153  7766 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.565s	user 0.012s	sys 0.001s
W20260709 12:24:32.945453  8143 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.588s	user 0.009s	sys 0.003s
W20260709 12:24:32.947971 31748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2930) took 1283 ms (client timeout 1514 ms). Trace:
W20260709 12:24:32.948448 31748 rpcz_store.cc:269] 0709 12:24:31.664945 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.680784 (+ 15839us) service_pool.cc:224] Handling call
0709 12:24:32.947726 (+1266942us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54002f76d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:32.947885 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2540580096}
W20260709 12:24:32.949501 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:32.918038  8230 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:32.950970 31747 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: [28.15-28.15]
I20260709 12:24:32.951915 31753 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:32.900977  7944 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:32.957010 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:32.963493 31670 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:32.960876 32375 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:32.961707 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:32.929131 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:32.903157  8182 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:32.968907 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:32.969964  8284 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.179s	user 0.001s	sys 0.000s
I20260709 12:24:32.971882 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3676) took 1005 ms. Trace:
W20260709 12:24:32.973453 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44394.3us
W20260709 12:24:32.951915  8289 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:24:32.850170  8274 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.970381 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:32.972286 31819 rpcz_store.cc:276] 0709 12:24:31.966333 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.966443 (+   110us) service_pool.cc:224] Handling call
0709 12:24:32.009204 (+ 42761us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.196505 (+187301us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.196537 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.211054 (+ 14517us) log.cc:844] Serialized 26 byte log entry
0709 12:24:32.318187 (+107133us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:32.318303 (+   116us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.318501 (+   198us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:32.971620 (+653119us) raft_consensus.cc:1685] finished
0709 12:24:32.971647 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:32.971816 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":173413,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":11159,"thread_start_us":275339,"threads_started":2,"wal-append.queue_time_us":402494}
I20260709 12:24:32.963232  7944 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:32.994989  7779 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.005s
I20260709 12:24:32.972426  7805 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:32.972506  8167 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:32.998014  8206 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:33.003372  8092 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:32.973690  8151 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:32.992905  8223 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:32.988878  7767 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.989391 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:32.997090  8143 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:32.963943  8271 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.046s
I20260709 12:24:32.958739  8230 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:32.992736  8174 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.034s
W20260709 12:24:33.035466 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 46225.7us
I20260709 12:24:32.990489  8182 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:33.050998 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 122454us
I20260709 12:24:33.021387  8151 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.026135  8223 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:33.056449 31792 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.640s (SENT) [suppressed 8 similar messages]
I20260709 12:24:33.043141  8167 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:33.076040 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:33.076534  8167 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.030084  7805 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:33.046604  8163 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:33.039485  7767 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.080097  8182 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:33.076886 32369 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:33.015528  8257 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.034s
I20260709 12:24:33.086822 32377 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:33.043995  7779 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:33.094877  8167 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:33.054663  8184 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:33.081087  8151 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:33.076145  8223 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:33.037559  8143 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.108752 32369 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:33.121464 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3643) took 1039 ms. Trace:
W20260709 12:24:33.121565 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 151251us
I20260709 12:24:33.121935 31750 rpcz_store.cc:276] 0709 12:24:32.081853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.377709 (+295856us) service_pool.cc:224] Handling call
0709 12:24:32.378002 (+   293us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.456488 (+ 78486us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:32.456827 (+   339us) log.cc:844] Serialized 14 byte log entry
0709 12:24:32.631503 (+174676us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.632169 (+   666us) log.cc:844] Serialized 26 byte log entry
0709 12:24:32.632314 (+   145us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:32.632351 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.632430 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.121243 (+488813us) raft_consensus.cc:1685] finished
0709 12:24:33.121265 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.121405 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":73051,"prepare.queue_time_us":2414,"prepare.run_cpu_time_us":544,"prepare.run_wall_time_us":5559,"raft.queue_time_us":82450,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":35934,"thread_start_us":172567,"threads_started":1,"wal-append.queue_time_us":333619}
I20260709 12:24:33.129040 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:33.130352 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:33.131893 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 75479.3us
I20260709 12:24:33.133244 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:33.120930  8275 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.513s	user 0.007s	sys 0.004s
I20260709 12:24:33.138824 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.083968  7944 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.123031  8264 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.148389  7944 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:33.148551  8264 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.123669  8206 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:33.151068  8206 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.135299  8299 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.026s
I20260709 12:24:33.131456  8274 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:33.137176  8163 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 16.11, Next index: 12, Last known committed idx: 11, Time since last communication: 0.018s
I20260709 12:24:33.109181  8025 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:33.156234  8163 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:33.136101  8155 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:33.154597  8274 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:33.158530  8226 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Stepping down as leader of term 16
I20260709 12:24:33.159004  8226 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 16 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:33.159438  8296 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.137s	user 0.001s	sys 0.000s
I20260709 12:24:33.159742  8206 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:33.159965  8226 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 16, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.160859 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3646) took 1037 ms. Trace:
I20260709 12:24:33.161123  8264 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:33.167999  8059 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.544s	user 0.003s	sys 0.008s
I20260709 12:24:33.168601  8059 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:33.139431  8271 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 20.11, Next index: 12, Last known committed idx: 10, Time since last communication: 0.016s
I20260709 12:24:33.161298 31745 rpcz_store.cc:276] 0709 12:24:32.123581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.705882 (+582301us) service_pool.cc:224] Handling call
0709 12:24:32.706260 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.706610 (+   350us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.711492 (+  4882us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.728003 (+ 16511us) log.cc:844] Serialized 26 byte log entry
0709 12:24:32.926134 (+198131us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:32.926187 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.926268 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.160616 (+234348us) raft_consensus.cc:1685] finished
0709 12:24:33.160640 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.160780 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":161745,"raft.queue_time_us":82,"raft.run_cpu_time_us":163,"raft.run_wall_time_us":10248,"thread_start_us":107957,"threads_started":1,"wal-append.queue_time_us":192650}
I20260709 12:24:33.136072 31792 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:33.123540  8273 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:33.185057 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 49055.9us
I20260709 12:24:33.181915 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.183858 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:33.127983  7766 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.201259  7766 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.192727  8274 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:33.202615  8274 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:33.203598 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:33.203665  8274 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.169169  8275 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:33.169658  7805 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:24:33.206269  7805 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:33.206763  7805 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:33.207683  7805 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.208757  7805 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:33.167892  7767 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:33.211884  8295 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:33.212380  8295 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:33.213369  8295 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.189949  8273 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.216288 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1536) took 1123 ms. Trace:
I20260709 12:24:33.216714 31818 rpcz_store.cc:276] 0709 12:24:32.092402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.138531 (+ 46129us) service_pool.cc:224] Handling call
0709 12:24:32.138955 (+   424us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.619390 (+480435us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.216021 (+596631us) spinlock_profiling.cc:243] Waited 457 ms on lock 0x7b54001987d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:33.216223 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1051666688}
I20260709 12:24:33.217456  8273 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:33.218029 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:33.201303  8135 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.204402 31823 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:33.205492 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3654) took 1079 ms. Trace:
I20260709 12:24:33.238677 31753 rpcz_store.cc:276] 0709 12:24:32.125716 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.232536 (+106820us) service_pool.cc:224] Handling call
0709 12:24:33.205326 (+972790us) spinlock_profiling.cc:243] Waited 699 ms on lock 0x7b54000b95d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:33.205434 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1608297728}
I20260709 12:24:33.240207 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:33.241811 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:33.244807 31753 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 28: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:33.223465  7766 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:33.224555  7756 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:33.224768 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.226298 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3677) took 1897 ms. Trace:
I20260709 12:24:33.247311 31676 rpcz_store.cc:276] 0709 12:24:31.328828 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.547737 (+218909us) service_pool.cc:224] Handling call
0709 12:24:31.548243 (+   506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:31.642436 (+ 94193us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.225962 (+1583526us) spinlock_profiling.cc:243] Waited 65.6 ms on lock 0x7b54001815d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:33.226220 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":150876928}
I20260709 12:24:33.228754 31750 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:33.192317  8303 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Stepping down as leader of term 20
W20260709 12:24:33.190318  8253 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.053s	user 0.000s	sys 0.000s
I20260709 12:24:33.258206 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:33.199371  8218 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.182617  8264 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.161666  8226 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:33.246717  8295 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:33.246477 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 157338us
I20260709 12:24:33.243503 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:33.257884 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:33.260270 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:33.237866  8228 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.218s	user 0.004s	sys 0.004s
I20260709 12:24:33.196079  8143 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:33.275669 31675 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:33.258826  8271 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:33.268575  8206 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:33.246294  7959 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.272486 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 17 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.267618  8012 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:33.272183 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:33.263530 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 116893us
I20260709 12:24:33.262401  7766 raft_consensus.cc:2764] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:24:33.283226 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:33.274274  8228 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.266s	user 0.005s	sys 0.004s
I20260709 12:24:33.287388  8228 consensus_queue.cc:1243] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 25.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:24:33.287132  8223 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.288192 31745 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:33.282226 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:33.290160 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:33.257093  8303 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 20 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:33.290834  8059 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:33.291265  8059 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:33.291716  8059 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:33.291329  8303 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 20.11, Last appended by leader: 11, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.292495 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:33.293682  8174 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:33.294076 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:33.294210  8174 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.294930  8275 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:33.295378  8275 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.295734  8275 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:33.296366  8174 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:33.297634 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:33.262490  8273 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:33.292631  8303 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:33.305116  8223 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:33.305780  8211 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:33.306250  8211 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:33.307152  8211 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.13, Last appended by leader: 13, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.308199  8211 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:33.311021  8230 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:33.298236  8174 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:24:33.283998 31753 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
W20260709 12:24:33.263427 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 81608.3us
I20260709 12:24:33.318656 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.300920  8012 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:33.301460  8222 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:33.303654 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:33.304271  8053 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:33.273703  8182 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:33.270807  8264 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.342553  8264 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:33.343634  8151 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.345755  8151 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:33.303833  8302 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260709 12:24:33.348969 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:33.256742  7756 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.320195 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:33.375371 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 55259.4us
I20260709 12:24:33.322306 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:33.376734 31753 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:33.280812  8184 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader election lost for term 17. Reason: could not achieve majority
I20260709 12:24:33.385332  8181 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:33.354794  8201 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:33.374353  8309 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:33.388809  7756 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:33.326092 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:33.389583 32377 raft_consensus.cc:2376] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27. Re-sending same reply.
W20260709 12:24:33.391167  8304 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.224s	user 0.002s	sys 0.000s
I20260709 12:24:33.388850  8309 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:33.392413  8309 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:33.392575 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3688) took 1002 ms. Trace:
I20260709 12:24:33.393018 31822 rpcz_store.cc:276] 0709 12:24:32.390232 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.941371 (+551139us) service_pool.cc:224] Handling call
0709 12:24:32.941734 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.959577 (+ 17843us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.959630 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.967562 (+  7932us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.089804 (+122242us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:33.090350 (+   546us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.090438 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.390546 (+300108us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:33.392265 (+  1719us) raft_consensus.cc:1685] finished
0709 12:24:33.392303 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.392515 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17324,"raft.queue_time_us":26637,"raft.run_cpu_time_us":154,"raft.run_wall_time_us":6633,"spinlock_wait_cycles":40083200,"thread_start_us":91764,"threads_started":1,"wal-append.queue_time_us":156927}
I20260709 12:24:33.395470 31822 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:33.396238 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3506) took 1004 ms. Trace:
I20260709 12:24:33.396627 31822 rpcz_store.cc:276] 0709 12:24:32.391757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.394945 (+1003188us) service_pool.cc:224] Handling call
0709 12:24:33.395269 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.395948 (+   679us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.396172 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:33.398132 31822 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:33.318557 31750 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:24:33.328039 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.316685  8155 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.323937  8222 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:33.325485  8012 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.329860  8218 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:33.333053 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:33.336552 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:33.416000 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:33.337966 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3636) took 1528 ms. Trace:
I20260709 12:24:33.417026 31746 rpcz_store.cc:276] 0709 12:24:31.809259 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.348551 (+539292us) service_pool.cc:224] Handling call
0709 12:24:32.386724 (+ 38173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:32.386910 (+   186us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.386941 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:32.386985 (+    44us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:32.387013 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:32.387071 (+    58us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.337774 (+950703us) spinlock_profiling.cc:243] Waited 37.8 ms on lock 0x7b54004047d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:33.337902 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":87006080}
I20260709 12:24:33.339457 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:33.298749  7959 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.305096  8273 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.382972 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:33.423000 32370 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:33.389052  8054 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:33.399296 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:33.427296 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.401616 31753 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:33.428989 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:33.404069 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3692) took 1007 ms. Trace:
W20260709 12:24:33.407603 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 79634.9us
I20260709 12:24:33.431852 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.432536 31822 rpcz_store.cc:276] 0709 12:24:32.397091 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.397543 (+1000452us) service_pool.cc:224] Handling call
0709 12:24:33.397907 (+   364us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.398602 (+   695us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.403984 (+  5382us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:33.410359 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 152228us
I20260709 12:24:33.419287 31825 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:33.418354 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:33.421355 32377 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:33.416615 31749 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:33.430748 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:33.437619 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:33.438263 31749 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:33.395027  8182 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:33.440135 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:33.440971 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:33.443359 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.439656 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3653) took 1314 ms. Trace:
I20260709 12:24:33.410343  8151 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:33.449323 31749 rpcz_store.cc:276] 0709 12:24:32.125465 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.438898 (+1313433us) service_pool.cc:224] Handling call
0709 12:24:33.439179 (+   281us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.439350 (+   171us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:33.439375 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:33.439415 (+    40us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:33.439453 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.439503 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.439603 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:33.453047 31746 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 13. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:33.451217 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3656) took 1325 ms. Trace:
I20260709 12:24:33.463214 31749 rpcz_store.cc:276] 0709 12:24:32.126303 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.450415 (+1324112us) service_pool.cc:224] Handling call
0709 12:24:33.450728 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.450916 (+   188us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:33.450942 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:33.450984 (+    42us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:33.451014 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.451066 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.451166 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:33.464622 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3657) took 1338 ms. Trace:
I20260709 12:24:33.465013 31746 rpcz_store.cc:276] 0709 12:24:32.126577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.452455 (+1325878us) service_pool.cc:224] Handling call
0709 12:24:33.452735 (+   280us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.453431 (+   696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.464564 (+ 11133us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:33.467497  8181 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:33.468080  8181 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:33.468511  8172 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:33.468901  8172 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:33.469005  8181 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.469766  8172 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.437808  8303 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:33.471665  8275 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:33.474020  8275 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:33.475078 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.475904  8184 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:33.477175  8264 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.477849  8264 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.480113  7959 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:33.480459  8264 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:33.480719  8269 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:33.464479 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:33.484676 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.485924 31825 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 20.
I20260709 12:24:33.488811 31825 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:33.490792 31825 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:33.492372 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.478310  8012 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:33.493469 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.426321  8054 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:33.495098 31825 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.502303 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.410132  8155 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.501509 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.504856 32370 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:24:33.451584  8182 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.506063  8182 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:33.507088 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:33.507918  8012 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:33.508292  8012 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.508671  8012 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:33.485749  7767 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:33.510754 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.511452 31825 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:33.520095 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.522004 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.487772  8303 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:33.513441 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3689) took 1123 ms. Trace:
I20260709 12:24:33.536063 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:33.538373 31825 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 15. (term mismatch)
I20260709 12:24:33.540172 31825 pending_rounds.cc:85] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 14
I20260709 12:24:33.538456 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.503127  7944 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:33.502722  8226 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:33.550155 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3690) took 1159 ms. Trace:
I20260709 12:24:33.544647 31825 pending_rounds.cc:107] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 26.15
I20260709 12:24:33.596458 31825 raft_consensus.cc:2889] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:33.597494 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.600198 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:33.505403  8059 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.532557  8232 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:33.540149  8230 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:33.544584  8245 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:33.559594  7756 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:33.556341  8228 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:24:33.561192  8273 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:33.628845 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 199971us
I20260709 12:24:33.629839 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.549487  8316 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:33.593744 31818 rpcz_store.cc:276] 0709 12:24:32.390858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.268517 (+877659us) service_pool.cc:224] Handling call
0709 12:24:33.279560 (+ 11043us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.279901 (+   341us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:33.279931 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:33.283301 (+  3370us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.415429 (+132128us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:33.415489 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.416439 (+   950us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.549931 (+133492us) raft_consensus.cc:1685] finished
0709 12:24:33.549954 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.550094 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":572,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":17107,"thread_start_us":125702,"threads_started":1,"wal-append.queue_time_us":264963}
I20260709 12:24:33.593029  8269 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:33.583721  8206 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.540727 31819 rpcz_store.cc:276] 0709 12:24:32.390493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.002759 (+612266us) service_pool.cc:224] Handling call
0709 12:24:33.021628 (+ 18869us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.021916 (+   288us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:33.042448 (+ 20532us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.194947 (+152499us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x7b180015c0a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:24:33.512997 (+318050us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:33.513077 (+    80us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:33.513143 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.513238 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.513391 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":16805,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":5499,"raft.queue_time_us":4692,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":10316,"spinlock_wait_cycles":3668736,"thread_start_us":305096,"threads_started":1}
I20260709 12:24:33.659232 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:33.662104 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:33.621121  7805 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:33.622360  8130 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:24:33.660545  7756 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:33.738555 31825 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:24:33.622565  8230 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:33.741616  8230 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:33.745407  7805 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:33.665019  8053 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:33.746006  7805 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:33.750957  7295 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:33.751691  7295 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.753750  7295 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:33.746933  8053 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.755861  8053 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:33.759641  8143 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:24:33.766979  8320 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
W20260709 12:24:33.768214  8318 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.153s	user 0.000s	sys 0.000s
I20260709 12:24:33.548401  8054 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.773895  8054 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:33.549053  8303 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:33.714602 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.664644  8226 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:33.604167  8271 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.060s
I20260709 12:24:33.622478  8245 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:33.675544 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.603243  8151 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.059s
W20260709 12:24:33.792043 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:33.800976 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.623696  7944 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:33.793385 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 352473us
I20260709 12:24:33.739667 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:33.739629  7756 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:33.734622  7963 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Leader election won for term 20
W20260709 12:24:33.784126  8312 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.213s	user 0.000s	sys 0.001s
I20260709 12:24:33.806259 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:33.809952 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:33.657166  8316 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.811316 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:33.833835 31825 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:24:33.688969  8206 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.662377  7959 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:33.792695  8143 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:33.736696  7767 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.071s
I20260709 12:24:33.838701 31819 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:33.803894  8245 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:33.689051  8225 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.123s
I20260709 12:24:33.840088  8232 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.018s
I20260709 12:24:33.848408  7959 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.807410  7805 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.859206 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:33.860426  7805 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:33.861042 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:33.861949 31819 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:33.854496  8059 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:33.863958  8275 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:33.832188 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:33.864306  8309 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.865929 31822 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:33.867780  8309 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:33.864457  8275 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.870517 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:33.871024  8275 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:33.874969 31826 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Stepping down as leader of term 24
W20260709 12:24:33.884603 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 362685us
I20260709 12:24:33.842095  8316 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:33.886675 31826 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:33.887506  7295 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
W20260709 12:24:33.891239 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:33.898177  7295 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:24:33.898900  7295 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:33.903090 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3658) took 1720 ms. Trace:
I20260709 12:24:33.905403 31749 rpcz_store.cc:276] 0709 12:24:32.182798 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.466852 (+1284054us) service_pool.cc:224] Handling call
0709 12:24:33.467180 (+   328us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.467391 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:33.662075 (+194684us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.902544 (+240469us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:33.902606 (+    62us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:33.902677 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.902758 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.902977 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23120,"prepare.queue_time_us":80518,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":113,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":109,"thread_start_us":402500,"threads_started":2}
I20260709 12:24:33.911453 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:33.914050 31749 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:33.915970 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:33.811383  7944 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.849009  8155 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:33.916188 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:33.885644  8232 consensus_queue.cc:1243] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 26.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:24:33.882862  7959 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:33.892783 31826 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:33.941392  8181 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:33.941874  8181 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:33.942294  8181 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:33.942620  7514 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:33.943864  7514 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:33.945240  8182 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.948035  8182 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:33.849195  8143 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.951435  8143 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:33.952054  8181 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:33.952662 31751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2935) took 2224 ms (client timeout 1640 ms). Trace:
W20260709 12:24:33.953058 31751 rpcz_store.cc:269] 0709 12:24:31.728439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.753275 (+ 24836us) service_pool.cc:224] Handling call
0709 12:24:33.952454 (+2199179us) spinlock_profiling.cc:243] Waited 202 ms on lock 0x7b54002f71d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:33.952564 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":465713024}
I20260709 12:24:33.937526  7295 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.958753  7295 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:33.964069  8269 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:33.982002 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:33.985430  8181 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:33.967670  8228 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:33.988763  8228 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:33.987279  8269 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:33.993474  8321 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.182s	user 0.000s	sys 0.001s
I20260709 12:24:33.995308 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3691) took 1604 ms. Trace:
W20260709 12:24:33.995589  8323 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.195s	user 0.001s	sys 0.000s
I20260709 12:24:33.996845 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:33.997591 31822 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
W20260709 12:24:33.985814  8313 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.252s	user 0.000s	sys 0.001s
I20260709 12:24:33.998472 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:33.995862 31824 rpcz_store.cc:276] 0709 12:24:32.391137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.350025 (+958888us) service_pool.cc:224] Handling call
0709 12:24:33.350380 (+   355us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:33.377677 (+ 27297us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:24:33.383988 (+  6311us) log.cc:844] Serialized 14 byte log entry
0709 12:24:33.607737 (+223749us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:33.620964 (+ 13227us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.621156 (+   192us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:24:33.621209 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.621363 (+   154us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.995004 (+373641us) raft_consensus.cc:1685] finished
0709 12:24:33.995033 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:33.995221 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22455,"prepare.queue_time_us":6121,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":34790,"raft.run_cpu_time_us":203,"raft.run_wall_time_us":9176,"thread_start_us":219349,"threads_started":1,"wal-append.queue_time_us":344879}
I20260709 12:24:34.004068 31825 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:34.004980 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:34.005715 31824 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
W20260709 12:24:34.009064  8222 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.588s	user 0.010s	sys 0.000s
I20260709 12:24:34.009627  8222 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:34.013016 31744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2907) took 2485 ms (client timeout 2122 ms). Trace:
W20260709 12:24:34.013458 31744 rpcz_store.cc:269] 0709 12:24:31.527109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.532502 (+  5393us) service_pool.cc:224] Handling call
0709 12:24:34.012773 (+2480271us) spinlock_profiling.cc:243] Waited 663 ms on lock 0x7b5400294fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:34.012938 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1525585152}
I20260709 12:24:34.016235 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3685) took 1626 ms. Trace:
I20260709 12:24:34.016672 31827 rpcz_store.cc:276] 0709 12:24:32.389432 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.868768 (+479336us) service_pool.cc:224] Handling call
0709 12:24:32.951543 (+ 82775us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.970958 (+ 19415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.970995 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.971561 (+   566us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.015596 (+ 44035us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:33.015636 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.015710 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:33.150752 (+135042us) raft_consensus.cc:1685] finished
0709 12:24:33.150784 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.015879 (+865095us) spinlock_profiling.cc:243] Waited 60.9 ms on lock 0x7b54004470d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:34.016155 (+   276us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18924,"raft.queue_time_us":67385,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":5362,"spinlock_wait_cycles":140056832,"thread_start_us":43639,"threads_started":1,"wal-append.queue_time_us":114808}
I20260709 12:24:34.024696  7756 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.014535 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:34.033298 31744 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:24:34.043489  8206 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:34.045680 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:34.048400 31744 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:24:34.055195 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:34.056082 31744 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 28
W20260709 12:24:34.031977  8012 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.514s	user 0.007s	sys 0.001s
I20260709 12:24:34.053995 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:34.083734 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3655) took 1957 ms. Trace:
I20260709 12:24:34.039677  7766 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.084231 31753 rpcz_store.cc:276] 0709 12:24:32.126013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.441340 (+1315327us) service_pool.cc:224] Handling call
0709 12:24:33.441669 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.441910 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:33.806394 (+364484us) log.cc:844] Serialized 14 byte log entry
0709 12:24:34.083349 (+276955us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:34.083410 (+    61us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:34.083468 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.083536 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.083672 (+   136us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":105924,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":38,"raft.queue_time_us":6005,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":107,"thread_start_us":594796,"threads_started":2}
I20260709 12:24:34.096158 31744 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:24:34.099617 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:34.127562 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1266) took 1001 ms. Trace:
I20260709 12:24:34.128064 31745 rpcz_store.cc:276] 0709 12:24:33.125922 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.260020 (+134098us) service_pool.cc:224] Handling call
0709 12:24:34.127370 (+867350us) spinlock_profiling.cc:243] Waited 15.2 ms on lock 0x7b54000f3cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:34.127499 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":34958208}
I20260709 12:24:34.129747 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:34.130153 31745 pending_rounds.cc:85] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 12
W20260709 12:24:34.100728 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 441544us
I20260709 12:24:34.084199  8264 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:34.087684  8257 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:34.133314 31745 pending_rounds.cc:107] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 22.13
I20260709 12:24:34.091638  8265 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:34.085268  8012 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.152711 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:34.133785 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 267913us
W20260709 12:24:34.177646  8300 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.042s	user 0.005s	sys 0.000s
I20260709 12:24:34.149775 31749 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:34.156514 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.139712 31745 raft_consensus.cc:2889] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:34.134464  8184 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:24:34.158466  8331 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.167s	user 0.000s	sys 0.001s
W20260709 12:24:34.158599  8334 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
W20260709 12:24:34.212786 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.163571 31751 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:34.165138  8012 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:34.167325  8332 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
W20260709 12:24:34.168231  8333 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
I20260709 12:24:34.096763  7766 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.140178  8181 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:34.142782  7756 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:34.140297  8265 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:34.143399  8257 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:34.148797 31752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2912) took 2620 ms (client timeout 2608 ms). Trace:
W20260709 12:24:34.228273 31752 rpcz_store.cc:269] 0709 12:24:31.528587 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:31.539665 (+ 11078us) service_pool.cc:224] Handling call
0709 12:24:34.148524 (+2608859us) spinlock_profiling.cc:243] Waited 843 ms on lock 0x7b54001a4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:34.148704 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1940014464}
W20260709 12:24:34.222589  8267 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.988s	user 0.017s	sys 0.021s
I20260709 12:24:34.187441 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:34.212246  8013 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:34.229918 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:34.214237  8184 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:34.213282  8329 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.211059  7963 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:34.234105 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:34.239189 31752 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:34.239401 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3649) took 2115 ms. Trace:
I20260709 12:24:34.187461 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3647) took 2063 ms. Trace:
I20260709 12:24:34.189296  8245 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.239888 31748 rpcz_store.cc:276] 0709 12:24:32.124404 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.952807 (+828403us) service_pool.cc:224] Handling call
0709 12:24:32.953136 (+   329us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.953582 (+   446us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:32.953622 (+    40us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:32.954156 (+   534us) log.cc:844] Serialized 26 byte log entry
0709 12:24:33.029112 (+ 74956us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:33.029172 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:33.029284 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:34.026566 (+997282us) spinlock_profiling.cc:243] Waited 46.2 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:34.237012 (+210446us) raft_consensus.cc:1685] finished
0709 12:24:34.237037 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.239327 (+  2290us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4541,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":9578,"spinlock_wait_cycles":106323712,"thread_start_us":59639,"threads_started":1,"wal-append.queue_time_us":119219}
I20260709 12:24:34.243101 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:34.243743 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.243680 31748 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:34.245280 31748 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:34.246163 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3666) took 1346 ms. Trace:
W20260709 12:24:34.235181 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47885.7us
I20260709 12:24:34.246870 31748 rpcz_store.cc:276] 0709 12:24:32.899709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.244430 (+1344721us) service_pool.cc:224] Handling call
0709 12:24:34.244847 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.245828 (+   981us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.246094 (+   266us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:34.235498 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.248828 31748 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:34.240064 31747 rpcz_store.cc:276] 0709 12:24:32.123858 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.776670 (+652812us) service_pool.cc:224] Handling call
0709 12:24:32.950712 (+174042us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:32.952126 (+  1414us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.187252 (+1235126us) spinlock_profiling.cc:243] Waited 165 ms on lock 0x7b54002f76d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:34.187395 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":379504512}
I20260709 12:24:34.251451 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:34.253633  8225 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:34.254256  8225 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.254818  8225 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:34.258977  8174 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:24:34.250296 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3667) took 1350 ms. Trace:
I20260709 12:24:34.268455 31748 rpcz_store.cc:276] 0709 12:24:32.899947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.247963 (+1348016us) service_pool.cc:224] Handling call
0709 12:24:34.248532 (+   569us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.249976 (+  1444us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.250229 (+   253us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:34.276434 31752 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
W20260709 12:24:34.303812 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55749.7us
I20260709 12:24:34.304916 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:34.224887  8181 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:34.302735  8230 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.561s	user 0.006s	sys 0.006s
I20260709 12:24:34.309948 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:34.315531 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:34.240458  8222 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:34.322778 31748 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:34.301987  7944 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:34.325053 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2942) took 1368 ms. Trace:
W20260709 12:24:34.326678 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 265851us
I20260709 12:24:34.310313  8013 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:34.330041 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:34.313478  8181 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:34.330734 31791 connection.cc:256] RPC call timeout handler was delayed by -1.3682690000000548s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2122s for a response
I20260709 12:24:34.339444 31671 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:34.331146 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:34.334440 31748 rpcz_store.cc:276] 0709 12:24:32.956512 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.313290 (+1356778us) service_pool.cc:224] Handling call
0709 12:24:34.322530 (+  9240us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.324637 (+  2107us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.324949 (+   312us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:34.344762 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:34.347666 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.350445 31673 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:24:34.347020 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:34.368440 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:34.372313 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:34.319747  8218 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260709 12:24:34.336225  7756 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.346974  7944 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:34.349426  8273 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:34.314940  8336 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.059s
I20260709 12:24:34.308202  8329 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.323457  8230 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.336614  8335 raft_consensus.cc:2764] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:24:34.384074 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:34.348318  8012 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:34.385573  8274 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.082s	user 0.002s	sys 0.006s
I20260709 12:24:34.388113 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3663) took 2038 ms. Trace:
I20260709 12:24:34.349375  8344 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:34.368268  8228 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:34.389635 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:34.346717  8257 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:34.408504  8329 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:34.412809  7766 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:34.412624  8225 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:34.413093  8339 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:24:34.402992  8269 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.413868  8269 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.368775  8184 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:34.415946  8269 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:34.417537  7805 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:34.418097  7805 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:34.418967  7805 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 23.11, Last appended by leader: 11, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.422338 31678 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:34.423151  8222 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:34.424341  8222 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:34.426798  8228 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:34.429018  8228 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:34.429643  8222 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:34.431061  8225 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:34.432220  8344 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:34.433321  8344 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:34.436529  8245 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:34.440014  8226 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.644s	user 0.008s	sys 0.004s
I20260709 12:24:34.440716  8226 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:34.442929  8226 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:34.444460  8226 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:34.444911  8226 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:34.445365  8275 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.548s	user 0.010s	sys 0.000s
I20260709 12:24:34.445695  8226 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.446873  8226 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:34.449604  8316 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.523s	user 0.006s	sys 0.005s
W20260709 12:24:34.450940 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44872.8us
I20260709 12:24:34.453122  8230 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:34.453536 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.406164 31744 rpcz_store.cc:276] 0709 12:24:32.350112 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.127554 (+1777442us) service_pool.cc:224] Handling call
0709 12:24:34.127856 (+   302us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.128156 (+   300us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:34.184975 (+ 56819us) log.cc:844] Serialized 14 byte log entry
0709 12:24:34.387369 (+202394us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:34.387579 (+   210us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:34.387630 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.387697 (+    67us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.388054 (+   357us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":54606,"prepare.queue_time_us":57458,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":32,"raft.queue_time_us":125,"raft.run_cpu_time_us":82,"raft.run_wall_time_us":84,"thread_start_us":201601,"threads_started":1}
I20260709 12:24:34.459399  8135 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:34.460690  8218 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:34.461295  8218 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.463683  8218 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:34.464037 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3661) took 2196 ms. Trace:
I20260709 12:24:34.464460 31751 rpcz_store.cc:276] 0709 12:24:32.267375 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.954068 (+1686693us) service_pool.cc:224] Handling call
0709 12:24:33.962817 (+  8749us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.215486 (+252669us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.463780 (+248294us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54003e30d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:34.463972 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":460837760}
I20260709 12:24:34.388775  8273 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:34.465803 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:34.466851 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:34.467804  8273 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:34.467813  8228 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:34.468997  8228 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.470073 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:34.471093 31744 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:34.471292  8228 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:34.472189 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:34.472870  8013 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:34.480795 31827 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:34.481525 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3699) took 1190 ms. Trace:
I20260709 12:24:34.383569  7756 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.483537 31744 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 31 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:24:34.483773  8222 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.484601  7756 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:34.484337  8222 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:34.484735 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:34.485360 31744 raft_consensus.cc:2376] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27. Re-sending same reply.
I20260709 12:24:34.386263  7944 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.375505  7963 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 20, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.489269 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:34.487933 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:34.492089 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3665) took 1592 ms. Trace:
I20260709 12:24:34.493314 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:34.492501 31745 rpcz_store.cc:276] 0709 12:24:32.899393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.129197 (+1229804us) service_pool.cc:224] Handling call
0709 12:24:34.129861 (+   664us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:34.200252 (+ 70391us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:34.200308 (+    56us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:34.213835 (+ 13527us) log.cc:844] Serialized 78 byte log entry
0709 12:24:34.382516 (+168681us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:34.490836 (+108320us) log.cc:844] Serialized 14 byte log entry
0709 12:24:34.491103 (+   267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.491173 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:34.491217 (+    44us) raft_consensus.cc:1685] finished
0709 12:24:34.491229 (+    12us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.492031 (+   802us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":109012,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":639,"raft.run_cpu_time_us":238,"raft.run_wall_time_us":3582,"thread_start_us":151443,"threads_started":1,"wal-append.queue_time_us":234035}
I20260709 12:24:34.474385 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:34.405333  8274 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.102s	user 0.002s	sys 0.007s
I20260709 12:24:34.476243 31822 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: [29.17-29.17]
I20260709 12:24:34.477922  8210 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:24:34.339215  8265 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.480048 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:34.487957  7944 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:34.482041 31827 rpcz_store.cc:276] 0709 12:24:33.291522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.018278 (+726756us) service_pool.cc:224] Handling call
0709 12:24:34.480432 (+462154us) spinlock_profiling.cc:243] Waited 22 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:24:34.480621 (+   189us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.481255 (+   634us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.481458 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":50594560}
I20260709 12:24:34.473380 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:34.481559 31748 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:34.499928 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3662) took 2208 ms. Trace:
I20260709 12:24:34.502126 31825 pending_rounds.cc:85] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 14
I20260709 12:24:34.506505 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:34.511765 31825 pending_rounds.cc:107] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 25.15
I20260709 12:24:34.512096 31825 raft_consensus.cc:2889] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:34.507175 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3696) took 1319 ms. Trace:
I20260709 12:24:34.513152 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3701) took 1221 ms. Trace:
I20260709 12:24:34.513906 31822 rpcz_store.cc:276] 0709 12:24:33.292108 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.032807 (+740699us) service_pool.cc:224] Handling call
0709 12:24:34.475794 (+442987us) spinlock_profiling.cc:243] Waited 6.07 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:24:34.475999 (+   205us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:34.512861 (+ 36862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.513097 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":13966592}
I20260709 12:24:34.512830 31824 rpcz_store.cc:276] 0709 12:24:33.187743 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.006410 (+818667us) service_pool.cc:224] Handling call
0709 12:24:34.038006 (+ 31596us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:34.038565 (+   559us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:34.038609 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:34.039963 (+  1354us) log.cc:844] Serialized 26 byte log entry
0709 12:24:34.372674 (+332711us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:34.372723 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.373108 (+   385us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:34.506944 (+133836us) raft_consensus.cc:1685] finished
0709 12:24:34.506966 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.507106 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":116,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":8088,"thread_start_us":278270,"threads_started":1,"wal-append.queue_time_us":372974}
I20260709 12:24:34.516592 31824 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:34.517395 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3518) took 1010 ms. Trace:
I20260709 12:24:34.518323 31821 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:34.518194 31827 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:34.519302 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3515) took 1013 ms. Trace:
I20260709 12:24:34.519809 31827 rpcz_store.cc:276] 0709 12:24:33.505868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.510099 (+1004231us) service_pool.cc:224] Handling call
0709 12:24:34.518022 (+  7923us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.518978 (+   956us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.519229 (+   251us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:34.517896 31824 rpcz_store.cc:276] 0709 12:24:33.506615 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.515949 (+1009334us) service_pool.cc:224] Handling call
0709 12:24:34.516344 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.517129 (+   785us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.517334 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:34.523125 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1545) took 1014 ms. Trace:
I20260709 12:24:34.523669 31824 rpcz_store.cc:276] 0709 12:24:33.508910 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.521862 (+1012952us) service_pool.cc:224] Handling call
0709 12:24:34.522423 (+   561us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.522683 (+   260us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:34.522727 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:34.522794 (+    67us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:34.522847 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.522926 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.523060 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:34.526144 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1546) took 1017 ms. Trace:
I20260709 12:24:34.527752 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1544) took 1019 ms. Trace:
I20260709 12:24:34.527949 32371 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
W20260709 12:24:34.528136 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48149.3us
I20260709 12:24:34.528261 31827 rpcz_store.cc:276] 0709 12:24:33.508659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.520822 (+1012163us) service_pool.cc:224] Handling call
0709 12:24:34.527226 (+  6404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.527426 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:34.527452 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:34.527497 (+    45us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:34.527531 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.527586 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.527695 (+   109us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:34.530238 31821 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3513) took 2135 ms (client timeout 1671 ms). Trace:
W20260709 12:24:34.530781 31821 rpcz_store.cc:269] 0709 12:24:32.394807 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.015257 (+1620450us) service_pool.cc:224] Handling call
0709 12:24:34.473135 (+457878us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:24:34.530167 (+ 57032us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":54403328}
I20260709 12:24:34.532966 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:34.533149 31827 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:34.534487 31744 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:24:34.534601 31827 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:34.535110 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1265) took 1409 ms. Trace:
I20260709 12:24:34.535465 31744 rpcz_store.cc:276] 0709 12:24:33.125588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.533909 (+1408321us) service_pool.cc:224] Handling call
0709 12:24:34.534183 (+   274us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.534871 (+   688us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.535039 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:34.536027 31821 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:34.536469 31821 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:34.531001 31824 rpcz_store.cc:276] 0709 12:24:33.511006 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.525085 (+1014079us) service_pool.cc:224] Handling call
0709 12:24:34.525465 (+   380us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.525698 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:34.525737 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:34.525799 (+    62us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:34.525847 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.525922 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.526079 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:34.538833 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:34.446099  8275 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:34.537287 31821 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.450270  8316 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.552829  8316 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:34.537829  8274 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:34.566622 31747 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: [31.21-31.21]
I20260709 12:24:34.508901  8222 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:34.516156  8225 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.061s
I20260709 12:24:34.583524  8225 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:34.583982  8225 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:34.584393  8181 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:34.584800  8225 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:34.586340 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:34.588522 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:34.589946 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:34.590086 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:34.591249 32372 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:34.591837  8275 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:34.592036  8181 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:34.541960  8265 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:34.595726 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:34.596864 32376 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 17: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:34.613926 32378 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 30: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:34.615835 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:34.616289 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:34.617486 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:34.619127  7944 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:34.614557  8316 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:34.624214 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:34.506703  8012 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:34.627709 32377 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:34.588528  8225 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:34.628046 32376 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 29: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:34.628603 32372 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: [23.16-23.16]
I20260709 12:24:34.630558 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:34.631217 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:34.639154 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:34.648649 32378 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:34.649793 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:34.654846 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 207885us
I20260709 12:24:34.655181 32375 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:24:34.653712 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3704) took 1330 ms. Trace:
I20260709 12:24:34.615717  8270 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.041s
I20260709 12:24:34.662706 31819 rpcz_store.cc:276] 0709 12:24:33.323253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.091005 (+767752us) service_pool.cc:224] Handling call
0709 12:24:34.132337 (+ 41332us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.132549 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:34.382951 (+250402us) log.cc:844] Serialized 14 byte log entry
0709 12:24:34.653083 (+270132us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:34.653165 (+    82us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:34.653234 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.653316 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.653631 (+   315us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":394799,"prepare.run_cpu_time_us":72,"prepare.run_wall_time_us":9896,"raft.queue_time_us":670,"thread_start_us":517577,"threads_started":2}
I20260709 12:24:34.654403  8264 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:34.656248  8132 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.669122  7756 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.011s
I20260709 12:24:34.679961  8132 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:34.672472  8257 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.011s
W20260709 12:24:34.683563 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.654620  8337 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:24:34.700939 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:34.693212  8299 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.539s	user 0.002s	sys 0.017s
I20260709 12:24:34.687412  8151 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.671737  8264 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.714818 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:34.717270 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:34.715698 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:34.725219 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1270) took 1569 ms. Trace:
I20260709 12:24:34.726011 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:34.728408 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.729382 31818 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:34.729820 31818 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:34.707167  8299 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.553s	user 0.002s	sys 0.018s
I20260709 12:24:34.729707 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:34.727200 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:34.725313 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:34.853192 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 117177us
I20260709 12:24:34.854616  7295 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:34.855573  7295 consensus_queue.cc:1243] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 24.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:24:34.868654 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:34.874528 31824 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:34.877278 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1540) took 2358 ms. Trace:
I20260709 12:24:34.877897 31820 rpcz_store.cc:276] 0709 12:24:32.536712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.414596 (+877884us) service_pool.cc:224] Handling call
0709 12:24:33.414904 (+   308us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.415084 (+   180us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:24:33.438487 (+ 23403us) log.cc:844] Serialized 14 byte log entry
0709 12:24:34.643440 (+1204953us) spinlock_profiling.cc:243] Waited 10.5 ms on lock 0x7b1800445360. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:24:34.876793 (+233353us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:34.876869 (+    76us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:34.876936 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.877028 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.877199 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":141055,"prepare.queue_time_us":21983,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":48,"spinlock_wait_cycles":24152960,"thread_start_us":91892,"threads_started":1}
W20260709 12:24:34.888069 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:34.891243 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:34.723752  8264 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:34.873441  7766 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:34.886129 31818 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:34.894515  7756 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20260709 12:24:34.895782  8357 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
I20260709 12:24:34.896054 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:34.906944  8012 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:34.727100  8257 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:34.734494  8230 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:34.719388  8151 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:34.912976  8274 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:24:34.917026  8350 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.482s	user 0.002s	sys 0.000s
I20260709 12:24:34.924080 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:34.935074  8299 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:34.926362 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:34.929008 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3514) took 1423 ms. Trace:
I20260709 12:24:34.931550 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:34.933768 31820 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:24:34.947903 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:34.949012 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:34.937431 31818 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:34.927425  7766 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.947196 31825 rpcz_store.cc:276] 0709 12:24:33.505593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.498048 (+992455us) service_pool.cc:224] Handling call
0709 12:24:34.501939 (+  3891us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:34.542420 (+ 40481us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:34.542453 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:34.542924 (+   471us) log.cc:844] Serialized 26 byte log entry
0709 12:24:34.736440 (+193516us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:34.887517 (+151077us) log.cc:844] Serialized 14 byte log entry
0709 12:24:34.928526 (+ 41009us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.928644 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:34.928708 (+    64us) raft_consensus.cc:1685] finished
0709 12:24:34.928728 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.928928 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40614,"prepare.queue_time_us":157935,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":46869,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":6041,"thread_start_us":122514,"threads_started":1,"wal-append.queue_time_us":279636}
I20260709 12:24:34.938997  8230 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.969712 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:34.970678 31824 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 27: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:34.971163 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:34.969731 31827 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:34.967670 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:34.974164 31825 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:34.976547 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2786) took 2024 ms. Trace:
I20260709 12:24:34.977126 31675 rpcz_store.cc:276] 0709 12:24:32.952544 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.957181 (+  4637us) service_pool.cc:224] Handling call
0709 12:24:32.957528 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.276090 (+318562us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.976280 (+1700190us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b5400391cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:34.976476 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":731203456}
I20260709 12:24:34.978618 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:34.979584 31819 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:34.981241 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:34.979807 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:34.727270 31744 rpcz_store.cc:276] 0709 12:24:33.156254 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.536149 (+1379895us) service_pool.cc:224] Handling call
0709 12:24:34.536439 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.536628 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:24:34.536889 (+   261us) log.cc:844] Serialized 14 byte log entry
0709 12:24:34.724755 (+187866us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:34.724832 (+    77us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:34.724909 (+    77us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.724995 (+    86us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.725149 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":246,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":40,"raft.queue_time_us":1549,"raft.run_cpu_time_us":124,"raft.run_wall_time_us":6272,"spinlock_wait_cycles":11520,"thread_start_us":187253,"threads_started":1,"wal-append.queue_time_us":395851}
I20260709 12:24:34.953930  7766 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:34.988855 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:34.992969 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3660) took 2749 ms. Trace:
I20260709 12:24:34.993199  8181 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:34.993769  8181 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:34.994220  8181 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:34.971733  8230 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:34.968336  7756 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:34.995883 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:35.003391 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:35.003783  8343 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:35.004442  8343 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:34.952395  8299 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.020970 31824 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:24:34.988735 31819 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 14. (term mismatch)
I20260709 12:24:34.993494 31749 rpcz_store.cc:276] 0709 12:24:32.243960 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.932438 (+1688478us) service_pool.cc:224] Handling call
0709 12:24:34.149526 (+217088us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.188771 (+ 39245us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.992716 (+803945us) spinlock_profiling.cc:243] Waited 216 ms on lock 0x7b54001a4fd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:34.992892 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":497869696}
I20260709 12:24:34.997133 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3516) took 1491 ms. Trace:
I20260709 12:24:34.977486  8012 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:34.989876  8257 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:24:34.997246  8025 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.026798 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:35.034773 31749 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:35.032939 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:35.035806 31748 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:35.036763 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:35.026348 31822 rpcz_store.cc:276] 0709 12:24:33.506149 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.515221 (+1009072us) service_pool.cc:224] Handling call
0709 12:24:34.515609 (+   388us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:34.516407 (+   798us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:34.516436 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:34.541859 (+ 25423us) log.cc:844] Serialized 26 byte log entry
0709 12:24:34.729605 (+187746us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:34.729647 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.731342 (+  1695us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:34.996898 (+265556us) raft_consensus.cc:1685] finished
0709 12:24:34.996921 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.997071 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1231,"raft.run_cpu_time_us":106,"raft.run_wall_time_us":10658,"thread_start_us":105173,"threads_started":1,"wal-append.queue_time_us":349603}
I20260709 12:24:35.027220 31819 pending_rounds.cc:85] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 13
I20260709 12:24:35.038756 31819 pending_rounds.cc:107] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 25.14
I20260709 12:24:35.039132 31819 raft_consensus.cc:2889] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:35.040225 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:35.040405 31825 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 32 because replica has last-logged OpId of term: 31 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:24:35.043689  8309 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.044291  8309 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.044736  8270 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:35.045168  8270 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:35.058017 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:35.061125 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:35.061662 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.064026 31744 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:35.065670 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:35.066483 31745 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:35.066622 31744 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:35.079818 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.165621 31672 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 30 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
W20260709 12:24:35.080864 31750 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1264) took 1955 ms (client timeout 1539 ms). Trace:
W20260709 12:24:35.166272 31750 rpcz_store.cc:269] 0709 12:24:33.125276 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.130067 (+  4791us) service_pool.cc:224] Handling call
0709 12:24:35.080642 (+1950575us) spinlock_profiling.cc:243] Waited 96.2 ms on lock 0x7b5400064d54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:35.080793 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":221210880}
I20260709 12:24:35.081600 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.167935 32369 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:24:35.168426 31674 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:35.083534 31670 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:35.085372 31677 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:35.086335 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:35.087118 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.167970 32373 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:35.170171 31673 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:24:35.087827 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.088889 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.172072 31669 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 28: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:35.167999 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:35.172858 31750 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:35.174976 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:35.098745  8155 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:35.180275 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:35.181438 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:35.099861  8298 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.923s	user 0.002s	sys 0.010s
I20260709 12:24:35.100731  8341 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
W20260709 12:24:35.106336  8355 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.695s	user 0.000s	sys 0.009s
I20260709 12:24:35.107136  7295 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:35.183475  7295 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:35.112000  8275 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:35.024869  7756 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.052199  8025 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.115938  8309 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:35.046067  8270 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:35.227613 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.119953  8151 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:34.510830 31753 rpcz_store.cc:276] 0709 12:24:32.291519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.127127 (+1835608us) service_pool.cc:224] Handling call
0709 12:24:34.127488 (+   361us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.127720 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:34.182977 (+ 55257us) log.cc:844] Serialized 14 byte log entry
0709 12:24:34.499386 (+316409us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:34.499442 (+    56us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:34.499493 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:34.499565 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:34.499868 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":57610,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":46,"raft.queue_time_us":86,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":5923,"thread_start_us":315881,"threads_started":1}
I20260709 12:24:35.123564 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1221) took 2067 ms. Trace:
I20260709 12:24:35.230973 32375 rpcz_store.cc:276] 0709 12:24:33.056159 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.056836 (+   677us) service_pool.cc:224] Handling call
0709 12:24:33.057299 (+   463us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.655586 (+1598287us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.123271 (+467685us) spinlock_profiling.cc:243] Waited 1.6 s on lock 0x7b54004984d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:35.123489 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3673895808}
I20260709 12:24:35.102392  7779 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:35.063675  8346 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:35.126911  8135 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.127112  7963 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:35.114187  8262 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:35.259366  8262 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.262214  8262 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:35.264271 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:35.265079 31673 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:35.268388  8130 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.268996  8130 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.270170  8273 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:35.128429  8343 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:35.268404 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.272174 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 23 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:35.272598 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.273247 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.129586  8053 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:35.130323 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:35.274878 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 144651us
I20260709 12:24:35.274686  8053 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.275532 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.130486  8316 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:35.276018 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:35.131417  8210 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.002s
W20260709 12:24:35.134680 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.226541  8226 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.285342 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:35.137741  8265 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.545s	user 0.011s	sys 0.000s
I20260709 12:24:35.138556  8299 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:35.149150 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:35.032492  8012 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.156208 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:35.157867 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:35.159488 32371 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
W20260709 12:24:35.161875 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 180702us
I20260709 12:24:35.162402 31822 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 24 index: 15. (index mismatch)
I20260709 12:24:35.161325 31748 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:35.344050 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:35.345203 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:35.164346 31818 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:24:35.151803  8163 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:35.223834  8275 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.092372  8201 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:35.093679 31821 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:35.176079 31750 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:35.188824 31820 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:35.194666 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:35.204999 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.219874 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.151913  8224 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:24:35.225344 31745 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:35.372655 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:35.225800 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.229213  7295 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.229759 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.231769 31753 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:24:35.376410 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1276) took 1414 ms. Trace:
I20260709 12:24:35.376736 31753 rpcz_store.cc:276] 0709 12:24:33.962358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.231146 (+1268788us) service_pool.cc:224] Handling call
0709 12:24:35.231468 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.376152 (+144684us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.376356 (+   204us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:35.241379 31744 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:35.377473 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.242442 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:35.291610  8270 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:35.192250  8341 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:35.179046  8155 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:35.250471 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:35.387624 32375 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:35.253567 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:35.258174  7963 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.275591 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:35.388777 32370 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
W20260709 12:24:35.389108 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 113599us
W20260709 12:24:35.131362  7944 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.512s	user 0.014s	sys 0.001s
I20260709 12:24:35.276237  8316 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:35.132253 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:35.276906  8130 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:35.277070  8053 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:35.277743 31677 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:35.278205 31752 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:35.400135 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3676) took 1209 ms. Trace:
I20260709 12:24:35.135804 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3668) took 2180 ms. Trace:
I20260709 12:24:35.400447 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:35.285671 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:35.402705 31821 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260709 12:24:35.403311 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 117719us
I20260709 12:24:35.404143 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:35.400462 31752 rpcz_store.cc:276] 0709 12:24:34.190386 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.277669 (+1087283us) service_pool.cc:224] Handling call
0709 12:24:35.277993 (+   324us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.399871 (+121878us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.400083 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:35.405709 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:35.249097  7779 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:35.348932 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:35.348073 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:35.081642 31746 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:35.410696 31744 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 21. (term mismatch)
I20260709 12:24:35.356433 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:35.270740  8273 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:35.281033  8327 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:24:35.353027  8184 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.934s	user 0.028s	sys 0.037s
I20260709 12:24:35.357800  8013 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.010s
I20260709 12:24:35.368600 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:35.350464  8163 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:35.384290 31750 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:35.445022 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2950) took 1111 ms. Trace:
I20260709 12:24:35.445554 31750 rpcz_store.cc:276] 0709 12:24:34.334132 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.383574 (+1049442us) service_pool.cc:224] Handling call
0709 12:24:35.384039 (+   465us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.444661 (+ 60622us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.444944 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:35.255577  7766 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.385564 31673 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:24:35.386463 31753 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:35.391459 31745 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:24:35.393987 31823 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:35.390321 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:35.395818 31670 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:35.403430 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3708) took 1771 ms. Trace:
I20260709 12:24:35.348199 32376 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:24:35.402212 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:35.408387 31752 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:35.279209  8135 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.351979 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:35.411237 31744 pending_rounds.cc:85] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 20
I20260709 12:24:35.400609 31747 rpcz_store.cc:276] 0709 12:24:32.955907 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.250809 (+1294902us) service_pool.cc:224] Handling call
0709 12:24:34.251261 (+   452us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:34.567334 (+316073us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.135488 (+568154us) spinlock_profiling.cc:243] Waited 315 ms on lock 0x7b54002134d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:35.135737 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":724278016}
I20260709 12:24:35.397388  7944 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:35.414906 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:35.483997 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.292029  8226 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:35.440739 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 84439.1us
I20260709 12:24:35.492826 31818 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.12-24.12]
I20260709 12:24:35.374431  8299 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.062s
W20260709 12:24:35.455646 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 65440.6us
I20260709 12:24:35.455166 31821 rpcz_store.cc:276] 0709 12:24:33.631602 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.531809 (+900207us) service_pool.cc:224] Handling call
0709 12:24:34.535887 (+  4078us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.093080 (+557193us) spinlock_profiling.cc:243] Waited 501 ms on lock 0x7b580039bfd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd310aa 00007fe8fcdb3447 00007fe8fcdcbfe7 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4
0709 12:24:35.403152 (+310072us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.403374 (+   222us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1152621952}
W20260709 12:24:35.442163  8184 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.025s	user 0.029s	sys 0.037s
I20260709 12:24:35.461509 31744 pending_rounds.cc:107] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 30.21
I20260709 12:24:35.506206 31744 raft_consensus.cc:2889] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:35.507038 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2954) took 1026 ms. Trace:
I20260709 12:24:35.289570  8346 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.507839  7756 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:35.507696  8273 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.509421  8273 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:35.509618 31669 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:24:35.509969  8346 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:35.511811  8012 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:35.512390  7779 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.513842  7779 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:35.515699  8230 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.516954  8135 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:35.503078  7805 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:35.446702  8274 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:35.454315  8130 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:35.438087  8327 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:35.487118  8224 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:35.491577  8341 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.505096  8343 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:24:35.397486  8316 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:35.413512 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 269318us
W20260709 12:24:35.488742 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 73920.5us
I20260709 12:24:35.443811  8309 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:35.397346  8054 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:35.492823 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.441126 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.497087 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:35.459879 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 110124us
I20260709 12:24:35.501827 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3677) took 1164 ms. Trace:
W20260709 12:24:35.440752  8271 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 28 current term: 28 is higher.
I20260709 12:24:35.501662 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:35.447244  8303 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:35.490973  8155 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 24.12, Last appended by leader: 12, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.523680 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.630558 31677 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 29: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:35.527344 31826 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:35.537969 31825 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:35.507355 31744 rpcz_store.cc:276] 0709 12:24:34.480855 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.385863 (+905008us) service_pool.cc:224] Handling call
0709 12:24:35.410220 (+ 24357us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.506772 (+ 96552us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.506980 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:35.556442 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:35.576584 31672 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:35.579838 31745 rpcz_store.cc:276] 0709 12:24:34.337920 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.384924 (+1047004us) service_pool.cc:224] Handling call
0709 12:24:35.385239 (+   315us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.490904 (+105665us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.501671 (+ 10767us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:35.576879 31673 raft_consensus.cc:2376] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28. Re-sending same reply.
W20260709 12:24:35.595109 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 93538.6us
I20260709 12:24:35.640131 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:35.598166 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3659) took 3382 ms. Trace:
I20260709 12:24:35.641757 31746 rpcz_store.cc:276] 0709 12:24:32.215930 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:33.486552 (+1270622us) service_pool.cc:224] Handling call
0709 12:24:35.081449 (+1594897us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.410872 (+329423us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.597907 (+187035us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b5400064d50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:35.598063 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3666922880}
I20260709 12:24:35.643944 31746 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 11. Preceding OpId from leader: term: 25 index: 12. (index mismatch)
W20260709 12:24:35.598724 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 250698us
I20260709 12:24:35.645936 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:35.611271 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.545634 31747 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: [27.19-27.19]
I20260709 12:24:35.577963 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:35.557185 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:35.655320 31746 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260709 12:24:35.658895 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 81014us
W20260709 12:24:35.662087 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:35.669888 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 113561us
I20260709 12:24:35.670611 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:35.666029 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:35.671203 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:35.671938 31823 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
W20260709 12:24:35.676592 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 119491us
W20260709 12:24:35.552059  8369 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.585s	user 0.002s	sys 0.001s
W20260709 12:24:35.574482  8181 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.580s	user 0.008s	sys 0.008s
I20260709 12:24:35.675709 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:35.553422  8264 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.618s	user 0.011s	sys 0.000s
I20260709 12:24:35.664276 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:35.529402  8172 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:35.525408  8226 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:35.526253  8327 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:35.520823  7805 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:35.596563  8167 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:35.518556  8230 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:35.698812  8379 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.596153  8303 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.574728  8054 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:35.536446  8341 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:35.593099  7963 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:35.710886 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:35.770428 31818 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:35.527088  7756 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:35.527070  8224 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:35.692171  7944 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.073s
I20260709 12:24:35.523089  8275 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:35.598079  8155 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:35.700028 31747 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28. Ops: [27.15-27.15]
W20260709 12:24:35.700685 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60680.8us
I20260709 12:24:35.709148 31824 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:35.711478 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:35.720949 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 22 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:35.723204 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52115.6us
I20260709 12:24:35.713816  8130 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.056s
I20260709 12:24:35.522020  8025 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:35.551365 31569 catalog_manager.cc:5697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 16 to 26. New cstate: current_term: 26 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:35.754951 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2794) took 1425 ms. Trace:
I20260709 12:24:35.772636  8054 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:35.755877 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.595930  8000 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:35.698191  8309 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:35.763499 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:35.764539 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.577972  8012 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.134s
I20260709 12:24:35.692391  8380 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election won for term 25
W20260709 12:24:35.771013 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.727038  8262 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.771499 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3673) took 2359 ms. Trace:
I20260709 12:24:35.520308  7766 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:35.818924 31751 rpcz_store.cc:276] 0709 12:24:33.412576 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.104170 (+1691594us) service_pool.cc:224] Handling call
0709 12:24:35.104639 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.104878 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:35.287532 (+182654us) log.cc:844] Serialized 14 byte log entry
0709 12:24:35.771149 (+483617us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:35.771208 (+    59us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:35.771252 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.771313 (+    61us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:35.771442 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":347882,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":50,"raft.queue_time_us":79775,"raft.run_cpu_time_us":85,"raft.run_wall_time_us":85,"thread_start_us":663889,"threads_started":3,"wal-append.queue_time_us":222983}
I20260709 12:24:35.820853 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:35.821513 31751 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:35.750209  7805 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:35.720202  8274 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:35.713244  8376 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:35.827754 31569 catalog_manager.cc:5697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 13 to 23. New cstate: current_term: 23 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:35.724424  8377 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:35.751334  8226 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:35.727375  8337 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.750389  7889 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:35.755131  8167 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:35.523707  8163 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.751019  8225 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:35.767616  8369 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:24:35.533455  8132 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:35.713451  8264 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.712226  8181 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.789296 31678 rpcz_store.cc:276] 0709 12:24:34.329268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:34.329437 (+   169us) service_pool.cc:224] Handling call
0709 12:24:34.331106 (+  1669us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:34.423866 (+ 92760us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.754515 (+1330649us) spinlock_profiling.cc:243] Waited 91 ms on lock 0x7b54001815d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:35.754850 (+   335us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":209315584}
I20260709 12:24:35.692749  7959 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:35.768751  8230 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.841166 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.842003 31678 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
W20260709 12:24:35.846484 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:35.848248 31824 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:35.803894  8275 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.030s
I20260709 12:24:35.767115  8013 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.028s
I20260709 12:24:35.816570 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.812589  8309 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:35.770306  8303 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.815745  7779 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.020s
I20260709 12:24:35.767025  8379 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.780058  7756 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.888331 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:35.893365  7756 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:35.803449  7963 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.042s
I20260709 12:24:35.837409 31569 catalog_manager.cc:5697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 20 to 22, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 22 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:24:35.838451  8344 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.401s	user 0.021s	sys 0.041s
I20260709 12:24:35.880734  7779 consensus_queue.cc:1243] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 27.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:24:35.853488 31672 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:35.854760 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:35.821154  8346 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.022s
I20260709 12:24:35.810135 31818 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:24:35.821147  8130 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.029s
I20260709 12:24:35.868110 32376 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:24:35.868203 31824 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:35.946365 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:35.869735 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:35.872668 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:35.944693 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 243654us
I20260709 12:24:35.948279 32375 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:35.853120  8230 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.874380 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:35.838009  8376 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:35.826644  8210 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.025s
I20260709 12:24:35.879019 31749 raft_consensus.cc:2368] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28.
I20260709 12:24:35.900247  8012 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:35.883925 31747 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
I20260709 12:24:35.885203 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:35.957821 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:35.931411  8151 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:24:35.893545  8381 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:24:35.964664 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:35.787202  8299 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.042s
I20260709 12:24:35.856143  8163 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:35.868182  8309 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:35.834074  8274 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.877089  8378 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.836467  8226 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:35.836639  8337 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.865643  7959 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:35.876667  8222 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:35.840116  8225 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:35.853242 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 247928us
I20260709 12:24:35.963310  8151 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:35.944242 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3686) took 1297 ms. Trace:
I20260709 12:24:35.967018  8343 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:35.898321  8167 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.951069 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:35.947592 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:35.935119  8344 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.498s	user 0.021s	sys 0.041s
I20260709 12:24:35.826977  8262 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.895980  8393 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:35.962508 31821 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:35.965479 31818 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:35.979336  8346 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:35.966398 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 190915us
W20260709 12:24:35.983168 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:35.984495 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 308875us
I20260709 12:24:35.938915  8053 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.032s
I20260709 12:24:35.839828  8224 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:35.995600 31569 catalog_manager.cc:5697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 19 to 21. New cstate: current_term: 21 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:35.940948 31678 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 30 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:24:35.962738  8012 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:36.006459 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 194287us
I20260709 12:24:35.874008 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:36.011083 31753 rpcz_store.cc:276] 0709 12:24:34.648646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.457997 (+809351us) service_pool.cc:224] Handling call
0709 12:24:35.458468 (+   471us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:35.458782 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:35.757316 (+298534us) log.cc:844] Serialized 14 byte log entry
0709 12:24:35.941587 (+184271us) log.cc:844] Serialized 14 byte log entry
0709 12:24:35.943885 (+  2298us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:35.943943 (+    58us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:35.943986 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.944061 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:35.944176 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":46020,"prepare.queue_time_us":126900,"prepare.run_cpu_time_us":82,"prepare.run_wall_time_us":87,"raft.queue_time_us":90280,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":5853,"spinlock_wait_cycles":7401856,"thread_start_us":434151,"threads_started":2,"wal-append.queue_time_us":216421}
I20260709 12:24:35.948279 32370 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 17: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:35.959107 31826 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:35.972720  7766 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:36.025738  8371 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:36.033946 31823 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
I20260709 12:24:36.035236 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:36.033346 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:36.035519 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:36.076030 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:36.001014  8378 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:36.017372  8343 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:36.220738 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3711) took 1125 ms. Trace:
I20260709 12:24:36.221366 31678 rpcz_store.cc:276] 0709 12:24:35.095718 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.852537 (+756819us) service_pool.cc:224] Handling call
0709 12:24:36.220662 (+368125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.098174 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:36.226902 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:36.227754 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3687) took 3767 ms. Trace:
I20260709 12:24:36.238497 31826 rpcz_store.cc:276] 0709 12:24:32.389954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:32.889852 (+499898us) service_pool.cc:224] Handling call
0709 12:24:32.900563 (+ 10711us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:33.874829 (+974266us) spinlock_profiling.cc:243] Waited 387 ms on lock 0x7b540036c520. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:24:35.526852 (+1652023us) spinlock_profiling.cc:243] Waited 654 ms on lock 0x7b58004ae7d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd310aa 00007fe8fcdb3447 00007fe8fcdcbfe7 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4
0709 12:24:36.156924 (+630072us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.157156 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2393673856}
W20260709 12:24:36.032603  8375 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.237s	user 0.000s	sys 0.001s
I20260709 12:24:36.228449 31827 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 29: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:36.242056 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3689) took 1179 ms. Trace:
I20260709 12:24:36.030220  8210 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.157346 31569 catalog_manager.cc:5697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 17 to 25, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 25 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:36.244524 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3685) took 1598 ms. Trace:
I20260709 12:24:36.030099  8344 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:36.242506 31675 rpcz_store.cc:276] 0709 12:24:35.062540 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.110815 (+ 48275us) service_pool.cc:224] Handling call
0709 12:24:35.111428 (+   613us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.111966 (+   538us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:35.112011 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:35.112748 (+   737us) log.cc:844] Serialized 26 byte log entry
0709 12:24:35.403950 (+291202us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:35.403988 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.404050 (+    62us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.241790 (+837740us) raft_consensus.cc:1685] finished
0709 12:24:36.241818 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.241989 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7145,"raft.run_cpu_time_us":128,"raft.run_wall_time_us":128,"thread_start_us":290686,"threads_started":1,"wal-append.queue_time_us":639404}
I20260709 12:24:36.245015 31752 rpcz_store.cc:276] 0709 12:24:34.646392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.457911 (+811519us) service_pool.cc:224] Handling call
0709 12:24:35.458354 (+   443us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:35.776714 (+318360us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:35.776750 (+    36us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:35.789864 (+ 13114us) log.cc:844] Serialized 52 byte log entry
0709 12:24:35.971251 (+181387us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:35.971299 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.971363 (+    64us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.244298 (+272935us) raft_consensus.cc:1685] finished
0709 12:24:36.244319 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.244470 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":66898,"raft.run_cpu_time_us":57,"raft.run_wall_time_us":57,"thread_start_us":434365,"threads_started":2,"wal-append.queue_time_us":148876}
I20260709 12:24:36.019563  8222 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:36.255069 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3713) took 1095 ms. Trace:
I20260709 12:24:36.248111 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:36.278992 31569 catalog_manager.cc:5697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 21 to 25. New cstate: current_term: 25 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:36.115955  8274 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:36.114363  8395 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:24:36.311493 31678 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:24:36.075897  7779 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:24:36.220669  8343 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:36.113809  8389 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.113525  8226 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:36.319082 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:36.227875  8002 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 25 current term: 25 is higher.
I20260709 12:24:36.037776  8346 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.133512  8262 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:36.131996  8230 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:36.234313  8012 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:36.136053  8225 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:36.138900  8412 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.308605 31820 rpcz_store.cc:276] 0709 12:24:35.159402 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.369932 (+210530us) service_pool.cc:224] Handling call
0709 12:24:35.370275 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.370704 (+   429us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:35.383072 (+ 12368us) log.cc:844] Serialized 14 byte log entry
0709 12:24:35.678505 (+295433us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:35.680927 (+  2422us) log.cc:844] Serialized 26 byte log entry
0709 12:24:35.681135 (+   208us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:35.681185 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:35.681285 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.244193 (+562908us) raft_consensus.cc:1685] finished
0709 12:24:36.244215 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.254996 (+ 10781us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10980,"prepare.queue_time_us":15068,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":20332,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":21256,"spinlock_wait_cycles":25573632,"thread_start_us":295077,"threads_started":1,"wal-append.queue_time_us":412029}
I20260709 12:24:36.319641  8181 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:36.283067  8264 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:36.359329  8264 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:36.336139 31675 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:24:36.360572 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3699) took 1270 ms. Trace:
I20260709 12:24:36.360930 31675 rpcz_store.cc:276] 0709 12:24:35.089892 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.308428 (+1218536us) service_pool.cc:224] Handling call
0709 12:24:36.360505 (+ 52077us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.325112 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3698) took 1235 ms. Trace:
I20260709 12:24:36.362465 31678 rpcz_store.cc:276] 0709 12:24:35.089597 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.242676 (+1153079us) service_pool.cc:224] Handling call
0709 12:24:36.325034 (+ 82358us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:36.339643 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 444802us
I20260709 12:24:36.339882 31569 catalog_manager.cc:5697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 12 to 19, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 19 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:36.227947  8000 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:36.328186  8155 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.541s	user 0.009s	sys 0.002s
I20260709 12:24:36.372080 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:36.124409  8303 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:36.156293  8387 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.175s
I20260709 12:24:36.332710  8399 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:36.335777  8273 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.822s	user 0.013s	sys 0.001s
W20260709 12:24:36.352289 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:36.278612  8210 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.327876  7779 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:36.283013  8299 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.062s
I20260709 12:24:36.116451  7959 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:36.357350  8380 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.011s	sys 0.019s
I20260709 12:24:36.358433  8379 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:36.242322  7766 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.359853  8264 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:36.339287  7759 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.242249  8371 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.117372  8054 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.344983  8013 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:36.348891  8184 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.309681  8344 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:36.383194 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:36.383666 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:36.383711 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 348537us
I20260709 12:24:36.389401  8346 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:36.354985  8230 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:36.377131  8341 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.602s	user 0.012s	sys 0.001s
I20260709 12:24:36.392331 31569 catalog_manager.cc:5697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 15 to 23, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 23 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:24:36.220324  8376 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:36.400045  8380 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.011s	sys 0.019s
I20260709 12:24:36.423007 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1393) took 1315 ms. Trace:
I20260709 12:24:36.451133 31678 rpcz_store.cc:276] 0709 12:24:35.107669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.375628 (+1267959us) service_pool.cc:224] Handling call
0709 12:24:36.422938 (+ 47310us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.386402  8225 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.343125  8389 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:36.461150 31569 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 14 to 22, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 22 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:36.463534 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:36.420885 31675 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:36.466152 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:36.466528 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:36.466904 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 309368us
I20260709 12:24:36.403254  8273 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:36.454785  8230 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:36.421583 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:36.423463  8402 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.391s	user 0.001s	sys 0.000s
I20260709 12:24:36.412025  8387 consensus_queue.cc:1243] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 20.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:24:36.467146 32371 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:24:36.417801  8337 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:36.469826 32375 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:36.419863  8415 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:36.470230  7805 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.632s	user 0.009s	sys 0.002s
I20260709 12:24:36.468966 31569 catalog_manager.cc:5697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 18 to 21, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 21 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:36.476004 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:36.470301 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1384) took 1371 ms. Trace:
I20260709 12:24:36.399787  8226 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.069s
I20260709 12:24:36.477566 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:36.500918 31675 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:36.501896 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:36.372238  8412 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:36.478400 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56916.8us
I20260709 12:24:36.419649  8013 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.450903  8380 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:36.419847  8385 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:36.509954 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:36.482717 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:36.474468  8172 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.749s	user 0.012s	sys 0.001s
I20260709 12:24:36.414234  7759 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:36.471704  8270 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.080s	user 0.011s	sys 0.002s
I20260709 12:24:36.479791  8390 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:36.411552  8414 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.103s
I20260709 12:24:36.483996  8222 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:36.487782  8264 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 20.11, Last appended by leader: 11, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.416455  8399 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.459558  7779 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.489739  8389 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:36.490826  7805 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:36.485741 31676 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:36.491454  8344 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:36.497114  8201 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:36.497309 31752 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 28. Ops: []
I20260709 12:24:36.499779 31753 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: [28.15-28.15]
I20260709 12:24:36.536093 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:36.470189  8418 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.504249 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:36.422873  8184 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:36.507925 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:36.505424 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:36.481178  8393 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:36.544054 31671 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 26 index: 14. (index mismatch)
I20260709 12:24:36.485641 31678 rpcz_store.cc:276] 0709 12:24:35.098735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.458902 (+1360167us) service_pool.cc:224] Handling call
0709 12:24:36.470235 (+ 11333us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.544723 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:36.464880  8000 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.513772  8378 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:36.514024 31672 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:36.512933 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 479607us
I20260709 12:24:36.552582 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:36.553313  8222 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.553908  8222 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:36.554657  8210 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:36.555213  8316 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.997s	user 0.012s	sys 0.000s
I20260709 12:24:36.555632  8316 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:36.556190  8222 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:36.518517 31747 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 16. (term mismatch)
I20260709 12:24:36.467203  8225 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:36.556721 31747 pending_rounds.cc:85] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 15
W20260709 12:24:36.556979 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51697.9us
I20260709 12:24:36.557380  8389 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:24:36.557775  8316 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:36.557863 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:36.508450  8410 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.557035 31747 pending_rounds.cc:107] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 23.16
I20260709 12:24:36.558765 31747 raft_consensus.cc:2889] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:36.559203 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:36.558753  8410 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.559976  8265 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:36.560410 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:36.560685  8265 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:36.561281  8410 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:36.504424  8420 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:36.561623 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:36.562836  7756 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:36.564162 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:36.564699 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:36.558185  8225 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:36.565716 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:36.500458  8415 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:36.525965  8399 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:36.510047 31569 catalog_manager.cc:5697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 15 to 18, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 18 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:36.511427 31827 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:36.582046 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:36.582617 31827 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:36.544809 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3706) took 1452 ms. Trace:
I20260709 12:24:36.512465 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:36.586066 31670 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 16.
I20260709 12:24:36.585753 31671 rpcz_store.cc:276] 0709 12:24:35.091970 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.543394 (+1451424us) service_pool.cc:224] Handling call
0709 12:24:36.543724 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.544550 (+   826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.544755 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.587605 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1386) took 1488 ms. Trace:
I20260709 12:24:36.587936 31670 rpcz_store.cc:276] 0709 12:24:35.099364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.586815 (+1487451us) service_pool.cc:224] Handling call
0709 12:24:36.587133 (+   318us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.587290 (+   157us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.587312 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.587367 (+    55us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:36.587402 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.587452 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.587542 (+    90us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.590032 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1394) took 1482 ms. Trace:
I20260709 12:24:36.539599 31746 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 17.
I20260709 12:24:36.484247  8303 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.558058  8270 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:36.561369  8265 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:36.568297 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:36.568857 31820 pending_rounds.cc:85] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 14
I20260709 12:24:36.596107 31820 pending_rounds.cc:107] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 27.15
I20260709 12:24:36.573100  7779 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:36.571094 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:36.571589 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:36.596904 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:36.524111 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3705) took 1432 ms. Trace:
I20260709 12:24:36.597227 31824 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:36.515486  8385 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:36.577361 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:36.591259 31671 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:36.534880  8201 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:36.592222 31827 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:24:36.592880 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:36.577119  8163 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.593s	user 0.012s	sys 0.000s
I20260709 12:24:36.590521 31670 rpcz_store.cc:276] 0709 12:24:35.107948 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.588994 (+1481046us) service_pool.cc:224] Handling call
0709 12:24:36.589401 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.589620 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.589660 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.589718 (+    58us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:36.589765 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.589834 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.589961 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.596410 31820 raft_consensus.cc:2889] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:36.606371 31821 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:36.557911  8389 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:36.603042 31822 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:24:36.597800 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:36.573504 31678 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:24:36.607903 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:36.597642 31676 rpcz_store.cc:276] 0709 12:24:35.091682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.523423 (+1431741us) service_pool.cc:224] Handling call
0709 12:24:36.523744 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.523901 (+   157us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.523916 (+    15us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:36.523944 (+    28us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:36.523969 (+    25us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.524001 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:36.524070 (+    69us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.599606 31569 catalog_manager.cc:5697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 21 to 24, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 24 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:36.609654 31569 catalog_manager.cc:5697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 22 to 24, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 24 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:36.608140 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3710) took 1512 ms. Trace:
I20260709 12:24:36.601722 31753 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:24:36.569193  8013 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:36.614493 31676 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:24:36.519400  8390 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:36.605374 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1389) took 1499 ms. Trace:
I20260709 12:24:36.606163 32370 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:36.570586 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:36.611972 31678 rpcz_store.cc:276] 0709 12:24:35.095406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.544841 (+1449435us) service_pool.cc:224] Handling call
0709 12:24:36.552647 (+  7806us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.607826 (+ 55179us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.608068 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.617100 31671 rpcz_store.cc:276] 0709 12:24:35.106403 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.587612 (+1481209us) service_pool.cc:224] Handling call
0709 12:24:36.587905 (+   293us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.605110 (+ 17205us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.605313 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:36.622530 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:36.623946 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:36.626842 31821 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:36.604434  8420 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:36.629963 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:36.634045 31827 raft_consensus.cc:2410] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 30 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 16.
I20260709 12:24:36.633394 31569 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 19 to 23, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 23 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:36.636093 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:36.630260 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:36.639844 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:36.641618 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:36.598354  8303 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.613437  8054 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:36.645812  7756 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:24:36.676802  8327 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.920s	user 0.008s	sys 0.004s
I20260709 12:24:36.677373  8327 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.631505  8270 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:24:36.638357  8265 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:36.647019 31569 catalog_manager.cc:5697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 13 to 21. New cstate: current_term: 21 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:36.652601 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:36.551245  8418 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.631932  8420 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:36.697221 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44755.3us
I20260709 12:24:36.681905  8387 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:36.680171 31674 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:36.663942  7805 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:36.689924  8371 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:36.637926  8316 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:24:36.690984 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:36.694258 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:36.696117 31672 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:24:36.684216 31670 raft_consensus.cc:1217] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Deduplicated request from leader. Original: 22.13->[24.14-24.14]   Dedup: 24.14->[]
I20260709 12:24:36.645700  8273 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:36.646013  8393 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
W20260709 12:24:36.699811  8422 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.160s	user 0.000s	sys 0.001s
I20260709 12:24:36.701220  8418 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:36.702000  8420 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:36.702555  7756 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:36.703421  8184 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:36.704010  8265 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:36.704766 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:36.894805 31822 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
W20260709 12:24:36.707170  8383 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.942s	user 0.000s	sys 0.007s
I20260709 12:24:36.893364  8416 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.722555 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3719) took 1554 ms. Trace:
I20260709 12:24:36.756850  8273 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:36.897840  8416 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.741390 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:36.743330 31569 catalog_manager.cc:5697] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 21 to 25. New cstate: current_term: 25 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:36.754797 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:36.756374 31670 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: [24.14-24.14]
I20260709 12:24:36.684331  8230 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:24:36.711187  8404 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.202s	user 0.000s	sys 0.001s
W20260709 12:24:36.710569  8309 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.712s	user 0.006s	sys 0.003s
W20260709 12:24:36.821666 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:36.839504 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:36.868043 31717 connection.cc:256] RPC call timeout handler was delayed by -1.5336319900002309s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2154s for a response
W20260709 12:24:36.959856 31717 connection.cc:256] RPC call timeout handler was delayed by -1.5182282959999611s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.23170000000000002s for a response
I20260709 12:24:36.888193 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:36.746503  8387 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:36.604846  8380 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.896834  8265 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:36.711696 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:36.897801 31825 rpcz_store.cc:276] 0709 12:24:35.172693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.348308 (+175615us) service_pool.cc:224] Handling call
0709 12:24:36.714573 (+1366265us) spinlock_profiling.cc:243] Waited 76.4 ms on lock 0x7b540037a854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:36.722450 (+  7877us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":175783680}
I20260709 12:24:36.638248  8399 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:36.898519  8273 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:36.890390  8414 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.975984  8414 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:36.978163  8414 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:36.900729 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1435) took 1041 ms. Trace:
I20260709 12:24:36.723394  8412 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:36.917546 31753 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:24:36.718349  8385 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:36.747644  8316 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:36.955359 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:36.957432 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 216121us
I20260709 12:24:36.993364 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:36.991825 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:36.995582 31669 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:36.964352 31569 catalog_manager.cc:5697] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 15 to 22, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 22 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:36.663023  8201 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:36.970968 31717 connection.cc:256] RPC call timeout handler was delayed by -1.4661501069999758s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2126s for a response
W20260709 12:24:36.997990 31717 connection.cc:256] RPC call timeout handler was delayed by -1.3679566330001762s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2546s for a response
W20260709 12:24:36.998471 31717 connection.cc:256] RPC call timeout handler was delayed by -1.1256940540001779s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2799s for a response
I20260709 12:24:36.972190  8387 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:36.982417 32376 rpcz_store.cc:276] 0709 12:24:35.859518 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.859654 (+   136us) service_pool.cc:224] Handling call
0709 12:24:35.860121 (+   467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.945263 (+ 85142us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.900347 (+955084us) spinlock_profiling.cc:243] Waited 7.74 ms on lock 0x7b54005c30d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:36.900645 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17795968}
I20260709 12:24:36.983202 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 17 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:36.971089  8230 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:24:36.971032 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 259419us
I20260709 12:24:36.971722  7805 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:36.982002  8002 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:37.001175 31678 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 16.
I20260709 12:24:37.001204 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:37.019770 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:37.020543  8232 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:37.020829  8270 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:37.020953 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:36.970979  8309 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.021660 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:37.021853  8415 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:37.022449  8415 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:37.022859  8415 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:37.005147  8416 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.023443 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:37.023731  8415 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.024698  8271 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:37.024803 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:37.025111  8415 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:37.022300 31825 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 17.
I20260709 12:24:37.025969 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:37.026324  8412 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:37.026733  8412 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:37.027168  8412 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:37.030840 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:37.031140  8226 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:37.004343 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:36.728945  8151 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.031767  8226 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.003554 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:36.766366  8389 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.034209  8226 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:37.025192  8271 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:37.035282  8271 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:37.023571  8309 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:37.018404 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:37.017444 31821 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:37.036432 31820 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:37.019814 31569 catalog_manager.cc:5697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 20 to 22, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 22 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:37.036755 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:37.032605  8151 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:36.973742  8393 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:37.044265 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:37.032940  8232 consensus_queue.cc:1175] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b95aeeaf4c51442b8092cff1de674f2b" responder_term: 27 status { last_received { term: 25 index: 16 } last_committed_idx: 16 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:36.998354  8316 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:36.991744  8327 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.038220 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:37.006558 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:37.070425 31569 catalog_manager.cc:5697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 14 to 17, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 17 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:24:37.079156 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:37.022233 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:37.084097 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:37.038115 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:37.045253 31822 raft_consensus.cc:2410] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 27 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:24:37.000353  8264 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.005537  8372 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:36.998136  8418 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.037971  7805 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.008751  8371 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:37.005589  8230 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:37.054553 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3679) took 2410 ms. Trace:
I20260709 12:24:37.070760  8425 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:37.057473 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:37.098892 31818 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:37.057863 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3544) took 1025 ms. Trace:
I20260709 12:24:37.071892 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:37.099977 31819 rpcz_store.cc:276] 0709 12:24:36.032702 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.034684 (+  1982us) service_pool.cc:224] Handling call
0709 12:24:36.035305 (+   621us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:36.581945 (+546640us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.581993 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:36.582584 (+   591us) log.cc:844] Serialized 26 byte log entry
0709 12:24:36.670252 (+ 87668us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:36.670293 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.670365 (+    72us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:37.057659 (+387294us) raft_consensus.cc:1685] finished
0709 12:24:37.057680 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.057807 (+   127us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":119627,"raft.queue_time_us":313333,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":10964,"thread_start_us":498039,"threads_started":2,"wal-append.queue_time_us":399676}
I20260709 12:24:37.085740 31569 catalog_manager.cc:5697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 19 to 22, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 22 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
W20260709 12:24:37.068740  8012 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.697s	user 0.011s	sys 0.000s
I20260709 12:24:37.090219 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:37.069504  8270 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:37.060817  7959 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.031s
I20260709 12:24:37.089563  8201 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:37.096681 31750 rpcz_store.cc:276] 0709 12:24:34.644569 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.446463 (+801894us) service_pool.cc:224] Handling call
0709 12:24:35.446843 (+   380us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.773870 (+327027us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:35.792699 (+ 18829us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.023764 (+231065us) log.cc:844] Serialized 14 byte log entry
0709 12:24:36.024502 (+   738us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:36.025343 (+   841us) log.cc:844] Serialized 26 byte log entry
0709 12:24:36.025549 (+   206us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:36.025599 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.025715 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:36.531280 (+505565us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x7b180048af00. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:36.712656 (+181376us) spinlock_profiling.cc:243] Waited 6.47 ms on lock 0x7b180048af00. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:37.054300 (+341644us) raft_consensus.cc:1685] finished
0709 12:24:37.054322 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.054480 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":19017,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":88,"raft.queue_time_us":79919,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":3314,"spinlock_wait_cycles":26314496,"thread_start_us":546236,"threads_started":2,"wal-append.queue_time_us":656551}
I20260709 12:24:37.106165 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:37.146572 31750 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:24:37.054716  8184 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.042s
I20260709 12:24:37.061990  8393 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.087669  8002 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:24:37.121943 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.103776 31820 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:37.143411 31678 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:37.094061  8309 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:24:37.106141 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.148211 31750 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:37.101101  8230 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:37.188369  7733 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:37.100762  8425 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:37.144018  8270 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:37.104560  8371 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:37.094277  8372 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.194253  7733 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:37.094463  8418 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.092167  8264 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.179759  8327 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.177965  8309 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:37.203490  7805 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:37.196671  8343 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.850s	user 0.012s	sys 0.000s
I20260709 12:24:37.145927  8012 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.100768  8226 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:37.201829  7959 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.201104  8270 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:37.197242  8392 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:24:37.270025 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 197472us
I20260709 12:24:37.587440 31675 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: [27.19-27.19]
W20260709 12:24:37.615022 32341 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.002s	sys 0.013s
W20260709 12:24:37.615634 32341 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.507s	user 0.002s	sys 0.013s
I20260709 12:24:37.601424 31746 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:37.601962 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.007s	sys 0.011s
W20260709 12:24:37.622764 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.008s	sys 0.011s
W20260709 12:24:37.623373 31749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1279) took 2994 ms (client timeout 2174 ms). Trace:
W20260709 12:24:37.608090 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:37.623988 31749 rpcz_store.cc:269] 0709 12:24:34.637885 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.039952 (+402067us) service_pool.cc:224] Handling call
0709 12:24:37.622714 (+2582762us) spinlock_profiling.cc:243] Waited 827 ms on lock 0x7b5400065254. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:37.622987 (+   273us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1902254848}
I20260709 12:24:37.148543 31569 catalog_manager.cc:5697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 13 to 21. New cstate: current_term: 21 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:37.628756 31569 catalog_manager.cc:5697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 15 to 16, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 16 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:24:37.630326 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:37.636018 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3723) took 1797 ms. Trace:
W20260709 12:24:37.623374 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 589950us
W20260709 12:24:37.637866 31674 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3707) took 2545 ms (client timeout 1512 ms). Trace:
I20260709 12:24:37.638185 31673 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:37.605284 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2968) took 1787 ms. Trace:
I20260709 12:24:37.596297 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:37.645946  7295 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
W20260709 12:24:37.592410 31672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1391) took 2485 ms (client timeout 2347 ms). Trace:
I20260709 12:24:37.270085  7733 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.598124 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1401) took 1847 ms. Trace:
W20260709 12:24:37.493597  7756 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.593s	user 0.012s	sys 0.000s
I20260709 12:24:37.462240  7759 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:37.206441  7766 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.649195 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1385) took 2550 ms. Trace:
I20260709 12:24:37.649626  7889 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:37.633791 31643 connection.cc:256] RPC call timeout handler was delayed by -1.1459161259999746s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2073s for a response
I20260709 12:24:37.465335  8167 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:37.220906  8002 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:24:37.250353  8433 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.142s	user 0.000s	sys 0.001s
I20260709 12:24:37.711145  8002 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:37.711719  8002 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:37.420799  8012 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.425760  8230 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.721405  8230 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:37.488073  8264 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:37.589514 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2803) took 1332 ms. Trace:
I20260709 12:24:37.722615 31670 rpcz_store.cc:276] 0709 12:24:36.256557 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.630736 (+374179us) service_pool.cc:224] Handling call
0709 12:24:36.631950 (+  1214us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:36.951475 (+319525us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.588591 (+637116us) spinlock_profiling.cc:243] Waited 52.1 ms on lock 0x7b540022ae54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:37.589198 (+   607us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":119899008}
I20260709 12:24:37.723196 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2802) took 1466 ms. Trace:
I20260709 12:24:37.724596 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:37.723740 31671 rpcz_store.cc:276] 0709 12:24:36.256271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.627639 (+371368us) service_pool.cc:224] Handling call
0709 12:24:36.627981 (+   342us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:36.628334 (+   353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:36.628366 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:36.628804 (+   438us) log.cc:844] Serialized 26 byte log entry
0709 12:24:36.756891 (+128087us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:36.756939 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.757010 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:37.722711 (+965701us) raft_consensus.cc:1685] finished
0709 12:24:37.722759 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.723028 (+   269us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":105,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":12014,"thread_start_us":127778,"threads_started":1,"wal-append.queue_time_us":458208}
I20260709 12:24:37.726053 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:37.304651  8271 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.699204 31676 rpcz_store.cc:276] 0709 12:24:35.750820 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.615534 (+864714us) service_pool.cc:224] Handling call
0709 12:24:37.592763 (+977229us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.593606 (+   843us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:37.593678 (+    72us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:37.593771 (+    93us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:37.593831 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.593947 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.597745 (+  3798us) spinlock_profiling.cc:243] Waited 977 ms on lock 0x7b54002e8450. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:37.597951 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":58282,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":339,"spinlock_wait_cycles":2246171520}
I20260709 12:24:37.726974 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:37.727632 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:37.729060 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 23 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:37.729617  8271 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:37.729723  8025 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:37.729924 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:37.209316  8303 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:37.380364  8273 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.452690  8316 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.730981  8376 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.730786  8025 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.731698  8376 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.732961  8316 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.269423  8309 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:37.230103  8327 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.733755  8275 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.272320  8343 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.673790 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:37.737972 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:37.738435 31750 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:37.738933 32378 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:37.734534  8275 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.297279  8371 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.739758 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:37.304185  8372 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:37.220188  8378 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:24:37.655440 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.741809 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:37.742547 31677 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:37.743458 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:37.653683  8132 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:37.230535  8437 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
I20260709 12:24:37.651778  7959 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.706867 31675 rpcz_store.cc:276] 0709 12:24:35.099033 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.574306 (+1475273us) service_pool.cc:224] Handling call
0709 12:24:36.575389 (+  1083us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:37.615453 (+1040064us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.648737 (+ 33284us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54002544d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:37.649103 (+   366us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2325144448}
I20260709 12:24:37.748714 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1239) took 1793 ms. Trace:
I20260709 12:24:37.304838  8389 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.749136 32377 rpcz_store.cc:276] 0709 12:24:35.955253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.955373 (+   120us) service_pool.cc:224] Handling call
0709 12:24:35.955670 (+   297us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:35.955990 (+   320us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:35.956016 (+    26us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:35.956398 (+   382us) log.cc:844] Serialized 26 byte log entry
0709 12:24:36.738960 (+782562us) spinlock_profiling.cc:243] Waited 5.64 ms on lock 0x7b180000f580. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:24:36.988675 (+249715us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:36.988737 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:36.988838 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:37.748135 (+759297us) raft_consensus.cc:1685] finished
0709 12:24:37.748178 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.748494 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":969,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":2939,"spinlock_wait_cycles":12976000,"thread_start_us":245242,"threads_started":1,"wal-append.queue_time_us":1051771}
I20260709 12:24:37.750104  8389 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.749464 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2806) took 1492 ms. Trace:
I20260709 12:24:37.751583 31675 rpcz_store.cc:276] 0709 12:24:36.257395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.748313 (+1490918us) service_pool.cc:224] Handling call
0709 12:24:37.748812 (+   499us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.749098 (+   286us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:37.749121 (+    23us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:37.749199 (+    78us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:37.749230 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.749282 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:37.749392 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:37.432638  8387 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.753600 31675 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 24 index: 12. Preceding OpId from leader: term: 25 index: 12. (term mismatch)
I20260709 12:24:37.754058 31675 pending_rounds.cc:85] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 11
I20260709 12:24:37.488425  8390 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:37.367718  8393 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:37.670913 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.677368 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:37.384554  8418 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.481413  8412 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.196321  8425 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:37.242244  8420 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:37.663391 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48714.7us
W20260709 12:24:37.671188 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.688606 31821 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.15-24.15]
W20260709 12:24:37.687400 31678 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.533s	user 0.018s	sys 0.003s
W20260709 12:24:37.696190 31674 rpcz_store.cc:269] 0709 12:24:35.092235 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.180660 (+ 88425us) service_pool.cc:224] Handling call
0709 12:24:37.637244 (+2456584us) spinlock_profiling.cc:243] Waited 1.3 s on lock 0x7b54002b0554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:37.637578 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2999517568}
I20260709 12:24:37.658829 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3702) took 1145 ms. Trace:
I20260709 12:24:37.654819  8184 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:37.695960 31747 rpcz_store.cc:276] 0709 12:24:35.817138 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.973030 (+155892us) service_pool.cc:224] Handling call
0709 12:24:36.010261 (+ 37231us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.559508 (+549247us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.604505 (+1044997us) spinlock_profiling.cc:243] Waited 508 ms on lock 0x7b54004033d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:37.604913 (+   408us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1168033152}
W20260709 12:24:37.696861 31672 rpcz_store.cc:269] 0709 12:24:35.107043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.034400 (+927357us) service_pool.cc:224] Handling call
0709 12:24:37.591806 (+1557406us) spinlock_profiling.cc:243] Waited 275 ms on lock 0x7b54002e8454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:37.592100 (+   294us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":632512896}
I20260709 12:24:37.695981 31823 rpcz_store.cc:276] 0709 12:24:35.838812 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.838943 (+   131us) service_pool.cc:224] Handling call
0709 12:24:35.839215 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:36.222709 (+383494us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.606315 (+1383606us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54004466d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:37.635698 (+ 29383us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":447369088}
I20260709 12:24:37.715077  7756 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:37.714953  7889 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.716264  7733 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:37.717676 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2804) took 1460 ms. Trace:
I20260709 12:24:37.769320 31746 rpcz_store.cc:276] 0709 12:24:36.513530 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.598518 (+ 84988us) service_pool.cc:224] Handling call
0709 12:24:36.598814 (+   296us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.619685 (+1020871us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.658295 (+ 38610us) spinlock_profiling.cc:243] Waited 1 s on lock 0x7b54001edcd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:37.658716 (+   421us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2304182144}
I20260709 12:24:37.730160 31676 raft_consensus.cc:2410] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 27 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 14.
I20260709 12:24:37.738013  8343 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.741446  8378 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:37.653254  8053 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.650866 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:37.763283 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:37.754431 31675 pending_rounds.cc:107] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 24.12
I20260709 12:24:37.770399 31678 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:37.779074 31675 raft_consensus.cc:2889] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:37.770674  8025 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:37.773545 31673 rpcz_store.cc:276] 0709 12:24:36.256827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.736997 (+480170us) service_pool.cc:224] Handling call
0709 12:24:37.637550 (+900553us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.706103 (+ 68553us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.717360 (+ 11257us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b54002b0550. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:37.717589 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2069782400}
I20260709 12:24:37.778396 31670 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 23.
I20260709 12:24:37.783002  8184 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.784364  7889 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.786177  8184 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:37.788774  8275 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.790967  8448 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:37.793119 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:37.792950  8053 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.794575  8378 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.795199 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:37.795847  8343 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.800467 31744 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 21. Current term is 22. Ops: []
I20260709 12:24:37.802024 31672 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:37.805356 31753 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:37.797326 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 67652.8us
I20260709 12:24:37.809460 31677 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
W20260709 12:24:37.800643 31717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.693s	user 0.003s	sys 0.006s
I20260709 12:24:37.816088 32377 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:37.817619  8273 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:37.817973  8053 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.819221 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2810) took 1560 ms. Trace:
I20260709 12:24:37.800779  8201 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:24:37.820065 31717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.731s	user 0.003s	sys 0.007s
I20260709 12:24:37.820447  8201 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:37.821074  8201 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:37.821743 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 799516us
I20260709 12:24:37.821779  8448 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.823587  8420 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.819764 31672 rpcz_store.cc:276] 0709 12:24:36.258535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.780027 (+1521492us) service_pool.cc:224] Handling call
0709 12:24:37.801762 (+ 21735us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.818863 (+ 17101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.819144 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:37.826676  8448 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:37.827121  8412 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:37.822085  8343 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.828190 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:37.831148  8445 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:24:37.762569  8132 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.832995  8275 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:37.833912 31669 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:24:37.835049  8275 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:37.835573  8275 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:37.836019  8155 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.836588  8155 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.836712  8275 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.838016  8275 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:37.812273 31671 raft_consensus.cc:2376] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28. Re-sending same reply.
I20260709 12:24:37.780885  7959 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:37.839924  7889 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.840605  7889 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.823589  8327 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.842839 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:37.843109  8376 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.846390  8389 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:37.848024 31719 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.003s	sys 0.007s
I20260709 12:24:37.819748 31823 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:24:37.820050 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:37.854938  8053 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:24:37.856938 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:37.814411 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2807) took 1552 ms. Trace:
I20260709 12:24:37.867612 31675 rpcz_store.cc:276] 0709 12:24:36.257659 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:37.752607 (+1494948us) service_pool.cc:224] Handling call
0709 12:24:37.753061 (+   454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.808250 (+ 55189us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.809587 (+  1337us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:37.839485  8270 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.833799 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:37.857832  8387 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.861330  8132 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:37.870651 31719 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.711s	user 0.003s	sys 0.007s
I20260709 12:24:37.867055 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:37.881611 31750 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:37.884778 31748 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 20.
W20260709 12:24:37.886461 31748 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 2965) took 2602 ms (client timeout 1917 ms). Trace:
W20260709 12:24:37.886910 31748 rpcz_store.cc:269] 0709 12:24:35.284134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.344966 (+ 60832us) service_pool.cc:224] Handling call
0709 12:24:37.886192 (+2541226us) spinlock_profiling.cc:243] Waited 2.52 s on lock 0x7b54002ac6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:37.886386 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5800885632}
I20260709 12:24:37.888249 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:37.889003 31821 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:37.889923 31748 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:37.890481 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3689) took 2970 ms. Trace:
I20260709 12:24:37.891011 31744 rpcz_store.cc:276] 0709 12:24:34.919968 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.633326 (+713358us) service_pool.cc:224] Handling call
0709 12:24:37.633307 (+1999981us) spinlock_profiling.cc:243] Waited 21 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:24:37.633835 (+   528us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.830715 (+196880us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:37.890218 (+ 59503us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x7b54002963d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:37.890395 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":338926208}
I20260709 12:24:37.892865 31744 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:24:37.893591  8376 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.894197  8376 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.896199 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:37.896574  8376 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:37.897931 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.590s	user 0.000s	sys 0.008s
W20260709 12:24:37.898442 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.715s	user 0.000s	sys 0.008s
I20260709 12:24:37.899330 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:37.899539 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 793391us
I20260709 12:24:37.900010  8167 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:37.900131 31822 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:37.900580  8167 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:37.900998  8167 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:37.884907  7805 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.884162 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:37.901206 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:37.907675 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74081.5us
I20260709 12:24:37.908671 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:37.909512 31819 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:37.910132  8420 raft_consensus.cc:2764] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:24:37.910943 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:37.910887 31749 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:37.913671  8343 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.914294 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:37.915189 32377 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:37.916757  7805 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.917001  8425 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.917943  8270 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.918926 31752 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:37.919344  7805 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:37.919958 31752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3698) took 1415 ms (client timeout 1865 ms). Trace:
I20260709 12:24:37.919452  8425 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:37.920425 31752 rpcz_store.cc:269] 0709 12:24:36.504088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:36.535831 (+ 31743us) service_pool.cc:224] Handling call
0709 12:24:37.629736 (+1093905us) spinlock_profiling.cc:243] Waited 2.51 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:24:37.919701 (+289965us) spinlock_profiling.cc:243] Waited 211 ms on lock 0x7b54004038d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:37.919878 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":490831488}
I20260709 12:24:37.923485  7889 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:37.886924  8155 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.926198 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:37.908115  8399 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.929000 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:37.929610  8399 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.919579 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:37.927413  8389 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:37.928153 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:37.928624  8130 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:37.932003 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:37.933332 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 83341.7us
W20260709 12:24:37.939026 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 159785us
I20260709 12:24:37.940146  8343 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:37.940935  8343 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.944072  8343 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:37.945930  8343 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:37.946452  8002 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:37.937376 31746 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:24:37.979758 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:37.978024 31745 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 20 index: 14. (index mismatch)
I20260709 12:24:37.951186  8132 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 25.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.004s
I20260709 12:24:37.936908  8420 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 12:24:37.990667  8343 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:37.990767  8155 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:37.991168  8343 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:37.992059 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:37.992560 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:37.992770 31826 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:37.994213 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:37.984223 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:37.984719 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 56719.7us
I20260709 12:24:37.985090 32342 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:37.986096 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 60050.1us
I20260709 12:24:37.987947 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:37.989070  7805 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:37.990460 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:37.992389  8343 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:37.978314 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:37.996080 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:37.978574 31822 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:37.932765  8053 consensus_queue.cc:1243] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 26.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:24:37.999430  7805 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:38.003123 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 191075us
I20260709 12:24:37.992769  8420 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:37.955502 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.008050  8420 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:38.014783 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 77222.5us
I20260709 12:24:37.928148  8412 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.079s
I20260709 12:24:38.021636 32377 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:38.023500  7295 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:38.023984  7295 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.024334  7295 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:38.006835  7805 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:38.008730  8002 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:38.046486 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:38.047037  7733 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 21.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:38.008597  8155 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:38.048877  7889 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:38.049876 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:38.050733  7959 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:38.051388  7959 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.051767  7959 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:38.019285  8201 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.034830  8132 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:38.037919  7295 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:38.039057 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 57844.7us
W20260709 12:24:38.039783 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 54857.4us
I20260709 12:24:38.040526 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.040979 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.041656 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.043891 31823 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:38.043892 31826 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:38.045866 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:38.050792 31824 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:38.029824  8309 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:38.151758 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 125304us
W20260709 12:24:38.097230 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.050485  7889 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:38.153033 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 126934us
I20260709 12:24:38.153710 31753 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:38.189709  7295 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:38.190722 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 142495us
I20260709 12:24:38.195587 31748 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:38.194059 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:38.199365 31821 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:38.071414  8132 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:38.205892 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:38.207365 31751 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 28. Ops: []
I20260709 12:24:38.208846 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.210007 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:38.215327 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.217737  8265 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:38.218173 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:38.218511  8265 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.218925  8445 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:38.221009  8265 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:38.223098  8054 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:38.224014  8309 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:38.228448 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:38.230741  8447 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:38.230866 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:38.231791 32370 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:38.231688  8457 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
I20260709 12:24:38.232964 31819 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:24:38.233126  8457 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:38.233600  8457 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:38.234310 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:38.235049 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:38.238672 31825 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:24:38.239908 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:38.201900  8343 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:38.237429 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:38.242337 31822 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:38.244195  8399 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:38.245155 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.245873 31671 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 16.
I20260709 12:24:38.246060 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.208916 31752 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 17.
I20260709 12:24:38.214360 31744 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:24:38.248847 31823 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:38.201332  8025 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.252383  8025 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:38.253286  8460 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
W20260709 12:24:38.255074 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55815.3us
I20260709 12:24:38.245193 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:38.256232 31678 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:24:38.206017  8132 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:38.261345 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.264868  8025 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:38.264993 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:38.266301 31819 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:38.266470 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.267273 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:38.271481 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74144.8us
I20260709 12:24:38.272568 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:38.273725  8446 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:38.274179  8446 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.274570  8446 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:38.267289 31671 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:38.285482 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.288915 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.289655 31749 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:38.291529  8201 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:38.295308 31746 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:24:38.298604 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 2966) took 2482 ms. Trace:
I20260709 12:24:38.290552 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.306198 31677 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:24:38.219681  8420 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:38.307272  8184 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:38.307605 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:38.307921  8184 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:38.309259 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:38.309121  8184 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.299196 31751 rpcz_store.cc:276] 0709 12:24:35.816612 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.822269 (+  5657us) service_pool.cc:224] Handling call
0709 12:24:37.626973 (+1804704us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:38.274001 (+647028us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:38.297794 (+ 23793us) spinlock_profiling.cc:243] Waited 560 ms on lock 0x7b5400065254. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:38.298402 (+   608us) spinlock_profiling.cc:243] Waited 1.8 s on lock 0x7b5400065250. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:38.298547 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5436849280}
I20260709 12:24:38.316586 31827 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:38.317027 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:38.318054 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 295496us
W20260709 12:24:38.321455 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.323817 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:38.324551 31818 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:38.226518  8054 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:38.330765 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.331033  8054 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.332923 32370 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:38.333289  8054 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:38.334669 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.335491 31746 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:38.316414 31749 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
W20260709 12:24:38.340138 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.341096 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.342672 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.343196 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.343386 31674 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:38.343879 31669 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:38.349586 31822 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:38.351191 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.354573 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.355885 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.234546  8457 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:38.359687 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.359761 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.358469 31752 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
W20260709 12:24:38.366672 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.359082  8412 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.371982 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.371086  8457 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:38.377485  8445 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:38.378707  8445 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.13, Last appended by leader: 13, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.379221  8012 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.380533  8012 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.381199 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:38.381883 31672 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:38.382284  8378 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:38.358069  8130 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.372090 31671 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
W20260709 12:24:38.382310 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 181042us
I20260709 12:24:38.388185 31746 raft_consensus.cc:2410] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 29 because replica has last-logged OpId of term: 26 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
I20260709 12:24:38.340855  8167 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.371447 31751 pending_rounds.cc:85] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 15
I20260709 12:24:38.390177 31751 pending_rounds.cc:107] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 26.16
I20260709 12:24:38.390457 31751 raft_consensus.cc:2889] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:38.390408 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:38.391206 32370 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
W20260709 12:24:38.386293 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 112744us
W20260709 12:24:38.386785 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 71728.1us
I20260709 12:24:38.396571 31753 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:24:38.397995 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.399839 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:38.382856  7756 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:38.395107 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:38.405054 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.406258 31821 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:38.398952 31750 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:38.392726  8265 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.352557  8446 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.408538 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.410187 31744 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:38.412464 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:38.416530 31675 raft_consensus.cc:2368] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27.
I20260709 12:24:38.417456 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.424889 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:38.427536  8462 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
W20260709 12:24:38.429462 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.430173 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.394737  8002 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.432077  7295 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:38.435503 31746 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:38.432355 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:38.437003 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48104.1us
I20260709 12:24:38.442085 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:38.403995  8309 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.431115  8448 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:38.411084  8130 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:38.451121 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:38.451850 31746 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:38.454905 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.455035  7295 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:38.455838 31744 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:38.455696 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 17 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:38.456149  7295 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.456650  8465 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:38.455744  7889 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:38.460635  7959 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.463004 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:38.463744 31678 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:24:38.463640  7805 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:38.465106 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:38.472189 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:38.473446  8420 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:38.473956  8420 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:38.474423  8420 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:38.479252  8387 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:38.479782  8387 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:38.480226  8387 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:38.481304  8167 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:38.482120 31672 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:38.485719 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:38.486591 31753 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:38.489137  8455 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.460s	user 0.002s	sys 0.002s
I20260709 12:24:38.488584 31746 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:38.490667 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.493867 31747 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:38.452083  8130 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:38.495106 31827 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:38.495332  8130 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.495496 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.497334  8449 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:38.446571  8012 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:38.490048 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.493698 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.499382  8376 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:38.500097  8376 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 21 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:38.501187  8376 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:38.503826  8309 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:38.504199 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3691) took 3410 ms. Trace:
I20260709 12:24:38.504742 31745 rpcz_store.cc:276] 0709 12:24:35.093397 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:35.637753 (+544356us) service_pool.cc:224] Handling call
0709 12:24:37.907736 (+2269983us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:37.981064 (+ 73328us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:38.503000 (+521936us) spinlock_profiling.cc:243] Waited 23.8 ms on lock 0x7b54002ac6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:38.503931 (+   931us) spinlock_profiling.cc:243] Waited 2.26 s on lock 0x7b54002ac6d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:38.504105 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5261696512}
I20260709 12:24:38.505630  8376 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:38.506199  8309 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:24:38.508314 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51149.6us
I20260709 12:24:38.509891 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.484719 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:38.511333 31750 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:24:38.478467 31818 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
W20260709 12:24:38.512382 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 78345.2us
I20260709 12:24:38.496065  8130 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:38.523797 31748 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:24:38.541940 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57622.3us
I20260709 12:24:38.543267 31677 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:38.400507  8412 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:38.466411  7805 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:38.439337  8265 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.556097 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:38.552534  8389 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:38.575850 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:38.584204 31826 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:38.522945  8002 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:38.598515 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 173755us
I20260709 12:24:38.602840 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:38.612407 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:38.558269  8448 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:38.612758 31748 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:38.588636  8389 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:38.619076 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:24:38.619256 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.604069  8012 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:38.625561 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:38.635208  8464 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 16 FOLLOWER]: Advancing to term 17
I20260709 12:24:38.579294  8457 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.042s
W20260709 12:24:38.626149  8155 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.011s	sys 0.019s
I20260709 12:24:38.624655  8389 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:38.646937 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 240971us
I20260709 12:24:38.629518  7756 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:38.614977  8449 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.032s
I20260709 12:24:38.643110  8465 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:38.649518  8155 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.597s	user 0.011s	sys 0.019s
I20260709 12:24:38.618613  8425 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.030s
I20260709 12:24:38.639012  8457 consensus_queue.cc:1243] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 24.14 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:24:38.675606 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:38.646545  7959 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:38.652485 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:38.676311 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62739.6us
I20260709 12:24:38.649986  8447 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:38.681402 31677 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:38.685297 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:38.645484  8467 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.033s
I20260709 12:24:38.683802 31753 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:24:38.665696  8465 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:38.711248  8130 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:38.712766  8002 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:38.712836 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:38.713790 31826 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
W20260709 12:24:38.715040 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.661266  8399 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 15, Time since last communication: 0.033s
I20260709 12:24:38.711375  8465 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:38.714795  8473 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.033s
I20260709 12:24:38.707162 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:38.707810  8446 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:38.709793 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:38.715071 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:38.695827 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 72602.2us
I20260709 12:24:38.675406  8155 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:38.718310  8273 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:38.691524  8270 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:38.722277 32370 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 31: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:38.720376 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:38.735507 31827 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:24:38.738467 32376 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:24:38.717744  8447 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:38.740661 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54270.7us
I20260709 12:24:38.744136 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:38.740981 31818 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:38.746178 31748 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:24:38.741127 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:38.723316  8309 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.025s
I20260709 12:24:38.750591  8412 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:38.757272 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.552s	user 0.011s	sys 0.023s
W20260709 12:24:38.758169 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.553s	user 0.011s	sys 0.023s
W20260709 12:24:38.758880 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 98332.4us
I20260709 12:24:38.746219  8467 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:38.770694 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:38.771756 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:38.751969  7889 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:38.797783 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.760519  8473 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 12:24:38.778614  8309 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Leader pre-election won for term 18
I20260709 12:24:38.804551  8309 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.805042  8309 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:38.779026  7889 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:38.778455  8468 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.130s
I20260709 12:24:38.809202  7889 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:38.810418  8467 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Stepping down as leader of term 22
W20260709 12:24:38.810787 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 101114us
I20260709 12:24:38.811208  8467 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:38.812386  8467 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.812956 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:38.815191  8468 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:38.814311 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:24:38.830087 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 86804us
I20260709 12:24:38.835325 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:38.781694  8412 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:24:38.790735 31792 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.670s (SENT) [suppressed 15 similar messages]
I20260709 12:24:38.798871  8201 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:38.845086  8486 raft_consensus.cc:2764] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:24:38.775198  8132 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.817816  8448 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:38.852186  7295 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:38.856469  7295 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:38.853922 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.845122 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:38.858310  8474 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.161s	user 0.000s	sys 0.001s
I20260709 12:24:38.858603 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:38.858786 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:38.858932 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 114919us
I20260709 12:24:38.872165 31827 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:24:38.859434  8471 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.079s
I20260709 12:24:38.861677 31750 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
W20260709 12:24:38.862061 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 111094us
I20260709 12:24:38.849644  8468 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:38.858649 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:38.840706  8412 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:38.877656  8468 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:38.878322  8468 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:38.878983 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:38.878659  8412 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:38.847935  8201 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.847518  8457 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:24:38.897812 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44681.3us
I20260709 12:24:38.903748 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:38.904161 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.904505 31752 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:24:38.871505  8445 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:38.909997 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:38.890004  8399 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:24:38.926963 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:38.874151  8012 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.889689  8201 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:38.919060 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:38.932791 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 118555us
I20260709 12:24:38.929927 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:38.911563  8486 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:38.912551  7733 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:38.848027  8447 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 21.12, Last appended by leader: 12, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:38.947000 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:38.830678  8155 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:38.947109 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 73175.7us
W20260709 12:24:38.950276 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 71375.8us
I20260709 12:24:38.943323 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:38.916535  8445 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.918247  8449 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:38.949213 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:38.927070  8275 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:38.927244  8132 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:38.963753 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:38.947535  8479 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.105s	user 0.000s	sys 0.000s
I20260709 12:24:38.958799  8273 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:38.949849  8481 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:24:38.971336  8472 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:24:38.965256 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 244964us
I20260709 12:24:38.974928  8327 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:38.948378  8002 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:38.950480  8265 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:38.962576  8445 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:38.967964  7889 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:38.978973  8448 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:24:38.947919  8452 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:38.964380 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:38.968375  8275 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.942711  8484 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:39.015137  7889 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:39.034582  8484 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:39.034888  7889 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:39.035179  8484 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:39.036276  8476 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.277s	user 0.000s	sys 0.002s
I20260709 12:24:38.989202  7733 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:39.038110  7733 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:39.009003 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 218289us
I20260709 12:24:38.938205  8270 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:39.012391 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:39.013411 31825 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:38.962950  8449 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:38.965956  8412 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:38.956723  8457 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.052s
I20260709 12:24:38.988154 31827 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:24:39.013250  7295 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:39.018016 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 68877.1us
I20260709 12:24:38.957985  8486 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:38.947674  8000 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.081s
I20260709 12:24:39.052997  8399 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:39.017998  8452 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:39.049381  8449 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:39.055794  8494 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:39.057116  8201 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.058470  8491 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.059158  8491 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.051564  7756 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.013s
I20260709 12:24:39.061503  8425 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:24:39.029877  8420 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.555s	user 0.008s	sys 0.003s
I20260709 12:24:39.008546  8002 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:39.008699  8327 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:39.016721  8448 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:39.074183  8448 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:24:39.031483 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 73560.4us
I20260709 12:24:39.075337 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:39.032348  8473 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
W20260709 12:24:39.033789  8387 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.553s	user 0.011s	sys 0.000s
I20260709 12:24:39.033399  8273 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:38.992177  8472 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:38.960091  8012 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.041172 31825 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:39.048418 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 96441.7us
W20260709 12:24:39.020696  8475 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
I20260709 12:24:39.067291  7295 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:39.110294 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59911.2us
I20260709 12:24:39.040431  8275 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:39.069396 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:39.072642  8002 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.073647  8327 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:39.077346 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:39.067701 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 77582.7us
I20260709 12:24:39.062011  8425 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:39.078915  8420 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.099449  8472 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:39.055346  8486 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:39.126658 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57376.6us
I20260709 12:24:39.128141 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:39.077981  8483 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:39.093811  8473 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:39.106748  8465 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:39.142082 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64822.9us
I20260709 12:24:39.077175  8468 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.141510 31825 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.146085  8130 raft_consensus.cc:2764] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:24:39.149641  8420 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:39.150153 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.151156 31825 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:39.153090 32370 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:24:39.153605 32378 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:39.099254  8387 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.154016 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:39.154598 32377 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:39.155968  8387 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:39.157598  8376 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.633s	user 0.010s	sys 0.000s
I20260709 12:24:39.158877  8452 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.160235  8452 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:39.080770  8270 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.167464  8309 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.168749  8467 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:39.144476  8473 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:39.172395 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:39.174870  8167 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:39.175338  8167 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:39.175107  8389 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.175652 31825 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 14. (term mismatch)
I20260709 12:24:39.175822  8167 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:39.177507  8012 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:39.177645 31745 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:39.179288  8201 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:39.179994  8201 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:39.184283 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:39.185333  8130 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:39.185249  7889 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.185701 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:39.185818  8130 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:39.186429  8130 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:39.186659  8486 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:39.176029 31825 pending_rounds.cc:85] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 13
I20260709 12:24:39.188254 31822 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:39.188261 31825 pending_rounds.cc:107] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 27.14
I20260709 12:24:39.188717 31825 raft_consensus.cc:2889] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:39.190263 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.137511  8378 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:24:39.205401 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:39.202373 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 55086.2us
I20260709 12:24:39.207533 32377 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:24:39.139914  8483 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:39.184283  8468 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:39.217723 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:39.218755 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:39.219641 31827 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 17. (term mismatch)
I20260709 12:24:39.235389 31827 pending_rounds.cc:85] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 16
I20260709 12:24:39.235737 31827 pending_rounds.cc:107] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 26.17
I20260709 12:24:39.236047 31827 raft_consensus.cc:2889] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:39.220436 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:39.221366 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:39.227067 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:39.236958 31826 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:39.222350 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:39.227458 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.232771 31791 leader_election.cc:400] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27.
I20260709 12:24:39.233394 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:39.238498 31791 leader_election.cc:403] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
W20260709 12:24:39.239689 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.236985 31824 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:39.230893 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:39.241127  8378 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:39.193039  8309 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:39.246570  8309 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.246711  8378 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.247112  8309 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.233960  8491 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:39.233791  8000 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.017s
I20260709 12:24:39.209563  8155 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.043s
I20260709 12:24:39.177549  8389 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:39.254994 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.241329 32376 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28. Ops: [25.13-27.14]
I20260709 12:24:39.257563 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.257761 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.258527 31672 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:39.258790 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.258924 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.259699 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.260231 31678 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:39.261806 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 72671.1us
I20260709 12:24:39.245180 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:39.262705 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:39.263384 31674 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
W20260709 12:24:39.210541  8464 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.575s	user 0.018s	sys 0.000s
I20260709 12:24:39.260344 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:39.278529 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.275089 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:39.287287 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:39.204439  8480 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:24:39.292572 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61802.8us
I20260709 12:24:39.293044 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.293408 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:39.303196 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:39.306986 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:39.234454  8483 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:39.301716 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 20 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:39.312043 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:39.314432 31824 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:24:39.315052 32378 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:39.309360 31678 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:39.187278  8425 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.333930 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.327956 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:39.337028 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:39.336603 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 72950.3us
I20260709 12:24:39.304443  8464 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Leader pre-election lost for term 17. Reason: could not achieve majority
I20260709 12:24:39.296238  8480 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:39.307075  8457 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:24:39.422966 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:39.402426 31749 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:39.396874 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:39.310708  8000 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.407552 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.327167  7889 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:39.498203 31669 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:39.418979 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:39.419957 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.422762 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:39.440716 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.433758 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.439754 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:39.454144 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:39.501504 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.454866 31753 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28. Ops: [27.14-27.14]
I20260709 12:24:39.424763 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.341173  8270 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.462116 31671 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
W20260709 12:24:39.509652 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.450517 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:39.510442 31676 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:39.510499 32372 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:39.422969 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.503679 31675 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:39.345146  8506 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.432039 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:39.521355 32373 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:39.522440 31824 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:39.406430  8480 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:39.522706 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.523849 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.527225 31671 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 32 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
I20260709 12:24:39.527140 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:39.514097 32378 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:39.513347 31677 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:39.528187 31748 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:24:39.524202 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:39.514737 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:39.529379 31749 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:39.529072 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:39.525300 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:39.496114  8457 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:39.531738 31824 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:39.440716  8468 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.535111 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:39.529273 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:39.530853 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:39.538123 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:39.538753 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:39.543167 31750 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:24:39.540163 31672 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:39.544736 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:39.553824 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:39.554978 31824 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:39.437690  8449 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.525791  8412 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27.
I20260709 12:24:39.568845 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.571549 32377 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:39.568136 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:39.570989 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.557634 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:39.541821  8457 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:24:39.574545 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:39.572526 31675 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:39.530902  7756 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:39.509243  8275 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.575536 31748 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 17 FOLLOWER]: Advancing to term 18
I20260709 12:24:39.565510  8447 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:24:39.637312 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 79784.2us
I20260709 12:24:39.568720  7733 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.546875  8273 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:39.564478  8486 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:39.541931  8000 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.517392  8270 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.533667  8130 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.542061  8399 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 12:24:39.541227  8167 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.663335  8412 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.664335  8167 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:39.664016  8412 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.665170 31678 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:39.666666 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:39.666538 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.667744 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.668483 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:39.668848 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.669099 31822 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:39.669939 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.670555  7805 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:39.671123  7805 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:39.675608  8504 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
I20260709 12:24:39.677431 32377 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 12. Preceding OpId from leader: term: 28 index: 14. (index mismatch)
I20260709 12:24:39.677968 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:39.687570 32374 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:24:39.688951 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:39.689400  8491 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:39.690202 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:39.530910  8506 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.653489 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.640985  8447 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:39.703088 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:39.655748 31674 raft_consensus.cc:2368] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29.
I20260709 12:24:39.656771 31677 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:39.656599 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:39.581421  8420 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:39.658689  7889 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
W20260709 12:24:39.658911  8501 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.260s	user 0.001s	sys 0.000s
I20260709 12:24:39.659250  8508 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:39.733368  8508 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:39.734840 31825 pending_rounds.cc:85] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 13
I20260709 12:24:39.734375  8508 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.735165 31825 pending_rounds.cc:107] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 24.14
I20260709 12:24:39.735435 31825 raft_consensus.cc:2889] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:39.659662  8512 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:39.737670 32375 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:24:39.638010  7756 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.739310 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.739950 31674 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:39.741046  7756 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:39.743459 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.744110 31670 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
W20260709 12:24:39.744405  8445 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.708s	user 0.002s	sys 0.008s
I20260709 12:24:39.744830  8445 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.745170 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.701105  7805 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:39.649782  8465 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:39.755189 31745 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 31: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:24:39.726485  8448 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.648s	user 0.002s	sys 0.005s
W20260709 12:24:39.693163 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:39.764523 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.669939 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:39.696981 31749 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:24:39.767058 31822 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:39.552093  8468 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.699071 32372 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:39.705212 31675 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:24:39.768532 31821 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:39.706974 32377 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 18: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:24:39.707964 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 63243.6us
I20260709 12:24:39.707741  7733 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.709072  8000 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:39.782019 31670 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:24:39.783880  7733 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:39.711781  8449 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:39.711658  8420 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.713018 31748 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 18.
I20260709 12:24:39.787849  8420 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:39.651564  8273 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.713629  8130 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:39.713735 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.790284  8273 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:39.790547 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.791769 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.792467 31677 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:39.793188 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.714026 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.795723 31673 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: [26.16-27.17]
I20260709 12:24:39.714313 31819 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:39.714534  8012 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:39.717216  8275 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:39.717557  8472 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.573s	user 0.006s	sys 0.001s
I20260709 12:24:39.719321  8309 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:39.720667  8002 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:39.721689 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:39.725179 31827 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: [20.14-20.14]
W20260709 12:24:39.778074  8473 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.607s	user 0.011s	sys 0.000s
I20260709 12:24:39.726337 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.731691 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:39.826052 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.826251 31671 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:39.661942  8389 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:39.755951 31751 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
W20260709 12:24:39.752189 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 449454us
I20260709 12:24:39.627763 31824 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:39.700132 31672 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:24:39.666548  8412 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:39.762240 32373 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:39.773883 32374 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:24:39.727943  8447 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:39.770051  7756 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.010s
I20260709 12:24:39.780593 31675 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:39.838040 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:39.798027 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:39.798429 31677 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
W20260709 12:24:39.771656  8467 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.602s	user 0.010s	sys 0.003s
I20260709 12:24:39.822568 31674 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
W20260709 12:24:39.829461 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 191106us
I20260709 12:24:39.831470 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:39.830274 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:39.833458 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:39.836117 32342 connection.cc:256] RPC call timeout handler was delayed by -1.4092526049998924s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20720000000000002s for a response
I20260709 12:24:39.827808 31822 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:39.765245  8448 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:39.852142 32342 connection.cc:256] RPC call timeout handler was delayed by -1.0020814160002374s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.33570000000000005s for a response
I20260709 12:24:39.855007 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:39.824887  8273 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:39.853854 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:39.857227 32370 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:39.861431 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:39.862126 31750 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:39.863155 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:39.864827 31748 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:39.866559 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:39.826838  8472 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.860410 31753 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: [20.14-20.14]
W20260709 12:24:39.869782 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.872545 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:39.873322 31747 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:39.831640  8445 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:39.879189 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:39.892135 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 214250us
I20260709 12:24:39.893896 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:39.894984 31826 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:24:39.895152 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.895792 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:39.896780  8389 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:39.897560  8389 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:39.902868  7295 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:39.903360  7295 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:39.903682  7295 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:39.903976 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:39.904112 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:39.909446 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:39.910413 31752 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:39.911904 31672 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:24:39.871485  8387 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:39.928073 31750 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:24:39.852082  8000 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.041s
I20260709 12:24:39.828693  8473 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.942167  8473 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:39.947572 31672 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: [24.13-26.14]
W20260709 12:24:39.948601 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 56374.3us
I20260709 12:24:39.931425  8130 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.020s
I20260709 12:24:39.952353 31674 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:39.953258 31670 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:39.914747  8371 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:39.955665 31717 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:39.905860  8499 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:39.962293 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:39.976676 31750 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:39.971205  7295 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.980741 31752 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:39.871233  8201 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.066s
I20260709 12:24:39.974699 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:39.878153  8273 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:39.890888  8420 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.063s
I20260709 12:24:39.898708  8389 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.859812  8486 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:39.978964  8054 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:24:39.994046  8483 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.651s	user 0.008s	sys 0.004s
I20260709 12:24:39.936757  8387 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:40.024252 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 70622.5us
I20260709 12:24:40.031721 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:40.028394 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 132672us
I20260709 12:24:40.029794 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:39.938892  8480 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.032665 32372 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 29 because replica has last-logged OpId of term: 27 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 18.
I20260709 12:24:40.033001 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:40.034291 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:40.036430 31752 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:24:40.036620 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:40.037698 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:40.015949  7756 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
I20260709 12:24:40.040266 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:40.043385 31752 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:40.044757 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:40.045603 31822 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:40.043195 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:40.063406 31669 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:40.065955 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:40.057307 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:39.983124  8523 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.067s
I20260709 12:24:40.057313 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:40.076282 31822 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:24:40.032545  8270 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:40.091357 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:40.198474 31827 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 31: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:40.071719 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:40.191557 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:40.200207 32372 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:24:40.203533 31747 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:24:39.991763  8518 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.088s
I20260709 12:24:40.217446 32373 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 19. (term mismatch)
W20260709 12:24:40.234208  8521 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.170s	user 0.001s	sys 0.000s
I20260709 12:24:40.026168  8499 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
W20260709 12:24:40.239501 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:40.030429  8483 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.218056 32373 pending_rounds.cc:85] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 18
I20260709 12:24:40.263955 32373 pending_rounds.cc:107] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 27.19
I20260709 12:24:40.264348 32373 raft_consensus.cc:2889] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:40.059718  8155 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:24:40.257455 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 242590us
I20260709 12:24:40.047334  8506 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:40.070173  8468 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:40.070425  8054 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:24:40.242772 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 171115us
I20260709 12:24:40.291349 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:40.040480  8519 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.219s
I20260709 12:24:40.249205  8201 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:40.243598  8000 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:24:40.312832 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 269718us
I20260709 12:24:40.313905 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:40.304857 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 238965us
I20260709 12:24:40.308081 31818 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:40.261498  8130 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:40.290153  8054 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.298187  8445 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:40.341739 32373 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:40.301141  8155 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:40.348379 31747 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:40.344154 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:40.327116  8420 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:40.350019  8512 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:40.358387 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:40.359030  8512 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.359475  8512 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:40.359745  8387 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:40.359984 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:40.361191  8273 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:40.362048  8528 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Stepping down as leader of term 20
I20260709 12:24:40.362448  8528 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:40.363525  8530 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:40.363932  8530 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:40.326913  8448 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:40.329912  8472 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:40.335143  8054 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:40.370528 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:40.371541 32372 raft_consensus.cc:2376] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30. Re-sending same reply.
I20260709 12:24:40.317611  8468 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 18, Time since last communication: 0.039s
I20260709 12:24:40.380923 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:40.384742 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:40.386613 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 430991us
I20260709 12:24:40.388839 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:40.387964 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:40.393934 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:40.370083  8467 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:40.379349  8130 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:40.370195  8273 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:40.374387  8201 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:40.398487 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:40.373989  8445 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.022s
W20260709 12:24:40.399791  8497 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.158s	user 0.007s	sys 0.000s
W20260709 12:24:40.400005  8531 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.108s	user 0.001s	sys 0.000s
I20260709 12:24:40.402765  8512 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.402872  8155 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:40.403745  8526 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.162s	user 0.000s	sys 0.001s
W20260709 12:24:40.517278  8447 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.646s	user 0.011s	sys 0.000s
I20260709 12:24:40.404345  8472 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:40.405680  8538 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:40.517784  8447 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.513345  8201 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:40.364753  8530 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.407658  8483 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:40.408232  8448 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:40.380436  8446 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:40.377188  8420 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:40.455021 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61193.4us
W20260709 12:24:40.442838  8327 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.306s	user 0.007s	sys 0.008s
I20260709 12:24:40.444842  8371 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:40.472676 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 73424.9us
I20260709 12:24:40.448859  8499 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:40.464797  8480 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:40.482738 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:40.492959  8484 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.457s	user 0.012s	sys 0.002s
I20260709 12:24:40.493660  8378 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
W20260709 12:24:40.503782  8457 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.867s	user 0.010s	sys 0.003s
W20260709 12:24:40.508898 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 110501us
I20260709 12:24:40.498184  8054 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.519253 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3758) took 1294 ms. Trace:
I20260709 12:24:40.519954 31746 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:24:40.520255 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3584) took 1008 ms. Trace:
I20260709 12:24:40.529825  7756 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:40.536968 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3760) took 1311 ms. Trace:
I20260709 12:24:40.513012  8449 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:40.537990  8448 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:40.538240  8420 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.520385  8273 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.539806 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:40.549849 31822 rpcz_store.cc:276] 0709 12:24:39.225644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.190742 (+965098us) service_pool.cc:224] Handling call
0709 12:24:40.191151 (+   409us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.232840 (+ 41689us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:40.332845 (+100005us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.536447 (+203602us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.536526 (+    79us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:40.536596 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.536693 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.536873 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8622,"prepare.queue_time_us":124676,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":53,"raft.queue_time_us":124,"raft.run_cpu_time_us":180,"raft.run_wall_time_us":5375,"thread_start_us":224157,"threads_started":2,"wal-append.queue_time_us":173825}
I20260709 12:24:40.571885 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:40.540339  8327 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:40.540083  8201 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.13, Last appended by leader: 13, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.517827  8472 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.613806  8472 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:40.506536  8000 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:40.615470  8000 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.511301  8519 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:24:40.617493  8515 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.885s	user 0.003s	sys 0.000s
I20260709 12:24:40.619578  8448 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:40.624218  8273 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:40.624992  8512 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:40.542591 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:40.627261  8447 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:40.523916  8002 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.695s	user 0.011s	sys 0.001s
I20260709 12:24:40.544284 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 18 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:40.627951 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 85439.8us
W20260709 12:24:40.628384 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 84196.1us
I20260709 12:24:40.629208 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:40.547501  8499 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:40.630812  8446 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader election lost for term 18. Reason: could not achieve majority
I20260709 12:24:40.568204  7295 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:40.547865 31825 rpcz_store.cc:276] 0709 12:24:39.224987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.734191 (+509204us) service_pool.cc:224] Handling call
0709 12:24:39.734547 (+   356us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:39.736414 (+  1867us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:39.736449 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:39.737033 (+   584us) log.cc:844] Serialized 52 byte log entry
0709 12:24:39.908723 (+171690us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:39.988738 (+ 80015us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.040540 (+ 51802us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.057884 (+ 17344us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.057978 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:40.518977 (+460999us) raft_consensus.cc:1685] finished
0709 12:24:40.519016 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.519189 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":68949,"prepare.queue_time_us":666592,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":97,"raft.queue_time_us":14650,"raft.run_cpu_time_us":402,"raft.run_wall_time_us":31358,"thread_start_us":285975,"threads_started":3,"wal-append.queue_time_us":295896}
I20260709 12:24:40.631525  8446 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:40.627951  8447 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:40.633885  8447 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:40.635064  8327 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:40.636797  8447 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:40.541901  8371 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.551875  8449 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:40.549412  8457 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.547271  8484 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.599795  8425 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:40.590266  8534 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:40.608001 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:40.612285 31674 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:40.640633  8483 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.548071 31821 rpcz_store.cc:276] 0709 12:24:39.511429 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.689950 (+178521us) service_pool.cc:224] Handling call
0709 12:24:40.520091 (+830141us) spinlock_profiling.cc:243] Waited 74.1 ms on lock 0x7b54000fbc54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:40.520191 (+   100us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":170380672}
I20260709 12:24:40.608027  8130 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:40.599870  8054 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:40.612782  8486 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:40.629455 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:40.668306 31820 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:40.631517  8499 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:40.677415 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:40.678792 31676 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:40.683079 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3581) took 1172 ms. Trace:
W20260709 12:24:40.660110  8508 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.923s	user 0.006s	sys 0.000s
W20260709 12:24:40.692548 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 63164.9us
I20260709 12:24:40.694082 31717 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:40.703850 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 95915.6us
I20260709 12:24:40.704737 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:40.659982  8506 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:40.668095  8273 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.714268 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3761) took 1488 ms. Trace:
W20260709 12:24:40.705221  8508 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.968s	user 0.007s	sys 0.000s
I20260709 12:24:40.681118  8167 raft_consensus.cc:2764] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:24:40.711098  8449 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:40.715565  7889 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.706837  8445 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:40.724145 31717 leader_election.cc:400] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29.
I20260709 12:24:40.743124 31820 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:24:40.744082 31717 leader_election.cc:403] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:24:40.669138  8483 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.755647 31676 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:40.757107 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:40.776206  7295 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:40.777196  7295 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.777773  7295 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:40.719344 31819 rpcz_store.cc:276] 0709 12:24:39.225993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.251712 (+1025719us) service_pool.cc:224] Handling call
0709 12:24:40.280375 (+ 28663us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.358374 (+ 77999us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:40.358684 (+   310us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.510536 (+151852us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.515347 (+  4811us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.515426 (+    79us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:40.515502 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.631130 (+115628us) spinlock_profiling.cc:243] Waited 4.37 ms on lock 0x7b180043d560. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:40.713998 (+ 82868us) spinlock_profiling.cc:243] Waited 70.1 ms on lock 0x7b54000fcb54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:40.714055 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.714202 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2385,"prepare.queue_time_us":87817,"prepare.run_cpu_time_us":79,"prepare.run_wall_time_us":5734,"raft.queue_time_us":5766,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":6175,"spinlock_wait_cycles":173869184,"thread_start_us":149699,"threads_started":1,"wal-append.queue_time_us":168891}
I20260709 12:24:40.782792 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3763) took 1555 ms. Trace:
I20260709 12:24:40.730865  8012 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 21. Reason: could not achieve majority
I20260709 12:24:40.733078  8484 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:40.785444 31824 rpcz_store.cc:276] 0709 12:24:39.227100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.614700 (+1387600us) service_pool.cc:224] Handling call
0709 12:24:40.615124 (+   424us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:40.615563 (+   439us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.615596 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:40.616120 (+   524us) log.cc:844] Serialized 26 byte log entry
0709 12:24:40.682282 (+ 66162us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:40.682308 (+    26us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.682358 (+    50us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:40.782493 (+100135us) raft_consensus.cc:1685] finished
0709 12:24:40.782521 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.782714 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":123,"raft.run_cpu_time_us":117,"raft.run_wall_time_us":119,"thread_start_us":65786,"threads_started":1,"wal-append.queue_time_us":160623}
I20260709 12:24:40.786854 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3587) took 1274 ms. Trace:
I20260709 12:24:40.791587 31819 rpcz_store.cc:276] 0709 12:24:39.512110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.785622 (+1273512us) service_pool.cc:224] Handling call
0709 12:24:40.786114 (+   492us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.786377 (+   263us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.786417 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.786477 (+    60us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:40.786525 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.786601 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.786787 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.790381 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:40.720351  8201 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.064s
I20260709 12:24:40.795326 31824 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 16. (index mismatch)
I20260709 12:24:40.706125  7756 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.803382 31819 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:40.803833 31819 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:40.733314  8273 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:40.821686 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:40.719980  8519 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:40.806035 31823 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3755) took 1580 ms (client timeout 1780 ms). Trace:
I20260709 12:24:40.822675 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3588) took 1291 ms. Trace:
I20260709 12:24:40.823155 31824 rpcz_store.cc:276] 0709 12:24:39.512322 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.791626 (+1279304us) service_pool.cc:224] Handling call
0709 12:24:40.794080 (+  2454us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.802929 (+  8849us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.803444 (+   515us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:40.823050 31823 rpcz_store.cc:269] 0709 12:24:39.224059 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.330394 (+106335us) service_pool.cc:224] Handling call
0709 12:24:40.804430 (+1474036us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54001e9bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:40.804621 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2517206272}
I20260709 12:24:40.777953  8545 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29.
I20260709 12:24:40.792124  8449 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:40.831246 31824 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 20 index: 11. Preceding OpId from leader: term: 22 index: 12. (index mismatch)
I20260709 12:24:40.832427 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3590) took 1315 ms. Trace:
I20260709 12:24:40.832535  7295 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:40.832808 31824 rpcz_store.cc:276] 0709 12:24:39.517308 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.824101 (+1306793us) service_pool.cc:224] Handling call
0709 12:24:40.830269 (+  6168us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.832095 (+  1826us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.832351 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.746037  7889 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:40.774899  8472 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:40.804090  8371 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:40.831182 31819 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:40.835541 31824 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:40.834106  8387 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.838500  8371 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.838605  8543 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.851910  8543 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:40.852615  7889 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:40.853579 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:40.854372 31753 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:40.854696 31752 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:24:40.854856  8012 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.841504  8273 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:40.855407 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:40.844089  8371 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:40.856266 31749 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:24:40.856401 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:40.844383  8457 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:40.845316 31744 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:40.846688 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:40.858888 31744 raft_consensus.cc:606] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Received request to transfer leadership to TS b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:40.858935 31751 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:40.846844 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:40.846004 31819 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:40.859864 31746 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:24:40.859369 31744 raft_consensus.cc:614] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:40.853609 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1584) took 1178 ms. Trace:
I20260709 12:24:40.839457 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1583) took 1302 ms. Trace:
I20260709 12:24:40.854249  8543 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:40.860945 31827 rpcz_store.cc:276] 0709 12:24:39.675349 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.852656 (+1177307us) service_pool.cc:224] Handling call
0709 12:24:40.853022 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.853215 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.853243 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.853298 (+    55us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:40.853351 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.853411 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.853525 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.861244 31823 rpcz_store.cc:276] 0709 12:24:39.537735 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.826365 (+1288630us) service_pool.cc:224] Handling call
0709 12:24:40.839092 (+ 12727us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.839237 (+   145us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.839253 (+    16us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.839288 (+    35us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:40.839307 (+    19us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.839339 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.839410 (+    71us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.857849 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3591) took 1217 ms. Trace:
I20260709 12:24:40.886323  7295 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:40.896907 31824 rpcz_store.cc:276] 0709 12:24:39.621750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.833581 (+1211831us) service_pool.cc:224] Handling call
0709 12:24:40.834796 (+  1215us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.838386 (+  3590us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.838610 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:40.845155  8511 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.125s	user 0.004s	sys 0.008s
I20260709 12:24:40.901378 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:40.902690 31752 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:40.903918 31746 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 31: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:40.862725  8545 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.905422 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 18 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:40.907832 31748 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 18: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:40.842501  8472 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:40.871245  8445 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:40.855497  8012 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:40.916442 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 211767us
W20260709 12:24:40.916996 31818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3746) took 1699 ms (client timeout 2053 ms). Trace:
I20260709 12:24:40.908557  8545 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.917111  8514 raft_consensus.cc:2764] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:24:40.926530 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:40.924520 31818 rpcz_store.cc:269] 0709 12:24:39.217283 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.217429 (+   146us) service_pool.cc:224] Handling call
0709 12:24:40.916686 (+1699257us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b540043d5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:40.916906 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2467257216}
I20260709 12:24:40.919265 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:40.920953 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3764) took 1155 ms. Trace:
I20260709 12:24:40.931090 31823 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:40.923421 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 62928.7us
I20260709 12:24:40.939859 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1585) took 1239 ms. Trace:
I20260709 12:24:40.841914  8387 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.942551 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:40.938549 31820 rpcz_store.cc:276] 0709 12:24:39.765203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.886539 (+121336us) service_pool.cc:224] Handling call
0709 12:24:40.920690 (+1034151us) spinlock_profiling.cc:243] Waited 763 ms on lock 0x7b54001671d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:40.920862 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1755341696}
I20260709 12:24:40.944334  8508 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.946955 31744 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:24:40.947067  8508 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.683801 31825 rpcz_store.cc:276] 0709 12:24:39.510510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.632148 (+1121638us) service_pool.cc:224] Handling call
0709 12:24:40.632493 (+   345us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.632701 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:24:40.633033 (+   332us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.668717 (+ 35684us) log.cc:844] Serialized 14 byte log entry
0709 12:24:40.682587 (+ 13870us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.682668 (+    81us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:40.682725 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.682818 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.682996 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":795,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"raft.queue_time_us":223,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":12099,"spinlock_wait_cycles":49220352,"thread_start_us":35048,"threads_started":1}
I20260709 12:24:40.949038 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:40.940269 31823 rpcz_store.cc:276] 0709 12:24:39.700834 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.906115 (+1205281us) service_pool.cc:224] Handling call
0709 12:24:40.922563 (+ 16448us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.939453 (+ 16890us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.939787 (+   334us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:40.952819 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:40.953969 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:40.954600 31820 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:40.851104  8519 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:40.960047 31818 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:40.944653  8472 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.968338 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:40.970362 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1275) took 1675 ms. Trace:
I20260709 12:24:40.951947  8473 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:40.957630  8130 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:40.974957 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:40.970880  8472 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:40.978273 32370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 1446) took 1771 ms (client timeout 1578 ms). Trace:
I20260709 12:24:40.978178 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:40.979606  8012 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:40.974043  8378 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.007s
I20260709 12:24:40.981698  8545 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:40.970937 32376 rpcz_store.cc:276] 0709 12:24:39.294670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.294846 (+   176us) service_pool.cc:224] Handling call
0709 12:24:39.316364 (+ 21518us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:39.316775 (+   411us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:39.316806 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:39.331422 (+ 14616us) log.cc:844] Serialized 26 byte log entry
0709 12:24:39.566100 (+234678us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:39.566141 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:39.566222 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:40.844533 (+1278311us) spinlock_profiling.cc:243] Waited 89.8 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:40.967566 (+123033us) raft_consensus.cc:1685] finished
0709 12:24:40.967587 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:40.970286 (+  2699us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":113,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":10275,"spinlock_wait_cycles":206430592,"thread_start_us":228891,"threads_started":1,"wal-append.queue_time_us":383082}
W20260709 12:24:40.984360 32370 rpcz_store.cc:269] 0709 12:24:39.205009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.205154 (+   145us) service_pool.cc:224] Handling call
0709 12:24:40.975958 (+1770804us) spinlock_profiling.cc:243] Waited 651 ms on lock 0x7b54003f3254. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:40.976075 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1497230208}
I20260709 12:24:40.972847  8534 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:40.989813 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:40.991029  8378 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:40.991431  8378 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:40.992019  8568 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.091s	user 0.001s	sys 0.000s
I20260709 12:24:40.992409  8378 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:40.993268  8509 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:40.993705  8509 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:40.991431 31746 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: [28.21-28.21]
W20260709 12:24:40.983376  8491 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.264s	user 0.016s	sys 0.028s
I20260709 12:24:40.992815  8309 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:40.999418 31746 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: [28.20-28.20]
I20260709 12:24:40.999316 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:40.982420  8130 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 30 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:41.003364 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:41.004558 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:40.998117  8491 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.279s	user 0.016s	sys 0.028s
I20260709 12:24:40.998163  8509 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:41.023805 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:41.025090 31820 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:41.025754 31746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:41.026361 31746 raft_consensus.cc:606] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:41.026749 31746 raft_consensus.cc:614] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:41.011711 31823 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:40.993716  8378 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:41.011016  8560 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:41.008330  8343 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.005s
W20260709 12:24:41.014947  8572 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.073s	user 0.000s	sys 0.001s
I20260709 12:24:41.015218 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3762) took 1788 ms. Trace:
I20260709 12:24:41.016348  8552 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:41.007889  8446 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:41.017270  8483 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:40.997174  8506 raft_consensus.cc:2764] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:24:41.142189 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.143376  8387 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:41.144292 31820 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 25 index: 15. (index mismatch)
I20260709 12:24:41.165702 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1586) took 1418 ms. Trace:
I20260709 12:24:41.166105 31820 rpcz_store.cc:276] 0709 12:24:39.746992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.026116 (+1279124us) service_pool.cc:224] Handling call
0709 12:24:41.143795 (+117679us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.165410 (+ 21615us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.165642 (+   232us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:41.022419  8130 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:41.145771 31819 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 16. (term mismatch)
I20260709 12:24:41.167742 31819 pending_rounds.cc:85] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 15
I20260709 12:24:41.167984 31819 pending_rounds.cc:107] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 24.16
I20260709 12:24:41.168195 31819 raft_consensus.cc:2889] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:41.168355 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:41.147514  8465 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.149122  8371 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.174891  8465 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.176453  8371 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:41.176160  8509 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.042914  8508 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:41.177110  8465 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:41.149547 31670 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:24:41.178330  8509 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:41.152128  8574 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.176s	user 0.000s	sys 0.009s
I20260709 12:24:41.183077 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3589) took 1670 ms. Trace:
W20260709 12:24:41.183012  7805 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 25 current term: 25 is higher.
I20260709 12:24:41.043419 31673 raft_consensus.cc:2368] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30.
I20260709 12:24:41.185029 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:41.024989 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3582) took 1514 ms. Trace:
I20260709 12:24:41.186918 31821 rpcz_store.cc:276] 0709 12:24:39.510848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.721081 (+1210233us) service_pool.cc:224] Handling call
0709 12:24:40.728329 (+  7248us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:40.917299 (+188970us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.917333 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:40.917919 (+   586us) log.cc:844] Serialized 26 byte log entry
0709 12:24:40.967645 (+ 49726us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:40.967683 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:40.967756 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:41.024797 (+ 57041us) raft_consensus.cc:1685] finished
0709 12:24:41.024824 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:41.024936 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":206878,"raft.queue_time_us":189204,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":5402,"thread_start_us":21671,"threads_started":1,"wal-append.queue_time_us":47908}
I20260709 12:24:41.189779 31821 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:41.191843 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.193017 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.153479  8472 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:41.194017 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.001302  8309 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:41.141911 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:41.059525 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 55026.4us
I20260709 12:24:41.154263 31674 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:41.197613 31675 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:41.197826 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:41.199097 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:41.157466 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 21 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:41.158746 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3592) took 1456 ms. Trace:
I20260709 12:24:41.160362 31822 rpcz_store.cc:276] 0709 12:24:39.226710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.593825 (+1367115us) service_pool.cc:224] Handling call
0709 12:24:40.594184 (+   359us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:40.669290 (+ 75106us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:40.669324 (+    34us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:40.669372 (+    48us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:40.669400 (+    28us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.014949 (+345549us) spinlock_profiling.cc:243] Waited 74.9 ms on lock 0x7b54003c9bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:41.015006 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:41.015154 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":172247424}
I20260709 12:24:41.202451 31822 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:41.203092 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3599) took 1304 ms. Trace:
I20260709 12:24:41.203393 31822 rpcz_store.cc:276] 0709 12:24:39.898864 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.201666 (+1302802us) service_pool.cc:224] Handling call
0709 12:24:41.202003 (+   337us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.202865 (+   862us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.203040 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:41.204748 31822 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:41.146560 31672 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:24:41.170296 31820 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:24:41.207827 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1587) took 1460 ms. Trace:
I20260709 12:24:41.207312  8528 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 20.14, Last appended by leader: 14, Current term: 20, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.208164 31820 rpcz_store.cc:276] 0709 12:24:39.747309 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.169663 (+1422354us) service_pool.cc:224] Handling call
0709 12:24:41.169972 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.207591 (+ 37619us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.207775 (+   184us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:41.171139  8522 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.090s	user 0.001s	sys 0.002s
I20260709 12:24:41.171998 31746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:41.209599 31746 raft_consensus.cc:606] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:41.210024 31746 raft_consensus.cc:614] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:41.158149  8560 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:41.211019  8560 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:41.214672 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2836) took 1797 ms. Trace:
I20260709 12:24:41.215243 31669 rpcz_store.cc:276] 0709 12:24:39.417003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.522265 (+105262us) service_pool.cc:224] Handling call
0709 12:24:41.214391 (+1692126us) spinlock_profiling.cc:243] Waited 536 ms on lock 0x7b54003917d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:41.214569 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1233939840}
I20260709 12:24:41.179186 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:41.218945  8510 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:40.994612  8534 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:41.221364  8510 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:41.221801  8425 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:41.183499 31819 rpcz_store.cc:276] 0709 12:24:39.512538 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.793004 (+1280466us) service_pool.cc:224] Handling call
0709 12:24:40.793351 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.182799 (+389448us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.183023 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:41.222371  8425 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:41.223857 31676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1420) took 1421 ms (client timeout 1728 ms). Trace:
I20260709 12:24:41.223994 31671 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:41.228492  7889 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:24:41.228960 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.229684  8457 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:41.228965  7889 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:41.232391  7889 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.233880  7889 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:41.236049 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:41.237002 31751 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:41.237898 32371 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:41.238164 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:41.239460 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:41.240368 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:41.240684 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:41.241351 31752 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 28: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:24:41.210140 31826 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3753) took 1986 ms (client timeout 1702 ms). Trace:
I20260709 12:24:41.242300 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:41.242619 31826 rpcz_store.cc:269] 0709 12:24:39.223449 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.238074 (+ 14625us) service_pool.cc:224] Handling call
0709 12:24:41.209866 (+1971792us) spinlock_profiling.cc:243] Waited 606 ms on lock 0x7b54004466d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:41.210047 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1393502592}
I20260709 12:24:41.245172 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:41.247171 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:41.247949 31746 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:41.251361  8464 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:41.251775  8464 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:41.252600  8464 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:41.252954 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.253666  8464 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:41.200796 31823 rpcz_store.cc:276] 0709 12:24:39.702611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:40.997687 (+1295076us) service_pool.cc:224] Handling call
0709 12:24:41.011351 (+ 13664us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.155203 (+143852us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.158655 (+  3452us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:41.257860 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.257905 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:41.258939 31823 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:41.259641 31750 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:41.260367 31750 raft_consensus.cc:606] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:41.262091 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3745) took 1688 ms. Trace:
I20260709 12:24:41.262513 31671 rpcz_store.cc:276] 0709 12:24:39.573519 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.849647 (+276128us) service_pool.cc:224] Handling call
0709 12:24:41.223821 (+1374174us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.224438 (+   617us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.261883 (+ 37445us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54002e7a50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:41.262021 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3099022208}
I20260709 12:24:41.263180 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.263944 31819 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
W20260709 12:24:41.224277 31676 rpcz_store.cc:269] 0709 12:24:39.802780 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:39.825786 (+ 23006us) service_pool.cc:224] Handling call
0709 12:24:41.223672 (+1397886us) spinlock_profiling.cc:243] Waited 840 ms on lock 0x7b54002e7a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:41.223798 (+   126us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1931918080}
I20260709 12:24:41.266809  8508 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.267390  8508 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.269874  8508 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:41.271873 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.224624  8425 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:41.222150  8534 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:41.274210  8541 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:41.275868 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:41.200892 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.258060 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.278792 31825 raft_consensus.cc:2410] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 28 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 14.
I20260709 12:24:41.282248  8530 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:41.284175 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8725620974f7465eb0371f48761a1625"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:41.285066 31752 raft_consensus.cc:606] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:41.285535 31752 raft_consensus.cc:614] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:41.190519 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3766) took 1424 ms. Trace:
I20260709 12:24:41.290947 31821 rpcz_store.cc:276] 0709 12:24:39.765840 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.189135 (+1423295us) service_pool.cc:224] Handling call
0709 12:24:41.189554 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.190206 (+   652us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.190445 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:41.312124 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:41.313138 31827 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:24:41.318336 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.319044 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:41.320395 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:41.320935  8378 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:41.321275 31825 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:41.321388  8378 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.321820  8378 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:41.322992 31827 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: [27.17-27.17]
I20260709 12:24:41.323201 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:41.324193 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.325003 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.325497 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.326303 31676 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:41.326385 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:41.327241 31822 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:41.327924 31676 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:41.328254 31674 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:41.328378 31676 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:41.328612 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.329274  7733 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.329722 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.330003 31674 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 25 index: 15. (term mismatch)
I20260709 12:24:41.329807  7733 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.330413 31674 pending_rounds.cc:85] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 14
I20260709 12:24:41.331550 31674 pending_rounds.cc:107] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 24.15
W20260709 12:24:41.331629 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.331831 31674 raft_consensus.cc:2889] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:41.332209  7733 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:41.333256  8449 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:24:41.333724 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.329306 31676 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.334911 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.336066 31676 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:41.324832 31827 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:41.325829 31825 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:41.324785  8560 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:41.331030 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:41.196993  8309 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:41.334961 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.367050 31823 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:41.367216 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:41.368602 31669 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:24:41.208916  8528 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:41.371038 31825 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:41.372901 31825 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 15. (term mismatch)
I20260709 12:24:41.373342 31825 pending_rounds.cc:85] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 14
I20260709 12:24:41.373615 31825 pending_rounds.cc:107] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 26.15
I20260709 12:24:41.290016 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.370394 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.373909 31825 raft_consensus.cc:2889] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:41.370275 31827 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:41.375203 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:41.376009 31749 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: [20.14-20.14]
I20260709 12:24:41.376320 31822 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:41.377202 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:41.376142 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.378844 31824 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 23: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:24:41.290010 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 277070us
I20260709 12:24:41.384792 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.385277 31823 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:41.386006  8560 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:41.370426  8449 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.016s
I20260709 12:24:41.386911 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:41.387554 31826 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:41.389073 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.389698 31824 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:41.390759 31824 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:41.367230  8387 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.024s
I20260709 12:24:41.366298  8534 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election won for term 21
I20260709 12:24:41.391662  8534 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:41.394279 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.394729  8273 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:41.395124 31822 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:41.396239 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.404675 31823 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:41.397914  8541 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:24:41.370491 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:24:41.405545 31749 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:41.406210 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:41.397953 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:41.398904 31672 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:41.394176  8534 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 19.13, Last appended by leader: 13, Current term: 21, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.403685 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:41.397538  8270 raft_consensus.cc:993] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:24:41.413533 31751 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:24:41.436685 31748 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:41.386695  8560 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:41.384078 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:41.427573 31676 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:41.428169 31826 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:41.447290 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:41.443717 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:41.448089 31749 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:24:41.448196 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 84051.5us
I20260709 12:24:41.449669 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:41.451781 31670 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 28: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:24:41.456761 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 72836us
W20260709 12:24:41.458918 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 52791.1us
I20260709 12:24:41.460459 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:41.467475 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:41.455479 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57625.2us
I20260709 12:24:41.469456 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:41.471097 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:41.471978 31748 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 28 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:24:41.473181 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:41.473737 31752 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:41.444451 31746 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:24:41.407567 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:41.419003  8309 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:41.491374 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:41.491838 31746 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:41.492237 31676 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:41.500697 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.501776 31675 raft_consensus.cc:2410] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 28 because replica has last-logged OpId of term: 27 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:24:41.502053 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:41.500829 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.503048 31674 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:41.506848 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:41.451601  8547 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:41.516610 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:41.516883 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 67289.7us
I20260709 12:24:41.517510 31669 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 28: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:41.509651 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:41.516885 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:41.520985  8399 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:41.521561 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:41.522375 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:41.522871  8012 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:41.524569  8012 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:41.526083  8012 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:41.480669  8491 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:41.526868  8491 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:41.527309  8491 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:41.446921  8499 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.069s
I20260709 12:24:41.528709  8499 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:41.529158  8499 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:41.481576  8516 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:41.455730  8420 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.075s
I20260709 12:24:41.455932  8449 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.076s
I20260709 12:24:41.532008 31717 leader_election.cc:400] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30.
I20260709 12:24:41.532388 31717 leader_election.cc:403] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Cancelling election due to peer responding with higher term
I20260709 12:24:41.533385  8579 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:24:41.533852  8541 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election lost for term 29. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30.
I20260709 12:24:41.534792 31670 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:41.534957  8579 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:41.535797  8579 consensus_queue.cc:1243] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 25.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:24:41.537060 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:41.538024 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:41.534502  8399 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:41.539615  8399 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.533380 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:41.544248  8387 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:41.544996  8387 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:41.516685  7889 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:41.547093  8389 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:41.547266  7889 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:41.547864  8389 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:41.548355  8389 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:41.549576  7889 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:41.550223  8517 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.055s
I20260709 12:24:41.551981 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:41.546543  8387 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:41.552892  8560 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:41.553058  8387 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.553305  8560 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:41.553488  8387 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:41.554576  8270 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:41.555151  8270 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:41.556048  8270 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.490151  8201 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:41.479918  8518 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:41.489907  8543 raft_consensus.cc:1081] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Signalling peer 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:24:41.519538  8519 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.482023  8000 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:41.550485  8389 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:41.560400  8000 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:41.560814  8000 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:41.561887 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:41.563658 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:41.564863 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 71265us
W20260709 12:24:41.565353 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.563557  8449 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:41.566751  8449 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:41.567449 31823 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:24:41.567574  8449 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.569808 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.566040 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:41.509339  8464 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.574894 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:41.576890 31749 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:24:41.577929 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:41.584117 31746 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:41.572057  8449 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:41.572314  8543 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:41.592110 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:41.595109 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:41.595942 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 396901us
I20260709 12:24:41.587890 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:41.571627  8541 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:41.571710  8389 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:41.598935 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:41.602521 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:41.603363 32376 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:41.599915 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.599956 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:41.559291  8519 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.532008  8420 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.055s
I20260709 12:24:41.554234  8560 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:41.608821 31744 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:41.608063 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:41.530140  8499 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.586432 31824 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:41204
I20260709 12:24:41.628551 31824 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:24:41.629014 31824 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:41.630460 31671 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
W20260709 12:24:41.625195 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 292408us
I20260709 12:24:41.635023  8184 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:24:41.636981  8184 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 26.20, Next index: 21, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:41.638003  8184 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:24:41.638909 31752 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:24:41.591825  8516 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:41.586187  8371 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.017s
W20260709 12:24:41.641486 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.591941  8518 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:41.629105 31642 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:41.599864  8012 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.639981 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:41.583189  8464 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:41.668715 32372 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 14. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:41.669904 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:41.670871 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:41.672611 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:41.623556  8530 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:41.676873 32377 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:41.679749 31671 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:41.671878 32369 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
W20260709 12:24:41.662434  8578 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:24:41.629941  8543 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:41.682835 32377 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:41.604301  8545 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.061s
I20260709 12:24:41.690037 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:41.701220 32369 raft_consensus.cc:1217] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Deduplicated request from leader. Original: 22.15->[24.16-24.16]   Dedup: 24.16->[]
I20260709 12:24:41.701848 32369 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: [24.16-24.16]
I20260709 12:24:41.701763 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:41.702528 31675 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:41.667865  8499 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:41.694591  8543 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:41.664265  8343 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.074s
I20260709 12:24:41.676770  8012 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:41.815059  8012 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:41.678117  8516 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:41.778990  8590 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.738031  8184 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:41.783735  8519 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:41.677264  8483 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:41.784881 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 94921.4us
I20260709 12:24:41.785583  8464 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:41.699352  8445 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:41.788889 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:41.667973  8130 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:41.790031  8584 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.094s	user 0.000s	sys 0.010s
W20260709 12:24:41.792569 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.839661 31675 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:24:41.682250  8560 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:41.793372  8554 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:24:41.690694  8547 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:41.789517 32342 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:41.797983  8387 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.806972  8587 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:41.692098  8552 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:41.668606  8582 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.113s
W20260709 12:24:41.667397  8446 consensus_queue.cc:1175] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "039d66a6c85e43c3936bf5017501b344" responder_term: 25 status { last_received { term: 23 index: 16 } last_committed_idx: 16 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:24:41.776727  8518 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.917378 31748 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:24:41.830425 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:41.834167  7889 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:41.826412  8580 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:24:41.843194 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:41.818169  8543 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:41.846196  8130 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:41.847335 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:41.828240  8184 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:41.817610  8517 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:41.896162 31819 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21. Ops: [20.14-20.14]
W20260709 12:24:41.896621 31677 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.699s	user 0.006s	sys 0.018s
I20260709 12:24:41.859133 31824 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:41.835604  8445 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:41.910892 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3594) took 2163 ms. Trace:
W20260709 12:24:41.913319 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 325518us
W20260709 12:24:41.978892 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 148554us
I20260709 12:24:41.897699  8552 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:41.894610  8587 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:41.897891  8547 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:41.993064 31677 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.796s	user 0.006s	sys 0.018s
I20260709 12:24:41.836279  8590 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:41.995270 31818 rpcz_store.cc:276] 0709 12:24:39.747670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.188781 (+1441111us) service_pool.cc:224] Handling call
0709 12:24:41.189129 (+   348us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:41.189366 (+   237us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:24:41.581849 (+392483us) log.cc:844] Serialized 14 byte log entry
0709 12:24:41.910013 (+328164us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:41.910173 (+   160us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:24:41.910315 (+   142us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:41.910522 (+   207us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:41.910792 (+   270us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4981,"prepare.queue_time_us":15468,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"raft.queue_time_us":9993,"raft.run_cpu_time_us":176,"raft.run_wall_time_us":62543,"thread_start_us":707624,"threads_started":2}
I20260709 12:24:41.996405 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:41.989374  8420 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:41.985217  8155 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:24:42.015542 31824 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:42.019011 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.023808 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:41.994577  8593 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:42.027504 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.002616  8343 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:42.000913  8517 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:41.984460  7889 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.034060 32373 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:24:42.033354  8541 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:42.015950  8273 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:42.040831 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.042316 32372 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:42.027365  8545 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:42.001407  8399 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260709 12:24:42.025924  8562 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:42.025583  8155 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:42.038208  8541 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:42.044534  8389 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:42.053725  8523 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:24:42.070837 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 504831us
I20260709 12:24:42.020364  8420 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.065828  8541 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:42.036659  8593 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.141613  8593 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:42.078281  8491 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.550s	user 0.013s	sys 0.002s
I20260709 12:24:42.142735  8491 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:42.145761  8491 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:42.058236  7756 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.026s
I20260709 12:24:42.048920  8552 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.086668  8155 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:42.060521  8445 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:42.099351  8387 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:42.099547  8130 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:42.100765  8378 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.778s	user 0.011s	sys 0.002s
I20260709 12:24:42.077620  8516 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:42.104121  8420 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:42.051498  8184 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:42.107596  8000 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.546s	user 0.012s	sys 0.000s
I20260709 12:24:42.124229  7889 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:42.129403  8541 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:42.135757  8594 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:24:42.081458  8602 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.060s
I20260709 12:24:42.138507  8590 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:42.029078  8587 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:42.154563 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:42.179466 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:42.188602 31825 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.019s	sys 0.042s
I20260709 12:24:42.068316  8273 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.021s
I20260709 12:24:42.105809 32375 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
W20260709 12:24:42.193624 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.019s	sys 0.057s
W20260709 12:24:42.221256 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.020s	sys 0.057s
I20260709 12:24:42.059695  8517 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:42.062408  8562 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:42.218516 31825 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.541s	user 0.019s	sys 0.043s
W20260709 12:24:42.192893 31716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.020s	sys 0.026s
W20260709 12:24:42.229342 31716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.020s	sys 0.026s
I20260709 12:24:42.231621 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:42.222144 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:42.232687 31752 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:42.233095 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 20 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:42.233601 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:42.234546 31746 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:24:42.240549 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.243196 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:42.209867  8552 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:42.075151  8547 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:42.255898 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:42.256753 31747 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 29: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:42.213801  8445 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:42.250057 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.246857 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:42.250308 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:42.192890  8378 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.251641  8516 raft_consensus.cc:2764] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:24:42.251987  8587 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:42.252142 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:42.255138 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.258214 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:42.258416 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:42.268930 31752 raft_consensus.cc:2368] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31.
W20260709 12:24:42.258853 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.648s	user 0.026s	sys 0.031s
I20260709 12:24:42.262543 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:42.245628  8449 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.646s	user 0.010s	sys 0.004s
I20260709 12:24:42.266099 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:42.284999  8562 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:42.267088  8603 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:42.287140 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.288985  8611 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:42.291261  8483 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:42.232565  8509 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:42.270963 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.665s	user 0.026s	sys 0.031s
I20260709 12:24:42.293020  8509 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:42.294869 31820 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:42.271193 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:42.301590 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:42.302368  8378 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:42.302757 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:42.303434 32370 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:24:42.303570 32376 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 31: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:24:42.306116 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.201790  8000 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.317587 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 18 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:42.302383 32369 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
I20260709 12:24:42.321620 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:42.322463 31747 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:42.317499 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:42.322155 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 532683us
I20260709 12:24:42.332401 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:42.320926 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.355410 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:42.363904 31747 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:24:42.337227  8562 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.366524  8562 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:42.369081  8562 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:42.374782 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.366595 31745 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:42.333968  8541 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:42.336764  8555 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:42.364302  8378 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:42.332773  8606 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Leader election won for term 18
W20260709 12:24:42.399174 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.000s	sys 0.017s
W20260709 12:24:42.401968 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.594s	user 0.000s	sys 0.017s
W20260709 12:24:42.400746 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.360707  8516 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
W20260709 12:24:42.403105 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 595357us
W20260709 12:24:42.405887 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.398761  8555 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:42.396847  8543 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:42.364063  8483 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.038s
I20260709 12:24:42.400456  8562 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:42.417665 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:42.433794 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.444553 31753 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:42.429591  8516 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:42.406401  8378 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:42.435232  8560 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.582s	user 0.012s	sys 0.003s
W20260709 12:24:42.434149  8499 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.615s	user 0.011s	sys 0.005s
I20260709 12:24:42.430438  8606 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:42.433605  8555 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:42.452682 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:42.459722 32377 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 25 index: 17. (term mismatch)
I20260709 12:24:42.461521 32377 pending_rounds.cc:85] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 16
I20260709 12:24:42.461961 32377 pending_rounds.cc:107] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 24.17
I20260709 12:24:42.462261 32377 raft_consensus.cc:2889] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:42.471601 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:42.474169 31818 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:42.438887  8582 consensus_queue.cc:1243] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 27.20 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
I20260709 12:24:42.489919  8620 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.490845 31818 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 21 index: 14. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:42.437640  8562 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:42.499842 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 134146us
I20260709 12:24:42.501281 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.493659  8560 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:42.508756 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:42.505134 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:42.510655 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:42.515185 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:42.524979 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:42.526371 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:42.512678  8155 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.513270  8603 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:42.535071 31819 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
W20260709 12:24:42.537035  8609 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
I20260709 12:24:42.530052  8617 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:42.495645  8620 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.529750  8000 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:42.539970  8541 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:24:42.524669  8518 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.034s
W20260709 12:24:42.598754 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62732.3us
I20260709 12:24:42.530525  8630 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:42.540076  8560 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:42.557454  8582 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:42.610133 32375 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32. Ops: []
W20260709 12:24:42.610620  8621 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:24:42.610482 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:42.616911 32372 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
W20260709 12:24:42.616202 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 106432us
I20260709 12:24:42.528220  8378 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.620013 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: true dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:42.611244 32373 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
W20260709 12:24:42.611435  8615 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.176s	user 0.001s	sys 0.000s
I20260709 12:24:42.537439  8611 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.063s
W20260709 12:24:42.612344 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 102515us
I20260709 12:24:42.535488  8425 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.059s
I20260709 12:24:42.534265  8624 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 26.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.080s
I20260709 12:24:42.542833  8155 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.606287  8630 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:42.622749 31677 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:42.556242  8619 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.075s
W20260709 12:24:42.619982 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.615162  8543 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:42.637725 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:42.638032 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:42.639528 31678 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 31: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:42.640048 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3611) took 1074 ms. Trace:
I20260709 12:24:42.640554 31826 rpcz_store.cc:276] 0709 12:24:41.566035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.566164 (+   129us) service_pool.cc:224] Handling call
0709 12:24:41.566517 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:41.592313 (+ 25796us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:41.592352 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:41.605760 (+ 13408us) log.cc:844] Serialized 26 byte log entry
0709 12:24:42.057806 (+452046us) raft_consensus.cc:1645] Marking committed up to 10
0709 12:24:42.057848 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.057922 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:42.639829 (+581907us) raft_consensus.cc:1685] finished
0709 12:24:42.639848 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.639978 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":53309,"raft.queue_time_us":36102,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":16935,"thread_start_us":374906,"threads_started":1,"wal-append.queue_time_us":718235}
I20260709 12:24:42.642524 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:42.643476 31675 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:42.644399  8631 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.645186 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:42.645958 31671 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:42.658092 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:42.660187 31675 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: [25.16-25.16]
I20260709 12:24:42.657857 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:42.665591 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:42.667938 31674 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
W20260709 12:24:42.642551  8622 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.114s	user 0.000s	sys 0.000s
I20260709 12:24:42.674072 31671 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:24:42.674880 31670 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
W20260709 12:24:42.678393  8628 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
I20260709 12:24:42.678995 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:42.679817 31671 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 18.
W20260709 12:24:42.680968  8626 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.063s	user 0.000s	sys 0.000s
W20260709 12:24:42.681219  8627 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:24:42.681015 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:42.682240  8518 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:42.683030  8623 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:24:42.683256  8518 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.685936  8518 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:42.657657  8387 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.687286 31671 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:42.687816  8387 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:42.690129  8387 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:42.690973  8636 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:42.691524  8636 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:42.637681  8630 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:42.694326 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1590) took 1053 ms. Trace:
I20260709 12:24:42.706174 31822 rpcz_store.cc:276] 0709 12:24:41.640943 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.681149 (+ 40206us) service_pool.cc:224] Handling call
0709 12:24:41.681479 (+   330us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:41.694354 (+ 12875us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:41.791437 (+ 97083us) log.cc:844] Serialized 14 byte log entry
0709 12:24:42.219806 (+428369us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:42.270465 (+ 50659us) log.cc:844] Serialized 26 byte log entry
0709 12:24:42.319065 (+ 48600us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:42.319126 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.319228 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:42.694104 (+374876us) raft_consensus.cc:1685] finished
0709 12:24:42.694126 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.694264 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":31431,"prepare.queue_time_us":706,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":52,"raft.queue_time_us":29836,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":92497,"thread_start_us":413075,"threads_started":1,"wal-append.queue_time_us":621634}
I20260709 12:24:42.591746  8606 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 18, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.662871  8631 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:42.694820  8621 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:24:42.620939  8582 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.713932 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3613) took 1144 ms. Trace:
I20260709 12:24:42.723541 31823 rpcz_store.cc:276] 0709 12:24:41.569343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.569483 (+   140us) service_pool.cc:224] Handling call
0709 12:24:41.569788 (+   305us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:42.014944 (+445156us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:42.014998 (+    54us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:42.039576 (+ 24578us) log.cc:844] Serialized 26 byte log entry
0709 12:24:42.441626 (+402050us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:42.441681 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.441766 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:42.713661 (+271895us) raft_consensus.cc:1685] finished
0709 12:24:42.713688 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.713860 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":87342,"raft.queue_time_us":95021,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":13699,"thread_start_us":727223,"threads_started":2,"wal-append.queue_time_us":440821}
W20260709 12:24:42.717908 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 91378us
I20260709 12:24:42.719381 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:42.731014 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:42.731278 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:42.733568 31671 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:42.729384 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:42.734877 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:42.735764 31671 raft_consensus.cc:2410] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 22 because replica has last-logged OpId of term: 20 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:24:42.742507 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:42.743484 31745 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 31: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:42.742185 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:42.714699  8562 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:42.745126 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:42.713280  8636 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:42.748723 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:42.754479 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.754657 31747 raft_consensus.cc:2376] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24. Re-sending same reply.
I20260709 12:24:42.757942 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:42.765821 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3615) took 1194 ms. Trace:
I20260709 12:24:42.743813  8445 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.773236 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.766407 31821 rpcz_store.cc:276] 0709 12:24:41.571361 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.571533 (+   172us) service_pool.cc:224] Handling call
0709 12:24:41.583481 (+ 11948us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:41.828592 (+245111us) spinlock_profiling.cc:243] Waited 4.17 ms on lock 0x7b4800130158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:24:42.017067 (+188475us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:42.017099 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:42.017619 (+   520us) log.cc:844] Serialized 26 byte log entry
0709 12:24:42.357211 (+339592us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:42.376021 (+ 18810us) log.cc:844] Serialized 14 byte log entry
0709 12:24:42.376288 (+   267us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.376391 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:42.753108 (+376717us) raft_consensus.cc:1685] finished
0709 12:24:42.765566 (+ 12458us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:42.765746 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26349,"prepare.queue_time_us":8735,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":47,"raft.queue_time_us":104907,"raft.run_cpu_time_us":2880,"raft.run_wall_time_us":189664,"spinlock_wait_cycles":9586560,"thread_start_us":738272,"threads_started":2,"wal-append.queue_time_us":432489}
I20260709 12:24:42.767493  8613 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:42.778095  8582 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:42.781085  8445 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:42.784171  8445 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:42.788780  8632 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.789508  8632 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.790239 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:42.791838 31678 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 17. (term mismatch)
I20260709 12:24:42.791973  8632 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:42.792205 31678 pending_rounds.cc:85] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 16
I20260709 12:24:42.792847 31678 pending_rounds.cc:107] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 27.17
I20260709 12:24:42.793128 31678 raft_consensus.cc:2889] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:42.794268  8634 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.118s	user 0.000s	sys 0.000s
I20260709 12:24:42.795074 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.796394  8620 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:42.796892 31717 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:42.798593 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:42.812417 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.816671 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.709723  8560 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:42.853089 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 79983.7us
I20260709 12:24:42.854264 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:42.883419 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 66856.1us
I20260709 12:24:42.884346 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:42.886804 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:42.890437 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:42.798908  8420 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:42.821041  8620 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:24:42.855610  8445 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:42.812644  8624 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:24:42.864305  8619 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:24:42.815629  8273 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Leader pre-election won for term 19
I20260709 12:24:42.864705  8636 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:42.849654  8582 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:42.862293  8614 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
W20260709 12:24:42.908124 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 150232us
I20260709 12:24:42.894817  8541 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.893337  8617 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:42.909965  8590 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:42.882661  8387 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.035s
I20260709 12:24:42.815173  8270 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.101s
I20260709 12:24:42.923314  8606 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:42.887326  8642 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.060s
I20260709 12:24:42.914798  8620 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:42.918303  8273 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:42.917105  8636 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:42.937170 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:42.918999  8445 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:42.939347 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 201134us
I20260709 12:24:42.926713  8590 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:42.934602  8648 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.030s
I20260709 12:24:42.967628 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:42.972901 32373 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:42.939082  8273 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:42.938927  8636 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:42.925763  8541 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:42.964617  8590 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:42.922796  8582 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.943775  8155 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:42.936230  8644 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.054s
I20260709 12:24:42.936864  8606 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:42.989503  8618 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.079s
I20260709 12:24:42.989890  8643 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:42.985589  8641 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.246s	user 0.000s	sys 0.001s
W20260709 12:24:43.004334 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 120057us
W20260709 12:24:43.017303 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.052337 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:43.068063 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:43.061568 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:43.069105 32376 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:43.007426  8648 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:43.068459 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:43.067660 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:43.066433 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.070308 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:43.073925 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:43.075537 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:43.076241 32369 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:43.055680  8642 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:24:43.076468 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:43.009295  8643 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:43.069130  8656 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:24:43.080185 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:43.013602  8445 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:43.078328  8653 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:24:43.072088  8657 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:24:43.092137  8648 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:43.092681  8270 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.093321  8602 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:43.114802  8602 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.117095  8602 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:43.120500  8387 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.121069  8387 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.121343  8425 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:43.121937  8425 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.122151  8270 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.123406  8387 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.124421  8270 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:43.095067  8610 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.708s	user 0.002s	sys 0.002s
I20260709 12:24:43.024186  8620 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.109474  8516 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:43.144832 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.105227  8657 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:43.126698 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.149749 31747 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:43.150681 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.152005 31747 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:43.140417 31752 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
W20260709 12:24:43.111876  8555 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.617s	user 0.010s	sys 0.003s
I20260709 12:24:43.153672  8555 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.154472  8643 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.155957  8555 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:43.104087  8658 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:24:43.145624 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.156952 31751 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 21.
I20260709 12:24:43.157034  8509 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:43.158552 31751 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:43.105027 31670 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:24:43.156320 31747 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:24:43.163519 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.165951 32371 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:43.167433  8446 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:43.167946  8446 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.169818  8446 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.170545  8270 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.171151  8270 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.173368  8555 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:43.173619  8270 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.173996  8555 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.166424 32371 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:43.174837  8582 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.175490 32371 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 26.19, Last appended by leader: 19, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.176159  8555 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:43.177126 32371 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:43.177624 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3770) took 1608 ms. Trace:
I20260709 12:24:43.178081 31748 rpcz_store.cc:276] 0709 12:24:41.569286 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.570465 (+  1179us) service_pool.cc:224] Handling call
0709 12:24:41.570786 (+   321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:41.918131 (+347345us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:43.177337 (+1259206us) spinlock_profiling.cc:243] Waited 346 ms on lock 0x7b54001a59d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:43.177564 (+   227us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":796147968}
I20260709 12:24:43.180583  8582 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:43.181193  8582 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.183280 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:43.183749  8582 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:43.184734  8612 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.701s	user 0.000s	sys 0.004s
I20260709 12:24:43.193082  8630 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.195066  8541 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.197763  8184 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:43.200742  8425 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.202060  8606 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:43.211166  8604 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.825s	user 0.003s	sys 0.000s
I20260709 12:24:43.104910 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:43.230957 31751 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
W20260709 12:24:43.233639 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 128810us
I20260709 12:24:43.235227 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:43.238523 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3617) took 1664 ms. Trace:
I20260709 12:24:43.239645 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:43.238971 31827 rpcz_store.cc:276] 0709 12:24:41.574289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.574418 (+   129us) service_pool.cc:224] Handling call
0709 12:24:41.574739 (+   321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:41.602917 (+ 28178us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:41.602953 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:41.603392 (+   439us) log.cc:844] Serialized 26 byte log entry
0709 12:24:42.080190 (+476798us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:42.080244 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.080321 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:43.094335 (+1014014us) spinlock_profiling.cc:243] Waited 12.7 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:43.236699 (+142364us) raft_consensus.cc:1685] finished
0709 12:24:43.236724 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:43.238456 (+  1732us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":33596,"raft.queue_time_us":35275,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":19120,"spinlock_wait_cycles":29173632,"thread_start_us":403773,"threads_started":1,"wal-append.queue_time_us":735226}
I20260709 12:24:43.203464  8657 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.236500  8534 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:43.242413  8606 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:43.242578 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1266) took 1453 ms. Trace:
I20260709 12:24:43.247612 32377 rpcz_store.cc:276] 0709 12:24:41.788851 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.789385 (+   534us) service_pool.cc:224] Handling call
0709 12:24:41.789679 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.489188 (+699509us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:43.242309 (+753121us) spinlock_profiling.cc:243] Waited 670 ms on lock 0x7b540074e854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:43.242512 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1540083072}
I20260709 12:24:43.232370  8425 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.256125  8534 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:43.257120  8657 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:43.262826  8579 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.263368  8579 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.265069  8273 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:43.266995  8616 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.880s	user 0.002s	sys 0.003s
I20260709 12:24:43.272989  8624 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:43.251137  8425 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.279440 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:43.296010 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:43.303229 32377 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
W20260709 12:24:43.284025 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.309930 31791 leader_election.cc:400] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31.
I20260709 12:24:43.310411 31791 leader_election.cc:403] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:24:43.314957 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:43.313055 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:43.319221 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:43.320837 32369 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:43.280900  8579 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.321780 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:43.312795  8603 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:43.311369  8541 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.302659  8624 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:43.331637  8509 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:43.335958  8624 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.337749  8624 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:43.325212  8602 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:43.339267  8602 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:43.339728  8602 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:43.345465 32369 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:24:43.331244 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:43.346848  8658 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:43.348006  8603 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:43.315883  8663 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31.
I20260709 12:24:43.327721  8606 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:24:43.330415  8534 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.024s
I20260709 12:24:43.357285 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3009) took 1125 ms. Trace:
I20260709 12:24:43.357897 31753 rpcz_store.cc:276] 0709 12:24:42.231670 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:42.234223 (+  2553us) service_pool.cc:224] Handling call
0709 12:24:42.251796 (+ 17573us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.445147 (+193351us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:43.356927 (+911780us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b54001a4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:43.357186 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":442718080}
I20260709 12:24:43.349833  8541 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.359133  8630 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:43.360874  8541 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.351697 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:43.352054  8603 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:43.369997  8603 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.371023 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.372725  8425 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:43.373392  8603 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:43.375206  8658 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:24:43.375641  8658 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:43.376529  8658 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.377628  8273 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:43.377874  8658 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:43.378154 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:43.372515  8663 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:43.368825 32371 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:24:43.270419  8636 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.372852 31751 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 21 index: 14. (index mismatch)
I20260709 12:24:43.382041 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1299) took 1821 ms. Trace:
I20260709 12:24:43.381774 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:43.385850 32371 rpcz_store.cc:276] 0709 12:24:41.560904 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.561027 (+   123us) service_pool.cc:224] Handling call
0709 12:24:41.801040 (+240013us) spinlock_profiling.cc:243] Waited 2.1 ms on lock 0x7b5800026d98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0eee 000000000036eb39 00007fe901d241e0 00007fe901ded0c4 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:24:41.801284 (+   244us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:43.165826 (+1364542us) spinlock_profiling.cc:243] Waited 691 ms on lock 0x7b5400545220. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:24:43.381528 (+215702us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:43.381982 (+   454us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1594014208}
I20260709 12:24:43.391429 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1592) took 1730 ms. Trace:
I20260709 12:24:43.391966 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.392052 31820 rpcz_store.cc:276] 0709 12:24:41.660865 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.806018 (+145153us) service_pool.cc:224] Handling call
0709 12:24:41.806339 (+   321us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:42.295345 (+489006us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:43.391088 (+1095743us) spinlock_profiling.cc:243] Waited 488 ms on lock 0x7b54000fb754. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:43.391344 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1122977152}
I20260709 12:24:43.393078 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:43.398025 32372 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31. Ops: [30.18-30.18]
I20260709 12:24:43.398700  8449 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:24:43.401505 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:43.394488  8343 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 27 current term: 27 is higher.
W20260709 12:24:43.403708 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:43.405179 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.406070 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.408736 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.394507  8663 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:43.409276  8425 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:43.409380 31671 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:43.409858  8663 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.410465 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.411087 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:43.411608 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:43.412328 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.412894 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:43.413182 31671 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:43.414249 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.414916 31675 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
W20260709 12:24:43.415699 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.416466 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.417253 31674 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28. Ops: [24.13-27.15]
I20260709 12:24:43.417622  8518 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:43.418146  8518 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.425743 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.420318 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.429828 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.429983 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.430038 31747 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 13. Preceding OpId from leader: term: 29 index: 14. (index mismatch)
I20260709 12:24:43.430241 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.428892 31819 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:24:43.424876 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.419193 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.426859 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.424944 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.418459 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.409952  8425 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.381093  8541 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:24:43.435933 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:43.422494 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.446789 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:43.447659 32369 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:43.448158 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:43.449468 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56447.3us
I20260709 12:24:43.450804 31749 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:24:43.451169 31822 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 13. Preceding OpId from leader: term: 29 index: 14. (index mismatch)
I20260709 12:24:43.384047  8273 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:43.452628 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:43.453195 31671 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:43.455482 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.370644  8590 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.456107 31822 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:24:43.460892 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.462714 31825 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:24:43.465854 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.469328 31792 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:43.471454 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.473047 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:43.473820 31824 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:43.420225  8519 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:43.418519  8630 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.484392 31822 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:43.419951  7733 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.485332 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.509735 31819 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:43.488445 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:43.489223 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55332.7us
I20260709 12:24:43.511106 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:43.489864 31744 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:43.423271  8631 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.420473  8000 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:43.522910 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.491454 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1463) took 1826 ms. Trace:
I20260709 12:24:43.492127 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.538084 31675 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: [25.15-25.15]
I20260709 12:24:43.488117 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:43.420055  8376 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:43.506603 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:43.539361 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51318.2us
I20260709 12:24:43.502681 31747 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:24:43.508529 31678 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 20 index: 14. Preceding OpId from leader: term: 21 index: 14. (term mismatch)
I20260709 12:24:43.540951 31678 pending_rounds.cc:85] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 13
I20260709 12:24:43.541201 31678 pending_rounds.cc:107] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 20.14
I20260709 12:24:43.541433 31678 raft_consensus.cc:2889] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:43.509639 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.542171 32371 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28. Ops: [27.18-27.18]
I20260709 12:24:43.514884 31825 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:24:43.515687 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 20 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.532650 31673 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 19. (index mismatch)
I20260709 12:24:43.531747 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.535552 32378 rpcz_store.cc:276] 0709 12:24:41.664750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:41.665119 (+   369us) service_pool.cc:224] Handling call
0709 12:24:41.665473 (+   354us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:41.665797 (+   324us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:41.665830 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:41.666349 (+   519us) log.cc:844] Serialized 26 byte log entry
0709 12:24:42.196786 (+530437us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:42.212788 (+ 16002us) log.cc:844] Serialized 14 byte log entry
0709 12:24:42.213138 (+   350us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:42.213213 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:43.266353 (+1053140us) spinlock_profiling.cc:243] Waited 10.6 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:43.458731 (+192378us) raft_consensus.cc:1685] finished
0709 12:24:43.491240 (+ 32509us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:43.491393 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11202,"prepare.queue_time_us":29441,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":4031,"raft.queue_time_us":187671,"raft.run_cpu_time_us":415,"raft.run_wall_time_us":64497,"spinlock_wait_cycles":24461696,"thread_start_us":392175,"threads_started":1,"wal-append.queue_time_us":692739}
I20260709 12:24:43.543937 31674 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:43.534340  8517 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.012s
I20260709 12:24:43.544121  8376 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.523300  7733 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.547048  8518 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:43.547801  7733 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.548921  8425 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.537871  8449 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.012s
I20260709 12:24:43.458185  8637 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:43.535384  8633 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:24:43.550943  8637 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.484313  8387 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.054s
I20260709 12:24:43.551208  8633 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:43.551457  8637 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:43.483033  8516 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 25.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.055s
I20260709 12:24:43.500844  8519 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:43.522152 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:43.533224 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:43.532656 31672 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:43.452421  8130 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.454056  8534 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:43.553479 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:43.556623 31671 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:24:43.518080 32369 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:24:43.552983 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:43.533758  8483 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.094s	user 0.006s	sys 0.010s
I20260709 12:24:43.521108  8002 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:43.588330 31671 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:24:43.570941 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.570197 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.579438 31824 raft_consensus.cc:2410] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 26 because replica has last-logged OpId of term: 25 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:24:43.583663 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:43.588693 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 77264.3us
I20260709 12:24:43.503149  8644 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.059s
I20260709 12:24:43.593508 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:43.492633  8273 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.596545 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:43.512205  8642 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.063s
I20260709 12:24:43.546522  8376 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.600126 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:43.595702  7733 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election won for term 22
W20260709 12:24:43.576517  8483 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.139s	user 0.008s	sys 0.010s
I20260709 12:24:43.567772  8519 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:43.551642  8633 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:43.592736 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 137777us
I20260709 12:24:43.575899  8657 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:43.521800  8630 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.588270  8378 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.022s
I20260709 12:24:43.516296  8652 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.092s
I20260709 12:24:43.591748  8387 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.574786  8130 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.612084  8273 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.613955 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:43.614714  8155 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:43.591802  8499 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:43.601560  8446 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:43.592556  8595 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.602171  8593 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:43.667418 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.608282  7733 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.534708  8651 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 24.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.087s
I20260709 12:24:43.678581 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.684165 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:43.688084 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 74268.5us
I20260709 12:24:43.697180  8465 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:43.599325  8631 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.072s
I20260709 12:24:43.596411  8534 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.607246  8541 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:43.640434  8677 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:43.668751  8560 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:43.720449 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 83964.7us
W20260709 12:24:43.698650 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 246913us
I20260709 12:24:43.637346  8657 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.705210 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:43.602516  8201 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.088s
I20260709 12:24:43.643615  8644 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:43.680019  8606 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:43.707414  8593 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:43.641852  8632 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.058s
I20260709 12:24:43.694082 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.740430 31751 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:43.712025 31747 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:43.667339  8611 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.066s
I20260709 12:24:43.741853 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.719286 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:43.742959 31826 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 32 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:24:43.744030 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:43.667629  8483 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:43.683178  8582 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.030s
I20260709 12:24:43.709753  8499 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:43.731709  8657 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:43.686493  8387 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.721076  8465 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:43.742739  8593 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:43.722736  8682 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.712370  7733 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:43.727437  8636 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:43.711567  8595 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.747371 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:43.755633 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:43.705327 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 121847us
I20260709 12:24:43.752419  8517 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:43.756729  8465 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:43.763760 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.751606  8376 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:43.767542  8499 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 24.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:43.782652 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58380.7us
I20260709 12:24:43.785552 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.768169  8640 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:24:43.760023  8590 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.771713  8130 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:43.773747  8595 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:43.776060  8682 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.736958  8560 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.779333 31826 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:24:43.784739  8376 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.831930  8376 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:43.832170 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.833204 31672 raft_consensus.cc:2410] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:24:43.835606  8560 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:43.787592  8517 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:43.841321  8517 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.743569  8602 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:43.765705  8675 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.192s	user 0.000s	sys 0.002s
I20260709 12:24:43.740170  8644 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:43.739805  8606 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:43.807850 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:43.817015 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:43.854327 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:43.809405  8663 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:43.822753 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:43.823066 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.824869 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.866659 31748 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:24:43.828650 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.868054 31826 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:43.769922  8387 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:43.788663  8673 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
I20260709 12:24:43.845165 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:43.788802  8652 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:43.825821 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:43.793431  8155 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:43.864471 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:43.844472  8602 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:43.877753 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:43.881903 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:43.869423  8560 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.825734  8650 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.140s
I20260709 12:24:43.875149  8155 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.825450  8677 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: INVALID_TERM, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.057s
I20260709 12:24:43.819582  8639 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.192s
I20260709 12:24:43.869855  8688 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:43.886469  8603 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.511s	user 0.012s	sys 0.000s
I20260709 12:24:43.815953  8541 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.879791  8630 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:43.902307 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 19 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:43.894800 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.875458  8686 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:43.910235  8002 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:43.895737  8155 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:43.895653 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 69928.2us
I20260709 12:24:43.882004  8516 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.026s
I20260709 12:24:43.892515  8528 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:43.882087  8642 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 15, Time since last communication: 0.024s
I20260709 12:24:43.889428  8201 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:43.888911  8643 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:24:43.925550 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 61185.6us
I20260709 12:24:43.926457 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:43.913494  8483 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:43.924983  8002 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:43.930459  8687 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.146s	user 0.000s	sys 0.001s
I20260709 12:24:43.931047  8693 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:43.934940 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.955660 31818 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
W20260709 12:24:43.938563 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60857.6us
I20260709 12:24:43.944907 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:43.927213  8534 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:43.957600  8678 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.188s	user 0.000s	sys 0.001s
I20260709 12:24:43.959043  8692 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election lost for term 22. Reason: could not achieve majority
I20260709 12:24:43.943583  8595 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:43.963065  8595 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:43.965397  8595 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:43.947846 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:43.892465  8560 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:43.950774 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 218280us
I20260709 12:24:43.967710 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:43.968205 31826 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:24:43.968950 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.968700 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.970856 31744 raft_consensus.cc:2410] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 27 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:24:43.944221  8643 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.025158  8643 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:43.989182 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 44406.1us
I20260709 12:24:44.005910 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:43.965548  8606 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:43.988090  8633 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:44.023300  8698 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:24:44.148603 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:43.970176  8693 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:44.151016 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 119774us
I20260709 12:24:44.152328 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:44.153693 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 227291us
I20260709 12:24:44.154590 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:44.157012 31744 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:43.916429  8688 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:43.991163  8658 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.613s	user 0.010s	sys 0.003s
W20260709 12:24:44.164710 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 197062us
I20260709 12:24:44.165432 31824 raft_consensus.cc:2410] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 31 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
I20260709 12:24:44.165460 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:44.147383  8483 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:43.954504  8519 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.181699  8682 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.182402  8682 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:44.182797  8682 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.182957  8697 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:44.183274  8682 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:44.183561  8697 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.183997  8697 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:44.182789  8704 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:44.186251  8483 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.189672 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:44.190366 32375 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:44.193472 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:44.203259 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:44.205864 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:44.171468  8632 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:44.212973 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:44.210116 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:44.191422  8644 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:44.188516  8483 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:44.219082 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 724040us
I20260709 12:24:44.226851 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:44.164070  8376 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.230618 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:44.232529 31819 raft_consensus.cc:2410] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 31 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 14.
I20260709 12:24:44.164101  8593 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.202060  8704 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:44.232364 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:44.246959 31819 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 15. (term mismatch)
I20260709 12:24:44.247678 31819 pending_rounds.cc:85] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 14
I20260709 12:24:44.247996 31819 pending_rounds.cc:107] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 27.15
I20260709 12:24:44.248299 31819 raft_consensus.cc:2889] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:44.251463 31678 raft_consensus.cc:2368] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24.
I20260709 12:24:44.234546  8273 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:44.236862  8704 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.239580  8517 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.264746 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:44.228076  8632 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.233070  8644 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.023s
W20260709 12:24:44.276398 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 75402.5us
I20260709 12:24:44.251178  8701 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:44.283671 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:44.284366 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 70007.2us
W20260709 12:24:44.285938 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53640.7us
I20260709 12:24:44.286696 31717 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 19 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:44.264955  8704 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:44.292133 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:44.317974 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:44.318998 31672 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:44.324156 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:44.293730  8683 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.273s	user 0.002s	sys 0.014s
I20260709 12:24:44.335819 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:44.288533  8701 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:44.294930  8707 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:44.275924  8517 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:44.335019  7733 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.565s	user 0.015s	sys 0.005s
I20260709 12:24:44.351987 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:44.352684 31819 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:44.353150 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:44.336221  8528 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:44.334488  8685 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:44.343106  8637 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.791s	user 0.013s	sys 0.009s
W20260709 12:24:44.349711  8465 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.556s	user 0.021s	sys 0.004s
I20260709 12:24:44.342514  8703 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Leader election won for term 19
I20260709 12:24:44.342353  8701 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:44.382130 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 95537.2us
I20260709 12:24:44.346904  8707 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.352795  8560 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.388177 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:44.389441 31819 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:44.390719 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:44.391911 31819 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:44.343910  8633 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.397215 32378 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:44.401829 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:44.375962  8713 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:44.349668  8002 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.359084  7733 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.388132  8707 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:44.412468 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:44.419116 31819 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 32 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 17.
I20260709 12:24:44.367100  8639 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.036s
I20260709 12:24:44.420885 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:44.402588  8642 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:44.429471 32375 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
W20260709 12:24:44.434027 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 146760us
I20260709 12:24:44.388290  8528 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.031s
I20260709 12:24:44.376971  8685 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:44.412174  8657 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.651s	user 0.013s	sys 0.000s
I20260709 12:24:44.379000  8637 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.382206  8465 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.409286  8483 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:44.409590  8713 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.405326  8703 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:44.439184 31675 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 21. Current term is 22. Ops: [21.14-21.14]
I20260709 12:24:44.415040  8376 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:44.444422 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 117908us
I20260709 12:24:44.434563  8534 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:44.427929  8688 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:44.436055  8593 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.478490 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:44.494248 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3805) took 1026 ms. Trace:
I20260709 12:24:44.471859  8686 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:44.494735 31823 rpcz_store.cc:276] 0709 12:24:43.467531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.493377 (+ 25846us) service_pool.cc:224] Handling call
0709 12:24:43.493637 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:43.742102 (+248465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:43.742137 (+    35us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:43.742182 (+    45us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:43.742217 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.494036 (+751819us) spinlock_profiling.cc:243] Waited 248 ms on lock 0x7b54003c91d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:44.494079 (+    43us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.494184 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":570695424}
I20260709 12:24:44.484086  8725 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:24:44.497920  8712 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.206s	user 0.001s	sys 0.000s
I20260709 12:24:44.512904 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:44.499264 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:44.501654  8710 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.181s	user 0.000s	sys 0.002s
I20260709 12:24:44.437115  8642 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.479887  8516 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.483769  8713 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:44.498970 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:44.444331  8657 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.498309  8693 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:44.514142 31820 raft_consensus.cc:2410] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 27 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
I20260709 12:24:44.495942 31821 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:44.492517  8534 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:44.515662 31823 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:44.495854  8632 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.476195  8483 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.502764  8593 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.526826 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:44.589247 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62509.3us
I20260709 12:24:44.590026 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:44.499212 31676 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:44.533062  8703 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 16.11, Last appended by leader: 11, Current term: 19, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.534209  8657 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:44.515285  8658 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.543615 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:44.546913 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:44.509008  8686 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:44.548895 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:44.549175 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:44.599951 31752 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:44.600296 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:44.531795  8652 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.655s	user 0.012s	sys 0.000s
I20260709 12:24:44.594259 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:44.597431 31823 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:24:44.597761 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:44.597939 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:44.637146 31749 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:44.638300  8686 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.516490  8499 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:44.528412  8694 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.601728 31819 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:24:44.549074  8693 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.631424 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:44.597998 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:44.554589  8517 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.632751  8658 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.643656  8685 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:44.646765  8643 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.621s	user 0.014s	sys 0.000s
I20260709 12:24:44.598915  8632 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:44.647915  7733 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:44.648429  8155 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.705s	user 0.008s	sys 0.004s
I20260709 12:24:44.597548  8688 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:44.648927  8534 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.707851  8483 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:44.711997  8682 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.528s	user 0.005s	sys 0.007s
I20260709 12:24:44.712508  8682 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.520936  8725 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:44.652522  8714 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.164s	user 0.000s	sys 0.001s
W20260709 12:24:44.653363  8727 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.142s	user 0.000s	sys 0.000s
W20260709 12:24:44.659256 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 59075.4us
I20260709 12:24:44.598716  8593 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.660702 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:44.632961  8652 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:44.530941  8516 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.694733  8643 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.691265  8632 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.695971  8155 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.696730  8688 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:44.650970  8730 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:44.735869 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:44.734818 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1601) took 1118 ms. Trace:
I20260709 12:24:44.538581  8697 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.742487  8643 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:44.742688  8509 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Stepping down as leader of term 18
I20260709 12:24:44.743163  8509 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 18 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:44.634449  8728 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:44.743877  8725 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:44.734810  8465 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:44.733000 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:44.736794 31827 rpcz_store.cc:276] 0709 12:24:43.616560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.958029 (+341469us) service_pool.cc:224] Handling call
0709 12:24:43.958342 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:43.958550 (+   208us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:24:44.473466 (+514916us) log.cc:844] Serialized 14 byte log entry
0709 12:24:44.539654 (+ 66188us) log.cc:844] Serialized 14 byte log entry
0709 12:24:44.734469 (+194815us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:44.734519 (+    50us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:24:44.734562 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:44.734621 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:44.734763 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":377799,"prepare.run_cpu_time_us":125,"prepare.run_wall_time_us":4246,"raft.queue_time_us":27994,"raft.run_cpu_time_us":198,"raft.run_wall_time_us":197,"thread_start_us":578142,"threads_started":3,"wal-append.queue_time_us":181127}
I20260709 12:24:44.740854  8516 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:44.741060  8731 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:24:44.741739 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:44.746330 32371 raft_consensus.cc:2410] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 27 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 16.
I20260709 12:24:44.744565  8155 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.737069 31673 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:44.689126  8693 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.743831  8697 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.756482  8688 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:44.757092  8723 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:44.757905  8642 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:44.678359  8694 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.693850  8704 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.732859  8632 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:44.759228 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:44.744088  8509 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 18.12, Last appended by leader: 12, Current term: 18, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:44.764899 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 104326us
I20260709 12:24:44.763746 31827 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:24:44.766769 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:44.770077 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:44.774698 32373 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 32: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:44.784050  8648 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:44.784736  8648 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:44.788161  8693 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:44.788868  8593 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.789793  8593 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:44.790261  8593 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.792364  8593 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:44.793788  8730 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:44.794773  8742 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:44.794538  8730 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.795428  8742 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.797873  8730 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.800073  8519 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.801599 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:44.802322 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:44.798039  8658 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:44.802466 31820 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:44.807063 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:44.805011 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:44.765407  8682 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.806910 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:44.809448 32403 heartbeater.cc:646] Failed to heartbeat to 127.29.201.254:37495 (8 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.29.201.254:37495 timed out after 15.000s (SENT)
I20260709 12:24:44.807777 32373 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:24:44.812542 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:44.764681  8723 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:44.842917 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:44.814002  8642 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:44.847234 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:44.841773 31827 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:44.841641 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:44.849396 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:44.849713 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:44.850993 31820 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:24:44.852698 32373 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:24:44.785880  8648 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.12, Last appended by leader: 12, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.867148  8700 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:44.867714  8700 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.868111  8700 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:44.867323  8701 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.846643  8688 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:44.846424  8657 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:44.847328  8642 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:44.873427 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:44.873682  8742 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:44.874579  8742 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:44.874858 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:44.875325  8742 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.875849  8552 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:44.875792  8742 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.876391  8552 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.878086  8742 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.880960 31670 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
W20260709 12:24:44.885185  8746 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
I20260709 12:24:44.871397  8648 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:44.893173  8701 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:44.874320  8707 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.878027  8632 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.898144 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:44.863013  8697 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.025s
I20260709 12:24:44.882244  8642 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:44.914018 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:44.915136 32373 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 30 because replica has last-logged OpId of term: 28 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 19.
I20260709 12:24:44.916916  8694 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:44.902076  8707 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:44.906739  8723 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.869469  8713 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:44.939119  8741 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:44.963740  8632 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.937731  8552 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:44.971235  8744 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:44.945848  8741 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.972084  8744 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:44.977243 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:44.942976  8704 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:44.981994 31820 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:44.995074  8744 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:45.011569  8713 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:45.016270  8741 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:45.021708 31824 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3643) took 1473 ms (client timeout 1830 ms). Trace:
W20260709 12:24:45.022231 31824 rpcz_store.cc:269] 0709 12:24:43.548552 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.599865 (+ 51313us) service_pool.cc:224] Handling call
0709 12:24:45.021456 (+1421591us) spinlock_profiling.cc:243] Waited 564 ms on lock 0x7b54003ca0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:45.021636 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1297849600}
W20260709 12:24:45.023478 31821 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3806) took 1555 ms (client timeout 2050 ms). Trace:
I20260709 12:24:45.023605 31825 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
W20260709 12:24:45.023959 31821 rpcz_store.cc:269] 0709 12:24:43.467748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.484299 (+ 16551us) service_pool.cc:224] Handling call
0709 12:24:45.023252 (+1538953us) spinlock_profiling.cc:243] Waited 959 ms on lock 0x7b54001978d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:45.023417 (+   165us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2206585216}
I20260709 12:24:45.024748 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:45.025908 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3639) took 1531 ms. Trace:
I20260709 12:24:45.026337 31825 rpcz_store.cc:276] 0709 12:24:43.494949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.547375 (+ 52426us) service_pool.cc:224] Handling call
0709 12:24:45.023395 (+1476020us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:45.024088 (+   693us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.025717 (+  1629us) spinlock_profiling.cc:243] Waited 1.48 s on lock 0x7b54001978d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:45.025848 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3393732864}
I20260709 12:24:45.027719 31750 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:24:45.029801 31750 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:24:45.029964 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:45.036456 31820 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:45.039057  8630 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.041949  8630 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:45.044539 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:45.046811 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1436) took 1152 ms. Trace:
I20260709 12:24:45.047780 31669 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:45.047389 31673 rpcz_store.cc:276] 0709 12:24:43.894323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.894493 (+   170us) service_pool.cc:224] Handling call
0709 12:24:45.046571 (+1152078us) spinlock_profiling.cc:243] Waited 817 ms on lock 0x7b54001d5954. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:45.046745 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1880113024}
I20260709 12:24:45.044512  8630 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:45.057528 31718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.830s (SENT) [suppressed 7 similar messages]
W20260709 12:24:45.065311 31677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1435) took 1280 ms (client timeout 1505 ms). Trace:
I20260709 12:24:44.998149  8593 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:45.067800 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3640) took 1533 ms. Trace:
I20260709 12:24:45.068405 31818 rpcz_store.cc:276] 0709 12:24:43.534475 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.970927 (+436452us) service_pool.cc:224] Handling call
0709 12:24:44.148778 (+177851us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:44.175767 (+ 26989us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:44.175817 (+    50us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:44.175859 (+    42us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:24:44.175901 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.067479 (+891578us) spinlock_profiling.cc:243] Waited 26.7 ms on lock 0x7b5400447ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:45.067547 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:45.067724 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":61404032}
W20260709 12:24:45.065961 31677 rpcz_store.cc:269] 0709 12:24:43.784787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:43.785196 (+   409us) service_pool.cc:224] Handling call
0709 12:24:45.065030 (+1279834us) spinlock_profiling.cc:243] Waited 380 ms on lock 0x7b54001b2bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:45.065208 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":874653056}
I20260709 12:24:45.078434 31753 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:24:45.080911  8593 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:45.087867 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:45.088855 31750 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:45.089404  8530 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:45.089870  8530 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:45.090720 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:45.090839  8530 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:45.093353 31750 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:45.093251 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:45.094103 32378 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:45.095785 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:45.096665 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.101882 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:45.102536 32376 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 25 index: 17. (index mismatch)
I20260709 12:24:45.111533 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:45.107184 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:45.113687 31750 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:45.118386 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:45.121966 32378 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:45.092288  8530 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:45.134943 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:45.143435 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:45.144281 32372 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:45.145903  8642 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:45.147037 32377 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:24:45.149533  8642 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:45.151292  8657 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:45.151487 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:45.151940 32374 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:45.112262  8700 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:45.155026 31749 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
I20260709 12:24:45.157091  8700 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:45.166669  8650 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.026s
I20260709 12:24:45.168273 31818 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: [29.14-29.14]
I20260709 12:24:45.170171 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:45.170461 31820 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:24:45.170943 31824 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:45.173436 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:45.178877 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:45.181622 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:45.189258 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 22 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:45.192152 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:45.196866 32372 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:24:45.200066 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:45.245755 31752 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:45.200278 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:45.201443 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:45.254135 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 488180us
I20260709 12:24:45.132652 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:45.215620 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:45.216318 31819 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:24:45.217254 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:45.214437 31825 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:24:45.223085 31746 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:45.283154 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:45.283931 31671 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:45.223349 31750 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:45.224310 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:45.199182  8516 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:45.227200  7889 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:24:45.230196 31745 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: [29.14-29.14]
I20260709 12:24:45.231681 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:45.241868 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:45.286667 31820 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:45.267433 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:45.287213 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:45.288028 31671 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:45.289178 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:45.289652  8593 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:45.290041 31820 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:24:45.182446 31822 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
W20260709 12:24:45.273910 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.275017 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:45.291139 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:45.290369  8593 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:45.299327  8593 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:45.299809 31823 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:45.304294 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:45.305876  8552 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:45.320135 31827 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:45.320751 31826 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:45.320955 31751 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:24:45.323910 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 191377us
I20260709 12:24:45.326958 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:45.327032 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:45.323571 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:45.364064 31824 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:45.369731 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:45.250260  8464 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:45.372881 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:45.237602  8705 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.090s
I20260709 12:24:45.242976  8702 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.094s
I20260709 12:24:45.322395  8636 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:45.395802 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:45.393635 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:45.336050  8631 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:45.332897  8632 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:45.396844 32374 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:45.338860  8657 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:45.390491  8201 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.004s	sys 0.049s
I20260709 12:24:45.411320 31824 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:24:45.377477  8464 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:45.424494  8657 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:45.427748 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:45.417088 31823 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
W20260709 12:24:45.386312 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 79053.9us
I20260709 12:24:45.433396  8632 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:45.441458  8721 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.480406  8721 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:45.422995  8201 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.004s	sys 0.049s
I20260709 12:24:45.480417  8632 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:45.499915 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:45.502157 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:45.503299 32371 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
W20260709 12:24:45.386839 32369 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s	user 0.017s	sys 0.028s
W20260709 12:24:45.507839 32369 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.784s	user 0.017s	sys 0.029s
I20260709 12:24:45.468580 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:45.491393  8658 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.026s	sys 0.032s
W20260709 12:24:45.510818  8658 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s	user 0.027s	sys 0.032s
I20260709 12:24:45.510792 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:45.511911 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:45.422981  8631 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:45.396709  8552 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.082s
I20260709 12:24:45.378479  8717 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 27.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.046s
I20260709 12:24:45.405320  8636 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:45.540310 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:45.544226 31670 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.626s	user 0.017s	sys 0.022s
W20260709 12:24:45.545110 31670 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.633s	user 0.018s	sys 0.023s
W20260709 12:24:45.537132 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 479615us
W20260709 12:24:45.524133 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 52907.5us
W20260709 12:24:45.413277  8688 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.506s	user 0.011s	sys 0.009s
I20260709 12:24:45.554895 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:45.559267 31823 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:45.555200 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:45.561892 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:45.532943  8534 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.000s	sys 0.024s
I20260709 12:24:45.490339  8721 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:45.490809  8723 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.000s	sys 0.016s
I20260709 12:24:45.579493 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:45.562781 31745 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:24:45.575245 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:45.581362 32371 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:45.581794 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:45.559638  8688 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.011s	sys 0.010s
I20260709 12:24:45.569070  8273 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:45.574115  8607 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:45.588608  8273 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:45.589146  8607 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:45.590391  8607 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:45.591326  8464 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:45.591544  8273 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:45.606200 31752 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: [18.12-18.12]
I20260709 12:24:45.551337  8756 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:45.620198 31752 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28. Ops: []
W20260709 12:24:45.571718  8534 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.668s	user 0.000s	sys 0.025s
I20260709 12:24:45.625401 32371 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:24:45.580053  8749 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:24:45.632216 31672 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.763s	user 0.009s	sys 0.023s
I20260709 12:24:45.621892  8756 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:45.333397 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
W20260709 12:24:45.648207 31678 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.780s	user 0.038s	sys 0.070s
I20260709 12:24:45.626813  8607 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:45.585702  8723 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.621s	user 0.000s	sys 0.018s
W20260709 12:24:45.648736 31678 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.780s	user 0.038s	sys 0.070s
I20260709 12:24:45.656582  8686 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:45.661764 31748 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:45.661906 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1276) took 1436 ms. Trace:
I20260709 12:24:45.665711 31751 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 28. Current term is 29. Ops: [28.18-28.18]
W20260709 12:24:45.670523 31672 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.806s	user 0.010s	sys 0.024s
I20260709 12:24:45.660975  8593 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:45.671168  8637 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:45.671792  8509 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 18 FOLLOWER]: Advancing to term 19
I20260709 12:24:45.662276 32369 rpcz_store.cc:276] 0709 12:24:44.226010 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.226572 (+   562us) service_pool.cc:224] Handling call
0709 12:24:44.238453 (+ 11881us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:44.661489 (+423036us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:44.661536 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:44.662149 (+   613us) log.cc:844] Serialized 26 byte log entry
0709 12:24:45.508483 (+846334us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:45.508557 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.508675 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:45.661676 (+153001us) raft_consensus.cc:1685] finished
0709 12:24:45.661700 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:45.661847 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":160679,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":7786,"thread_start_us":1210310,"threads_started":2,"wal-append.queue_time_us":978566}
W20260709 12:24:45.688884 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.646704  8749 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:45.693317 31676 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.813s	user 0.013s	sys 0.019s
W20260709 12:24:45.693816 31676 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.013s	sys 0.019s
I20260709 12:24:45.648700  8756 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:45.697268 32375 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.012s	sys 0.018s
W20260709 12:24:45.697702 32375 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.718s	user 0.013s	sys 0.018s
I20260709 12:24:45.698177 32372 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: [25.17-25.17]
W20260709 12:24:45.699120  8726 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.729s	user 0.017s	sys 0.018s
W20260709 12:24:45.702863  8726 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.733s	user 0.018s	sys 0.018s
I20260709 12:24:45.694494  8721 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:45.693526  8749 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:24:45.645974  8631 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:45.740338 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2878) took 1456 ms. Trace:
I20260709 12:24:45.740783 31675 rpcz_store.cc:276] 0709 12:24:44.288682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.291393 (+  2711us) service_pool.cc:224] Handling call
0709 12:24:44.291749 (+   356us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:44.475504 (+183755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.740040 (+1264536us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b540017d9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:45.740271 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":338489728}
I20260709 12:24:45.740020  8002 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:45.748078 31747 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.723s	user 0.012s	sys 0.019s
W20260709 12:24:45.748997 31747 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.724s	user 0.013s	sys 0.019s
I20260709 12:24:45.747133  8378 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
W20260709 12:24:45.749614 31674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.863s	user 0.012s	sys 0.021s
I20260709 12:24:45.729135  8657 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:45.774755 31674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.888s	user 0.014s	sys 0.022s
I20260709 12:24:45.777014  8593 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:45.779490 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:45.771914  8768 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
I20260709 12:24:45.769495  8636 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:45.748605  8002 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:45.772060  8378 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:45.811028 31669 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.717s	user 0.010s	sys 0.028s
I20260709 12:24:45.828378 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:45.829191 31669 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.735s	user 0.010s	sys 0.028s
I20260709 12:24:45.811102  8378 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:45.812453  8593 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:45.834229  8593 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:45.837249  8593 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:45.860316 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:45.876771 31746 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:45.878114 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:45.878706 31746 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:45.872437 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:45.873895 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 318783us
I20260709 12:24:45.838481  8002 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:45.879736 31751 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:24:45.881013 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:45.881312 31749 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:45.888384 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:45.895483 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:45.940729 31643 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.596s	user 0.009s	sys 0.024s
W20260709 12:24:45.941215 31643 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.009s	sys 0.024s
W20260709 12:24:45.941599 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 615153us
I20260709 12:24:45.958377  8649 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.959043  8649 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:45.961153  8649 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:45.962265  8649 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:45.942835  8528 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:45.967329  8528 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:45.969291  8201 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.000s
W20260709 12:24:45.970824  8763 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.197s	user 0.000s	sys 0.001s
W20260709 12:24:45.972121  8754 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.133s	user 0.000s	sys 0.000s
I20260709 12:24:45.973441  8201 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 27.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:45.973838  8721 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:45.974548  8721 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:45.984485  8723 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:45.985620 31674 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:24:45.985327  8723 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:45.989650  8723 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:45.938807  8658 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:24:45.990813  8658 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:45.991202  8658 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:24:45.991621 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.873571  8726 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 11, Time since last communication: 0.064s
I20260709 12:24:45.893805  8636 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:45.973474  8762 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:24:46.012872 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:45.879364  8703 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.064s
I20260709 12:24:46.003983 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:46.005967 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:45.997576 31576 catalog_manager.cc:5697] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 15 to 18, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 18 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:45.933692  8637 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:46.003155 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:46.015112 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:46.016337  8765 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
W20260709 12:24:46.020574 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:45.942967  8518 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:46.048161 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45083.1us
I20260709 12:24:46.049355 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:46.054229 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:46.055228 31671 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:46.059403 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.069504 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.661s	user 0.005s	sys 0.023s
W20260709 12:24:46.075376 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.667s	user 0.006s	sys 0.023s
I20260709 12:24:46.035286  8270 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.044057  8642 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.045754  8702 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election lost for term 23. Reason: could not achieve majority
I20260709 12:24:46.032361  8657 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.013998  8636 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:46.080850 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.010s	sys 0.015s
W20260709 12:24:46.104605 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.011s	sys 0.015s
W20260709 12:24:46.105151 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 593306us
I20260709 12:24:46.052538  8518 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:46.079340  8530 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:24:46.040025  8650 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:46.143450 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:46.080338 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:46.096661 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:46.039157  8579 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:46.046252  8528 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:46.082105  8270 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:46.106192 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:46.160133 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54082.6us
I20260709 12:24:46.162164 31671 raft_consensus.cc:2410] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 29 because replica has last-logged OpId of term: 28 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:24:46.164206 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:46.053237  8593 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:46.033106  8637 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:46.138749 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.167869 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.133839 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:46.145844 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:46.101408 31746 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
W20260709 12:24:46.147779 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67548.4us
I20260709 12:24:46.164851 31824 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:46.171664 31675 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
W20260709 12:24:46.172232 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.007s	sys 0.032s
W20260709 12:24:46.172688 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.007s	sys 0.032s
W20260709 12:24:46.173292 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 593869us
I20260709 12:24:46.098755  8642 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:46.142755  8650 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:46.171591 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:46.222937 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:46.170852 31674 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 30 because replica has last-logged OpId of term: 28 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 19.
I20260709 12:24:46.118722  8657 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:46.145337  8607 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:46.230309 31824 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
W20260709 12:24:46.231534 31641 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.009s	sys 0.029s
I20260709 12:24:46.162403  8530 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:46.234076 31641 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.658s	user 0.009s	sys 0.029s
I20260709 12:24:46.232357 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:46.167009  8593 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:46.169965  8637 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:46.239434 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:46.240788 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:46.233448 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 61952.9us
W20260709 12:24:46.231918 31752 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.602s	user 0.008s	sys 0.026s
W20260709 12:24:46.222252  8425 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.665s	user 0.015s	sys 0.023s
I20260709 12:24:46.189788  8528 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
W20260709 12:24:46.165544  8562 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s	user 0.014s	sys 0.022s
I20260709 12:24:46.244618 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:46.158679  8579 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:46.228859  8534 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:46.234710  8607 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:46.237541  8464 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:24:46.294337 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.239527  8593 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:46.301095 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:46.307071 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62542.8us
I20260709 12:24:46.308341 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:46.308764 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:46.292026  8644 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:46.250913  8518 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:46.249073 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:46.317438 31675 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:46.176113  8378 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:46.263114  8723 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:46.261394  8201 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:46.269332  8184 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:46.236908  8002 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:46.245800  8425 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.750s	user 0.016s	sys 0.023s
I20260709 12:24:46.275617  8579 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:46.243026  8530 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:46.281016  8509 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.595s	user 0.010s	sys 0.000s
I20260709 12:24:46.281467  8534 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:46.287169 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:46.288728  8636 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:46.291406  8761 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.126s	user 0.000s	sys 0.000s
W20260709 12:24:46.266845  8767 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.173s	user 0.001s	sys 0.000s
I20260709 12:24:46.266049  8657 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:46.264117  8607 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:46.261865  8713 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:24:46.267647  8769 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
I20260709 12:24:46.247985  8528 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:46.319049 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:46.390607 31674 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:46.391495 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 72586.5us
I20260709 12:24:46.339665  8184 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:46.320581 32373 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.634s	user 0.014s	sys 0.010s
W20260709 12:24:46.412657 32373 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.015s	sys 0.010s
W20260709 12:24:46.320817 31753 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.667s	user 0.025s	sys 0.040s
W20260709 12:24:46.413784 31753 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.760s	user 0.025s	sys 0.040s
I20260709 12:24:46.321455 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:46.243408 31752 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.614s	user 0.008s	sys 0.026s
W20260709 12:24:46.302314  8562 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.892s	user 0.015s	sys 0.023s
I20260709 12:24:46.325174 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:46.250643  8270 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:46.416836  8778 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:46.417078  8595 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:46.417596  8778 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:46.417695  8595 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:46.418725  8778 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:46.419334  8184 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:46.423588 31678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.728s	user 0.015s	sys 0.029s
W20260709 12:24:46.424008 31678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.761s	user 0.015s	sys 0.029s
W20260709 12:24:46.335846 31818 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.911s	user 0.031s	sys 0.024s
W20260709 12:24:46.425055 31818 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.013s	user 0.031s	sys 0.024s
I20260709 12:24:46.316495  8702 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:46.420157  8595 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:46.334470 31672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.642s	user 0.009s	sys 0.027s
I20260709 12:24:46.375236 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:46.442910 31672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.764s	user 0.009s	sys 0.028s
I20260709 12:24:46.443181 31673 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:46.445196 31673 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:24:46.323153  8723 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:46.326225  8464 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.024s
W20260709 12:24:46.319870 32344 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.651s	user 0.019s	sys 0.044s
I20260709 12:24:46.326824  8721 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:46.456056 32344 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.787s	user 0.020s	sys 0.045s
I20260709 12:24:46.428742  8562 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:46.458083  8721 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:46.458806  8721 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:46.455096 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3817) took 1606 ms. Trace:
I20260709 12:24:46.460996 31672 rpcz_store.cc:276] 0709 12:24:44.849119 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.849243 (+   124us) service_pool.cc:224] Handling call
0709 12:24:44.849570 (+   327us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:44.849770 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:45.678843 (+829073us) log.cc:844] Serialized 14 byte log entry
0709 12:24:46.454496 (+775653us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:46.454585 (+    89us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:46.454728 (+   143us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.454829 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.455012 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12176,"prepare.queue_time_us":33766,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"thread_start_us":1578294,"threads_started":2,"wal-append.queue_time_us":750227}
W20260709 12:24:46.397723 31791 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.924s	user 0.016s	sys 0.059s
I20260709 12:24:46.393206 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:46.299801  8650 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 29.14, Last appended by leader: 14, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:46.375964  8425 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.051s
I20260709 12:24:46.316224  8644 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:46.332383  8693 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:46.440053 31674 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
W20260709 12:24:46.327883  8541 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.721s	user 0.009s	sys 0.040s
W20260709 12:24:46.441004 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.507637 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.454306  8702 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:46.437692 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3818) took 1577 ms. Trace:
I20260709 12:24:46.406302  8528 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:46.467206 31791 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.993s	user 0.018s	sys 0.061s
W20260709 12:24:46.511143 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 1.0376e+06us
W20260709 12:24:46.509125 32342 thread.cc:641] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.026s	sys 0.018s
W20260709 12:24:46.512204 32342 thread.cc:608] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.026s	sys 0.018s
W20260709 12:24:46.501363 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 108259us
W20260709 12:24:46.464177  8552 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.893s	user 0.019s	sys 0.024s
I20260709 12:24:46.441185  8651 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:46.442643 31671 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:46.460289  8774 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.026s
I20260709 12:24:46.424381  8642 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:46.513553 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:46.514679 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:46.501857 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1329) took 1525 ms. Trace:
I20260709 12:24:46.606114 32375 rpcz_store.cc:276] 0709 12:24:44.976657 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.976873 (+   216us) service_pool.cc:224] Handling call
0709 12:24:44.978460 (+  1587us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:44.978842 (+   382us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:44.978876 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:44.979404 (+   528us) log.cc:844] Serialized 26 byte log entry
0709 12:24:45.698173 (+718769us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:45.698215 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.698288 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:46.501555 (+803267us) raft_consensus.cc:1685] finished
0709 12:24:46.501581 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.501775 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":267,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":2734,"thread_start_us":718173,"threads_started":1,"wal-append.queue_time_us":1033717}
I20260709 12:24:46.479166  8562 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:46.508436 31678 rpcz_store.cc:276] 0709 12:24:44.860588 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.860755 (+   167us) service_pool.cc:224] Handling call
0709 12:24:44.861085 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:44.861317 (+   232us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:24:45.662652 (+801335us) log.cc:844] Serialized 14 byte log entry
0709 12:24:46.437106 (+774454us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:46.437195 (+    89us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:24:46.437291 (+    96us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.437409 (+   118us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.437601 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15739,"prepare.queue_time_us":36131,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":2062,"raft.queue_time_us":56511,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":15148,"thread_start_us":1555903,"threads_started":2}
W20260709 12:24:46.516516 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 626972us
I20260709 12:24:46.610684 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:46.487771  8602 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:24:46.506876  8541 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.905s	user 0.010s	sys 0.040s
I20260709 12:24:46.520455 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:46.523020  8582 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:46.535768 31676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.828s	user 0.015s	sys 0.035s
W20260709 12:24:46.624500 31676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.921s	user 0.016s	sys 0.036s
I20260709 12:24:46.625746 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3816) took 1777 ms. Trace:
I20260709 12:24:46.626035 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:46.540295 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:46.544917 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.628867 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.546087 31671 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:46.557168 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.587999 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:46.611685 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 97121.5us
I20260709 12:24:46.617920 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:46.629058 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:46.631489 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:46.631476 31746 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:46.626355 31676 rpcz_store.cc:276] 0709 12:24:44.848709 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:44.848841 (+   132us) service_pool.cc:224] Handling call
0709 12:24:44.864069 (+ 15228us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:44.864310 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:45.703119 (+838809us) log.cc:844] Serialized 14 byte log entry
0709 12:24:46.625163 (+922044us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:46.625267 (+   104us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:46.625345 (+    78us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.625464 (+   119us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.625660 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3996,"prepare.queue_time_us":48582,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"thread_start_us":1735686,"threads_started":2}
I20260709 12:24:46.528055  8650 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:46.628410 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3804) took 1609 ms. Trace:
W20260709 12:24:46.594610  8552 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.982s	user 0.019s	sys 0.025s
I20260709 12:24:46.650919 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:46.637179 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:46.636251 31747 rpcz_store.cc:276] 0709 12:24:45.018969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.023303 (+  4334us) service_pool.cc:224] Handling call
0709 12:24:45.023675 (+   372us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:45.024123 (+   448us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:45.024156 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:45.024846 (+   690us) log.cc:844] Serialized 52 byte log entry
0709 12:24:45.768711 (+743865us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:24:45.789593 (+ 20882us) log.cc:844] Serialized 14 byte log entry
0709 12:24:45.789883 (+   290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.789984 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:46.628134 (+838150us) raft_consensus.cc:1685] finished
0709 12:24:46.628163 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.628335 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":13889,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":68,"raft.queue_time_us":18449,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":7769,"thread_start_us":724147,"threads_started":1,"wal-append.queue_time_us":1021517}
I20260709 12:24:46.649083 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:46.607605  8516 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.664801 32378 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 28. Current term is 30. Ops: []
I20260709 12:24:46.646030 31671 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:46.667008 31670 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.567s	user 0.012s	sys 0.027s
I20260709 12:24:46.666913 32377 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:46.613715  8562 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:46.667586 31670 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s	user 0.012s	sys 0.028s
I20260709 12:24:46.613873  8530 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:46.735981 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 85175.1us
I20260709 12:24:46.748265 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:46.671205 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:46.664778 32376 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28. Ops: [27.18-27.18]
I20260709 12:24:46.771129 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:46.771795 31824 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:46.773411 31824 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:46.723668 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:46.666816 31671 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:46.774998 32369 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 28: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:46.614262  8713 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:46.623289  8201 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:46.635607  8721 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:46.520884  8723 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:46.793752 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:46.667948  8688 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.646152  8649 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:46.727963 32375 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:24:46.658119  8464 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:46.637117  8595 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:46.779438 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:46.654547  8657 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:46.733130  8756 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.029s	user 0.015s	sys 0.000s
I20260709 12:24:46.792493 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:46.608104  8651 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:46.668381  8552 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:46.800871 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:46.735975 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.656349  8582 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:46.804301 31677 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.757s	user 0.018s	sys 0.027s
I20260709 12:24:46.787674  8713 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:46.767827  8677 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:46.768343  8509 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:24:46.794610  8631 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.830s	user 0.029s	sys 0.024s
I20260709 12:24:46.735374  8607 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:46.667476  8642 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:46.786612 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:46.875342 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 88854.3us
W20260709 12:24:46.876103 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.636s	user 0.016s	sys 0.027s
I20260709 12:24:46.877666  8509 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:46.782192  8636 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:46.653781  8702 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:46.811962 31821 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31. Ops: [30.18-30.18]
I20260709 12:24:46.853225 31826 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 29 because replica has last-logged OpId of term: 26 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 12.
I20260709 12:24:46.854655 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:46.803756 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:46.883419 31820 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:24:46.867857  8631 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.904s	user 0.029s	sys 0.024s
W20260709 12:24:46.859248 31677 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.814s	user 0.018s	sys 0.027s
I20260709 12:24:46.734797  8516 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:46.864364 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.718s	user 0.025s	sys 0.041s
I20260709 12:24:46.878274  8509 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:46.860908  8705 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.604s	user 0.010s	sys 0.036s
I20260709 12:24:46.881376 32374 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
W20260709 12:24:46.887061 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.741s	user 0.026s	sys 0.041s
W20260709 12:24:46.876499 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.636s	user 0.016s	sys 0.027s
W20260709 12:24:46.890399 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 651442us
W20260709 12:24:46.890750 31717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.581s	user 0.008s	sys 0.037s
W20260709 12:24:46.891182 31717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.581s	user 0.008s	sys 0.037s
I20260709 12:24:46.891563 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:46.891750 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 590715us
I20260709 12:24:46.869724  8534 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:46.855195  8611 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.027s
I20260709 12:24:46.862047  8464 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:46.892767 31745 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.018s	sys 0.033s
W20260709 12:24:46.920382 31745 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.652s	user 0.018s	sys 0.033s
I20260709 12:24:46.888109 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:46.800145  8721 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:46.894702 32375 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: [31.19-31.19]
I20260709 12:24:46.857687  8657 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:46.893003  8705 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.638s	user 0.010s	sys 0.037s
W20260709 12:24:46.885453 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.647s	user 0.015s	sys 0.027s
W20260709 12:24:46.977399 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.739s	user 0.015s	sys 0.028s
I20260709 12:24:46.957540  8611 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Stepping down as leader of term 23
W20260709 12:24:46.896154  8777 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.297s	user 0.001s	sys 0.000s
W20260709 12:24:46.896049  8786 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.189s	user 0.000s	sys 0.001s
W20260709 12:24:46.895892  8784 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.143s	user 0.000s	sys 0.001s
I20260709 12:24:46.845016  8688 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:46.857569  8756 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:46.881873  8644 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:46.915963  8789 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.234s	user 0.000s	sys 0.001s
W20260709 12:24:46.915313  8686 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.234s	user 0.015s	sys 0.000s
I20260709 12:24:46.893136 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:46.855451  8562 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:46.955864 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.001379 31819 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:46.879760  8607 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:46.867506  8677 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:46.895583  8787 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:46.916635  8804 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:46.884073  8636 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:46.923285  8534 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:46.895650  8780 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.286s
W20260709 12:24:46.962265 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:46.882263  8642 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:46.890230  8552 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.062s
W20260709 12:24:46.966888  8658 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.975s	user 0.014s	sys 0.000s
I20260709 12:24:46.895454  8782 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.305s
W20260709 12:24:46.975620 31752 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.553s	user 0.014s	sys 0.015s
W20260709 12:24:47.047492 31752 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.015s	sys 0.015s
I20260709 12:24:47.048312 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3047) took 1826 ms. Trace:
I20260709 12:24:46.916044 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3822) took 1746 ms. Trace:
I20260709 12:24:46.921434 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3049) took 1698 ms. Trace:
I20260709 12:24:46.958034  8378 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:46.967410  8779 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.323s	user 0.000s	sys 0.001s
I20260709 12:24:47.051743 31745 rpcz_store.cc:276] 0709 12:24:45.223550 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.691936 (+468386us) service_pool.cc:224] Handling call
0709 12:24:45.692359 (+   423us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:45.692632 (+   273us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:45.708341 (+ 15709us) log.cc:844] Serialized 14 byte log entry
0709 12:24:46.268483 (+560142us) spinlock_profiling.cc:243] Waited 908 us on lock 0x7b18004d9320. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:24:46.920981 (+652498us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:46.921054 (+    73us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:46.921116 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.921206 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.921363 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15383,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":1594,"raft.queue_time_us":151,"spinlock_wait_cycles":8528128,"thread_start_us":652199,"threads_started":1}
I20260709 12:24:47.064208 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:47.050735 31753 rpcz_store.cc:276] 0709 12:24:45.169242 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.169436 (+   194us) service_pool.cc:224] Handling call
0709 12:24:45.169735 (+   299us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:45.169946 (+   211us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:45.213885 (+ 43939us) log.cc:844] Serialized 14 byte log entry
0709 12:24:45.654030 (+440145us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b1800473bc0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:24:46.915357 (+1261327us) log.cc:844] Serialized 14 byte log entry
0709 12:24:46.915602 (+   245us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:46.915671 (+    69us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:46.915734 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.915824 (+    90us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:46.915979 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":43633,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":46,"raft.queue_time_us":48527,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":8217,"spinlock_wait_cycles":3373312,"thread_start_us":1247338,"threads_started":2,"wal-append.queue_time_us":1309561}
I20260709 12:24:47.065254 31821 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
W20260709 12:24:46.984735 31751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s	user 0.010s	sys 0.013s
W20260709 12:24:46.997439 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 104401us
I20260709 12:24:46.981434  8611 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:47.069906 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:46.935420  8713 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:47.013967 31673 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.554s	user 0.011s	sys 0.010s
I20260709 12:24:47.006052  8663 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.048789 31752 rpcz_store.cc:276] 0709 12:24:45.222263 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.628355 (+406092us) service_pool.cc:224] Handling call
0709 12:24:45.628664 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:45.628882 (+   218us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:45.629199 (+   317us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.047872 (+1418673us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:47.047931 (+    59us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:47.047978 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:47.048065 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:47.048235 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":155,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":33,"raft.queue_time_us":44090,"raft.run_cpu_time_us":71,"raft.run_wall_time_us":71,"spinlock_wait_cycles":12126464,"thread_start_us":1418189,"threads_started":2,"wal-append.queue_time_us":827544}
W20260709 12:24:47.051469  8790 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 12:24:47.098752 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:47.099900 31672 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
W20260709 12:24:47.084595 31751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s	user 0.011s	sys 0.014s
W20260709 12:24:47.085346 31673 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.012s	sys 0.010s
I20260709 12:24:47.086359  8796 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:47.087299  8801 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:47.089502 31674 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:47.091284 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:47.093204 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:47.019385  8562 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:47.123601 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:47.007115 31818 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.545s	user 0.008s	sys 0.012s
W20260709 12:24:47.096127 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 303706us
I20260709 12:24:47.104775 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:47.096920  8644 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:47.098505 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 255938us
I20260709 12:24:47.102360 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:47.096606  8786 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:24:47.118709 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:47.120337 31671 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:46.989846  8464 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.125556 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:47.097221 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:47.126428 31818 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.671s	user 0.009s	sys 0.013s
W20260709 12:24:47.127915 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.128971 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:47.130425 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60590.5us
I20260709 12:24:47.054513  8705 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.068s
I20260709 12:24:47.053288  8658 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.098603  8686 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.179164 31670 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 18. Current term is 19. Ops: []
I20260709 12:24:47.100360  8782 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:47.195668 31678 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: []
W20260709 12:24:47.156399 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:47.157490  8791 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.189s	user 0.000s	sys 0.000s
W20260709 12:24:47.158340  8794 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.111s	user 0.000s	sys 0.000s
I20260709 12:24:47.163815  8582 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:47.103860  8631 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.065s
I20260709 12:24:47.166963 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:47.172092  8509 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:47.176011  8607 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:47.129316  8801 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:47.174077  8800 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
I20260709 12:24:47.174149  8642 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:47.174853  8590 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:47.178323  8658 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:47.175498  8701 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:47.135488  8644 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:47.128623  8775 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.232s	user 0.000s	sys 0.001s
I20260709 12:24:47.171903 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:47.306649 32375 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
W20260709 12:24:47.152494  8806 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:24:47.188287 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:47.332108 31671 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:47.288690  8590 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:47.332935  8590 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:47.333506 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:47.189867 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:47.335004 31674 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:47.199273 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3828) took 1190 ms. Trace:
I20260709 12:24:47.338771  8818 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.127899  8781 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.339638  8818 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:47.340041  8781 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:47.206164 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 64849.4us
I20260709 12:24:47.343329 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:47.343942  8772 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.021s	user 0.007s	sys 0.001s
W20260709 12:24:47.181779  8650 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.535s	user 0.005s	sys 0.003s
W20260709 12:24:47.178506  8795 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
W20260709 12:24:47.180037  8637 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.932s	user 0.013s	sys 0.001s
I20260709 12:24:47.254374  8579 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.008s
I20260709 12:24:47.254963 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:47.357278 31670 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:47.255340  8425 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:47.262089  8509 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.172326  8798 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:47.171933  8677 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:47.137734  8799 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:47.183126  8632 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:24:47.176338  8657 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:47.291070 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 374577us
I20260709 12:24:47.392935 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:47.394150 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.123452  8802 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.072s
I20260709 12:24:47.358126  8804 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:47.183092  8721 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:47.181555  8688 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:47.258841  8797 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.282s	user 0.001s	sys 0.000s
W20260709 12:24:47.246928  8528 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.726s	user 0.007s	sys 0.005s
I20260709 12:24:47.102486  8663 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.308887 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:47.418540 31678 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:24:47.419842 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:47.305015  8801 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:47.310075  8582 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.235016  8782 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:47.262602  8749 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.535s	user 0.004s	sys 0.013s
W20260709 12:24:47.314064  8593 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.999s	user 0.014s	sys 0.000s
I20260709 12:24:47.306069  8756 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:47.306419  8701 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:47.205282  8686 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:47.183189  8611 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.325692  8649 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:47.263890  8607 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:47.330906  8642 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 12:24:47.345441 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3826) took 2457 ms. Trace:
I20260709 12:24:47.468264 31669 rpcz_store.cc:276] 0709 12:24:44.888118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.048816 (+160698us) service_pool.cc:224] Handling call
0709 12:24:45.051021 (+  2205us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:45.092993 (+ 41972us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:45.093046 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:45.093792 (+   746us) log.cc:844] Serialized 26 byte log entry
0709 12:24:45.829803 (+736011us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:45.829868 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:45.829945 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:47.170950 (+1341005us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:47.345063 (+174113us) raft_consensus.cc:1685] finished
0709 12:24:47.345092 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:47.345352 (+   260us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":41317,"raft.queue_time_us":54618,"raft.run_cpu_time_us":162,"raft.run_wall_time_us":29891,"spinlock_wait_cycles":33033600,"thread_start_us":735503,"threads_started":1,"wal-append.queue_time_us":1149750}
I20260709 12:24:47.346282 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3840) took 1062 ms. Trace:
I20260709 12:24:47.338893 31677 rpcz_store.cc:276] 0709 12:24:46.008517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.011047 (+  2530us) service_pool.cc:224] Handling call
0709 12:24:46.011376 (+   329us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:46.011576 (+   200us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:46.044688 (+ 33112us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.198709 (+1154021us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:47.198803 (+    94us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:47.198879 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:47.198981 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:47.199169 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":39932,"prepare.run_cpu_time_us":94,"prepare.run_wall_time_us":2957,"raft.queue_time_us":109668,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":13962,"thread_start_us":1152866,"threads_started":2,"wal-append.queue_time_us":946501}
I20260709 12:24:47.349419 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:47.347062 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:47.355863 31674 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
W20260709 12:24:47.348729 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 416993us
I20260709 12:24:47.348232  8774 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:47.387027  8799 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:47.413753  8775 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.056s	user 0.001s	sys 0.000s
W20260709 12:24:47.426460 31824 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.013s	sys 0.004s
W20260709 12:24:47.424841 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 532294us
I20260709 12:24:47.390048  8632 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:47.454792 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:47.455016 31672 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 28. Current term is 30. Ops: []
I20260709 12:24:47.367357  8637 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.387024  8726 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.042s
I20260709 12:24:47.384331  8509 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:47.459636 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 310525us
I20260709 12:24:47.384611  8425 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:47.428246  8528 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:47.472708 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 125752us
W20260709 12:24:47.530021 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 184981us
I20260709 12:24:47.473281 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:47.632710 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:47.633862 31824 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.768s	user 0.025s	sys 0.012s
I20260709 12:24:47.633924 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:47.631398 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3827) took 1504 ms. Trace:
W20260709 12:24:47.632743 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 490751us
I20260709 12:24:47.635371 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:47.639758 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:47.620759 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:47.536427 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3828) took 1409 ms. Trace:
W20260709 12:24:47.635340 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.645455 31678 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 23: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:47.647066 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:47.644313 31750 rpcz_store.cc:276] 0709 12:24:46.127522 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.130111 (+  2589us) service_pool.cc:224] Handling call
0709 12:24:46.130602 (+   491us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:46.131128 (+   526us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:46.131171 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:46.131839 (+   668us) log.cc:844] Serialized 26 byte log entry
0709 12:24:46.481760 (+349921us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:46.481806 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.481903 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:47.536167 (+1054264us) raft_consensus.cc:1685] finished
0709 12:24:47.536193 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:47.536356 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":14954,"raft.queue_time_us":4732,"raft.run_cpu_time_us":156,"raft.run_wall_time_us":6639,"thread_start_us":316008,"threads_started":1,"wal-append.queue_time_us":720417}
I20260709 12:24:47.647800 31752 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:47.472060 31675 rpcz_store.cc:276] 0709 12:24:46.284329 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.417434 (+133105us) service_pool.cc:224] Handling call
0709 12:24:46.417765 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:46.417975 (+   210us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:46.418264 (+   289us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.345876 (+927612us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:47.345939 (+    63us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:47.346008 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:47.346091 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:47.346215 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":21450,"prepare.queue_time_us":4237,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":50,"raft.queue_time_us":122193,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":41742,"thread_start_us":927226,"threads_started":2,"wal-append.queue_time_us":709551}
I20260709 12:24:47.649817 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:47.650214 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:47.688570 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:47.464987  8002 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:47.488168  8756 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:47.635105 31751 rpcz_store.cc:276] 0709 12:24:46.127129 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.135754 (+  8625us) service_pool.cc:224] Handling call
0709 12:24:46.409427 (+273673us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:24:46.409781 (+   354us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:46.427566 (+ 17785us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:46.427599 (+    33us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:46.456446 (+ 28847us) log.cc:844] Serialized 52 byte log entry
0709 12:24:47.111042 (+654596us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:47.557662 (+446620us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.559932 (+  2270us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:47.560046 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:47.560107 (+    61us) raft_consensus.cc:1685] finished
0709 12:24:47.609911 (+ 49804us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:47.631223 (+ 21312us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15803,"prepare.queue_time_us":123297,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":68,"raft.queue_time_us":33355,"raft.run_cpu_time_us":305,"raft.run_wall_time_us":26520,"spinlock_wait_cycles":23930624,"thread_start_us":977294,"threads_started":2,"wal-append.queue_time_us":716161}
W20260709 12:24:47.692910 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48715.5us
I20260709 12:24:47.696192  8642 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:47.715864 31824 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:47.698387 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:47.699002 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:47.457491  8582 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.722105 31753 raft_consensus.cc:2376] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27. Re-sending same reply.
I20260709 12:24:47.715932 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:47.646189  8799 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:47.699767 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:47.612712  8701 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.700448 31749 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "181ddae071d549aab328e3c1d8553734"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:47.700966 31669 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:47.732892 31669 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:47.732913 31749 raft_consensus.cc:606] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Received request to transfer leadership to TS b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:47.698500  8832 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.460135  8593 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.704005  8822 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.033s
I20260709 12:24:47.483136  8818 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:47.708285  8509 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:47.710150  8637 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:47.711274  8817 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.248s	user 0.000s	sys 0.001s
I20260709 12:24:47.591032  8807 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:47.610428  8562 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:47.717638 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:47.714164 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:47.634227  8798 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:47.687278  8809 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:24:47.715107 31823 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:24:47.717281  8002 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:47.714262 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:47.733502 31749 raft_consensus.cc:614] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting request to transer leadership while not leader
W20260709 12:24:47.739319 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44723.6us
I20260709 12:24:47.741225 31752 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:24:47.673496  8787 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:47.751572  8644 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:47.752826  8726 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:47.758067 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:47.758420 31675 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:47.460824  8749 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:47.803637 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 89573.8us
I20260709 12:24:47.802143 31821 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:47.749519  8590 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.813862 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:47.760270 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 62543us
I20260709 12:24:47.712271  8756 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:47.762933 31669 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.782099 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1487) took 1128 ms. Trace:
I20260709 12:24:47.785812 31670 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:47.820544 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:47.823472 32371 rpcz_store.cc:276] 0709 12:24:46.654030 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.654343 (+   313us) service_pool.cc:224] Handling call
0709 12:24:46.780136 (+125793us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:46.780450 (+   314us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:46.780478 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:46.780523 (+    45us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:46.780561 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.780618 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:47.781930 (+1001312us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x7b54003f3250. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:47.782042 (+   112us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":90248,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":9238,"spinlock_wait_cycles":260193152}
I20260709 12:24:47.824545 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:47.781380  8804 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:47.716297 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:47.834767 31750 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 26 because replica has last-logged OpId of term: 25 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 12:24:47.712074  8632 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:47.764719  8701 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:47.750090  8840 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:24:47.817014 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 102838us
I20260709 12:24:47.849546 32375 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
W20260709 12:24:47.760941 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 111177us
I20260709 12:24:47.788314  8834 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:47.789506 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3835) took 2548 ms. Trace:
I20260709 12:24:47.801998  8726 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:47.759115  8832 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.816941  8637 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:47.837486 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:47.842878  8582 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:47.846485  8723 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:47.848031  8590 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:47.825232  8756 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:47.822808  8642 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:47.791679  8562 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:47.864692 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:47.816147  8787 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:47.851471 31818 rpcz_store.cc:276] 0709 12:24:45.232987 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.248361 (+ 15374us) service_pool.cc:224] Handling call
0709 12:24:45.248760 (+   399us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:45.249387 (+   627us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:45.337453 (+ 88066us) log.cc:844] Serialized 14 byte log entry
0709 12:24:46.425573 (+1088120us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:46.435156 (+  9583us) log.cc:844] Serialized 26 byte log entry
0709 12:24:46.435413 (+   257us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:47.144954 (+709541us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.431144 (+286190us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:47.445959 (+ 14815us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:47.763675 (+317716us) raft_consensus.cc:1685] finished
0709 12:24:47.763707 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:47.781049 (+ 17342us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":821042,"prepare.run_cpu_time_us":106,"prepare.run_wall_time_us":110,"raft.queue_time_us":173599,"raft.run_cpu_time_us":466,"raft.run_wall_time_us":29456,"thread_start_us":1979913,"threads_started":3,"wal-append.queue_time_us":1410812}
I20260709 12:24:47.866786 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:47.835780  8607 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.018s
I20260709 12:24:47.793605  8798 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.835822  8579 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election won for term 27
W20260709 12:24:47.853714  8828 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.220s	user 0.000s	sys 0.000s
I20260709 12:24:47.836701  8818 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 19.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.017s
I20260709 12:24:47.825435  8644 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:47.835455  8804 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:47.841228  8703 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:47.848165  8842 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.010s
I20260709 12:24:47.853735  8602 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:47.856890  8834 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:47.858083  8637 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:47.855053  8726 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:47.866449  8657 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:47.864935  8509 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:24:47.867259  8839 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.061s	user 0.001s	sys 0.001s
I20260709 12:24:47.857821  8831 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.022s
I20260709 12:24:47.887878  8642 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:47.888108  8562 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:47.888383  8642 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:47.888490  8798 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:47.875089  8602 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:47.879976  8593 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:47.875350  8703 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:47.880029  8787 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.21, Last appended by leader: 21, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.878532  8637 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:47.883136  8726 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:47.883668  8651 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:47.884146  8425 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:47.884213  8595 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:47.867925  8590 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.899395  8595 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:47.899669  8590 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.900559  8677 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:47.900478  8595 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.18, Last appended by leader: 18, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.901947  8595 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:47.890594  8831 consensus_queue.cc:1243] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 28.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
W20260709 12:24:47.904004  8788 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.102s	user 0.003s	sys 0.000s
I20260709 12:24:47.903754  8642 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.905114  8642 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:47.874231  8804 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:47.872162  8644 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:47.880797  8749 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:47.873718  8579 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:47.912197  8811 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:24:47.908838  8644 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:47.912693  8811 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:47.915107  8752 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:47.915678  8752 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:47.892868  8832 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:47.893790  8834 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:47.894317  8703 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:47.901057  8677 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:47.885251  8835 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Stepping down as leader of term 21
I20260709 12:24:47.923236  8835 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:47.923326  8677 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:47.913641  8811 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:47.924309  8785 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.467s	user 0.005s	sys 0.000s
I20260709 12:24:47.924759  8677 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:47.925021  8811 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:47.924281  8835 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 21, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:47.929030  8835 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:47.917824  8752 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:47.905939  8799 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:47.934496  8774 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:47.934993  8782 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:47.919946  8783 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:47.935276  8774 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:47.936651  8799 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:47.933961  8834 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:47.996790 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:47.999392 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.002950 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:48.005152 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.005239 31670 raft_consensus.cc:2410] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 29 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 17.
I20260709 12:24:48.007135 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:48.008781 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3845) took 1722 ms. Trace:
W20260709 12:24:48.010951 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.009342 31673 rpcz_store.cc:276] 0709 12:24:46.285954 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.454011 (+168057us) service_pool.cc:224] Handling call
0709 12:24:46.454395 (+   384us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:24:47.103066 (+648671us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:47.485487 (+382421us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.759082 (+273595us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:24:47.790519 (+ 31437us) log.cc:844] Serialized 78 byte log entry
0709 12:24:47.790709 (+   190us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:47.795079 (+  4370us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.007467 (+212388us) log.cc:844] Serialized 14 byte log entry
0709 12:24:48.008011 (+   544us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.008163 (+   152us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:48.008250 (+    87us) raft_consensus.cc:1685] finished
0709 12:24:48.008261 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.008561 (+   300us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32019,"prepare.queue_time_us":884741,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":61,"raft.queue_time_us":104072,"raft.run_cpu_time_us":2518,"raft.run_wall_time_us":46043,"thread_start_us":1227647,"threads_started":4,"wal-append.queue_time_us":165723}
I20260709 12:24:47.803683 31675 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 24. Current term is 25. Ops: [24.18-24.18]
W20260709 12:24:48.014657 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 163670us
I20260709 12:24:48.030572 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:48.031574 31675 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:48.033653 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:48.035502 31673 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:48.037583 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:48.038478 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:48.040555 31670 raft_consensus.cc:2376] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25. Re-sending same reply.
W20260709 12:24:48.042711 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 192042us
I20260709 12:24:48.044132 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:48.047127 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:48.054152 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 187328us
I20260709 12:24:48.055984 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:48.059216 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:48.062435 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:48.076342 31818 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:48.085693 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:48.086799 32375 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:48.088141 32371 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:48.076362 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:48.090617 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1475) took 3002 ms. Trace:
I20260709 12:24:48.091208 32373 rpcz_store.cc:276] 0709 12:24:45.088486 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:45.093970 (+  5484us) service_pool.cc:224] Handling call
0709 12:24:45.094331 (+   361us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:45.095263 (+   932us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:45.095289 (+    26us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:45.112544 (+ 17255us) log.cc:844] Serialized 52 byte log entry
0709 12:24:45.686535 (+573991us) spinlock_profiling.cc:243] Waited 21.1 ms on lock 0x7b18003504a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:24:46.413290 (+726755us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:46.413364 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:46.413504 (+   140us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:47.903419 (+1489915us) spinlock_profiling.cc:243] Waited 4.54 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:48.089924 (+186505us) raft_consensus.cc:1685] finished
0709 12:24:48.089981 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.090357 (+   376us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":11891,"raft.run_cpu_time_us":166,"raft.run_wall_time_us":29823,"spinlock_wait_cycles":58873600,"thread_start_us":726464,"threads_started":1,"wal-append.queue_time_us":1507667}
I20260709 12:24:48.094027 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:48.078691 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:48.095633 31824 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:48.080135 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3842) took 1310 ms. Trace:
I20260709 12:24:48.096650 31825 rpcz_store.cc:276] 0709 12:24:46.769787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.772294 (+  2507us) service_pool.cc:224] Handling call
0709 12:24:46.772592 (+   298us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:46.782322 (+  9730us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:24:47.129863 (+347541us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.783658 (+653795us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:47.791610 (+  7952us) log.cc:844] Serialized 26 byte log entry
0709 12:24:47.791972 (+   362us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:24:47.792029 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:47.792131 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:48.079565 (+287434us) raft_consensus.cc:1685] finished
0709 12:24:48.079603 (+    38us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.079942 (+   339us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":52703,"prepare.queue_time_us":390007,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":46961,"raft.run_cpu_time_us":259,"raft.run_wall_time_us":8462,"thread_start_us":817670,"threads_started":2,"wal-append.queue_time_us":625624}
I20260709 12:24:48.085253 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:48.101871 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:48.105741 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:48.107131  8270 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:48.108924  8270 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:48.108319  8425 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.002s
I20260709 12:24:48.111286  8270 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.13, Last appended by leader: 13, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:47.893428  8516 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:48.174003 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.117275  8541 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:48.118284  8562 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:48.182574  8562 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.184923 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3848) took 1408 ms. Trace:
I20260709 12:24:47.911680  8579 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.118780  8782 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.194156  8562 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.195739 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 16 index: 11 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:48.197345 31824 raft_consensus.cc:2410] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 20 because replica has last-logged OpId of term: 18 index: 12, which is greater than that of the candidate, which has last-logged OpId of term: 16 index: 11.
I20260709 12:24:48.120033  8590 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:48.115871 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:48.199652 31673 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 33 because replica has last-logged OpId of term: 31 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 18.
I20260709 12:24:48.121680  8595 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:48.117395  8723 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:48.135150  8705 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:48.141273  8813 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:48.141562 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:48.142429 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 80314.8us
I20260709 12:24:48.143111  8832 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:48.144353  8811 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.145892  8831 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:48.146327  8835 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:47.939641  8752 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:48.121842  8851 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:48.157141 31676 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:48.156334 31576 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 14 to 18, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 18 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:47.938567  8783 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:48.159562  8799 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:48.160974  8804 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.168668  8607 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:48.168972  8701 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:47.870198  8528 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.230684  8783 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.231760  8804 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:48.181589 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3850) took 1301 ms. Trace:
I20260709 12:24:48.233808 31822 rpcz_store.cc:276] 0709 12:24:46.879629 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.880378 (+   749us) service_pool.cc:224] Handling call
0709 12:24:46.880796 (+   418us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:46.929638 (+ 48842us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:46.929683 (+    45us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:46.929747 (+    64us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:46.929795 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.180871 (+1251076us) spinlock_profiling.cc:243] Waited 48.6 ms on lock 0x7b54001982d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:48.180969 (+    98us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.181321 (+   352us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":111688576}
I20260709 12:24:48.202517  8723 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:48.218958 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:48.236305 31678 raft_consensus.cc:2393] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:48.219146  8811 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.236760  8723 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.219915  8602 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.193986  8782 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.224668  8799 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:48.230058  8701 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.239598  8783 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:48.244447  8602 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.243798  8799 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.239601  8811 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.246176  8686 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:48.242555 31669 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:48.249379  8701 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:48.252501 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:48.253573 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:48.257999 31819 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3841) took 1488 ms (client timeout 1525 ms). Trace:
W20260709 12:24:48.274827 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 59704.6us
I20260709 12:24:48.185683 31827 rpcz_store.cc:276] 0709 12:24:46.776186 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.776422 (+   236us) service_pool.cc:224] Handling call
0709 12:24:46.776785 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:46.933461 (+156676us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:47.286724 (+353263us) log.cc:844] Serialized 14 byte log entry
0709 12:24:47.471386 (+184662us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:47.543267 (+ 71881us) log.cc:844] Serialized 26 byte log entry
0709 12:24:47.633028 (+ 89761us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:47.633069 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.183761 (+550692us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54000fc654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:48.183885 (+   124us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:48.184034 (+   149us) raft_consensus.cc:1685] finished
0709 12:24:48.184199 (+   165us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.184638 (+   439us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5383,"prepare.queue_time_us":239091,"prepare.run_cpu_time_us":30,"prepare.run_wall_time_us":31,"raft.queue_time_us":67743,"raft.run_cpu_time_us":395,"raft.run_wall_time_us":24529,"spinlock_wait_cycles":350007552,"thread_start_us":468906,"threads_started":2,"wal-append.queue_time_us":454898}
I20260709 12:24:48.242413  8705 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.285481  8813 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.253664  8774 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.294239 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:48.295414 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:48.370537 31750 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:24:48.296047 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:48.297089 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:48.297616  8270 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:48.298024  8841 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.298229 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:48.314054  8781 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:48.315268  8637 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.297961  8851 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:48.317070  8829 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.528s	user 0.002s	sys 0.000s
I20260709 12:24:48.320304 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 2892) took 1171 ms. Trace:
I20260709 12:24:48.320608 32374 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:48.321906 32372 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:48.326990  8663 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:48.326822  8644 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.339726  8632 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:48.342684  8611 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:48.388679 31672 rpcz_store.cc:276] 0709 12:24:47.149043 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.149412 (+   369us) service_pool.cc:224] Handling call
0709 12:24:47.166462 (+ 17050us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:47.642330 (+475868us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.319765 (+677435us) spinlock_profiling.cc:243] Waited 271 ms on lock 0x7b540022b854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:48.320069 (+   304us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":622152960}
I20260709 12:24:48.330472 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:48.411041 31826 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 27: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:48.354871 32369 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 16 index: 11. Preceding OpId from leader: term: 19 index: 12. (index mismatch)
I20260709 12:24:48.356419  8552 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:48.360131 31823 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
W20260709 12:24:48.364203 31819 rpcz_store.cc:269] 0709 12:24:46.769501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:46.772911 (+  3410us) service_pool.cc:224] Handling call
0709 12:24:48.257617 (+1484706us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54001e9bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:48.257799 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":336411904}
I20260709 12:24:48.367053 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:48.324535  8636 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:48.372478 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:48.381187 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.361559  8595 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:48.386591 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 211928us
I20260709 12:24:48.378408 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:48.413779 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:48.421041 31748 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:48.422435  8002 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:48.423240 32371 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:48.422920  8595 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.424530 32375 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 29: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:48.424902 32373 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:48.426294 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:48.443324 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:48.444974 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:48.447507 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:48.441757 31752 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:24:48.450270 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:48.463183 31822 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 25. Current term is 26. Ops: [25.19-25.19]
I20260709 12:24:48.462898  8703 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:48.464411 31569 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42072 (request call id 245) took 40804 ms (client timeout 15000 ms). Trace:
I20260709 12:24:48.465412  8703 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.467223 31818 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:24:48.468755 31749 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: []
W20260709 12:24:48.468806  8756 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.599s	user 0.000s	sys 0.016s
I20260709 12:24:48.470256  8781 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:48.451834 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:48.471148 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:48.470947  8781 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.477305  8721 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:48.486464 31823 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:48.471510 32376 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 19.
I20260709 12:24:48.483387 31745 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: [26.16-26.16]
I20260709 12:24:48.492666 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:48.499362 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:48.424814  8841 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.500142 31824 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:24:48.509246 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:48.433322  8851 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:48.491256  8844 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:48.469599  8756 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.435779  8552 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.518733 31827 raft_consensus.cc:2410] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 29 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 17.
I20260709 12:24:48.513417 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:48.448323  8270 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:48.531863 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 75834.6us
I20260709 12:24:48.527900 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:48.528669 31826 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:48.547228 31672 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:48.548274 31820 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:24:48.550729 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:48.508970  8663 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:48.553105 32377 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 28: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:48.460080 31746 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:48.555112 31746 raft_consensus.cc:606] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:48.555868  8844 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.556555 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:48.558732  8796 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.559376  8796 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.561599  8796 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:48.519089  8721 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.562675 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:48.563468 31818 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:48.564752  8721 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.565455  8782 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.566068 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:48.566955 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 197293us
I20260709 12:24:48.567374 31749 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:48.558337  8844 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:48.571475 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:48.571789 31819 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 31 index: 18. (term mismatch)
I20260709 12:24:48.585059 31819 pending_rounds.cc:85] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 17
I20260709 12:24:48.575192  8774 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:48.576529  8642 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.671s	user 0.013s	sys 0.006s
I20260709 12:24:48.579432  8799 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:48.580193  8851 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:24:48.580417 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 82811.4us
W20260709 12:24:48.580888  8677 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.655s	user 0.012s	sys 0.010s
I20260709 12:24:48.581084  8787 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:48.552668  8595 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.593943 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:48.582103 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:48.523697  8002 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:24:48.586226 31824 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 18 index: 12. Preceding OpId from leader: term: 19 index: 12. (term mismatch)
I20260709 12:24:48.585726 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:48.596846 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:48.585350 31819 pending_rounds.cc:107] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 30.18
I20260709 12:24:48.595980 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:48.603089 31819 raft_consensus.cc:2889] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:48.603681 31822 raft_consensus.cc:2410] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 28 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:24:48.605552 32375 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: [25.17-25.17]
I20260709 12:24:48.576586  8552 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.610373 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:48.602298 31824 pending_rounds.cc:85] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 11
W20260709 12:24:48.613567 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.611827 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:48.613763 31824 pending_rounds.cc:107] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 18.12
I20260709 12:24:48.616097 31824 raft_consensus.cc:2889] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:48.595988  8705 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.619465 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:48.619421 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:48.623876 31678 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:48.628046 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:48.629480 31821 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 16. (term mismatch)
I20260709 12:24:48.629673  8644 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:48.629616 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:48.630159 31821 pending_rounds.cc:85] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 15
I20260709 12:24:48.630570 31821 pending_rounds.cc:107] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 27.16
I20260709 12:24:48.630934 31821 raft_consensus.cc:2889] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:48.632197 31826 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:48.632714 32373 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: [23.15-23.15]
I20260709 12:24:48.633738  8841 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.634073 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:48.635854 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:48.635015 32372 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:48.611840  8807 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:48.609665  8637 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:48.639396 31827 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:24:48.645342 31748 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:24:48.635106 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:48.647754 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1449) took 1444 ms. Trace:
I20260709 12:24:48.590783  8663 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.638778 31820 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:24:48.652045 31669 rpcz_store.cc:276] 0709 12:24:47.191249 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:47.700485 (+509236us) service_pool.cc:224] Handling call
0709 12:24:47.700836 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:48.241628 (+540792us) spinlock_profiling.cc:243] Waited 416 ms on lock 0x7b58009afdd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd310aa 00007fe8fcdb3447 00007fe8fcdcbfe7 00007fe8fcdcd7de 00007fe8fcdd7b49 00007fe8fcdd8416 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4
0709 12:24:48.609663 (+368035us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:24:48.609702 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:48.609746 (+    44us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:48.609780 (+    34us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:48.617661 (+  7881us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:48.635710 (+ 18049us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3175,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":9053,"spinlock_wait_cycles":957169024}
W20260709 12:24:48.659153  8530 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260709 12:24:48.652756 32343 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:48.666447 31822 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:24:48.668047 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:48.649078  8807 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:48.643436  8813 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.655905  8541 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.646783  8677 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:48.674281 32376 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:48.674827 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:48.675170 32378 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:48.660691  8841 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:48.672444 31745 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:24:48.677788  8807 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:48.688483 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 106554us
W20260709 12:24:48.687355 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66584.9us
I20260709 12:24:48.687556 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:48.685184  8787 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:48.699095  8756 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:48.706939 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52000.8us
I20260709 12:24:48.682113  8541 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.707904 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:48.709265  8270 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:48.464813 31569 rpcz_store.cc:269] 0709 12:24:07.660272 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:19.392104 (+11731832us) service_pool.cc:224] Handling call
0709 12:24:48.464116 (+29072012us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:38.676705 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:38.842908 (+166203us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:38.851611 (+  8703us) write_op.cc:435] Acquired schema lock
0709 12:24:38.860110 (+  8499us) tablet.cc:662] Decoding operations
0709 12:24:38.862618 (+  2508us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:38.906381 (+ 43763us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:38.911458 (+  5077us) tablet.cc:685] Acquiring locks for 22 operations
0709 12:24:38.948543 (+ 37085us) tablet.cc:701] Row locks acquired
0709 12:24:38.948600 (+    57us) write_op.cc:260] PREPARE: finished
0709 12:24:38.948772 (+   172us) write_op.cc:270] Start()
0709 12:24:38.948941 (+   169us) write_op.cc:276] Timestamp: P: 1783599878948703 usec, L: 0
0709 12:24:38.948973 (+    32us) op_driver.cc:348] REPLICATION: starting
0709 12:24:38.949791 (+   818us) log.cc:844] Serialized 8721 byte log entry
0709 12:24:39.538973 (+589182us) op_driver.cc:464] REPLICATION: finished
0709 12:24:40.079284 (+540311us) write_op.cc:301] APPLY: starting
0709 12:24:40.231783 (+152499us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:40.720790 (+489007us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:40.735397 (+ 14607us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:41.030012 (+294615us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:41.181714 (+151702us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=44,key_file_lookups=44,delta_file_lookups=0,mrs_lookups=0
0709 12:24:41.181786 (+    72us) write_op.cc:312] APPLY: finished
0709 12:24:41.182305 (+   519us) log.cc:844] Serialized 196 byte log entry
0709 12:24:41.515067 (+332762us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:41.573843 (+ 58776us) write_op.cc:454] Released schema lock
0709 12:24:41.587519 (+ 13676us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:41.666068 (+ 78549us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:41.670343 (+  4275us) write_op.cc:454] Released schema lock
0709 12:24:42.303709 (+633366us) spinlock_profiling.cc:243] Waited 2.92 ms on lock 0x7b180006af00. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
Metrics: {"mutex_wait_us":23192,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":454683,"apply.run_cpu_time_us":47852,"apply.run_wall_time_us":1584224,"cfile_cache_hit":90,"cfile_cache_hit_bytes":96043,"cfile_init":1,"lbm_read_time_us":710,"lbm_reads_lt_1ms":4,"num_ops":22,"prepare.queue_time_us":277208,"prepare.run_cpu_time_us":9911,"prepare.run_wall_time_us":328200,"raft.queue_time_us":256439,"raft.run_cpu_time_us":29732,"raft.run_wall_time_us":499680,"replication_time_us":548477,"spinlock_wait_cycles":6706560,"thread_start_us":716173,"threads_started":5,"wal-append.queue_time_us":566249,"wal-append.run_cpu_time_us":42305,"wal-append.run_wall_time_us":2224107}]]}
I20260709 12:24:48.702956  8781 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.688455  8841 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:48.704427 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 109653us
I20260709 12:24:48.705317 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:48.703832  8858 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.061s	user 0.000s	sys 0.001s
I20260709 12:24:48.691039  8813 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:48.731642 31823 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:48.737874 31822 raft_consensus.cc:2410] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 32 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:24:48.747155  8649 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:48.747774 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.734959  8270 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:24:48.751703 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.759488 31750 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
W20260709 12:24:48.752020 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44234.9us
I20260709 12:24:48.766441 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:48.775491  8841 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 23.17, Last appended by leader: 17, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.785303  8663 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:48.789618 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:48.790913 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 67268.7us
I20260709 12:24:48.792130 31824 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
W20260709 12:24:48.801581 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 48796.7us
I20260709 12:24:48.814648  8541 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:48.815651 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:48.817116 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:48.817909 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:48.821144 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:48.822252 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 128545us
W20260709 12:24:48.822763 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 144462us
I20260709 12:24:48.826169 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:48.828107 32374 raft_consensus.cc:2368] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33.
I20260709 12:24:48.829735  8693 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20260709 12:24:48.834906  8851 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:24:48.793812  8270 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:48.837785  8270 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.840039 31824 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
W20260709 12:24:48.840831 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:48.841955 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.839349 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:48.844329 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:48.845825 31669 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:24:48.846283 31673 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
W20260709 12:24:48.847291 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.847476 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:48.847942 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.848218 31676 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:48.848517 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:48.851002  8726 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:48.851432  8726 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:48.852335  8726 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.853808  8726 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:48.850345 31792 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:48.860069 31822 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
W20260709 12:24:48.861392 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 65277us
I20260709 12:24:48.865548 31675 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:24:48.865837  8663 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:48.866953  8530 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.867074 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:48.868189 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:48.870477 31821 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 31: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:48.867216  8693 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:24:48.873149 31745 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:48.875584  8541 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:48.868937  8663 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:48.880066 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:48.885242 32378 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:24:48.887035  8787 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.887756  8787 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.889127 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:48.890142 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:48.887036 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47199us
W20260709 12:24:48.891945 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 52695.1us
I20260709 12:24:48.867774  8530 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.895121 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:48.893381 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:48.890722 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:48.896131 31746 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:24:48.901012 31753 raft_consensus.cc:2410] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:24:48.901188 31748 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
W20260709 12:24:48.887111  8862 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.093s	user 0.000s	sys 0.001s
I20260709 12:24:48.907294 31677 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:48.883071  8541 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:48.885749  8663 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:48.909502 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:48.882242  8632 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:48.890722  8677 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:48.915886  8749 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 18, Time since last communication: 0.034s
I20260709 12:24:48.917848  8602 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:24:48.913348  8541 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:48.919242  8803 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.010s
I20260709 12:24:48.892104  8787 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:48.923403 31669 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:48.927635 31792 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:48.940824 31749 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 16. (index mismatch)
I20260709 12:24:48.940886 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:48.941668 31751 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:48.939922 31718 leader_election.cc:400] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33.
I20260709 12:24:48.946475 31718 leader_election.cc:403] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Cancelling election due to peer responding with higher term
I20260709 12:24:48.947176 31748 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:24:48.948693 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:48.951565 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:48.942317 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:48.940886  8002 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:48.953130  8002 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:48.953727  8002 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:48.956501 31676 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
W20260709 12:24:48.959957 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 45275.6us
I20260709 12:24:48.940300  8579 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.070s
I20260709 12:24:48.970247 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:48.973366 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:48.976824 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:48.963061  8602 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election lost for term 32. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33.
I20260709 12:24:48.961732  8787 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:24:48.948065  8632 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:48.987576 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 26 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:48.987953 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:48.994406 32376 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 31 because replica has last-logged OpId of term: 28 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 19.
I20260709 12:24:48.995092 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:49.004367 31823 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
W20260709 12:24:49.013037 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.013554 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:49.018865 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:49.012465 31669 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:48.988502  8787 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.020506 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:49.026371  8528 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:48.981277  8807 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:48.979892  8723 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.037s
I20260709 12:24:49.049043 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:49.055451 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:49.058779 31823 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:49.058956 31820 raft_consensus.cc:2368] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33.
I20260709 12:24:49.039559  8787 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:24:49.051606  8863 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 12:24:49.064810 31747 raft_consensus.cc:2368] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27.
I20260709 12:24:49.067788 31750 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:49.050520  8595 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:49.048601  8528 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:49.069209 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 74189.8us
I20260709 12:24:49.075691 31750 raft_consensus.cc:606] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:49.076187 31750 raft_consensus.cc:614] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:49.081967 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:49.078706 32342 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:49.084659  8530 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:49.070112 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.068518  8677 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:49.075086  8528 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:49.085901  8611 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.657s	user 0.010s	sys 0.005s
W20260709 12:24:49.085448  8865 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.092s	user 0.000s	sys 0.000s
I20260709 12:24:49.101545 31820 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:24:49.107641 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 96408.2us
I20260709 12:24:49.099009 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:49.105569 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:49.089815  8595 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:49.108523  8579 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:24:49.096422  8677 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:49.123865 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47102.6us
I20260709 12:24:49.129436 31825 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 15. (term mismatch)
I20260709 12:24:49.129146 31678 pending_rounds.cc:85] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 16
I20260709 12:24:49.135990 31678 pending_rounds.cc:107] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 25.17
I20260709 12:24:49.136404 31678 raft_consensus.cc:2889] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:49.129909 31825 pending_rounds.cc:85] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 14
I20260709 12:24:49.138370 31825 pending_rounds.cc:107] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 23.15
I20260709 12:24:49.138798 31825 raft_consensus.cc:2889] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:49.119823  8874 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:49.121537  8813 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:49.114519  8781 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.026s
I20260709 12:24:49.128829  8677 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:49.135327  8552 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:49.175931  8201 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.084s
I20260709 12:24:49.135672  8579 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 33 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:49.173302  8562 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.165349  8874 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:49.182749  8866 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.055s	user 0.000s	sys 0.001s
W20260709 12:24:49.199062 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.201737 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43200 (request call id 3848) took 1162 ms. Trace:
I20260709 12:24:49.167070  8813 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:49.165761  8851 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:49.202296 31675 rpcz_store.cc:276] 0709 12:24:48.032785 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.033283 (+   498us) service_pool.cc:224] Handling call
0709 12:24:49.194774 (+1161491us) spinlock_profiling.cc:243] Waited 820 ms on lock 0x7b5400391cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:49.194936 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1885089664}
I20260709 12:24:49.206506 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:49.210239 31673 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:49.212337 31820 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:24:49.212634 31745 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:49.215243 31745 raft_consensus.cc:606] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:49.215724 31745 raft_consensus.cc:614] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:24:49.189225  8632 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:49.194584  8552 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:49.231671 31823 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
W20260709 12:24:49.236649 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.210022  8813 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:49.237205 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:49.240990  8579 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:49.244607  8867 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.123s	user 0.000s	sys 0.001s
I20260709 12:24:49.245620 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:49.250850  8807 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:49.244412 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.253903 31822 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:49.241190  8552 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.233690  8595 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.200662  8562 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:49.277779 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:49.283116 31822 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:49.284554  8562 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:49.252581  8637 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.285539  8002 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.270977  8811 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.009s
I20260709 12:24:49.285660  8637 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.288218 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.288517  8637 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:49.278970  8874 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:49.277801 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.235124  8649 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:49.293469 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56377.1us
W20260709 12:24:49.305511 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.306337 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:49.289750  8811 consensus_queue.cc:1243] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 25.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:24:49.309010  8781 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:49.311033  8721 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:24:49.314302 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:49.316272  8590 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:49.318323 31822 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:49.313786  8781 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.319875 31673 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:49.267874  8541 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.332988 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.333848 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.333853 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:49.336177 31827 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:49.343670 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:49.344486 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:49.327236  8799 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.724s	user 0.011s	sys 0.003s
W20260709 12:24:49.353016 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 156479us
I20260709 12:24:49.344628 31672 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:24:49.364483 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.366133 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.328742  8781 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:49.276242  8851 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:49.367291 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.363061 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:49.367930 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.369267 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.372488 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.374567 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.329636  8721 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:49.377170 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:49.380553 31671 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 31 because replica has last-logged OpId of term: 29 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 13.
I20260709 12:24:49.379393 31717 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:49.381314 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.355355  8874 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:49.389048 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.390700 31672 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:49.392446  8637 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.393988 31753 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:49.395041  8649 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.400862 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.401197  8649 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:49.402381 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:49.403342 31824 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
W20260709 12:24:49.405421  8881 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:24:49.411078  8590 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.411859  8590 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:49.413305 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 83139.9us
W20260709 12:24:49.415845  8868 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.288s	user 0.001s	sys 0.001s
I20260709 12:24:49.349274  8663 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.423420 31744 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:24:49.424142 31744 raft_consensus.cc:606] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:49.424353 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.419011 31822 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
W20260709 12:24:49.463944 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49767.5us
I20260709 12:24:49.396919  8703 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:49.396477  8781 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.395068  8637 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:49.501623 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 138647us
I20260709 12:24:49.411883  8425 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:49.502378 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:49.513839 31822 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:49.511931 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:49.508651 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:49.519440 31751 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:49.467381  8807 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.524698  8637 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:49.527807  8632 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:49.513427  8425 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:49.524763  8803 raft_consensus.cc:2764] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election decision vote started in defunct term 26: won
I20260709 12:24:49.532274 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:49.533640 31827 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:24:49.496163  8530 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:24:49.535323 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:49.539865 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:49.665411 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.542713 31746 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 26 index: 20. Preceding OpId from leader: term: 30 index: 22. (index mismatch)
I20260709 12:24:49.514216  8590 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:49.551714 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:49.551931  8002 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:49.552433  8890 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:24:49.553503 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.553397 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:49.555271  8703 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:49.557101  8663 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:49.559262  8611 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:49.559722  8851 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:49.560559  8721 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:49.560624 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:49.562505 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:49.717278 31820 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:49.562831 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:49.717723 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.563915  8749 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:49.564312 31819 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:49.717355 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.722425 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.565730 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:49.510601  8781 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.575842 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.665935  8637 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.578001 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3862) took 1435 ms. Trace:
I20260709 12:24:49.727872 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:49.728204 31825 rpcz_store.cc:276] 0709 12:24:48.143113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.143419 (+   306us) service_pool.cc:224] Handling call
0709 12:24:48.144048 (+   629us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:49.172219 (+1028171us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.577728 (+405509us) spinlock_profiling.cc:243] Waited 976 ms on lock 0x7b5400196ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:49.577935 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2244377984}
I20260709 12:24:49.574339  8804 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:49.579016  8541 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:49.579486 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.580552 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 18 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.580919 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:49.737287 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 183961us
I20260709 12:24:49.581046  8562 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:49.739485 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:49.739823 31752 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:49.593530 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.602145 31744 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28. Ops: [27.16-27.16]
W20260709 12:24:49.605623  8883 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.288s	user 0.001s	sys 0.000s
I20260709 12:24:49.505113  8677 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:49.640201 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.616008 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:49.752672 31824 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:49.616997  8528 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.503s	user 0.011s	sys 0.000s
I20260709 12:24:49.661373  8632 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:49.666925 32377 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:49.710547 32376 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 25 index: 18. (term mismatch)
I20260709 12:24:49.759169 32376 pending_rounds.cc:85] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 17
I20260709 12:24:49.759428 32376 pending_rounds.cc:107] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 24.18
W20260709 12:24:49.672710 31821 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.585s	user 0.015s	sys 0.019s
W20260709 12:24:49.760058 31821 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.673s	user 0.016s	sys 0.019s
I20260709 12:24:49.732254  8649 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:49.676283 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3843) took 1122 ms. Trace:
W20260709 12:24:49.671351 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 119697us
I20260709 12:24:49.665344 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:24:49.717265 31823 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:24:49.718272 31826 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:24:49.723204 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.723409 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.767050 31676 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:49.724645 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:49.731877 31678 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:49.671296  8886 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.711289  8811 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:49.739836  8562 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:49.784363 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:49.785907 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46988.9us
I20260709 12:24:49.787201 31669 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:49.793284 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:49.763557  8632 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:49.740149 32374 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:49.744783 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.739352 31818 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:24:49.750800 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:49.751430 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.756441 31827 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:24:49.787771 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 261691us
I20260709 12:24:49.711735 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:49.788726 31673 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:49.787839 31675 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:49.800232 31822 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
W20260709 12:24:49.690493 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.759658 32376 raft_consensus.cc:2889] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:49.733631  8804 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:49.765548 31642 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:49.764424 31747 rpcz_store.cc:276] 0709 12:24:48.546484 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.571028 (+ 24544us) service_pool.cc:224] Handling call
0709 12:24:49.614967 (+1043939us) spinlock_profiling.cc:243] Waited 479 ms on lock 0x7b54001a4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:49.668909 (+ 53942us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1101498112}
I20260709 12:24:49.750774  8611 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.778434  8590 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:49.676345  8807 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:49.837493 31820 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:49.755514  8749 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:49.781735  8811 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.809273 32378 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
W20260709 12:24:49.672941  8787 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.595s	user 0.012s	sys 0.000s
I20260709 12:24:49.850778 31826 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 13. Preceding OpId from leader: term: 30 index: 14. (index mismatch)
I20260709 12:24:49.839586 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.760763  8528 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:49.802435  8595 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:24:49.851181 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 64190.7us
I20260709 12:24:49.859468 32343 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:49.853459 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:49.863154 31748 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:49.787122  8637 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:49.841181  8879 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:49.838404  8590 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:49.844900  8201 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:49.876792 31678 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 29 index: 14. Preceding OpId from leader: term: 30 index: 14. (term mismatch)
I20260709 12:24:49.879623 31678 pending_rounds.cc:85] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 13
I20260709 12:24:49.880024 31678 pending_rounds.cc:107] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 29.14
I20260709 12:24:49.880369 31678 raft_consensus.cc:2889] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:49.845865  8749 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:49.893288 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.852170  8530 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:24:49.884266  8804 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.885381  8579 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:49.885648 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:24:49.886941 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.922827 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.892204 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.886652 31671 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:49.857935  8787 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:49.926832 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:49.918876  8749 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:49.935968  8803 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:24:49.936408  8611 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:49.936582  8803 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:49.939420 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 80085.4us
I20260709 12:24:49.944684  8528 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:49.945587  8528 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:49.934505  8787 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:49.953716 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 413934us
I20260709 12:24:49.956719 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.866175  8552 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.054s
I20260709 12:24:49.895201 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.897949 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:49.973795 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:49.921247  8890 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:49.898986 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:49.974924  8602 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:49.975272  8890 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:49.975661  8602 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:49.899616 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:49.903183 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.976961  8602 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:49.900149 31791 connection.cc:256] RPC call timeout handler was delayed by -1.3209939919997851s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15710000000000002s for a response
W20260709 12:24:49.978238 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0914939919998687s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1826s for a response
I20260709 12:24:49.904810  8781 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:49.911056 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 136191us
I20260709 12:24:49.957424 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:49.979031 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:49.911890  8595 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:49.979557  8804 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:49.979918  8595 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.980597 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:49.981920  8002 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:49.982121  8595 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:49.912640 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:49.859218  8649 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 12:24:49.982837  8663 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:49.984555  8637 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:49.985049  8663 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:49.986982 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:49.866101  8703 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 25.12, Last appended by leader: 12, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:49.915969  8813 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.639s	user 0.013s	sys 0.000s
I20260709 12:24:49.916435 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:49.852576  8811 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:49.892280 32374 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:24:49.801283  8425 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.794739  8886 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.881275  8201 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.958911  8579 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:49.959669 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:49.882992  8590 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:49.937875  8803 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:49.960542  8893 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.100s	user 0.000s	sys 0.001s
I20260709 12:24:49.956516 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3683) took 1042 ms. Trace:
I20260709 12:24:49.967445  8677 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:49.968763  8807 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:49.966573 31748 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 26 index: 19. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:24:49.990603  8813 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:50.011102 31673 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:50.012719 31824 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:49.971691 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:49.972046  8895 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.097s	user 0.001s	sys 0.000s
I20260709 12:24:49.972971 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:49.975754  8890 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:49.995357 31676 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 17. (index mismatch)
I20260709 12:24:49.970826 31749 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:24:49.983179  8783 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.006s
I20260709 12:24:49.877238  8562 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:49.907152 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:49.979604 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:49.982455  8002 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:49.998808 31675 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:24:49.987882  8874 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.597s	user 0.011s	sys 0.000s
W20260709 12:24:50.023664 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50787.3us
I20260709 12:24:50.025139 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:50.005388 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:50.006525 31671 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 25 index: 18. (index mismatch)
W20260709 12:24:49.956563 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 95192.6us
W20260709 12:24:50.002421 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.035264 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1345) took 1033 ms. Trace:
I20260709 12:24:49.993645  8425 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:24:49.998154  8897 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:24:49.998484  8579 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 25.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:50.055583 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 68683.8us
I20260709 12:24:49.997109  8886 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:50.057663  8902 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 18, Time since last communication: 0.007s
I20260709 12:24:50.007404 31821 rpcz_store.cc:276] 0709 12:24:48.924510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.924706 (+   196us) service_pool.cc:224] Handling call
0709 12:24:48.936670 (+ 11964us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:48.960175 (+ 23505us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:49.064515 (+104340us) log.cc:844] Serialized 14 byte log entry
0709 12:24:49.760459 (+695944us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:49.789403 (+ 28944us) log.cc:844] Serialized 26 byte log entry
0709 12:24:49.795657 (+  6254us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:49.795718 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.956131 (+160413us) spinlock_profiling.cc:243] Waited 5.95 ms on lock 0x7b54001c89d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:49.956177 (+    46us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:49.956222 (+    45us) raft_consensus.cc:1685] finished
0709 12:24:49.956315 (+    93us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:49.956452 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10593,"prepare.queue_time_us":160943,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":3081,"raft.queue_time_us":16018,"raft.run_cpu_time_us":455,"raft.run_wall_time_us":33538,"spinlock_wait_cycles":13696384,"thread_start_us":767346,"threads_started":2,"wal-append.queue_time_us":777411}
I20260709 12:24:50.022775  8002 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:50.020174  8813 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:50.060226 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:24:50.091322 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 111841us
I20260709 12:24:50.091993  8270 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:50.007983  8677 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:50.051615  8781 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:49.997380  8201 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:24:50.100035 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 238204us
I20260709 12:24:50.005438  8803 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:50.043294 32371 rpcz_store.cc:276] 0709 12:24:48.986577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.986741 (+   164us) service_pool.cc:224] Handling call
0709 12:24:48.987151 (+   410us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:49.004110 (+ 16959us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:49.004145 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:49.020173 (+ 16028us) log.cc:844] Serialized 26 byte log entry
0709 12:24:49.311266 (+291093us) spinlock_profiling.cc:243] Waited 13.5 ms on lock 0x7b180047b360. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:24:49.665693 (+354427us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:49.665733 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:49.665804 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:49.959787 (+293983us) spinlock_profiling.cc:243] Waited 16.3 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:50.014713 (+ 54926us) raft_consensus.cc:1685] finished
0709 12:24:50.014748 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:50.019428 (+  4680us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":881,"raft.queue_time_us":1282,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":26340,"spinlock_wait_cycles":68515968,"thread_start_us":354232,"threads_started":1,"wal-append.queue_time_us":754464}
I20260709 12:24:50.060997 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:50.039964  8528 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.065s
I20260709 12:24:50.034991  8595 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:50.067591  8611 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:50.061721  8562 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:50.003839  8749 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:50.105494 31675 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 33 because replica has last-logged OpId of term: 31 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 18.
I20260709 12:24:50.051564  8552 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.057s
I20260709 12:24:50.157533 31672 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33. Ops: []
I20260709 12:24:50.004590  8632 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:49.999827  8896 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.107s
I20260709 12:24:50.067045  8901 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.083s
I20260709 12:24:50.066758  8807 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.042s
I20260709 12:24:50.169111 31674 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:50.167311  8896 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:50.170264  8896 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:50.171568  8896 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:50.173317  8896 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:50.174269  8901 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:50.174836  8901 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:50.175006  8632 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:50.175232  8901 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:50.176469  8552 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:50.177031  8552 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:50.182055  8749 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:50.158833  8663 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.184194  8663 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:50.186311  8663 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:50.160084  8595 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:50.212960  8595 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:50.150559  8813 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:50.213626  8813 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:50.214042  8813 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:50.099884  8787 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.058s
I20260709 12:24:50.099673  8886 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.059s
W20260709 12:24:50.228758 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:50.187026 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 126910us
I20260709 12:24:50.115278 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:50.235062 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.188092  8552 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:50.188148 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.196455  8811 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:50.199018 31642 leader_election.cc:400] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27.
W20260709 12:24:50.202888 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 100762us
W20260709 12:24:50.247411 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 222355us
I20260709 12:24:50.248651 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:50.106979  8879 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.127s
I20260709 12:24:50.107502  8781 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:50.108819  8677 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:50.254324 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 55451.9us
I20260709 12:24:50.260298 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:50.261233 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:50.253571 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:50.103371  8874 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.143s
I20260709 12:24:50.262583 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:50.262971 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.264374 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:50.273573 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:50.251281  8663 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:50.274915 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:50.255533  8552 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:24:50.286504 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.287889 31749 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:50.256392  8781 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:50.290242 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 181007us
I20260709 12:24:50.272956  8595 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:50.243112  8721 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.028s
I20260709 12:24:50.295634 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:50.297783 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 258243us
I20260709 12:24:50.245640  8905 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:24:50.300431 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:50.302778 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1348) took 1114 ms. Trace:
I20260709 12:24:50.303989 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:50.302842 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:50.307103 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:50.304307 31750 rpcz_store.cc:276] 0709 12:24:49.181207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.181391 (+   184us) service_pool.cc:224] Handling call
0709 12:24:49.181797 (+   406us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:49.214925 (+ 33128us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:49.214975 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:49.235260 (+ 20285us) log.cc:844] Serialized 52 byte log entry
0709 12:24:49.577184 (+341924us) spinlock_profiling.cc:243] Waited 975 us on lock 0x7b18002f9da0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:24:49.919382 (+342198us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:50.255773 (+336391us) log.cc:844] Serialized 14 byte log entry
0709 12:24:50.263733 (+  7960us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.263853 (+   120us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:50.263920 (+    67us) raft_consensus.cc:1685] finished
0709 12:24:50.273245 (+  9325us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:50.295460 (+ 22215us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":40837,"raft.queue_time_us":62618,"raft.run_cpu_time_us":370,"raft.run_wall_time_us":43422,"spinlock_wait_cycles":2242432,"thread_start_us":641043,"threads_started":2,"wal-append.queue_time_us":727977}
I20260709 12:24:50.291095  8552 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:50.343978 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:50.363958 32371 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:50.345701 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:50.365161 31826 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:50.318753 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:50.320416 31748 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:50.300812  8841 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:50.358006 31824 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:50.290320  8804 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.031s
I20260709 12:24:50.390028 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:50.386126 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:50.302158  8595 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:50.350983  8552 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:50.297837  8902 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:50.407043 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:50.410235 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 95233.2us
W20260709 12:24:50.419106 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 175918us
I20260709 12:24:50.409163  8595 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:50.422804 31826 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:24:50.424064 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:50.412056  8902 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:50.422523  8804 consensus_queue.cc:1243] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 31.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:24:50.453299  8632 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:50.461611 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.460225  8889 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:50.463725  8889 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:50.445930  8902 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:50.471167  8908 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.112s	user 0.000s	sys 0.000s
I20260709 12:24:50.422173  8425 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:50.488332 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:50.439493  8637 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:24:50.492321  8677 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:50.424616  8841 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.022s
I20260709 12:24:50.496001  8916 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:50.496099 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 72187.2us
I20260709 12:24:50.446902  8879 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:50.504056  8425 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:50.485760  8663 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:50.526757 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 140781us
I20260709 12:24:50.528920 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:50.529363 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:50.484376  8528 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:50.538218 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:50.548521 31675 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:50.565958 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:50.566764 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 19 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:50.566890 32374 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:50.567660 31822 raft_consensus.cc:2410] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 23 because replica has last-logged OpId of term: 21 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 19 index: 13.
I20260709 12:24:50.569070 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:50.571128 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:50.572595  8425 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:50.573763 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 265734us
I20260709 12:24:50.569764 32376 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:50.574533 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:50.575205 31820 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:24:50.575412 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:24:50.577965 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 56456.5us
I20260709 12:24:50.576104 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:50.589534 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:50.590981 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:50.587391 31826 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:50.531046  8919 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:24:50.598554 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:50.596472 31677 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:24:50.599445 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:50.564080  8528 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:50.487183  8590 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:50.597456 31827 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:50.598340 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:50.600219 31820 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:50.589534 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:50.603071 31818 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:50.598798  8918 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:50.615633  8918 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:50.616397 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:50.618747  8879 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:50.619930  8425 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:50.620348  8425 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:50.621279  8425 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:50.624313 32373 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: [29.21-29.21]
I20260709 12:24:50.625839  8425 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:50.627054  8918 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:50.632578  8901 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:50.634091 32376 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:24:50.526302  8902 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:50.637295 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:50.638476  8637 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:50.638800  8902 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:50.639568 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:50.640483  8002 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.536s	user 0.011s	sys 0.000s
I20260709 12:24:50.640801 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:50.640923  8002 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:50.641449 31819 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:24:50.644025 32371 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:24:50.646322 32374 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 33: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:50.615669 31827 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:50.642396 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:50.678860 31819 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:50.658843 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:50.684113 31826 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:50.655349 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:50.608546  8528 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:50.666646 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:50.691704 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:50.642864  8905 raft_consensus.cc:2764] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election decision vote started in defunct term 24: won
I20260709 12:24:50.573346  8663 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:50.660418 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:50.531322  8916 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:50.697875 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:50.703715 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48454.2us
I20260709 12:24:50.704833 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:50.687691 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 297753us
I20260709 12:24:50.691974 31820 raft_consensus.cc:2368] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29.
I20260709 12:24:50.673362 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:50.675541  8921 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.726182  8921 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:50.728627  8921 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:50.695165  8923 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:50.729619 31819 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:50.729884  8923 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:50.732937  8923 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:50.699060  8649 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.035s
I20260709 12:24:50.734763  8649 consensus_queue.cc:1243] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 30.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 19
W20260709 12:24:50.736282  8201 consensus_meta.cc:287] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.577s	user 0.009s	sys 0.001s
I20260709 12:24:50.736901  8201 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:50.737282  8552 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:50.742424  8916 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:50.702854  8562 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:50.743335  8562 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:50.743736  8562 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:50.744767  8703 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:24:50.745216  8703 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:50.748099 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:50.749019 31824 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 25: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:50.749888 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:50.751904 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:50.754135  8912 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:24:50.706375 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:50.715091  8813 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.501s	user 0.004s	sys 0.007s
I20260709 12:24:50.715345  8590 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:50.715696  8783 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:24:50.723057  8632 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:50.724362  8901 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:50.750588 31824 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 31: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:50.756418  8663 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:50.757561 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52863us
I20260709 12:24:50.804096 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:50.759234  8677 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:24:50.762106  8611 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.601s	user 0.010s	sys 0.000s
I20260709 12:24:50.761924 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:50.746121  8703 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:50.764111  8890 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.674s	user 0.010s	sys 0.001s
W20260709 12:24:50.729578  8803 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.620s	user 0.002s	sys 0.006s
I20260709 12:24:50.757679  8889 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:50.708011 31827 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:50.770551 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.825762 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:50.831341 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.831472 31746 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
W20260709 12:24:50.770656 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 110346us
I20260709 12:24:50.779680  8637 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:50.832759 31718 leader_election.cc:400] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33.
I20260709 12:24:50.777472  8201 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:50.781488 31820 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: [26.16-26.16]
W20260709 12:24:50.794116 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 87817us
I20260709 12:24:50.844775 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:50.845013 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:50.758543  8922 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:24:50.813179 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.816529 31675 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33. Ops: [32.19-32.19]
I20260709 12:24:50.811616  8907 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:50.822315 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:50.776135  8552 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:50.825663 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:50.829233 31752 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:50.764147  8811 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:50.833391 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.781126 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:50.788112  8783 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:50.847759 31744 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: [28.20-29.21]
I20260709 12:24:50.848574 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:50.850088  8922 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:50.822458  8602 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:50.825472  8611 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election lost for term 23. Reason: could not achieve majority
I20260709 12:24:50.864393 31826 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:50.850292  8703 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:24:50.793766  8813 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:50.899366 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:50.832338  8890 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:50.900463 31744 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:50.872753  8783 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:24:50.869645 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:50.883744 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:50.913437 31749 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:50.858747  8907 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:50.921298  8907 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:50.923540  8002 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:50.860002  8595 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:50.888052  8781 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.585s	user 0.011s	sys 0.001s
I20260709 12:24:50.874897  8632 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:50.889511  8851 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:50.893078  8879 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:24:50.867180  8803 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:50.875023  8889 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:50.880450  8922 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:50.996475 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:50.897496  8602 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:50.907099  8787 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:50.967794 31669 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:50.935688 31792 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:51.095512 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:51.096035 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 160462us
I20260709 12:24:51.096374 31826 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:24:51.031982 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 162431us
W20260709 12:24:51.033389 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 153212us
I20260709 12:24:51.031620 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:50.897548  8637 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:51.036123 31670 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: []
I20260709 12:24:51.053485 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:50.817620 31671 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
W20260709 12:24:51.102941 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 161666us
I20260709 12:24:50.849625 31818 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
W20260709 12:24:51.068703 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 264623us
I20260709 12:24:51.042939  8677 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:51.077106  8781 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.078176  8851 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:51.098362 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:51.104408 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:51.048086  8920 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
I20260709 12:24:51.102352  8611 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election won for term 29
W20260709 12:24:51.155606  8933 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.101s	user 0.000s	sys 0.000s
I20260709 12:24:51.157990  8781 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:51.157446 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:51.158571 32372 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:51.159732  8595 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:51.160312 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 56038.3us
W20260709 12:24:51.161351  8925 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
I20260709 12:24:51.118854  8530 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:51.100100  8803 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:51.242338 31827 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:51.243673 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:51.130923 31643 leader_election.cc:400] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29.
I20260709 12:24:51.085877  8528 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.171068 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:51.119084  8934 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:51.124693  8002 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.216341  8611 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:51.124095  8637 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.170964  8918 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:51.186830 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:51.188284  8920 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:51.193032  8562 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.195101  8851 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.195757  8874 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:51.153545  8886 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:51.190490  8890 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:51.153642  8902 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.049s
I20260709 12:24:51.202983  8602 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:51.212898 31746 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
W20260709 12:24:51.232071 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 113676us
I20260709 12:24:51.244906 31643 leader_election.cc:403] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Cancelling election due to peer responding with higher term
I20260709 12:24:51.257493 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:51.257704 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:51.096227  8879 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:51.263129 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:51.271500 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 104188us
I20260709 12:24:51.298861 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:51.278981 31826 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:24:51.252198  8595 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:51.260185  8934 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.259795  8907 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.300669 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:51.300951 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 173835us
W20260709 12:24:51.302434 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44843us
I20260709 12:24:51.315456  8943 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:51.316934  8528 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:51.317245 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:51.318010 31827 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:51.251895  8950 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.058s
I20260709 12:24:51.272320  8813 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:51.309118 31820 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:24:51.294200 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:51.322381 31823 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:51.336858 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:51.337574 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:51.341861 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:51.345549 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:51.345875 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:51.347189 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:51.347887 31823 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:51.348206 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:51.348868 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54812.4us
I20260709 12:24:51.349602 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:51.349676 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:51.350246 31826 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:24:51.351177 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:51.302384  8579 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:51.302007  8602 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.305323  8890 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:51.353215 31826 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:51.308917  8934 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:51.304783  8874 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:51.280213  8918 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.309098  8595 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.308427  8851 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:24:51.361285 31791 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.286535  8002 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.362246 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:51.351904  8919 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:24:51.355698  8811 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:24:51.369498 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1636) took 1073 ms. Trace:
I20260709 12:24:51.354279  8804 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:51.355798  8781 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:51.389390  8910 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:51.389892  8910 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:51.390378  8910 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:51.394987  8530 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:51.395586  8530 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:51.396025  8002 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:51.399326  8562 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:51.399948  8947 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:24:51.280049  8920 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.415503  8950 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:24:51.416077  8950 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:51.416488  8950 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:24:51.416553  8703 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.515s	user 0.008s	sys 0.002s
I20260709 12:24:51.353652  8930 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 22.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.064s
I20260709 12:24:51.419562 32377 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:51.394975 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1634) took 1101 ms. Trace:
W20260709 12:24:51.427774 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:51.448959 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 77842.9us
I20260709 12:24:51.421141 31821 rpcz_store.cc:276] 0709 12:24:50.293509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.293684 (+   175us) service_pool.cc:224] Handling call
0709 12:24:50.310941 (+ 17257us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:50.601700 (+290759us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:50.601734 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:50.664629 (+ 62895us) log.cc:844] Serialized 26 byte log entry
0709 12:24:51.034663 (+370034us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:51.034727 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.036074 (+  1347us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.394708 (+358634us) raft_consensus.cc:1685] finished
0709 12:24:51.394732 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.394902 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38900,"raft.queue_time_us":61248,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":6203,"thread_start_us":530993,"threads_started":2,"wal-append.queue_time_us":593704}
I20260709 12:24:51.379454 31823 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:24:51.374117 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:51.380954 31825 rpcz_store.cc:276] 0709 12:24:50.295933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.297079 (+  1146us) service_pool.cc:224] Handling call
0709 12:24:50.311690 (+ 14611us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:50.387324 (+ 75634us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:50.387373 (+    49us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:50.405354 (+ 17981us) log.cc:844] Serialized 26 byte log entry
0709 12:24:50.666579 (+261225us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:50.666621 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.666715 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.369229 (+702514us) raft_consensus.cc:1685] finished
0709 12:24:51.369266 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.369424 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13745,"raft.queue_time_us":65009,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":13214,"thread_start_us":224058,"threads_started":1,"wal-append.queue_time_us":441032}
I20260709 12:24:51.386153 31827 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:24:51.474977 31820 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:51.475889 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:51.476495 31827 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:51.418772  8595 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:51.388674  8804 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:51.341603  8611 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.477659 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:51.490679 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 145203us
I20260709 12:24:51.498973 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:51.419663  8902 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:51.363335  8602 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.490545  8811 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:51.503505  8907 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:51.360914  8803 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.497014  8804 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:51.506268  8890 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election lost for term 28. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29.
I20260709 12:24:51.506876  8943 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:51.507848 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 57962.4us
I20260709 12:24:51.509261  8924 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:24:51.515568  8964 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:24:51.519249  8920 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:51.506323  8632 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:51.527372  8787 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:51.527877 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:51.527029  8637 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:51.527724  8874 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.16, Last appended by leader: 16, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:51.530871  8917 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.955s	user 0.002s	sys 0.004s
I20260709 12:24:51.474395  8781 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:51.484364  8948 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:51.535746  8902 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:51.539508 31827 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:24:51.384408  8944 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.152s
W20260709 12:24:51.568265 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 69440.6us
W20260709 12:24:51.571837 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 222229us
I20260709 12:24:51.605509 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:51.581382  8579 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:51.559808 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:51.639675 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:51.558689  8562 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.642670 31820 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:51.562691  8907 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.405921  8879 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.584862  8787 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:51.582592  8949 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:24:51.652989 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47573.4us
I20260709 12:24:51.587793  8948 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:51.644137  8902 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:51.683344  8940 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:51.684754  8945 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:51.678018  8948 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.685451  8945 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:51.686129  8945 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:51.583846  8950 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.671669  8579 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.663905  8595 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:51.648185  8721 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.028s
I20260709 12:24:51.664426  8923 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:51.670225  8886 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.671558 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3891) took 1434 ms. Trace:
I20260709 12:24:51.544158  8811 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.656122  8632 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:51.702577 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50167.1us
I20260709 12:24:51.588835  8919 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:51.568661  8943 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:51.762898 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:51.764194 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.772403 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:51.775786 31669 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260709 12:24:51.781009 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:51.781548 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.782308 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:51.655115  8907 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:51.710263  8964 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.061s	user 0.001s	sys 0.000s
I20260709 12:24:51.761739 31674 rpcz_store.cc:276] 0709 12:24:50.236908 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.237441 (+   533us) service_pool.cc:224] Handling call
0709 12:24:50.252476 (+ 15035us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:50.252950 (+   474us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:50.252994 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:50.254141 (+  1147us) log.cc:844] Serialized 26 byte log entry
0709 12:24:50.443277 (+189136us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:50.443315 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:50.443395 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.525721 (+1082326us) spinlock_profiling.cc:243] Waited 5.06 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:51.655209 (+129488us) raft_consensus.cc:1685] finished
0709 12:24:51.655236 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.671452 (+ 16216us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":745,"raft.queue_time_us":846,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":9380,"spinlock_wait_cycles":11645696,"thread_start_us":142990,"threads_started":1,"wal-append.queue_time_us":244783}
I20260709 12:24:51.686537  8948 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:51.784916 31823 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:24:51.785130 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:51.795257 31677 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:51.784948 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 225239us
I20260709 12:24:51.795473 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:51.796130 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:51.715344  8783 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.785s	user 0.007s	sys 0.006s
I20260709 12:24:51.668691  8949 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:51.789572 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:51.791510 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:51.792384 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3885) took 1828 ms. Trace:
I20260709 12:24:51.799357 31675 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:24:51.803577 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:51.804227 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:51.656479  8562 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.808032 31750 raft_consensus.cc:2410] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:24:51.811599 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:51.813249 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:51.814180 31751 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:51.712726  8693 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.088s
I20260709 12:24:51.800943 31670 rpcz_store.cc:276] 0709 12:24:49.963614 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:49.965211 (+  1597us) service_pool.cc:224] Handling call
0709 12:24:49.965567 (+   356us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:51.103716 (+1138149us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.792063 (+688347us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b540036f1d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:51.792291 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2336425344}
I20260709 12:24:51.803397 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:51.820667 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:51.822129 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:51.815972 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:51.827230 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:51.835310  8966 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:51.835940  8968 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:51.836565  8968 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:51.837469  8959 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
I20260709 12:24:51.801044  8971 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.839277 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3879) took 1240 ms. Trace:
I20260709 12:24:51.839344  8971 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.840709  8907 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:51.839825 31824 rpcz_store.cc:276] 0709 12:24:50.599321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.797102 (+197781us) service_pool.cc:224] Handling call
0709 12:24:50.797394 (+   292us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:50.839538 (+ 42144us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:24:50.839572 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:50.847983 (+  8411us) log.cc:844] Serialized 26 byte log entry
0709 12:24:51.530308 (+682325us) spinlock_profiling.cc:243] Waited 942 us on lock 0x7b18003d90c0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:24:51.581872 (+ 51564us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:51.581929 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.582029 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.821548 (+239519us) raft_consensus.cc:1685] finished
0709 12:24:51.821573 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.839190 (+ 17617us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":36844,"raft.run_cpu_time_us":136,"raft.run_wall_time_us":9295,"spinlock_wait_cycles":2166144,"thread_start_us":51305,"threads_started":1,"wal-append.queue_time_us":795444}
I20260709 12:24:51.841370  8907 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:51.841693  8971 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:51.843333 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:51.843868  8950 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:51.845086 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:51.845638 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:51.846578 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3888) took 1221 ms. Trace:
I20260709 12:24:51.847188  8425 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:51.838713  8944 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:51.847043 31818 rpcz_store.cc:276] 0709 12:24:50.625392 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.137978 (+512586us) service_pool.cc:224] Handling call
0709 12:24:51.218505 (+ 80527us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:51.233830 (+ 15325us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:51.233872 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:51.234478 (+   606us) log.cc:844] Serialized 26 byte log entry
0709 12:24:51.496805 (+262327us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:51.497426 (+   621us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.497499 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.846341 (+348842us) raft_consensus.cc:1685] finished
0709 12:24:51.846363 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.846516 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":32602,"raft.run_cpu_time_us":243,"raft.run_wall_time_us":16682,"thread_start_us":233743,"threads_started":1,"wal-append.queue_time_us":520779}
W20260709 12:24:51.835575  8963 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:24:51.847043 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:51.850716 31670 raft_consensus.cc:2368] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28.
I20260709 12:24:51.851543 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:51.851965 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:51.853176 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:51.853507 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.853899 31745 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:51.857089  8968 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:51.842532  8971 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:24:51.858453 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:51.859429 31818 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 29: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:51.859359 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:51.860648 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:51.861189 31750 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:51.867149 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:51.869103 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:51.870657  8813 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:51.871165  8813 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:51.871585  8922 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.769s	user 0.011s	sys 0.000s
I20260709 12:24:51.872049  8813 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.872133  8922 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.873690 31822 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:51.877364 31749 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:24:51.882982 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 54087.2us
W20260709 12:24:51.850593  8961 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.295s	user 0.001s	sys 0.000s
I20260709 12:24:51.887634 31749 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 28 index: 21. (index mismatch)
I20260709 12:24:51.888995 31745 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:51.776654  8886 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.841987  8907 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.784687  8632 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.894490 31750 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:24:51.802819  8949 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:51.785434  8923 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:51.897004  8889 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.800s	user 0.011s	sys 0.000s
I20260709 12:24:51.923056 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:51.924845 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3894) took 1143 ms. Trace:
I20260709 12:24:51.804649  8783 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:51.932782 31671 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 12. Preceding OpId from leader: term: 27 index: 13. (index mismatch)
I20260709 12:24:51.937551 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:51.815135  8530 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.941870 31676 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 30 index: 21. (term mismatch)
I20260709 12:24:51.942416 31676 pending_rounds.cc:85] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 20
I20260709 12:24:51.943038 31676 pending_rounds.cc:107] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 29.21
I20260709 12:24:51.943418 31676 raft_consensus.cc:2889] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:51.919166  8201 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:51.925482 31672 rpcz_store.cc:276] 0709 12:24:50.780993 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.781698 (+   705us) service_pool.cc:224] Handling call
0709 12:24:50.783476 (+  1778us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:50.783966 (+   490us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:24:51.252345 (+468379us) log.cc:844] Serialized 14 byte log entry
0709 12:24:51.362044 (+109699us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:51.404237 (+ 42193us) log.cc:844] Serialized 26 byte log entry
0709 12:24:51.404493 (+   256us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:51.404548 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.404653 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.924538 (+519885us) raft_consensus.cc:1685] finished
0709 12:24:51.924573 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:51.924760 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13229,"prepare.queue_time_us":529337,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":49,"raft.queue_time_us":23699,"raft.run_cpu_time_us":314,"raft.run_wall_time_us":18290,"thread_start_us":510569,"threads_started":2,"wal-append.queue_time_us":128775}
W20260709 12:24:51.945855 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:51.952536 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:51.818142  8934 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.895792  8930 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.040s
I20260709 12:24:51.806349  8781 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.941239 31670 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
W20260709 12:24:51.963585 32377 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.513s	user 0.008s	sys 0.004s
I20260709 12:24:51.845178  8910 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:51.934652  8944 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:51.975150 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:51.967062 32377 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:51.941581  8918 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:51.947886  8201 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:51.955657  8896 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:51.946226  8602 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:51.981427 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:51.932452  8948 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.040s
W20260709 12:24:51.970297 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 178879us
I20260709 12:24:51.928550  8693 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.041s
I20260709 12:24:51.999475 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:51.917101  8966 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.052s
I20260709 12:24:51.938972  8851 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.042s
I20260709 12:24:52.002579 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:51.998955 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 114804us
W20260709 12:24:52.008545 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 68716.9us
I20260709 12:24:52.013761 31672 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 32 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
I20260709 12:24:52.025112 31669 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 28: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:52.022814 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:52.026448 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:52.000684  8943 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:52.000434  8595 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:52.031620 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:52.033280 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:51.937762  8919 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:52.041765 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:52.045039 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:52.055832 32378 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33. Ops: []
I20260709 12:24:52.058288 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:52.046350  8813 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:52.060479  8813 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:52.062613 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:52.065294  8552 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:52.062947  8813 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:52.068044 32342 leader_election.cc:400] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28.
I20260709 12:24:52.068992 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:52.068810  8201 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 24.15, Last appended by leader: 15, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:52.069160 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:52.069738 32374 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 25: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:52.069929  8201 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:52.069602  8945 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.071740 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:52.072733 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:52.075843 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:52.082046 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:51.980672  8944 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.027539  8896 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:52.084921 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:52.073957  8952 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:52.081041  8930 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:52.096422 31752 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:52.097622 31669 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:24:52.096227 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:52.083782  8920 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:52.037810  8968 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.053s
W20260709 12:24:52.095628  8804 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.547s	user 0.012s	sys 0.001s
I20260709 12:24:52.100143  8803 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:24:52.106222  8783 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:52.112003 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:52.125700 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:52.126423 31669 raft_consensus.cc:2393] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:52.155650 31751 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:24:52.156454 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:52.114743  8552 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Leader pre-election won for term 20
I20260709 12:24:52.104038  8922 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:52.114979  8562 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:52.161783 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:52.114770  8984 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:24:52.127871  8987 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:52.124665  8920 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:52.171337  8987 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:52.172138  8920 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.172314  8987 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:52.173635  8987 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:52.122694  8952 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:52.175642  8952 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:52.187333 31677 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:52.159826  8950 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.188865 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:52.118506  8804 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:24:52.191865  8985 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.095s	user 0.001s	sys 0.000s
W20260709 12:24:52.192562  8978 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.177s	user 0.001s	sys 0.000s
I20260709 12:24:52.197813 31674 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:52.198613  8986 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:52.169858  8923 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:52.200927  8787 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:52.179831  8984 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:52.164969  8595 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:52.070879  8889 consensus_queue.cc:1175] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 25 status { last_received { term: 22 index: 14 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.15-24.15]" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:24:52.208984  8981 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
W20260709 12:24:52.209995  8989 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.085s	user 0.000s	sys 0.000s
I20260709 12:24:52.219558  8944 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:52.220158 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.222599 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:52.226475  8945 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:52.170162  8896 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:52.191053  8934 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:52.163756  8552 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:52.232102  8924 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:52.234221  8002 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:24:52.234400 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 438320us
I20260709 12:24:52.237160  8910 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:52.238865  8943 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.242673  8968 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:52.247869 31821 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 31 index: 19. (index mismatch)
I20260709 12:24:52.249027 31823 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: [31.18-31.18]
I20260709 12:24:52.249768 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:52.250746 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:52.253911  8902 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.575s	user 0.012s	sys 0.000s
I20260709 12:24:52.256621 31824 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 12. Preceding OpId from leader: term: 27 index: 13. (index mismatch)
I20260709 12:24:52.390710 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:52.257879  8590 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.259862 31820 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:52.402458 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:52.403232 31820 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:52.261075 31827 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260709 12:24:52.276122 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 321727us
I20260709 12:24:52.282936 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:52.407583 31677 raft_consensus.cc:2410] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 29 because replica has last-logged OpId of term: 27 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:24:52.305524 31750 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:52.240515  8940 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:52.321528  8787 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:52.321872  8804 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:52.293593  8950 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.303709  8890 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:52.324484  8984 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.285916  8949 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.308589  8986 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:52.337733 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 210681us
I20260709 12:24:52.340945  8552 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:52.375485 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:52.452306 31821 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:52.348572  8943 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.454802  8943 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:52.348706 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:52.382604 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:52.384197 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:52.385203  8993 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.207s	user 0.000s	sys 0.001s
I20260709 12:24:52.385891 31672 raft_consensus.cc:2410] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 25 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:24:52.386453  8942 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:24:52.347460  8945 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.387408  8813 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.408689 32378 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: []
I20260709 12:24:52.389179  8874 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:52.444180 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.443398 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:52.572280 31827 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:52.445634 31820 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:52.575685 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:52.443142 31824 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:24:52.447793 32375 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31. Ops: []
W20260709 12:24:52.439411  8749 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.656s	user 0.010s	sys 0.007s
I20260709 12:24:52.425190  8940 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:52.456701 31819 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:24:52.447691  8803 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:52.439930  8924 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:52.444916  8851 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:52.608547  8992 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:24:52.609061  8851 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.608991  8924 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:52.611493  8975 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.828s	user 0.003s	sys 0.000s
I20260709 12:24:52.612109  8851 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:52.612623  8924 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:52.615500  8874 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.560938 31821 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:24:52.623658  8907 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:52.623819  8949 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:52.624146  8890 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:52.625113 31823 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:24:52.628233 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57870.4us
I20260709 12:24:52.629719 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:52.564728  8945 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:52.565739 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 1508) took 1218 ms. Trace:
I20260709 12:24:52.408699  8590 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:52.445199  8944 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:52.404385  8902 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:52.567226  8903 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:52.587210  9004 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:52.460928  8942 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:52.592815  8915 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.017s
I20260709 12:24:52.604187 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1354) took 1759 ms. Trace:
I20260709 12:24:52.632280 32377 rpcz_store.cc:276] 0709 12:24:51.347526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.347973 (+   447us) service_pool.cc:224] Handling call
0709 12:24:52.565499 (+1217526us) spinlock_profiling.cc:243] Waited 70.7 ms on lock 0x7b54004cfed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:52.565638 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":162650368}
I20260709 12:24:52.642058 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:52.643754 31820 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
W20260709 12:24:52.646165 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 196542us
I20260709 12:24:52.662974 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 1498) took 2111 ms. Trace:
I20260709 12:24:52.622378  8813 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:52.667589 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 318925us
I20260709 12:24:52.570497  8943 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:24:52.624758  8749 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.842s	user 0.011s	sys 0.007s
I20260709 12:24:52.631011  8992 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:52.570837  8804 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:52.673143 31748 rpcz_store.cc:276] 0709 12:24:50.844289 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.845603 (+  1314us) service_pool.cc:224] Handling call
0709 12:24:50.873543 (+ 27940us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:51.075740 (+202197us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:51.075914 (+   174us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:51.100514 (+ 24600us) log.cc:844] Serialized 52 byte log entry
0709 12:24:51.270148 (+169634us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:51.270194 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.270277 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:51.849885 (+579608us) spinlock_profiling.cc:243] Waited 4.99 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:51.982357 (+132472us) raft_consensus.cc:1685] finished
0709 12:24:51.982394 (+    37us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:52.603905 (+621511us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x7b54004042d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:52.604085 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":18726,"raft.queue_time_us":197874,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":202,"spinlock_wait_cycles":74853504,"thread_start_us":235127,"threads_started":2,"wal-append.queue_time_us":255466}
I20260709 12:24:52.676275 32372 rpcz_store.cc:276] 0709 12:24:50.528352 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.528602 (+   250us) service_pool.cc:224] Handling call
0709 12:24:52.639479 (+2110877us) spinlock_profiling.cc:243] Waited 629 ms on lock 0x7b540071d4d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:52.639674 (+   195us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1445996288}
I20260709 12:24:52.688098 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:52.689893 32376 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:52.710197 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:52.624689  8890 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:52.691298 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:52.691491 31827 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:24:52.711855 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:52.672583 31819 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: []
I20260709 12:24:52.712807 31824 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31. Ops: [30.14-30.14]
I20260709 12:24:52.701112 31823 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:24:52.715154 32372 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
W20260709 12:24:52.722762 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:52.645056  8693 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.062s
I20260709 12:24:52.713305 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:52.690063  8896 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:52.710522  8924 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:52.797035 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:52.758220  8992 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.758999  8590 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:52.755151  8879 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election won for term 22
W20260709 12:24:52.762893 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74920.4us
I20260709 12:24:52.680737  9004 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.763828  8811 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:52.587277  8986 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.768934  8902 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:52.769488  8803 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 30.14, Last appended by leader: 14, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.770493  8942 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:52.755743  8949 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:52.772688  8874 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:52.772835 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 62712.3us
I20260709 12:24:52.705698  8851 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.069s
I20260709 12:24:52.669736  8948 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.044s
I20260709 12:24:52.773466  8916 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:52.693094  8721 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.086s
I20260709 12:24:52.776284  8886 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:52.766261  8632 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:52.779431  8987 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.605s	user 0.013s	sys 0.000s
W20260709 12:24:52.781373  8979 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.562s	user 0.000s	sys 0.001s
W20260709 12:24:52.787248 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 74024.8us
I20260709 12:24:52.663021  8944 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.786312  8945 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:52.789919 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:52.790374  8903 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:52.791669 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:52.795812 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:52.947394 31669 raft_consensus.cc:2410] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 30 because replica has last-logged OpId of term: 27 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 17.
I20260709 12:24:52.721685  8804 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:52.795462 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:52.950999 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:52.951157 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 155853us
I20260709 12:24:52.893972  8949 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:52.894032  8916 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:52.791172  8990 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 23.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.006s
I20260709 12:24:52.852939  9003 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.093s
I20260709 12:24:52.809311 32377 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:24:52.811996 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:52.785162  8890 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.814610 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:52.826208 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:52.826777  8879 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:24:52.827597  8974 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.989s	user 0.003s	sys 0.000s
I20260709 12:24:52.984282 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:52.836634  9008 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.380s	user 0.001s	sys 0.000s
I20260709 12:24:52.985947 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:52.825760  8552 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:52.797045  8907 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:52.856717 32370 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:52.989624 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:52.990510 31818 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:52.991839 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:52.857744 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:52.953281  8903 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:52.950567  9004 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:52.858057  8940 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:52.881945 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:52.910594 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:52.911296 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:52.911995 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:52.922297 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:52.999046 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:52.999820 31750 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:52.906036  8693 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:24:52.799015 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:52.809136 31677 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
W20260709 12:24:53.003373 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 204450us
I20260709 12:24:52.958205 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:52.912490  8924 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:52.922137  8992 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:52.880959 32378 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:52.943749  8998 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.089s
I20260709 12:24:53.013074 32374 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 19. Current term is 20. Ops: []
I20260709 12:24:52.985162  8943 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.029s
I20260709 12:24:53.037050 31825 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 24: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:53.003540  8590 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:24:53.002607  8693 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:53.062284 32370 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.15-24.15]
I20260709 12:24:53.004442 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:53.119228 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 114872us
I20260709 12:24:53.119336  8804 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:53.120152  8804 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:53.120252 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:53.120704  8804 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.121217  8804 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:53.121726  8721 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:24:53.121941 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:53.123944 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1647) took 2073 ms. Trace:
I20260709 12:24:53.125114  8990 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:53.125880  8990 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:52.955242  8950 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:52.984843  8922 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
W20260709 12:24:52.994242 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 179701us
I20260709 12:24:53.150554 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:52.983127  8811 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.005376  9004 raft_consensus.cc:2764] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:24:53.005623  8907 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:53.001157 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:53.041433  8998 consensus_queue.cc:1243] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 32.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
W20260709 12:24:53.126560  9016 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.174s	user 0.001s	sys 0.000s
I20260709 12:24:53.113488 32378 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:53.114095 31677 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:53.124419 31826 rpcz_store.cc:276] 0709 12:24:51.050963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.380707 (+329744us) service_pool.cc:224] Handling call
0709 12:24:51.381047 (+   340us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:51.381425 (+   378us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:51.381481 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:51.388768 (+  7287us) log.cc:844] Serialized 26 byte log entry
0709 12:24:51.652057 (+263289us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:24:51.652104 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:51.652180 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:52.826937 (+1174757us) spinlock_profiling.cc:243] Waited 10.4 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:53.123570 (+296633us) raft_consensus.cc:1685] finished
0709 12:24:53.123598 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.123861 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":90,"raft.run_cpu_time_us":151,"raft.run_wall_time_us":15310,"spinlock_wait_cycles":23869184,"thread_start_us":117064,"threads_started":1,"wal-append.queue_time_us":319146}
I20260709 12:24:53.174943 31818 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:53.011466  8924 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:52.990197  8952 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.774s	user 0.003s	sys 0.007s
I20260709 12:24:52.996565 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:24:53.113366 31642 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.701s (SENT) [suppressed 13 similar messages]
I20260709 12:24:53.200271 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:53.196156  8811 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:53.211742 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:53.127924  8890 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:53.129611  8552 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:53.148051  8590 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:53.223392 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 226923us
W20260709 12:24:53.170097  8980 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.940s	user 0.008s	sys 0.020s
W20260709 12:24:53.219446  8952 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.003s	user 0.003s	sys 0.008s
I20260709 12:24:53.104809  8902 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.161s
I20260709 12:24:53.213557  8922 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:53.221710  8966 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.997s	user 0.026s	sys 0.042s
W20260709 12:24:53.234814  8980 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.046s	user 0.008s	sys 0.026s
I20260709 12:24:53.256901 32371 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 30. Ops: []
I20260709 12:24:53.069281  8949 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:53.261057 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 147726us
I20260709 12:24:53.273018 31818 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
W20260709 12:24:53.282431 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 131971us
I20260709 12:24:53.283464 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:53.274533 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:53.253206  8966 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.028s	user 0.026s	sys 0.042s
I20260709 12:24:53.127152  8990 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:53.235242  8918 consensus_queue.cc:1059] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:24:53.206947  8916 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.202721  8879 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 22, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:53.285478 31818 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:53.316699 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:53.231621  9019 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:24:53.332775 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 49384.1us
W20260709 12:24:53.358158  9018 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
W20260709 12:24:53.359411 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.259153  8940 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:53.360980  8851 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:53.361862 31670 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1485) took 1526 ms (client timeout 1706 ms). Trace:
I20260709 12:24:53.361732  8851 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:53.362305 31670 rpcz_store.cc:269] 0709 12:24:51.835663 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.851639 (+ 15976us) service_pool.cc:224] Handling call
0709 12:24:53.361666 (+1510027us) spinlock_profiling.cc:243] Waited 88.5 ms on lock 0x7b540017cfd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:53.361791 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":203515776}
I20260709 12:24:53.365031  8851 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:53.366219  8940 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:53.353044  9019 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:53.369633  9019 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:53.373684 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:53.374570 31826 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:53.376502 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:53.368798 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:53.384327  9004 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.385175  8940 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:24:53.384977  9004 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.386799  8950 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:53.388542  8721 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.389096  8721 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.382969 31826 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:24:53.318452  8907 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:53.391783  8721 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:53.392376  8721 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.395169  8942 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:53.396090 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:53.396952  8948 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:53.398929  8948 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.387367  8950 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:53.401366  8948 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:53.401675  8950 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:53.396745 31826 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:24:53.392834  8721 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.403743  9004 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:53.355365  9029 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.102s	user 0.000s	sys 0.001s
W20260709 12:24:53.409132 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.385000 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:53.409183 31818 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:53.405442  8721 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:53.417550 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:53.414942 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:53.258468  8922 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:53.328934  8944 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:53.432793 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:53.435053 31826 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:53.289247  9030 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:24:53.445585 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60668.3us
I20260709 12:24:53.446779 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:53.420570  9004 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:53.440697  8940 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.421126  8945 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:53.448711  8997 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.256s	user 0.000s	sys 0.008s
I20260709 12:24:53.483067  8945 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.487107  8940 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.491201  8890 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:53.456079  9027 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
W20260709 12:24:53.457892  9021 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.261s	user 0.000s	sys 0.001s
I20260709 12:24:53.458833 31818 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
W20260709 12:24:53.460075  9031 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:24:53.490607 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1482) took 1676 ms. Trace:
I20260709 12:24:53.500195 31678 rpcz_store.cc:276] 0709 12:24:51.814139 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.831855 (+ 17716us) service_pool.cc:224] Handling call
0709 12:24:51.910275 (+ 78420us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:51.910511 (+   236us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:24:51.932224 (+ 21713us) log.cc:844] Serialized 14 byte log entry
0709 12:24:53.212078 (+1279854us) log.cc:844] Serialized 14 byte log entry
0709 12:24:53.212524 (+   446us) log.cc:844] Serialized 14 byte log entry
0709 12:24:53.489825 (+277301us) spinlock_profiling.cc:243] Waited 6.68 ms on lock 0x7b18000419e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:24:53.490042 (+   217us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:53.490102 (+    60us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:24:53.490159 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.490238 (+    79us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.490527 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":292111,"prepare.queue_time_us":702517,"prepare.run_cpu_time_us":207,"prepare.run_wall_time_us":14514,"raft.queue_time_us":296176,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":9363,"spinlock_wait_cycles":15357312,"thread_start_us":1263592,"threads_started":3,"wal-append.queue_time_us":409837}
I20260709 12:24:53.421406  9003 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:53.504441  9003 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.505286 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3704) took 1259 ms. Trace:
I20260709 12:24:53.505738 31825 rpcz_store.cc:276] 0709 12:24:52.245778 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.249452 (+  3674us) service_pool.cc:224] Handling call
0709 12:24:53.505031 (+1255579us) spinlock_profiling.cc:243] Waited 683 ms on lock 0x7b54001c93d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:53.505208 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1571095424}
I20260709 12:24:53.440305  8721 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:53.342787  8590 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.365489  8693 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.468015  8811 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:53.468685  8903 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:53.536370 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:53.536633  8994 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:53.537007 31748 raft_consensus.cc:2376] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30. Re-sending same reply.
I20260709 12:24:53.537523 31752 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:53.537225  8994 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:53.530416  8992 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.519s	user 0.009s	sys 0.000s
I20260709 12:24:53.538427 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:53.440373  8924 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:53.472460  8530 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:53.474748 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1314) took 1425 ms. Trace:
I20260709 12:24:53.432771  8966 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.445914  8851 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 12:24:53.433053  9033 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:53.491993  8890 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.493165 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:53.509330 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1353) took 1455 ms. Trace:
W20260709 12:24:53.526500 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 79843us
I20260709 12:24:53.521070  8602 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:53.524878  9004 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:53.527858 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:53.528957 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.533252 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1475) took 2991 ms. Trace:
I20260709 12:24:53.535358  8945 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:53.535740 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3717) took 1217 ms. Trace:
I20260709 12:24:53.535969  8940 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:53.468719  8922 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:53.540613 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:53.540104 32376 rpcz_store.cc:276] 0709 12:24:52.051164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.051852 (+   688us) service_pool.cc:224] Handling call
0709 12:24:52.052282 (+   430us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:52.710533 (+658251us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.474380 (+763847us) spinlock_profiling.cc:243] Waited 637 ms on lock 0x7b540074e854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:53.474615 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1465808640}
I20260709 12:24:53.545842 31826 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:24:53.550868 32371 rpcz_store.cc:276] 0709 12:24:52.053963 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.054594 (+   631us) service_pool.cc:224] Handling call
0709 12:24:52.054987 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:53.259176 (+1204189us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.508988 (+249812us) spinlock_profiling.cc:243] Waited 1.2 s on lock 0x7b54006473d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:53.509251 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2763447040}
I20260709 12:24:53.551666 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:53.550915 31746 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:24:53.553920 31825 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:53.554796 31676 rpcz_store.cc:276] 0709 12:24:50.541338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.541471 (+   133us) service_pool.cc:224] Handling call
0709 12:24:50.577326 (+ 35855us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:51.943977 (+1366651us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.532938 (+1588961us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x7b54002e8454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:53.533151 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3127314176}
I20260709 12:24:53.556391 31823 rpcz_store.cc:276] 0709 12:24:52.318460 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.382294 (+ 63834us) service_pool.cc:224] Handling call
0709 12:24:53.535566 (+1153272us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54004466d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:53.535673 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":388823936}
I20260709 12:24:53.537880  8903 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:53.540333  9030 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:53.603967 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:53.540723  8811 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.542448  9007 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:53.606992 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55436.5us
I20260709 12:24:53.608042 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:53.609695 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:53.610534 31823 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:53.525962  8721 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.527909  9019 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:53.619153 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 196369us
I20260709 12:24:53.620024 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:53.622844 31825 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:53.586079  8924 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:53.623346 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 303346us
I20260709 12:24:53.552374  8966 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:53.615012  8940 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:53.635133 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:53.585973  8602 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:53.643261 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:53.641775 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:53.645792 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:53.614092  8811 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:53.647930 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3718) took 1317 ms. Trace:
I20260709 12:24:53.648039 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:53.648509 31750 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:24:53.633720  8924 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:53.649269  8943 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:53.649639 31821 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:53.650846 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:53.651492 31750 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:53.653527 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:53.654126 31746 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:53.654400 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:53.655345  8992 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:53.655961  8992 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.656672  8002 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:53.657801  8002 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:53.658200  8992 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:53.640853  9033 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:53.648850 32371 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
W20260709 12:24:53.667131 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3885) took 3042 ms (client timeout 3087 ms). Trace:
I20260709 12:24:53.621802  9003 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:53.667675 31822 rpcz_store.cc:269] 0709 12:24:50.624582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:50.761604 (+137022us) service_pool.cc:224] Handling call
0709 12:24:53.666922 (+2905318us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540030ded4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:53.667043 (+   121us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2427990912}
I20260709 12:24:53.659011 31753 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:53.666713 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3716) took 1359 ms. Trace:
I20260709 12:24:53.677618 31824 rpcz_store.cc:276] 0709 12:24:52.306942 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.709099 (+402157us) service_pool.cc:224] Handling call
0709 12:24:52.709457 (+   358us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:52.713257 (+  3800us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.666453 (+953196us) spinlock_profiling.cc:243] Waited 3.07 ms on lock 0x7b540043d5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:53.666658 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7068416}
I20260709 12:24:53.665614 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:53.679193 31750 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:53.680721 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:53.689237 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:53.668708 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:53.658164  8002 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:24:53.661777  8804 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.540s	user 0.012s	sys 0.003s
I20260709 12:24:53.691115 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:53.691401 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:53.691515  8804 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:53.691650  8002 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.691165 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:53.692378 31670 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:24:53.692827 31677 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:24:53.692901  8002 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:53.656682  9004 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:53.693888  8804 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:53.658835  8721 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:53.660128  8966 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:53.658583  8922 raft_consensus.cc:2764] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election decision vote started in defunct term 27: won
I20260709 12:24:53.696431 31669 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:53.696584 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:53.697052 31676 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:24:53.648299 31827 rpcz_store.cc:276] 0709 12:24:52.330317 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.711799 (+381482us) service_pool.cc:224] Handling call
0709 12:24:52.712138 (+   339us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:52.777759 (+ 65621us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:52.989199 (+211440us) log.cc:844] Serialized 14 byte log entry
0709 12:24:53.220583 (+231384us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:53.221288 (+   705us) log.cc:844] Serialized 52 byte log entry
0709 12:24:53.221501 (+   213us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:53.252450 (+ 30949us) log.cc:844] Serialized 14 byte log entry
0709 12:24:53.487708 (+235258us) spinlock_profiling.cc:243] Waited 24.1 ms on lock 0x7b18003c5620. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:24:53.647257 (+159549us) log.cc:844] Serialized 14 byte log entry
0709 12:24:53.647537 (+   280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.647688 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:53.647733 (+    45us) raft_consensus.cc:1685] finished
0709 12:24:53.647744 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.647873 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":74290,"prepare.queue_time_us":597233,"prepare.run_cpu_time_us":125,"prepare.run_wall_time_us":127,"raft.queue_time_us":180541,"raft.run_cpu_time_us":544,"raft.run_wall_time_us":63694,"spinlock_wait_cycles":62663168,"thread_start_us":755204,"threads_started":3,"wal-append.queue_time_us":305792}
I20260709 12:24:53.699400 31824 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:53.700147 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3720) took 1308 ms. Trace:
I20260709 12:24:53.700565 31824 rpcz_store.cc:276] 0709 12:24:52.392035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.698835 (+1306800us) service_pool.cc:224] Handling call
0709 12:24:53.699171 (+   336us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:53.699869 (+   698us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.700081 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:53.615880  9007 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.703341 31822 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: [30.18-31.19]
I20260709 12:24:53.708250 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:53.709065 31749 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
I20260709 12:24:53.709144 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:53.640625  8530 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:53.712451 31827 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:24:53.660605  8994 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:53.695997  8890 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:53.963482 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3727) took 1161 ms. Trace:
W20260709 12:24:53.874155 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.743821 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:53.965092 31751 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:53.966857 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:53.968205  8879 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:53.968963 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1665) took 1161 ms. Trace:
I20260709 12:24:53.696586 31825 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:54.075836 31827 rpcz_store.cc:276] 0709 12:24:52.807148 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.748391 (+941243us) service_pool.cc:224] Handling call
0709 12:24:53.968215 (+219824us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:53.968477 (+   262us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:53.968516 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:53.968579 (+    63us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:53.968621 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.968720 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:53.968898 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.076552 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3904) took 1232 ms. Trace:
W20260709 12:24:53.840451 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.748994 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:53.970861 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:54.077003 31825 rpcz_store.cc:276] 0709 12:24:52.844039 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.612153 (+768114us) service_pool.cc:224] Handling call
0709 12:24:54.076472 (+464319us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:53.750437  8530 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:53.972417 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:53.972689 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:53.973804 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:53.769035  8940 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:24:53.964231 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0117004290000295s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1509s for a response
I20260709 12:24:54.081274 31745 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 29: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:53.975102 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.078549 31825 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31. Ops: [30.19-30.19]
I20260709 12:24:54.121692 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3906) took 1275 ms. Trace:
W20260709 12:24:53.877766 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.705917  9033 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:54.122169 31825 rpcz_store.cc:276] 0709 12:24:52.845877 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.077803 (+1231926us) service_pool.cc:224] Handling call
0709 12:24:54.078257 (+   454us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:54.121422 (+ 43165us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.121632 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:54.123430 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:53.976689 31824 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:53.798877 31750 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:24:53.855769 31753 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:53.963980 31822 rpcz_store.cc:276] 0709 12:24:52.802581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.697826 (+895245us) service_pool.cc:224] Handling call
0709 12:24:53.698222 (+   396us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:53.707886 (+  9664us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:53.963415 (+255529us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.121243 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:54.125737 31822 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:54.126469 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3912) took 1277 ms. Trace:
I20260709 12:24:54.126855 31822 rpcz_store.cc:276] 0709 12:24:52.848939 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.125262 (+1276323us) service_pool.cc:224] Handling call
0709 12:24:54.125566 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.126203 (+   637us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.126415 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.128337 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3913) took 1279 ms. Trace:
I20260709 12:24:54.128638 31822 rpcz_store.cc:276] 0709 12:24:52.849270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.127611 (+1278341us) service_pool.cc:224] Handling call
0709 12:24:54.127881 (+   270us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.128047 (+   166us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.128072 (+    25us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.128114 (+    42us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:54.128143 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.128192 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.128289 (+    97us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.129886 31822 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:54.130394 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3914) took 1280 ms. Trace:
I20260709 12:24:54.084633 31827 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
W20260709 12:24:54.119624 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0072004290000223s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1514s for a response
I20260709 12:24:54.137462 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:53.966307 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:54.137687 31822 rpcz_store.cc:276] 0709 12:24:52.849611 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.129465 (+1279854us) service_pool.cc:224] Handling call
0709 12:24:54.129720 (+   255us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.130192 (+   472us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.130349 (+   157us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.122773  9040 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.133263 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:54.136233 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:54.141496 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:54.138427 31751 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:54.142160 32376 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:53.969775 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:54.137458 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3903) took 1293 ms. Trace:
I20260709 12:24:53.727231  8924 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:54.140220 32378 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:24:54.141952 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3915) took 1292 ms. Trace:
I20260709 12:24:54.143004 31749 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:24:54.080600 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 109813us
I20260709 12:24:54.142957 31827 rpcz_store.cc:276] 0709 12:24:52.843652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.076949 (+1233297us) service_pool.cc:224] Handling call
0709 12:24:54.084343 (+  7394us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.137124 (+ 52781us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.137387 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.147534 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:54.118914  8971 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.078135  9030 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:54.146606 31822 rpcz_store.cc:276] 0709 12:24:52.849889 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.139298 (+1289409us) service_pool.cc:224] Handling call
0709 12:24:54.141444 (+  2146us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.141638 (+   194us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.141669 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.141710 (+    41us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:54.141742 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.141792 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.141900 (+   108us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:54.146412 31672 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 2921) took 2255 ms (client timeout 1803 ms). Trace:
I20260709 12:24:54.157003 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:54.158829 31672 rpcz_store.cc:269] 0709 12:24:51.881906 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.032904 (+150998us) service_pool.cc:224] Handling call
0709 12:24:54.137390 (+2104486us) spinlock_profiling.cc:243] Waited 341 ms on lock 0x7b5400046ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:54.137520 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":783385856}
I20260709 12:24:54.147678 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:54.154593 31751 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
W20260709 12:24:54.134159  8907 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s	user 0.047s	sys 0.085s
W20260709 12:24:54.146759 31644 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.021s	sys 0.033s
W20260709 12:24:54.168977 31644 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.021s	sys 0.033s
I20260709 12:24:54.124433  8943 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:53.746496  9007 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:54.156208 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:54.139950  8916 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.135790  8890 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:54.162086 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:54.147872  9035 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.167497 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:54.168774 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3916) took 1318 ms. Trace:
W20260709 12:24:54.126260  8920 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.909s	user 0.000s	sys 0.050s
I20260709 12:24:54.134027  9033 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:54.180274  8002 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:54.140661  8940 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:54.161276 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:54.148456  9040 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:54.301226 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:54.302296 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:54.306293 31827 rpcz_store.cc:276] 0709 12:24:52.850208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.147166 (+1296958us) service_pool.cc:224] Handling call
0709 12:24:54.168073 (+ 20907us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.168340 (+   267us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.168379 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.168440 (+    61us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:54.168485 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.168557 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.168695 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.301802 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1358) took 1908 ms. Trace:
I20260709 12:24:54.302362 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:54.301748 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:54.309079 32376 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 30: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:54.307725 31745 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:24:54.145516  8923 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.183449 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3919) took 1332 ms. Trace:
I20260709 12:24:54.238802 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:54.310432 31822 rpcz_store.cc:276] 0709 12:24:52.851195 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.158070 (+1306875us) service_pool.cc:224] Handling call
0709 12:24:54.158361 (+   291us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.167833 (+  9472us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:54.167863 (+    30us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.183046 (+ 15183us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:54.183114 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.183203 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.183369 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9346,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103}
W20260709 12:24:54.312022 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 155888us
I20260709 12:24:54.148689  8804 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.307631 32374 rpcz_store.cc:276] 0709 12:24:52.393582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.393722 (+   140us) service_pool.cc:224] Handling call
0709 12:24:52.440256 (+ 46534us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:53.013623 (+573367us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.301494 (+1287871us) spinlock_profiling.cc:243] Waited 572 ms on lock 0x7b5400595454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:54.301731 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1316680064}
I20260709 12:24:54.307637 32377 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:54.309442 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:54.303196 32369 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: [25.18-25.18]
I20260709 12:24:54.185645 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:54.319054 32378 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:24:54.338949 31677 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:54.181524 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1667) took 1373 ms. Trace:
I20260709 12:24:54.339882 31824 rpcz_store.cc:276] 0709 12:24:52.808088 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.976143 (+1168055us) service_pool.cc:224] Handling call
0709 12:24:53.976550 (+   407us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.178199 (+201649us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.178251 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.178341 (+    90us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:54.178390 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.178492 (+   102us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.181432 (+  2940us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":148,"raft.run_cpu_time_us":96,"raft.run_wall_time_us":97}
I20260709 12:24:54.237421 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.342710 31673 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:54.238246 31671 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:54.150484 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:54.344265 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 193856us
I20260709 12:24:54.312994 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:54.168641  8907 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.774s	user 0.047s	sys 0.085s
I20260709 12:24:54.357342 31750 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 34: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:54.345499 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:54.250118 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:54.288570 31752 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.011s	sys 0.045s
W20260709 12:24:54.361526 31752 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.701s	user 0.011s	sys 0.045s
I20260709 12:24:54.162020  8924 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:24:54.167289  8990 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.680s	user 0.005s	sys 0.023s
W20260709 12:24:54.366729 31643 connection.cc:256] RPC call timeout handler was delayed by -1.0528356559998429s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.30870000000000003s for a response
W20260709 12:24:54.305482  8920 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.089s	user 0.000s	sys 0.065s
I20260709 12:24:54.368505  8002 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:54.371831 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:54.376101 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:54.419812 31823 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:54.391388 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3714) took 2096 ms. Trace:
I20260709 12:24:54.394184 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3730) took 1266 ms. Trace:
I20260709 12:24:54.422567 31821 rpcz_store.cc:276] 0709 12:24:53.128209 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.550677 (+422468us) service_pool.cc:224] Handling call
0709 12:24:54.393969 (+843292us) spinlock_profiling.cc:243] Waited 45 ms on lock 0x7b54003c96d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:54.394117 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":103454592}
W20260709 12:24:54.397009  9023 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.284s	user 0.003s	sys 0.001s
I20260709 12:24:54.397557 31825 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:54.407582  9019 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:54.290491  8590 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.023s	sys 0.034s
I20260709 12:24:54.193814  8562 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:54.413904 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1668) took 1436 ms. Trace:
I20260709 12:24:54.415161 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:54.435429 31750 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:24:54.435199 31827 rpcz_store.cc:276] 0709 12:24:52.977580 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.357001 (+1379421us) service_pool.cc:224] Handling call
0709 12:24:54.413089 (+ 56088us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.413399 (+   310us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.413443 (+    44us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.413513 (+    70us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:54.413561 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.413635 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.413814 (+   179us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.424813 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:54.438136  8562 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:54.438726 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:54.440418 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:54.440794  8562 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:54.441761 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:54.442212 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:54.425747 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:54.305697  8890 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:54.442818 31826 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:54.426946 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:54.443449 31751 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:24:54.427045 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.299350  8595 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.413424 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:54.412636 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:54.422408 31820 rpcz_store.cc:276] 0709 12:24:52.295190 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.676554 (+381364us) service_pool.cc:224] Handling call
0709 12:24:52.677003 (+   449us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:53.658868 (+981865us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:53.658900 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:53.658942 (+    42us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:53.658971 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.391111 (+732140us) spinlock_profiling.cc:243] Waited 982 ms on lock 0x7b54000fbc54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:54.391161 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.391317 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2257520384}
I20260709 12:24:54.248658  8530 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:54.409467  8990 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.927s	user 0.013s	sys 0.040s
W20260709 12:24:54.427757  8590 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.024s	sys 0.034s
I20260709 12:24:54.365342  9007 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:54.452107 31825 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:24:54.442351 31827 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:24:54.168881  8971 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:54.368258  9046 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:24:54.449877 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 281540us
I20260709 12:24:54.461563 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.336335  9033 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:54.475080 31820 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
W20260709 12:24:54.475603 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.487881 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:54.586910 31677 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:24:54.214129  8916 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:54.591387 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:54.581282 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.365113  8923 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:54.365139  8804 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:54.592206 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.590911 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.599858 31745 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
W20260709 12:24:54.602533  9044 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.129s	user 0.000s	sys 0.000s
I20260709 12:24:54.194227  8943 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:54.603423 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.446440  8998 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:54.604247 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 151291us
I20260709 12:24:54.648120 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:54.358458  9035 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:54.584903 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:54.609791 31826 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
W20260709 12:24:54.610787 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.531s	user 0.017s	sys 0.029s
W20260709 12:24:54.650135 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.571s	user 0.017s	sys 0.029s
W20260709 12:24:54.650509 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 683527us
I20260709 12:24:54.488039  8721 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.612375 31820 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:24:54.247225  9038 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.207s
I20260709 12:24:54.615677 32370 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:54.443082  8890 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:54.474025  9019 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.142s
I20260709 12:24:54.641649 31671 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
W20260709 12:24:54.680492 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:54.675411 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:54.673296 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 364620us
I20260709 12:24:54.676543 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:54.606220  8949 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:54.610944  8992 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:54.678093 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:54.678359 31820 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:54.681241 31677 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:24:54.612931  9046 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:54.580255  8595 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:54.621706  9007 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:54.683480 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.699913 31674 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:24:54.689435 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:54.692358 31827 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 33. Current term is 34. Ops: []
I20260709 12:24:54.695698 31822 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:24:54.701457 31673 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:54.689268 31820 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:24:54.703478 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.705555 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:54.706286 31745 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:54.611431  8976 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.072s
I20260709 12:24:54.707100 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:54.707140 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:54.677989  8907 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:54.715487  9040 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:54.716425  9040 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 30.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:54.717159  9040 consensus_queue.cc:1243] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 31.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 13
I20260709 12:24:54.718191  9007 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:54.719565 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:54.720366 31826 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:54.723547  8923 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:54.680266  8721 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:54.699295  8992 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:54.728470  8971 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:54.728996  8992 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:54.729727  8992 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:24:54.730120  8992 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:54.730568  8804 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:54.733095  9035 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:54.738349 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1373) took 1084 ms. Trace:
I20260709 12:24:54.694916  8949 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:54.739003 31752 rpcz_store.cc:276] 0709 12:24:53.654013 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.659373 (+  5360us) service_pool.cc:224] Handling call
0709 12:24:53.659786 (+   413us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:53.660142 (+   356us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:53.660173 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:53.660677 (+   504us) log.cc:844] Serialized 26 byte log entry
0709 12:24:54.361922 (+701245us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:24:54.361961 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.362031 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:54.738047 (+376016us) raft_consensus.cc:1685] finished
0709 12:24:54.738077 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.738263 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1300,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":102,"thread_start_us":700846,"threads_started":1,"wal-append.queue_time_us":919851}
I20260709 12:24:54.741600  8949 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:54.741957  9046 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:54.747898  8999 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:54.748503  8999 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:54.753954  8595 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:54.756067  8943 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:54.709508 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:54.761831 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:54.765614 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 59119.7us
W20260709 12:24:54.775367 31749 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.624s	user 0.012s	sys 0.000s
I20260709 12:24:54.775981 31749 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:54.776577 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1377) took 1064 ms. Trace:
I20260709 12:24:54.690651  8813 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Leader pre-election lost for term 20. Reason: could not achieve majority
I20260709 12:24:54.776978 31749 rpcz_store.cc:276] 0709 12:24:53.712035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.712205 (+   170us) service_pool.cc:224] Handling call
0709 12:24:54.776425 (+1064220us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:54.779556 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:54.781426 31749 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:24:54.767042 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:54.784682 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 77658.4us
I20260709 12:24:54.785706 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:54.785918 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.699173  8986 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:24:54.788230 31826 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
W20260709 12:24:54.791463 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 478535us
I20260709 12:24:54.792464 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:54.797255 31752 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:54.823897 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.758919  8425 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:54.749519  8999 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:54.674770  8998 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:54.846751 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54258.8us
I20260709 12:24:54.848585 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:54.848743 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.849978 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:54.786962  9049 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.182s	user 0.000s	sys 0.001s
W20260709 12:24:54.834087 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 67193.6us
I20260709 12:24:54.830135  8952 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:54.708627  9019 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.129s
I20260709 12:24:54.869304 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.870179 31677 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:24:54.871384 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.873668 31749 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:54.825795  8907 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:54.859023 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:54.823616  8562 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.040s
I20260709 12:24:54.863763 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.866240 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:54.895924 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 20 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:54.896430 31673 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:24:54.878086  8916 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:54.878355 31670 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:54.791776  8986 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:54.884938 31744 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 24 index: 13. (index mismatch)
W20260709 12:24:54.857930 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 163143us
I20260709 12:24:54.834098  8425 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:54.786036  8923 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:54.851908  9035 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:24:54.896713 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:54.883107  8940 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:54.830996  8889 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 23.16, Next index: 17, Last known committed idx: 14, Time since last communication: 0.066s
I20260709 12:24:54.900322  8943 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:54.875183  8999 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:54.908776 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:54.911202  8002 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:54.911700  8002 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:54.912127  8002 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:54.912055 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:54.851706  9045 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.049s
I20260709 12:24:54.874039  9060 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:54.896047  8952 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:54.914873 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:54.904254 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:54.922943 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:54.926059 31670 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 22 index: 15. (index mismatch)
I20260709 12:24:54.904956  8425 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:54.903807  8923 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:54.960656  8923 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:54.930742 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1380) took 1183 ms. Trace:
I20260709 12:24:54.909124  8595 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader election lost for term 20. Reason: could not achieve majority
W20260709 12:24:54.956970 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 52791.9us
I20260709 12:24:54.964963 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:54.965039 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1375) took 1299 ms. Trace:
I20260709 12:24:54.921631  8002 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:54.867789  9047 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.069s
I20260709 12:24:54.903151  8907 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:54.836251  8992 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.14, Last appended by leader: 14, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:54.939446 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:54.967621 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52858.4us
I20260709 12:24:54.985296 31717 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:54.995035 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:54.995960 31670 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 29: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:24:54.957999 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 373162us
I20260709 12:24:54.961915 31751 rpcz_store.cc:276] 0709 12:24:53.747496 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.444024 (+696528us) service_pool.cc:224] Handling call
0709 12:24:54.444379 (+   355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.444568 (+   189us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:24:54.444857 (+   289us) log.cc:844] Serialized 14 byte log entry
0709 12:24:54.927349 (+482492us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.927409 (+    60us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:24:54.927500 (+    91us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.927587 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.930647 (+  3060us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":1175,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":2010,"raft.queue_time_us":60795,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":83,"thread_start_us":482049,"threads_started":2,"wal-append.queue_time_us":253779}
I20260709 12:24:54.999982 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.001781 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:54.923761  8940 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:24:55.002851  8940 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:55.003387  8940 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:55.002363 31751 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:55.016917 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 162026us
I20260709 12:24:54.913429  9062 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:54.926056  8924 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:24:54.992156 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:55.020228 31669 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: []
I20260709 12:24:55.020660 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.021472 31749 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:24:55.023115  8002 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:55.022493 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:54.959997  9035 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:54.913668  9038 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.075s
I20260709 12:24:55.020558 31752 raft_consensus.cc:2376] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26. Re-sending same reply.
I20260709 12:24:55.032635 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:54.962844  8841 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:55.041385 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.045629 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3734) took 1914 ms. Trace:
I20260709 12:24:55.046556 31749 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 34 because replica has last-logged OpId of term: 32 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 19.
I20260709 12:24:55.049077 31751 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:55.049594 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.018383  8804 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:55.069651 31750 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: [26.17-26.17]
I20260709 12:24:55.070159 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.028077  8721 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:55.072309 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:55.072952 31749 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:55.043998  9035 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.043678  8562 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:24:55.035318  8992 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:55.048915  9000 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:24:55.077291 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 223811us
I20260709 12:24:55.082058 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.042160  8924 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:55.092283  8924 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 23.16, Last appended by leader: 16, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.093261  9033 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.076282  8952 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.095731 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:55.096462 32375 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:24:55.097358 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:55.043589  8841 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:55.085860  9063 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 12:24:55.087577  8907 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.074445  8889 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 25.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.016s
I20260709 12:24:55.091055  8721 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:24:55.100593 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:55.082043  9035 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:55.110605 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:55.082114  8562 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:55.082906  9000 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.098493 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:55.117992 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:55.084235  9007 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:24:55.132457 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 139880us
I20260709 12:24:55.104408  8998 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.113336  8841 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:55.113144  9030 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.148042  8002 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.148689  8002 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.130270 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:55.134821  9060 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.138710  8890 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.133740  9081 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:55.141610  9033 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:55.133386  9000 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:55.065299  8986 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.151026 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:55.143093  9038 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.048s
W20260709 12:24:55.157447 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46978.9us
I20260709 12:24:55.161445 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:55.159178 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:55.185655 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:55.187003 31676 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: []
I20260709 12:24:55.188170 32375 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:24:55.168318  8002 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:55.189106 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:55.189109  8002 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:55.191064  9086 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:55.191581  9086 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:55.157415  9007 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.035s
I20260709 12:24:55.192147  9086 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:55.193197  9007 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:24:55.193706  9007 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:55.194120  9007 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:55.183563  9019 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:55.202916  8562 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 23.16, Last appended by leader: 16, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.204434  8562 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:24:55.183743  9045 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:55.207239  9045 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.207448  8890 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.207592  9045 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:55.209872 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.210026 31643 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:55.211627 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 93703.6us
I20260709 12:24:55.212471 31826 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
W20260709 12:24:55.213074 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51806.4us
I20260709 12:24:55.214576 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.215322 31826 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:55.216246 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.152851  8922 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.159225  9062 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:55.222183 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3732) took 2092 ms. Trace:
I20260709 12:24:55.222677 31823 rpcz_store.cc:276] 0709 12:24:53.129693 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.420812 (+1291119us) service_pool.cc:224] Handling call
0709 12:24:54.421265 (+   453us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:54.421750 (+   485us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.421795 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:54.437985 (+ 16190us) log.cc:844] Serialized 26 byte log entry
0709 12:24:54.776658 (+338673us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:54.776716 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.776817 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.221972 (+445155us) raft_consensus.cc:1685] finished
0709 12:24:55.221992 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.222124 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":13020,"raft.queue_time_us":2690,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":11407,"thread_start_us":338199,"threads_started":1,"wal-append.queue_time_us":430046}
I20260709 12:24:55.155405  8998 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:55.187520  8943 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:24:55.159271  9047 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 29.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.060s
I20260709 12:24:55.154585  9030 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.226465  9091 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:55.228171  8986 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:55.183928  8889 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.060s
W20260709 12:24:55.230036 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.156931  9060 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.214035  8890 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.017s
I20260709 12:24:55.231055  9076 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:55.183861  9033 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:55.237651 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:55.216487  9082 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.083s	user 0.000s	sys 0.001s
I20260709 12:24:55.216684  8721 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:55.243579 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.218894 31826 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:55.216949  9019 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:24:55.237448  9071 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.136s	user 0.001s	sys 0.001s
W20260709 12:24:55.249379 31818 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3910) took 2401 ms (client timeout 2208 ms). Trace:
I20260709 12:24:55.252588 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:55.264081 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 79135.2us
I20260709 12:24:55.157063  9081 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:55.265668 31818 rpcz_store.cc:269] 0709 12:24:52.848057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.412749 (+564692us) service_pool.cc:224] Handling call
0709 12:24:55.249084 (+1836335us) spinlock_profiling.cc:243] Waited 38.3 ms on lock 0x7b5400166cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:55.249271 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":88199424}
I20260709 12:24:55.265241 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.266397 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:55.267042 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.267040 31826 raft_consensus.cc:2376] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29. Re-sending same reply.
I20260709 12:24:55.255972  9066 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:55.233433  8922 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.268302 31818 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:55.266390 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:55.272971 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:55.275844 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:55.273144 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.256644  8721 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:55.291254  8924 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election won for term 23
W20260709 12:24:55.291920 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 75722.9us
I20260709 12:24:55.270308  9066 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.241926  9085 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.031s
I20260709 12:24:55.286073 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:55.293251 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.268065  8804 raft_consensus.cc:2764] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:24:55.347295 31826 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:55.284988  9067 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:55.285426  9091 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.038s
I20260709 12:24:55.274887  8425 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.322331  8841 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.296797  8721 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:55.324817  9089 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.042s
I20260709 12:24:55.327426  8940 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.371541  9089 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:24:55.371944  9089 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:55.372651  8940 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:55.372782  9089 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.328646 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.374255 31818 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:24:55.375347 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 22 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.376689 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.299161  8924 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.377787  8924 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:55.379030  9085 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:24:55.379716  9084 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:55.343663 31823 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:55.381714  9084 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:55.337690  8803 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:55.382710  9084 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:55.379158 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 93170.7us
I20260709 12:24:55.388449  9060 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:55.389011  8991 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 3.190s	user 0.000s	sys 0.017s
I20260709 12:24:55.391040 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1484) took 3556 ms. Trace:
I20260709 12:24:55.348141  9066 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:55.391446 31675 rpcz_store.cc:276] 0709 12:24:51.835205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:51.865442 (+ 30237us) service_pool.cc:224] Handling call
0709 12:24:51.865922 (+   480us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:51.866380 (+   458us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:51.866416 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:51.897562 (+ 31146us) log.cc:844] Serialized 26 byte log entry
0709 12:24:52.100118 (+202556us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:24:52.100169 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.102388 (+  2219us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.388207 (+3285819us) spinlock_profiling.cc:243] Waited 95.2 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:55.390720 (+  2513us) raft_consensus.cc:1685] finished
0709 12:24:55.390742 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.390975 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4110,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":88,"spinlock_wait_cycles":218874624,"thread_start_us":158906,"threads_started":1,"wal-append.queue_time_us":218754}
I20260709 12:24:55.394099  8976 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:55.394569  8976 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:55.395730  8976 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:55.397177  8976 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:55.353413  9009 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:55.399678  9009 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:55.400424  9088 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.171s	user 0.000s	sys 0.001s
I20260709 12:24:55.400712  9009 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.401700 31747 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 14. Preceding OpId from leader: term: 31 index: 15. (index mismatch)
I20260709 12:24:55.402372 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:55.402863  9072 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.177s	user 0.003s	sys 0.000s
I20260709 12:24:55.403753 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:55.404693 31818 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:24:55.309927 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:55.352085 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:55.406977 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55008.7us
I20260709 12:24:55.409680 31823 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:24:55.256763  9033 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.347874  9030 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:55.357537  8425 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:55.360276 31752 raft_consensus.cc:2368] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 29. Current term is 30.
I20260709 12:24:55.390285 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:55.360710  8811 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.362679 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.363005  9093 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:55.350302  9067 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:24:55.366469  9022 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 2.235s	user 0.002s	sys 0.002s
I20260709 12:24:55.373387  8940 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.431038  8940 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.432287  9035 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.283692  8943 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:55.379680  9085 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:55.435951 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:55.436846 32372 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:55.436240  9085 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.438441 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.438766  9085 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:55.439379 31818 raft_consensus.cc:2376] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34. Re-sending same reply.
I20260709 12:24:55.440632 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.441572 31818 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:55.422082  8841 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.344025 32375 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:24:55.360682 31745 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
W20260709 12:24:55.411603 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 101835us
I20260709 12:24:55.417541  9067 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:55.046337 31821 rpcz_store.cc:276] 0709 12:24:53.131188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.424263 (+1293075us) service_pool.cc:224] Handling call
0709 12:24:54.485475 (+ 61212us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.485716 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:54.901645 (+415929us) log.cc:844] Serialized 14 byte log entry
0709 12:24:55.044657 (+143012us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:55.044866 (+   209us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:55.045006 (+   140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.045216 (+   210us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.045509 (+   293us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":534303,"raft.queue_time_us":116,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":9455,"spinlock_wait_cycles":33199488,"thread_start_us":431310,"threads_started":2}
I20260709 12:24:55.349058  8923 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.412134 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:55.413589 31827 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
W20260709 12:24:55.415217 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 193999us
I20260709 12:24:55.350453 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.402360  9009 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:55.414964  8803 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:55.434947  8943 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.442618 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.415966  8949 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:24:55.446111 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.463295 31818 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:24:55.459838 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1660) took 3069 ms. Trace:
I20260709 12:24:55.453471 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:55.450012 31671 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:24:55.471165 31671 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 25 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:55.457520 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.413281  9019 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:24:55.466949 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54927.6us
I20260709 12:24:55.481186 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.464460 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.507449 31826 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 25.
I20260709 12:24:55.467411 31819 rpcz_store.cc:276] 0709 12:24:52.391268 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:52.712987 (+321719us) service_pool.cc:224] Handling call
0709 12:24:52.713310 (+   323us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:52.764359 (+ 51049us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:52.764401 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:52.765052 (+   651us) log.cc:844] Serialized 26 byte log entry
0709 12:24:52.944390 (+179338us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:52.944432 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:52.944502 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.365553 (+2421051us) spinlock_profiling.cc:243] Waited 27.9 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:55.459478 (+ 93925us) raft_consensus.cc:1685] finished
0709 12:24:55.459506 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.459740 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":47621,"raft.queue_time_us":50366,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":106,"spinlock_wait_cycles":64067328,"thread_start_us":178919,"threads_started":1,"wal-append.queue_time_us":246333}
I20260709 12:24:55.480264 31671 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 25.15, Last appended by leader: 15, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.505709 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:24:55.513144 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 237358us
I20260709 12:24:55.505553 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:55.448658  9052 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.838s	user 0.001s	sys 0.001s
I20260709 12:24:55.468532  8943 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:55.516995 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:55.453266  9090 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
I20260709 12:24:55.446957  8552 raft_consensus.cc:3055] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:55.522552 31671 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:55.459631  8940 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:55.456131  9033 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:55.466750  8949 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.523675 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:55.415042  9007 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.529835 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:55.530061 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.481472  9093 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.532183 31818 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:24:55.481197  8948 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:24:55.480783  8803 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:55.528553 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:55.519650  8952 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:55.479601  8924 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.449882  9086 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.542440 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:55.544216 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.457249  9000 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.540557 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:55.534022  8949 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:55.541380  9033 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:55.575060  8948 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:55.575551  8948 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.575896  8948 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:55.575992  9000 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.550570 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.551424 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:55.577791 32342 leader_election.cc:336] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.706s (SENT)
I20260709 12:24:55.578382 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:55.579102 31751 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 25. Current term is 26. Ops: []
I20260709 12:24:55.579483  8940 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:55.548195  8886 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
I20260709 12:24:55.539065  8562 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.014s
I20260709 12:24:55.548321  8999 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:54.965618 31746 rpcz_store.cc:276] 0709 12:24:53.665184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.666350 (+  1166us) service_pool.cc:224] Handling call
0709 12:24:53.666710 (+   360us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:53.666943 (+   233us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:24:54.162755 (+495812us) log.cc:844] Serialized 14 byte log entry
0709 12:24:54.958705 (+795950us) log.cc:844] Serialized 14 byte log entry
0709 12:24:54.959085 (+   380us) log.cc:844] Serialized 14 byte log entry
0709 12:24:54.959364 (+   279us) log.cc:844] Serialized 14 byte log entry
0709 12:24:54.964548 (+  5184us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:54.964623 (+    75us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:24:54.964688 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:54.964797 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:54.964963 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4880,"prepare.queue_time_us":742576,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":50,"raft.queue_time_us":65562,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":6970,"thread_start_us":1235364,"threads_started":3,"wal-append.queue_time_us":572246}
I20260709 12:24:55.547060 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.522416  9004 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:24:55.641506  9004 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.558525  9093 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:55.641991  9004 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:24:55.568248 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.570947  8803 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.538425  9055 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:55.573408 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.655033 31818 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:24:55.529708  8552 raft_consensus.cc:740] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:55.577857 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:55.595119 31751 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:24:55.654966 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:55.657145 32371 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:24:55.595610  9013 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:55.596458  9045 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.665972  9045 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.667035  8562 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.667179 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.667748  8562 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.668581 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.669845 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:55.598282 31753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3910) took 2062 ms (client timeout 1569 ms). Trace:
I20260709 12:24:55.599923 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.602576 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.671160 31819 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:55.670817 31753 rpcz_store.cc:269] 0709 12:24:53.535493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.538195 (+  2702us) service_pool.cc:224] Handling call
0709 12:24:55.598043 (+2059848us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x7b54001a59d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:55.598195 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":275846912}
I20260709 12:24:55.672573 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:55.673727 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:55.674253 31753 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:55.604537 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:55.606455  8851 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 19.12, Next index: 13, Last known committed idx: 11, Time since last communication: 0.000s
W20260709 12:24:55.675009 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 70569.4us
I20260709 12:24:55.675403  8851 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Stepping down as leader of term 19
I20260709 12:24:55.675836  8851 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 19 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:55.676177  8924 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:55.676667  8977 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:55.677187  8977 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.543596  9035 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:55.677552  8977 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:55.677722  9093 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:24:55.676759  8851 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 19, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.678952  8851 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 19 FOLLOWER]: Advancing to term 20
I20260709 12:24:55.679399  9086 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.680756 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.681406 31670 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:55.610005  8922 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:55.610754  9007 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.612061  8923 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.660219  9060 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.612643 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3916) took 1302 ms. Trace:
I20260709 12:24:55.614849  9019 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.684247  9060 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.685575 31827 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:55.620390 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:55.620800 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.624576 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.625379 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.545184  8952 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.532251  8841 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:24:55.627521 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3911) took 2779 ms. Trace:
I20260709 12:24:55.631363 31671 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 26 index: 16. (index mismatch)
I20260709 12:24:55.640223 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.632764 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:55.682411 31753 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:24:55.653899  9030 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:55.628206 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:55.665957  8552 consensus_queue.cc:260] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.671345  9055 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:55.689926 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57279.6us
I20260709 12:24:55.694451 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.703954 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:55.704391 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.695384 31819 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
W20260709 12:24:55.704839 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:55.687192 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 66887.4us
I20260709 12:24:55.684259 31676 rpcz_store.cc:276] 0709 12:24:54.310568 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.381637 (+ 71069us) service_pool.cc:224] Handling call
0709 12:24:54.382106 (+   469us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:55.188246 (+806140us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.612310 (+424064us) spinlock_profiling.cc:243] Waited 805 ms on lock 0x7b540022b354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:55.612554 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1850637696}
I20260709 12:24:55.706344 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:55.689049 31674 raft_consensus.cc:2410] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 25 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:24:55.689759 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3935) took 1039 ms. Trace:
I20260709 12:24:55.690739 31749 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:24:55.692345 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:55.708487 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.689393 31825 rpcz_store.cc:276] 0709 12:24:52.848439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.123571 (+1275132us) service_pool.cc:224] Handling call
0709 12:24:54.397412 (+273841us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.489334 (+ 91922us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.627286 (+1137952us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54003c96d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:55.627448 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":580340608}
W20260709 12:24:55.659246 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.709134 31746 raft_consensus.cc:2368] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25.
I20260709 12:24:55.707463 31671 rpcz_store.cc:276] 0709 12:24:54.650140 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.899083 (+248943us) service_pool.cc:224] Handling call
0709 12:24:54.913215 (+ 14132us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:55.449884 (+536669us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x7b540035e420. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:24:55.689515 (+239631us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.689706 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":237164928}
I20260709 12:24:55.654373  8999 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:55.709283  8552 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:55.690165  8922 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:55.687433  9066 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.713265 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:55.687350  8971 raft_consensus.cc:2764] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:24:55.726966 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.696727 31677 raft_consensus.cc:2368] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25.
I20260709 12:24:55.728677 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.728755 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:55.721500 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.729882 31825 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:55.627578  9097 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:55.723230 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.727293 31791 leader_election.cc:400] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25.
I20260709 12:24:55.716154 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:55.686620  9060 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:55.731117 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.736665 31818 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:55.733639 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:55.689255  8952 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:55.711365  8562 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:55.725607 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:55.735440 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.689442  8841 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:55.733925 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.746572 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:55.733227 31791 leader_election.cc:403] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Cancelling election due to peer responding with higher term
I20260709 12:24:55.739362 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.740842  8602 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.741272  9066 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.734586  8922 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:55.718470  9093 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:55.743146 31750 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: [27.16-27.16]
I20260709 12:24:55.746021 31825 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:55.746152 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:55.693648  9030 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.751667 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.758322 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:55.774144 31674 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 25: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:55.689459  8721 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.744343  9097 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.776824  9097 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:55.777205  9055 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 25.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.777872  8922 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.767005  9060 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election lost for term 24. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25.
I20260709 12:24:55.747473 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.768327 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.780957 31825 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:55.781251 31675 raft_consensus.cc:2410] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:24:55.734791  9035 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:55.767994 31677 raft_consensus.cc:2376] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29. Re-sending same reply.
I20260709 12:24:55.730758  8924 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:55.784227 31751 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:55.760452  8966 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:55.795392 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.760100  8992 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:55.765082  8945 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:24:55.807988 31818 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:24:55.808833  8924 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.809502  9093 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:55.816283  9067 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.817158  9035 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 30 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:55.769838  8972 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 25.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.079s
I20260709 12:24:55.830943 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3918) took 1419 ms. Trace:
I20260709 12:24:55.857474 31669 rpcz_store.cc:276] 0709 12:24:54.411899 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.412104 (+   205us) service_pool.cc:224] Handling call
0709 12:24:54.412508 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:55.028020 (+615512us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.830671 (+802651us) spinlock_profiling.cc:243] Waited 605 ms on lock 0x7b5400391cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:55.830868 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1391369088}
I20260709 12:24:55.833070  8943 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.833839 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3930) took 1248 ms. Trace:
I20260709 12:24:55.860057 31673 rpcz_store.cc:276] 0709 12:24:54.585797 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.901863 (+316066us) service_pool.cc:224] Handling call
0709 12:24:54.902215 (+   352us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:54.907301 (+  5086us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:54.907348 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:54.907992 (+   644us) log.cc:844] Serialized 26 byte log entry
0709 12:24:55.833147 (+925155us) spinlock_profiling.cc:243] Waited 22.3 ms on lock 0x7b18003ba940. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1
0709 12:24:55.833333 (+   186us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:55.833372 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.833467 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.833516 (+    49us) raft_consensus.cc:1685] finished
0709 12:24:55.833535 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.833771 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22156,"raft.queue_time_us":86045,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":11840,"spinlock_wait_cycles":51253632,"thread_start_us":33071,"threads_started":1,"wal-append.queue_time_us":145716}
I20260709 12:24:55.835999 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.837353 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.864140 31819 raft_consensus.cc:2410] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 31 because replica has last-logged OpId of term: 28 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 17.
I20260709 12:24:55.837677  9060 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:55.838430 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:55.839437  8811 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:55.839529 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.840180  8841 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:55.841190 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.841813  8916 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:24:55.842337  8562 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:55.845857 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1669) took 2837 ms. Trace:
I20260709 12:24:55.793145  8999 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:55.819157  9046 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 26 current term: 26 is higher.
I20260709 12:24:55.798166  8966 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:55.856282 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:55.807158  8992 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:55.865602 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.903141 31671 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 29: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:55.903694 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.904675 31818 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:55.847877  8945 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:55.867400 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.859093 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.869529 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3934) took 1437 ms. Trace:
I20260709 12:24:55.872229 31825 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:55.781648  8977 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:55.883270 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45012.1us
I20260709 12:24:55.884392 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:55.885816 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.885838 31822 rpcz_store.cc:276] 0709 12:24:53.008817 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.311929 (+1303112us) service_pool.cc:224] Handling call
0709 12:24:54.312279 (+   350us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.701584 (+389305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.845583 (+1143999us) spinlock_profiling.cc:243] Waited 383 ms on lock 0x7b540037a354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:55.845790 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":880837376}
I20260709 12:24:55.894285 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:55.894556 31821 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 32: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:55.941744 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:55.943011 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.940905 31673 raft_consensus.cc:2410] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 26 because replica has last-logged OpId of term: 24 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 22 index: 14.
I20260709 12:24:55.943778 31822 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:55.945044 31819 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 28: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:55.945211 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:55.946069 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.888012  9003 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:24:55.947415 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:55.950716 31669 raft_consensus.cc:1240] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: [28.17-28.17]
I20260709 12:24:55.942291 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.951781 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:55.952193 31818 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:55.903599  9004 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:55.948560 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 64308.5us
I20260709 12:24:55.949842 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:55.962033 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:55.962809  8998 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:55.962989  9068 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:55.963246  8998 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:55.963596  9068 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:55.963698  8998 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:55.964401 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:55.894163  8562 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.967128  8562 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:55.967928 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:55.971800  8945 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:24:55.971948  8977 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.973595  9068 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.910070  8992 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:55.882274  8916 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:55.963737 31673 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:55.974900  9084 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:55.975855  9084 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:55.976588  8851 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:55.976686  9084 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.977290  8851 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.977514  8916 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:55.978713  8916 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:55.957792 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:55.979447 31825 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:55.981405  8851 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:55.982707 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:55.982723 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:55.983318  9102 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.132s	user 0.000s	sys 0.001s
I20260709 12:24:55.983609 31825 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:55.904570  8966 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:55.974452  8998 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:55.898743  8929 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:55.989063  8998 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.997891 31672 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:55.999642 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:56.004820  9081 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:56.005471 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3919) took 1569 ms. Trace:
I20260709 12:24:56.005875 31744 rpcz_store.cc:276] 0709 12:24:54.435731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.605442 (+169711us) service_pool.cc:224] Handling call
0709 12:24:54.605841 (+   399us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:54.900114 (+294273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.005191 (+1105077us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54000b81d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:56.005401 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":640750464}
I20260709 12:24:56.006472 31675 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:56.008445 31675 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:24:55.877406  9060 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:56.016415 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:56.018258 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 53958.6us
I20260709 12:24:55.946913 31827 rpcz_store.cc:276] 0709 12:24:54.432509 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.868390 (+1435881us) service_pool.cc:224] Handling call
0709 12:24:55.868818 (+   428us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:55.869084 (+   266us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:55.869124 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:55.869188 (+    64us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:55.869238 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.869314 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:55.869439 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:56.027034 31822 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:56.019181 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:56.026203 31818 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:56.032100 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:56.051532  9067 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:55.940537  8922 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:55.940877  8721 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:56.053568 31825 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:24:56.055603 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:56.059672 31818 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:55.950132  9030 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:55.959021  9066 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:55.950300  9046 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:56.068872 31820 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
W20260709 12:24:56.064404 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:56.052736  9104 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.086s	user 0.000s	sys 0.000s
I20260709 12:24:55.901381  8948 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:56.067124 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48020.5us
I20260709 12:24:56.057420 31746 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:55.939519  8949 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:56.072643 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.078977 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:56.079981 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:56.080895 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.083462 31672 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 16. Preceding OpId from leader: term: 24 index: 17. (index mismatch)
I20260709 12:24:55.939245  8952 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:56.091773 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:56.093308 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:56.087370 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:56.094482 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:56.092935 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 23 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:56.096374 31672 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:24:56.056017  8632 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:24:56.109902 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 394437us
W20260709 12:24:56.102747 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 379581us
I20260709 12:24:56.103515 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:56.103962 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:56.111871 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:56.071676  8851 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:24:56.110033 32374 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:24:56.078513  9009 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:24:56.133042 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:56.070101  9067 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:55.954615  9035 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.114588 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:56.097872 31791 leader_election.cc:400] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25.
I20260709 12:24:56.119307 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:56.108409 31818 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
W20260709 12:24:56.121157  9108 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:24:56.075626  9066 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:56.148231  9066 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.124027  8952 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:56.075495  9030 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:56.150494  9030 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.109529  9033 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:56.152343  9033 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.152783  9033 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:56.095645  9081 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:56.090471  9093 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:56.132580  9009 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.127789 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:56.154702  9009 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:56.158550 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:56.159502 31827 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:24:56.148159 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:56.163985 31674 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:56.129179 31672 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 23.
I20260709 12:24:56.129524  9060 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:56.130864  9097 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:56.131273  9004 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:56.071849  8924 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:56.133788 32378 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:24:56.134403 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1672) took 1014 ms. Trace:
I20260709 12:24:56.134591 32371 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
W20260709 12:24:56.136552 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58978.9us
I20260709 12:24:56.194167 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:56.195014 31826 rpcz_store.cc:276] 0709 12:24:55.120407 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.133312 (+1012905us) service_pool.cc:224] Handling call
0709 12:24:56.133695 (+   383us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.133956 (+   261us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.133995 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:56.134059 (+    64us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:56.134109 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.134186 (+    77us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:56.134328 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:56.197255 31826 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:56.198104 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1673) took 1077 ms. Trace:
I20260709 12:24:56.135188  8781 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:24:56.142020 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:24:56.201282 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59379.1us
I20260709 12:24:56.198485 31826 rpcz_store.cc:276] 0709 12:24:55.120646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.196571 (+1075925us) service_pool.cc:224] Handling call
0709 12:24:56.197005 (+   434us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.197772 (+   767us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.198034 (+   262us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:56.143311 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.143421  9067 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:56.144459  9075 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:56.144852  8693 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:56.145264  8943 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:56.065953  8922 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.142412 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:56.217631 31818 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:56.136921  8851 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:56.192996  9040 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:56.213980 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 116344us
I20260709 12:24:56.218948 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:56.205618 31826 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:24:56.220657 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:56.237030 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1674) took 1116 ms. Trace:
I20260709 12:24:56.242987 31826 rpcz_store.cc:276] 0709 12:24:55.120874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.204760 (+1083886us) service_pool.cc:224] Handling call
0709 12:24:56.205164 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.236623 (+ 31459us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.236951 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:56.244959 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:56.227658 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:56.213200 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:56.238780 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:56.238693 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:56.214900  8425 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:56.207170  9060 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:56.254143 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:56.265743 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1389) took 1239 ms. Trace:
I20260709 12:24:56.256423 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:56.266183 31752 rpcz_store.cc:276] 0709 12:24:55.026189 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.049304 (+ 23115us) service_pool.cc:224] Handling call
0709 12:24:56.243132 (+1193828us) spinlock_profiling.cc:243] Waited 309 ms on lock 0x7b54002f71d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:56.265675 (+ 22543us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":711607552}
I20260709 12:24:56.270350 31819 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:56.271973 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:56.216425  8781 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:56.375162 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 105626us
I20260709 12:24:56.222468  8924 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:56.378070 31827 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:24:56.380280 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:56.381027 31827 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:24:56.397691 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 125790us
I20260709 12:24:56.239974  9047 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:56.411550 31819 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:24:56.201089  9097 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:56.265834  8943 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:56.265373  8948 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:56.417547 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:56.271201  8912 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.164s
I20260709 12:24:56.396968  8920 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.039s
W20260709 12:24:56.442610 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.443778  9091 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:56.444387  9091 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:56.445508  9091 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.271399  8201 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:56.226226  9112 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.199s
I20260709 12:24:56.437841 31827 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:24:56.448658  9081 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:56.451538  9067 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:56.453656  9097 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:56.409420  8924 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.483908 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:56.484658 31747 raft_consensus.cc:2376] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28. Re-sending same reply.
W20260709 12:24:56.463779 31751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.017s	sys 0.026s
I20260709 12:24:56.374518  8425 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:56.468219 31827 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:56.486918 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3745) took 1361 ms. Trace:
I20260709 12:24:56.487424 31827 rpcz_store.cc:276] 0709 12:24:55.125072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.467359 (+1342287us) service_pool.cc:224] Handling call
0709 12:24:56.467906 (+   547us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.486554 (+ 18648us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.486852 (+   298us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:56.489018 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:56.415226  9002 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:24:56.241216  9040 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.273763  9093 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.187s
I20260709 12:24:56.476429 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:56.481637 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:56.485801 31751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.017s	sys 0.026s
W20260709 12:24:56.498785 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.492080 31827 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: []
I20260709 12:24:56.379441  9060 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.497116 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:56.408365  9046 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.073s
I20260709 12:24:56.265036  8851 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.473474  9085 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:24:56.500033 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3746) took 1374 ms. Trace:
I20260709 12:24:56.501561 32369 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 32: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:56.504521 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:56.485280  9030 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:56.421114  8562 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.069s
I20260709 12:24:56.479660  8945 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:56.785589 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 3110) took 1060 ms. Trace:
I20260709 12:24:56.771726 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:56.480966  8787 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Stepping down as leader of term 26
W20260709 12:24:56.777570 31673 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.002s	sys 0.009s
I20260709 12:24:56.486534  9091 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:56.613726 31822 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29. Ops: []
W20260709 12:24:56.799866 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.529s	user 0.008s	sys 0.024s
W20260709 12:24:56.764052 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:56.807431 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.008s	sys 0.024s
I20260709 12:24:56.801049 31746 rpcz_store.cc:276] 0709 12:24:55.725395 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.725527 (+   132us) service_pool.cc:224] Handling call
0709 12:24:56.785293 (+1059766us) spinlock_profiling.cc:243] Waited 307 ms on lock 0x7b54000f2dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:56.785511 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":705203072}
W20260709 12:24:56.809103 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 555005us
W20260709 12:24:56.749137 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 1.00144e+06us
I20260709 12:24:56.810087 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:56.811645 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:56.820433 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:56.494736  9002 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:56.550812 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:56.785338 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.803227 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1671) took 1683 ms. Trace:
I20260709 12:24:56.509450 31827 rpcz_store.cc:276] 0709 12:24:55.125306 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.490137 (+1364831us) service_pool.cc:224] Handling call
0709 12:24:56.490500 (+   363us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.499769 (+  9269us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.499957 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:56.825158 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:56.824927 31825 rpcz_store.cc:276] 0709 12:24:55.120118 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.118166 (+998048us) service_pool.cc:224] Handling call
0709 12:24:56.118488 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.118701 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:24:56.466161 (+347460us) log.cc:844] Serialized 14 byte log entry
0709 12:24:56.802838 (+336677us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:56.802904 (+    66us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:24:56.802951 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.803025 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:56.803164 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":598992,"prepare.run_cpu_time_us":62,"prepare.run_wall_time_us":60,"raft.queue_time_us":5261,"thread_start_us":605457,"threads_started":2}
I20260709 12:24:56.558410 31753 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 33. Current term is 34. Ops: []
I20260709 12:24:56.676317 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3731) took 3547 ms. Trace:
I20260709 12:24:56.828367 31824 rpcz_store.cc:276] 0709 12:24:53.128981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.341540 (+1212559us) service_pool.cc:224] Handling call
0709 12:24:55.250179 (+908639us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:55.250662 (+   483us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:55.250706 (+    44us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:55.265023 (+ 14317us) log.cc:844] Serialized 26 byte log entry
0709 12:24:55.515262 (+250239us) raft_consensus.cc:1645] Marking committed up to 11
0709 12:24:55.515315 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:55.515404 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:55.676174 (+160770us) raft_consensus.cc:1685] finished
0709 12:24:55.676195 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:56.675903 (+999708us) spinlock_profiling.cc:243] Waited 908 ms on lock 0x7b5400166cd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:56.676224 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3073,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":9558,"spinlock_wait_cycles":2088413056,"thread_start_us":215360,"threads_started":1,"wal-append.queue_time_us":268349,"wal-append.run_cpu_time_us":2155,"wal-append.run_wall_time_us":1127274}
I20260709 12:24:56.830340 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:56.631022  9089 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:24:56.808123 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3754) took 1653 ms. Trace:
I20260709 12:24:56.831861 31822 rpcz_store.cc:276] 0709 12:24:55.153992 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.613086 (+1459094us) service_pool.cc:224] Handling call
0709 12:24:56.613522 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.807053 (+193531us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.807271 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:56.833784  8530 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.834410  8530 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.510291  9067 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.325s
I20260709 12:24:56.837080  8530 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:56.734421  9055 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:56.669224  9116 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.219s	user 0.000s	sys 0.001s
W20260709 12:24:56.589836  8916 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.611s	user 0.015s	sys 0.007s
W20260709 12:24:56.587656  8940 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.960s	user 0.009s	sys 0.012s
W20260709 12:24:56.745376  8693 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.528s	user 0.010s	sys 0.000s
W20260709 12:24:56.768289  9033 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.615s	user 0.013s	sys 0.008s
I20260709 12:24:56.822731  8002 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:24:56.856320  8002 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:56.508435  9062 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.335s
I20260709 12:24:56.831573  9002 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:24:56.829375 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:56.758617  9130 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:56.506549  8948 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:24:56.839128 31822 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:24:56.844120  8943 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.840235 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:56.615056  8924 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:56.652971  8922 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:56.723186  9120 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.267s	user 0.002s	sys 0.002s
I20260709 12:24:56.509684 31818 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
W20260709 12:24:56.806866 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 302423us
I20260709 12:24:56.866305  8940 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:56.544034  9081 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.613971  9040 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:56.612955  9060 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:56.693693  8949 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:56.842238 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:56.546495  9085 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 33 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:56.817569  9084 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:24:56.606719  8425 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.822753  9068 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:56.806901  8787 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:24:56.748049  9009 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.593s	user 0.019s	sys 0.000s
I20260709 12:24:56.831136  9030 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:56.865684 31748 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
W20260709 12:24:56.757325  8966 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.772s	user 0.010s	sys 0.016s
I20260709 12:24:56.866766 31824 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:56.838296  9093 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:56.838739  9097 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.899096  8943 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:56.873162 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3939) took 1167 ms. Trace:
I20260709 12:24:56.874555 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1680) took 1465 ms. Trace:
I20260709 12:24:56.946874  8787 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:56.947181 31822 rpcz_store.cc:276] 0709 12:24:55.409327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.833326 (+1423999us) service_pool.cc:224] Handling call
0709 12:24:56.833612 (+   286us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.874216 (+ 40604us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.874479 (+   263us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:56.948300  8787 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:56.948917 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:56.802855 31673 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.003s	sys 0.009s
I20260709 12:24:56.943207  8966 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:56.878350 31749 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:56.951983  8972 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.876022  8002 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 21.14, Last appended by leader: 14, Current term: 23, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.954123  8945 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:56.955192  9140 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.057s	user 0.000s	sys 0.001s
I20260709 12:24:56.888435  8552 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:56.898310 31744 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:24:56.852749  9131 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:56.863680  8942 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:56.869488  8998 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:56.927626  9040 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:56.555732  8201 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
W20260709 12:24:56.903185 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:56.549759  8781 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:56.930573  9030 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:56.904979  9126 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.057s	user 0.001s	sys 0.000s
I20260709 12:24:56.881837  9130 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:56.906844  8992 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.923s	user 0.009s	sys 0.016s
I20260709 12:24:56.852288  9055 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:56.864035  9033 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:56.984058 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:56.908474 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1503) took 1249 ms. Trace:
I20260709 12:24:56.902824  8922 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:56.987397  8922 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:56.989920  9033 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:56.990727  8922 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:56.909888  9117 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.153s	user 0.000s	sys 0.001s
I20260709 12:24:56.873082  9062 consensus_queue.cc:1243] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 28.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
I20260709 12:24:56.992547  9055 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:56.960004  9105 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:56.993322 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3115) took 1033 ms. Trace:
I20260709 12:24:56.956081  9097 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:56.920475 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 78339.5us
I20260709 12:24:56.995023 31827 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: [25.19-27.20]
I20260709 12:24:56.995383 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:56.914016  8924 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:56.996187  9131 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:56.997205  8924 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.871799 31672 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:24:56.999579  8924 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:56.921697  9085 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.000998  9130 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:56.980396  9030 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:56.953142  8966 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:56.946360 31678 rpcz_store.cc:276] 0709 12:24:55.705500 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.960716 (+255216us) service_pool.cc:224] Handling call
0709 12:24:55.995451 (+ 34735us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:55.995772 (+   321us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:24:56.485533 (+489761us) log.cc:844] Serialized 14 byte log entry
0709 12:24:56.869504 (+383971us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:56.869579 (+    75us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:24:56.869649 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.872953 (+  3304us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:56.873088 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1379,"prepare.queue_time_us":131368,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":34,"raft.queue_time_us":1548,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":7759,"spinlock_wait_cycles":10984320,"thread_start_us":846034,"threads_started":2,"wal-append.queue_time_us":394242}
I20260709 12:24:56.902752 31818 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: [32.19-32.19]
I20260709 12:24:57.004577 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3939) took 1528 ms. Trace:
I20260709 12:24:57.005090 31818 rpcz_store.cc:276] 0709 12:24:55.476494 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.902127 (+1425633us) service_pool.cc:224] Handling call
0709 12:24:56.902495 (+   368us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.004182 (+101687us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.004495 (+   313us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:56.902467  8948 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:56.984936 31669 rpcz_store.cc:276] 0709 12:24:55.658837 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.951687 (+292850us) service_pool.cc:224] Handling call
0709 12:24:55.951947 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.003558 (+ 51611us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.003610 (+    52us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:56.003670 (+    60us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:56.003717 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.908206 (+904489us) spinlock_profiling.cc:243] Waited 51.3 ms on lock 0x7b54001b2bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:56.908260 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:56.908410 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":118027008}
I20260709 12:24:56.996743  9068 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:56.918174  9081 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:56.963647  8998 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:56.993743 31751 rpcz_store.cc:276] 0709 12:24:55.959707 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.959893 (+   186us) service_pool.cc:224] Handling call
0709 12:24:55.960193 (+   300us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:55.960524 (+   331us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:55.960553 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:55.961060 (+   507us) log.cc:844] Serialized 26 byte log entry
0709 12:24:56.499153 (+538093us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:24:56.499194 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.499262 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:56.909282 (+410020us) spinlock_profiling.cc:243] Waited 3.62 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:56.992979 (+ 83697us) raft_consensus.cc:1685] finished
0709 12:24:56.993004 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:56.993254 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":900,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"spinlock_wait_cycles":8336640,"thread_start_us":537796,"threads_started":1,"wal-append.queue_time_us":770298}
I20260709 12:24:56.952648  8972 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.007422 31822 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
W20260709 12:24:57.031350 32342 leader_election.cc:336] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 2.103s (SENT)
I20260709 12:24:57.100632 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:56.941761  9009 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:57.102018 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 70741.7us
I20260709 12:24:56.969515  8552 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.008092  8992 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.103014 31751 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:57.071180  8943 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:57.071493  9081 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:57.071733  9131 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:57.078823 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 72447.5us
I20260709 12:24:57.083148  9068 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:24:57.083461 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 99483.6us
I20260709 12:24:56.970840  8201 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:56.964231  9040 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:57.097476 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:56.984504  8945 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.091s
I20260709 12:24:57.110037 31751 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:24:57.132501 31751 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:56.944965  9093 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.119283 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:57.126119  8912 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:57.136895 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:57.148267 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1376) took 3451 ms. Trace:
W20260709 12:24:57.143503 31745 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3927) took 2377 ms (client timeout 1818 ms). Trace:
I20260709 12:24:57.148777 31748 rpcz_store.cc:276] 0709 12:24:53.687924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:53.688323 (+   399us) service_pool.cc:224] Handling call
0709 12:24:53.688670 (+   347us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.930865 (+3242195us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.139123 (+208258us) spinlock_profiling.cc:243] Waited 3.18 s on lock 0x7b5400000d54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:57.139407 (+   284us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":7305572224}
I20260709 12:24:57.144348 31670 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:57.146752 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3937) took 1599 ms. Trace:
W20260709 12:24:57.149210 31745 rpcz_store.cc:269] 0709 12:24:54.752965 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:54.761519 (+  8554us) service_pool.cc:224] Handling call
0709 12:24:57.129841 (+2368322us) spinlock_profiling.cc:243] Waited 598 ms on lock 0x7b5400064854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:57.130051 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1374861184}
I20260709 12:24:57.150586 31750 rpcz_store.cc:276] 0709 12:24:55.547007 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.547214 (+   207us) service_pool.cc:224] Handling call
0709 12:24:55.547555 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:55.771687 (+224132us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.146342 (+1374655us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54001ed7d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:57.146663 (+   321us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":449148288}
I20260709 12:24:57.140209  9055 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:24:57.151907 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:57.152485  9055 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:57.152767 32371 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:24:57.132151  8851 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.031s
I20260709 12:24:57.141759  9084 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.013s
I20260709 12:24:57.167578 31744 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 33 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
I20260709 12:24:57.129808  9040 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:57.013756  9069 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.136s
I20260709 12:24:57.094169  8949 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.055s
I20260709 12:24:57.169202 31747 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:57.147660  9131 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:57.170015 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 50848.4us
I20260709 12:24:57.179951 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:57.181833 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 148731us
I20260709 12:24:57.174961 32371 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:24:57.180169  8851 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:57.183207  8949 consensus_queue.cc:1243] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 26.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:24:57.153903  9055 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.182991  9131 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.186478  9004 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:24:57.202909  9004 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:57.203925  9131 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:57.203832  9004 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.19, Last appended by leader: 19, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.204775 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:57.209983  8945 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:57.211807 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:57.213032  8945 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:57.205207  9004 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:57.205735  8851 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:57.191309  8425 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.208257  8949 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:57.208729 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:57.248260  9067 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:24:57.248955  9067 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:57.181691  8952 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.251129  8952 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.251914  8552 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:57.176007  8966 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.020s
I20260709 12:24:57.168215  8201 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:57.254762 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.255044  8201 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:57.255146  9009 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:57.258023 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:57.180295  8948 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 22.12, Last appended by leader: 12, Current term: 25, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.238399  9134 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:57.218932  9069 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 27.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.043s
W20260709 12:24:57.271719 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.220782  9000 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.042s
W20260709 12:24:57.294947 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.240757  9085 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.026s
I20260709 12:24:57.269951  8952 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:57.263639  8949 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.279141  8552 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:24:57.261509  8425 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.332513  8972 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:57.340497  9002 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.311834  8916 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.343533  8916 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:57.280346  9124 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.627s	user 0.005s	sys 0.000s
W20260709 12:24:57.280994 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.281436 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3954) took 1060 ms. Trace:
W20260709 12:24:57.348451 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.348294 31744 rpcz_store.cc:276] 0709 12:24:56.220759 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.973624 (+752865us) service_pool.cc:224] Handling call
0709 12:24:56.973970 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.168098 (+194128us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.281152 (+113054us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b54000f2dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:57.281349 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":444544384}
W20260709 12:24:57.283159  9145 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.204s	user 0.001s	sys 0.001s
I20260709 12:24:57.285022  8693 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Stepping down as leader of term 22
I20260709 12:24:57.288125 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:57.291311 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:57.292520 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.361169 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:57.362116 31748 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:57.363195 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:57.293421 31821 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 3938) took 1817 ms (client timeout 1568 ms). Trace:
I20260709 12:24:57.348073  8916 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:57.363817 31821 rpcz_store.cc:269] 0709 12:24:55.476115 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.532138 (+ 56023us) service_pool.cc:224] Handling call
0709 12:24:57.293108 (+1760970us) spinlock_profiling.cc:243] Waited 331 ms on lock 0x7b540030e8d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:57.293319 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":760811904}
I20260709 12:24:57.364423 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:57.294080 31826 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: []
I20260709 12:24:57.250046  9067 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.261267  9009 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.300297 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:57.301972 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:57.349696 31677 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:24:57.366842 31748 raft_consensus.cc:2410] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 27 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:24:57.363448 31750 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:57.302517  9097 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:57.379024 31750 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: [26.17-26.17]
I20260709 12:24:57.391834 31750 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:57.395233 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.396085 31750 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:57.379515 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:57.390478 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:57.383459 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.311605  9075 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.310873  9069 consensus_queue.cc:1243] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 28.18 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 17
W20260709 12:24:57.384352 31752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 3947) took 1221 ms (client timeout 1529 ms). Trace:
W20260709 12:24:57.401440 31752 rpcz_store.cc:269] 0709 12:24:56.162686 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.273055 (+110369us) service_pool.cc:224] Handling call
0709 12:24:57.291125 (+1018070us) spinlock_profiling.cc:243] Waited 55.9 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:24:57.384255 (+ 93130us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":128642304}
I20260709 12:24:57.414691 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:24:57.415833 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 67820.8us
I20260709 12:24:57.417837 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:57.365597  9089 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.532s	user 0.012s	sys 0.000s
I20260709 12:24:57.420027 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:57.424973 31678 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:57.425052 31750 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 20.
I20260709 12:24:57.418550 31821 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:57.427845 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:57.428260 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.351302  9068 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.429886 32343 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:57.429572 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:57.353662  8693 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 22 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:57.436254 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.441182 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.438057 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:57.439770 31675 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 34: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:57.365808  8425 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:57.444521 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.381819  9091 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:57.380774  9033 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:57.443068 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:57.448633 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:57.442250 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.448757 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:57.445437 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.450507 31678 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 32: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:57.450431 31821 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 19.
I20260709 12:24:57.451099 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.452649 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:57.453131 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 23 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.454670 31674 raft_consensus.cc:2410] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 27 because replica has last-logged OpId of term: 25 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 17.
I20260709 12:24:57.456236 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.457813 31821 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:24:57.423825  9069 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.458613 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3941) took 1981 ms. Trace:
I20260709 12:24:57.456430  9091 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.479604 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.463294 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:57.466699 31821 rpcz_store.cc:276] 0709 12:24:55.477100 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.457148 (+1980048us) service_pool.cc:224] Handling call
0709 12:24:57.457564 (+   416us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.458319 (+   755us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.458549 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:57.460311 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:57.468881 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:24:57.473366 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.437983  9093 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:57.436968  8903 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.485479 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:57.416160  9009 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.487484 31675 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:24:57.486277 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:57.486379 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:57.462195  8787 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.513s	user 0.014s	sys 0.000s
I20260709 12:24:57.484758 32371 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 20.
I20260709 12:24:57.462667 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:57.487435 31821 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 19.
I20260709 12:24:57.508191 31674 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:57.508950 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:57.464520  9002 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:57.494982 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.428966  9075 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:57.505168 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.494131 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:57.510733 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3952) took 1301 ms. Trace:
I20260709 12:24:57.522564 31746 rpcz_store.cc:276] 0709 12:24:56.218847 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.870566 (+651719us) service_pool.cc:224] Handling call
0709 12:24:56.871061 (+   495us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:56.883328 (+ 12267us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.883378 (+    50us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:56.887614 (+  4236us) log.cc:844] Serialized 52 byte log entry
0709 12:24:56.970054 (+ 82440us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:56.970128 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.970244 (+   116us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:57.282352 (+312108us) spinlock_profiling.cc:243] Waited 16.6 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:57.490771 (+208419us) raft_consensus.cc:1685] finished
0709 12:24:57.490795 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.510659 (+ 19864us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11703,"raft.queue_time_us":16354,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":3340,"spinlock_wait_cycles":38127872,"thread_start_us":52181,"threads_started":1,"wal-append.queue_time_us":166509}
I20260709 12:24:57.466385  8992 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:57.515384 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52827.4us
I20260709 12:24:57.509965 31821 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:24:57.519512 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.531451 31675 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:57.532372 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.448529  8552 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.044s
I20260709 12:24:57.439110  9040 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.483736  9154 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.532903 31821 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:57.465762  9033 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.538169 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3942) took 2060 ms. Trace:
W20260709 12:24:57.538839  9151 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 12:24:57.539100  9156 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:57.539902  9156 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:57.540572  9040 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:57.541227  9154 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.543871  8912 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:57.544411  8912 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:57.545351  8912 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.445071  9068 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.546836 31745 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:57.548164  8949 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:57.503844  9113 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:57.548954  9113 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:57.549363  9113 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:57.504611  9000 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:57.504137  8425 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:57.550441  9000 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.533311 31677 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:24:57.548329 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.552737  9000 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:57.548532  9068 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:57.538607 31821 rpcz_store.cc:276] 0709 12:24:55.477381 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.529626 (+2052245us) service_pool.cc:224] Handling call
0709 12:24:57.529936 (+   310us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.537887 (+  7951us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.538111 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:57.544142 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3760) took 2152 ms. Trace:
I20260709 12:24:57.580859 31824 rpcz_store.cc:276] 0709 12:24:55.391536 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.830449 (+1438913us) service_pool.cc:224] Handling call
0709 12:24:56.866475 (+ 36026us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.931449 (+ 64974us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.543904 (+612455us) spinlock_profiling.cc:243] Waited 35.7 ms on lock 0x7b54003c91d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:57.544072 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":81995136}
W20260709 12:24:57.560638 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 142111us
W20260709 12:24:57.612964 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.627184 31669 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:57.627750 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3944) took 2149 ms. Trace:
I20260709 12:24:57.628337 31821 rpcz_store.cc:276] 0709 12:24:55.477981 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.626773 (+2148792us) service_pool.cc:224] Handling call
0709 12:24:57.627139 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.627358 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.627387 (+    29us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:57.627434 (+    47us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:57.627465 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.627533 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.627672 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:57.630682 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47326.5us
I20260709 12:24:57.627748 31669 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:57.482448  9069 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:57.648710 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 218780us
W20260709 12:24:57.658305 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.662889 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:57.664189  8949 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Leader pre-election won for term 21
I20260709 12:24:57.664354 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:57.665107  8949 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:57.665750  8949 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 20 FOLLOWER]: Advancing to term 21
I20260709 12:24:57.666978  9000 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:57.667327 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:57.667479  9000 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:57.668978  9000 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:57.514226  8903 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.638335 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:57.676259 31669 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:57.686112 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47872us
I20260709 12:24:57.520469  8851 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.103s
I20260709 12:24:57.684285 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:57.684532 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:57.693707 31669 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:57.685201 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:57.625237  9002 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.702585  9002 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.632067  8803 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.700871  9075 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:57.701964  9030 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.700s	user 0.010s	sys 0.003s
I20260709 12:24:57.624559  9131 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:57.705421  9002 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:57.689116 32343 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:57.718446  9134 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:57.719055 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3951) took 1509 ms. Trace:
I20260709 12:24:57.755591 31747 rpcz_store.cc:276] 0709 12:24:56.209152 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.831632 (+622480us) service_pool.cc:224] Handling call
0709 12:24:56.835710 (+  4078us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.180477 (+344767us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.718781 (+538304us) spinlock_profiling.cc:243] Waited 333 ms on lock 0x7b54002f62d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:57.718980 (+   199us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":766125440}
I20260709 12:24:57.756382 31669 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:24:57.734905 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:57.694038  8916 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:24:57.758229 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.759135  8916 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:57.760214 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.760594  8916 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:57.764513  9066 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.107s	user 0.010s	sys 0.016s
W20260709 12:24:57.765057  9066 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.125s	user 0.010s	sys 0.016s
W20260709 12:24:57.769690 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 80656.2us
I20260709 12:24:57.681998  9062 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.066s
W20260709 12:24:57.777384 31820 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3769) took 1796 ms (client timeout 2238 ms). Trace:
W20260709 12:24:57.779282 31820 rpcz_store.cc:269] 0709 12:24:55.981078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.999228 (+ 18150us) service_pool.cc:224] Handling call
0709 12:24:57.777135 (+1777907us) spinlock_profiling.cc:243] Waited 68.3 ms on lock 0x7b540043eed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:57.777303 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":157144832}
I20260709 12:24:57.781078 31820 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:57.681686  8787 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 30.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.062s
I20260709 12:24:57.788578 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1682) took 2104 ms. Trace:
I20260709 12:24:57.790825 31820 rpcz_store.cc:276] 0709 12:24:55.684599 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.780427 (+2095828us) service_pool.cc:224] Handling call
0709 12:24:57.780837 (+   410us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.787758 (+  6921us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.788416 (+   658us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:57.797888 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 112776us
I20260709 12:24:57.770555  9166 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.798967 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:57.809808 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1683) took 1982 ms. Trace:
W20260709 12:24:57.818107 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 128590us
I20260709 12:24:57.820286 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:57.823081 32371 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 34: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:24:57.828111 31792 connection.cc:256] RPC call timeout handler was delayed by -1.1662568100005046s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17500000000000002s for a response
I20260709 12:24:57.817874 31820 rpcz_store.cc:276] 0709 12:24:55.827180 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.808529 (+1981349us) service_pool.cc:224] Handling call
0709 12:24:57.809075 (+   546us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.809382 (+   307us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.809428 (+    46us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:57.809495 (+    67us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:57.809547 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.809615 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.809739 (+   124us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:57.833498 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1684) took 1967 ms. Trace:
I20260709 12:24:57.831881  9091 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:57.819126  9097 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:57.743975  9030 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.729317  8803 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.839465 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:57.769505  9131 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.825295  8811 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:57.843750 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3740) took 2720 ms. Trace:
I20260709 12:24:57.894164 31826 rpcz_store.cc:276] 0709 12:24:55.123761 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.406798 (+1283037us) service_pool.cc:224] Handling call
0709 12:24:57.293873 (+887075us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.365122 (+ 71249us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.843490 (+478368us) spinlock_profiling.cc:243] Waited 887 ms on lock 0x7b540030e8d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:57.843661 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2039072128}
W20260709 12:24:57.846110 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.847493 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:57.848248 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:57.848888 32374 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:24:57.798441  9161 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Leader pre-election won for term 21
W20260709 12:24:57.824201 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 330170us
I20260709 12:24:57.854765 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:57.855785 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:57.856212 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:57.863453  9170 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.144s	user 0.000s	sys 0.001s
W20260709 12:24:57.875635  9121 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.151s	user 0.000s	sys 0.011s
W20260709 12:24:57.876928  9159 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.124s	user 0.001s	sys 0.000s
W20260709 12:24:57.877792  9155 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.181s	user 0.000s	sys 0.001s
I20260709 12:24:57.822844  9002 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:57.896759 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3946) took 2018 ms. Trace:
I20260709 12:24:57.898077  8811 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:57.855659 31820 rpcz_store.cc:276] 0709 12:24:55.866367 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.832591 (+1966224us) service_pool.cc:224] Handling call
0709 12:24:57.832932 (+   341us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.833125 (+   193us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.833153 (+    28us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:57.833202 (+    49us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:57.833237 (+    35us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.833297 (+    60us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.833432 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:57.839285  8952 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:57.930871 31826 rpcz_store.cc:276] 0709 12:24:55.878424 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.895222 (+2016798us) service_pool.cc:224] Handling call
0709 12:24:57.896237 (+  1015us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.896438 (+   201us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.896464 (+    26us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:57.896507 (+    43us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:57.896539 (+    32us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.896596 (+    57us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.896702 (+   106us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:57.932749 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3947) took 2054 ms. Trace:
I20260709 12:24:57.911686 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3761) took 2407 ms. Trace:
I20260709 12:24:57.935392 31826 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:24:57.913626 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:57.916407 31673 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 19 index: 13. Preceding OpId from leader: term: 23 index: 15. (index mismatch)
I20260709 12:24:57.936311 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3948) took 2057 ms. Trace:
I20260709 12:24:57.824538  9166 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.905776  9161 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:57.900355  9148 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.623s	user 0.003s	sys 0.000s
W20260709 12:24:57.922318 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.925159 31825 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:57.908627 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:57.936758 31826 rpcz_store.cc:276] 0709 12:24:55.879051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.934537 (+2055486us) service_pool.cc:224] Handling call
0709 12:24:57.935052 (+   515us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.936002 (+   950us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.936231 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:57.951869 31826 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: []
I20260709 12:24:57.952770 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3951) took 2072 ms. Trace:
I20260709 12:24:57.952433 31673 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: [27.18-27.18]
I20260709 12:24:57.953342 31826 rpcz_store.cc:276] 0709 12:24:55.879862 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.951164 (+2071302us) service_pool.cc:224] Handling call
0709 12:24:57.951595 (+   431us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.952410 (+   815us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.952695 (+   285us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:57.934929 31824 rpcz_store.cc:276] 0709 12:24:55.504689 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.681309 (+2176620us) service_pool.cc:224] Handling call
0709 12:24:57.719165 (+ 37856us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.911288 (+192123us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.911324 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:57.911376 (+    52us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:57.911418 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.911499 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.911624 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":191570,"threads_started":1}
I20260709 12:24:57.933418 31820 rpcz_store.cc:276] 0709 12:24:55.878767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.931469 (+2052702us) service_pool.cc:224] Handling call
0709 12:24:57.931803 (+   334us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.932022 (+   219us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.932049 (+    27us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:57.932095 (+    46us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:57.932128 (+    33us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.932453 (+   325us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.932662 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:57.956260 31820 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:24:57.956880 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3768) took 1986 ms. Trace:
W20260709 12:24:57.918951 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.900624  9131 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:57.946444 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:57.927848  8201 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.672s	user 0.007s	sys 0.007s
I20260709 12:24:57.926501  9009 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:57.942278  9002 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:57.893450  9097 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.950577  9161 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 20 FOLLOWER]: Advancing to term 21
W20260709 12:24:57.960873 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 52373us
I20260709 12:24:57.967916 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1509) took 1155 ms. Trace:
I20260709 12:24:58.019748 31672 rpcz_store.cc:276] 0709 12:24:56.812653 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.812808 (+   155us) service_pool.cc:224] Handling call
0709 12:24:56.813170 (+   362us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.998121 (+184951us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.967624 (+969503us) spinlock_profiling.cc:243] Waited 58.3 ms on lock 0x7b54000479d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:57.967848 (+   224us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":134012672}
I20260709 12:24:57.969403 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1508) took 1160 ms. Trace:
I20260709 12:24:58.021708 31670 rpcz_store.cc:276] 0709 12:24:56.809134 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.810811 (+  1677us) service_pool.cc:224] Handling call
0709 12:24:57.969145 (+1158334us) spinlock_profiling.cc:243] Waited 262 ms on lock 0x7b5400046ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:57.969333 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":603238144}
I20260709 12:24:57.969929 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3944) took 1813 ms. Trace:
I20260709 12:24:58.023119 31749 rpcz_store.cc:276] 0709 12:24:56.156114 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.161010 (+  4896us) service_pool.cc:224] Handling call
0709 12:24:56.161414 (+   404us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.952507 (+791093us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.969678 (+1017171us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b54004056d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:57.969866 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1648226176}
I20260709 12:24:57.971153 31675 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 26 index: 15. (term mismatch)
I20260709 12:24:58.024500 31675 pending_rounds.cc:85] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 14
I20260709 12:24:58.024766 31675 pending_rounds.cc:107] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 24.15
I20260709 12:24:58.025030 31675 raft_consensus.cc:2889] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:57.973640 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1679) took 2751 ms. Trace:
I20260709 12:24:57.900996  8998 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:57.957226 31820 rpcz_store.cc:276] 0709 12:24:55.970046 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.955381 (+1985335us) service_pool.cc:224] Handling call
0709 12:24:57.955776 (+   395us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.956625 (+   849us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.956823 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:24:58.002868 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55080.8us
I20260709 12:24:58.069177 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:58.069991 31820 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:58.070981 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:57.987129 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:58.071846 31820 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:24:57.997568 31821 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 30 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:24:58.072786 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:58.073799  8851 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
W20260709 12:24:58.075806 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 88763.7us
I20260709 12:24:58.028843 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3940) took 2552 ms. Trace:
I20260709 12:24:57.987743 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:57.997176 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3779) took 1093 ms. Trace:
I20260709 12:24:58.008399 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1685) took 2042 ms. Trace:
W20260709 12:24:58.010583 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 172965us
I20260709 12:24:58.079698 31822 rpcz_store.cc:276] 0709 12:24:56.903658 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.948512 (+ 44854us) service_pool.cc:224] Handling call
0709 12:24:57.996960 (+1048448us) spinlock_profiling.cc:243] Waited 57.4 ms on lock 0x7b540043d5d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:57.997089 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":132072192}
I20260709 12:24:58.079787 31824 rpcz_store.cc:276] 0709 12:24:55.965563 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.955291 (+1989728us) service_pool.cc:224] Handling call
0709 12:24:58.007649 (+ 52358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.007927 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.007970 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:58.008032 (+    62us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:24:58.008082 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.008163 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.008315 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:58.077266 31826 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 13. Preceding OpId from leader: term: 31 index: 15. (index mismatch)
W20260709 12:24:58.004930 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 317865us
I20260709 12:24:57.996904 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1497) took 2343 ms. Trace:
I20260709 12:24:58.078824 31818 rpcz_store.cc:276] 0709 12:24:55.476801 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.006159 (+1529358us) service_pool.cc:224] Handling call
0709 12:24:57.006549 (+   390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.174944 (+168395us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:57.174978 (+    34us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.186811 (+ 11833us) log.cc:844] Serialized 26 byte log entry
0709 12:24:57.422204 (+235393us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:57.422256 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.905610 (+483354us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b54004475d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:57.905667 (+    57us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.015912 (+110245us) raft_consensus.cc:1685] finished
0709 12:24:58.015936 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.028745 (+ 12809us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":386341760,"thread_start_us":234944,"threads_started":1,"wal-append.queue_time_us":341945}
W20260709 12:24:57.967322  9160 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.140s	user 0.001s	sys 0.000s
I20260709 12:24:58.007508 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:24:58.120685 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 113284us
I20260709 12:24:58.121704 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:58.088605 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.124607 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:58.088783 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3953) took 2208 ms. Trace:
I20260709 12:24:58.112128 31818 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:24:57.988065 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:58.125851 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3777) took 1660 ms. Trace:
W20260709 12:24:57.974205 32370 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 1522) took 1882 ms (client timeout 1563 ms). Trace:
I20260709 12:24:58.126399 31818 rpcz_store.cc:276] 0709 12:24:56.465734 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.111345 (+1645611us) service_pool.cc:224] Handling call
0709 12:24:58.111714 (+   369us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.125554 (+ 13840us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.125790 (+   236us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:58.129688 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1689) took 1466 ms. Trace:
I20260709 12:24:58.111151 31827 rpcz_store.cc:276] 0709 12:24:55.221792 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.836024 (+1614232us) service_pool.cc:224] Handling call
0709 12:24:56.836501 (+   477us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:56.995474 (+158973us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:57.973354 (+977880us) spinlock_profiling.cc:243] Waited 158 ms on lock 0x7b54002080d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:57.973577 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":363667328}
I20260709 12:24:58.117882 31824 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:58.130122 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:58.130795 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3774) took 1927 ms. Trace:
W20260709 12:24:58.126588 32370 rpcz_store.cc:269] 0709 12:24:56.091722 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.091873 (+   151us) service_pool.cc:224] Handling call
0709 12:24:57.973995 (+1882122us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54003f3254. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:57.974137 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2495308672}
I20260709 12:24:57.963222  8966 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:24:58.085273 31671 rpcz_store.cc:276] 0709 12:24:55.653239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.943755 (+290516us) service_pool.cc:224] Handling call
0709 12:24:55.964979 (+ 21224us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:56.052996 (+ 88017us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.053033 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:56.071069 (+ 18036us) log.cc:844] Serialized 52 byte log entry
0709 12:24:56.416473 (+345404us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:56.416522 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:56.416619 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:57.771841 (+1355222us) spinlock_profiling.cc:243] Waited 1.45 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:57.996560 (+224719us) raft_consensus.cc:1685] finished
0709 12:24:57.996592 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:57.996831 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":113098,"raft.queue_time_us":89652,"raft.run_cpu_time_us":143,"raft.run_wall_time_us":8929,"spinlock_wait_cycles":3328256,"thread_start_us":302280,"threads_started":1,"wal-append.queue_time_us":623935}
I20260709 12:24:57.983538  9038 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:24:57.978013  8912 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:58.090993 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3772) took 1944 ms. Trace:
I20260709 12:24:57.988835  8803 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:58.130169 31818 rpcz_store.cc:276] 0709 12:24:56.663712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.127999 (+1464287us) service_pool.cc:224] Handling call
0709 12:24:58.128446 (+   447us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.128687 (+   241us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.128723 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:58.129408 (+   685us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:58.129448 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.129511 (+    63us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.129630 (+   119us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:58.069285  9131 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.134126 31827 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:24:58.157997 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3780) took 1254 ms. Trace:
I20260709 12:24:58.158459 31827 rpcz_store.cc:276] 0709 12:24:56.903946 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.133511 (+1229565us) service_pool.cc:224] Handling call
0709 12:24:58.133809 (+   298us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.157590 (+ 23781us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.157910 (+   320us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:58.125464 31826 rpcz_store.cc:276] 0709 12:24:55.880364 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.954544 (+2074180us) service_pool.cc:224] Handling call
0709 12:24:58.076911 (+122367us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.083471 (+  6560us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.083678 (+   207us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:57.967617  9069 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:57.973026  9084 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:24:58.131485 31824 rpcz_store.cc:276] 0709 12:24:56.203082 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.081445 (+1878363us) service_pool.cc:224] Handling call
0709 12:24:58.117634 (+ 36189us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.130518 (+ 12884us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.130730 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:58.015769  9035 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.075067  9033 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:57.995267  8945 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:58.136664  8890 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:24:57.972222  9166 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:58.146413 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.092031  8948 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:58.152176 31818 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:24:58.227736 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1690) took 1102 ms. Trace:
I20260709 12:24:58.228132 31818 rpcz_store.cc:276] 0709 12:24:57.125439 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.151288 (+1025849us) service_pool.cc:224] Handling call
0709 12:24:58.151750 (+   462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.227426 (+ 75676us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.227676 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:58.092459  8992 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.153627 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:58.180783  9174 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.017066  9165 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.130s
I20260709 12:24:58.237744  9165 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:58.238155  9165 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:58.135651  8998 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:58.139745 31822 rpcz_store.cc:276] 0709 12:24:56.138009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.081431 (+1943422us) service_pool.cc:224] Handling call
0709 12:24:58.081873 (+   442us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.082127 (+   254us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.082170 (+    43us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:58.082233 (+    63us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:58.082297 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.082382 (+    85us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.082548 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:24:58.138243  9162 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.082s
I20260709 12:24:58.188941  9154 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:58.189008  9069 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:58.183108  9084 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:58.225971  9033 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:58.192760  9113 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.643s	user 0.012s	sys 0.000s
I20260709 12:24:58.193079  9035 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.193586 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3758) took 2802 ms. Trace:
I20260709 12:24:58.194307 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1681) took 2784 ms. Trace:
I20260709 12:24:58.194981  9171 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
W20260709 12:24:58.195284  9118 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.623s	user 0.000s	sys 0.019s
W20260709 12:24:58.195684 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:57.977180  9000 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.136842  9038 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:57.984638  8693 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 22.15, Last appended by leader: 15, Current term: 22, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.150259  9177 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.136965  9105 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:24:58.135758  9178 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:24:58.229589 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3942) took 2680 ms. Trace:
I20260709 12:24:57.990702  9030 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:58.216423  8942 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.129s	user 0.003s	sys 0.033s
I20260709 12:24:58.196389  8945 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:58.216245  8966 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.226780  8948 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:58.235054  8992 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.188303  9002 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:58.241626 32369 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:58.156045  9091 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.080s
I20260709 12:24:58.192538 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3945) took 2713 ms. Trace:
I20260709 12:24:58.372306 31821 rpcz_store.cc:276] 0709 12:24:55.479393 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.629780 (+2150387us) service_pool.cc:224] Handling call
0709 12:24:57.630265 (+   485us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:58.073113 (+442848us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.192224 (+119111us) spinlock_profiling.cc:243] Waited 367 ms on lock 0x7b54000fad54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:58.192461 (+   237us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":843925760}
I20260709 12:24:58.368558 32369 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:58.267522 31824 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:24:58.184744  8803 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.071s
I20260709 12:24:58.182653  9131 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.304239 31825 rpcz_store.cc:276] 0709 12:24:55.390956 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.827267 (+1436311us) service_pool.cc:224] Handling call
0709 12:24:56.827539 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.950130 (+1122591us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.193305 (+243175us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b54001982d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:58.193526 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2523632896}
I20260709 12:24:58.305058 31819 rpcz_store.cc:276] 0709 12:24:55.409672 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.864143 (+1454471us) service_pool.cc:224] Handling call
0709 12:24:56.864533 (+   390us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.998895 (+134362us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:56.998948 (+    53us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:56.999015 (+    67us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:56.999066 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.194066 (+1195000us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b540043cbd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:58.194118 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.194251 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":308390784}
I20260709 12:24:58.305721 31818 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: [25.16-27.17]
W20260709 12:24:58.378472 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 295545us
I20260709 12:24:58.310583 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:58.374681 32369 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.380688 32369 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:58.308641  9113 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.384131 32378 raft_consensus.cc:2393] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:58.280164  8916 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.075s
W20260709 12:24:58.278980  8949 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.613s	user 0.019s	sys 0.000s
I20260709 12:24:58.386654  8949 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.387028  9038 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.388554  9038 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:58.269484  9154 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.349731 31753 rpcz_store.cc:276] 0709 12:24:55.549419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:55.690271 (+140852us) service_pool.cc:224] Handling call
0709 12:24:55.690556 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:56.828251 (+1137695us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.229264 (+1401013us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x7b54002134d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:58.229512 (+   248us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1995233536}
I20260709 12:24:58.243026  9174 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.241199  8811 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.303851  9033 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.280969  8952 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.078s
W20260709 12:24:58.352943  9093 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.650s	user 0.010s	sys 0.036s
W20260709 12:24:58.358668  8942 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.272s	user 0.003s	sys 0.033s
I20260709 12:24:58.370718  8948 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:58.370393  8903 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:58.369365  8693 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
W20260709 12:24:58.398980 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 328378us
I20260709 12:24:58.368178  9105 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 22.12, Next index: 13, Last known committed idx: 12, Time since last communication: 0.034s
I20260709 12:24:58.390053  8949 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:58.378403  9097 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:58.337211  9165 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.355504  9177 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.369864  8966 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.360608  9084 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.409843  9171 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:58.410338  8916 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:24:58.410816  9093 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.708s	user 0.010s	sys 0.038s
I20260709 12:24:58.400434  9154 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:58.453696  9181 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.185s	user 0.001s	sys 0.000s
I20260709 12:24:58.457188  9174 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:58.458066  9113 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:58.448367  9165 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:58.461217  9035 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:58.433308  9171 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:58.438655  9105 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:24:58.441366  9084 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:24:58.458160  9161 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.477139  9171 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.479136  9171 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:58.498679  9178 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:24:58.531947 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3969) took 1144 ms. Trace:
I20260709 12:24:58.533211 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 3975) took 1043 ms. Trace:
W20260709 12:24:58.331660 31643 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.955s (SENT) [suppressed 31 similar messages]
I20260709 12:24:58.533013 31674 rpcz_store.cc:276] 0709 12:24:57.386935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.526147 (+139212us) service_pool.cc:224] Handling call
0709 12:24:57.526612 (+   465us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.530303 (+  3691us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.530351 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.531484 (+  1133us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.034271 (+502787us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:58.034346 (+    75us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.035015 (+   669us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.530790 (+495775us) raft_consensus.cc:1685] finished
0709 12:24:58.530889 (+    99us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.531526 (+   637us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3049,"raft.queue_time_us":3189,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":5494,"thread_start_us":406901,"threads_started":1,"wal-append.queue_time_us":560274}
I20260709 12:24:58.533878 31669 rpcz_store.cc:276] 0709 12:24:57.490136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.773445 (+283309us) service_pool.cc:224] Handling call
0709 12:24:57.773829 (+   384us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.789312 (+ 15483us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.789365 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.790657 (+  1292us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.361872 (+571215us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:24:58.531737 (+169865us) log.cc:844] Serialized 14 byte log entry
0709 12:24:58.532508 (+   771us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.532659 (+   151us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.532778 (+   119us) raft_consensus.cc:1685] finished
0709 12:24:58.532794 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.533044 (+   250us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":19378,"raft.run_cpu_time_us":189,"raft.run_wall_time_us":18299,"thread_start_us":506429,"threads_started":1,"wal-append.queue_time_us":616614}
W20260709 12:24:58.547299 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 474530us
I20260709 12:24:58.548095 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:58.550257 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.551432 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:58.553025 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.553563 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.548061 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.555272 31749 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 27 index: 18. (index mismatch)
I20260709 12:24:58.556384 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.556819 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.557641 31749 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:58.557760 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.559376 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:24:58.553580 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.560422 31750 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:24:58.561573 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.550493 31744 raft_consensus.cc:2410] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 32 because replica has last-logged OpId of term: 31 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 18.
I20260709 12:24:58.563285 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.564249 31751 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:58.565836 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.567056 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.568217 31751 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:24:58.568287 31821 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:58.405977 31824 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 22 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
W20260709 12:24:58.571110 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.571198 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:58.572547 31824 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:58.573307 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1691) took 1382 ms. Trace:
I20260709 12:24:58.576612 32377 raft_consensus.cc:2368] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21.
I20260709 12:24:58.573820 31826 rpcz_store.cc:276] 0709 12:24:57.190517 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.161105 (+970588us) service_pool.cc:224] Handling call
0709 12:24:58.161586 (+   481us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:58.389063 (+227477us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.389091 (+    28us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:58.389667 (+   576us) log.cc:844] Serialized 52 byte log entry
0709 12:24:58.571409 (+181742us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:58.572164 (+   755us) log.cc:844] Serialized 14 byte log entry
0709 12:24:58.572619 (+   455us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.572723 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.572810 (+    87us) raft_consensus.cc:1685] finished
0709 12:24:58.572821 (+    11us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.573073 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4371,"raft.queue_time_us":270438,"raft.run_cpu_time_us":233,"raft.run_wall_time_us":13213,"thread_start_us":407848,"threads_started":2,"wal-append.queue_time_us":90232}
I20260709 12:24:58.592950 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.565459 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.594193 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:58.594825 31749 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 33 because replica has last-logged OpId of term: 31 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
I20260709 12:24:58.587033 31750 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 32 because replica has last-logged OpId of term: 31 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 13.
I20260709 12:24:58.564251 31744 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:24:58.606388 31746 raft_consensus.cc:2368] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21.
I20260709 12:24:58.613489  8949 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.643442  8992 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:58.440699  9066 consensus_queue.cc:1175] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "b95aeeaf4c51442b8092cff1de674f2b" responder_term: 27 status { last_received { term: 26 index: 15 } last_committed_idx: 14 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:24:58.648121  9137 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.559s	user 0.004s	sys 0.010s
I20260709 12:24:58.442756  9097 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.028s
I20260709 12:24:58.656163  9004 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.657423  9004 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.639406 31824 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: [27.18-27.18]
I20260709 12:24:58.666790 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.437067  8916 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:58.467767  9105 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:58.682564  8890 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:58.700843 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:58.700991  8890 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.15, Last appended by leader: 15, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.701683  9161 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:58.685019 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:58.704794 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1678) took 3486 ms. Trace:
W20260709 12:24:58.690999 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.540s	user 0.000s	sys 0.007s
W20260709 12:24:58.707114 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.000s	sys 0.007s
I20260709 12:24:58.698313  9093 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:24:58.708055 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 597818us
I20260709 12:24:58.708380  9093 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.710017 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:24:58.403036  8945 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 27.17, Last appended by leader: 17, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:58.690747 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 297126us
I20260709 12:24:58.711237  8992 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:58.712064  8945 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:58.685053  8966 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:58.715245 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:58.716187  8916 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.717680 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:58.691437 31826 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 32: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:24:58.720579  8966 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:58.721127  8966 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:24:58.721170  9000 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:58.721709  8966 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:58.723910  9131 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:58.724500 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:58.707167 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.718080  9154 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:58.722467  9000 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:24:58.614738  8949 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.688561 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1520) took 1050 ms. Trace:
W20260709 12:24:58.683691  9204 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.053s	user 0.000s	sys 0.001s
I20260709 12:24:58.731828 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:58.698072  9209 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:24:58.729562  9000 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:58.731721 31675 rpcz_store.cc:276] 0709 12:24:57.637990 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.895160 (+257170us) service_pool.cc:224] Handling call
0709 12:24:57.970738 (+ 75578us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.025486 (+ 54748us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.688044 (+662558us) spinlock_profiling.cc:243] Waited 75.2 ms on lock 0x7b5400046ad0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:58.688351 (+   307us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":173023488}
I20260709 12:24:58.736037 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.705394 31823 rpcz_store.cc:276] 0709 12:24:55.217935 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.661708 (+1443773us) service_pool.cc:224] Handling call
0709 12:24:57.778036 (+1116328us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.778369 (+   333us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.778421 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.799077 (+ 20656us) log.cc:844] Serialized 26 byte log entry
0709 12:24:58.141029 (+341952us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:58.141083 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.141180 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.569383 (+428203us) raft_consensus.cc:1685] finished
0709 12:24:58.569424 (+    41us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.704188 (+134764us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b540043eed0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:24:58.704516 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2566145408,"thread_start_us":294077,"threads_started":1,"wal-append.queue_time_us":544713}
I20260709 12:24:58.738996 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:58.741704 31744 raft_consensus.cc:2410] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 30 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:24:58.744158 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.745827 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:58.748952 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:58.753129  8912 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.602s	user 0.009s	sys 0.013s
I20260709 12:24:58.755234 31750 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 32 index: 19. (term mismatch)
I20260709 12:24:58.755921 31750 pending_rounds.cc:85] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 18
I20260709 12:24:58.756289 31750 pending_rounds.cc:107] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 31.19
I20260709 12:24:58.756668 31750 raft_consensus.cc:2889] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:58.740792  9000 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:24:58.758952 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:58.762712  8949 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:58.764148 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:58.765210 31823 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:58.767143  9105 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 24.13, Last appended by leader: 13, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.769168  9105 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:58.770947 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:58.772550 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:58.773516 31822 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 19.
I20260709 12:24:58.756120 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:58.774133 31826 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:24:58.774243 31825 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:58.707100  9097 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:58.755321  9004 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:58.737605 31792 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:58.778311 31821 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:24:58.780051 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:58.788831 31819 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:24:58.792966  9157 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.086s	user 0.004s	sys 0.001s
W20260709 12:24:58.799631  9169 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.563s	user 0.002s	sys 0.000s
I20260709 12:24:58.803686 31746 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 28: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:58.805620 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1688) took 2362 ms. Trace:
W20260709 12:24:58.796849  9168 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.864s	user 0.003s	sys 0.000s
I20260709 12:24:58.812831 31826 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:24:58.788997  8992 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:24:58.810113 31820 rpcz_store.cc:276] 0709 12:24:56.442900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.084144 (+1641244us) service_pool.cc:224] Handling call
0709 12:24:58.084527 (+   383us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:58.349169 (+264642us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:58.349271 (+   102us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:58.382804 (+ 33533us) log.cc:844] Serialized 52 byte log entry
0709 12:24:58.691312 (+308508us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:24:58.691454 (+   142us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.691597 (+   143us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.805194 (+113597us) raft_consensus.cc:1685] finished
0709 12:24:58.805214 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.805426 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1260,"raft.queue_time_us":320778,"raft.run_cpu_time_us":190,"raft.run_wall_time_us":16255,"thread_start_us":288765,"threads_started":2,"wal-append.queue_time_us":401833}
I20260709 12:24:58.810444  9193 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:24:58.827569  9193 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.828271  9193 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.828902 31752 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:58.831406 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1392) took 1472 ms. Trace:
I20260709 12:24:58.831877 31745 rpcz_store.cc:276] 0709 12:24:57.359252 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.359589 (+   337us) service_pool.cc:224] Handling call
0709 12:24:57.359980 (+   391us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:57.547303 (+187323us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.830907 (+1283604us) spinlock_profiling.cc:243] Waited 187 ms on lock 0x7b54004042d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:58.831223 (+   316us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":429147392}
I20260709 12:24:58.832391  9193 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:58.816270 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.818027 31821 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: [27.16-27.16]
W20260709 12:24:58.819288 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 112656us
I20260709 12:24:58.819425  9033 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:58.837436  8948 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.839156 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1402) took 1411 ms. Trace:
I20260709 12:24:58.820364 31819 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:24:58.822443  9177 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:58.825403 31825 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:24:58.826438  9093 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:58.826694 31820 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:24:58.823818  8992 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:58.856178 31747 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:58.839736 31748 rpcz_store.cc:276] 0709 12:24:57.419623 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:57.420669 (+  1046us) service_pool.cc:224] Handling call
0709 12:24:57.421012 (+   343us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:57.549223 (+128211us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:57.549269 (+    46us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:57.549773 (+   504us) log.cc:844] Serialized 26 byte log entry
0709 12:24:57.732309 (+182536us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:24:57.732373 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.795796 (+1063423us) spinlock_profiling.cc:243] Waited 2.8 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:24:58.830152 (+ 34356us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b5400000d54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:24:58.830231 (+    79us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:24:58.830286 (+    55us) raft_consensus.cc:1685] finished
0709 12:24:58.830302 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.830511 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1954,"raft.run_cpu_time_us":87,"raft.run_wall_time_us":87,"spinlock_wait_cycles":300380928,"thread_start_us":182110,"threads_started":1,"wal-append.queue_time_us":303328}
I20260709 12:24:58.871197  8992 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:58.856710 31745 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:24:58.857424 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:58.858254 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:24:58.881412 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 493889us
I20260709 12:24:58.883498  9075 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:58.885064  9203 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:58.885625  9203 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:58.885855  9004 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.868314  8811 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:24:58.870278  8998 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.605s	user 0.016s	sys 0.000s
I20260709 12:24:58.886710 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:58.890580 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.890823 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 23 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.895190  8916 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:24:58.886737  9203 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.896268 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:58.897064  9203 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:58.898090 31744 raft_consensus.cc:2410] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 25 because replica has last-logged OpId of term: 24 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 16.
I20260709 12:24:58.898409 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:58.899046  9193 raft_consensus.cc:2764] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election decision vote started in defunct term 25: won
I20260709 12:24:58.899501 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.899744  9093 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:24:58.906204 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:58.889089 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 171703us
I20260709 12:24:58.907001 31749 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: [24.13-24.13]
I20260709 12:24:58.904896 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.886680  9004 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.904726  9177 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:24:58.908037  8966 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:58.908831 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:58.910591 31643 leader_election.cc:336] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.568s (SENT)
I20260709 12:24:58.910667  9004 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:58.911463 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:58.915205  9002 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:24:58.910323  8966 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:58.920226  9093 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 30 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:24:58.921490 31792 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:58.923566 32369 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 28 index: 18. (term mismatch)
I20260709 12:24:58.923998 32369 pending_rounds.cc:85] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 17
I20260709 12:24:58.923851  9193 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:58.924437 32369 pending_rounds.cc:107] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 27.18
I20260709 12:24:58.924837 32369 raft_consensus.cc:2889] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:58.925045  9193 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:24:58.925304  9179 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.693s	user 0.001s	sys 0.000s
I20260709 12:24:58.925740 32369 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1529) took 2081 ms. Trace:
I20260709 12:24:58.927043 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:58.927973  9110 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 28 current term: 28 is higher.
I20260709 12:24:58.928661  9035 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:58.929870  8992 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.930788  8949 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.930946  8945 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:58.930951 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:58.931941  8949 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.932137  8945 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.932411  9067 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:24:58.936642  9035 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:58.939853 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:58.936659  9004 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.009s
I20260709 12:24:58.947067  9000 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.948253 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:58.949720 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.949877 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:58.951596  9177 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:24:58.953177  9177 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:24:58.953920  9177 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:24:58.955284 31748 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 17. (term mismatch)
I20260709 12:24:58.955695 31748 pending_rounds.cc:85] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 16
I20260709 12:24:58.955963 31748 pending_rounds.cc:107] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 27.17
I20260709 12:24:58.956214 31748 raft_consensus.cc:2889] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:58.956624 31792 leader_election.cc:400] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21.
I20260709 12:24:58.956985 31792 leader_election.cc:403] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:58.957623  9035 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:24:58.958194  9000 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:58.959676  8949 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:58.960687  9180 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:24:58.921867  9093 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.14, Last appended by leader: 14, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:58.967873  9002 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 26.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:58.968798  9002 consensus_queue.cc:1243] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 27.15 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:24:58.970391  9202 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:58.970847  9202 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:24:58.972170  9202 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 26.14, Last appended by leader: 14, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:58.974180  9202 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:58.975160  9033 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:58.978948  9161 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:24:58.982213  9113 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:58.926144 32369 rpcz_store.cc:276] 0709 12:24:56.844069 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:56.844206 (+   137us) service_pool.cc:224] Handling call
0709 12:24:56.863183 (+ 18977us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.241490 (+1378307us) spinlock_profiling.cc:243] Waited 129 ms on lock 0x7b540071d7a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcd4cdc0 00007fe8fcdd71c0 00007fe8fcdd8231 00007fe8fcdd4d8f 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c
0709 12:24:58.925413 (+683923us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.925641 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":297206016}
W20260709 12:24:58.983801 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.984858 31753 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
W20260709 12:24:58.987156 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.940805 31750 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
W20260709 12:24:58.988838 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:58.989465 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3782) took 1848 ms. Trace:
W20260709 12:24:59.021456 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.026279  8945 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:24:59.031427 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.031567  9033 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:59.032600 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.034902 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:59.035831 31675 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:59.041200  9112 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:24:59.041388  9009 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:59.041849  9112 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:59.042374  9009 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:54.772724 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:24:59.055007 31825 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: [25.19-27.20]
I20260709 12:24:59.055822 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.057039 31826 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:24:59.056555 31570 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 15 to 17, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 17 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:24:59.058341 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:59.058835 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 123533us
I20260709 12:24:59.060030  8945 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:24:59.062278  9180 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:59.062897  9180 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:59.063092 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.063977 31672 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
W20260709 12:24:59.064216 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 73126.6us
I20260709 12:24:59.066053  8803 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:59.067970  8803 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:59.069504 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:59.069974  8693 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.647s	user 0.013s	sys 0.001s
I20260709 12:24:59.070659 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.071318 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.071501 31676 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
W20260709 12:24:59.050388 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.071034 31751 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 22 index: 15. Preceding OpId from leader: term: 23 index: 15. (term mismatch)
I20260709 12:24:59.071064 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:59.074155 31751 pending_rounds.cc:85] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 14
I20260709 12:24:59.076129 31751 pending_rounds.cc:107] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 22.15
I20260709 12:24:59.076426 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.072131 31669 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:24:59.076455 31751 raft_consensus.cc:2889] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:58.995679 31827 rpcz_store.cc:276] 0709 12:24:57.141274 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:58.159530 (+1018256us) service_pool.cc:224] Handling call
0709 12:24:58.159947 (+   417us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:24:58.160221 (+   274us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:24:58.401925 (+241704us) log.cc:844] Serialized 14 byte log entry
0709 12:24:58.813143 (+411218us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:24:58.834001 (+ 20858us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:24:58.889291 (+ 55290us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:58.918267 (+ 28976us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:24:58.989248 (+ 70981us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1572,"prepare.queue_time_us":291474,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":45,"raft.queue_time_us":38839,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":184,"thread_start_us":541029,"threads_started":2,"wal-append.queue_time_us":370195}
I20260709 12:24:59.063848  8948 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:59.092353 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.081655  9161 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.105893 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.107213 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.106889 31669 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29. Ops: []
W20260709 12:24:59.115829 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.093040  8811 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.124645 31677 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 22. Current term is 23. Ops: [21.14-22.15]
I20260709 12:24:59.117520 31676 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:24:59.120868 31674 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:59.081532  8803 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:59.103585  9033 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:59.090539  8949 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.027s
I20260709 12:24:59.139923 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.117348  8201 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.143621 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.147686 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.120023  8945 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.148105 31748 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 29 because replica has last-logged OpId of term: 25 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 15.
I20260709 12:24:59.149762 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.142722 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:59.156991 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.157032 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.158991 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.162289 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.166937 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:59.107372  9161 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:59.172695 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:59.175208 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:59.173658 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:59.155646  9193 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.180308 31820 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:59.175846 31826 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:24:59.147393  8590 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.045s
I20260709 12:24:59.182158 31827 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:24:59.147392  8811 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:59.192344 31669 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:24:59.193714  9000 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:59.196352  9165 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:59.196946  9165 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:59.191614  8948 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.198843 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:59.199446  9038 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.198879  9210 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.124773  9112 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:59.207049 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.188963  9166 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:59.192030 31819 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:24:59.222795 31676 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 13. Preceding OpId from leader: term: 25 index: 13. (term mismatch)
I20260709 12:24:59.223515 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 20 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.169963  8945 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:59.217356 31677 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:59.223182 31676 pending_rounds.cc:85] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 12
I20260709 12:24:59.228514 31676 pending_rounds.cc:107] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 24.13
I20260709 12:24:59.228874 31676 raft_consensus.cc:2889] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:59.228883 31749 raft_consensus.cc:2368] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
W20260709 12:24:59.243005  9084 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.601s	user 0.014s	sys 0.001s
I20260709 12:24:59.243813 31753 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:24:59.244015  9210 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:59.246760  8890 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:24:59.247488 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.248313 31673 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:24:59.250878 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 24 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.251586  8803 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:59.252002  9038 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:59.252143  9166 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:59.254900  9161 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:59.255491  9180 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:59.178182  8201 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:59.257812  9180 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:59.257920  9112 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:24:59.258066  8552 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:59.254361  9038 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:59.263011 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:59.259584 31744 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
W20260709 12:24:59.265966 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.258823  8552 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:59.273610  8552 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:59.262604  9178 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.012s
I20260709 12:24:59.276701 31669 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 19 FOLLOWER]: Advancing to term 21
I20260709 12:24:59.215516  8948 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:59.281325  9185 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.868s	user 0.004s	sys 0.000s
I20260709 12:24:59.281548  8948 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:59.286463 31749 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
I20260709 12:24:59.238108  9000 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.223124  9193 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:59.295624  8977 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:24:59.269176  8916 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:59.299122  8916 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:24:59.300907 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:59.301323  8916 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:59.302109 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:24:59.295925 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:59.302832  9075 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.000s
I20260709 12:24:59.304095 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:59.305075  9075 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:59.305483  9075 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:59.305629 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:59.305923  9075 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:59.306221  8945 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:59.306861  9203 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:24:59.307642  9203 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:24:59.307968 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:59.308248  9203 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:24:59.321834 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:59.322834 32369 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 21: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:24:59.323544  9009 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:59.323767 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:59.324445 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:59.325068 31819 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:24:59.325145 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:59.325177 31826 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:24:59.327600 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:59.328389  8977 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:59.330838  9000 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:59.333787  9166 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:59.335003  9210 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:59.329028  8977 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:59.337436  9191 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:24:59.337924  9191 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:24:59.339128  8977 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:59.340739  9035 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21.
I20260709 12:24:59.342149  8201 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:59.349915  9191 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:59.351462  9191 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:59.351737 31752 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:24:59.353431 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:59.354801 31672 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:24:59.356838 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 22 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:59.357692 31670 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:24:59.362612  8977 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:24:59.363273 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:59.363636  8977 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:24:59.364015 31671 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:59.364123  8977 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:24:59.365106  8977 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:24:59.364276 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62250.4us
I20260709 12:24:59.366530  8977 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:59.369652 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:59.369828 31824 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 21.
W20260709 12:24:59.371168 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.371109 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.346341  8992 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:59.371886 31675 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:24:59.372606 31673 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:59.372689 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:59.373812 31576 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 154) took 83113 ms (client timeout 15000 ms). Trace:
I20260709 12:24:59.374037 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.366155 31674 raft_consensus.cc:2410] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 27 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
W20260709 12:24:59.292016 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 182128us
I20260709 12:24:59.371910 31676 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
W20260709 12:24:59.385466 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 79932.4us
I20260709 12:24:59.369668 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:59.390476 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:59.392282 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.392997 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:59.391476 32378 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:24:59.402967 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:59.402462 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:59.426174 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.432710 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.434171 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:59.436412 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.437237 31752 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:24:59.391006  9165 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:59.438614 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:59.441706 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:24:59.433501 31746 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:24:59.447412 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.448657 31669 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:24:59.451931 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 21 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:24:59.460937 31751 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:24:59.430444  9000 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:24:59.473110 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:59.473523 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.468753 31744 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 21: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:24:59.464437 32378 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:59.470750 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:59.475131 31821 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
W20260709 12:24:59.478245 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 108688us
I20260709 12:24:59.387552  8201 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.075s
I20260709 12:24:59.481351 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.482059 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:59.487542 31752 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:59.485486 31669 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:24:59.496096 31746 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:24:59.497290 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:59.476989 31570 catalog_manager.cc:5697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 15 to 16, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 16 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:24:59.505082 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:59.506705 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:59.507793 31753 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:24:59.510012 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:24:59.516247 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:24:59.512753 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:59.520752 31752 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 30. Current term is 31. Ops: []
I20260709 12:24:59.480822  8811 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:59.526130 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.532335 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:59.532538 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62013.4us
I20260709 12:24:59.533066 31823 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:24:59.501902  9000 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.533957 31746 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
W20260709 12:24:59.374389 31576 rpcz_store.cc:269] 0709 12:23:36.329009 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:36.443303 (+114294us) service_pool.cc:224] Handling call
0709 12:24:59.373523 (+82930220us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:48.710260 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:48.767437 (+ 57177us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:48.767525 (+    88us) write_op.cc:435] Acquired schema lock
0709 12:24:48.767556 (+    31us) tablet.cc:662] Decoding operations
0709 12:24:48.783237 (+ 15681us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:48.783399 (+   162us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:48.783448 (+    49us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:24:48.784098 (+   650us) tablet.cc:701] Row locks acquired
0709 12:24:48.784169 (+    71us) write_op.cc:260] PREPARE: finished
0709 12:24:48.784447 (+   278us) write_op.cc:270] Start()
0709 12:24:48.784666 (+   219us) write_op.cc:276] Timestamp: P: 1783599888784353 usec, L: 0
0709 12:24:48.784729 (+    63us) op_driver.cc:348] REPLICATION: starting
0709 12:24:48.785752 (+  1023us) log.cc:844] Serialized 925 byte log entry
0709 12:24:49.840953 (+1055201us) op_driver.cc:464] REPLICATION: finished
0709 12:24:50.106489 (+265536us) write_op.cc:301] APPLY: starting
0709 12:24:50.106609 (+   120us) tablet.cc:1367] starting BulkCheckPresence
0709 12:24:50.466487 (+359878us) tablet.cc:1370] finished BulkCheckPresence
0709 12:24:50.466620 (+   133us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:24:50.468043 (+  1423us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:24:50.468306 (+   263us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:24:50.468374 (+    68us) write_op.cc:312] APPLY: finished
0709 12:24:50.468675 (+   301us) log.cc:844] Serialized 33 byte log entry
0709 12:24:50.468895 (+   220us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:50.468988 (+    93us) write_op.cc:454] Released schema lock
0709 12:24:50.469065 (+    77us) write_op.cc:341] FINISH: Updating metrics
0709 12:24:50.548571 (+ 79506us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:24:50.563829 (+ 15258us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5485779,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":241395,"apply.run_cpu_time_us":7070,"apply.run_wall_time_us":380984,"cfile_cache_hit":10,"cfile_cache_hit_bytes":12483,"num_ops":2,"prepare.queue_time_us":352163,"prepare.run_cpu_time_us":68953,"prepare.run_wall_time_us":166522,"raft.queue_time_us":427288,"raft.run_cpu_time_us":11166,"raft.run_wall_time_us":239922,"replication_time_us":1011878,"thread_start_us":838744,"threads_started":4,"wal-append.queue_time_us":149996,"wal-append.run_cpu_time_us":87052,"wal-append.run_wall_time_us":2533876}]]}
W20260709 12:24:59.542593 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:59.545840 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44307.9us
I20260709 12:24:59.549362 31669 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:24:59.533797 31744 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 31. Current term is 32. Ops: []
I20260709 12:24:59.559269 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.560791 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.562031 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:24:59.565084 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:24:59.567631 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45198.6us
I20260709 12:24:59.566529 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:59.577253 31670 raft_consensus.cc:2410] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 31 because replica has last-logged OpId of term: 29 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
I20260709 12:24:59.570844  9186 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:24:59.566916  9165 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.613641 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:24:59.583642  8916 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:59.621254 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:24:59.623692 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 110074us
W20260709 12:24:59.626550 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.557361  9000 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:59.575536  8890 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.041s
I20260709 12:24:59.628322  9009 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:24:59.632627 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:59.571866  8811 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.047s
I20260709 12:24:59.635322 31670 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 31 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:24:59.596998  9202 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.033s
I20260709 12:24:59.625336  9161 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.647357 31670 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 28 index: 16. (term mismatch)
I20260709 12:24:59.647861 31670 pending_rounds.cc:85] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 15
I20260709 12:24:59.648228 31670 pending_rounds.cc:107] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 27.16
I20260709 12:24:59.648501 31670 raft_consensus.cc:2889] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:24:59.619304  9165 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:59.632897  9009 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:24:59.658463 31672 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 33 index: 19. (term mismatch)
I20260709 12:24:59.660750 31672 pending_rounds.cc:85] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 18
I20260709 12:24:59.661056 31672 pending_rounds.cc:107] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 32.19
I20260709 12:24:59.661319 31672 raft_consensus.cc:2889] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:24:59.660210 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:59.662902 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 96476.3us
I20260709 12:24:59.639595  9093 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:24:59.666608 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:59.669274 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:24:59.662945  9009 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:24:59.646366  9193 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:59.728201 31673 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
W20260709 12:24:59.729265 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.730212 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:59.649684  9045 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.049s
W20260709 12:24:59.742098 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 108093us
W20260709 12:24:59.746717 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 86120.8us
I20260709 12:24:59.687024  9203 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:24:59.745849 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:24:59.728471  9178 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:24:59.668380  9075 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:24:59.769898 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:24:59.768386 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 79251.3us
I20260709 12:24:59.775876 31672 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:24:59.668717  9161 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:59.740701  9045 consensus_queue.cc:1243] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 25.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:24:59.732622  9089 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:24:59.790710 31576 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42070 (request call id 228) took 21536 ms (client timeout 15000 ms). Trace:
W20260709 12:24:59.796684 31576 rpcz_store.cc:269] 0709 12:24:38.239430 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.765424 (+21525994us) service_pool.cc:211] Skipping call since client already timed out
0709 12:24:59.775380 (+  9956us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:24:59.793571 31820 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:24:59.787218 31717 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:59.765031  8803 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:59.740403  9112 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:24:59.758667  9033 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:59.773829  9002 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:24:59.812546 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:59.771675  9178 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:24:59.816864 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 50046.8us
I20260709 12:24:59.818548 31819 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:24:59.820652 31671 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:24:59.820881 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:24:59.831969 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51361.9us
I20260709 12:24:59.830340 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:59.839932 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:24:59.821071  9178 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:24:59.844537 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 170595us
I20260709 12:24:59.843552 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:24:59.845165 31576 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42072 (request call id 246) took 20407 ms (client timeout 15000 ms). Trace:
W20260709 12:24:59.847234 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:24:59.821198  8992 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:24:59.849408  8992 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:24:59.850065  8992 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:24:59.850987 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:59.848683 31576 rpcz_store.cc:269] 0709 12:24:39.430874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.828123 (+20397249us) service_pool.cc:211] Skipping call since client already timed out
0709 12:24:59.838197 (+ 10074us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:24:59.853258 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:59.856532 31819 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:24:59.859681 31819 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 22 index: 12. Preceding OpId from leader: term: 25 index: 13. (index mismatch)
I20260709 12:24:59.862589 31819 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:24:59.804886  9203 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.032s
W20260709 12:24:59.871599 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 76360us
I20260709 12:24:59.884167 31642 leader_election.cc:400] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
W20260709 12:24:59.884369 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 301488us
I20260709 12:24:59.884583 31642 leader_election.cc:403] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 20 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:24:59.844882  8945 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:24:59.847628  9084 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:24:59.895850 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64435.4us
I20260709 12:24:59.830742  9112 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:24:59.894363  8945 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:24:59.892642  9225 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.160s	user 0.001s	sys 0.000s
I20260709 12:24:59.831171  9033 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:24:59.907238 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:59.889487  9162 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:24:59.890645  9000 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:24:59.910367 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:24:59.915993  9236 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.095s	user 0.000s	sys 0.001s
I20260709 12:24:59.917148  9112 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:24:59.919078  9224 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.256s	user 0.001s	sys 0.000s
I20260709 12:24:59.919658  9033 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:24:59.900454  9002 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election lost for term 20. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 20. Current term is 21.
I20260709 12:24:59.907951  8945 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:24:59.922833  9105 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:24:59.910043  9162 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:59.923837  9089 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.925583 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:59.927919 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:24:59.935559  9230 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:24:59.936018  8916 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:24:59.936676 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:59.937295 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:24:59.938839  9093 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:24:59.940440  8948 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:24:59.903064  8903 raft_consensus.cc:2764] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:24:59.898972  9084 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.012094  9084 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:24:59.944682 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 67609.6us
I20260709 12:25:00.015584 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:24:59.959162 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:00.018868 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 65971.2us
I20260709 12:25:00.019762 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:00.021183 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:00.021968  9203 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 31.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:00.023489  8890 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:24:59.901355  9165 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:00.027550 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:00.029064 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:24:59.962886  9226 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:24:59.965356 32376 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:24:59.966724 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:00.032125 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:00.033339 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:59.968364  9161 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:00.034816 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:00.035431 31821 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:25:00.035517 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:59.952251  9000 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:24:59.970847 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:00.037508 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66775.3us
I20260709 12:25:00.038190 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:00.039186 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:24:59.974193 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:24:59.992221 32377 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:25:00.001952 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:00.004774 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 100132us
I20260709 12:24:59.999013 32373 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:25:00.007951 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:00.036134 31827 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:25:00.034390 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:00.045944 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:00.047060 31671 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 35 because replica has last-logged OpId of term: 31 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 18.
W20260709 12:25:00.052378  9237 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:25:00.055413 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:00.055663 32371 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 30 index: 15. (index mismatch)
I20260709 12:24:59.993031  9162 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:24:59.998195  8916 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:00.077394 32342 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:00.044919  9166 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:00.045063  9000 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.036s
W20260709 12:25:00.083009 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61930.3us
I20260709 12:25:00.088096  9035 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:00.092110  9252 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.042976  9249 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:00.092770  9069 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:00.094156  9004 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:00.095386  9009 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:00.096341  9075 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:00.096599  9246 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:00.042820  8949 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:25:00.099162  9227 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.428s	user 0.000s	sys 0.003s
I20260709 12:25:00.059643  8890 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260709 12:25:00.114959 31820 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
W20260709 12:25:00.118345 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:00.150041 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 132272us
W20260709 12:25:00.146263 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 68949.6us
I20260709 12:24:59.998728  9089 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:00.150936 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:00.021210  8992 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:00.134513  9249 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.173025 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:00.173102  8945 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:00.174034  9161 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:00.142616  9252 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:00.180171 31827 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: [30.14-31.15]
I20260709 12:25:00.188036 31677 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:00.050431  8948 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:00.191010  9075 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:00.191488  8948 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:00.187193 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 143097us
I20260709 12:25:00.188164 31820 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:25:00.194442  8948 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.004s
I20260709 12:25:00.199219 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:00.200681 31819 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:00.200878  9161 raft_consensus.cc:697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:00.042589  9084 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:00.211357 31752 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 27 index: 16. (index mismatch)
I20260709 12:25:00.202523 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:00.203424 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:00.204208 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:00.146983  9246 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:00.216424  9203 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:00.191720  9075 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:00.213241 31750 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:00.222795 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:00.233985 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:00.229293 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74178.1us
I20260709 12:25:00.187228  9249 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:00.202234  9093 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:00.242170 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:00.239880 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 194001us
I20260709 12:25:00.236127  8903 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:00.286973 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:00.227376  9246 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:00.297662 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:25:00.287657 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 145136us
I20260709 12:25:00.299371 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:00.301337 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:00.296890 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 155068us
I20260709 12:25:00.303714 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:00.316074 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:00.245587  9260 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:00.317521  9035 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:00.304453  9000 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:00.267645  8811 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.075s
I20260709 12:25:00.320966  9004 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:00.287350  9177 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.048s
I20260709 12:25:00.317770  9180 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:00.320245  8948 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:00.354532 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:00.313978  9261 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:00.311940  9210 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.166s
I20260709 12:25:00.478006  9165 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:25:00.623896 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 411561us
I20260709 12:25:00.474522  9004 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:00.628513 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 274206us
I20260709 12:25:00.482106  9180 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:00.331203  9178 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:00.490595  9000 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:00.628875  9165 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:00.639128  9180 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:00.636637  9004 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:00.632761  9009 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:00.650147  8949 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:25:00.724263 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 423005us
W20260709 12:25:00.734694 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 109592us
I20260709 12:25:00.743736 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:00.720695  9165 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:00.749259  9262 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.151s	user 0.001s	sys 0.000s
W20260709 12:25:00.750195  9264 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.131s	user 0.000s	sys 0.000s
W20260709 12:25:00.751143  9256 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.448s	user 0.001s	sys 0.000s
I20260709 12:25:00.663249  9261 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.084s
I20260709 12:25:00.729995  8992 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:00.764896 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:00.663813  8948 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.084s
I20260709 12:25:00.635123  9161 consensus_queue.cc:237] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:00.802088 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:00.664379  9263 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.089s
I20260709 12:25:00.722404  8945 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:00.756479  9084 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:00.816412 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 181142us
I20260709 12:25:00.815590 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:00.764093  9252 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:00.819234 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 64996.2us
I20260709 12:25:00.779227  9075 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:00.838769 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.593s	user 0.005s	sys 0.009s
W20260709 12:25:00.839267 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.593s	user 0.006s	sys 0.009s
W20260709 12:25:00.839546 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 593694us
W20260709 12:25:00.839972 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 96067.2us
I20260709 12:25:00.841324 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:00.841934 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:00.810702  8998 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:00.845149 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:00.846818 31753 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:00.809532  9089 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:00.923066 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 81348.8us
I20260709 12:25:00.959122 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:00.996481 31791 leader_election.cc:400] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 20 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 20. Current term is 21.
I20260709 12:25:00.845222  9075 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:00.957463  9177 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:01.030007 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 70997.2us
W20260709 12:25:01.033627 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.509s	user 0.001s	sys 0.013s
W20260709 12:25:01.034154 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.510s	user 0.001s	sys 0.013s
I20260709 12:25:00.993930  9035 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:01.022574 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:01.072469 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:01.061655 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 228226us
I20260709 12:25:00.843356  9000 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.104372 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:01.118741  9254 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.633s	user 0.002s	sys 0.000s
I20260709 12:25:01.120301  9283 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:01.121186  9283 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.066068  9210 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:01.081770  8998 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:01.079015  9260 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:01.139961 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 67783.2us
W20260709 12:25:01.140863 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66464.1us
I20260709 12:25:01.024595  9075 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.146821 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:25:01.158890 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 312553us
I20260709 12:25:01.129146  9178 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:01.140151  9275 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:01.168946  9275 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:01.169430  9275 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:01.169489  9276 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:01.170192  9276 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.083187  9177 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.086298  9035 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.165915  9045 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.172776  9276 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:01.167652  9267 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.415s	user 0.002s	sys 0.000s
W20260709 12:25:01.238179 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 92071.3us
W20260709 12:25:01.242849 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 159659us
I20260709 12:25:01.239753 31676 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
W20260709 12:25:01.240984 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 94330.3us
I20260709 12:25:01.255369 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:01.240525  8948 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:01.264341 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:01.267491 31821 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
W20260709 12:25:01.271720  9229 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.498s	user 0.004s	sys 0.028s
I20260709 12:25:01.271865 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:01.259686 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:01.276894 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:01.277581 32373 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:01.278427 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3978) took 2074 ms. Trace:
I20260709 12:25:01.279714 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:01.278863 31748 rpcz_store.cc:276] 0709 12:24:59.204055 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.333184 (+129129us) service_pool.cc:224] Handling call
0709 12:24:59.333566 (+   382us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:24:59.334025 (+   459us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:59.334057 (+    32us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:24:59.362740 (+ 28683us) log.cc:844] Serialized 52 byte log entry
0709 12:24:59.571744 (+209004us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:59.997663 (+425919us) log.cc:844] Serialized 14 byte log entry
0709 12:24:59.997936 (+   273us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.998006 (+    70us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:01.247281 (+1249275us) spinlock_profiling.cc:243] Waited 9.84 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:01.278119 (+ 30838us) raft_consensus.cc:1685] finished
0709 12:25:01.278223 (+   104us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.278358 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":38324,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":495,"raft.run_cpu_time_us":181,"raft.run_wall_time_us":182,"spinlock_wait_cycles":22628096,"thread_start_us":604361,"threads_started":2,"wal-append.queue_time_us":301035}
I20260709 12:25:01.280321 32371 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:01.246807  8949 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:01.288898 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:01.154745  9260 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.298092  9202 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:01.302922  9202 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:01.304225  9202 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 31.15, Last appended by leader: 15, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.305657  9202 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:01.291038 31821 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
W20260709 12:25:01.333242 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.254860  9276 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.244582  8992 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.039s
W20260709 12:25:01.344452 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56720.6us
I20260709 12:25:01.260692  9280 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.345917 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:01.246177  9045 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.364343  9045 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:01.356719 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.365144  9288 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:01.364506  9269 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.491s	user 0.001s	sys 0.000s
I20260709 12:25:01.365738  9288 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.367868  9035 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:01.368069  9288 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:01.368489  9035 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:01.369378  9272 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.289s	user 0.000s	sys 0.001s
I20260709 12:25:01.369190 32371 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:25:01.369050  9035 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.371265  9283 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:01.371379  9035 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:01.371793 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3990) took 1427 ms. Trace:
I20260709 12:25:01.371989  9035 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:01.373047  9271 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.528s	user 0.000s	sys 0.001s
I20260709 12:25:01.372534  9035 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.290740  8949 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.375980 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:01.380333 31669 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
W20260709 12:25:01.364205 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:01.347258  9191 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.995s	user 0.007s	sys 0.013s
W20260709 12:25:01.368608  9274 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.511s	user 0.000s	sys 0.002s
I20260709 12:25:01.372211 31820 rpcz_store.cc:276] 0709 12:24:59.944561 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.194523 (+249962us) service_pool.cc:224] Handling call
0709 12:25:00.198553 (+  4030us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:00.217276 (+ 18723us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:00.220473 (+  3197us) log.cc:844] Serialized 14 byte log entry
0709 12:25:00.479071 (+258598us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:00.642046 (+162975us) log.cc:844] Serialized 52 byte log entry
0709 12:25:00.642275 (+   229us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:00.642337 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.642436 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:01.371525 (+729089us) raft_consensus.cc:1685] finished
0709 12:25:01.371549 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.371723 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":19034,"prepare.queue_time_us":8848,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":44,"raft.queue_time_us":22654,"raft.run_cpu_time_us":271,"raft.run_wall_time_us":5081,"thread_start_us":254250,"threads_started":1,"wal-append.queue_time_us":623794}
I20260709 12:25:01.388264 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:01.381000  8945 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 27.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.018s
I20260709 12:25:01.388319 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:01.262179  8998 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.113s
I20260709 12:25:01.413064 31752 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:25:01.346518  9286 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:01.401647 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3985) took 1484 ms. Trace:
I20260709 12:25:01.283676  9263 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.095s
W20260709 12:25:01.403559  9278 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.401s	user 0.002s	sys 0.000s
I20260709 12:25:01.426292  9191 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.004s
W20260709 12:25:01.399646  9279 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.121s	user 0.001s	sys 0.000s
I20260709 12:25:01.388391  8949 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:01.423108 31826 rpcz_store.cc:276] 0709 12:24:59.916896 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.057549 (+140653us) service_pool.cc:224] Handling call
0709 12:25:00.071477 (+ 13928us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:00.109283 (+ 37806us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:00.109333 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:00.130305 (+ 20972us) log.cc:844] Serialized 26 byte log entry
0709 12:25:00.312184 (+181879us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:00.312250 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.312369 (+   119us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:01.401272 (+1088903us) raft_consensus.cc:1685] finished
0709 12:25:01.401320 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.401548 (+   228us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37213,"raft.queue_time_us":37381,"raft.run_cpu_time_us":185,"raft.run_wall_time_us":17133,"thread_start_us":154836,"threads_started":1,"wal-append.queue_time_us":526459}
I20260709 12:25:01.378207  8992 consensus_queue.cc:1243] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 31.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
W20260709 12:25:01.439395 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51168.3us
I20260709 12:25:01.384442  9283 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:01.349591  9276 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.476384  9276 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:01.480331 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3980) took 1565 ms. Trace:
I20260709 12:25:01.481141  9302 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:01.481750  8916 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.352s	user 0.008s	sys 0.003s
I20260709 12:25:01.481918  9302 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.485162  9302 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:01.486763  9289 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:01.487691  9238 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.548s	user 0.003s	sys 0.027s
I20260709 12:25:01.487511  9177 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:01.488678  9177 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:01.489296  9177 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.491631 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3796) took 2097 ms. Trace:
I20260709 12:25:01.492062 31824 rpcz_store.cc:276] 0709 12:24:59.394095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.394360 (+   265us) service_pool.cc:224] Handling call
0709 12:24:59.394816 (+   456us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:59.433115 (+ 38299us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:24:59.433171 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:59.481765 (+ 48594us) log.cc:844] Serialized 26 byte log entry
0709 12:24:59.759292 (+277527us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:24:59.759347 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.759442 (+    95us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:01.486957 (+1727515us) spinlock_profiling.cc:243] Waited 3.52 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:01.491195 (+  4238us) raft_consensus.cc:1685] finished
0709 12:25:01.491230 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.491535 (+   305us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5153,"raft.queue_time_us":27600,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":12778,"spinlock_wait_cycles":8104320,"thread_start_us":223347,"threads_started":1,"wal-append.queue_time_us":382477}
I20260709 12:25:01.493970 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:01.381245  9280 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:01.495615  9228 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.720s	user 0.033s	sys 0.023s
I20260709 12:25:01.487432  9289 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:01.473146  9223 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.745s	user 0.029s	sys 0.000s
I20260709 12:25:01.505977  9280 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:01.507184 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:01.507967  9290 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.261s	user 0.000s	sys 0.001s
I20260709 12:25:01.443902  9300 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:01.510361  9035 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:01.507861  9260 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:01.515394  9075 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:01.516361  9289 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:01.517043  9180 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.785s	user 0.008s	sys 0.004s
I20260709 12:25:01.480778 31827 rpcz_store.cc:276] 0709 12:24:59.915275 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.047523 (+132248us) service_pool.cc:224] Handling call
0709 12:25:00.152044 (+104521us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:00.189103 (+ 37059us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.480115 (+1291012us) spinlock_profiling.cc:243] Waited 98.4 ms on lock 0x7b540043d5d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:01.480261 (+   146us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":226207104}
I20260709 12:25:01.422381  9286 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.503289 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:01.504383 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:01.506428 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3979) took 2302 ms. Trace:
I20260709 12:25:01.450995  8912 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:01.482215  8916 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.525823 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:01.517529  9180 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.518422  8948 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:01.522977 31745 rpcz_store.cc:276] 0709 12:24:59.204311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.344063 (+139752us) service_pool.cc:224] Handling call
0709 12:24:59.344426 (+   363us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:59.344951 (+   525us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:59.344983 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:59.345503 (+   520us) log.cc:844] Serialized 26 byte log entry
0709 12:24:59.607356 (+261853us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:24:59.630869 (+ 23513us) log.cc:844] Serialized 14 byte log entry
0709 12:24:59.631079 (+   210us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.631153 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:01.494799 (+1863646us) spinlock_profiling.cc:243] Waited 5.53 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:01.497092 (+  2293us) raft_consensus.cc:1685] finished
0709 12:25:01.506173 (+  9081us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.506324 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9210,"prepare.queue_time_us":32798,"prepare.run_cpu_time_us":75,"prepare.run_wall_time_us":5604,"raft.queue_time_us":10303,"raft.run_cpu_time_us":206,"raft.run_wall_time_us":5139,"spinlock_wait_cycles":12724992,"thread_start_us":261474,"threads_started":1,"wal-append.queue_time_us":305567}
I20260709 12:25:01.466238  9283 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.521896  9286 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:01.527525  9273 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.472s	user 0.000s	sys 0.001s
W20260709 12:25:01.527050  9165 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.757s	user 0.008s	sys 0.003s
I20260709 12:25:01.565045 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:01.574456 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:01.533802  8998 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:01.527045  8912 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.538563  9263 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:01.580875 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3992) took 1353 ms. Trace:
I20260709 12:25:01.586581  9265 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:01.587059  8945 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:01.587568  9292 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:01.588241  9283 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:01.587563  8945 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.589835  8945 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:01.595873  8916 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:01.599162  9177 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:01.601588  9210 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:01.602201  9210 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:01.604022 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1340) took 2390 ms. Trace:
I20260709 12:25:01.604463 32375 rpcz_store.cc:276] 0709 12:24:59.148752 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.157162 (+  8410us) service_pool.cc:224] Handling call
0709 12:24:59.207857 (+ 50695us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:24:59.255700 (+ 47843us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:24:59.255733 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:24:59.256304 (+   571us) log.cc:844] Serialized 26 byte log entry
0709 12:24:59.514004 (+257700us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:24:59.514052 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:24:59.514120 (+    68us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:01.472350 (+1958230us) spinlock_profiling.cc:243] Waited 6.02 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:01.527706 (+ 55356us) raft_consensus.cc:1685] finished
0709 12:25:01.527738 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.531988 (+  4250us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":17519,"raft.run_cpu_time_us":188,"raft.run_wall_time_us":188,"spinlock_wait_cycles":13854976,"thread_start_us":257259,"threads_started":1,"wal-append.queue_time_us":373978}
W20260709 12:25:01.606884  9294 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
I20260709 12:25:01.542254  9300 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:01.608278  9300 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.609905  8945 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.568923  9191 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.038s
I20260709 12:25:01.584815 31825 rpcz_store.cc:276] 0709 12:25:00.227095 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.235926 (+  8831us) service_pool.cc:224] Handling call
0709 12:25:00.236334 (+   408us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:00.239004 (+  2670us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:00.239043 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:00.326502 (+ 87459us) log.cc:844] Serialized 26 byte log entry
0709 12:25:00.720063 (+393561us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:00.720118 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.720207 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:01.580584 (+860377us) raft_consensus.cc:1685] finished
0709 12:25:01.580615 (+    31us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.580797 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":4556,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":41464,"thread_start_us":232558,"threads_started":1,"wal-append.queue_time_us":348699}
I20260709 12:25:01.624389 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:01.577041  9035 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:01.625232 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 227204us
W20260709 12:25:01.595873  9162 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.499s	user 0.006s	sys 0.007s
I20260709 12:25:01.543294  8948 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.636260 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3982) took 1720 ms. Trace:
I20260709 12:25:01.636525  9180 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:01.614120  8912 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:01.638717 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3986) took 1721 ms. Trace:
I20260709 12:25:01.578238  9165 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.610605  9300 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:01.621105  9249 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.373s	user 0.007s	sys 0.007s
I20260709 12:25:01.644783 31822 rpcz_store.cc:276] 0709 12:24:59.917214 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.059020 (+141806us) service_pool.cc:224] Handling call
0709 12:25:00.114166 (+ 55146us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:00.153726 (+ 39560us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:00.153777 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:00.181132 (+ 27355us) log.cc:844] Serialized 52 byte log entry
0709 12:25:00.498166 (+317034us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:00.498220 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.498328 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:01.638434 (+1140106us) raft_consensus.cc:1685] finished
0709 12:25:01.638457 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.638646 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10715,"raft.queue_time_us":4470,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":7543,"thread_start_us":275124,"threads_started":1,"wal-append.queue_time_us":546978}
I20260709 12:25:01.640725 31823 rpcz_store.cc:276] 0709 12:24:59.915917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.050783 (+134866us) service_pool.cc:224] Handling call
0709 12:25:00.102750 (+ 51967us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.180162 (+ 77412us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:00.200936 (+ 20774us) log.cc:844] Serialized 14 byte log entry
0709 12:25:00.614888 (+413952us) log.cc:844] Serialized 14 byte log entry
0709 12:25:00.622987 (+  8099us) log.cc:844] Serialized 14 byte log entry
0709 12:25:00.637277 (+ 14290us) log.cc:844] Serialized 14 byte log entry
0709 12:25:00.637540 (+   263us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:00.637606 (+    66us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:00.637675 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.608679 (+971004us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b1800144ac0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:01.635259 (+ 26580us) spinlock_profiling.cc:243] Waited 13.3 ms on lock 0x7b1800144ac0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419
0709 12:25:01.635937 (+   678us) spinlock_profiling.cc:243] Waited 44.5 ms on lock 0x7b54004475d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:01.635990 (+    53us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:01.636172 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":8820,"prepare.queue_time_us":1163712,"prepare.run_cpu_time_us":134,"prepare.run_wall_time_us":138,"raft.queue_time_us":4064,"raft.run_cpu_time_us":205,"raft.run_wall_time_us":34157,"spinlock_wait_cycles":176180096,"thread_start_us":284828,"threads_started":1,"wal-append.queue_time_us":522157}
I20260709 12:25:01.587332  9265 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:01.678894 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 407151us
W20260709 12:25:01.678776  9246 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.352s	user 0.014s	sys 0.000s
I20260709 12:25:01.643365  9035 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.642287  9162 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.707839 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:01.705564 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:01.712731 31746 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:01.711046 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:01.716851 31749 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:01.721563  8948 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:01.702301  9246 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.724155  9246 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:01.724993 31819 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
W20260709 12:25:01.725509  9275 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.555s	user 0.006s	sys 0.005s
I20260709 12:25:01.725950  9275 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.728000  9105 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.733625  9105 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.735062 32370 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:01.735904  9105 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:01.736436 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1412) took 1773 ms. Trace:
I20260709 12:25:01.736884 32370 rpcz_store.cc:276] 0709 12:24:59.962933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.963103 (+   170us) service_pool.cc:224] Handling call
0709 12:24:59.963477 (+   374us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:01.735519 (+1772042us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.736182 (+   663us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b540055da54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:01.736372 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4073522944}
I20260709 12:25:01.737758 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:01.736433  9162 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:01.682551  9210 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 25.18, Last appended by leader: 18, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.729431  8945 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:01.719559  9307 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:01.706753  9177 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.024s
I20260709 12:25:01.731403  9165 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:01.674592  9249 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.728521  9035 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:01.729295  9286 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:01.750866 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 174335us
I20260709 12:25:01.741712  9033 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:25:01.750023  9004 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.014s	user 0.013s	sys 0.001s
I20260709 12:25:01.740849  9246 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:01.745759  9292 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:01.770604 31744 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: [32.19-32.19]
I20260709 12:25:01.759070  8803 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:01.772187  8803 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:01.773152 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:01.774132  9180 raft_consensus.cc:2764] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election decision vote started in defunct term 26: won
I20260709 12:25:01.783396  8916 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.784184  8916 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.786809 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:01.783285 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:01.788004  9105 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:01.789875  9105 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.788874 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:01.791263 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1541) took 1837 ms. Trace:
I20260709 12:25:01.791782 31677 rpcz_store.cc:276] 0709 12:24:59.948226 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.966868 (+ 18642us) service_pool.cc:224] Handling call
0709 12:24:59.967261 (+   393us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:00.189411 (+222150us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.778538 (+1589127us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54001b2bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:01.785920 (+  7382us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":493527936}
I20260709 12:25:01.761300  9275 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:01.750434  8949 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.797487  8949 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:01.795056  9202 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:01.798450  9202 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.789863  9289 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:01.798905  9202 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:01.799104  9289 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:01.795603  9165 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:01.799628  9289 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:01.769032  9004 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:01.806768 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:01.753890  9307 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:01.830783 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:01.838423 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:01.846081 32370 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:01.808002  9004 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:01.771489  9246 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.846443  9282 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.007s
I20260709 12:25:01.805209  9292 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:01.788522  8992 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 27.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.052s
I20260709 12:25:01.864465 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:01.866099 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:01.877213 31823 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:01.879460 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:01.882972 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:01.883503  9307 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:01.884497 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:01.885295 32371 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:01.887667  9246 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:01.888487  9292 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:01.889685  9265 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:01.842730  9298 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:01.897003 32370 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:01.846151  9275 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:01.918870 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:01.920884 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:01.921670 31677 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:25:01.922041 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:01.920800 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:01.924630 31671 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 28: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:01.906760 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:01.933702 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:01.934582 32371 raft_consensus.cc:2368] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32.
I20260709 12:25:01.934701 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:01.935679 31669 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:25:01.937801 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:01.939877 31674 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 29 index: 18. Preceding OpId from leader: term: 30 index: 18. (term mismatch)
I20260709 12:25:01.919832 31819 raft_consensus.cc:2376] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32. Re-sending same reply.
W20260709 12:25:01.912761  9161 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.742s	user 0.028s	sys 0.024s
I20260709 12:25:01.898620  9177 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:01.896318  8949 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:01.945539 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:01.940455 31674 pending_rounds.cc:85] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 17
I20260709 12:25:01.946798 31674 pending_rounds.cc:107] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 29.18
I20260709 12:25:01.947189 31674 raft_consensus.cc:2889] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:01.903385  9275 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:01.953922 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 121447us
I20260709 12:25:01.873479  9180 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.069s
I20260709 12:25:01.868216  9276 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.071s
I20260709 12:25:01.955565 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:01.957563 31672 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:25:01.973189 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:01.958479 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:01.973675 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:01.974099 31675 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:25:01.974506 31752 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:25:01.975872 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:01.979427 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:01.980512 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:01.950225  9161 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.780s	user 0.029s	sys 0.024s
I20260709 12:25:01.982811 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:25:01.956637 31791 connection.cc:256] RPC call timeout handler was delayed by -2.1071120709998468s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1563s for a response
I20260709 12:25:01.953644  9307 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:01.952428  8949 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:01.987190 31791 connection.cc:256] RPC call timeout handler was delayed by -1.5761120709998977s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21530000000000002s for a response
I20260709 12:25:01.988165 31673 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:25:02.005074 31791 connection.cc:256] RPC call timeout handler was delayed by -1.5194185780001135s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17190000000000003s for a response
I20260709 12:25:01.951061  8945 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:01.956912  9275 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:01.952162  9177 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:02.025184 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:01.996467 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:02.005721 31791 connection.cc:256] RPC call timeout handler was delayed by -1.4490485070000432s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1804s for a response
I20260709 12:25:02.009166 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 21 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:02.010464 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:02.031965 31678 raft_consensus.cc:2410] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 29 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 15.
I20260709 12:25:02.012295 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:02.029762 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:02.032969 31820 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:25:01.999167  9035 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:01.999598  9246 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:02.035913 31791 connection.cc:256] RPC call timeout handler was delayed by -1.22764850700014s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.205s for a response
W20260709 12:25:02.036947 31791 connection.cc:256] RPC call timeout handler was delayed by -1.1378120709996438s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.264s for a response
W20260709 12:25:02.024077  9310 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.231s	user 0.001s	sys 0.000s
I20260709 12:25:02.029677 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:02.013950  9038 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:02.023916  9307 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:02.028232  9177 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:02.060106 31820 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:02.076885 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:02.022971  9105 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:02.032694  9281 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.023222  8945 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:02.086963 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 57466.2us
W20260709 12:25:02.087893 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 78819.2us
I20260709 12:25:02.094446 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:02.107023 32371 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:02.043259  9246 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.095320  9307 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 30.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.073035  9286 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:02.098771  9281 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.085165  9202 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:02.108319 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:02.109064 31823 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:25:02.127192 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:02.127985 31823 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
W20260709 12:25:02.105739  9328 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.052s	user 0.001s	sys 0.000s
I20260709 12:25:02.072738  9161 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader election lost for term 21. Reason: could not achieve majority
I20260709 12:25:02.112939 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:02.129046 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:02.045621  9035 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.075387  9038 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.136982 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:02.137478 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:02.138845 31827 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:02.107055 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:02.151546  8945 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:02.152897 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:02.154251 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:02.154863 31827 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:25:02.155771 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:02.156338 31827 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:02.157183 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:02.157696 31827 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:25:02.159328 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:02.160153 31823 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 32: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:02.129391  9105 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.162111 31823 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:02.164263 32371 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:02.164810  9105 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:02.187290  9333 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.075s	user 0.001s	sys 0.000s
W20260709 12:25:02.245836 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 97508.7us
W20260709 12:25:02.247937  9325 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:25:02.249429  9202 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:02.250237  9202 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:02.250648  9202 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:02.251076  9202 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:02.251178  9323 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.174s	user 0.000s	sys 0.000s
I20260709 12:25:02.253592  9105 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:02.254884  9326 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.220s	user 0.002s	sys 0.000s
I20260709 12:25:02.246771 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:02.260222 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:02.263300 31643 leader_election.cc:400] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32.
I20260709 12:25:02.263715 31643 leader_election.cc:403] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:02.172814  9338 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:02.239935  9093 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:02.265941  9338 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.268906  9348 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:25:02.269732  9009 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:02.270223  9009 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.244043  9161 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 25.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.027s
I20260709 12:25:02.270692  9009 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:02.272011  8803 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.280037  9281 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:02.272511  9246 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:02.262303  9333 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.074s	user 0.002s	sys 0.000s
W20260709 12:25:02.293483  8912 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.624s	user 0.013s	sys 0.006s
I20260709 12:25:02.294186  9105 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:02.298955  9275 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:02.341648 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 268241us
I20260709 12:25:02.345270 31823 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: [29.16-29.16]
W20260709 12:25:02.347470 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 83061.6us
I20260709 12:25:02.300946  8803 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:02.294178  8912 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.362365  9275 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:02.477334  9343 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:02.433974 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:02.372383 31827 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:02.480460 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:02.488682 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:02.406909  9038 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:02.390368  9289 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.570s	user 0.009s	sys 0.014s
I20260709 12:25:02.494433  9105 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260709 12:25:02.495769 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61938.1us
I20260709 12:25:02.493196 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:02.496338 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:02.515062 31827 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:02.492746  9275 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:02.497786 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:02.494483  9339 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:02.499207  9336 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.499576  8803 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:02.501264  8912 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:02.522418 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:02.503300  9338 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:02.504446  9038 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:02.491648  8201 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:02.503906  9348 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32.
I20260709 12:25:02.366063  9112 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.139s
I20260709 12:25:02.505954  9009 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.565366  9009 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:02.510025 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34010 (request call id 1546) took 1406 ms. Trace:
I20260709 12:25:02.566071 31676 rpcz_store.cc:276] 0709 12:25:01.103652 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.103899 (+   247us) service_pool.cc:224] Handling call
0709 12:25:02.509738 (+1405839us) spinlock_profiling.cc:243] Waited 134 ms on lock 0x7b54001d5954. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:02.509901 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":308851584}
I20260709 12:25:02.567430 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:02.479669  9035 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:02.515542 32370 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:02.516790 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:02.382256  9084 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:02.523718 32377 raft_consensus.cc:2393] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:02.522346  9281 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:02.574084 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 103860us
I20260709 12:25:02.554802 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:02.512970  9069 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:02.533354 31827 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:02.555357 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:02.485121  9343 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:02.517469  9275 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:02.569203 31675 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35. Ops: [34.23-34.23]
I20260709 12:25:02.553944 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:02.579329 31672 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:25:02.577091 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:02.580281 31827 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 29: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:02.557960  8201 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:02.513140  9289 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:02.577654 31678 raft_consensus.cc:2410] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
I20260709 12:25:02.578868  9298 raft_consensus.cc:2764] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:25:02.589008 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:02.589666 31672 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:02.590008 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 73325us
I20260709 12:25:02.524875  8803 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:02.585886 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:02.581053  9275 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:02.586683  8201 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:02.592552  9330 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.492s	user 0.005s	sys 0.008s
I20260709 12:25:02.604104 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:02.607214 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:02.578742  9084 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:02.579155  9281 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:02.619233 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:02.525075  9336 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:02.579471  9069 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:02.602037  8803 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:02.637212 31676 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:02.611351  9035 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:02.639580 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:02.594786  9038 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.027s
I20260709 12:25:02.611101  9193 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:02.625095  9112 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.020s
I20260709 12:25:02.645661  9298 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:02.647229  9298 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:02.647657  9298 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:02.615702  9161 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.024s
I20260709 12:25:02.657840 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:02.658486 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:02.655143  8811 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:02.663913 31672 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:25:02.664630 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:02.665297 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:02.666554 31672 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:02.670516 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:02.671209 31673 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:02.644114  9035 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.674573 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:02.669392  8811 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.667886  9246 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:02.668574  8590 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:02.724709 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:02.668301  9180 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:02.726353  9180 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:02.726943  9180 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:02.727277  9289 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:02.729462  8811 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:02.730468  9352 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
W20260709 12:25:02.737649  9327 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.854s	user 0.009s	sys 0.005s
W20260709 12:25:02.739125  9353 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.127s	user 0.001s	sys 0.000s
W20260709 12:25:02.730499  9321 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.777s	user 0.002s	sys 0.002s
I20260709 12:25:02.740238  8201 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.740954 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3827) took 1255 ms. Trace:
I20260709 12:25:02.742303  8201 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:02.741309 31821 rpcz_store.cc:276] 0709 12:25:01.485466 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.541117 (+ 55651us) service_pool.cc:224] Handling call
0709 12:25:01.573676 (+ 32559us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.616282 (+ 42606us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.616338 (+    56us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.639178 (+ 22840us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.748704 (+109526us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:01.748761 (+    57us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.748897 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.729803 (+980906us) spinlock_profiling.cc:243] Waited 87.5 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:02.740660 (+ 10857us) raft_consensus.cc:1685] finished
0709 12:25:02.740680 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.740891 (+   211us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3090,"raft.queue_time_us":14373,"raft.run_cpu_time_us":94,"raft.run_wall_time_us":95,"spinlock_wait_cycles":201331968,"thread_start_us":76415,"threads_started":1,"wal-append.queue_time_us":213676}
I20260709 12:25:02.739408 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3829) took 1253 ms. Trace:
I20260709 12:25:02.744139 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3987) took 2584 ms. Trace:
I20260709 12:25:02.744635 31744 rpcz_store.cc:276] 0709 12:25:00.160052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.160256 (+   204us) service_pool.cc:224] Handling call
0709 12:25:00.195066 (+ 34810us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.771167 (+1576101us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:02.743859 (+972692us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400404cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:02.744074 (+   215us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3622762368}
I20260709 12:25:02.743747 31822 rpcz_store.cc:276] 0709 12:25:01.485888 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.673011 (+187123us) service_pool.cc:224] Handling call
0709 12:25:01.673475 (+   464us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.685448 (+ 11973us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.685479 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.717641 (+ 32162us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.839779 (+122138us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:01.839846 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.839952 (+   106us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.725864 (+885912us) spinlock_profiling.cc:243] Waited 189 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:02.739086 (+ 13222us) raft_consensus.cc:1685] finished
0709 12:25:02.739106 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.739336 (+   230us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5129,"raft.queue_time_us":20594,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":17792,"spinlock_wait_cycles":434618752,"thread_start_us":80285,"threads_started":1,"wal-append.queue_time_us":134462}
I20260709 12:25:02.746259  9303 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.746905 31673 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:02.747061  9303 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.747934 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:02.748533 31673 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:02.712644  9165 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:02.749631  9303 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:02.753212 31822 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:02.766829  9084 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:02.716718  9354 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260709 12:25:02.716853  9277 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.833s	user 0.011s	sys 0.005s
I20260709 12:25:02.717097  9069 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:02.718601  8916 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:02.720010  9320 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.876s	user 0.012s	sys 0.006s
W20260709 12:25:02.765019  9317 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.884s	user 0.000s	sys 0.012s
W20260709 12:25:02.722784 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 48343.8us
W20260709 12:25:02.765357  9177 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.678s	user 0.012s	sys 0.000s
I20260709 12:25:02.753873 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:02.712512 31672 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:02.752501 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:02.773125 31676 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
W20260709 12:25:02.757068  9202 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.505s	user 0.010s	sys 0.000s
I20260709 12:25:02.749356 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:02.721122  9324 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.736s	user 0.001s	sys 0.003s
I20260709 12:25:02.788512 31821 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 28 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:25:02.711876  9300 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:02.760958  8948 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.790261 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:02.794276 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:02.764740  9336 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:02.761420  8992 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.753420  9165 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.761272  9210 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:02.763708  9281 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:02.759276  9289 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:02.833600 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:02.752825  9343 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:02.754196  9035 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:02.836901 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:02.871057 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:02.869982 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 3978) took 2955 ms. Trace:
I20260709 12:25:02.793083  9300 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:02.795058  9069 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:02.837648 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:02.885361 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:02.860925  9084 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:02.885979 31672 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:02.884852  9303 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:02.870076  9210 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.873842 31818 rpcz_store.cc:276] 0709 12:24:59.914717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.041817 (+127100us) service_pool.cc:224] Handling call
0709 12:25:00.042138 (+   321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:00.074900 (+ 32762us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:00.074931 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:00.087720 (+ 12789us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.637461 (+1549741us) spinlock_profiling.cc:243] Waited 14.9 ms on lock 0x7b18004c7380. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:25:01.718521 (+ 81060us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:01.718560 (+    39us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.720229 (+  1669us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.719434 (+999205us) spinlock_profiling.cc:243] Waited 194 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:02.869726 (+150292us) raft_consensus.cc:1685] finished
0709 12:25:02.869751 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.869917 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":32355,"raft.queue_time_us":36081,"raft.run_cpu_time_us":102,"raft.run_wall_time_us":103,"spinlock_wait_cycles":480417408,"thread_start_us":80496,"threads_started":1,"wal-append.queue_time_us":1674455}
I20260709 12:25:02.836112  8948 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.890175  9303 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 33 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:02.892637  9303 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:02.890175 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56694.5us
I20260709 12:25:02.894233 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:02.895208 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:02.896509 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:02.884469 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47903.8us
I20260709 12:25:02.835614  9246 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:02.888152  9069 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:02.891626  8912 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:02.895344  9193 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:02.897639  8912 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:02.897739  9193 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:02.898226  9193 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:02.898609  8912 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.899950  8912 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:02.890332 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 3991) took 2722 ms. Trace:
I20260709 12:25:02.893121  9210 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:02.902809 31747 rpcz_store.cc:276] 0709 12:25:00.167600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:00.180952 (+ 13352us) service_pool.cc:224] Handling call
0709 12:25:00.201601 (+ 20649us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:00.214512 (+ 12911us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:00.214564 (+    52us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:00.229234 (+ 14670us) log.cc:844] Serialized 26 byte log entry
0709 12:25:00.654171 (+424937us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:00.654225 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:00.654313 (+    88us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.716533 (+2062220us) spinlock_profiling.cc:243] Waited 4.15 ms on lock 0x7b24002a9ae8. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9b6e9c4 00007fe8f9b6cacb 00007fe8f9d27dc9 00007fe8f9d27a55 00007fe8f9d2709d 00007fe8fc95910e 00007fe8fc97088c 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221
0709 12:25:02.888882 (+172349us) raft_consensus.cc:1685] finished
0709 12:25:02.888903 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:02.890261 (+  1358us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":27565,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":8735,"spinlock_wait_cycles":9550080,"thread_start_us":367463,"threads_started":1,"wal-append.queue_time_us":525994}
I20260709 12:25:02.897372 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:02.904742  9298 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:02.897373  9246 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.912247 31673 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:02.793288  9202 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.902118  8803 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:02.792750  9177 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.870446  8992 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:02.934719 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 315540us
I20260709 12:25:02.880952  9347 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:02.906409  9069 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 31.15, Last appended by leader: 15, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:02.942953 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:02.922823  9210 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:25:02.944309 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:02.934194  9283 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.981s	user 0.008s	sys 0.016s
I20260709 12:25:02.925700  9084 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:02.938441  9318 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.884s	user 0.011s	sys 0.002s
I20260709 12:25:02.872816  9289 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:02.952829 31672 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:25:02.952873 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:02.932117  9300 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:02.953155  9283 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.002s	user 0.008s	sys 0.016s
W20260709 12:25:02.981381 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62086.9us
I20260709 12:25:02.983736  9035 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:02.987684 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:02.989653 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 93203.8us
I20260709 12:25:02.951705  9210 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:02.990048 31673 raft_consensus.cc:2376] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32. Re-sending same reply.
I20260709 12:25:02.995759 31673 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:02.996932 31673 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:02.998164 31673 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:02.999782  8948 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:03.000442  9165 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:03.001636  9289 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:02.978699  9347 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:03.002321  9347 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:03.002724  8992 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:03.002838  9210 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:03.003681  9363 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:03.004124  9363 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:03.004474  9363 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:02.977670  9298 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:03.000453  9246 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:02.991250  9180 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.010324  9180 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:03.007016  9084 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.007066  9275 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:03.011178  9358 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
W20260709 12:25:03.014292 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61490.7us
I20260709 12:25:02.992540 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:03.016126 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.016875 32370 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:25:03.023815 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.024088  9370 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:03.024541  9370 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:03.024889  9370 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:03.033226  9300 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:03.036727 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 103645us
I20260709 12:25:03.047230 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:03.049592  9306 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:03.050196  9306 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:03.051187  9306 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:03.022804 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.002734  9347 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:03.038725 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.038972 31673 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 33. Current term is 34. Ops: [33.19-33.19]
I20260709 12:25:02.999890  8803 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:02.999796  9202 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:03.041975 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.013561  9193 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.063922 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:03.055454  9322 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.045s	user 0.000s	sys 0.004s
I20260709 12:25:03.065963 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:03.040876  9275 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:03.067795 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:03.072772 31673 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:03.076087 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.077427 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3825) took 1592 ms. Trace:
I20260709 12:25:03.077893 31824 rpcz_store.cc:276] 0709 12:25:01.485031 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.518670 (+ 33639us) service_pool.cc:224] Handling call
0709 12:25:01.523429 (+  4759us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.535911 (+ 12482us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:01.574310 (+ 38399us) log.cc:844] Serialized 14 byte log entry
0709 12:25:01.753331 (+179021us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.759439 (+  6108us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.759646 (+   207us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:01.759696 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.759800 (+   104us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:02.767469 (+1007669us) spinlock_profiling.cc:243] Waited 12.3 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:03.070437 (+302968us) raft_consensus.cc:1685] finished
0709 12:25:03.077200 (+  6763us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:03.077355 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":63730,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":56,"raft.queue_time_us":11061,"raft.run_cpu_time_us":338,"raft.run_wall_time_us":30579,"spinlock_wait_cycles":28351744,"thread_start_us":126348,"threads_started":1,"wal-append.queue_time_us":368603}
I20260709 12:25:03.080185  9193 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:03.082829 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3828) took 1597 ms. Trace:
I20260709 12:25:03.083256 31825 rpcz_store.cc:276] 0709 12:25:01.485679 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:01.643436 (+157757us) service_pool.cc:224] Handling call
0709 12:25:01.644613 (+  1177us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:01.662648 (+ 18035us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:01.662699 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:01.669850 (+  7151us) log.cc:844] Serialized 26 byte log entry
0709 12:25:01.844940 (+175090us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:01.844977 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:01.845038 (+    61us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:03.054763 (+1209725us) spinlock_profiling.cc:243] Waited 36.3 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:03.082336 (+ 27573us) raft_consensus.cc:1685] finished
0709 12:25:03.082359 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:03.082603 (+   244us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":117,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":6675,"spinlock_wait_cycles":83514752,"thread_start_us":144336,"threads_started":1,"wal-append.queue_time_us":273818}
I20260709 12:25:03.100927 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.102280 32369 raft_consensus.cc:2393] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 29: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:03.071610  9300 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:03.103437 32378 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:03.104316 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:03.105279 31673 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:03.105284 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.107827 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:03.110056 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.110783 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.111070 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.111271 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.111917 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.076875  9303 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 27.13, Next index: 14, Last known committed idx: 13, Time since last communication: 0.038s
I20260709 12:25:03.112625 32377 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:03.112689 32372 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 17. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:25:03.114970 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.117033 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.117705 32378 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:25:03.111877 32376 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:03.120987 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:03.122880  9202 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.123144 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:03.124557 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:03.123169 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:03.123476  9283 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:03.124955 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:03.095697  9249 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:03.126436 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.128162 32378 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:03.124506 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.127054 31678 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:25:03.129062 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4008) took 1231 ms. Trace:
I20260709 12:25:03.135651 31823 rpcz_store.cc:276] 0709 12:25:01.897174 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.193495 (+296321us) service_pool.cc:224] Handling call
0709 12:25:02.239099 (+ 45604us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:02.346085 (+106986us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.128820 (+782735us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54001c89d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:03.129003 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":243233024}
I20260709 12:25:03.152719 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.152719 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.152966 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.131247  9202 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.155629  9370 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.157677  9202 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:03.158224  9249 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:03.160893 31744 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 31: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:03.169997 32378 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:03.185726 31673 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:25:03.158691  9370 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:03.207468 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:03.153923  9283 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:03.208894 31673 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:03.118219  9363 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:03.172519  9261 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 1.859s	user 0.019s	sys 0.066s
W20260709 12:25:03.179926  9276 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.171s	user 0.009s	sys 0.021s
W20260709 12:25:03.217443  9276 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.218s	user 0.010s	sys 0.021s
I20260709 12:25:03.212345 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:03.215503  9283 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.16, Last appended by leader: 16, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.162649  9249 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:03.219305 31673 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
W20260709 12:25:03.218566  9261 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 1.922s	user 0.020s	sys 0.067s
I20260709 12:25:03.251566 32373 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28. Ops: []
W20260709 12:25:03.272331  9369 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.126s	user 0.000s	sys 0.001s
W20260709 12:25:03.274276  9376 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.143s	user 0.001s	sys 0.000s
I20260709 12:25:03.325920 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:03.329003 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:03.302161  9363 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:03.332451 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.334347 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.336598 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.336845 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 24 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:03.339221 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:03.338275 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:03.337325 31825 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:25:03.335034 31823 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:03.340909 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:03.334086  9084 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.008s
I20260709 12:25:03.345420 31818 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:03.348052  9383 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.006s
I20260709 12:25:03.353209 32378 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: [31.22-31.22]
I20260709 12:25:03.361284 31672 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32"
dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:40834
I20260709 12:25:03.364391 31678 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:03.365475  9210 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.369218 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.369517  9033 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:03.369964 31820 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:03.369977  9033 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:03.405516 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 122364us
I20260709 12:25:03.399721 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:03.408862 31717 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.571s (SENT) [suppressed 22 similar messages]
I20260709 12:25:03.427713 31818 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:03.347043  9347 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.430670 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.433913 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.335639  9306 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.094s
I20260709 12:25:03.435388 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.435439 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.437601 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.439020 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.441180  9210 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:03.447566 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.449841 31745 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:03.452045 31753 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:03.457671 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.487288 31824 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:03.532195 31672 raft_consensus.cc:606] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:03.541874 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.542896 31745 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 21.
I20260709 12:25:03.448230  9384 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:03.543061 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.547084 31818 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:25:03.559288 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:03.559962 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:03.560678 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:03.561410 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:03.552459 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.562906  9291 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:03.563227 31748 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:03.563519  9291 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:03.564774  9291 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.13, Last appended by leader: 13, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.570956 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.571739 31745 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:03.573450 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.574590 31753 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:03.575676 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.577008 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.577613  9033 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.578106 31750 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:03.579110 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.579716 31750 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:03.580426  9347 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:03.561339 31674 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 21.
I20260709 12:25:03.581663 31674 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 29. Ops: []
I20260709 12:25:03.537554  9372 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 30.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.045s
I20260709 12:25:03.582197 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4022) took 1017 ms. Trace:
I20260709 12:25:03.580675 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.585011 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.586452 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.587097 31753 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:03.587673  9372 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:03.588128  9372 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:03.589246 31676 pending_rounds.cc:85] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 18
I20260709 12:25:03.589008  9372 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:03.589629 31676 pending_rounds.cc:107] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 31.19
I20260709 12:25:03.589912 31676 raft_consensus.cc:2889] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:03.590380  9372 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:03.593466 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.595587 31826 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:03.598294 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.598982 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.600344 31827 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:03.601342 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:03.601907 31825 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:03.604859 31826 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:03.605973  9289 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:25:03.606491 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:03.606920  9289 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:03.607244  9289 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:03.608069  9289 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:03.608623  9334 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.448s	user 0.001s	sys 0.005s
I20260709 12:25:03.614159 31825 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:03.599010 31821 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:25:03.620975  9280 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:03.621831  9038 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:03.627074 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:03.627213 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:03.630105 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.631083 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.631852 31753 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:25:03.553520 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:03.633713 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:03.635172 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:03.635293  9337 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:03.684949  9337 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:03.685272 31819 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:03.636795 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:03.673879 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.686939 31745 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:25:03.687114 31744 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:03.638863 31792 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:03.639627 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.688088 31818 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:25:03.640151  9249 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:03.635195 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:03.680473 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:03.691258 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 52497us
I20260709 12:25:03.691498 31820 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:03.691742 31753 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:03.692050 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:03.692409 31672 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:03.609090  9289 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:03.693007 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4030) took 1126 ms. Trace:
I20260709 12:25:03.582535 31674 rpcz_store.cc:276] 0709 12:25:02.564492 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.581192 (+1016700us) service_pool.cc:224] Handling call
0709 12:25:03.581501 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.581993 (+   492us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.582145 (+   152us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:03.646373  9347 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:03.649008  9177 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:03.653674 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:03.707129 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.633095  9276 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:03.706182 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:03.654589 31748 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:03.668154 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.671195 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:03.671693 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:03.709586 31826 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:03.635876 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.687347 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4028) took 1121 ms. Trace:
I20260709 12:25:03.675488 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:03.676505 31823 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:03.689438 31822 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 33: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:03.689349 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:03.644279 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:03.712076 31821 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:03.710809 31670 rpcz_store.cc:276] 0709 12:25:02.566155 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.686304 (+1120149us) service_pool.cc:224] Handling call
0709 12:25:03.686670 (+   366us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.686914 (+   244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.686953 (+    39us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:03.687012 (+    59us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:03.687060 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.687135 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:03.687267 (+   132us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:03.677889 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:03.680955 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 53811.1us
I20260709 12:25:03.682147 31678 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:25:03.693475 31672 rpcz_store.cc:276] 0709 12:25:02.566730 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.691938 (+1125208us) service_pool.cc:224] Handling call
0709 12:25:03.692232 (+   294us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.692826 (+   594us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.692961 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:03.719075 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.720005 31753 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:25:03.722898 31823 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:03.725852 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.727651 31827 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:03.729950 31819 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:03.731519 31826 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:03.657976  8949 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:03.732563 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.734203  8949 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:03.744172 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.746229 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.747258 31748 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:25:03.667763  9033 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:03.691000  9283 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:03.763134 31821 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:25:03.675052  9280 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:03.675195  9038 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:03.705772 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.756014 31675 raft_consensus.cc:1275] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:03.707727  9345 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:03.707554  9180 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:03.715034  9363 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.047s
I20260709 12:25:03.772110  9370 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:03.690968  9265 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 26.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.063s
I20260709 12:25:03.772678  9370 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:03.753576  9177 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:03.685930  9337 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:03.776829 31642 leader_election.cc:336] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.678s (SENT)
I20260709 12:25:03.777457 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:03.773921  9370 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.14, Last appended by leader: 14, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.749238 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.764443 31824 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:03.760855 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.779573  9283 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:03.784698 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 25 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.759431  9166 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:03.783994 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 106207us
I20260709 12:25:03.835425 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:03.776039  9345 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.786942  9337 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:03.826797 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.856151 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.857452  9162 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:03.857614 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.857980  9162 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:03.858850  8590 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:03.859319  8590 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:03.858958  9162 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:03.860499  9162 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:03.860160  8590 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:03.861415  8590 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:03.856245 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.862480  9364 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:03.863066  9364 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:03.863680 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.864432 31753 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:03.864085  9364 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.865324 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.846747  9345 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:03.867028 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:03.872231 31670 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 18. Preceding OpId from leader: term: 27 index: 19. (index mismatch)
I20260709 12:25:03.873729 31670 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: [25.14-25.14]
I20260709 12:25:03.881000  9261 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:03.882110  9261 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:03.882540  9261 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.883034  9261 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:03.883999  9383 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:03.885272  9210 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:03.885627  9210 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:03.886443  9210 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.813192 31820 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:03.850229  9038 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:03.825521  9306 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Leader pre-election won for term 22
I20260709 12:25:03.891680  9306 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.891455  9383 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:03.892095  9306 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:03.894747  8948 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:03.894794  9000 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:03.895371  8948 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:03.870493 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:03.896245  8948 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.891481  9038 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:03.897217  9386 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:03.897871  9386 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:03.898718  8992 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:03.899371  8992 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:03.899855  8992 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:03.849035 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.902962 31752 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:03.903280  9303 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:03.903764 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:03.904702 31752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1422) took 1903 ms (client timeout 1664 ms). Trace:
I20260709 12:25:03.890138  9384 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 25.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 12:25:03.905160 31752 rpcz_store.cc:269] 0709 12:25:02.001311 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.011959 (+ 10648us) service_pool.cc:224] Handling call
0709 12:25:03.904487 (+1892528us) spinlock_profiling.cc:243] Waited 1.87 s on lock 0x7b54002954d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:03.904601 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4301230336}
I20260709 12:25:03.897280  8948 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:03.858278 31744 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:03.907984 31670 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:25:03.906581 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.842725  9166 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.917677 31670 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:03.902400  9383 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.008s
I20260709 12:25:03.930023 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.951658 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1553) took 1380 ms. Trace:
I20260709 12:25:03.952231 31669 rpcz_store.cc:276] 0709 12:25:02.571624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.707904 (+1136280us) service_pool.cc:224] Handling call
0709 12:25:03.716934 (+  9030us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.717182 (+   248us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.717218 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:03.717280 (+    62us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:25:03.717325 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.717379 (+    54us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:03.951405 (+234026us) spinlock_profiling.cc:243] Waited 8.7 ms on lock 0x7b54001810d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:03.951578 (+   173us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":20011264}
I20260709 12:25:03.958513 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:03.949977 31576 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 17 to 26, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 26 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:03.963567 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.966120 31750 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 31: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:03.967971 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:03.968871 31669 raft_consensus.cc:2376] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33. Re-sending same reply.
I20260709 12:25:03.970268 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:03.970386  9386 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:03.971881  9386 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:03.972090 31672 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:25:03.907020  9249 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:03.908396  9281 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:03.976889 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:03.977808 31669 raft_consensus.cc:2376] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29. Re-sending same reply.
I20260709 12:25:03.979542 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.916783  9364 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:03.982870 31745 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:04.000842 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:04.003575 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:04.013566  9394 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.062s	user 0.000s	sys 0.000s
I20260709 12:25:04.018462  9291 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:04.019408  9261 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:04.021677  9261 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:03.908200  9280 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:04.025745 31745 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:03.986267  9398 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:04.032256 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:04.036085  9396 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.065s	user 0.002s	sys 0.000s
I20260709 12:25:04.036468 31678 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 33 because replica has last-logged OpId of term: 32 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 18.
W20260709 12:25:04.036800 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.037179 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:03.930410  9283 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:03.982378  9249 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:04.024122  9383 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:04.064203 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.077512  9084 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:04.057937 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:04.085330 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:04.091559 31748 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 32 because replica has last-logged OpId of term: 31 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 21.
I20260709 12:25:04.107029 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:04.038523  9398 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:04.108515 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:04.114195 31676 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.518s	user 0.019s	sys 0.016s
W20260709 12:25:04.115417 31676 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.519s	user 0.020s	sys 0.017s
W20260709 12:25:04.121279 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.105437  9084 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:04.136526 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:04.119652 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1421) took 2004 ms. Trace:
I20260709 12:25:04.144490 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1549) took 1753 ms. Trace:
I20260709 12:25:04.086306  9306 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:04.162127 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:04.115995  8903 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 2.113s	user 0.003s	sys 0.040s
W20260709 12:25:04.160180  9165 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.012s	sys 0.034s
I20260709 12:25:04.108476  8992 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:04.109675  9383 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:04.159837  9180 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:04.122022  9355 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:04.175814 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:04.188191 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:04.189019 31744 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:04.145243 32371 rpcz_store.cc:276] 0709 12:25:02.391425 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.391562 (+   137us) service_pool.cc:224] Handling call
0709 12:25:02.391943 (+   381us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.392235 (+   292us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:03.659386 (+1267151us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.143896 (+484510us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:04.144128 (+   232us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:04.144201 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.144282 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.144431 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1266786,"prepare.queue_time_us":1268464,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"thread_start_us":484091,"threads_started":1}
I20260709 12:25:04.213984 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:04.171051  8903 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 2.187s	user 0.003s	sys 0.040s
W20260709 12:25:04.179592  9165 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.584s	user 0.013s	sys 0.034s
I20260709 12:25:04.228055 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:04.230772 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:04.244973 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:04.245415 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.246176 31749 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
W20260709 12:25:04.252609 31791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.011s	sys 0.008s
W20260709 12:25:04.255607 31791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.012s	sys 0.008s
W20260709 12:25:04.258464 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 614214us
I20260709 12:25:04.270915 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 31 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:04.139446 32374 rpcz_store.cc:276] 0709 12:25:02.115076 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:02.115240 (+   164us) service_pool.cc:224] Handling call
0709 12:25:02.133020 (+ 17780us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:02.133323 (+   303us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:25:02.145567 (+ 12244us) log.cc:844] Serialized 14 byte log entry
0709 12:25:03.658779 (+1513212us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x7b180045b080. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf 00007fe8fcdef038
0709 12:25:03.659018 (+   239us) log.cc:844] Serialized 14 byte log entry
0709 12:25:03.676820 (+ 17802us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b180045b080. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:04.119050 (+442230us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:04.119121 (+    71us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:04.119180 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.119410 (+   230us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.119579 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1491076,"prepare.queue_time_us":1528128,"prepare.run_cpu_time_us":87,"prepare.run_wall_time_us":90,"raft.queue_time_us":2798,"raft.run_cpu_time_us":130,"raft.run_wall_time_us":3106,"spinlock_wait_cycles":55574528,"thread_start_us":457024,"threads_started":1}
I20260709 12:25:04.235335  9210 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:04.203409  9180 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:04.264870 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:04.279729 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:04.299463 31825 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.006s	sys 0.022s
W20260709 12:25:04.280488 31677 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.555s	user 0.005s	sys 0.038s
W20260709 12:25:04.300232 31677 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.582s	user 0.005s	sys 0.038s
I20260709 12:25:04.304250 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:04.183074  9398 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:04.306849  9398 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:04.277638  9165 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Stepping down as leader of term 25
W20260709 12:25:04.310127 31674 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.534s	user 0.010s	sys 0.018s
W20260709 12:25:04.310524 31671 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.016s	sys 0.025s
W20260709 12:25:04.313395 31671 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.627s	user 0.016s	sys 0.025s
W20260709 12:25:04.313192 31674 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.550s	user 0.010s	sys 0.018s
W20260709 12:25:04.299894 31825 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.622s	user 0.006s	sys 0.022s
I20260709 12:25:04.302903  9347 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:25:04.310398  9363 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.538s	user 0.002s	sys 0.022s
W20260709 12:25:04.311394  9395 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.100s	user 0.001s	sys 0.000s
I20260709 12:25:04.322098 31576 catalog_manager.cc:5697] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 14 to 25, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 25 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:04.327615 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4024) took 1762 ms. Trace:
I20260709 12:25:04.319550 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 3005) took 1284 ms. Trace:
I20260709 12:25:04.330557 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:04.333479 31674 rpcz_store.cc:276] 0709 12:25:03.035207 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.748232 (+713025us) service_pool.cc:224] Handling call
0709 12:25:03.762558 (+ 14326us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.762783 (+   225us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:03.763027 (+   244us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.319222 (+556195us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:04.319278 (+    56us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:04.319319 (+    41us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.319381 (+    62us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.319488 (+   107us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":79312,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":39,"raft.queue_time_us":9633,"thread_start_us":553916,"threads_started":1}
W20260709 12:25:04.329500  9363 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.557s	user 0.002s	sys 0.023s
W20260709 12:25:04.342846 32341 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.527s	user 0.011s	sys 0.015s
I20260709 12:25:04.234709  9084 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:04.358598 32341 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.646s	user 0.016s	sys 0.015s
I20260709 12:25:04.313071  9165 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:04.362623 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.332667 31671 rpcz_store.cc:276] 0709 12:25:02.565003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.676905 (+1111902us) service_pool.cc:224] Handling call
0709 12:25:03.677195 (+   290us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.677393 (+   198us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:03.685753 (+  8360us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.327151 (+641398us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:04.327299 (+   148us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:04.327347 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.327428 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.327551 (+   123us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9043,"prepare.queue_time_us":8783,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":9150,"raft.run_cpu_time_us":193,"raft.run_wall_time_us":13663,"thread_start_us":628099,"threads_started":1}
W20260709 12:25:04.364660 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:04.360546 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.520s	user 0.012s	sys 0.019s
W20260709 12:25:04.365811 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.525s	user 0.012s	sys 0.019s
W20260709 12:25:04.367035 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.662s	user 0.010s	sys 0.010s
I20260709 12:25:04.320181  9347 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:04.368074 31753 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:04.367305 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 590469us
I20260709 12:25:04.302889  9345 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:04.401327 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 95356.2us
W20260709 12:25:04.367882 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.663s	user 0.010s	sys 0.010s
I20260709 12:25:04.354207  9180 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:04.403411 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 698267us
I20260709 12:25:04.364665  9383 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:04.376955  9347 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:04.397794  9084 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:04.431174 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.533s	user 0.024s	sys 0.015s
W20260709 12:25:04.433423 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.535s	user 0.024s	sys 0.015s
W20260709 12:25:04.433849 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 535732us
W20260709 12:25:04.434342 31792 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.725s	user 0.024s	sys 0.028s
W20260709 12:25:04.434836 31792 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.726s	user 0.024s	sys 0.028s
I20260709 12:25:04.438454 31673 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4019) took 1876 ms. Trace:
I20260709 12:25:04.446905 31753 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
W20260709 12:25:04.435595 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 729448us
I20260709 12:25:04.450765 31792 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:04.451138 31819 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 29. Ops: []
I20260709 12:25:04.453269 31825 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:25:04.455658 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:04.438934 31673 rpcz_store.cc:276] 0709 12:25:02.561753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.425677 (+863924us) service_pool.cc:224] Handling call
0709 12:25:03.426537 (+   860us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:03.429479 (+  2942us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.429508 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:03.430020 (+   512us) log.cc:844] Serialized 26 byte log entry
0709 12:25:03.938244 (+508224us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:03.981147 (+ 42903us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.438065 (+456918us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.438158 (+    93us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.438205 (+    47us) raft_consensus.cc:1685] finished
0709 12:25:04.438223 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.438379 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":42690,"prepare.run_cpu_time_us":44,"prepare.run_wall_time_us":46,"raft.queue_time_us":78653,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":22835,"thread_start_us":923430,"threads_started":2,"wal-append.queue_time_us":795938}
I20260709 12:25:04.456036 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:04.457765 32371 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:04.458954 31791 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.538s (SENT)
I20260709 12:25:04.462764 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:04.463043 31820 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:25:04.466048 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:04.440634  9069 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.425s	user 0.022s	sys 0.015s
I20260709 12:25:04.468528 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:04.467296 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.012s	sys 0.012s
W20260709 12:25:04.472574 31717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.576s	user 0.003s	sys 0.022s
W20260709 12:25:04.474517 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.012s	sys 0.012s
W20260709 12:25:04.474617 31717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.578s	user 0.003s	sys 0.022s
W20260709 12:25:04.474907 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 595340us
I20260709 12:25:04.469755 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:04.477267 31821 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 13. Preceding OpId from leader: term: 31 index: 15. (index mismatch)
I20260709 12:25:04.477267 32373 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:25:04.416858  9162 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:25:04.483700 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:04.491509 32374 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
W20260709 12:25:04.473414  9069 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.470s	user 0.022s	sys 0.016s
I20260709 12:25:04.434899  8992 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:04.460741  9289 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.754s	user 0.013s	sys 0.000s
I20260709 12:25:04.497519  9289 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:04.481473 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:04.493664 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:04.485971  9370 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.625s	user 0.022s	sys 0.021s
W20260709 12:25:04.479099 32370 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.561s	user 0.022s	sys 0.035s
W20260709 12:25:04.512050 32370 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.595s	user 0.022s	sys 0.035s
I20260709 12:25:04.479737  9363 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:04.501410 32375 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:04.515669 31745 raft_consensus.cc:1217] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Deduplicated request from leader. Original: 23.14->[24.15-28.17]   Dedup: 24.15->[26.16-28.17]
I20260709 12:25:04.502192 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:04.516480 31745 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: [24.15-28.17]
I20260709 12:25:04.502458 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:04.518321 31824 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 22: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:25:04.492286  8803 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.508s	user 0.014s	sys 0.026s
I20260709 12:25:04.514796 31744 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:04.514737 32371 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:25:04.546659 31750 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:04.491719  9306 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:04.522128 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1432) took 1169 ms. Trace:
I20260709 12:25:04.501061 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:25:04.518712 31641 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.501s	user 0.006s	sys 0.045s
W20260709 12:25:04.554986 31641 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.537s	user 0.006s	sys 0.045s
I20260709 12:25:04.547937 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:04.545804 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 86882.1us
W20260709 12:25:04.492486  9382 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.644s	user 0.015s	sys 0.028s
I20260709 12:25:04.460958  9383 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 27.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260709 12:25:04.524011 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1356) took 1440 ms. Trace:
W20260709 12:25:04.511423  9370 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.022s	sys 0.021s
I20260709 12:25:04.555071 31748 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:04.446231  9165 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:04.521891  9289 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:04.579385 31818 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:04.566505 32370 rpcz_store.cc:276] 0709 12:25:03.083593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.096653 (+ 13060us) service_pool.cc:224] Handling call
0709 12:25:03.097112 (+   459us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.097392 (+   280us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:03.912153 (+814761us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.523553 (+611400us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:04.523625 (+    72us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:04.523685 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.523779 (+    94us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.523937 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":573395,"prepare.queue_time_us":563573,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":28,"thread_start_us":844539,"threads_started":2}
I20260709 12:25:04.569841  9306 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:25:04.554199 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.551908  9409 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.022s
W20260709 12:25:04.552575 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 84228.3us
I20260709 12:25:04.551964  9166 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:04.568596  9382 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.722s	user 0.019s	sys 0.028s
I20260709 12:25:04.572866  9265 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:04.577548 32375 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
W20260709 12:25:04.568414  9401 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.218s	user 0.000s	sys 0.002s
W20260709 12:25:04.605211 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.605950 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:04.520534  9363 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:04.592141 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:04.645555 31744 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:04.499707  9408 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:04.569869  9404 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:25:04.573777  8590 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:04.551555 32372 rpcz_store.cc:276] 0709 12:25:03.352757 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.352917 (+   160us) service_pool.cc:224] Handling call
0709 12:25:03.356641 (+  3724us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:03.430289 (+ 73648us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.430323 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:03.440024 (+  9701us) log.cc:844] Serialized 52 byte log entry
0709 12:25:03.788085 (+348061us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:03.845344 (+ 57259us) log.cc:844] Serialized 14 byte log entry
0709 12:25:03.845618 (+   274us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:03.845717 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.521903 (+676186us) raft_consensus.cc:1685] finished
0709 12:25:04.521923 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.522061 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23144,"prepare.queue_time_us":58030,"prepare.run_cpu_time_us":69,"prepare.run_wall_time_us":19415,"raft.queue_time_us":210969,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":5361,"thread_start_us":347621,"threads_started":1,"wal-append.queue_time_us":695776}
I20260709 12:25:04.622376 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:04.606473  9370 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.615986  9165 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:04.524197  9180 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.082s
I20260709 12:25:04.581526  9289 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:04.646247  9363 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:04.654807 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:04.655670 31748 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:04.656566 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:04.581148  9306 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:04.556038  9412 raft_consensus.cc:2764] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:25:04.653796 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4027) took 2088 ms. Trace:
I20260709 12:25:04.581940  9202 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:04.628298  9281 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:04.647037  9400 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.271s	user 0.000s	sys 0.001s
I20260709 12:25:04.663041 31825 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 28: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:25:04.621062 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45239.2us
I20260709 12:25:04.658152  9289 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:04.663959 31677 rpcz_store.cc:276] 0709 12:25:02.565811 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.678679 (+1112868us) service_pool.cc:224] Handling call
0709 12:25:03.685208 (+  6529us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:03.685447 (+   239us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:04.301443 (+615996us) log.cc:844] Serialized 14 byte log entry
0709 12:25:04.653114 (+351671us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:04.653171 (+    57us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:04.653220 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.653583 (+   363us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.653713 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":5721,"prepare.queue_time_us":71634,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":74,"thread_start_us":933294,"threads_started":2}
I20260709 12:25:04.671653 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:04.663553  8590 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:25:04.664474  9405 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.240s	user 0.000s	sys 0.001s
I20260709 12:25:04.647858  8992 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:04.690733 31748 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: [29.16-29.16]
W20260709 12:25:04.676141  9403 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.268s	user 0.001s	sys 0.001s
I20260709 12:25:04.721732 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4021) took 2157 ms. Trace:
I20260709 12:25:04.723062 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4004) took 1076 ms. Trace:
I20260709 12:25:04.723479 31748 rpcz_store.cc:276] 0709 12:25:03.646164 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.671318 (+1025154us) service_pool.cc:224] Handling call
0709 12:25:04.690396 (+ 19078us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:04.722786 (+ 32390us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.723004 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:04.726724  9418 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.024s
I20260709 12:25:04.729063  9289 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:04.731101  8992 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:04.732903  9289 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:04.733325  8992 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:04.738932  9345 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:04.677507  9291 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.063s
I20260709 12:25:04.658334  9370 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:04.675189  9281 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:04.743387  8590 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:04.745193  9281 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:04.722133 31676 rpcz_store.cc:276] 0709 12:25:02.564222 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.588551 (+1024329us) service_pool.cc:224] Handling call
0709 12:25:03.588991 (+   440us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:03.595223 (+  6232us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.595253 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:03.596047 (+   794us) log.cc:844] Serialized 52 byte log entry
0709 12:25:04.118613 (+522566us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:04.118698 (+    85us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.118775 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.721496 (+602721us) raft_consensus.cc:1685] finished
0709 12:25:04.721518 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.721668 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3907,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":111,"thread_start_us":522125,"threads_started":1,"wal-append.queue_time_us":778784}
I20260709 12:25:04.748245 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3154) took 1313 ms. Trace:
I20260709 12:25:04.659121  9265 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:04.697026 31674 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:04.677839 31675 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:04.748778 31747 rpcz_store.cc:276] 0709 12:25:03.434947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.446957 (+ 12010us) service_pool.cc:224] Handling call
0709 12:25:03.475504 (+ 28547us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:03.537152 (+ 61648us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:03.537189 (+    37us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:03.547802 (+ 10613us) log.cc:844] Serialized 52 byte log entry
0709 12:25:04.048316 (+500514us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:04.048414 (+    98us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.048508 (+    94us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.747942 (+699434us) raft_consensus.cc:1685] finished
0709 12:25:04.747963 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.748169 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12648,"raft.queue_time_us":13417,"raft.run_cpu_time_us":161,"raft.run_wall_time_us":7870,"thread_start_us":464640,"threads_started":1,"wal-append.queue_time_us":791440}
I20260709 12:25:04.752645 31753 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:04.753557 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4014) took 1104 ms. Trace:
W20260709 12:25:04.695606 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 323091us
W20260709 12:25:04.697038 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 221447us
W20260709 12:25:04.699322  9411 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
W20260709 12:25:04.689751  9402 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.228s	user 0.000s	sys 0.001s
I20260709 12:25:04.764590 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:04.691599 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:04.753955 31753 rpcz_store.cc:276] 0709 12:25:03.648924 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.751988 (+1103064us) service_pool.cc:224] Handling call
0709 12:25:04.752404 (+   416us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.753240 (+   836us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.753485 (+   245us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:04.695642  9166 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:04.758550 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 192259us
I20260709 12:25:04.759475  9345 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:04.808785 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51519.5us
I20260709 12:25:04.826577 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:04.826968 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 65497.9us
W20260709 12:25:04.810341 31791 leader_election.cc:336] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.571s (SENT)
I20260709 12:25:04.837154 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:04.856570 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 165052us
W20260709 12:25:04.857679 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 93165.5us
I20260709 12:25:04.858956 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:04.864969 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 363980us
I20260709 12:25:04.866060 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:04.857336 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:04.880511 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.762403  9249 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:04.895345 32342 connection.cc:256] RPC call timeout handler was delayed by -1.411210889999893s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1664s for a response
W20260709 12:25:04.893919 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:04.851724  9210 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.553s	user 0.012s	sys 0.000s
W20260709 12:25:04.906534 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.899729 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:04.756356  9383 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.108s
I20260709 12:25:04.762234  9069 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.132s
I20260709 12:25:04.899370 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:04.863353  9291 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:04.897313  9084 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:04.812300  9345 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:04.900671  9276 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:04.900254  9420 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:04.899355  9426 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:04.968744 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:04.893093  9412 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:04.951683 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:04.952997 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:04.836599  9166 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:04.893159  9038 consensus_queue.cc:1048] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.061s
I20260709 12:25:04.929468  9249 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 25.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.048s
I20260709 12:25:04.962182  9283 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:04.882993  9425 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:04.982512  9425 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:04.983024  9425 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:04.983390  9345 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:04.983747 31745 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
W20260709 12:25:04.984716  9422 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.287s	user 0.000s	sys 0.001s
I20260709 12:25:04.761324  9306 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:04.986614  9427 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:04.987443  9306 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:04.961690  9291 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:04.988415  9363 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:04.989626 32372 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:04.989880  9291 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:04.924115  9202 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.047s
I20260709 12:25:04.991003  9202 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:04.991348  9202 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:04.991607  9249 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:04.992153  9202 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:04.992194  9249 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:04.976576  9426 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:04.995179  9426 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:04.996448  9432 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.181s	user 0.001s	sys 0.000s
I20260709 12:25:04.997685 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3162) took 1430 ms. Trace:
W20260709 12:25:04.997917  8948 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.091s	user 0.014s	sys 0.000s
I20260709 12:25:04.961110  9276 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:04.999011  9276 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:04.994042  9249 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:04.998203 31749 rpcz_store.cc:276] 0709 12:25:03.567669 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.362442 (+794773us) service_pool.cc:224] Handling call
0709 12:25:04.362829 (+   387us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:04.363195 (+   366us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:04.363225 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:04.384224 (+ 20999us) log.cc:844] Serialized 26 byte log entry
0709 12:25:04.680684 (+296460us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:04.680743 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.680839 (+    96us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:04.997349 (+316510us) raft_consensus.cc:1685] finished
0709 12:25:04.997385 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:04.997598 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":80,"raft.run_cpu_time_us":182,"raft.run_wall_time_us":5100,"thread_start_us":223725,"threads_started":1,"wal-append.queue_time_us":392220}
I20260709 12:25:05.001880 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3164) took 1433 ms. Trace:
W20260709 12:25:05.002264  9398 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.688s	user 0.011s	sys 0.001s
I20260709 12:25:05.002825  9180 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:05.003187  9406 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.535s	user 0.003s	sys 0.003s
I20260709 12:25:05.003441  9249 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:05.003988 31751 raft_consensus.cc:2410] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 29 because replica has last-logged OpId of term: 28 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 15.
I20260709 12:25:05.006803  9166 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:05.007087  9363 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:04.960799  9386 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.047s
I20260709 12:25:05.008710  9386 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:05.009078  9386 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:05.009171  9417 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:05.009749  9417 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:04.976337  9370 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:05.014168 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:04.940980  9265 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.20, Last appended by leader: 20, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.002329 31746 rpcz_store.cc:276] 0709 12:25:03.568206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.335148 (+766942us) service_pool.cc:224] Handling call
0709 12:25:04.335548 (+   400us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:04.335967 (+   419us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:04.336002 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:04.336521 (+   519us) log.cc:844] Serialized 26 byte log entry
0709 12:25:04.701286 (+364765us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:04.701346 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.701443 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:05.001620 (+300177us) raft_consensus.cc:1685] finished
0709 12:25:05.001643 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.001815 (+   172us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1205,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":91,"thread_start_us":364312,"threads_started":1,"wal-append.queue_time_us":585901}
I20260709 12:25:05.018046  9417 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:04.973487 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4034) took 1808 ms. Trace:
I20260709 12:25:05.038584 31675 rpcz_store.cc:276] 0709 12:25:03.165290 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.828090 (+662800us) service_pool.cc:224] Handling call
0709 12:25:03.828396 (+   306us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.755779 (+927383us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:04.973217 (+217438us) spinlock_profiling.cc:243] Waited 849 ms on lock 0x7b54002e8454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:04.973406 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1951565184}
I20260709 12:25:04.954574 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:05.040627 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 86176.8us
I20260709 12:25:05.040071 31576 catalog_manager.cc:5697] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 17 to 27, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 27 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:05.041525 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.041826 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:05.010010  9386 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:05.023742  9162 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:05.025441  9421 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:05.027072  9425 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:04.974983  9412 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:05.030038  9280 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:05.031945 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.033202  9347 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.593s	user 0.009s	sys 0.000s
I20260709 12:25:05.036923 31744 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:05.048761  9303 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:05.042058 31576 catalog_manager.cc:5697] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 22 to 31, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 31 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:05.078073 31748 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:05.058154 31752 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:25:05.050768  9413 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.067157 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:05.076383 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.081991 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.083443  9249 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:05.083849 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.086161 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:05.087213  9249 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:05.087440  9347 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.088896 31821 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:05.077502  9412 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:05.102797  9202 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:05.070461  9421 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 24 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:05.108707  9421 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:05.114804  9383 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:05.117234  9303 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:05.117800  9303 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:05.118180  9303 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:05.118825  9413 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.065246  9162 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:05.123867  9162 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.126085  9162 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:05.130678  9249 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:05.030074  9408 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:05.153034 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:05.153256 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:05.153301 31792 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.155184  9180 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:05.152029  9276 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:05.157971  9345 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:05.158838  9166 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:05.159170  9180 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:05.159926  9180 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:05.161441  9180 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.130062  9303 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.162166 31717 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.110352  9421 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:05.179283 31825 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:05.090443  9347 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:05.149134  9413 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:05.231520 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.185125  9166 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:05.252844  9425 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:05.187994  9347 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:05.262466 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:05.265633 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 455284us
I20260709 12:25:05.273730 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.254446  8948 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:05.281198 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:05.269568  9347 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:05.285385  9347 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:05.287582  9360 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:05.288473  9360 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:25:05.290372 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 309684us
I20260709 12:25:05.293643  9289 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.294512  9289 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:05.296823  9289 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:05.297453  9289 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:05.298274 31753 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:25:05.298060  9289 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.298906 31753 raft_consensus.cc:606] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:05.300550  9289 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:05.308782  9180 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:05.312417 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:05.313221 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:05.313621 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:05.313879 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:05.314565 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:05.315440 32378 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:05.315680 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:05.316550 32377 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:05.318359 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:05.319355 32377 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 29: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:05.320423 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.321511  8590 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:05.322064 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:05.323122  9383 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.323984  9383 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:05.328338  9289 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:05.328629 31751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 4037) took 1471 ms (client timeout 1636 ms). Trace:
I20260709 12:25:05.329177  9289 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:05.329399 31751 rpcz_store.cc:269] 0709 12:25:03.857411 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.903486 (+ 46075us) service_pool.cc:224] Handling call
0709 12:25:05.328293 (+1424807us) spinlock_profiling.cc:243] Waited 1.09 s on lock 0x7b54001ed7d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:05.328531 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2516291328}
I20260709 12:25:05.331033  9337 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:05.331681  9337 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:05.334406  9337 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:05.335465  9337 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.336508 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:05.337499 31823 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:05.337988  9289 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:05.339365  9165 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.340204  9165 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.347319  8903 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:05.348100 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4041) took 1489 ms. Trace:
I20260709 12:25:05.348726 31748 rpcz_store.cc:276] 0709 12:25:03.858574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.724437 (+865863us) service_pool.cc:224] Handling call
0709 12:25:05.077899 (+353462us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:05.093243 (+ 15344us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.347791 (+254548us) spinlock_profiling.cc:243] Waited 353 ms on lock 0x7b54001a59d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:05.348011 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":812142464}
I20260709 12:25:05.347968  8903 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.351281 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:05.351691  8903 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:05.353310 32370 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:25:05.360553 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:05.362121 31822 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:25:05.363626 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:05.364686 31822 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:05.367527 31821 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 15. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
I20260709 12:25:05.369247 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:05.371747 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:05.373600 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:05.375017 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:05.375829 31825 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:05.383816 31675 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:25:05.388199 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:05.388942 31674 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 21 FOLLOWER]: Advancing to term 22
I20260709 12:25:05.391523 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:05.391579 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:05.392575  9289 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:05.394004 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:05.394362 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:05.395210 31677 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:05.396618 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:05.397938 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:05.400099 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 21 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:05.401449 31676 raft_consensus.cc:2410] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 24 because replica has last-logged OpId of term: 23 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 21 index: 14.
W20260709 12:25:05.403729 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 81757us
I20260709 12:25:05.404858 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:05.405915  9413 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:05.406853  8590 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:05.408418 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.409319 31677 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 31: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:05.410334 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:05.289949  9360 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.411931  9360 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:05.412460 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:05.414959  9417 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.415562  9417 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.416361  9038 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:05.417451  9038 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:05.417719  9417 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:05.417788  9038 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:05.418186  9165 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:05.358250 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:05.419178  9038 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.420361 31824 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:05.420817  9038 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:05.425091 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:05.425945 31675 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:05.415586 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:05.427470  9383 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:05.427597  9413 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:05.428251  9413 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.336585  9337 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:05.428773  9413 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:05.370317 31821 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:05.421336 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:05.256147  9303 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:05.264258  9420 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:05.249836  9398 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.041s
I20260709 12:25:05.254168  9283 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:05.431852 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:05.473773 31821 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:05.267798  9276 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:05.267526  9439 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:05.436107 31824 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:05.477631 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.476825 31824 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:05.457859 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:05.461447  9289 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:05.461855 32375 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
W20260709 12:25:05.451171 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.467761 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 186646us
I20260709 12:25:05.498241 31822 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 28 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:25:05.269920  9438 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:05.492316 31674 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:25:05.440392 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:05.505277 31678 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
W20260709 12:25:05.455776 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:05.451182 31716 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.016s	sys 0.002s
W20260709 12:25:05.506194 31716 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.566s	user 0.016s	sys 0.002s
I20260709 12:25:05.430550  9337 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:05.453505 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 554164us
I20260709 12:25:05.446090 31670 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:05.451479 31823 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:05.474910  9347 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:05.275637  9408 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:05.479087 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:05.523272 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:05.524272 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:05.500126 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:05.520382 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4042) took 1649 ms. Trace:
W20260709 12:25:05.451134 31750 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.001s	sys 0.017s
I20260709 12:25:05.527350 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:05.527853 31750 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.588s	user 0.001s	sys 0.017s
I20260709 12:25:05.510262 31675 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:05.450423  9370 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:05.511716 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:05.508785 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 94239.9us
I20260709 12:25:05.442390 32378 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 22.
I20260709 12:25:05.514014 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:05.500480 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:05.534808 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:05.450008  9413 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.515185  9166 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:05.468654 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 74773.2us
I20260709 12:25:05.505743  9418 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:05.444574  9442 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:05.540110 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61293.9us
I20260709 12:25:05.532394 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.540872 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:05.545571 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.546646 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:05.546913 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:05.548238 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:05.550180 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:05.476034  9420 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:05.570065 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.552150 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:05.572986 31672 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:05.507550  9303 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.034s
I20260709 12:25:05.522184  9306 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:05.508296  9383 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.034s
I20260709 12:25:05.508461  9291 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.034s
I20260709 12:25:05.580274 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:05.582880 31669 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:05.437397  8590 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:05.566583 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.548506  9355 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:05.589545 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:05.544108  8992 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:05.558908  9165 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:05.504221  9439 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:05.544625  8948 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:05.618870  9451 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:05.620728 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 25 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:05.640314 32377 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 28 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 17.
I20260709 12:25:05.621798 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:05.624606  8992 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 33 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:05.625272  9439 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:05.612934  9442 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:05.655139  9442 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:05.655633  9442 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:05.656256  9420 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:05.542559  9370 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.627056 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:05.658795 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.659157 32374 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 32: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:05.550765  9166 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:05.668406  9202 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.565s	user 0.008s	sys 0.004s
W20260709 12:25:05.669486  9412 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.570s	user 0.010s	sys 0.000s
I20260709 12:25:05.670136  9440 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:25:05.628593  9424 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.952s	user 0.008s	sys 0.008s
I20260709 12:25:05.629436  9413 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:05.630040  9165 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:05.633365 31669 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:05.634244 32371 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 32 because replica has last-logged OpId of term: 30 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 14.
W20260709 12:25:05.635787  9431 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.702s	user 0.001s	sys 0.001s
W20260709 12:25:05.637391 31752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 3170) took 1697 ms (client timeout 1527 ms). Trace:
I20260709 12:25:05.640950 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:05.533823  9408 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:05.519533  9180 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.090s
I20260709 12:25:05.648687 31675 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:05.513418  9438 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.559201  9445 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:05.671747 31672 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:25:05.602664  9418 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:25:05.674398 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 142118us
W20260709 12:25:05.682288 31752 rpcz_store.cc:269] 0709 12:25:03.939600 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:03.963269 (+ 23669us) service_pool.cc:224] Handling call
0709 12:25:05.637206 (+1673937us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b54001ed2d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:05.637320 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2316669696}
I20260709 12:25:05.721939 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.730216 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:05.731173 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:05.677222  9440 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:05.677728  9439 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:05.739562 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.680583  8948 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.590889  9283 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.13, Last appended by leader: 13, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:05.753211 32342 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.722698  9418 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:05.677510  9412 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.739813  9180 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:05.736027  9451 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:05.739209  9439 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:05.773552  9265 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:05.774083  9438 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:05.774832  9438 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:05.775597  9438 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
W20260709 12:25:05.777654  9421 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.596s	user 0.005s	sys 0.005s
W20260709 12:25:05.743324 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 197829us
I20260709 12:25:05.778770  9165 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.779430 31717 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.779671  9347 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:05.780228  9383 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:05.780969 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.783118  9383 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:05.790531 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:05.770942  9360 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 26.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.020s
I20260709 12:25:05.791421 31825 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:05.791329 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:05.755072  8948 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:05.794423 32373 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:05.792584 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:05.761474  9418 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:05.792654 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.789458 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:05.797205 32370 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:05.765558  9355 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
W20260709 12:25:05.805931 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 252309us
I20260709 12:25:05.806851 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.796255  9439 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:05.764503  9413 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.037s
W20260709 12:25:05.839306 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46723us
I20260709 12:25:05.770779  8590 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.065s
I20260709 12:25:05.792884  9363 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.047s
I20260709 12:25:05.852455 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:05.864233 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.865094  9438 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:05.796648  9451 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:05.849129  9180 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:05.848881  9434 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:05.869486 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:05.770599  8992 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:05.871097 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:05.872917 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:05.860431  9280 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.767s	user 0.007s	sys 0.007s
I20260709 12:25:05.858709  9166 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:05.872427 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:05.873759 31748 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:05.860175  9165 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:05.868021  9438 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:05.850541  9417 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.022s
I20260709 12:25:05.868201  9421 raft_consensus.cc:2764] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election decision vote started in defunct term 30: won
I20260709 12:25:05.863600  9355 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:05.888183 31752 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:25:05.870041  9162 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.691s	user 0.011s	sys 0.001s
I20260709 12:25:05.888463 31753 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:25:05.891081 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:05.892163 31746 raft_consensus.cc:2368] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 28. Current term is 29.
I20260709 12:25:05.892832 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:05.899276 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:05.902280 32372 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
W20260709 12:25:05.906718 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 133598us
I20260709 12:25:05.908792 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:05.909133 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:05.909780 32375 raft_consensus.cc:2376] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33. Re-sending same reply.
I20260709 12:25:05.914876 31792 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:05.889572  9420 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:05.891559  9355 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:25:05.944133 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 151611us
I20260709 12:25:05.945642 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:05.926257  9408 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:25:05.955250  9398 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:05.952754 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:05.957962 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58766.9us
W20260709 12:25:05.958588 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 169232us
I20260709 12:25:05.971338 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:05.953491  9414 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.339s	user 0.005s	sys 0.012s
I20260709 12:25:06.015796  9473 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:06.026952 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 112147us
I20260709 12:25:05.948563  9420 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:06.045251  9180 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:06.072139 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:06.018791  9454 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:25:06.023114  9408 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:06.047282  9472 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:06.028645  9451 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:06.169529 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 198152us
W20260709 12:25:06.182704 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:06.139698  9462 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.295s	user 0.000s	sys 0.001s
I20260709 12:25:06.108527  9464 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:06.088738  9408 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:06.116483  9265 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:06.231009 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 60281.1us
I20260709 12:25:06.019245  9438 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.119436  9472 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:06.248507 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:06.249519 32375 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:06.148150  9347 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.061177  9473 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.302251 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:06.238919  9486 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:06.251070  9474 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.373s	user 0.004s	sys 0.002s
W20260709 12:25:06.253258  9454 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.094s	user 0.000s	sys 0.000s
I20260709 12:25:06.239861  9451 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.252138  9494 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:06.240051  9476 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:06.241012  9408 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:06.251847  9468 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.316s	user 0.001s	sys 0.004s
I20260709 12:25:06.300244  9464 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:06.141603  9355 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.364691 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:06.366434 31753 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:25:06.371111 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:06.340730  9483 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:06.252381  9265 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.348366  9487 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.341199  9486 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.408807 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:06.405606 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:06.348142  9451 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:06.413224 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.375408  9038 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.322826  9180 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.427397 31748 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 33 because replica has last-logged OpId of term: 31 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 21.
W20260709 12:25:06.428074 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 320214us
I20260709 12:25:06.307196  9420 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:06.452404 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 43680.6us
W20260709 12:25:06.447943  9498 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:25:06.475728  9486 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:06.453344  9479 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:06.380862  9483 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.440912  9166 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.377502  9472 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 28.21, Last appended by leader: 21, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.443126 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:06.451294  9480 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.125s	user 0.000s	sys 0.000s
I20260709 12:25:06.463805  9473 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:06.493829  8948 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.689s	user 0.008s	sys 0.004s
W20260709 12:25:06.469617  9442 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.813s	user 0.007s	sys 0.003s
I20260709 12:25:06.492424 31818 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 24 index: 18. Preceding OpId from leader: term: 29 index: 21. (index mismatch)
I20260709 12:25:06.288725  9347 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.508330  9494 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:06.459973  9451 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:06.511044  9495 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
I20260709 12:25:06.483829  9493 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.509223  9479 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:06.642833  9493 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.643486  9479 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:06.645083  9493 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:06.645516  9434 consensus_meta.cc:287] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.770s	user 0.007s	sys 0.004s
I20260709 12:25:06.645982  9434 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:06.646071 31753 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:25:06.485743  9038 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.402737  9487 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:05.532550 31747 rpcz_store.cc:276] 0709 12:25:03.870957 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.809368 (+938411us) service_pool.cc:224] Handling call
0709 12:25:04.855971 (+ 46603us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.856184 (+   213us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:05.254589 (+398405us) log.cc:844] Serialized 14 byte log entry
0709 12:25:05.519989 (+265400us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:05.520047 (+    58us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:05.520097 (+    50us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.520180 (+    83us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:05.520311 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":621165,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":4875,"raft.queue_time_us":13758,"spinlock_wait_cycles":19005184,"thread_start_us":600040,"threads_started":2}
I20260709 12:25:06.652799  9487 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:06.529292  9166 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.579463 31748 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:25:06.550529  9442 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:06.582767 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.551071  8948 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.656353  9347 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:06.656911  9347 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.657018  8948 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:06.657366  9347 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.531951  9464 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.484087  9478 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.638907  9500 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.641160 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:06.638427  9439 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.789s	user 0.014s	sys 0.001s
I20260709 12:25:06.649335  9493 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.650725  9265 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:06.650888  9038 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:06.659392 31748 raft_consensus.cc:606] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:06.663183 31749 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:06.640269  9486 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:06.673369  9478 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:06.665217  9418 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.859s	user 0.016s	sys 0.003s
I20260709 12:25:06.690528 31748 raft_consensus.cc:614] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:06.693313  9487 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.693841  9442 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:06.715085  9355 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:06.719978  9166 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:06.694274  9485 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:06.723894  9485 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:06.696965 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55907.5us
I20260709 12:25:06.679018  9347 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:06.727414  9485 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:06.709785 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:06.678259  9439 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.731345 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1749) took 1140 ms. Trace:
I20260709 12:25:06.731824 31823 rpcz_store.cc:276] 0709 12:25:05.590447 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.590604 (+   157us) service_pool.cc:224] Handling call
0709 12:25:05.590955 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:05.742055 (+151100us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:05.742086 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:05.742129 (+    43us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:05.742159 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.731092 (+988933us) spinlock_profiling.cc:243] Waited 151 ms on lock 0x7b54001987d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:06.731137 (+    45us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.731275 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":347015936}
I20260709 12:25:06.733270  9439 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:06.733989 31826 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:06.736119 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 1439) took 1226 ms. Trace:
I20260709 12:25:06.736617 32371 rpcz_store.cc:276] 0709 12:25:05.509380 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.509507 (+   127us) service_pool.cc:224] Handling call
0709 12:25:06.735798 (+1226291us) spinlock_profiling.cc:243] Waited 99.6 ms on lock 0x7b5400546354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:06.736016 (+   218us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":229163776}
I20260709 12:25:06.711668  9483 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:06.704468  8948 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:06.712798  9503 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:06.744115  9503 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.704752  9486 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.748461  9486 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:06.749464  9437 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.171s	user 0.004s	sys 0.000s
I20260709 12:25:06.722841  9408 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.752871  9408 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:06.673914  9484 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.753713  9484 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.754504 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1362) took 1442 ms. Trace:
I20260709 12:25:06.755146 32373 rpcz_store.cc:276] 0709 12:25:05.312270 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.312401 (+   131us) service_pool.cc:224] Handling call
0709 12:25:05.312731 (+   330us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:05.794920 (+482189us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.754195 (+959275us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b540031d654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:06.754433 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1106369792}
I20260709 12:25:06.757339  9484 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:06.758699  9503 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:06.759124  9386 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:06.765058 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 1.21485e+06us
I20260709 12:25:06.766070 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:06.767578 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:06.769690 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 1.22602e+06us
I20260709 12:25:06.772671  9412 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:06.769321 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:06.777045 31826 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:25:06.687080  9472 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:06.782202 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:06.784655 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:06.790167 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4043) took 2918 ms. Trace:
I20260709 12:25:06.793603 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:06.796979 31576 catalog_manager.cc:5697] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 22 to 30, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 30 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:06.799147 31821 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:06.808090  9503 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.816367  9501 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:25:06.816903 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44011us
I20260709 12:25:06.823153 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:06.817657 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:06.823505  9501 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:06.824177  9501 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:06.824864  9503 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.825348  9501 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.817749 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:06.827473 32371 raft_consensus.cc:2368] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32.
I20260709 12:25:06.831445  9479 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.802028 31822 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:25:06.827524  9503 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:06.843106 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46903.1us
I20260709 12:25:06.828470  9501 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:06.840794 31792 leader_election.cc:400] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32.
I20260709 12:25:06.851279 31792 leader_election.cc:403] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
I20260709 12:25:06.858395 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:06.859192 31818 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:25:06.861147  9479 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:06.864871  9398 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.849s	user 0.011s	sys 0.007s
I20260709 12:25:06.865506  9398 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:06.867077 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:06.867991 31820 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
W20260709 12:25:06.835521  9428 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 2.019s	user 0.006s	sys 0.001s
I20260709 12:25:06.875368 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:06.872166 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:06.872786 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 145808us
I20260709 12:25:06.877136 32371 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:25:06.798954  9337 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:06.769356  9370 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:06.878549 32373 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:06.799681  9505 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:06.696079  9493 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.896848 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 3015) took 2273 ms. Trace:
I20260709 12:25:06.904706 31671 rpcz_store.cc:276] 0709 12:25:04.623898 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.647959 (+ 24061us) service_pool.cc:224] Handling call
0709 12:25:04.659527 (+ 11568us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:04.670664 (+ 11137us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:04.670697 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:04.671162 (+   465us) log.cc:844] Serialized 26 byte log entry
0709 12:25:05.024758 (+353596us) spinlock_profiling.cc:243] Waited 5.09 ms on lock 0x7b4800080058. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:25:05.215767 (+191009us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:05.215840 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.215963 (+   123us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:06.730141 (+1514178us) spinlock_profiling.cc:243] Waited 6.04 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:06.896548 (+166407us) raft_consensus.cc:1685] finished
0709 12:25:06.896576 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.896745 (+   169us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":10753,"raft.queue_time_us":11522,"raft.run_cpu_time_us":6662,"raft.run_wall_time_us":349437,"spinlock_wait_cycles":25599104,"thread_start_us":468564,"threads_started":1,"wal-append.queue_time_us":815780}
W20260709 12:25:06.899569  9448 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 1.309s	user 0.003s	sys 0.007s
I20260709 12:25:06.900767  9441 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:06.787949  9487 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.869827  9449 raft_consensus.cc:2764] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:25:06.874460  9413 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.034s
I20260709 12:25:06.707162  9418 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.901584 31746 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:06.683789  9500 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:06.902849  9505 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.856840  9488 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.844408  9484 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 18, Time since last communication: 0.067s
I20260709 12:25:06.871368  9202 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:06.911173 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:06.909130  9487 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:06.879822 31792 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:06.913691  9440 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.115s	user 0.013s	sys 0.023s
W20260709 12:25:06.928047  9440 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.135s	user 0.014s	sys 0.023s
I20260709 12:25:06.928766  9503 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:06.930559  9503 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:06.931133  9503 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:06.908840  9413 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 33 LEADER]: Stepping down as leader of term 33
W20260709 12:25:06.929505 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 49761.6us
I20260709 12:25:06.876824  9464 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32.
I20260709 12:25:06.910871  8803 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:06.944526 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:06.944880 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 121821us
I20260709 12:25:06.909747  9084 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:06.907166  9441 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:06.914666  9280 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.906605  9493 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:06.971758 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:06.935343  9440 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 30 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:06.962872  9210 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:07.090783 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 95933us
I20260709 12:25:07.331809 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:07.091639 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.332778 31670 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:07.334041 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.093313 31672 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:07.094261 31675 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 26 index: 15. (index mismatch)
I20260709 12:25:06.943403  9283 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.936041  9289 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:07.092397 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.334743 31670 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:07.342859 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.343246 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.344079 31670 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:25:07.345341 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.347649 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:06.993847  9420 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:06.960186  9413 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:07.117463 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.118690 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.120481  9464 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:07.121479 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:07.131048 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 22 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:06.997190  8803 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:06.992149  9084 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:06.921301  9488 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.386472 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.387279 31669 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:25:07.388213 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.389520 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.123365  9337 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.013s
I20260709 12:25:06.946885  9370 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:07.085845  9427 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:06.918092  9453 consensus_queue.cc:1059] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.212s
W20260709 12:25:07.426249 31791 leader_election.cc:336] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.558s (SENT)
I20260709 12:25:07.426908 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:06.915194  9445 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.172s
I20260709 12:25:06.992084  9509 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.161064  9441 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.15, Last appended by leader: 15, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:07.104993  9276 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:06.920739  9425 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:07.322571 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.336398 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.475270 31675 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:07.476251 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:07.359457 31670 raft_consensus.cc:2410] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 31 because replica has last-logged OpId of term: 29 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 15.
I20260709 12:25:07.362712 31643 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:07.369612 31677 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 35: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:07.479859 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.481076 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:07.378052 31671 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 22: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:07.482290 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.484191 31671 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33. Ops: [32.16-32.16]
I20260709 12:25:07.330314  9280 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:07.427574  9427 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:07.334311  8948 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.595s	user 0.010s	sys 0.007s
I20260709 12:25:07.316859  9421 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:07.315022  9438 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:07.420481 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.525772 31669 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:25:07.373733  9513 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.274s	user 0.000s	sys 0.008s
I20260709 12:25:07.534132  9450 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:07.322929  9449 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:07.536762  9449 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:07.537209  9449 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.537604  9449 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:07.537751  9345 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:07.540814 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1431) took 1177 ms. Trace:
I20260709 12:25:07.541383 31746 rpcz_store.cc:276] 0709 12:25:06.363051 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.363206 (+   155us) service_pool.cc:224] Handling call
0709 12:25:06.363612 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:06.914153 (+550541us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.540439 (+626286us) spinlock_profiling.cc:243] Waited 538 ms on lock 0x7b54004038d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:07.540728 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1236576000}
I20260709 12:25:07.363718  9420 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.370422  8903 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:07.342180  9363 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:07.468027 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 698732us
I20260709 12:25:07.560329 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:07.388892  9493 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.381506 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.364250  9210 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.531530  9421 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:07.532099 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 169533us
I20260709 12:25:07.569159 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:07.531678  9438 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.376969  9289 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:07.370786  9283 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.582662 31669 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35. Ops: [34.23-34.23]
I20260709 12:25:07.427179  9477 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:07.571318 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:07.528401  9427 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:07.578593 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:07.387410  9464 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.538532  9345 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.534983  9450 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.576635 31672 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 34: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:07.469743  9386 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.585865 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:07.558691  9420 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:07.549528  9484 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.531566  8948 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:07.592378 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:07.607519 31673 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
W20260709 12:25:07.603760 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.514235  9276 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.581626  9453 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:07.513936  9425 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:07.590720  9202 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:07.591312  9481 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:07.587291  9418 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:07.472436  9370 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.617666  9345 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.632242  9276 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:07.633400  8803 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:07.527040  9084 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.637128  9280 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:07.637590  9425 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:07.638146  9501 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.790s	user 0.010s	sys 0.008s
I20260709 12:25:07.652010  9084 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:07.638718  9500 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:07.652277  9210 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:07.613952  9386 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.654479  9386 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:07.640686 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48410.1us
I20260709 12:25:07.474339  9509 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.590531  9478 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:07.656549 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:07.649633 31670 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:07.652546  9345 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:07.581085  9363 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:07.580013  9493 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.640734  9202 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:07.674321 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:07.675271  9337 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:07.675787  9518 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:07.676458  9421 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:07.644856  9445 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:07.680984 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:07.648569  8948 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.664671  9517 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.587304  9413 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:07.565654  9440 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 28.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.658604  9509 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:07.671476  9512 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.267s	user 0.000s	sys 0.000s
I20260709 12:25:07.621924  9484 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:07.689975 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:07.671264  9520 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:07.683166  9202 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:07.701642 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:07.688856  9453 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:25:07.671718  9499 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.249s	user 0.000s	sys 0.001s
W20260709 12:25:07.688876  9503 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.757s	user 0.014s	sys 0.003s
W20260709 12:25:07.704183 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 217930us
I20260709 12:25:07.712981 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:07.697523  9445 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:07.718545 32373 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:07.700489  9289 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:07.722333 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:07.691882  8948 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.707078  9518 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:07.711975  9337 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:07.661070  9481 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:07.740319 31791 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.618s (SENT)
I20260709 12:25:07.694283  9517 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:07.714051  8992 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.337s	user 0.010s	sys 0.016s
I20260709 12:25:07.598758  9477 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:07.747470 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:07.719555  9500 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:07.755285 31791 leader_election.cc:336] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.633s (SENT)
I20260709 12:25:07.755966 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:07.715391  8803 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.035s
I20260709 12:25:07.738488  9489 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.763926  8803 consensus_queue.cc:1243] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 25.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:25:07.764569  8803 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:07.764304  9489 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.764914  8803 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:07.751047  9337 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.015s
I20260709 12:25:07.713475  9503 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:07.768004  9503 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:07.746156  9355 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.031s	user 0.017s	sys 0.000s
I20260709 12:25:07.770267  9355 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:07.743376  9289 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:07.757978  9526 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:07.774997  9523 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:07.775692  9523 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:07.776288  9481 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:07.750177  9520 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.027s
I20260709 12:25:07.777907  9520 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:07.778281  9520 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:07.739687  9518 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:07.779218  9518 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:07.783250  9450 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:07.762300 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1449) took 1971 ms. Trace:
I20260709 12:25:07.791734 31677 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:07.795755  9477 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:07.760098  9421 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:07.798974  9445 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.801554  8903 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:07.804538  9363 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:07.807775  9484 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:07.808709  9464 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:07.812649  9517 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:07.761300  9500 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:07.814893 32371 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
I20260709 12:25:07.815140  9370 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:07.785781 32377 rpcz_store.cc:276] 0709 12:25:05.790941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.792301 (+  1360us) service_pool.cc:224] Handling call
0709 12:25:05.792622 (+   321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:06.025723 (+233101us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:06.025776 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:06.077135 (+ 51359us) log.cc:844] Serialized 26 byte log entry
0709 12:25:06.335016 (+257881us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:06.335054 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.335190 (+   136us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:07.761981 (+1426791us) raft_consensus.cc:1685] finished
0709 12:25:07.762029 (+    48us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:07.762216 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17085,"raft.queue_time_us":120358,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":15289,"thread_start_us":460484,"threads_started":2,"wal-append.queue_time_us":808141}
I20260709 12:25:07.816140  9500 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:07.816813  9489 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:07.791309  9523 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:07.791414  9283 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:07.828380  9521 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.138s	user 0.003s	sys 0.000s
W20260709 12:25:07.830613 32372 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 1564) took 2526 ms (client timeout 2254 ms). Trace:
W20260709 12:25:07.836318 32372 rpcz_store.cc:269] 0709 12:25:05.304532 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.371285 (+ 66753us) service_pool.cc:224] Handling call
0709 12:25:07.830281 (+2458996us) spinlock_profiling.cc:243] Waited 529 ms on lock 0x7b5400426dd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:07.830464 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1217623936}
I20260709 12:25:07.837813  9441 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:07.841463  8590 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:07.842026  8590 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:07.843187  8590 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:07.844048  9449 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.844738  8590 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:07.853751  9450 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:07.854573  9450 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.856657 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:07.857405 31819 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
I20260709 12:25:07.860466 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.860716  9449 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:07.863971  9518 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:07.867563 31674 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:25:07.862219 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:07.887801 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:07.888710 31822 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
W20260709 12:25:07.751420  8992 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.379s	user 0.011s	sys 0.016s
I20260709 12:25:07.892203 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:07.894110 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.895712 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:07.896555 31752 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:07.898315 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:07.883658 31673 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:25:07.893997 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:07.907344 31746 raft_consensus.cc:2376] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29. Re-sending same reply.
I20260709 12:25:07.892865 31751 raft_consensus.cc:2376] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32. Re-sending same reply.
I20260709 12:25:07.909204 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:07.906221 31822 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:07.915021 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:07.918308 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:07.918666 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:07.919529 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:07.920485 31745 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 29: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:07.919559 31819 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:07.922416 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:07.923273 31746 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 35.
I20260709 12:25:07.924309 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 25 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:07.925299 31746 raft_consensus.cc:2410] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 29 because replica has last-logged OpId of term: 26 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 15.
I20260709 12:25:07.915993 31751 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 25.
I20260709 12:25:07.928385 31751 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31. Ops: [30.15-30.15]
W20260709 12:25:07.933655 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.933876 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:07.935876 31825 raft_consensus.cc:2368] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29.
I20260709 12:25:07.932034 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:07.938311 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:07.882020  8948 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:07.884120  9303 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.938588 31819 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
W20260709 12:25:07.948426 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 349109us
I20260709 12:25:07.951972 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:07.903576  9509 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:07.963891 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:07.964345 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:07.965003 31824 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:07.965335 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:07.961120 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:07.931648  9472 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.096s	user 0.014s	sys 0.000s
I20260709 12:25:07.903182  9528 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:07.965911 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:07.968458 31824 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:25:07.972065 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:07.965914 31825 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:07.974611 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:07.953603  9180 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:07.981822 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:07.983431 31673 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:07.982688 31825 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:07.989480 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:07.993799 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 271529us
I20260709 12:25:07.980206 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:07.993764 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:07.968456  9303 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.018275 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:07.970248  9509 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:08.048488 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54806.9us
I20260709 12:25:08.094882 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:08.096043  9537 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:08.096647  9537 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:08.058548 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 78449.4us
I20260709 12:25:08.034376 31673 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:08.078320 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.088066 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.099139 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:07.972815  9528 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:08.098958 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.096540 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:08.120713 31751 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:08.097347  9537 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:08.015558  9434 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:08.097901 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:08.121249 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.018167  9084 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:07.989372  9180 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.133684 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:08.126120 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:08.056703 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 334905us
I20260709 12:25:08.126911 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:08.143209 31746 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:25:08.143564 32377 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:25:08.137878  9477 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 30.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:25:08.144100 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:08.144246 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:08.092334  9518 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:08.137288 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 377899us
I20260709 12:25:08.145020 31746 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:25:08.145556 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:08.146147 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 129980us
I20260709 12:25:08.140053 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:08.146277 32372 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:08.146387 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:08.148195 31745 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:08.099145  9536 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:08.149130 31744 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:25:08.146471 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:08.147606 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:08.147938 31791 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.660s (SENT)
I20260709 12:25:08.132448  9289 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:08.162981 31819 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
W20260709 12:25:08.177345 31826 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:34642 (request call id 1750) took 2517 ms (client timeout 1813 ms). Trace:
W20260709 12:25:08.177835 31826 rpcz_store.cc:269] 0709 12:25:05.640511 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.640676 (+   165us) service_pool.cc:224] Handling call
0709 12:25:08.157359 (+2516683us) spinlock_profiling.cc:243] Waited 1.05 s on lock 0x7b540043eed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:08.157490 (+   131us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2417067264}
I20260709 12:25:08.158030 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:08.127348  9484 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.127504  9500 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:08.162657 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:08.145229  9518 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:08.157054 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1451) took 1913 ms. Trace:
I20260709 12:25:08.146170 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:08.191658 31673 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:08.127096  9363 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.231050 31746 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:08.198227 31744 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 33 index: 22. (index mismatch)
W20260709 12:25:08.163141  9427 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.552s	user 0.012s	sys 0.000s
I20260709 12:25:08.148679  9425 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:08.242165 31822 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:25:08.245941 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:08.247411 31822 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:08.233862 31791 leader_election.cc:336] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.680s (SENT)
I20260709 12:25:08.295372 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:08.296577 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62746.7us
I20260709 12:25:08.160086  9536 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:08.254292  9461 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 2.572s	user 0.005s	sys 0.019s
I20260709 12:25:08.194545  9276 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:08.255193 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:08.183104  9289 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.227560  9525 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:08.163875  9445 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:08.138739  9084 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:08.267658  9529 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.272s	user 0.000s	sys 0.001s
I20260709 12:25:08.136559  9434 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.182085  9440 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:08.269367  9447 raft_consensus.cc:3055] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:08.270874  9303 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.374287 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:08.157793  9535 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:08.276648 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.375114 31821 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:08.375242 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4059) took 2972 ms. Trace:
I20260709 12:25:08.376096 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:08.272158  9427 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.278323 32372 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:08.228523  9484 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:08.287561  9532 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260709 12:25:08.287256 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:08.227380  9500 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.229215  9518 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:08.298520 31791 leader_election.cc:336] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.742s (SENT)
W20260709 12:25:08.411623 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 113126us
I20260709 12:25:08.317561 31822 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:08.324069 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:08.414000 31718 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.520s (SENT) [suppressed 34 similar messages]
I20260709 12:25:08.360910 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:08.361414 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 217372us
I20260709 12:25:08.415251 31826 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 30 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:25:08.271821  9363 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.326538  9421 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:08.327629  9536 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:08.229903 32370 rpcz_store.cc:276] 0709 12:25:06.243224 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.243393 (+   169us) service_pool.cc:224] Handling call
0709 12:25:06.243665 (+   272us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:06.342029 (+ 98364us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:06.507920 (+165891us) log.cc:844] Serialized 14 byte log entry
0709 12:25:06.840874 (+332954us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:06.840941 (+    67us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:06.840992 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.156777 (+1315785us) spinlock_profiling.cc:243] Waited 98.1 ms on lock 0x7b54004312d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:08.156829 (+    52us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:08.156988 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":11278,"prepare.queue_time_us":36021,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":1495,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":15711,"spinlock_wait_cycles":225676032,"thread_start_us":483678,"threads_started":2,"wal-append.queue_time_us":456719}
I20260709 12:25:08.376696 31821 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:08.435575 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:08.436513 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:08.437762 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:08.438108  9427 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:08.359766  9289 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:08.361925  9202 consensus_meta.cc:287] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.647s	user 0.010s	sys 0.002s
I20260709 12:25:08.439013  9202 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:08.440764  9463 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 2.676s	user 0.000s	sys 0.019s
I20260709 12:25:08.441545  9202 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:08.361490  9445 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.442237  9445 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:08.409726 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:08.443363 31718 leader_election.cc:400] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29.
I20260709 12:25:08.443518 31670 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 29. Current term is 30. Ops: [29.21-29.21]
I20260709 12:25:08.443742 31718 leader_election.cc:403] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:08.443984  9496 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:08.444464  9496 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.444792  9496 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:08.444839 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1576) took 1108 ms. Trace:
I20260709 12:25:08.445416  8948 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:25:08.445629  9487 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:08.414464 31673 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:08.447651  9488 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.417990 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:08.449205 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:08.449853 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:08.414011 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:08.451682  9547 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.155s	user 0.001s	sys 0.000s
I20260709 12:25:08.367295  9276 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:08.452417  9276 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:08.420575 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:08.453081 31819 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 34 because replica has last-logged OpId of term: 32 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 18.
I20260709 12:25:08.438441 31821 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:25:08.422834 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:08.374697  9440 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:08.379079  9420 consensus_meta.cc:287] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.753s	user 0.010s	sys 0.003s
I20260709 12:25:08.434478 31744 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:08.454970 31826 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:25:08.445264 31670 rpcz_store.cc:276] 0709 12:25:07.336682 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.679703 (+343021us) service_pool.cc:224] Handling call
0709 12:25:07.693385 (+ 13682us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:08.444014 (+750629us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.444591 (+   577us) spinlock_profiling.cc:243] Waited 750 ms on lock 0x7b54000474d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:08.444782 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1724547328}
I20260709 12:25:08.453246 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:08.450317 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:08.375625 31676 rpcz_store.cc:276] 0709 12:25:05.402710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.423246 (+ 20536us) service_pool.cc:224] Handling call
0709 12:25:05.423714 (+   468us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:05.462093 (+ 38379us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:05.462135 (+    42us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:05.462915 (+   780us) log.cc:844] Serialized 52 byte log entry
0709 12:25:05.503690 (+ 40775us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:05.504611 (+   921us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:05.504806 (+   195us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:08.148804 (+2643998us) spinlock_profiling.cc:243] Waited 2.86 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:08.367993 (+219189us) raft_consensus.cc:1685] finished
0709 12:25:08.368017 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:08.375161 (+  7144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":37894,"raft.queue_time_us":39484,"raft.run_cpu_time_us":103,"raft.run_wall_time_us":103,"spinlock_wait_cycles":6583680,"thread_start_us":40288,"threads_started":1,"wal-append.queue_time_us":128611}
I20260709 12:25:08.374464  9447 raft_consensus.cc:740] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:08.462736 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:08.461315 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:08.411901  9500 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:08.460207 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:08.431743  9441 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.593s	user 0.014s	sys 0.000s
I20260709 12:25:08.465070 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:08.464962 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:08.467231 31824 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:25:08.421841  9518 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:08.473635 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:08.474514  9363 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.476866 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4071) took 1159 ms. Trace:
W20260709 12:25:08.477111  8590 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.632s	user 0.013s	sys 0.000s
I20260709 12:25:08.478494 31747 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:08.490267 31670 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:08.468989  9306 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:08.472223 31821 raft_consensus.cc:2376] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32. Re-sending same reply.
W20260709 12:25:08.482985 31823 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.611s	user 0.024s	sys 0.043s
I20260709 12:25:08.458920  9417 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:08.459260  9440 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:08.485059 31669 rpcz_store.cc:276] 0709 12:25:07.317895 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.583916 (+266021us) service_pool.cc:224] Handling call
0709 12:25:07.869489 (+285573us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:07.869745 (+   256us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:07.869782 (+    37us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:07.869857 (+    75us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:07.869903 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.869979 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:08.476656 (+606677us) spinlock_profiling.cc:243] Waited 285 ms on lock 0x7b54000465d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:08.476793 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":655751424}
I20260709 12:25:08.491369 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:08.506744 31744 raft_consensus.cc:2376] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28. Re-sending same reply.
I20260709 12:25:08.495355 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:08.508370 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:08.508463 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:08.471324  9166 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:08.405354  9535 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.509827 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:08.510620  9166 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:08.510790  9303 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:08.511296  9303 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.500453 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4065) took 1184 ms. Trace:
W20260709 12:25:08.502902 31823 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.631s	user 0.025s	sys 0.043s
I20260709 12:25:08.516376 31819 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:08.515357 31671 rpcz_store.cc:276] 0709 12:25:07.316003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.485169 (+169166us) service_pool.cc:224] Handling call
0709 12:25:07.485478 (+   309us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:07.557214 (+ 71736us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:07.557280 (+    66us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:07.578331 (+ 21051us) log.cc:844] Serialized 26 byte log entry
0709 12:25:07.955479 (+377148us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:07.955519 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.955595 (+    76us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:08.500224 (+544629us) raft_consensus.cc:1685] finished
0709 12:25:08.500247 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:08.500388 (+   141us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":71224,"raft.queue_time_us":71878,"raft.run_cpu_time_us":178,"raft.run_wall_time_us":21037,"thread_start_us":325548,"threads_started":1,"wal-append.queue_time_us":378420}
I20260709 12:25:08.503821 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:08.504456 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.517863 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.479816 31826 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:25:08.478708  9509 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:08.521382 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 338834us
I20260709 12:25:08.511708  9303 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:08.491941  9531 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.509780  9398 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:25:08.525332 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 149153us
I20260709 12:25:08.498114  9528 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.506211  9417 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.505300  9180 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:08.522424  9533 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.340s	user 0.000s	sys 0.001s
I20260709 12:25:08.497097  9427 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.511513  9488 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:08.498590  9507 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:08.584136 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.462606  9420 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.511727  9166 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.589358 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:08.589632 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:08.672029 31744 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:08.589999  9084 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:08.590515  9540 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.225s	user 0.000s	sys 0.000s
W20260709 12:25:08.591281  9554 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.132s	user 0.001s	sys 0.000s
W20260709 12:25:08.591609 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:08.592357  9545 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.123s	user 0.000s	sys 0.000s
W20260709 12:25:08.593061  9550 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.161s	user 0.000s	sys 0.000s
I20260709 12:25:08.593716 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:08.688706 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:08.599817  9484 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.520321  8590 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:08.502173  9306 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:08.604290  9470 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.610025  8948 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29.
I20260709 12:25:08.612262 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:08.612550 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 188553us
I20260709 12:25:08.643419 31670 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:25:08.706851 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 3036) took 1007 ms. Trace:
I20260709 12:25:08.612826  9289 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.648577  9425 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:08.644218 31576 catalog_manager.cc:5697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 25 to 31. New cstate: current_term: 31 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:08.653337  9303 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.654305  9520 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.613008  9398 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:08.659574 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:08.660274 31675 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4066) took 1344 ms. Trace:
I20260709 12:25:08.662518 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:08.679147 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 28 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:08.729672 32378 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 32 because replica has last-logged OpId of term: 30 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 14.
I20260709 12:25:08.728447 31675 rpcz_store.cc:276] 0709 12:25:07.316321 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.498121 (+181800us) service_pool.cc:224] Handling call
0709 12:25:07.561499 (+ 63378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:07.833875 (+272376us) spinlock_profiling.cc:243] Waited 5.1 ms on lock 0x7b4800080058. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:25:07.975217 (+141342us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:07.975253 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:07.988069 (+ 12816us) log.cc:844] Serialized 26 byte log entry
0709 12:25:08.118548 (+130479us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:08.118609 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.659952 (+541343us) spinlock_profiling.cc:243] Waited 79.6 ms on lock 0x7b54001d5954. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:08.660003 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:08.660052 (+    49us) raft_consensus.cc:1685] finished
0709 12:25:08.660068 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:08.660205 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22970,"raft.queue_time_us":27648,"raft.run_cpu_time_us":2649,"raft.run_wall_time_us":163021,"spinlock_wait_cycles":194809472,"thread_start_us":402981,"threads_started":2,"wal-append.queue_time_us":159980}
I20260709 12:25:08.649753  9417 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:08.732570 32377 raft_consensus.cc:2368] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29.
I20260709 12:25:08.651348  9531 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.680449 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:08.690199 31824 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 13. Preceding OpId from leader: term: 29 index: 14. (index mismatch)
I20260709 12:25:08.693643 31746 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:08.734150 32373 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:25:08.695703 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:08.668042  9488 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:08.690068  9537 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.547s	user 0.010s	sys 0.000s
W20260709 12:25:08.700170 31717 connection.cc:256] RPC call timeout handler was delayed by -1.075231150000036s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1819s for a response
I20260709 12:25:08.704474 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:08.707353 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:08.707193 31670 rpcz_store.cc:276] 0709 12:25:07.699287 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:08.470175 (+770888us) service_pool.cc:224] Handling call
0709 12:25:08.490134 (+ 19959us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.706545 (+216411us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.706780 (+   235us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:08.707152 31792 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:08.716555 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.522897  9447 consensus_queue.cc:260] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.699343  9306 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:08.759486  9561 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:08.760070  9561 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:08.760546  9561 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:08.762770 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58439us
I20260709 12:25:08.764201  9265 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:08.606427  9276 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:25:08.768991 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 61914us
I20260709 12:25:08.769862  9276 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:08.771732  9560 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:08.772559  9560 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:25:08.743633 31717 connection.cc:256] RPC call timeout handler was delayed by -1.0626311500000156s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18330000000000002s for a response
I20260709 12:25:08.698467  9535 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:08.699783  9487 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.673452  9427 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:08.746093  9556 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.135s	user 0.000s	sys 0.000s
I20260709 12:25:08.746843  9289 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.748777  9303 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.709699  9528 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.755157  9520 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:08.727602  9558 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:08.756965 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 49737.3us
I20260709 12:25:08.710657  8590 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:08.706211  9249 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.758564 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:08.676431  9507 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.845381 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:08.782343 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:08.800691  9383 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.800881 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:08.801211 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:08.803762  9515 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.217s	user 0.012s	sys 0.016s
I20260709 12:25:08.804188  9487 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.800937 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:08.854455 31820 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:08.773710  9560 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.732395  9398 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:08.774256  9438 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:08.742719  9488 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:08.736291  9434 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:08.832549  9552 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.163s	user 0.001s	sys 0.000s
I20260709 12:25:08.833323  9427 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.833204  9534 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:08.726387  9470 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.888705  9470 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:08.851410 32373 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:25:08.782413  9496 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.892185  9496 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.809298  9537 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.836998  9417 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.898079 32370 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35. Ops: []
W20260709 12:25:08.879236 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.850304  9249 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.901014  9500 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.903764  9536 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.905597 32374 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:08.906766  9531 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:08.907567  8590 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.843470  9528 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.911795  8803 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.914887 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1376) took 1196 ms. Trace:
I20260709 12:25:08.914669  9528 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:08.915307 32371 rpcz_store.cc:276] 0709 12:25:07.718163 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.733389 (+ 15226us) service_pool.cc:224] Handling call
0709 12:25:07.738164 (+  4775us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:07.815295 (+ 77131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:08.914592 (+1099297us) spinlock_profiling.cc:243] Waited 76.3 ms on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:08.914821 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":175486208}
I20260709 12:25:08.917562  9360 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:08.918090  9360 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:08.918263  9355 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:08.918850  9355 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.919129  9360 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.920214  9360 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:08.921257  9355 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:08.921875  9355 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:08.923249 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:08.924001 32377 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:25:08.925231 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:08.928601 32374 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:08.933233 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:08.934228 32377 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:08.902302  9249 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.884105  9306 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.925269  9505 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.878224  9438 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.061s
I20260709 12:25:08.844775  9558 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:08.860783  9383 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.940904  9417 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.941200  9445 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.943729  9420 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:08.953764  9558 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:08.954592  9450 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.931177  9440 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.064138 32377 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
W20260709 12:25:08.809597 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 299815us
I20260709 12:25:09.069367 31718 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:09.072265  9445 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:09.074944  9487 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:09.076328 31674 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 3023) took 1985 ms (client timeout 1679 ms). Trace:
W20260709 12:25:09.021903 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:08.849861 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.023790 31751 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
W20260709 12:25:09.078012 31674 rpcz_store.cc:269] 0709 12:25:07.116358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.121228 (+  4870us) service_pool.cc:224] Handling call
0709 12:25:09.075798 (+1954570us) spinlock_profiling.cc:243] Waited 473 ms on lock 0x7b54003926d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:09.076047 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1088380288}
I20260709 12:25:08.881906 31820 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
W20260709 12:25:09.025461 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.029301 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:09.036672 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:09.036753 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:09.086267 32374 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:25:09.081121 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:09.042531 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 260181us
W20260709 12:25:09.087132 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 238678us
I20260709 12:25:08.930269  9355 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.043406  9210 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:08.970705  9488 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:08.954843  9383 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:09.090502  9383 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:09.090987  9383 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.091382  9383 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:09.092068  9488 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:08.961966  9500 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:08.972942  9508 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 2.054s	user 0.006s	sys 0.009s
I20260709 12:25:08.973848  9507 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:08.950239  9518 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:08.979131  9514 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.504s	user 0.004s	sys 0.000s
I20260709 12:25:08.972913  9536 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:09.088526 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.107640 31744 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:09.109181 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.109995 31744 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:09.112689 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.115234  9505 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 31.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
W20260709 12:25:09.116990 31753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1433) took 2746 ms (client timeout 1627 ms). Trace:
W20260709 12:25:09.117553 31753 rpcz_store.cc:269] 0709 12:25:06.370521 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.370716 (+   195us) service_pool.cc:224] Handling call
0709 12:25:09.116547 (+2745831us) spinlock_profiling.cc:243] Waited 274 ms on lock 0x7b5400065254. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:09.116725 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":630045056}
W20260709 12:25:09.021270 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 180529us
I20260709 12:25:09.121713 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3191) took 1221 ms. Trace:
I20260709 12:25:09.065727  9440 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:09.024056 31819 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:25:09.085270 32376 raft_consensus.cc:2376] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28. Re-sending same reply.
I20260709 12:25:09.086570  9441 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.124308  9441 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:09.126137 31827 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 3896) took 2882 ms (client timeout 1819 ms). Trace:
W20260709 12:25:09.126586 31827 rpcz_store.cc:269] 0709 12:25:06.243491 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.243836 (+   345us) service_pool.cc:224] Handling call
0709 12:25:09.125682 (+2881846us) spinlock_profiling.cc:243] Waited 1.69 s on lock 0x7b54001c89d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:09.125888 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3882430208}
I20260709 12:25:09.127372 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:09.128460 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.128700 31820 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:09.088526 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:09.131163 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:09.132155 32376 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:08.968765  9537 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:09.133180 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:09.136307  9440 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:09.136960  9440 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:09.137558  9440 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:09.140983  9558 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:09.141703  9558 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.142189  9558 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:09.142781 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:09.143781 32369 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:25:09.144097 32372 raft_consensus.cc:2410] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 33 because replica has last-logged OpId of term: 32 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 22.
I20260709 12:25:09.115723 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1435) took 2738 ms. Trace:
I20260709 12:25:09.146889 31749 rpcz_store.cc:276] 0709 12:25:06.378577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.378840 (+   263us) service_pool.cc:224] Handling call
0709 12:25:06.379229 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:06.696312 (+317083us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.115172 (+2418860us) spinlock_profiling.cc:243] Waited 279 ms on lock 0x7b54002954d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:09.115499 (+   327us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":642518784}
I20260709 12:25:09.148695  9507 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:09.149410  9507 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:09.153563 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:09.156292  9355 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:09.156667 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:09.157737 32377 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:25:09.157917 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:09.167531 32374 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:09.106107 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.154500 32369 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
W20260709 12:25:09.161810  9553 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.514s	user 0.003s	sys 0.001s
I20260709 12:25:09.163115  9449 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.183867  9449 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:09.184949 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.185861 31751 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:06.803608 31750 rpcz_store.cc:276] 0709 12:25:03.871284 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:04.820790 (+949506us) service_pool.cc:224] Handling call
0709 12:25:04.881655 (+ 60865us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:04.881926 (+   271us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:05.528273 (+646347us) log.cc:844] Serialized 14 byte log entry
0709 12:25:06.720492 (+1192219us) spinlock_profiling.cc:243] Waited 3.71 ms on lock 0x7b180037e760. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a
0709 12:25:06.789518 (+ 69026us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:06.789611 (+    93us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:06.789699 (+    88us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:06.789828 (+   129us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:06.790067 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":109332,"prepare.queue_time_us":651669,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":78700,"spinlock_wait_cycles":18180224,"thread_start_us":647173,"threads_started":2,"wal-append.queue_time_us":191339}
I20260709 12:25:09.188526 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:09.188009  9507 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.16, Last appended by leader: 16, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:09.189375 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.189819  9535 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:09.190003  9522 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.127341  9441 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:09.191207  9522 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.193543 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3192) took 1292 ms. Trace:
I20260709 12:25:09.194711  9522 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:09.197160  9413 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:09.190390 31750 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:09.098502  9518 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:09.164347  9563 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.320s	user 0.000s	sys 0.001s
I20260709 12:25:09.164966  9548 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:09.114362 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.149338 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.176571 31746 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:09.177402 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:09.179268 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 22 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.181416 32372 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:25:09.154124  9493 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:09.182176 31677 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 3034) took 1762 ms (client timeout 2067 ms). Trace:
I20260709 12:25:09.164412  9520 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.200083 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:08.980671  9561 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:09.231884 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:09.232334 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:09.232930 31826 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:09.233134 31822 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:09.234040 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:09.182296 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:09.209756 31677 rpcz_store.cc:269] 0709 12:25:07.419479 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.480849 (+ 61370us) service_pool.cc:224] Handling call
0709 12:25:09.181800 (+1700951us) spinlock_profiling.cc:243] Waited 147 ms on lock 0x7b5400391cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:09.181992 (+   192us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":338554752}
I20260709 12:25:09.235116  9507 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:09.238719 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1563) took 3691 ms. Trace:
I20260709 12:25:09.240378 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.241796 31749 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 26.
I20260709 12:25:09.239758 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.239120 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:09.122357 31752 rpcz_store.cc:276] 0709 12:25:07.900066 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.900225 (+   159us) service_pool.cc:224] Handling call
0709 12:25:07.900583 (+   358us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.139756 (+239173us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:08.233339 (+ 93583us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.433987 (+200648us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:08.434055 (+    68us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:08.434108 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.120292 (+686184us) spinlock_profiling.cc:243] Waited 19.3 ms on lock 0x7b18004a6440. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419
0709 12:25:09.121040 (+   748us) spinlock_profiling.cc:243] Waited 239 ms on lock 0x7b5400294fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:09.121110 (+    70us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.121447 (+   337us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":88023,"prepare.queue_time_us":109480,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":53220,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":5290,"spinlock_wait_cycles":594029056,"thread_start_us":146153,"threads_started":1,"wal-append.queue_time_us":268127}
I20260709 12:25:09.226220 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:09.226622 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:09.223718 32376 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:09.227785  9449 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:09.227543  9520 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.231256 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:09.235224 31825 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:09.227062 31753 raft_consensus.cc:2368] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27.
W20260709 12:25:09.248217 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 45897.9us
I20260709 12:25:09.235109 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:09.243423 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:09.253316 31824 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:09.248795  9453 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:09.259795 31820 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:09.259594 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:09.239380 31678 rpcz_store.cc:276] 0709 12:25:05.546649 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:05.559023 (+ 12374us) service_pool.cc:224] Handling call
0709 12:25:05.599018 (+ 39995us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:06.920850 (+1321832us) raft_consensus.cc:1534] Early marking committed up to index 14
0709 12:25:06.920884 (+    34us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:06.966161 (+ 45277us) log.cc:844] Serialized 52 byte log entry
0709 12:25:07.463661 (+497500us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:07.463705 (+    44us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:07.827643 (+363938us) spinlock_profiling.cc:243] Waited 3.32 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:09.237897 (+1410254us) spinlock_profiling.cc:243] Waited 489 ms on lock 0x7b54001d5e54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:09.237986 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:09.238056 (+    70us) raft_consensus.cc:1685] finished
0709 12:25:09.238073 (+    17us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.238405 (+   332us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":930714,"raft.queue_time_us":855384,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":5550,"spinlock_wait_cycles":1132059904,"thread_start_us":331241,"threads_started":1,"wal-append.queue_time_us":556038,"wal-append.run_cpu_time_us":6697,"wal-append.run_wall_time_us":1175704}
I20260709 12:25:09.255342 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.267359 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.259866 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.272199 31744 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:09.255091 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:09.275089 32343 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:09.277639 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1437) took 2784 ms. Trace:
I20260709 12:25:09.277820 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:09.278745  9162 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:09.279291  9162 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.279702  9162 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:09.193980 31748 rpcz_store.cc:276] 0709 12:25:07.901188 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.901385 (+   197us) service_pool.cc:224] Handling call
0709 12:25:07.936107 (+ 34722us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.135832 (+199725us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:08.174960 (+ 39128us) log.cc:844] Serialized 14 byte log entry
0709 12:25:08.433468 (+258508us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:08.433529 (+    61us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:08.433577 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.192998 (+759421us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x7b54002ac6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:09.193087 (+    89us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.193355 (+   268us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20131,"prepare.queue_time_us":97324,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":54,"raft.queue_time_us":28582,"raft.run_cpu_time_us":109,"raft.run_wall_time_us":111,"spinlock_wait_cycles":458763520,"thread_start_us":193729,"threads_started":1,"wal-append.queue_time_us":332060}
I20260709 12:25:09.283849 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.284934 31746 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:09.286850 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:09.287192 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.287482 31826 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:09.288820 31818 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:09.293318  9520 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:09.299392 31821 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:09.300411 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:09.301770  9441 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:25:09.304471  9441 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:09.304744 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.305142  9441 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:09.305791  9441 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:09.304353 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.312948 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.312402 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.252250  9561 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:09.319298 31825 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:09.320673  9577 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.094s	user 0.000s	sys 0.001s
I20260709 12:25:09.343031  9537 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:09.322245  9447 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:09.344733 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:09.344981 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:09.346239  9561 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:09.278249 31747 rpcz_store.cc:276] 0709 12:25:06.493092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:06.652521 (+159429us) service_pool.cc:224] Handling call
0709 12:25:06.652872 (+   351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:08.487735 (+1834863us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.276935 (+789200us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54004051d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:09.277304 (+   369us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4198075136}
I20260709 12:25:09.346892  9561 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:09.347779 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.348393  8803 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:09.346201 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:09.349484 31823 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:09.347687  9561 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:09.351717 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 174603us
I20260709 12:25:09.322317 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.331353 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:09.331419  9280 raft_consensus.cc:2764] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
W20260709 12:25:09.330884 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.331137 31824 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:09.333974 31826 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
W20260709 12:25:09.334249 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 56649.1us
I20260709 12:25:09.198545  9398 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:09.337523  9518 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:09.338831 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:09.340104 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.342181 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.307518  9453 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:09.348598 31747 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:09.305887  9306 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:09.353087 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.326496 31576 catalog_manager.cc:5697] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 19 to 28, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 28 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:09.346907  9537 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.360993  9445 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:25:09.355278 31825 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:09.362303 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:09.362870 31752 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
I20260709 12:25:09.386530 31818 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:25:09.384099 31748 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:25:09.391736 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.392706 31745 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 33: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:09.398332 31746 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:09.404419 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:09.405083 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.403517 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.408583 31825 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 34: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:09.385299  9518 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:09.412153 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:09.416235 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:09.396010  9537 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:09.422310 31819 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 31: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:09.422948 31822 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:09.424388 31744 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:25:09.426795 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:09.427727 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:09.428316 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 66228.8us
I20260709 12:25:09.428794 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:09.429184 31827 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:25:09.429383 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:09.430812  9355 raft_consensus.cc:2764] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election decision vote started in defunct term 27: won
I20260709 12:25:09.431092  8903 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:09.431659  8903 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:09.432101  8903 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:09.426471 31744 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 13. Preceding OpId from leader: term: 29 index: 14. (index mismatch)
I20260709 12:25:09.428828 31752 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:25:09.439823 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:09.435389 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.447830 31821 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:09.446691 32343 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:09.450055  9283 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:09.456314  9445 consensus_queue.cc:1243] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 31.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:25:09.461437  9180 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:09.461997  9180 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.462432  9180 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:09.462941 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:09.463716 31822 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 35
W20260709 12:25:09.467018 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 50967.1us
I20260709 12:25:09.468636 31751 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:09.471705 31820 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
I20260709 12:25:09.473291 31750 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:09.475208 31823 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35. Ops: [34.20-34.20]
I20260709 12:25:09.478868  9276 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:09.478871  9520 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:09.480176  9520 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.404672  9453 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.486832  9441 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.461750  9398 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:09.489148  9441 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:09.490352 31576 catalog_manager.cc:5697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 18 to 26. New cstate: current_term: 26 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:09.492672 31576 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 16 to 27. New cstate: current_term: 27 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:09.493803 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.494737 31673 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:09.494874 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.496040 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.496241  8803 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:09.496848 31677 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:25:09.496447  9518 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.497925 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:09.498217 31822 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
I20260709 12:25:09.499511 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.500976 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.501811 31670 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:25:09.502138 31753 raft_consensus.cc:2393] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:09.503190 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.496896  8803 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:09.504521 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.505097 31753 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:09.505627  8803 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:09.509093 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4066) took 1015 ms. Trace:
I20260709 12:25:09.509528 31750 rpcz_store.cc:276] 0709 12:25:08.494065 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.409175 (+915110us) service_pool.cc:224] Handling call
0709 12:25:09.460348 (+ 51173us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.508812 (+ 48464us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.509013 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:09.509990 31676 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:25:09.510764 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.511981 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.512419 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.512786 31749 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 33 because replica has last-logged OpId of term: 32 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 15.
I20260709 12:25:09.513175 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.516196 31747 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:09.513151 31674 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
W20260709 12:25:09.517696 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 65247.4us
I20260709 12:25:09.519726 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 4055) took 1052 ms. Trace:
I20260709 12:25:09.520021 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.520179 31749 rpcz_store.cc:276] 0709 12:25:08.467452 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.253893 (+786441us) service_pool.cc:224] Handling call
0709 12:25:09.513814 (+259921us) spinlock_profiling.cc:243] Waited 198 ms on lock 0x7b54003e3ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:09.519620 (+  5806us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":456387840}
I20260709 12:25:09.523303 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.523974 31677 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:09.524955 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:09.525645 31670 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:09.525887  9421 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:09.526881 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:09.528131  9441 raft_consensus.cc:2764] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election decision vote started in defunct term 33: won
I20260709 12:25:09.528566  9280 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.529289  9280 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:09.530079  9202 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:09.530510  9202 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:25:09.517454 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 79997us
W20260709 12:25:09.533835 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 91518.1us
I20260709 12:25:09.541177 31746 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:09.437548 31748 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:25:09.543744 31748 raft_consensus.cc:606] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:09.547441 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:09.555384 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 160098us
I20260709 12:25:09.556496 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:09.558987  9202 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.565246 31748 raft_consensus.cc:614] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:09.571214 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.571640  9280 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:09.572527 31677 raft_consensus.cc:2410] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 30 because replica has last-logged OpId of term: 28 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 16.
I20260709 12:25:09.574757 31642 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:09.576900 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.577514 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:09.578514 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.578604 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.579384 31674 raft_consensus.cc:2368] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28.
I20260709 12:25:09.579649 31675 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:09.580325 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.581656 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.581804 31745 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 32. Ops: []
I20260709 12:25:09.582291  9484 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.582479 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4067) took 1088 ms. Trace:
I20260709 12:25:09.581857 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:09.582880  9551 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:09.566454  9276 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:09.586280  9276 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:09.582931 31745 rpcz_store.cc:276] 0709 12:25:08.494344 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.437591 (+943247us) service_pool.cc:224] Handling call
0709 12:25:09.541329 (+103738us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.582223 (+ 40894us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.582403 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:09.586731  9276 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:09.591639  9440 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:09.544006  9355 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:09.595170  9355 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:09.596401  9355 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.598759 31752 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:25:09.599453 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4069) took 1102 ms. Trace:
I20260709 12:25:09.553478 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:09.599838 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.600674 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.601648 31669 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 34: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:09.599888 31752 rpcz_store.cc:276] 0709 12:25:08.496662 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.474344 (+977682us) service_pool.cc:224] Handling call
0709 12:25:09.544308 (+ 69964us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:09.599180 (+ 54872us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.599386 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:09.602371  9558 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:09.600737  9536 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:09.603397  9500 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:09.603546  9536 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:09.604092  9536 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:09.604827  9558 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:09.605639 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.605841  9520 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:09.604040  9500 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:09.607612  9518 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:09.607402  9500 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:09.595124 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.609023 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.609644 31674 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:25:09.584219  9202 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:09.600929 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59801.3us
I20260709 12:25:09.599743 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.612625 32343 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:09.600435 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55941us
I20260709 12:25:09.614300  9537 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.616559 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:09.616927  9537 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:09.617693  9537 consensus_queue.cc:1059] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:25:09.582984  9484 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.601143 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1573) took 2232 ms. Trace:
I20260709 12:25:09.626485 31824 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:09.626557 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:09.638958 31823 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:09.628194 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 26 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.629086 31748 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:25:09.619483  9360 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:09.641891  9360 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:09.641376 31748 raft_consensus.cc:606] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:09.612653 31671 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:09.628810 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.636039 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:09.622381 32373 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:09.627521 31676 raft_consensus.cc:2376] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28. Re-sending same reply.
I20260709 12:25:09.644562 31677 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:09.652328 31670 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:09.658468 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.629863  9162 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:09.659157 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 100496us
I20260709 12:25:09.659981 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:09.619308  9280 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:09.660765  9162 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:09.661264 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.661365  9280 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:09.662034  9447 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:09.664749 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.667125 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.666816  9360 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.667769 31745 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:09.621280 31672 rpcz_store.cc:276] 0709 12:25:07.318897 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:07.613577 (+294680us) service_pool.cc:224] Handling call
0709 12:25:07.635117 (+ 21540us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:07.663734 (+ 28617us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:07.663769 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:07.664275 (+   506us) log.cc:844] Serialized 26 byte log entry
0709 12:25:09.180574 (+1516299us) spinlock_profiling.cc:243] Waited 4.43 ms on lock 0x7b1800302920. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:25:09.330997 (+150423us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:09.331070 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.331972 (+   902us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:09.531287 (+199315us) raft_consensus.cc:1685] finished
0709 12:25:09.531308 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:09.550751 (+ 19443us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":28201,"raft.queue_time_us":29584,"raft.run_cpu_time_us":164,"raft.run_wall_time_us":5470,"spinlock_wait_cycles":10185984,"thread_start_us":69703,"threads_started":1,"wal-append.queue_time_us":1699176}
I20260709 12:25:09.618599  9537 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:09.627609  9413 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:09.675062 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.631266  9440 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:09.642040  9536 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.685415 31745 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:09.689059 31748 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:25:09.691012 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.688436 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.700217 31642 leader_election.cc:400] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27.
I20260709 12:25:09.700803 31642 leader_election.cc:403] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:09.692625 31671 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:25:09.702771 31677 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:09.706837 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.709683 31676 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 31: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:09.664551  9276 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.713225 31747 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:09.717971 31752 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:25:09.718904 31752 raft_consensus.cc:606] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:09.719353 31752 raft_consensus.cc:614] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:09.691120  9360 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:09.722651 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.724860 31746 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 31: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:09.699061  9413 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 35 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:09.727072 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.730362 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.730362 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:09.730914 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 71067.3us
I20260709 12:25:09.731942 31744 raft_consensus.cc:2368] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32.
I20260709 12:25:09.734170 31745 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 29: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:09.730767 31576 catalog_manager.cc:5697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 21 to 30. New cstate: current_term: 30 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:09.737039 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.732393 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:09.741235 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:09.741897 31677 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:25:09.742700 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260709 12:25:09.742952 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.743865 31672 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:09.745086 31673 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:09.747138 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:09.748780 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:09.703626  9561 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:09.753023 31753 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:25:09.739722 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:09.756307 31677 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:09.761080 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:25:09.758760  9383 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.667s	user 0.012s	sys 0.003s
I20260709 12:25:09.760902  9535 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:09.737248  9537 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.033s
W20260709 12:25:09.782426 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:09.785370 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.784469 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.767041  9520 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 27
I20260709 12:25:09.790452 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:09.795166 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.798117 31678 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 29 index: 22. (index mismatch)
I20260709 12:25:09.795190 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:09.790096 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 118646us
I20260709 12:25:09.799543 31642 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:09.799779 31674 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:09.786564  9500 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:09.805457  9500 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:09.805884  9500 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:09.807696  9276 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:09.808779 31673 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 30: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:09.809856 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:09.810425 31673 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:09.814790 31671 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:09.816293 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.773016  9383 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:09.826691 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 87096.1us
I20260709 12:25:09.828508 31673 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:09.820839 31669 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:09.833039 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.797554  9551 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.797955  9535 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:09.792337  9484 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:09.830560 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:09.837754 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:09.791018  9561 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:25:09.847271 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.845959 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:09.849504 31678 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:09.848013 31676 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 22.
I20260709 12:25:09.826050  9445 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:09.853664 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.855471 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.839926  9280 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:09.872071 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.848272  9276 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.875168 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.877427 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:09.878050 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:09.882915 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.883698 31671 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
W20260709 12:25:09.884727 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 85287.7us
I20260709 12:25:09.887665  9536 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:09.850754  9180 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:09.892891  9180 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:09.865375  9484 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:09.893844  9484 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.895907  9383 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:09.896979 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:09.898072 31674 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:09.899391 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:25:09.904896 31791 connection.cc:256] RPC call timeout handler was delayed by -1.7495367470000929s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21250000000000002s for a response
W20260709 12:25:09.905447 31791 connection.cc:256] RPC call timeout handler was delayed by -1.7018086210000547s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2066s for a response
I20260709 12:25:09.906349 31670 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 33 because replica has last-logged OpId of term: 32 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 22.
I20260709 12:25:09.912607 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:09.913532 31673 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:09.915913 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:25:09.919942 31570 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42070 (request call id 226) took 104946 ms (client timeout 15000 ms). Trace:
I20260709 12:25:09.925649 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:09.920578  9484 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:09.914462 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:25:09.876470 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 46084.9us
I20260709 12:25:09.927242 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:09.928438 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 375495us
W20260709 12:25:09.925009 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 73438.6us
I20260709 12:25:09.931661 31671 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:09.933696 31823 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:09.937050 31673 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:09.938545 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:09.939558 31821 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:09.938966 32343 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:09.941781 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 89785.4us
I20260709 12:25:09.942878 31673 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:09.942690 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:09.944058  8803 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:09.952493 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:09.948809 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:09.881067  9280 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:09.958299 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:09.959200 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:09.959728 31824 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:09.961513 31673 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:25:09.961717 31746 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:25:09.928777  9445 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:09.850664  9413 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.959314 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:09.857898  9551 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.923374  9276 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:09.975293 31717 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:09.964921  9398 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:09.967907  9537 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:09.988754 31676 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:09.973214  9453 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:25:09.989635  8903 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.557s	user 0.015s	sys 0.000s
I20260709 12:25:09.996726 31822 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:09.995255 31675 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:09.990569 31576 catalog_manager.cc:5697] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 16 to 26. New cstate: current_term: 26 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:09.997735 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:10.012495 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 65487.5us
I20260709 12:25:09.984645  9398 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.018772 31824 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:25:09.996511  9453 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:09.986950  9484 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:10.004575  9537 raft_consensus.cc:697] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:10.076167 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:10.021740  9398 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:10.019622  8803 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:10.087567  9447 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:10.089483  9536 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:10.089753 31672 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:25:10.087838  9084 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:10.090451  9084 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:10.088255  9580 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.064s	user 0.000s	sys 0.001s
I20260709 12:25:10.090890  9084 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:10.091149 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:10.092188 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4070) took 1595 ms. Trace:
I20260709 12:25:10.092583 31751 rpcz_store.cc:276] 0709 12:25:08.496933 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:09.479754 (+982821us) service_pool.cc:224] Handling call
0709 12:25:09.480421 (+   667us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:09.481036 (+   615us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:09.481075 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:09.486809 (+  5734us) log.cc:844] Serialized 26 byte log entry
0709 12:25:09.761351 (+274542us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:09.761400 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:09.761481 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:10.091999 (+330518us) raft_consensus.cc:1685] finished
0709 12:25:10.092019 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:10.092133 (+   114us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":798,"raft.run_cpu_time_us":192,"raft.run_wall_time_us":189,"thread_start_us":273314,"threads_started":1,"wal-append.queue_time_us":473733}
I20260709 12:25:10.088752 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:10.090001  9536 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.096462  9536 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:10.100064  9447 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.102756  9447 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:10.104965  9447 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:10.100901 32343 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:10.113587 31823 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:25:10.000376  9280 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:10.115803 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 142406us
I20260709 12:25:10.034986  9484 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.116369 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:10.032321  9453 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:10.108336 31576 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 17 to 24, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 24 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:10.129721 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:10.134789 31826 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
W20260709 12:25:10.130800 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56946.3us
I20260709 12:25:10.077342  9162 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:10.008039  8903 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:10.099817  9084 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:10.141222 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 61498.2us
W20260709 12:25:10.172209 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 266298us
I20260709 12:25:10.148039 31678 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
I20260709 12:25:09.991429  9535 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:10.142998  9484 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:10.190658 31823 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 30. Ops: []
W20260709 12:25:10.186861 31791 connection.cc:256] RPC call timeout handler was delayed by -1.5058488470003795s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17200000000000001s for a response
I20260709 12:25:10.201864 31673 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:25:10.202451 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:10.206985 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 133181us
I20260709 12:25:10.210836  9548 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:10.210336 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:10.213644 31791 connection.cc:256] RPC call timeout handler was delayed by -1.3814577359999021s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
W20260709 12:25:10.214221 31791 connection.cc:256] RPC call timeout handler was delayed by -1.3679577359998802s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15400000000000003s for a response
W20260709 12:25:10.214613 31791 connection.cc:256] RPC call timeout handler was delayed by -1.1630159559999811s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1857s for a response
W20260709 12:25:10.215009 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0054874820002624s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2049s for a response
I20260709 12:25:10.215934 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:10.216890  9180 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:10.217694  9561 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:10.220021 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 22 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:10.220855  9561 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Leader election lost for term 22. Reason: could not achieve majority
I20260709 12:25:10.224826  9360 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:10.225930  9360 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:10.228305  9084 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:10.223230 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:10.229543  9283 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:10.231451  9561 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:10.233721  9276 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:10.234580  9276 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:10.243779  8903 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:10.197520 31576 catalog_manager.cc:5697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 18 to 27. New cstate: current_term: 27 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:10.241520 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:10.247130 31576 catalog_manager.cc:5697] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 18 to 25, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 25 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:10.249058  9561 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:10.249703  9561 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:10.249161 31576 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 22 to 26, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 26 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:10.250160  9561 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:10.251235  9383 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.251811  9383 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:10.142838  9162 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:25:10.256435 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 80986.2us
W20260709 12:25:10.270505 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 78169.6us
I20260709 12:25:10.272971 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:10.273816 32370 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:10.274325 31823 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:25:10.273129 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:10.254428  9383 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:10.275796 32371 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:25:10.282603  9551 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:10.293269 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:10.279975  9518 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.672s	user 0.012s	sys 0.002s
I20260709 12:25:10.298772 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:10.299872 31823 raft_consensus.cc:2368] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29.
I20260709 12:25:10.121963  9445 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:09.924702 31570 rpcz_store.cc:269] 0709 12:23:25.028618 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:25.066310 (+ 37692us) service_pool.cc:224] Handling call
0709 12:25:09.919654 (+104853344us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:24:59.921323 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:24:59.922067 (+   744us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:24:59.922125 (+    58us) write_op.cc:435] Acquired schema lock
0709 12:24:59.922143 (+    18us) tablet.cc:662] Decoding operations
0709 12:24:59.922615 (+   472us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:24:59.922730 (+   115us) write_op.cc:641] Partition lock acquired for write op
0709 12:24:59.922758 (+    28us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:24:59.923217 (+   459us) tablet.cc:701] Row locks acquired
0709 12:24:59.923265 (+    48us) write_op.cc:260] PREPARE: finished
0709 12:24:59.923452 (+   187us) write_op.cc:270] Start()
0709 12:24:59.923604 (+   152us) write_op.cc:276] Timestamp: P: 1783599899923386 usec, L: 0
0709 12:24:59.923650 (+    46us) op_driver.cc:348] REPLICATION: starting
0709 12:24:59.924330 (+   680us) log.cc:844] Serialized 925 byte log entry
0709 12:25:01.635278 (+1710948us) op_driver.cc:464] REPLICATION: finished
0709 12:25:01.974498 (+339220us) write_op.cc:301] APPLY: starting
0709 12:25:01.986811 (+ 12313us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:02.091148 (+104337us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:02.091264 (+   116us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:02.092584 (+  1320us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:02.092861 (+   277us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:25:02.092933 (+    72us) write_op.cc:312] APPLY: finished
0709 12:25:02.093248 (+   315us) log.cc:844] Serialized 33 byte log entry
0709 12:25:02.093457 (+   209us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:02.093549 (+    92us) write_op.cc:454] Released schema lock
0709 12:25:02.093615 (+    66us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:02.094100 (+   485us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:02.094146 (+    46us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":2442429,"reported_tablets":50,"spinlock_wait_cycles":2432,"child_traces":[["op",{"apply.queue_time_us":277833,"apply.run_cpu_time_us":6142,"apply.run_wall_time_us":148394,"cfile_cache_hit":10,"cfile_cache_hit_bytes":12483,"num_ops":2,"prepare.queue_time_us":329725,"prepare.run_cpu_time_us":10603,"prepare.run_wall_time_us":125046,"raft.queue_time_us":479500,"raft.run_cpu_time_us":34146,"raft.run_wall_time_us":118170,"replication_time_us":1711435,"thread_start_us":545338,"threads_started":4,"wal-append.queue_time_us":280961,"wal-append.run_cpu_time_us":30926,"wal-append.run_wall_time_us":2883161}]]}
I20260709 12:25:10.303174 31576 catalog_manager.cc:5697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 21 to 29. New cstate: current_term: 29 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:10.315115 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:10.334062 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 233225us
I20260709 12:25:10.295684  9383 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:10.338512 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:10.284351  9536 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:10.352152 31676 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
W20260709 12:25:10.355125 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 182698us
I20260709 12:25:10.363932 31826 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:10.365314 31822 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:25:10.366569 31718 leader_election.cc:400] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29.
I20260709 12:25:10.370241 31576 catalog_manager.cc:5697] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 22 to 28, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 28 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:10.344779  9383 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.376077 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:10.377233  9306 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:10.377633  9306 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.377962  9306 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:10.289912  9537 consensus_queue.cc:237] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:10.350914  9500 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.545s	user 0.012s	sys 0.000s
I20260709 12:25:10.343231  9586 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:10.334126  9162 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:10.399710  9162 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:10.401621 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 115030us
W20260709 12:25:10.404309 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 98606.8us
I20260709 12:25:10.289047  8803 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:10.393206  9383 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:10.423466  9581 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
W20260709 12:25:10.429025 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53051.7us
I20260709 12:25:10.430234 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:10.431181  9507 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:10.433665 31717 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:10.354132  9360 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 30.15, Last appended by leader: 15, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:10.452564 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:10.454463 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:10.455129 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.459800 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:10.464514 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:10.464980 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:10.465380 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.467170 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:10.471868 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:10.472031 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:10.474442 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61394.3us
I20260709 12:25:10.475337 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:10.403249  9500 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:10.462289  9180 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:10.478972  8903 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:10.492965  9561 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:10.493671  9536 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:10.495208 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.495107 31820 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 36 because replica has last-logged OpId of term: 34 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 22.
W20260709 12:25:10.497857 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 172326us
W20260709 12:25:10.494621 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 284434us
I20260709 12:25:10.493783  9276 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:10.452085  9355 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.032s
I20260709 12:25:10.482007  9507 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:10.475090  9306 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:10.522437  8903 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:10.540469  9591 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.089s	user 0.000s	sys 0.000s
I20260709 12:25:10.531251  9180 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:10.525462  8803 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:10.541215  8903 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:10.545465 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 111902us
W20260709 12:25:10.529016  9520 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.725s	user 0.012s	sys 0.000s
I20260709 12:25:10.541926  9180 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:10.553485  9180 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:10.554800 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 101144us
I20260709 12:25:10.494598  9536 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
I20260709 12:25:10.499963  9586 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.047s
I20260709 12:25:10.570499 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:10.583037 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 172032us
I20260709 12:25:10.585747 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:10.586500 31823 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:10.556936  9520 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 26. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27.
I20260709 12:25:10.555508  9447 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
W20260709 12:25:10.588747 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 111993us
I20260709 12:25:10.590924 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:10.594139 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:10.595775 31819 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:10.598275 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:10.598564 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:10.601929 31827 raft_consensus.cc:2393] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:10.604550 31643 leader_election.cc:400] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29.
I20260709 12:25:10.604954 31643 leader_election.cc:403] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:10.605681 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:10.577235  9561 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:10.611629 32372 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:25:10.591578  9507 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:10.610827  9551 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.009s
W20260709 12:25:10.622815 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.625238 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:10.634971 31819 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:10.639129 32370 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:10.610584 31677 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:25:10.615420 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:10.616321 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 66452.5us
I20260709 12:25:10.643497  9593 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:10.644116  9593 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.644542  9593 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:10.613979  9520 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 28. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29.
I20260709 12:25:10.659047 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:10.666484  9584 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.654s	user 0.002s	sys 0.002s
I20260709 12:25:10.666855  9592 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Leader pre-election won for term 23
I20260709 12:25:10.667348  9592 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.667722  9592 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:10.669606  9605 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:10.670312  9605 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:10.671403  9605 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 27.19, Last appended by leader: 19, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:10.674716  9084 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:10.675411 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:10.677944  9276 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:10.680533  9507 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:10.680968  9507 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:10.681890  9507 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:10.684630 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:10.692881 31751 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 22.
I20260709 12:25:10.685812 31747 raft_consensus.cc:1217] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Deduplicated request from leader. Original: 23.14->[24.15-28.17]   Dedup: 24.15->[26.16-28.17]
I20260709 12:25:10.686558 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:10.694271  9597 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:10.694767 31746 raft_consensus.cc:2368] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29.
I20260709 12:25:10.687348 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:10.696094 31744 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 36 because replica has last-logged OpId of term: 34 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 22.
I20260709 12:25:10.694337 31747 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: [24.15-28.17]
W20260709 12:25:10.698565 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.703084  9597 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.703852  9597 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:10.706739  9551 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.707621  9551 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:10.708590  9603 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:25:10.663761  8803 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.043s
I20260709 12:25:10.716773 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:10.718982 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 72961.3us
I20260709 12:25:10.720212 31642 leader_election.cc:400] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29.
W20260709 12:25:10.723730 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48383us
I20260709 12:25:10.741685 31823 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:25:10.744035 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:10.744434 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 146327us
I20260709 12:25:10.744506 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:10.747429 31751 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:10.744828 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 145129us
I20260709 12:25:10.743913 31745 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:25:10.753435 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:10.754093 31744 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:10.757679 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:10.715590  9280 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:10.763645 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:10.728020  9500 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:10.756106  9306 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:10.780416 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 181663us
I20260709 12:25:10.786594 31642 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:10.751941  9276 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:10.759207  9355 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:10.799062 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:10.802513 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:10.792292  9586 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.792054  8903 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:10.816381  9597 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:10.823722  9586 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:10.828256 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:10.829145 31823 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:25:10.834049 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:10.803578  9520 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:10.838202 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:10.826368  9586 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:10.857765 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 130063us
I20260709 12:25:10.872351 32373 raft_consensus.cc:1240] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 30. Ops: []
I20260709 12:25:10.849906  9413 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:10.875144 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 94148.1us
I20260709 12:25:10.876703 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:10.832034  9355 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:10.831475  8903 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:25:10.883198 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49833.4us
I20260709 12:25:10.866484  9500 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.843401  9520 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:10.885349  9180 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:10.887605  9500 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:10.894424  9610 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:25:10.896042  9500 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:10.897166  9484 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.637s	user 0.010s	sys 0.001s
I20260709 12:25:10.897599  9484 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:10.899183  9551 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:10.899267  9355 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:10.905726 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:10.906790 32370 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 28: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:10.905856  9536 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:25:10.907490  9398 consensus_meta.cc:287] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.765s	user 0.012s	sys 0.001s
I20260709 12:25:10.907941  9398 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:10.910120  9398 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:10.910943  9283 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:10.911378  9283 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:25:10.911545  9453 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.745s	user 0.013s	sys 0.000s
I20260709 12:25:10.912010  9453 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:10.912281  9283 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:10.913563  9283 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:10.914278  9453 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:10.914706  9520 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:10.915480 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:10.947438 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 65399.2us
I20260709 12:25:10.947803 32370 raft_consensus.cc:2368] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28.
W20260709 12:25:10.915835 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 101496us
I20260709 12:25:10.916643 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:10.914727 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:10.918246 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:10.918931 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:10.920718 32369 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:25:10.921384 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:10.921743 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:10.922009  9598 consensus_queue.cc:1175] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 30 status { last_received { term: 30 index: 16 } last_committed_idx: 16 last_received_current_leader { term: 30 index: 16 } }
I20260709 12:25:10.923013 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:10.965157 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:10.966952 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:10.969168 31819 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:25:10.970613 32377 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 34: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:10.889654  9522 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:10.931385  9484 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:10.886502  8903 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:10.981678 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:10.976516 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:10.986111 31824 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:25:10.932775  9561 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:10.891585  9586 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:10.989391 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:25:10.989822 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:10.990429 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:10.990686 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:10.984095 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:11.011435 31674 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:11.012941 31678 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:25:10.956482  9548 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.023190 31670 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
W20260709 12:25:11.030056 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 48485.4us
I20260709 12:25:11.021437 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:11.026779 31669 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:10.978191  9594 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.009714  9413 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:11.018373  9383 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.600s	user 0.008s	sys 0.007s
W20260709 12:25:11.025769  9162 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.625s	user 0.009s	sys 0.005s
W20260709 12:25:11.054353 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 128341us
W20260709 12:25:11.054710 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47307.5us
W20260709 12:25:11.050019  9596 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.187s	user 0.001s	sys 0.001s
I20260709 12:25:11.069182 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:11.069897 31676 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:11.072816 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 107239us
I20260709 12:25:11.074316 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:11.075182 31824 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:10.980338  9593 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.077883  9593 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:11.078451  9622 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:11.078907 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:11.079579 31752 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:11.081810  9180 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:11.081910 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:11.083010 32378 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 33. Current term is 34. Ops: [33.22-33.22]
I20260709 12:25:11.084794 31748 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:11.061582  9383 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.084362 31744 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
W20260709 12:25:11.091691  9618 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:25:11.101959 31748 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
W20260709 12:25:11.103426  9619 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.056s	user 0.000s	sys 0.000s
I20260709 12:25:11.105094 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:11.107537  9447 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:11.104015  9383 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:11.018069  9561 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.117419 31824 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
I20260709 12:25:11.047433  9606 raft_consensus.cc:2764] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election decision vote started in defunct term 33: won
I20260709 12:25:11.113749 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:11.163851 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50190us
I20260709 12:25:11.164044  9535 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:25:11.164479 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:11.158852 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 272585us
I20260709 12:25:11.164780  9535 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:11.159736 32374 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:25:11.166293  9535 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:11.175563 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 43985.6us
I20260709 12:25:11.064100  9276 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:11.062320  9616 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:11.131930  9580 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:25:11.133473  9447 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.172381  9561 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:11.184809 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:11.043154  9548 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:11.161846  9602 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.359s	user 0.001s	sys 0.001s
W20260709 12:25:11.183231 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51732.4us
I20260709 12:25:11.047155  9594 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.174768  9586 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:11.189743  9596 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:25:11.135130  9453 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.074s
I20260709 12:25:11.210968  9383 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:11.189633  9447 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:11.184051  9616 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.075621  9306 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.115s
I20260709 12:25:11.230104  9616 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:11.229168 31677 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:11.197587 31676 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
W20260709 12:25:11.235737  9599 log.cc:927] Time spent T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.376s	user 0.001s	sys 0.001s
I20260709 12:25:11.237716  9604 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.238394  9604 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.240801  9604 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:11.241962  9613 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.189s	user 0.001s	sys 0.000s
W20260709 12:25:11.244189 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 85656.2us
I20260709 12:25:11.245121 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:11.246121  9561 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:11.246878  9180 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:11.249361  9561 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:11.250152  9561 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:11.253253 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:11.221489  9383 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:11.259087  9383 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:11.266954 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53664.7us
I20260709 12:25:11.268059 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:11.275660 31751 consensus_meta.cc:287] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.519s	user 0.010s	sys 0.001s
I20260709 12:25:11.215802  9586 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:11.276129 31751 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
W20260709 12:25:11.281059 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61311.8us
I20260709 12:25:11.283280 31642 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:11.182462  9622 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.090s
I20260709 12:25:11.291280 31717 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:11.287187  9548 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:11.289332  9447 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.301872  9447 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:11.274933  9283 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.303129  9283 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.304260  8803 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:11.304471 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:11.304872  8803 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:11.305837 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:11.305656  9283 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:11.308074  9594 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:11.312216  9592 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.644s	user 0.011s	sys 0.000s
I20260709 12:25:11.312736 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:11.312561  9180 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:25:11.226049  9606 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.318495 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:11.320472 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:11.324352 31753 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:11.326500 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:11.326759  9306 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:11.295044  8903 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.330179  8903 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:11.328106 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:11.330935  9626 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.098s	user 0.003s	sys 0.000s
I20260709 12:25:11.332132  9398 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:11.333076  9633 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.090s	user 0.001s	sys 0.000s
I20260709 12:25:11.333319  9594 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:11.336685 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:11.337620 31751 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:25:11.312448  9507 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:11.280053  9084 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:11.340695 31744 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:11.336006 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:11.341570 31717 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:11.350854 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:11.351725 31752 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:11.354475 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:11.353565 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:11.357487 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:11.333820  9594 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.315445  9520 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.360354 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:11.363512 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:11.334012  9592 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.349509  9616 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.372610  9627 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:11.372957  9520 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:11.373251 31753 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:11.373420  9627 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 35 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:11.373898  9592 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:11.374754  9629 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:11.375268  9629 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:11.375672 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:11.356608  9622 raft_consensus.cc:2764] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election decision vote started in defunct term 32: won
I20260709 12:25:11.376405 31745 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:11.334362  9606 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.364027  9597 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:11.378918  9597 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.305855  8803 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.375730  9629 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:11.356464  9306 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:11.379621  9597 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:11.385072  9548 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:25:11.385939  9636 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:25:11.355368  9180 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:11.391829 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 59900.2us
I20260709 12:25:11.363601  9398 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:11.408241 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:11.364286  9594 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:11.406687 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:11.410893 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:11.366694  9598 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:11.363627  9355 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:11.410137 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:11.416266 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:11.416419 31821 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:25:11.408993 31824 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:11.416944 31753 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:11.405515  8903 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:11.378823  9606 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:11.416031 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 23 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:11.416641 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48269.4us
I20260709 12:25:11.411687 32372 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:11.404973  9084 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:11.422400 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 24 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:11.425246 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:11.426049 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:11.426122 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:11.431260 31818 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:25:11.431959 32376 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:11.432883 31819 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:11.433008 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:11.433283 31819 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:11.434334 31826 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:25:11.435889 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:11.436652 32370 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:11.439239 32377 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:25:11.439386 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:11.444727 31827 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:11.421279  9355 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.454329 31753 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:11.444553 31819 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.429617  9398 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:11.459197 31824 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:11.459976 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 23 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:11.467329 32376 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 22 FOLLOWER]: Advancing to term 23
I20260709 12:25:11.468765 31819 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:11.470337 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:11.482785 31753 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:25:11.425976  9586 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.431710  8903 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.438781  9598 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:11.451841  9355 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:11.500588 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.488946 32370 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:11.374538  9627 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.453105  9084 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:11.469501  9447 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:11.506110 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 90172.7us
I20260709 12:25:11.525697 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:11.508714 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:11.512622 31819 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:25:11.511489 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:11.469169  9616 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:11.520757 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64774.8us
I20260709 12:25:11.531415 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:11.459606  9180 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.521512 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:11.536695 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:11.541352 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:11.534772 32376 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:25:11.522063 31818 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:11.424024  9621 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.082s
I20260709 12:25:11.518452  9413 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.514472  9360 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.021s
I20260709 12:25:11.543591 32369 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 28: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:11.544633 31671 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:25:11.496999  9484 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.048s
I20260709 12:25:11.560675  9616 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:11.560875 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:11.562036 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1578) took 1537 ms. Trace:
I20260709 12:25:11.518468  9598 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.564335  9598 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:11.565028  9598 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:11.565572  9598 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:11.511945  9606 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.047s
I20260709 12:25:11.555249  9447 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.527223  9644 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 27.19, Next index: 20, Last known committed idx: 18, Time since last communication: 0.033s
I20260709 12:25:11.562357 31718 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:11.562440 32373 rpcz_store.cc:276] 0709 12:25:10.024719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.031112 (+  6393us) service_pool.cc:224] Handling call
0709 12:25:10.112455 (+ 81343us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:10.872955 (+760500us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.561759 (+688804us) spinlock_profiling.cc:243] Waited 757 ms on lock 0x7b540071d4d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:11.561948 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1741233024}
I20260709 12:25:11.554534  9180 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.524531  9561 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.587014  9084 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 25.13, Last appended by leader: 13, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.588307  9180 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:11.589059  9561 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:11.543172  9202 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:11.567616  9640 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.099s	user 0.002s	sys 0.000s
I20260709 12:25:11.599994 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:11.602463  9535 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:11.603226 31819 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:25:11.604035  9283 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:11.603086  9535 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.571731  9551 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:11.608006 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45723.2us
W20260709 12:25:11.609606 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 70847.3us
I20260709 12:25:11.610837 31827 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:25:11.566778  9276 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:11.610687 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:11.619531 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:11.612541 31745 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 23.
I20260709 12:25:11.622692 31822 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:25:11.624718 31818 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:25:11.623178 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:11.625325 31824 raft_consensus.cc:1240] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:11.554127  9383 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.557657  9413 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.633314  9649 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:11.613914  9551 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.631371 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:11.604579  9283 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.622014  9622 raft_consensus.cc:2764] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election decision vote started in defunct term 22: won
I20260709 12:25:11.616297  9642 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.637090  9649 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.638128  9645 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.093s
W20260709 12:25:11.639102 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 326435us
I20260709 12:25:11.612818  9202 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.635855 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:11.566591  9598 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.623795  9276 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.636693  9551 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:11.653631 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:11.595301  9398 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.649441  9649 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:11.661767 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:11.662482 32372 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:11.666507 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:11.667199 32370 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:11.666954  9656 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:11.667414 31678 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:25:11.652240  9621 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:11.671000  8903 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:11.668311 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:11.694986 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 63749.7us
I20260709 12:25:11.695642 32373 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 31 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:25:11.671598  9306 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:11.672165 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:11.672834  9413 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:11.674615  9202 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:11.675132  9283 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:11.678769  9647 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.044s
I20260709 12:25:11.655794  9643 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:11.650781  9644 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.018s
I20260709 12:25:11.691298 32372 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:25:11.647828  9594 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.702951 31752 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 24.
I20260709 12:25:11.683724 32370 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:11.708233 31669 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:25:11.689110  9656 consensus_queue.cc:1243] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 30.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
W20260709 12:25:11.714439 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 78669.7us
I20260709 12:25:11.721324 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:11.721623 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:11.651995  9642 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:11.728865 32374 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: [29.22-29.22]
I20260709 12:25:11.710449  9202 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election won for term 30
W20260709 12:25:11.733783 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 192514us
I20260709 12:25:11.712375  9360 raft_consensus.cc:2764] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:25:11.720863  9644 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:11.742098 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:11.747345 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:11.728372  9561 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:11.747155 31717 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:11.703611  9537 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.033s
W20260709 12:25:11.756827 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.759383 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:11.759728 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:11.760378 31677 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:11.764057  9383 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:11.764390 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:11.765702 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:11.767648  9639 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.163s	user 0.000s	sys 0.001s
I20260709 12:25:11.750103  9561 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:11.762495 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:11.768461  9638 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.007s
W20260709 12:25:11.836581 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74220.1us
I20260709 12:25:11.767189  9648 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:11.741081  9180 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.023s
I20260709 12:25:11.766840  9604 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:11.812546 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 83622.5us
I20260709 12:25:11.821475 31822 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 32
W20260709 12:25:11.849491 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:11.766479  9283 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:11.822906 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4086) took 1107 ms. Trace:
I20260709 12:25:11.851554 31827 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:11.822351 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59212.9us
W20260709 12:25:11.834784 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 94311.8us
I20260709 12:25:11.855257 31642 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 23 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:11.856897 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 24 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:11.845609 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:11.859436 31820 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:11.822772  9522 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:11.836505  9561 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:11.850605 31746 rpcz_store.cc:276] 0709 12:25:10.715640 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.715816 (+   176us) service_pool.cc:224] Handling call
0709 12:25:10.716157 (+   341us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:10.746209 (+ 30052us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:10.746237 (+    28us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:10.761225 (+ 14988us) log.cc:844] Serialized 26 byte log entry
0709 12:25:11.022480 (+261255us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:11.381930 (+359450us) spinlock_profiling.cc:243] Waited 2.96 ms on lock 0x7b18004e9be0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:11.821784 (+439854us) spinlock_profiling.cc:243] Waited 6.73 ms on lock 0x7b18004e9be0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf
0709 12:25:11.821999 (+   215us) log.cc:844] Serialized 14 byte log entry
0709 12:25:11.822503 (+   504us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:11.822606 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:11.822678 (+    72us) raft_consensus.cc:1685] finished
0709 12:25:11.822691 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:11.822838 (+   147us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":212,"prepare.queue_time_us":138723,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":43,"raft.queue_time_us":81934,"raft.run_cpu_time_us":227,"raft.run_wall_time_us":19095,"spinlock_wait_cycles":22275712,"thread_start_us":337060,"threads_started":2,"wal-append.queue_time_us":371266}
I20260709 12:25:11.814332  9644 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:11.821563  9654 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:25:11.868006 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 120967us
I20260709 12:25:11.841203  9648 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:11.846939  9413 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:11.855221  9535 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:11.863106  9643 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:11.866994  9656 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Leader election won for term 23
I20260709 12:25:11.865797 31791 leader_election.cc:400] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32.
I20260709 12:25:11.921527 31791 leader_election.cc:403] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:11.904583  9383 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.901087  9621 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:11.932145 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 73146.7us
I20260709 12:25:11.911053  9202 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:11.931242  9643 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:11.939863  9383 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:11.945571  9623 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
W20260709 12:25:11.947193  9659 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.110s	user 0.000s	sys 0.001s
I20260709 12:25:11.949344 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:11.861321  9664 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:11.991072 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:11.958005  9656 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:11.958266  9665 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.959388  9283 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:11.960472 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:12.000674 31827 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:11.965816 31752 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
W20260709 12:25:11.970556 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61905us
I20260709 12:25:12.001130 31752 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:11.970952 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:11.973192 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:12.013829 31749 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:25:12.017822 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:11.973860 31822 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:11.974740 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:11.917246  9413 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:11.926321  9598 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader election won for term 24
I20260709 12:25:11.983184 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:11.927914  9276 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:11.986935 31820 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:11.862107  9653 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.266s
I20260709 12:25:11.988405 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:11.992224 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 291683us
I20260709 12:25:11.992283 31751 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:11.994831 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:12.019802 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:12.023773  9413 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:12.043089 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.043715 31672 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:12.046870  9383 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:12.026647  9283 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.040109 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.048355 31670 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:25:12.048627 31821 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:25:12.049302  9283 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:12.049618 31670 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:12.050036 31824 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:25:12.039343 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:25:12.017506 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 270313us
I20260709 12:25:12.026901 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:12.116727 31819 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:25:12.055799 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61042.6us
I20260709 12:25:12.026466 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.117568 31673 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:11.987397  9447 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.116531 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:12.077039 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.048379 31669 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:12.078672 31677 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
I20260709 12:25:12.121356 31823 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
I20260709 12:25:12.121331 31672 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:11.967819  9649 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.023707 31752 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:12.088972  9652 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.166s	user 0.001s	sys 0.000s
I20260709 12:25:12.023108  9645 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 31. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32.
I20260709 12:25:11.993558  9642 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:12.075837  9665 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:25:12.078222  9659 log.cc:927] Time spent T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:25:12.093051  9180 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:12.089340  9646 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:25:12.104696  9507 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.760s	user 0.015s	sys 0.004s
W20260709 12:25:12.147287  9629 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.762s	user 0.007s	sys 0.008s
I20260709 12:25:12.165853 31671 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:12.105161  9638 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.106617  9598 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:25:12.108808 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 122616us
I20260709 12:25:12.111924  9551 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:12.113337  9673 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.063s	user 0.000s	sys 0.001s
I20260709 12:25:12.090096  9643 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 24.17, Last appended by leader: 17, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.026335  9621 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:11.921025  9655 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.268s
I20260709 12:25:12.190286 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:12.191619 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:12.117100 31674 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:12.122408 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.192744 31673 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 27 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 14.
I20260709 12:25:12.079372 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.193616 31676 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:12.125890 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:12.194391 31748 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:12.126761 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:12.191054 31744 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:12.195710 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.196194 31753 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: [26.15-26.15]
I20260709 12:25:12.131633 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:12.134327 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:12.136299 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:12.198119 32369 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 35: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:12.025508  9664 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:12.138073 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.143318 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.145712 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:12.199101 31819 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:12.199275 31677 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:12.150226 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:12.153158 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.154821 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:12.155472 31752 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:12.200279 31749 raft_consensus.cc:2393] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:12.201936 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.202143 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.029889  9654 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 35 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:12.038796  9202 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:12.162997 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 46558.8us
I20260709 12:25:12.164333  9667 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:25:12.156352  9662 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.294s	user 0.000s	sys 0.002s
I20260709 12:25:12.151479  9084 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.026777  9648 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.127733  9548 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.045s
I20260709 12:25:12.154790  9180 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:12.039391  9597 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.657s	user 0.010s	sys 0.004s
I20260709 12:25:12.197209 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.146709 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:12.204087 31818 raft_consensus.cc:2410] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 31 because replica has last-logged OpId of term: 29 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 20.
I20260709 12:25:12.207620 31752 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 28 index: 20. (index mismatch)
I20260709 12:25:12.208803 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:12.175199  9669 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.209061 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:12.213070 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.230551 31673 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 35: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:12.213532 31819 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:12.216250 31745 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 25 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:12.217907 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:12.232240  9180 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:12.233003 32373 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:12.218441 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:12.234023 31670 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:25:12.235114 31744 raft_consensus.cc:2368] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30.
I20260709 12:25:12.219753 31751 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:25:12.232034  9670 consensus_queue.cc:1059] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.101s
I20260709 12:25:12.231375 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:12.212841  9447 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.240965 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:12.242192 31818 raft_consensus.cc:1217] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Deduplicated request from leader. Original: 26.14->[28.15-31.17]   Dedup: 28.15->[29.16-31.17]
I20260709 12:25:12.236565 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:12.233027 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.238853 31819 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:25:12.244558 31672 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:12.233412  9674 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.173549  9629 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.255811 31748 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:25:12.256356 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.239292 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.243889 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:12.249329 32371 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 34: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:25:12.248544 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.259544 31676 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:25:12.243412 31818 pending_rounds.cc:85] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 15
I20260709 12:25:12.262215 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:12.265326 31819 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:12.190346  9638 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.265945 31819 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:12.234671  9586 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.240942  9611 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:12.282757  9611 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:12.283849  9611 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.187098  9656 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 19.12, Last appended by leader: 12, Current term: 23, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.284727  9664 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 25.19, Last appended by leader: 19, Current term: 27, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.285774  9654 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:12.266479 31791 connection.cc:256] RPC call timeout handler was delayed by -1.2278809680001359s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1512s for a response
I20260709 12:25:12.270817  9646 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:12.251996  9447 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.261075  9642 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.271286  9653 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:12.274227  9649 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:12.222512  9398 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.039s
I20260709 12:25:12.312870  9447 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.279070  9638 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.314792  9668 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:12.314914  9642 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.316306  9665 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:12.317250  9642 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.317615  9551 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.279894 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:12.319689 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:12.322018 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:12.280550  9605 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:12.323149  9398 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:12.323249  9653 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:12.323686  9398 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:12.323808  9653 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:12.323939  9629 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:12.265230  9674 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.324661  9398 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:12.324833  9653 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.22, Last appended by leader: 22, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.326002  9398 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:12.326329  9653 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:12.236611  9597 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.236799  9667 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:12.288805 31791 connection.cc:256] RPC call timeout handler was delayed by -1.2188809680001214s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1522s for a response
I20260709 12:25:12.260138 31818 pending_rounds.cc:107] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 30.16
W20260709 12:25:12.242442  9616 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 28 current term: 28 is higher.
I20260709 12:25:12.237746  9606 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:12.355667  9684 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.356611  9684 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.291323 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:12.293457 32376 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:12.279103 31819 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 16, Last appended: 29.18, Last appended by leader: 18, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.261960  9648 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:12.233870  9592 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.034s
I20260709 12:25:12.308807 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:12.310062 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:12.364789 31826 raft_consensus.cc:2368] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35.
I20260709 12:25:12.310824 31825 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:12.366391 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.279946 32370 raft_consensus.cc:2410] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 31 because replica has last-logged OpId of term: 29 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 20.
I20260709 12:25:12.326719  9674 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.327396 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.227290  9084 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:12.328406  9355 consensus_meta.cc:287] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.789s	user 0.013s	sys 0.000s
I20260709 12:25:12.330087  9649 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:12.240969  8903 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.051s
I20260709 12:25:12.331339  9668 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:12.245049  9669 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.330567  9665 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:12.335703 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:12.405387  9665 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:12.405809 32371 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 28: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:12.405908  9665 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:12.336208 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:12.410382  9084 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.356158  9667 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.342495 31669 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
W20260709 12:25:12.348667 31791 connection.cc:256] RPC call timeout handler was delayed by -1.1342809679999846s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16160000000000002s for a response
I20260709 12:25:12.348804 31818 raft_consensus.cc:2889] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:12.347868  9561 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.369675 31819 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:12.370340 32377 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
W20260709 12:25:12.379375 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 70643.6us
I20260709 12:25:12.383715  9447 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.394218  9597 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:12.426589 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:12.356033  9606 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.265219  9598 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 24, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.403478  9684 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:12.415860 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0946809679999205s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.166s for a response
W20260709 12:25:12.442138 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0775809679998929s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16790000000000002s for a response
W20260709 12:25:12.442523 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0739809679998871s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1683s for a response
W20260709 12:25:12.442924 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0739809679998871s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1683s for a response
I20260709 12:25:12.417176  9674 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:12.434368 31826 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 27 index: 14. (index mismatch)
W20260709 12:25:12.437796 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 101682us
I20260709 12:25:12.390100  9180 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.451494 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:12.455051 31718 leader_election.cc:400] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35.
I20260709 12:25:12.455499 31718 leader_election.cc:403] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Cancelling election due to peer responding with higher term
W20260709 12:25:12.417793 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 129971us
I20260709 12:25:12.421592  9551 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:12.462826 31791 leader_election.cc:400] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 27. Current term is 28.
I20260709 12:25:12.439317  9398 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:12.444069  9518 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:12.471201 31823 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:12.473654 31669 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30. Ops: [29.17-29.17]
W20260709 12:25:12.474362 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.474388 31674 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:12.382668  9355 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.390188  9413 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.455106  9684 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.478603  9561 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:12.478981  9684 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.480302 31826 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 30 index: 15. Preceding OpId from leader: term: 32 index: 16. (index mismatch)
I20260709 12:25:12.481176  9684 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:12.482223 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.483016 31826 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:12.438248  9447 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.485042 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.485759 31826 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:25:12.487323 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:12.488332 31826 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:12.488263 31718 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:12.489286  8803 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:12.489427  9383 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:12.489899  9383 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.490265  9383 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:12.492753  9507 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.018s
W20260709 12:25:12.495723 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.440164  9084 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.029s
I20260709 12:25:12.462348  9674 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:12.468544  9676 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:12.510399 32375 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:12.469484  9398 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:12.510461 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47809.7us
I20260709 12:25:12.516677  9656 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.496955  9447 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.518118 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:12.458575  9664 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.055s
I20260709 12:25:12.504575  9605 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:12.533090 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 100615us
I20260709 12:25:12.513895  9398 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:12.545369 31826 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:12.550604 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.551358 31826 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
W20260709 12:25:12.559001 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 46780.7us
I20260709 12:25:12.559314 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:12.560235 31827 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:12.561272 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.561893 31827 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
W20260709 12:25:12.563336  9693 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:25:12.525467  9629 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.015s
I20260709 12:25:12.568459  9674 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.569245 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:12.569962 31673 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:12.574788 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:12.575475 32377 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:12.575867 32378 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 35 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35. Ops: [34.20-34.20]
I20260709 12:25:12.531821  9500 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.015s
I20260709 12:25:12.579806  9500 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.580384  9500 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.581707  9665 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.539053  9690 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:12.583034  9690 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:12.583333  9180 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:12.583468  9690 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:12.560055 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:12.529729  9656 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.566264 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.606072 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:12.611424 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:12.618440 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.606441 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:12.620165 31675 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:12.621062 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:12.639256 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:12.640286 32378 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:12.642216 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:12.644348  9084 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:12.644896  9084 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.645349  9084 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:25:12.647382 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.661022 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:12.661754 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:12.662366 31673 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:25:12.664590 32377 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
W20260709 12:25:12.668236 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 108272us
I20260709 12:25:12.669450 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:12.579792  9606 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:12.673218 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:12.675969 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:12.678998 31673 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:25:12.622102  9453 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:12.637207  9180 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:12.666723  9592 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:12.665973  9669 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:12.684715  9640 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:25:12.667919  9586 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:12.671943  9597 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:12.731731  9697 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:25:12.741482  9665 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.686334  9453 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.733446  9586 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:12.742931 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.743148  9586 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:12.743453 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.744269 31822 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:12.745486 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.746376 31822 raft_consensus.cc:2410] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 34 because replica has last-logged OpId of term: 32 index: 16, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 15.
W20260709 12:25:12.748354  9692 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 12:25:12.749327 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.750074 31822 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
W20260709 12:25:12.750653 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 81310.5us
I20260709 12:25:12.751526 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:12.752470  9648 raft_consensus.cc:2764] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election decision vote started in defunct term 28: won
I20260709 12:25:12.753228  9398 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.754776  9679 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:12.755364  9679 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:12.756520  9679 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.758179  9679 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:12.760684  9398 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.732080  9638 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:12.690703 31643 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:12.739794 31825 raft_consensus.cc:2410] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 34 because replica has last-logged OpId of term: 33 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 21.
I20260709 12:25:12.753602 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:12.767688  9648 raft_consensus.cc:2764] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:25:12.768988 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:12.770439  9648 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:12.770730  9606 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:12.771138 31825 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35. Ops: [34.20-34.20]
I20260709 12:25:12.680496  9668 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.092s
I20260709 12:25:12.779475 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:12.693224 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:12.736949 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47965.1us
I20260709 12:25:12.774386 31825 raft_consensus.cc:1275] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:25:12.743053  9453 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:12.785697 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:12.783761 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.787732  9648 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.790947  9648 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.795475  9654 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:12.795816  9648 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:12.796231  9654 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.798892  9648 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:12.799484  9648 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:12.799945  9648 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:12.800386 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 109864us
I20260709 12:25:12.802098  9283 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:12.802953 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:12.803756  9276 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.790513 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:12.804247  9276 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.805030  9660 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:12.805506  9660 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:12.805830  9660 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:12.807240  9678 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:12.807471  9276 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.807909  9594 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:12.808581  9629 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:12.809156 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:12.809919 31827 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:12.804262 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:12.809160  9629 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:12.823820 31675 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
I20260709 12:25:12.821326 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:12.825519 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1605) took 1156 ms. Trace:
I20260709 12:25:12.828400 31678 rpcz_store.cc:276] 0709 12:25:11.668717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.668892 (+   175us) service_pool.cc:224] Handling call
0709 12:25:11.702012 (+ 33120us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:11.702396 (+   384us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:11.702425 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:11.714091 (+ 11666us) log.cc:844] Serialized 26 byte log entry
0709 12:25:11.867663 (+153572us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:11.920080 (+ 52417us) log.cc:844] Serialized 14 byte log entry
0709 12:25:12.559726 (+639646us) spinlock_profiling.cc:243] Waited 2.07 ms on lock 0x7b18002f42e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:25:12.812066 (+252340us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.812143 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:12.812187 (+    44us) raft_consensus.cc:1685] finished
0709 12:25:12.825185 (+ 12998us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:12.825427 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7735,"prepare.queue_time_us":51301,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":49,"raft.queue_time_us":109604,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":13199,"spinlock_wait_cycles":4768512,"thread_start_us":377227,"threads_started":2,"wal-append.queue_time_us":208925}
I20260709 12:25:12.807796  9678 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:12.834072 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:12.736657  9592 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:12.823323  9518 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:12.836287  9283 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:12.851876 31674 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:12.843552  9593 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:12.845645 32377 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:12.789789  9669 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.849831 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:12.821070  9654 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.856405 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:12.862133 31820 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:12.784956  9597 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.875377 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:12.871213 31827 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 36.
I20260709 12:25:12.848592  9535 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:12.886723 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:12.910116 31820 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
W20260709 12:25:12.912312 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 62628.2us
I20260709 12:25:12.913154 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:12.944512 31827 raft_consensus.cc:1240] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: [28.18-28.18]
I20260709 12:25:12.948220 31827 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:12.945767 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:12.858022  9355 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.953482  8803 consensus_queue.cc:1059] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 26.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:12.954901  8803 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:12.956163  8803 raft_consensus.cc:3055] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:12.956692  8803 raft_consensus.cc:740] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:12.958879  9447 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:12.959596  9447 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:12.960969  9447 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.880312  9180 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:12.957974  8803 consensus_queue.cc:260] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.974665  9084 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.912788  9678 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:12.978060  9518 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:12.980907 31823 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:25:12.888101  9535 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:12.969481  9662 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.080s	user 0.000s	sys 0.000s
I20260709 12:25:12.965152  9447 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:12.873936  9669 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:12.883086  9593 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 33 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:25:13.045066 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:12.966238  9561 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:12.978057  9690 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:13.015733  9383 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.525s	user 0.010s	sys 0.001s
W20260709 12:25:13.026513  9652 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.085s	user 0.001s	sys 0.000s
I20260709 12:25:13.073057  9383 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:13.030427  9535 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:12.903288  9283 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 27.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:13.064364  8803 raft_consensus.cc:3060] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:13.120078 31822 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 31. Current term is 32. Ops: []
W20260709 12:25:13.118029  9687 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:25:13.129045  9691 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:13.162791  9484 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
I20260709 12:25:13.136569  9669 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:13.137560  9383 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:13.137297  9660 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:13.163659  9484 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:13.143569  9586 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:13.152431  9413 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:13.153839 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 3060) took 1032 ms. Trace:
W20260709 12:25:13.154291  9676 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.640s	user 0.006s	sys 0.003s
I20260709 12:25:13.155448  9597 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:12.970410  9629 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:13.158645  9690 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:13.160820 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:13.030969  9518 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:13.161796 32343 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:13.165078 31669 rpcz_store.cc:276] 0709 12:25:12.121208 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.422952 (+301744us) service_pool.cc:224] Handling call
0709 12:25:12.424494 (+  1542us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.475374 (+ 50880us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.153579 (+678205us) spinlock_profiling.cc:243] Waited 48.9 ms on lock 0x7b54001b35d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:13.153779 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":112376704}
I20260709 12:25:13.115975  9678 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:13.158496 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:13.170498 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 224801us
I20260709 12:25:13.171412 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:13.163548  9453 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:13.164367  9484 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:25:13.175418  9712 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.115s	user 0.000s	sys 0.000s
I20260709 12:25:13.178725  9535 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:13.179564 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.179603 31825 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:13.060365  9592 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:13.163620  9717 consensus_queue.cc:1048] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.036s
I20260709 12:25:13.200639  9084 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:13.173406  9594 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:13.173638  9536 raft_consensus.cc:2804] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:13.174233  9676 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election lost for term 34. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35.
W20260709 12:25:13.212380 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.205819 31791 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:13.203780  9690 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:13.245734 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:13.210325  9691 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.031s
I20260709 12:25:13.221868  9413 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.028s
I20260709 12:25:13.238142  9536 raft_consensus.cc:697] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:13.241622  9283 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:13.231580  9594 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:13.261577 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 65086us
I20260709 12:25:13.243750  9718 raft_consensus.cc:2764] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election decision vote started in defunct term 30: won
I20260709 12:25:13.271741 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:13.272502 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:13.335107 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:13.260788  9721 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:13.337380 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:13.337651 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 1597) took 1200 ms. Trace:
I20260709 12:25:13.342185 32370 rpcz_store.cc:276] 0709 12:25:12.131073 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.131256 (+   183us) service_pool.cc:224] Handling call
0709 12:25:13.274957 (+1143701us) spinlock_profiling.cc:243] Waited 82 ms on lock 0x7b54004c4354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:13.331575 (+ 56618us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":188581248}
I20260709 12:25:13.256721  9655 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.074s
I20260709 12:25:13.235718  9593 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.23, Last appended by leader: 23, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:13.375423 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:13.343631  9707 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.315s	user 0.001s	sys 0.000s
I20260709 12:25:13.254122  9484 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:13.402277  9716 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.069s	user 0.001s	sys 0.000s
I20260709 12:25:13.341746  9721 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:13.275056  9283 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:13.376461  9691 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.400187  9597 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:13.433876  9484 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:13.439647  9701 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:13.440362  9701 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:13.426959  9721 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:13.447759  9723 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
W20260709 12:25:13.449481  9722 log.cc:927] Time spent T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
W20260709 12:25:13.450134  9710 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.295s	user 0.000s	sys 0.001s
I20260709 12:25:13.432322  9691 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:13.452200 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4116) took 1304 ms. Trace:
I20260709 12:25:13.453398  9691 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:13.452579 31818 rpcz_store.cc:276] 0709 12:25:12.147881 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.241471 (+ 93590us) service_pool.cc:224] Handling call
0709 12:25:12.241925 (+   454us) raft_consensus.cc:1471] Updating replica for 3 ops
0709 12:25:12.540250 (+298325us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:12.673977 (+133727us) log.cc:844] Serialized 14 byte log entry
0709 12:25:12.850558 (+176581us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:12.853457 (+  2899us) log.cc:844] Serialized 52 byte log entry
0709 12:25:12.853626 (+   169us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:12.853669 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:12.853750 (+    81us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:13.451934 (+598184us) raft_consensus.cc:1685] finished
0709 12:25:13.451955 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:13.452113 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3103,"prepare.queue_time_us":255851,"prepare.run_cpu_time_us":29,"prepare.run_wall_time_us":29,"raft.queue_time_us":54279,"raft.run_cpu_time_us":212,"raft.run_wall_time_us":2430,"thread_start_us":341532,"threads_started":3,"wal-append.queue_time_us":338171}
I20260709 12:25:13.454974  9518 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:13.422353  9708 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.293s	user 0.000s	sys 0.001s
I20260709 12:25:13.461771  9701 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:13.462534  9717 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.465459 31827 pending_rounds.cc:85] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 17
I20260709 12:25:13.508147 31827 pending_rounds.cc:107] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 29.18
I20260709 12:25:13.508543 31827 raft_consensus.cc:2889] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:13.468909  9594 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:13.470767  9721 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:13.472998  9656 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:13.455621  9536 consensus_queue.cc:237] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.17, Last appended by leader: 17, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:13.480257  9648 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.679s	user 0.013s	sys 0.000s
I20260709 12:25:13.492214 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:13.534250 31818 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:13.495020 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1809) took 1182 ms. Trace:
I20260709 12:25:13.454919  9726 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.185s
I20260709 12:25:13.535104 31822 rpcz_store.cc:276] 0709 12:25:12.312601 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.751230 (+438629us) service_pool.cc:224] Handling call
0709 12:25:12.751579 (+   349us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.122198 (+370619us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.494676 (+372478us) spinlock_profiling.cc:243] Waited 368 ms on lock 0x7b54001987d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:13.494935 (+   259us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":846863104}
I20260709 12:25:13.498255 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:13.499115 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.549208 32370 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
W20260709 12:25:13.524619 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:13.507341  9679 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.722s	user 0.004s	sys 0.007s
W20260709 12:25:13.515579  9714 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.084s	user 0.000s	sys 0.000s
I20260709 12:25:13.551371 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:13.581857 31818 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
W20260709 12:25:13.570420 32342 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 2.651s (SENT) [suppressed 14 similar messages]
I20260709 12:25:13.584571 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.585238 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.585278 31748 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:13.591576 31678 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:25:13.595589 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:13.516177  9717 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:13.612830 31748 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:13.573843  9648 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:13.618427 31748 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 28 index: 14. Preceding OpId from leader: term: 32 index: 16. (index mismatch)
I20260709 12:25:13.591236  9734 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:13.597280  9728 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.028s
I20260709 12:25:13.642071  9413 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:13.643292 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1469) took 1981 ms. Trace:
I20260709 12:25:13.643340 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:13.644141 31748 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:25:13.642489  9413 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:13.646921 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1588) took 2236 ms. Trace:
I20260709 12:25:13.647336 32375 rpcz_store.cc:276] 0709 12:25:11.406651 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.410167 (+  3516us) service_pool.cc:224] Handling call
0709 12:25:11.428518 (+ 18351us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.513766 (+1085248us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.642598 (+1128832us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b540053a554. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:13.642806 (+   208us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2487457664}
I20260709 12:25:13.647863  9413 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:13.643795 32376 rpcz_store.cc:276] 0709 12:25:11.662238 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:11.662419 (+   181us) service_pool.cc:224] Handling call
0709 12:25:11.667997 (+  5578us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.361531 (+693534us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.643080 (+1281549us) spinlock_profiling.cc:243] Waited 625 ms on lock 0x7b54004272d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:13.643235 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1437793536}
I20260709 12:25:13.649890  9413 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:13.651422  9717 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:13.624235  9734 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:13.652652  9734 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:13.654017  9648 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:13.658708  9727 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.257s	user 0.000s	sys 0.003s
I20260709 12:25:13.659305  9721 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:13.662773 31747 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
I20260709 12:25:13.664372  9500 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:13.668020  9621 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:13.668470  9621 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:13.669296  9621 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:13.670348  9621 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:13.671587  9701 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:13.677263  9660 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:13.682374 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 3061) took 1560 ms. Trace:
I20260709 12:25:13.697528 31674 rpcz_store.cc:276] 0709 12:25:12.121493 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.499693 (+378200us) service_pool.cc:224] Handling call
0709 12:25:12.500112 (+   419us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.852339 (+352227us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.682075 (+829736us) spinlock_profiling.cc:243] Waited 352 ms on lock 0x7b540017d9d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:13.682296 (+   221us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":808528128}
I20260709 12:25:13.695744 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.704767 31748 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:25:13.694993  9701 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:25:13.707604 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.708114 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:13.710491 31747 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 35: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:13.713336  9180 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:13.717653  9180 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:13.718717  9180 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:13.719914  9180 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:13.754419 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.758140 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4113) took 1753 ms. Trace:
I20260709 12:25:13.758597 31671 rpcz_store.cc:276] 0709 12:25:12.005185 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.018123 (+ 12938us) service_pool.cc:224] Handling call
0709 12:25:12.042354 (+ 24231us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:12.171707 (+129353us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:13.757895 (+1586188us) spinlock_profiling.cc:243] Waited 123 ms on lock 0x7b54000465d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:13.758077 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":283438848}
W20260709 12:25:13.781055 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.811308 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.812633 31747 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 30 because replica has last-logged OpId of term: 29 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 21.
I20260709 12:25:13.817240 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 26 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.818933 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.819780 31753 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:13.761191  9606 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.821017  9606 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:13.823091 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.823787 31744 raft_consensus.cc:2376] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34. Re-sending same reply.
I20260709 12:25:13.824878 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.825577 31751 raft_consensus.cc:2376] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35. Re-sending same reply.
I20260709 12:25:13.826683 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:13.828281 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.829007 31752 raft_consensus.cc:2376] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32. Re-sending same reply.
I20260709 12:25:13.830179 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:13.830955  9606 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:13.859710 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:13.860669 32370 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:13.866058 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:13.866940 31747 raft_consensus.cc:2376] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35. Re-sending same reply.
I20260709 12:25:13.881162  9678 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:13.881801  9678 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:13.884074  9678 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:13.885246  9413 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:13.885043  9500 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:13.885881  9413 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:13.888216  9413 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:13.760643  9645 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:13.863067 31792 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:13.901218 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:13.903731 31753 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
W20260709 12:25:13.904731 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 150229us
I20260709 12:25:13.907236 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:13.754539  9701 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:13.891255  9594 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.031s
I20260709 12:25:13.896903  9645 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:13.902891  9606 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:13.909489  9500 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:13.918362 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:13.919862 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:13.920856 31753 raft_consensus.cc:2368] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 34. Current term is 35.
I20260709 12:25:13.921864  9616 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:13.923324  9616 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:13.912428  9717 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:13.905396  9413 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.014s
I20260709 12:25:13.927742 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.928571 31753 raft_consensus.cc:2368] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 35. Current term is 36.
I20260709 12:25:13.934048 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:13.935003 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:13.936674 32370 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:25:13.920406  9606 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:13.924834  9611 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:13.942327 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:13.943409 31753 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:13.944933 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:13.945559 31822 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 35
I20260709 12:25:13.946157 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:13.946887 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:13.947389 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:13.947613 31824 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:13.947968 32371 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:13.949884 32376 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:25:13.929363  9701 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:13.951038 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:13.951108 31826 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:13.958416 31822 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
W20260709 12:25:13.959439 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.960970  9645 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:13.961603 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:13.962596  9283 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:13.964339  9535 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:13.941272  9606 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:13.967206  9734 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:13.969587  9734 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:13.973562 31747 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30. Ops: [29.18-29.18]
I20260709 12:25:13.973940 31750 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 27 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:25:13.975066 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:13.976533 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.977205 31752 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:13.979048 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:13.980171 31745 raft_consensus.cc:2410] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 35 because replica has last-logged OpId of term: 34 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 19.
I20260709 12:25:13.981187 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 30 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:13.981878 31745 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:25:13.983006 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.983757 31751 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:13.987454 31744 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30. Ops: [29.17-29.17]
I20260709 12:25:13.988847 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:13.989511 31744 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:13.991011 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:13.991276 31748 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:13.991897 31750 raft_consensus.cc:2410] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 31 because replica has last-logged OpId of term: 30 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
I20260709 12:25:13.992587 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:13.993346 31748 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 35.
W20260709 12:25:13.995126 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 75533.7us
I20260709 12:25:13.995762 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.000164 31824 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:14.000813 31749 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 33. Current term is 34. Ops: [33.22-33.22]
W20260709 12:25:14.005174 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.005963 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.013258 31752 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:25:14.015704 31751 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:14.017282 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:14.018036 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:14.025038 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.032996 31745 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 27.
I20260709 12:25:14.026340 31751 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:14.037890 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:14.039040 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:14.040669 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.042133  9734 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:25:14.043037  9734 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:14.043130  9535 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:14.043426  9734 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:14.043836  9535 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.044302  9734 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.045187 31744 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:14.045576  9734 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:14.046186 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.047444 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:14.047767 31745 raft_consensus.cc:2368] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 35. Current term is 36.
I20260709 12:25:14.029804 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:14.028354 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 110069us
I20260709 12:25:14.030611 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.048424 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:14.031320 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:14.048996 31749 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:14.049234 31747 raft_consensus.cc:2376] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29. Re-sending same reply.
I20260709 12:25:14.032012 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.049639 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:14.048183 31677 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 26. Current term is 27. Ops: []
I20260709 12:25:14.050184 31748 raft_consensus.cc:2368] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 34. Current term is 35.
I20260709 12:25:14.053767 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:14.055264 32371 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
W20260709 12:25:14.056682 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.056757  9675 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:14.057425 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.057397  9675 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.058708  9645 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:14.059633 31671 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 32. Ops: []
I20260709 12:25:14.060253 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:14.060519  9675 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:14.071822 31674 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 34: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:14.059830 31746 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 35 index: 20. (term mismatch)
I20260709 12:25:14.072427 31746 pending_rounds.cc:85] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 19
I20260709 12:25:14.072701 31746 pending_rounds.cc:107] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 34.20
I20260709 12:25:14.072968 31746 raft_consensus.cc:2889] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:14.069900 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:14.073915 31678 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:14.079005  9745 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.079277  9535 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:14.079717  9745 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:14.080329  9742 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
I20260709 12:25:14.082222  9745 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:14.082098  9592 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:14.070487 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:14.086342 31676 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:25:14.087270 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:14.060390 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:14.084635 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.088893 31677 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 35.
I20260709 12:25:14.089179 31678 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:25:14.090587 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:14.091225 31791 leader_election.cc:400] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30.
I20260709 12:25:14.091572 31791 leader_election.cc:403] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:14.052472  9520 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:14.092568  9180 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:14.085427  9645 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:14.093986  9520 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.100747  9520 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:14.095773 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:14.085423  9611 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.113359 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:14.094740  9745 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:14.121421 31752 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:14.095950  9726 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30.
I20260709 12:25:14.124722 31750 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:14.124619 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.126013 31752 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:14.127148 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.127393 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:14.128357 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.128916 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:14.130926 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:14.131317 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:14.131902 31751 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:14.133436  9648 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:14.134975 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.135380 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.115072  9645 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:14.135550 31753 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:25:14.136162 31750 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:25:14.137621 31749 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:25:14.139945 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.141119 31748 raft_consensus.cc:2410] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 30 because replica has last-logged OpId of term: 29 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 21.
I20260709 12:25:14.142851 31744 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 34: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:14.115425  9747 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:14.175917 31750 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:14.156665 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:14.158452 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:14.124403  9180 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:14.111229  9561 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.021s
I20260709 12:25:14.195156 31752 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 29
I20260709 12:25:14.211189 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.192119 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:14.207985 31749 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:25:14.212913 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:14.213493 31749 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:14.216257 31751 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:14.217502 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.217914 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.219952 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:14.134769  9721 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election won for term 36
W20260709 12:25:14.222205 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 172666us
I20260709 12:25:14.223279 31717 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:14.226354 31753 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 16. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:14.155678  9616 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.228534 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:14.231626 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:14.232774 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.118101  9611 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.208711  9748 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.237334 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:14.209255  9180 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:14.210547  9726 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:14.222314  9718 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:14.240594 31752 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:14.225066  9721 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.225600  9753 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:14.234892  9745 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:14.248215  9721 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:14.250783  9745 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.240443  9355 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:14.243222  9748 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.261207  9748 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:14.262153  9748 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:14.262694  9748 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:14.263615  9748 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.268512  9606 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.242506  9180 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:14.273121  9721 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.275434  9734 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:14.276122  9734 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.247684 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.278340 31752 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:14.279971  9520 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.280639  9611 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:14.281236  9611 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.281790  9611 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:14.283614 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.280652  9520 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.287719  9753 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.275580  9751 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:14.290458 31748 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:25:14.290784 31749 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:14.209456  9747 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.247704  9717 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:14.244048  9718 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:14.305558 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.239094  9616 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.306849 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.294433  9520 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:14.337759  9592 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:14.344441  9586 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:14.324204  9701 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.360206 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.300215  9718 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:14.376343 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:14.377647 31678 raft_consensus.cc:2410] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 29 because replica has last-logged OpId of term: 28 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 26 index: 17.
I20260709 12:25:14.376474 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:14.379050 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:14.379642 31670 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:14.324970  9717 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:14.295001  9751 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.387037 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:14.388067 31673 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:25:14.307209  9611 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:14.354727  9753 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:14.395054 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:14.401320 31749 raft_consensus.cc:2376] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29. Re-sending same reply.
I20260709 12:25:14.402714 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.393347 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:14.399392 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:14.404633 31672 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:25:14.358152  9413 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:14.410077 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.378264  9606 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:14.413113  9520 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.413977 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:14.421027 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:14.358410  9592 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.438104 31670 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:25:14.448655 31744 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:14.450969 31744 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:14.449780 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:14.381471  9701 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.453814 31744 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 32.23, Last appended by leader: 23, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.409498  9726 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 32 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:14.464689 31744 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:14.428021  9745 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:14.450030  9675 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:14.462411  9548 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.021s
I20260709 12:25:14.462546  9606 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.484725  9611 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:14.459697  9756 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:14.485837  9606 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.459224  9660 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.031s
I20260709 12:25:14.488288  9606 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:14.466363  9679 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:14.418784  9413 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:14.511243 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.447126  9520 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.518728  9763 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:14.519381  9763 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.521701  9763 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:14.488497  9717 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.497905  9606 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:14.523285 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1466) took 1012 ms. Trace:
I20260709 12:25:14.523190  9606 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.523717 31746 rpcz_store.cc:276] 0709 12:25:13.510598 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.510961 (+   363us) service_pool.cc:224] Handling call
0709 12:25:13.511301 (+   340us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.073529 (+562228us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.523051 (+449522us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x7b5400000d54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:14.523222 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1260701568}
I20260709 12:25:14.507588  9679 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:14.525199  9679 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:14.525290 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.526077 31746 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:14.525573  9726 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.526890  9757 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:14.528452  9701 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:14.529263  9747 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:14.528473  9413 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:14.535893 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 123051us
I20260709 12:25:14.477339  9180 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.539072 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:14.525610  9606 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:14.515220  9750 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.549335 31744 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 33 index: 24. (index mismatch)
I20260709 12:25:14.552466 31746 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:25:14.536479  9745 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:14.564935  9747 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:14.606107 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 146891us
I20260709 12:25:14.606371 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:14.607816 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.610328 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:14.607371 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:14.571625  9745 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:14.542783  9586 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.060s
I20260709 12:25:14.538385  9757 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.567255  9537 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.043s
I20260709 12:25:14.626689  9675 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.078s
I20260709 12:25:14.705365  9675 consensus_queue.cc:1243] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 31.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:25:14.625264  9656 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:14.563401  9413 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.042s
I20260709 12:25:14.635397 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:14.634349 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:14.712565 31818 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:14.713099 31824 raft_consensus.cc:2410] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 36 because replica has last-logged OpId of term: 33 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:25:14.713876 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:14.670648 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:14.715081 31825 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:14.716302 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:14.717042 31825 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:14.671550 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:14.672565 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.674783 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:14.569067  9750 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.718654 31823 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:14.679733  9520 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:14.679462  9679 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.639982  9453 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:14.682922 32374 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:25:14.683648 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:14.684451 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:14.605978  9764 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.020s
I20260709 12:25:14.685516  9757 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:14.687119 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:14.697369  9769 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.124s	user 0.000s	sys 0.000s
I20260709 12:25:14.698486  9745 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:14.612946  9747 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.700403  9751 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:14.702118  9180 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:14.702940  9592 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:14.604911  9770 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.738219 31818 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:14.714381 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:14.609455  9749 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.242s	user 0.001s	sys 0.000s
I20260709 12:25:14.765563 31825 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:25:14.770712 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:14.828091 31826 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:14.774706 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 149520us
W20260709 12:25:14.786137 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.801126 31827 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:14.818171 31823 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:14.615748 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:14.825040 31824 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:14.832118 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:14.840157 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.815429  9775 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.855283  9180 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:14.830509  9745 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:14.863688  9679 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:14.867254  9750 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:14.820340  9675 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.052s
I20260709 12:25:14.874310 31826 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:14.822062  9772 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:14.879022 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 47025.3us
W20260709 12:25:14.870273 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 234793us
I20260709 12:25:14.888454 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:14.829442  9678 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.052s
I20260709 12:25:14.871567  9520 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:14.828263  9770 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:14.912811 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:14.939904 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:14.940987 32378 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:14.892624  9621 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:14.876401  9180 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.893640  9656 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:14.946452 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:14.935520  9749 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:25:14.950441 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:14.959260 32371 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:25:14.874396  9775 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:14.929664  9638 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:14.971318  9638 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:14.987753  9780 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.102s	user 0.002s	sys 0.000s
I20260709 12:25:14.960050  9772 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:14.960822  9621 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 35 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:15.005947 31677 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:15.007746 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:15.008477 31677 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:15.009485 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.010895 31670 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 33. Current term is 34. Ops: []
W20260709 12:25:15.013070 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.015848 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:15.016340 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 400681us
I20260709 12:25:14.997287  9355 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.018005 31669 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:15.018034 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.000083  9772 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.022817 31678 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
W20260709 12:25:15.013535  9718 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.612s	user 0.011s	sys 0.003s
I20260709 12:25:14.999006  9638 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:15.007678  9592 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.020313 31672 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:15.024191 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:15.020287 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:15.020781 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 188786us
I20260709 12:25:15.029047 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3963) took 2445 ms. Trace:
I20260709 12:25:15.016687  9747 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:15.035565 31643 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:15.026407 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.038707  9679 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:15.039425  9679 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:15.040647  9679 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:15.046490 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.051483 31669 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:15.033565 31820 rpcz_store.cc:276] 0709 12:25:12.583681 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:12.952084 (+368403us) service_pool.cc:224] Handling call
0709 12:25:12.952374 (+   290us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:12.963334 (+ 10960us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:12.968874 (+  5540us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.024190 (+1055316us) spinlock_profiling.cc:243] Waited 5.84 ms on lock 0x7b180042b440. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:25:14.385160 (+360970us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:14.394466 (+  9306us) log.cc:844] Serialized 26 byte log entry
0709 12:25:14.394727 (+   261us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:14.394782 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.394880 (+    98us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:15.018702 (+623822us) raft_consensus.cc:1685] finished
0709 12:25:15.028810 (+ 10108us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.028963 (+   153us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9467,"prepare.queue_time_us":590,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":113616,"raft.run_cpu_time_us":260,"raft.run_wall_time_us":36854,"spinlock_wait_cycles":13438336,"thread_start_us":369753,"threads_started":3,"wal-append.queue_time_us":1263124}
I20260709 12:25:15.059515 31677 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 36.
I20260709 12:25:15.026072  9772 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:15.071197 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.024721  9675 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.037s
I20260709 12:25:15.033919  9586 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.094542 31677 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 19. Preceding OpId from leader: term: 27 index: 20. (index mismatch)
I20260709 12:25:15.096977 31678 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33. Ops: []
W20260709 12:25:15.062417  9765 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.005s	sys 0.019s
I20260709 12:25:15.024502  9355 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.099897 31672 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:25:15.098307 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:15.030105  9718 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.106560 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:15.096132  9756 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.036s	sys 0.050s
I20260709 12:25:15.034339  9592 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:15.101197  9765 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.617s	user 0.005s	sys 0.020s
I20260709 12:25:15.096879  9690 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.098526  9520 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:25:15.129261  9785 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.097s	user 0.000s	sys 0.000s
I20260709 12:25:15.114219 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:15.135535 31824 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: []
W20260709 12:25:15.139102  9787 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.069s	user 0.000s	sys 0.001s
I20260709 12:25:15.140405 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.141426 31823 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:15.142400 31827 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:25:15.142915 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:15.112056  9756 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.625s	user 0.036s	sys 0.051s
I20260709 12:25:15.143169 31820 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:25:15.140416 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:15.146648 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.147523 31820 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:15.149402 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 19 index: 12 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.150214 31824 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 23.
I20260709 12:25:15.153317 31821 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:15.146325 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.156214 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.157091 31826 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:15.158363 31820 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 30 index: 17. (term mismatch)
I20260709 12:25:15.158887 31820 pending_rounds.cc:85] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 16
W20260709 12:25:15.158908 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 98697.8us
I20260709 12:25:15.159240 31820 pending_rounds.cc:107] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 29.17
I20260709 12:25:15.159657 31820 raft_consensus.cc:2889] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:15.160705 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.161378 31677 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:15.090040  9621 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.104028  9586 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:15.167685 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 147464us
I20260709 12:25:15.168769 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.130929  9726 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:15.172335 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.172350 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:15.148409  9561 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:15.145036  9756 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:15.182368 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1604) took 1826 ms. Trace:
I20260709 12:25:15.211125  9726 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.165107  9791 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:15.209828 31820 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:25:15.212286  9791 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:15.214969 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:15.212875 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 72550.3us
I20260709 12:25:15.220669 31820 raft_consensus.cc:2410] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 31 because replica has last-logged OpId of term: 30 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 17.
I20260709 12:25:15.220697 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:15.224690 31824 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:15.131971  9690 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.228327 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:15.232188 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:15.210589 32373 rpcz_store.cc:276] 0709 12:25:13.355787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:13.355991 (+   204us) service_pool.cc:224] Handling call
0709 12:25:13.375912 (+ 19921us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:13.376140 (+   228us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:25:13.417262 (+ 41122us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.825893 (+1408631us) spinlock_profiling.cc:243] Waited 4.49 ms on lock 0x7b18004310e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:25:15.031096 (+205203us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.174438 (+143342us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:15.174518 (+    80us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:25:15.174579 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.174688 (+   109us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.182280 (+  7592us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":37871,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":95,"raft.queue_time_us":40976,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":111,"spinlock_wait_cycles":10322176,"thread_start_us":345549,"threads_started":2}
I20260709 12:25:15.212689  9791 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:15.243386 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 29 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:15.253610 31820 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:25:15.247475 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:15.254659 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:15.256460 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:15.175442  9784 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.077s	user 0.001s	sys 0.000s
I20260709 12:25:15.241571 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:15.177867  9775 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:15.259137 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.262764 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:15.261065 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:15.177644  9770 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:15.209722  9679 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:15.174968  9537 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:15.263374 31748 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:15.209554  9180 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:15.266517 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:15.273133 31821 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 35. Current term is 36. Ops: [35.23-35.23]
I20260709 12:25:15.213588  9747 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.268136 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:15.241832  9535 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:15.240372  9777 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.286530  9747 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.288671 31678 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 32 index: 21. (index mismatch)
I20260709 12:25:15.292898 31820 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
W20260709 12:25:15.293265  9782 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.182s	user 0.000s	sys 0.001s
I20260709 12:25:15.317277 31717 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:15.321758 31748 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 35.
W20260709 12:25:15.287757 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46252.1us
I20260709 12:25:15.227311  9561 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.246097  9520 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:15.290154  9777 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.281337  9180 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:15.296507  9756 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:15.298082  9788 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
I20260709 12:25:15.301735 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.348743 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.302881 31670 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:15.349472 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:15.303110 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4149) took 1004 ms. Trace:
I20260709 12:25:15.350265 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.303362  9770 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 14, Last known committed idx: 12, Time since last communication: 0.000s
W20260709 12:25:15.303721  9797 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.136s	user 0.001s	sys 0.000s
I20260709 12:25:15.294001  9668 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:15.280555  9535 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:15.305258  9690 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:15.307185  9592 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:15.350226 31753 rpcz_store.cc:276] 0709 12:25:14.299136 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.299279 (+   143us) service_pool.cc:224] Handling call
0709 12:25:14.299715 (+   436us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.408339 (+108624us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.408370 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:14.988595 (+580225us) spinlock_profiling.cc:243] Waited 5.99 ms on lock 0x7b2c00001158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:25:14.989024 (+   429us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.110784 (+121760us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:15.120880 (+ 10096us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.121185 (+   305us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.121292 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:15.302764 (+181472us) raft_consensus.cc:1685] finished
0709 12:25:15.302887 (+   123us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.303042 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23852,"prepare.queue_time_us":12479,"prepare.run_cpu_time_us":41,"prepare.run_wall_time_us":42,"raft.queue_time_us":105350,"raft.run_cpu_time_us":177,"raft.run_wall_time_us":7088,"spinlock_wait_cycles":29651072,"thread_start_us":222229,"threads_started":2,"wal-append.queue_time_us":220567}
I20260709 12:25:15.311731 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.311872  9726 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:15.270166  9679 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.324301 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:15.360775 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.361305 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.329181  9803 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:15.329572  9355 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:15.329339  9794 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 33.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:15.329157  9795 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:15.331538 31824 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:15.331634 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.352891  9757 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.333350 31818 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 29 index: 18. (index mismatch)
I20260709 12:25:15.398231 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1618) took 1021 ms. Trace:
I20260709 12:25:15.334237  9811 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:15.334877 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.415458 31819 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:25:15.335388  9718 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:15.336824  9678 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:15.336923  9772 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.337877 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:15.364879  9690 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:15.338809 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 166529us
I20260709 12:25:15.344404  9180 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.346920 32343 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:15.431679 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 93900.1us
I20260709 12:25:15.432618 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:15.310979 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.357285 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.362238 31825 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:25:15.435050 31825 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:15.365365 31821 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:15.281462  9537 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.355435  9535 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:15.438789 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:15.377681 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53444us
I20260709 12:25:15.378568  9592 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:15.380169  9721 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:15.380966  9777 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:15.383533  9734 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:15.384277  9791 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.385929 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.388168 31826 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 35: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:15.332741 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.467862 31827 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:25:15.392155 31673 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:15.395876 32376 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 1402) took 1027 ms. Trace:
I20260709 12:25:15.469465 32376 rpcz_store.cc:276] 0709 12:25:14.368053 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.368727 (+   674us) service_pool.cc:224] Handling call
0709 12:25:15.395517 (+1026790us) spinlock_profiling.cc:243] Waited 232 ms on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:15.395766 (+   249us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":533727616}
I20260709 12:25:15.398216 31674 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:25:15.400355 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4147) took 1108 ms. Trace:
I20260709 12:25:15.405596 31671 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:25:15.403261 31676 rpcz_store.cc:276] 0709 12:25:14.376446 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.377006 (+   560us) service_pool.cc:224] Handling call
0709 12:25:14.398939 (+ 21933us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.467541 (+ 68602us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:25:14.601675 (+134134us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.011677 (+410002us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:15.011737 (+    60us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:15.011803 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.397863 (+386060us) spinlock_profiling.cc:243] Waited 68.4 ms on lock 0x7b54002b1454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:15.397937 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.398142 (+   205us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":144581,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":48,"raft.queue_time_us":1343,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":5940,"spinlock_wait_cycles":157209344,"thread_start_us":493267,"threads_started":2,"wal-append.queue_time_us":487721}
I20260709 12:25:15.367638  9679 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:15.431720 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 84908.2us
I20260709 12:25:15.353993 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:15.441449 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:15.443287 32370 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 19 index: 12. Preceding OpId from leader: term: 23 index: 13. (index mismatch)
I20260709 12:25:15.357601  9799 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 33.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.085s
W20260709 12:25:15.442520  9745 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.567s	user 0.011s	sys 0.001s
I20260709 12:25:15.430928  9811 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:15.453474  9592 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.443230  9776 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:15.497941 31746 pending_rounds.cc:85] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 21
I20260709 12:25:15.508888 31746 pending_rounds.cc:107] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 31.22
I20260709 12:25:15.509294 31746 raft_consensus.cc:2889] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:15.454767  9355 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:15.500324 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:15.491133 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:15.463191  9453 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:15.471477 31752 rpcz_store.cc:276] 0709 12:25:14.291617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.291808 (+   191us) service_pool.cc:224] Handling call
0709 12:25:14.305584 (+ 13776us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:14.325760 (+ 20176us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:14.463475 (+137715us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.900773 (+437298us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:14.900846 (+    73us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:14.900911 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.399974 (+499063us) spinlock_profiling.cc:243] Waited 19.9 ms on lock 0x7b54002968d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:15.400050 (+    76us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.400269 (+   219us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":4297,"prepare.queue_time_us":129596,"prepare.run_cpu_time_us":49,"prepare.run_wall_time_us":51,"raft.queue_time_us":4521,"raft.run_cpu_time_us":148,"raft.run_wall_time_us":1142,"spinlock_wait_cycles":45689600,"thread_start_us":527962,"threads_started":2,"wal-append.queue_time_us":570926}
I20260709 12:25:15.427891  9757 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.517251 31753 raft_consensus.cc:2368] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31.
I20260709 12:25:15.454897  9548 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.050s
I20260709 12:25:15.522785 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:15.523614 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 169714us
I20260709 12:25:15.504120  9621 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:15.529829 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.513543  9592 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:15.535022  9813 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:15.535270 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:15.535929 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.548651 31673 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 32: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:15.536406 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.537410 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.520020  9453 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:15.513847  9806 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.488188  9775 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.041s
I20260709 12:25:15.555927 31678 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:15.453699  9520 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.525113  9355 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:25:15.551033 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:15.552632 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 120112us
I20260709 12:25:15.530123  9548 consensus_queue.cc:1243] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 25.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 14
I20260709 12:25:15.553520 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:15.521680  9798 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 32.16, Next index: 17, Last known committed idx: 15, Time since last communication: 0.033s
I20260709 12:25:15.530619  9656 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.532490  9814 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.564169  9776 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:15.565816  9795 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:15.567308 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:15.568945 31752 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:15.525831  9777 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:15.570472 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:15.591109 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:25:15.567976 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 49977.5us
I20260709 12:25:15.571264 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.554206  9453 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:15.557087  9747 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:15.556571  9621 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:15.596470 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:15.573410  9793 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 11, Time since last communication: 0.028s
I20260709 12:25:15.605777  9355 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.597589  9690 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:15.609871  9690 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:15.610319  9690 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:15.612475  9771 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:15.575083  9721 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:15.573295  9813 consensus_queue.cc:1243] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 24.16 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 15
I20260709 12:25:15.599679 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:15.617658 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:15.621827 31747 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:15.622570 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:15.623294 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:15.624238  9799 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:15.624702  9799 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.624770 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:15.625109  9799 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:15.627665  9638 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:15.628237  9638 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.628599  9638 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:15.641563 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:15.568284  9811 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.15, Last appended by leader: 15, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.595624  9745 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:15.667485 31791 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:15.613773  9718 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:15.598333  9798 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.063s
I20260709 12:25:15.639492  9812 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.618924  9822 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.089s
I20260709 12:25:15.589690  9814 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.654223  9819 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.613129  9771 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:15.589056  9656 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:15.729158 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60916.2us
I20260709 12:25:15.558959  9806 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.651381  9726 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:15.661055 31747 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 19. Preceding OpId from leader: term: 29 index: 20. (index mismatch)
I20260709 12:25:15.708994 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:15.666294 31752 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:15.740556 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:15.760094 31673 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:25:15.741168 31671 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: [27.14-27.14]
W20260709 12:25:15.691453  9816 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.141s	user 0.001s	sys 0.000s
I20260709 12:25:15.725848  9813 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:25:15.682256  9815 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
W20260709 12:25:15.671361  9759 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.188s	user 0.009s	sys 0.003s
I20260709 12:25:15.729393  9180 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:15.691347  9721 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:15.692420  9718 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:15.831112  9718 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:15.833470  9679 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.834717 31677 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:15.682952  9772 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:15.690819  9777 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 23.15, Next index: 16, Last known committed idx: 15, Time since last communication: 0.124s
W20260709 12:25:15.692113  9684 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.610s	user 0.012s	sys 0.053s
I20260709 12:25:15.737725  9823 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.080s
I20260709 12:25:15.819088 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4129) took 1693 ms. Trace:
I20260709 12:25:15.846900 31745 rpcz_store.cc:276] 0709 12:25:14.125341 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.169053 (+ 43712us) service_pool.cc:224] Handling call
0709 12:25:14.223430 (+ 54377us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:14.229947 (+  6517us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.229982 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:14.238681 (+  8699us) log.cc:844] Serialized 52 byte log entry
0709 12:25:14.384693 (+146012us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:14.384755 (+    62us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.384858 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:15.575437 (+1190579us) spinlock_profiling.cc:243] Waited 5.76 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:15.818736 (+243299us) raft_consensus.cc:1685] finished
0709 12:25:15.818763 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.819019 (+   256us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":38233,"raft.run_cpu_time_us":43,"raft.run_wall_time_us":43,"spinlock_wait_cycles":13246208,"thread_start_us":113666,"threads_started":1,"wal-append.queue_time_us":235617}
I20260709 12:25:15.741947  9819 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.807482  9756 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.066s
W20260709 12:25:15.822093  9802 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.493s	user 0.000s	sys 0.003s
I20260709 12:25:15.822737  9586 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:15.655303  9826 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.175s
I20260709 12:25:15.668370  9747 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:15.687592  9621 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.16, Last appended by leader: 16, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.755314  9812 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.813392  9757 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:15.879943  9757 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.880432  9757 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.830485  9791 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:15.884805  9794 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:15.885340  9794 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:15.670805  9824 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.170s
I20260709 12:25:15.888005  9824 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:15.888451  9824 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:15.889570  9824 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.859071 31820 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:15.865123  9775 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:15.930540 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:15.822896  9535 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:15.861696 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.938319 31642 leader_election.cc:400] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31.
I20260709 12:25:15.938843 31642 leader_election.cc:403] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:15.938972 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:15.865172 31673 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
W20260709 12:25:15.939911 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 230981us
I20260709 12:25:15.839421  9180 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:15.942269 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:15.839264  9813 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:15.871449  9747 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.875195  9616 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:15.876226 31825 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:25:15.876940  9755 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:15.879172  9548 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:15.891768  9786 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.728s	user 0.002s	sys 0.000s
I20260709 12:25:15.892921  9561 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:15.886344  9794 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.895752 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:15.896827 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1487) took 1492 ms. Trace:
I20260709 12:25:15.845209  9721 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:15.867565  9771 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 27.14, Last appended by leader: 14, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.904536  9717 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:25:15.911062 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.917476 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:15.859314 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 25 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:15.850318  9684 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.768s	user 0.013s	sys 0.054s
I20260709 12:25:15.992487 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:15.927937  9592 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.994104  9811 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:15.994414 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:15.995067  9592 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:15.929232  9718 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.995762  9592 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:15.997153  9718 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:15.882539  9757 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:15.860769 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:15.998083 31674 raft_consensus.cc:1275] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 18. (term mismatch)
I20260709 12:25:15.998049 31818 raft_consensus.cc:2376] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35. Re-sending same reply.
I20260709 12:25:15.937701 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:15.864115 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:15.999505 31822 raft_consensus.cc:2376] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34. Re-sending same reply.
I20260709 12:25:16.000712 31672 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:15.929092  9812 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:15.945902 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:15.914820  9777 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:15.949379 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:15.960376 31791 leader_election.cc:400] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 35. Current term is 36.
I20260709 12:25:15.961855 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:15.920778  9826 consensus_queue.cc:1243] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 33.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:25:15.968145 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:15.968811 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:15.967797  9755 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:16.024514 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:16.025228 32373 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 35.
I20260709 12:25:15.967648 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:15.920147  9726 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:15.980249 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:15.980473 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:15.980958 31748 raft_consensus.cc:1217] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Deduplicated request from leader. Original: 23.14->[24.15-30.18]   Dedup: 24.15->[26.16-30.18]
I20260709 12:25:15.984892 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:15.994843  9811 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:15.998536 31674 pending_rounds.cc:85] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 17
I20260709 12:25:16.001624 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:15.968961  9668 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31.
I20260709 12:25:15.980194  9795 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.030560 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1489) took 1624 ms. Trace:
I20260709 12:25:16.032681 31674 pending_rounds.cc:107] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 28.18
I20260709 12:25:16.033859 31748 pending_rounds.cc:85] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 15
I20260709 12:25:16.034138 31674 raft_consensus.cc:2889] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:15.943466 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:16.002367 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:16.035493 32370 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:16.008751 31670 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 33 index: 24. (index mismatch)
I20260709 12:25:16.012123 31791 leader_election.cc:403] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Cancelling election due to peer responding with higher term
I20260709 12:25:15.977361  9765 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:16.034303 31748 pending_rounds.cc:107] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 25.16
I20260709 12:25:16.045486 31748 raft_consensus.cc:2889] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:16.045612 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:16.026907 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:16.037675 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 68983.8us
I20260709 12:25:15.944588  9775 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:15.977818  9413 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.058s
I20260709 12:25:16.014185  9794 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:16.034242 31673 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:25:16.033439 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:16.002272  9717 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:16.048118 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:15.987901  9561 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:16.033905 31750 rpcz_store.cc:276] 0709 12:25:14.405949 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.413280 (+  7331us) service_pool.cc:224] Handling call
0709 12:25:14.413670 (+   390us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:14.605311 (+191641us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.605359 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:14.607541 (+  2182us) log.cc:844] Serialized 26 byte log entry
0709 12:25:15.003407 (+395866us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:15.003449 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.003524 (+    75us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:15.891108 (+887584us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:16.012494 (+121386us) raft_consensus.cc:1685] finished
0709 12:25:16.012514 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.030467 (+ 17953us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":94224,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":1961,"spinlock_wait_cycles":8206592,"thread_start_us":525413,"threads_started":2,"wal-append.queue_time_us":476546}
I20260709 12:25:15.996381  9592 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:16.054491 31671 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 30. Ops: [29.16-29.16]
I20260709 12:25:16.054957 31675 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
I20260709 12:25:16.055852 31669 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:15.981446  9752 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.054s
I20260709 12:25:15.983220 31749 rpcz_store.cc:276] 0709 12:25:14.403991 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:14.404134 (+   143us) service_pool.cc:224] Handling call
0709 12:25:14.404530 (+   396us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:14.430734 (+ 26204us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:14.430789 (+    55us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:14.460940 (+ 30151us) log.cc:844] Serialized 52 byte log entry
0709 12:25:14.828934 (+367994us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:14.872194 (+ 43260us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.879089 (+  6895us) log.cc:844] Serialized 14 byte log entry
0709 12:25:14.879989 (+   900us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:14.880089 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:14.980880 (+100791us) spinlock_profiling.cc:243] Waited 5.29 ms on lock 0x7b18000efb00. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:15.881069 (+900189us) spinlock_profiling.cc:243] Waited 3.8 ms on lock 0x7b18000efb00. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:15.896619 (+ 15550us) raft_consensus.cc:1685] finished
0709 12:25:15.896642 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:15.896771 (+   129us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36587,"prepare.queue_time_us":157348,"prepare.run_cpu_time_us":91,"prepare.run_wall_time_us":2413,"raft.queue_time_us":61541,"raft.run_cpu_time_us":545,"raft.run_wall_time_us":20729,"spinlock_wait_cycles":20908032,"thread_start_us":313616,"threads_started":1,"wal-append.queue_time_us":407317}
I20260709 12:25:16.069540 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:15.984956  9616 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.057s
I20260709 12:25:16.117502 31671 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 26 index: 18. (index mismatch)
I20260709 12:25:15.964583  9813 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:16.085829 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 52463.5us
I20260709 12:25:16.031510  9771 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:16.101794 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53808us
I20260709 12:25:16.184263 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:16.031095  9684 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:16.101796 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 151453us
I20260709 12:25:15.980003  9548 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.105543 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:16.049559  9621 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:16.116148 32370 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
W20260709 12:25:16.199635 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 87503.3us
I20260709 12:25:16.189749 31750 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
I20260709 12:25:16.030354  9777 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.050005  9765 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:16.080470  9757 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:16.059305  9180 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:16.189253 31791 leader_election.cc:400] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 34. Current term is 35.
I20260709 12:25:16.219008 31791 leader_election.cc:403] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:16.084127  9822 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:16.122879 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 161118us
W20260709 12:25:16.220047  9638 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.591s	user 0.008s	sys 0.007s
I20260709 12:25:16.221208  9638 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:16.222483 32371 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:16.223708  9638 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:16.210764  9755 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.226068  9755 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:16.227636 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:16.229287  9831 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.266s	user 0.001s	sys 0.000s
W20260709 12:25:15.946077  9745 consensus_queue.cc:1175] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 32 status { last_received { term: 30 index: 16 } last_committed_idx: 16 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:16.153607  9773 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.156355  9750 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:16.184989  9792 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Leader pre-election won for term 24
I20260709 12:25:16.049947  9795 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:16.201259  9679 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:16.241539 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52446us
I20260709 12:25:16.117046  9561 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.243813 31819 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:16.199785  9621 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:16.184778  9840 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election lost for term 35. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 35. Current term is 36.
I20260709 12:25:16.207724  9593 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:16.246481 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:16.220002  9765 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:16.308404 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61993.1us
I20260709 12:25:16.232292  9842 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:25:16.282820 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 388192us
W20260709 12:25:16.290544 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.322446 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:16.300226 31678 raft_consensus.cc:2468] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
W20260709 12:25:16.251960 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 302655us
I20260709 12:25:16.212255  9611 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.043s
I20260709 12:25:16.309707 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:16.325702 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:16.249068  9830 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.235s	user 0.001s	sys 0.000s
W20260709 12:25:16.320999 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 100126us
I20260709 12:25:16.309625 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:16.244513  9770 raft_consensus.cc:2764] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:25:16.244442  9718 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election lost for term 34. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 34. Current term is 35.
I20260709 12:25:16.207712  9684 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.252676  9838 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.248886  9853 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:16.356338 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 46769.1us
I20260709 12:25:16.398168 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:16.289364  9621 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:16.250721  9750 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:16.345151  9535 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:16.364435 31822 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:16.241439  9776 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.103s
I20260709 12:25:16.325467  9679 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:16.326618  9842 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.394151  9413 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:16.309190  9593 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:16.489018 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 103779us
I20260709 12:25:16.412338 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:16.218276  9721 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:16.421284 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:16.367113 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:16.352115  9840 raft_consensus.cc:2764] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election decision vote started in defunct term 31: won
I20260709 12:25:16.501159 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:16.501731  9849 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 29.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:16.242807  9773 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.503381  9819 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:16.504514  9773 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:16.385360  9825 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:25:16.244874  9717 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.23, Last appended by leader: 23, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:16.473845 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 75786.3us
I20260709 12:25:16.506120 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:16.387509  9537 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:16.506399  9355 consensus_queue.cc:1048] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:16.507102  9537 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:25:16.510758  9792 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:16.511359  9792 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:16.468811 31677 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4145) took 1226 ms. Trace:
I20260709 12:25:16.518527 31677 rpcz_store.cc:276] 0709 12:25:15.242301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.242502 (+   201us) service_pool.cc:224] Handling call
0709 12:25:15.242978 (+   476us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:15.835105 (+592127us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.468548 (+633443us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b54002b0a54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:16.468745 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1360322944}
I20260709 12:25:16.520407 31677 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:16.520617 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:16.469812 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:16.375031 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 218501us
I20260709 12:25:16.522770  9548 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:16.522874 31792 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:16.290660  9592 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.525463 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:16.233552  9822 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.527693 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:16.528410 31825 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:16.528667 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:16.242086  9726 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 27.14, Last appended by leader: 14, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.359292  9844 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:16.502014 31822 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
W20260709 12:25:16.499678 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 80491us
I20260709 12:25:16.535465 31826 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 36: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:16.490052 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:16.524410  9840 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:16.541496 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:16.476655  9690 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.866s	user 0.016s	sys 0.004s
I20260709 12:25:16.543520 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:16.604895 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 63528.5us
I20260709 12:25:16.472209  9679 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.480654  9718 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:16.494015  9593 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:16.488692  9842 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:16.465668  9747 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:16.465960  9812 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:16.637778 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 131760us
I20260709 12:25:16.605270 31822 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:16.638615 31822 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:16.639989 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:16.421780  9838 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:16.609840 31672 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.005s	sys 0.005s
W20260709 12:25:16.611804 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 427658us
I20260709 12:25:16.468205  9851 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:16.447225  9853 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:16.645767 31672 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.006s	sys 0.005s
W20260709 12:25:16.651290 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 129721us
W20260709 12:25:16.656873 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 166902us
I20260709 12:25:16.611518  9679 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:16.670918 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:16.611413  9413 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:16.477727  9180 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.639675 31822 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 23.15, Last appended by leader: 15, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.620609  9840 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:16.611119  9775 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:16.597743  9847 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:25:16.694074 31827 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:16.655334  9718 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:16.611773  9690 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:16.694554  9846 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.479s	user 0.000s	sys 0.002s
W20260709 12:25:16.685242  9843 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.451s	user 0.001s	sys 0.000s
I20260709 12:25:16.706136 31822 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:16.679738  9809 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:16.608785  9844 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:16.683893  9413 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:16.681205  9813 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:16.715458 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57674.8us
W20260709 12:25:16.703450  9799 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 1.074s	user 0.011s	sys 0.004s
I20260709 12:25:16.728018 31825 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:16.695292  9777 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:16.734323 31827 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:16.664125  9812 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:16.703146  9840 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:16.707767  9851 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:16.620191  9750 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 29.22, Last appended by leader: 22, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:16.707607  9795 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:16.756332 31821 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:16.725193  9413 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:16.769521 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.760s	user 0.008s	sys 0.016s
W20260709 12:25:16.770520 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.783s	user 0.009s	sys 0.017s
I20260709 12:25:16.748389 31825 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
W20260709 12:25:16.772984 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:16.777624  9755 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.551s	user 0.009s	sys 0.000s
I20260709 12:25:16.750490  9836 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:16.782718  9832 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.164s
I20260709 12:25:16.783234  9836 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.785706  9690 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:16.785842  9836 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:16.778939  9878 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:16.795984 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:16.760686  9561 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:16.797951 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:16.770280  9835 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.516s	user 0.000s	sys 0.001s
W20260709 12:25:16.810945 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 267473us
I20260709 12:25:16.812860 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:16.743165  9799 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:16.772085  9851 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:25:16.807782  9866 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.338s	user 0.000s	sys 0.001s
I20260709 12:25:16.803162  9826 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:16.823101 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 129936us
I20260709 12:25:16.800275  9684 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:16.836194 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:16.836994 31820 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:16.840076 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:16.841481 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:16.842744 31822 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 23 index: 15. Preceding OpId from leader: term: 24 index: 16. (index mismatch)
I20260709 12:25:16.842612 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:16.846890 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:16.848351 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:16.849435 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:16.817930  9832 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:16.851361 31751 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:25:16.858188 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:16.857064 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:16.860113 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:16.861364 31749 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:25:16.862944 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:16.884855 31750 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:16.885995 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:16.887413 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:16.888919 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:16.890169 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:16.895084  9851 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.22, Last appended by leader: 22, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:16.826169  9848 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:16.900255  9848 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:16.900722  9848 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:16.902725  9621 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.866802  9822 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:16.870765  9872 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.050s	user 0.000s	sys 0.000s
W20260709 12:25:16.827196  9803 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.005s	sys 0.005s
W20260709 12:25:16.871795  9856 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.103s	user 0.000s	sys 0.000s
W20260709 12:25:16.872843  9852 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:25:16.873785  9853 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:16.873994  9453 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.259s	user 0.004s	sys 0.006s
W20260709 12:25:16.853572  9765 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.504s	user 0.010s	sys 0.000s
W20260709 12:25:16.881680  9820 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 1.199s	user 0.000s	sys 0.006s
I20260709 12:25:16.883118  9812 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:16.903920  9792 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:16.904616  9806 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:16.863849  9838 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:16.908463 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:16.909730 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:16.981326 31749 raft_consensus.cc:2393] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 28: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:16.912588 31822 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:16.921695 31820 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:16.938831 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:16.940119 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4157) took 1445 ms. Trace:
I20260709 12:25:16.949707  9413 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:16.954674  9734 raft_consensus.cc:936] Couldn't advance consensus term.: Illegal state: Can't advance term to: 24 current term: 24 is higher.
I20260709 12:25:16.854853  9721 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:16.940513  9803 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s	user 0.005s	sys 0.006s
W20260709 12:25:16.969507 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:16.984251 31746 rpcz_store.cc:276] 0709 12:25:15.494357 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.497275 (+  2918us) service_pool.cc:224] Handling call
0709 12:25:15.497721 (+   446us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:15.517074 (+ 19353us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.517118 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:15.517704 (+   586us) log.cc:844] Serialized 52 byte log entry
0709 12:25:15.588301 (+ 70597us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:25:15.602179 (+ 13878us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.639875 (+ 37696us) log.cc:844] Serialized 14 byte log entry
0709 12:25:15.647922 (+  8047us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:15.648013 (+    91us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:15.900858 (+252845us) spinlock_profiling.cc:243] Waited 3.66 ms on lock 0x7b4800232758. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:25:15.906806 (+  5948us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b18004e6e80. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:16.900012 (+993206us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b18004e6e80. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:16.939829 (+ 39817us) raft_consensus.cc:1685] finished
0709 12:25:16.939854 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:16.940044 (+   190us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24379,"prepare.queue_time_us":313042,"prepare.run_cpu_time_us":92,"prepare.run_wall_time_us":94,"raft.queue_time_us":69449,"raft.run_cpu_time_us":3275,"raft.run_wall_time_us":281858,"spinlock_wait_cycles":58950528,"thread_start_us":38907,"threads_started":1,"wal-append.queue_time_us":147225}
W20260709 12:25:16.994019 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 85607.2us
I20260709 12:25:17.000169 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:16.997226 31825 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:17.015051 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.021814 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:17.023557  9790 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:17.024031 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4167) took 1069 ms. Trace:
I20260709 12:25:17.024477 31753 rpcz_store.cc:276] 0709 12:25:15.955061 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.955213 (+   152us) service_pool.cc:224] Handling call
0709 12:25:15.977400 (+ 22187us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:15.986278 (+  8878us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:15.986310 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:15.986802 (+   492us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.122428 (+135626us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:16.122487 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.122587 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.023781 (+901194us) raft_consensus.cc:1685] finished
0709 12:25:17.023810 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.023959 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":734,"raft.queue_time_us":2653,"raft.run_cpu_time_us":144,"raft.run_wall_time_us":4697,"thread_start_us":75457,"threads_started":1,"wal-append.queue_time_us":695629}
I20260709 12:25:17.024081  9790 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:17.028738  9790 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.16, Last appended by leader: 16, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:17.030125  9794 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.927s	user 0.013s	sys 0.000s
I20260709 12:25:17.040102 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:17.040493 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:16.969103  9453 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:16.991469  9871 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:16.996264  9838 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:16.969333  9765 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.054744 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:17.057902 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.059191 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.059844  9814 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:17.060294 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.061463 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.063416  9656 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:17.051949  9684 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.019s
I20260709 12:25:17.071557  9844 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:16.999542  9721 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.077116  9721 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:17.079452  9842 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.080039  9734 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:17.079454 31674 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4160) took 1027 ms. Trace:
I20260709 12:25:17.080906 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:17.087836  9734 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:17.056274  9871 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.070874  9881 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:17.090207  9881 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:17.090617  9881 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.090979 31669 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:25:17.082193  9792 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:17.083360  9621 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:17.085182  9765 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:17.087152  9453 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:17.088541  9734 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:17.094801  9355 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:17.095260  9355 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:17.096264  9355 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.097471  9355 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:17.090989  9881 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:17.101943 31678 raft_consensus.cc:1240] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 33. Current term is 34. Ops: [33.24-33.24]
I20260709 12:25:17.087879 31674 rpcz_store.cc:276] 0709 12:25:16.052412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.053766 (+  1354us) service_pool.cc:224] Handling call
0709 12:25:16.054056 (+   290us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.283747 (+229691us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.283798 (+    51us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:16.314554 (+ 30756us) log.cc:844] Serialized 26 byte log entry
0709 12:25:16.915768 (+601214us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:16.915837 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:16.915951 (+   114us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.079133 (+163182us) raft_consensus.cc:1685] finished
0709 12:25:17.079169 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.079365 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":124445,"raft.queue_time_us":234322,"raft.run_cpu_time_us":211,"raft.run_wall_time_us":36467,"spinlock_wait_cycles":8960,"thread_start_us":671692,"threads_started":2,"wal-append.queue_time_us":680150}
I20260709 12:25:17.103248 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.104449 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.105983 31674 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:25:17.060227  9848 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:17.108067  9757 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.874s	user 0.006s	sys 0.007s
I20260709 12:25:17.090615  9871 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:17.110406 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.082968  9826 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:17.112540  9757 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:17.113031  9717 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:17.113459 31825 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 29. Current term is 30. Ops: []
W20260709 12:25:17.111225  9771 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.908s	user 0.004s	sys 0.008s
I20260709 12:25:17.062453 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.117138 31676 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:17.109403  9848 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:17.119182  9842 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:17.119915  9679 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.123430  9679 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:17.118456 31676 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 23. Current term is 24. Ops: []
W20260709 12:25:17.131126 31673 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.598s	user 0.008s	sys 0.020s
I20260709 12:25:17.130050 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1492) took 1073 ms. Trace:
W20260709 12:25:17.135605 31673 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.603s	user 0.008s	sys 0.020s
I20260709 12:25:17.135676 31745 rpcz_store.cc:276] 0709 12:25:16.056620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.056774 (+   154us) service_pool.cc:224] Handling call
0709 12:25:16.057154 (+   380us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:16.309804 (+252650us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.309851 (+    47us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:16.325056 (+ 15205us) log.cc:844] Serialized 52 byte log entry
0709 12:25:16.610958 (+285902us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:16.611310 (+   352us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.129636 (+518326us) spinlock_profiling.cc:243] Waited 252 ms on lock 0x7b54002954d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:17.129687 (+    51us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.129731 (+    44us) raft_consensus.cc:1685] finished
0709 12:25:17.129745 (+    14us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.129987 (+   242us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":34800,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":4966,"spinlock_wait_cycles":579927296,"thread_start_us":187909,"threads_started":1,"wal-append.queue_time_us":312589}
I20260709 12:25:17.128788  9799 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:17.141000  9500 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:17.145395  9500 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:17.147379  9500 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.147691 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1614) took 1498 ms. Trace:
I20260709 12:25:17.138594  9734 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.131860  9593 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.126587  9881 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.148241 32371 rpcz_store.cc:276] 0709 12:25:15.649619 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.649813 (+   194us) service_pool.cc:224] Handling call
0709 12:25:15.691949 (+ 42136us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.222874 (+530925us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.147417 (+924543us) spinlock_profiling.cc:243] Waited 530 ms on lock 0x7b54003f3254. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:17.147631 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1219571072}
I20260709 12:25:17.163139  9500 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:17.171128 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4162) took 1118 ms. Trace:
I20260709 12:25:17.171646 31672 rpcz_store.cc:276] 0709 12:25:16.053015 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.053167 (+   152us) service_pool.cc:224] Handling call
0709 12:25:16.053593 (+   426us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.516809 (+463216us) spinlock_profiling.cc:243] Waited 35.1 ms on lock 0x7b4800080058. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:25:16.678224 (+161415us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.678260 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:16.951356 (+273096us) spinlock_profiling.cc:243] Waited 1.54 ms on lock 0x7b2c000d7278. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:25:16.952052 (+   696us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.139376 (+187324us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:17.139422 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.139495 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.170862 (+ 31367us) raft_consensus.cc:1685] finished
0709 12:25:17.170890 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.171056 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":130611,"raft.queue_time_us":190897,"raft.run_cpu_time_us":2495,"raft.run_wall_time_us":267210,"spinlock_wait_cycles":84352000,"thread_start_us":756606,"threads_started":2,"wal-append.queue_time_us":202619}
I20260709 12:25:17.174084 31827 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 35. Current term is 36. Ops: [35.23-35.23]
I20260709 12:25:17.174463 31825 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 34. Current term is 35. Ops: [34.22-34.22]
I20260709 12:25:17.184000  9734 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:17.187418 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:17.189697 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:17.196971 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:17.197700 31826 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:17.200026  9881 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:17.201735  9593 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:17.290704 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1615) took 1299 ms. Trace:
I20260709 12:25:17.203092  9824 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:17.211498  9790 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:17.215171 32371 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:17.224732  9413 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:17.226058 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.228700  9721 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.242604 31826 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
I20260709 12:25:17.302678 32343 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:17.241060  9848 consensus_queue.cc:1175] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 33 status { last_received { term: 32 index: 16 } last_committed_idx: 15 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 32. Current term is 33. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:25:17.241784  9857 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.621s	user 0.040s	sys 0.035s
I20260709 12:25:17.259795 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4159) took 1207 ms. Trace:
W20260709 12:25:17.262789 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 587086us
I20260709 12:25:17.306963 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:17.307654 31826 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:17.307693 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:17.308034 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:17.309165 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:17.299786  9757 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.010s
W20260709 12:25:17.266412 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 545726us
I20260709 12:25:17.268021  9809 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:17.268738 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:17.269055 31671 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:17.310994 31672 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:25:17.312140 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:17.263494  9868 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:17.275185 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 3987) took 1418 ms. Trace:
I20260709 12:25:17.286697 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.288043 31674 raft_consensus.cc:1240] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:17.291172 32372 rpcz_store.cc:276] 0709 12:25:15.990853 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.997466 (+  6613us) service_pool.cc:224] Handling call
0709 12:25:15.997869 (+   403us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.945988 (+948119us) spinlock_profiling.cc:243] Waited 1.84 ms on lock 0x7b1800195b20. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcdcdbfb 00007fe8fcdd864b
0709 12:25:16.946185 (+   197us) raft_consensus.cc:1534] Early marking committed up to index 12
0709 12:25:16.946571 (+   386us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.132564 (+185993us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.134449 (+  1885us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.134614 (+   165us) raft_consensus.cc:1645] Marking committed up to 12
0709 12:25:17.134684 (+    70us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.134764 (+    80us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.277202 (+142438us) raft_consensus.cc:1685] finished
0709 12:25:17.290460 (+ 13258us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.290618 (+   158us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3528,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":103934,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":46046,"spinlock_wait_cycles":4228480,"thread_start_us":357174,"threads_started":2,"wal-append.queue_time_us":278143}
W20260709 12:25:17.309725  9857 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.698s	user 0.040s	sys 0.035s
I20260709 12:25:17.319356 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:17.314644 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:17.269919 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.304899 31670 rpcz_store.cc:276] 0709 12:25:16.052072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.052258 (+   186us) service_pool.cc:224] Handling call
0709 12:25:16.052611 (+   353us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:16.058969 (+  6358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:16.059020 (+    51us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:16.112233 (+ 53213us) log.cc:844] Serialized 52 byte log entry
0709 12:25:16.624788 (+512555us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:16.668468 (+ 43680us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.259044 (+590576us) spinlock_profiling.cc:243] Waited 1.64 ms on lock 0x7b1800347ec0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:25:17.259266 (+   222us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.259365 (+    99us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.259417 (+    52us) raft_consensus.cc:1685] finished
0709 12:25:17.259532 (+   115us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.259707 (+   175us) inbound_call.cc:177] Queueing success response
0709 12:25:17.291308 (+ 31601us) spinlock_profiling.cc:243] Waited 3.65 ms on lock 0x7b1800347ec0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8557 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
Metrics: {"mutex_wait_us":94989,"prepare.queue_time_us":30696,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":60,"raft.queue_time_us":785736,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":41392,"spinlock_wait_cycles":12157568,"thread_start_us":378693,"threads_started":1,"wal-append.queue_time_us":679805}
I20260709 12:25:17.316663 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:17.316378 31821 rpcz_store.cc:276] 0709 12:25:15.856648 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.859901 (+  3253us) service_pool.cc:224] Handling call
0709 12:25:15.860269 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.757710 (+897441us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.274937 (+517227us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b54002675d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:17.275122 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2060087552}
I20260709 12:25:17.321668 31826 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:17.317924 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:17.316089 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:17.315470  9757 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Stepping down as leader of term 33
W20260709 12:25:17.324645 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 54820.6us
I20260709 12:25:17.327239 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:17.328236 31826 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:17.322885 32371 raft_consensus.cc:2410] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 28 because replica has last-logged OpId of term: 27 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 19.
I20260709 12:25:17.325888 31820 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:17.328446 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.326341  9757 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:17.331660  9757 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.22, Last appended by leader: 22, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.332913  9757 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:17.332724  9809 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 26.15, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.333495 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:17.334693 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:17.335342 31818 raft_consensus.cc:2368] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31.
I20260709 12:25:17.335373 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:17.336859 31818 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:25:17.338284 31818 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:25:17.338995 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:17.339681 31827 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:17.341508 31820 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:17.343245 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.344604 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:17.346882 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.351024 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:17.351156 32374 raft_consensus.cc:1240] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:17.352108 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:17.352448 32371 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:17.320499  9799 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:17.295980  9865 consensus_queue.cc:1059] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 32.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.039s
I20260709 12:25:17.356446 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:17.357800 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4165) took 1035 ms. Trace:
I20260709 12:25:17.311386  9721 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.368769 31826 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:17.370004  9721 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:17.370570  9873 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.370934 31678 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 25 index: 13. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
I20260709 12:25:17.371124  9873 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.372496 31678 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:25:17.373366  9873 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:17.378979  9880 consensus_queue.cc:1059] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:17.322049  9413 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.042s
I20260709 12:25:17.341652  9881 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:17.354651  9838 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:17.355691 32342 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.364743 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:17.366736 31823 rpcz_store.cc:276] 0709 12:25:16.322320 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.328553 (+  6233us) service_pool.cc:224] Handling call
0709 12:25:16.344575 (+ 16022us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.344833 (+   258us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:16.421411 (+ 76578us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.201411 (+780000us) spinlock_profiling.cc:243] Waited 32.1 ms on lock 0x7b1800386260. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a 00007fe8fcddfa2a
0709 12:25:17.357379 (+155968us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:17.357434 (+    55us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:17.357489 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.357622 (+   133us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.357742 (+   120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3715,"prepare.queue_time_us":76158,"prepare.run_cpu_time_us":45,"prepare.run_wall_time_us":47,"raft.queue_time_us":3850,"spinlock_wait_cycles":73899392,"thread_start_us":151729,"threads_started":1}
I20260709 12:25:17.355994 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.401589  9873 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:17.402858 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:17.342607 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:17.402895  9873 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:17.404059  9873 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 27 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:17.404186  9842 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:17.405028  9873 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.405543 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:25:17.407254 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51375us
I20260709 12:25:17.408146 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.366382  9851 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:17.419204 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:17.391374 31672 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
I20260709 12:25:17.398025  9798 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:17.477663 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 58599us
I20260709 12:25:17.474656  9500 raft_consensus.cc:3055] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 33 LEADER]: Stepping down as leader of term 33
W20260709 12:25:17.482687 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74616.7us
I20260709 12:25:17.483165  9834 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:17.484429  9734 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
W20260709 12:25:17.484894 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 95243.2us
I20260709 12:25:17.406332  9873 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:17.400549  9871 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:25:17.485781 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 130140us
I20260709 12:25:17.440299  9842 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election won for term 27
W20260709 12:25:17.495339 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.495796 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.414613  9881 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:17.489135 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.487634  9355 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:17.497153 32372 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:25:17.498273 31820 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:25:17.491080 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:17.476612  9799 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:17.487682  9500 raft_consensus.cc:740] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:17.509436 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.514999  9853 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:17.515681  9853 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:17.515499  9500 consensus_queue.cc:260] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.24, Last appended by leader: 24, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.516438 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.516938  9500 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:17.516660  9853 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.517781 31791 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.518822  9838 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:17.519078 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.519621  9838 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:17.520027  9721 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:17.520697  9684 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:17.521414  9721 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:17.523294 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:17.524030  9721 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:17.524446  9721 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.524912  9721 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:17.526984  9453 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:17.527727  9520 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:17.507316  9871 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 33 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:17.531384  9799 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 26.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.535482 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.537191  9721 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.495523  9842 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.546360  9842 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:17.549345 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:17.549937 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:17.550138  9792 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.550552 31745 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:17.550791  9792 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.489754  9798 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:17.543285 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.545226 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 898587us
I20260709 12:25:17.553787 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.545382  9747 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:17.569190  9734 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.000s
I20260709 12:25:17.569608 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.569801  9747 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:17.570519  9747 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:17.570761  9180 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:17.571156 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.571856  9747 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:17.572198 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:17.573659 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:17.574442 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 76976.2us
I20260709 12:25:17.574880 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:17.575487 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.576069 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:17.576181 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.576742 31750 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:17.577859  9887 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:17.552158 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:17.549981 31753 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:17.579442 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:17.579900 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:17.580350  9798 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:17.554126 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.527086 31643 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:17.583261 31745 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:17.556515 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:17.559067 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.552867 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 25 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:17.565521  9771 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:17.553790  9621 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:17.567512  9753 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.567889  9794 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:17.594135 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67112.8us
I20260709 12:25:17.524260  9881 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:17.568215  9721 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:17.580410  9826 raft_consensus.cc:3055] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:25:17.587074 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.588032 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.560024 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.588835 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:17.592146 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:17.597049 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4166) took 1649 ms. Trace:
I20260709 12:25:17.596592 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:17.598125 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:17.598658  9826 raft_consensus.cc:740] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:17.610747 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.599746 31674 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:17.600644 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:17.601488 31751 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:17.605458 31746 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:17.607371 31745 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:25:17.623936 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4166) took 1301 ms. Trace:
I20260709 12:25:17.625003 31643 leader_election.cc:400] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31.
I20260709 12:25:17.625372 31643 leader_election.cc:403] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Cancelling election due to peer responding with higher term
I20260709 12:25:17.626914 32371 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 36.
I20260709 12:25:17.602742 31747 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:17.601946 31748 rpcz_store.cc:276] 0709 12:25:15.947499 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:15.954514 (+  7015us) service_pool.cc:224] Handling call
0709 12:25:15.980730 (+ 26216us) raft_consensus.cc:1471] Updating replica for 4 ops
0709 12:25:16.058146 (+ 77416us) raft_consensus.cc:1534] Early marking committed up to index 15
0709 12:25:16.117659 (+ 59513us) log.cc:844] Serialized 14 byte log entry
0709 12:25:16.356548 (+238889us) raft_consensus.cc:1539] Triggering prepare for 3 ops
0709 12:25:16.357335 (+   787us) log.cc:844] Serialized 78 byte log entry
0709 12:25:16.357501 (+   166us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:16.357762 (+   261us) log.cc:844] Serialized 14 byte log entry
0709 12:25:16.897638 (+539876us) spinlock_profiling.cc:243] Waited 6.9 ms on lock 0x7b18003cf340. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:16.898704 (+  1066us) spinlock_profiling.cc:243] Waited 11.1 ms on lock 0x7b18003cf340. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:17.274310 (+375606us) spinlock_profiling.cc:243] Waited 11 ms on lock 0x7b18003cf340. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:25:17.291062 (+ 16752us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.539195 (+248133us) spinlock_profiling.cc:243] Waited 1.83 ms on lock 0x7b18003cf340. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe900ca958f 00007fe8f9ce8571 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd
0709 12:25:17.596706 (+ 57511us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.596798 (+    92us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.596844 (+    46us) raft_consensus.cc:1685] finished
0709 12:25:17.596857 (+    13us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.596994 (+   137us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":2102820,"prepare.queue_time_us":58741,"prepare.run_cpu_time_us":173,"prepare.run_wall_time_us":14386,"raft.queue_time_us":963159,"raft.run_cpu_time_us":525,"raft.run_wall_time_us":39110,"spinlock_wait_cycles":70874880,"thread_start_us":119240,"threads_started":1,"wal-append.queue_time_us":322547}
I20260709 12:25:17.626845 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:17.631389 31670 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:17.633802 31678 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 36: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:17.625802 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:17.624461 31819 rpcz_store.cc:276] 0709 12:25:16.322592 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.528862 (+206270us) service_pool.cc:224] Handling call
0709 12:25:16.540505 (+ 11643us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:16.692446 (+151941us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:16.807716 (+115270us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.134539 (+326823us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.134771 (+   232us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:17.134819 (+    48us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:17.136556 (+  1737us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.228377 (+ 91821us) spinlock_profiling.cc:243] Waited 3.46 ms on lock 0x7b1800326440. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:17.342920 (+114543us) spinlock_profiling.cc:243] Waited 2.42 ms on lock 0x7b1800326440. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419
0709 12:25:17.623706 (+280786us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x7b54001c89d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:17.623756 (+    50us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.623874 (+   118us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":6065,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":2167,"raft.queue_time_us":183,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":6537,"spinlock_wait_cycles":362355200,"thread_start_us":433046,"threads_started":2,"wal-append.queue_time_us":337341}
I20260709 12:25:17.645605 31748 raft_consensus.cc:1240] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:17.647094 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 64120.3us
I20260709 12:25:17.648658 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.589501  9868 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:17.659189 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.667299 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59480.6us
I20260709 12:25:17.668649 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:17.672149 31748 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:17.676970 31670 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 35.
I20260709 12:25:17.678944 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:17.598968  9592 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:17.681416 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:17.673533 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:17.691847 31822 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:25:17.693219 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:17.685156 31751 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
new_leader_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:35284
I20260709 12:25:17.623948  9611 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.688586 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:17.640014  9791 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.593s	user 0.028s	sys 0.062s
I20260709 12:25:17.530321  9871 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.598721  9753 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.681542  9616 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.623847  9679 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:17.624045  9865 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:17.595906  9621 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.636559  9721 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.696187 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:17.638224  9803 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:17.687544  9844 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:17.596246  9771 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.623594  9765 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:17.666847  9757 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:17.691217  9675 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:17.682881  9764 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.059s
I20260709 12:25:17.607857  9794 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.598814  9877 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.126s
W20260709 12:25:17.717017  9791 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.677s	user 0.028s	sys 0.063s
I20260709 12:25:17.637480  9854 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 34.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.087s
I20260709 12:25:17.683244  9868 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.702534  9881 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:17.766019  9548 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:17.766460  9548 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:17.767638  9868 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:17.745844 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:17.750921 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:17.715734  9834 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:17.754441 31753 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 30 index: 16. (term mismatch)
I20260709 12:25:17.728102 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:17.665339  9748 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.075s
I20260709 12:25:17.755981 31748 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
I20260709 12:25:17.774925 31753 pending_rounds.cc:85] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 15
I20260709 12:25:17.775242 31753 pending_rounds.cc:107] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 29.16
I20260709 12:25:17.727424  9679 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:17.729439  9823 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:17.703843  9775 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.046s
I20260709 12:25:17.704303  9684 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.046s
I20260709 12:25:17.794653 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:17.754899  9765 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.762571  9592 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:17.767323  9548 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.805621 31749 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 17. (term mismatch)
I20260709 12:25:17.806243 31749 pending_rounds.cc:85] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 16
I20260709 12:25:17.806452 31749 pending_rounds.cc:107] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 31.17
I20260709 12:25:17.806694 31749 raft_consensus.cc:2889] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:17.735839  9616 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.775518 31753 raft_consensus.cc:2889] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:17.776078 31824 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 36 index: 24. (index mismatch)
I20260709 12:25:17.776576 31822 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:17.776834 31820 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 35. Current term is 36. Ops: [35.20-35.20]
W20260709 12:25:17.749430 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 73228us
I20260709 12:25:17.778236 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:17.779832 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:17.831777 31826 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 28: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:17.780476 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:17.781625 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:17.782835 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:17.785627 31819 raft_consensus.cc:1240] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33. Ops: [32.20-32.20]
I20260709 12:25:17.789098 32372 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:25:17.793236 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:17.720305  9857 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.038s
I20260709 12:25:17.718300  9611 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.715518  9824 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.048s
I20260709 12:25:17.810105 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:17.748134  9803 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.818674 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:17.743844  9721 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:17.871838 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 53340us
I20260709 12:25:17.833946 31750 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 36.
I20260709 12:25:17.834919 32378 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 36: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:17.836823 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:17.783797  9679 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:17.784469  9834 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.837435 31818 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 31: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:25:17.837203 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 86354.4us
I20260709 12:25:17.789395  9770 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:17.794617  9586 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:17.844028 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1494) took 1779 ms. Trace:
I20260709 12:25:17.770332  9826 consensus_queue.cc:260] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.789543  9823 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:17.802871  9905 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.287s	user 0.001s	sys 0.000s
I20260709 12:25:17.742357  9865 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:17.791584  9881 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:17.800582  9684 consensus_queue.cc:1243] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 34.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
W20260709 12:25:17.775499 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47485.6us
I20260709 12:25:17.850354  9548 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:17.856422  9794 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:17.948880 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:17.858266 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:17.859046 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:17.859933  9902 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.233s	user 0.001s	sys 0.000s
I20260709 12:25:17.860088  9765 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:17.755132  9757 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.899387  9834 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:17.896320 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:17.893946 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:17.985915 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 92147.4us
I20260709 12:25:17.886765 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:17.908934 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:17.928160  9896 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.079s	user 0.001s	sys 0.000s
I20260709 12:25:17.912142 31752 rpcz_store.cc:276] 0709 12:25:16.064184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:16.065998 (+  1814us) service_pool.cc:224] Handling call
0709 12:25:16.118745 (+ 52747us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:16.119126 (+   381us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:16.219787 (+100661us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.543722 (+1323935us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.639744 (+ 96022us) spinlock_profiling.cc:243] Waited 45.9 ms on lock 0x7b2c00001158. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fb6cea11 00007fe8fb6de372 00007fe8fce10a01 00007fe8fcdd543a 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2
0709 12:25:17.650001 (+ 10257us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.652069 (+  2068us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:17.747186 (+ 95117us) log.cc:844] Serialized 14 byte log entry
0709 12:25:17.747457 (+   271us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.747539 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:17.843745 (+ 96206us) raft_consensus.cc:1685] finished
0709 12:25:17.843780 (+    35us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:17.843956 (+   176us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1152569,"prepare.queue_time_us":169619,"prepare.run_cpu_time_us":63,"prepare.run_wall_time_us":68,"raft.queue_time_us":1160272,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":26264,"spinlock_wait_cycles":105506048,"thread_start_us":211571,"threads_started":2,"wal-append.queue_time_us":570856}
W20260709 12:25:17.945510 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:17.914454  9790 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.611s	user 0.004s	sys 0.007s
I20260709 12:25:17.802507  9745 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:17.999265  9757 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:17.990343  9734 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.000264  9734 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:18.004045  9865 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:18.005053 31751 raft_consensus.cc:606] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Received request to transfer leadership to TS 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:18.005502 31751 raft_consensus.cc:614] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260709 12:25:17.988235 31750 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:17.836373  9809 consensus_queue.cc:1175] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "039d66a6c85e43c3936bf5017501b344" responder_term: 34 status { last_received { term: 34 index: 20 } last_committed_idx: 20 last_received_current_leader { term: 34 index: 20 } }
I20260709 12:25:17.986829 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:17.917727  9586 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:17.901094  9792 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:17.987339 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 100668us
I20260709 12:25:17.924937  9823 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:17.996611  9848 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:17.988994  9812 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:18.023383 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:18.028050  9799 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:18.029793  9721 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:17.952904  9764 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:17.996068  9906 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.078s	user 0.000s	sys 0.001s
I20260709 12:25:17.919661  9656 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.090s
I20260709 12:25:18.002461  9734 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:18.025583 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:17.985438  9794 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:17.939348  9842 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:17.993129  9775 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.057736 31643 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:18.058874 32378 pending_rounds.cc:85] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 21
I20260709 12:25:18.065765 32378 pending_rounds.cc:107] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 34.22
I20260709 12:25:18.066212 32378 raft_consensus.cc:2889] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:18.058614 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:18.009914  9851 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:18.067013 32370 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:25:18.024688  9586 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:18.069285 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:18.079838 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:17.923480  9772 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:18.072118 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:18.089584 31669 raft_consensus.cc:1240] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 26. Current term is 27. Ops: [26.18-26.18]
I20260709 12:25:18.089519 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:18.069757  9862 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:18.090927  9857 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:18.092324  9881 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.16, Last appended by leader: 16, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:18.106997  9912 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.167s	user 0.001s	sys 0.000s
W20260709 12:25:18.108075  9904 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.197s	user 0.001s	sys 0.000s
I20260709 12:25:18.111202  9812 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:18.081797  9794 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:18.113932  9794 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:18.111804  9812 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:18.114221  9611 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:18.119827 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:18.083686  9854 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:17.994642  9592 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:18.129020 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:18.164515 31752 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:18.051993  9520 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.056s
I20260709 12:25:18.085597  9734 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:25:18.134825 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 218465us
I20260709 12:25:18.090351  9772 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:18.050727  9764 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:18.085887  9771 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:18.084096  9775 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:18.123644  9721 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:18.131963 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:18.160403  9862 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:18.174984 31750 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:18.173164  9734 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:18.175874  9916 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.116s	user 0.000s	sys 0.000s
I20260709 12:25:18.083473  9851 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:18.133257  9857 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
W20260709 12:25:18.181852 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.161087 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:18.176982  9873 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.682s	user 0.011s	sys 0.000s
I20260709 12:25:18.177361  9772 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:18.202283 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:18.192337 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 60469.4us
I20260709 12:25:18.210260 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:18.198096 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:18.199004 31747 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:25:18.161949  9656 consensus_queue.cc:1059] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.035s
I20260709 12:25:18.211418 31750 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:25:18.213864 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:18.220584 31752 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:18.196964  9734 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:18.193637  9520 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:18.240794 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:18.200937  9684 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:18.254686 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:18.257241 31822 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: [28.16-28.16]
I20260709 12:25:18.243680  9803 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:18.257329 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:18.248299 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:18.262079 31824 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 30 index: 16. (index mismatch)
I20260709 12:25:18.263619 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:18.264499 31824 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:18.241000  9873 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.028s
I20260709 12:25:18.269359 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:18.212265  9862 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:18.270272 31827 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 36: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:18.197019  9721 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:18.249444  9500 consensus_meta.cc:287] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.731s	user 0.013s	sys 0.000s
I20260709 12:25:18.248482  9520 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:18.273713  9907 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:25:18.281527  9915 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.330s	user 0.001s	sys 0.000s
I20260709 12:25:18.298389 31750 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:18.284708  9854 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:18.300395 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:18.304488 31750 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
W20260709 12:25:18.312439  9920 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.153s	user 0.001s	sys 0.000s
I20260709 12:25:18.299497  9660 raft_consensus.cc:3055] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:18.323807 31752 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
W20260709 12:25:18.325106 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.513s	user 0.006s	sys 0.010s
W20260709 12:25:18.325453 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.513s	user 0.006s	sys 0.010s
I20260709 12:25:18.326287 31752 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:18.332901 31750 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 27 index: 14. Preceding OpId from leader: term: 28 index: 15. (index mismatch)
W20260709 12:25:18.305269  9925 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:25:18.275722  9684 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:18.324751  9660 raft_consensus.cc:740] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:18.330140  9771 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:18.329746  9842 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:18.329743  9854 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 36 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:18.436414  9757 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:18.405215 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60069.5us
I20260709 12:25:18.349839  9922 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:18.445747 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:18.396188  9748 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.569s	user 0.007s	sys 0.012s
I20260709 12:25:18.480724 32373 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:18.458194 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:18.540117 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.557413 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:18.587702 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:18.704079 32374 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 21. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:25:18.760643 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:18.407735  9860 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:18.406688  9500 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.433585  9180 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.781056 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1857) took 1153 ms. Trace:
W20260709 12:25:18.379654 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 177434us
W20260709 12:25:18.704665  9915 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.325s	user 0.000s	sys 0.000s
I20260709 12:25:18.424314  9771 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:18.825157 31823 rpcz_store.cc:276] 0709 12:25:17.627390 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.629592 (+  2202us) service_pool.cc:224] Handling call
0709 12:25:17.636951 (+  7359us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:17.637283 (+   332us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:17.637312 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.643813 (+  6501us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.911703 (+267890us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b180046ccc0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:25:18.034045 (+122342us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:25:18.034103 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.034208 (+   105us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:18.705691 (+671483us) raft_consensus.cc:1685] finished
0709 12:25:18.758071 (+ 52380us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:18.780862 (+ 22791us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7542,"raft.run_cpu_time_us":169,"raft.run_wall_time_us":29836,"spinlock_wait_cycles":36117376,"thread_start_us":122083,"threads_started":1,"wal-append.queue_time_us":535397}
I20260709 12:25:18.400785  9360 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:18.336297  9520 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:18.804463 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:18.843003 31750 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
W20260709 12:25:18.810307 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 253032us
I20260709 12:25:18.844467 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:18.705142  9775 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:18.811559  9748 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.996s	user 0.014s	sys 0.019s
I20260709 12:25:18.832865 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 23 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:18.815186  9721 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:18.833134  9180 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:18.829088  9881 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.811017  9922 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:18.812044  9660 consensus_queue.cc:260] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:18.810276  9764 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:25:18.813081  9822 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:18.826400  9851 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:18.847904  9520 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:18.829413  9500 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:18.870697  9755 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:18.855259 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:18.872315 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:18.888625 32370 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.025s	sys 0.036s
I20260709 12:25:18.873705 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 24 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:18.813347  9860 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:25:18.831585  9747 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.042s	user 0.015s	sys 0.043s
W20260709 12:25:18.876603 31825 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.605s	user 0.027s	sys 0.033s
W20260709 12:25:18.877274 31824 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.583s	user 0.050s	sys 0.084s
W20260709 12:25:18.859768  9927 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.304s	user 0.000s	sys 0.000s
W20260709 12:25:18.895962 31825 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.624s	user 0.027s	sys 0.034s
I20260709 12:25:18.807221  9757 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:18.858836  9935 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.427s	user 0.001s	sys 0.000s
W20260709 12:25:18.848505  9921 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.513s	user 0.000s	sys 0.001s
W20260709 12:25:18.849316  9941 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
W20260709 12:25:18.850205  9923 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.412s	user 0.000s	sys 0.001s
I20260709 12:25:18.895044  9860 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:18.894881  9851 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:18.847354  9834 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.835s	user 0.012s	sys 0.000s
W20260709 12:25:18.883178  9871 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.591s	user 0.029s	sys 0.049s
W20260709 12:25:18.890463 32370 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.025s	sys 0.036s
I20260709 12:25:18.843497  9283 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:18.893869  9180 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:18.861611  9771 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:18.898463 31824 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.608s	user 0.050s	sys 0.084s
W20260709 12:25:18.900751 31744 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.570s	user 0.013s	sys 0.003s
W20260709 12:25:18.901423 31827 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.578s	user 0.001s	sys 0.015s
I20260709 12:25:18.880548  9721 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.945791  9721 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:18.947196  9922 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:18.948089  9721 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:18.903057 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:18.903642  9822 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.949090  9939 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.949601  9822 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:18.949801  9939 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:18.891069  9675 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.132s	user 0.009s	sys 0.005s
W20260709 12:25:18.871641  9656 consensus_queue.cc:1175] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "4d48c83c761f48079bc6ea176649dda0" responder_term: 29 status { last_received { term: 26 index: 15 } last_committed_idx: 15 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: [28.16-28.16]" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:18.905941 32372 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:25:18.881489  9881 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:18.864948  9854 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:18.908989 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:18.848078  9360 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:18.972180  9283 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:18.864555  9857 consensus_queue.cc:1059] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
I20260709 12:25:18.824715  9929 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.087s
I20260709 12:25:18.903841  9940 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader election lost for term 24. Reason: could not achieve majority
W20260709 12:25:18.887061  9853 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 1.076s	user 0.025s	sys 0.041s
W20260709 12:25:18.900494  9747 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.112s	user 0.015s	sys 0.043s
W20260709 12:25:18.914899  9812 consensus_meta.cc:287] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.800s	user 0.009s	sys 0.004s
I20260709 12:25:18.938890  9180 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:18.914093  9660 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:18.942834 31827 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.619s	user 0.001s	sys 0.015s
W20260709 12:25:18.943102 31744 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.613s	user 0.013s	sys 0.003s
I20260709 12:25:18.950871  9771 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:18.951965  9822 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:18.949214 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 46266.7us
I20260709 12:25:18.880769  9803 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:18.949601  9684 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:18.936422  9873 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:18.954358 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1503) took 1270 ms. Trace:
I20260709 12:25:18.955830  9500 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.022908 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:18.965559  9823 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.910s	user 0.011s	sys 0.000s
W20260709 12:25:18.966378  9586 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.872s	user 0.010s	sys 0.001s
I20260709 12:25:18.933593  9834 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:18.974666  9943 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:18.954207  9936 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:25:18.993556 31750 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 23. Current term is 24. Ops: []
I20260709 12:25:18.980669 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:18.985185  9283 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:18.987653  9180 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.023826  9757 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:18.934028  9871 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.649s	user 0.030s	sys 0.050s
I20260709 12:25:19.023557 31748 rpcz_store.cc:276] 0709 12:25:17.683916 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.694417 (+ 10501us) service_pool.cc:224] Handling call
0709 12:25:18.954139 (+1259722us) spinlock_profiling.cc:243] Waited 18.8 ms on lock 0x7b54000b90d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:18.954264 (+   125us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":43147136}
W20260709 12:25:18.994112  9923 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.075s	user 0.000s	sys 0.001s
W20260709 12:25:18.996481  9679 consensus_meta.cc:287] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.077s	user 0.010s	sys 0.004s
W20260709 12:25:18.987138  9853 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.189s	user 0.025s	sys 0.042s
I20260709 12:25:19.032821 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:18.987533  9812 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:18.981545  9772 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.743s	user 0.009s	sys 0.004s
I20260709 12:25:18.976763  9675 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election lost for term 30. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31.
I20260709 12:25:18.984586  9360 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:18.987012 31675 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 3103) took 2094 ms (client timeout 2182 ms). Trace:
I20260709 12:25:18.985284  9857 raft_consensus.cc:3055] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:19.022302  9803 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:18.978832  9881 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:19.032387  9940 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:25:19.047587 31752 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.702s	user 0.013s	sys 0.028s
W20260709 12:25:19.077699 31752 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.732s	user 0.013s	sys 0.029s
I20260709 12:25:18.937223  9851 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.030256  9751 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 32.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.013s
I20260709 12:25:19.059918  9180 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:19.062983 31669 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4174) took 1505 ms. Trace:
I20260709 12:25:19.079962 31669 rpcz_store.cc:276] 0709 12:25:17.557586 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.557708 (+   122us) service_pool.cc:224] Handling call
0709 12:25:17.566887 (+  9179us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.090183 (+523296us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.062679 (+972496us) spinlock_profiling.cc:243] Waited 522 ms on lock 0x7b5400181fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:19.062910 (+   231us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1201521024}
I20260709 12:25:19.062882  9772 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.030408  9873 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.063593  9942 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.044795  9943 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.038168  9586 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.061352  9679 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.057790  9593 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:19.071097 31675 rpcz_store.cc:269] 0709 12:25:16.892914 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.054458 (+161544us) service_pool.cc:224] Handling call
0709 12:25:18.986711 (+1932253us) spinlock_profiling.cc:243] Waited 539 ms on lock 0x7b5400253ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:18.986920 (+   209us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1239297792}
I20260709 12:25:19.034675  9822 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.017s
I20260709 12:25:19.037151  9823 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.059567  9757 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:19.085311 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1417) took 1027 ms. Trace:
I20260709 12:25:19.077107  9803 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:19.074532  9857 raft_consensus.cc:740] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:19.080155  9940 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:19.091799 31747 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 32. Current term is 33. Ops: [32.19-32.19]
I20260709 12:25:19.092151 32378 rpcz_store.cc:276] 0709 12:25:18.057739 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.058247 (+   508us) service_pool.cc:224] Handling call
0709 12:25:18.058613 (+   366us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:18.070397 (+ 11784us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.070428 (+    31us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:18.081174 (+ 10746us) log.cc:844] Serialized 52 byte log entry
0709 12:25:18.404225 (+323051us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:25:18.404286 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.404395 (+   109us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.085054 (+680659us) raft_consensus.cc:1685] finished
0709 12:25:19.085076 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.085235 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1795,"raft.queue_time_us":2635,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":4816,"thread_start_us":314565,"threads_started":1,"wal-append.queue_time_us":744556}
I20260709 12:25:19.102540 31748 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:25:19.101596 31678 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 3112) took 1519 ms. Trace:
I20260709 12:25:19.104961 31670 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:19.105129 31678 rpcz_store.cc:276] 0709 12:25:17.582543 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.634712 (+ 52169us) service_pool.cc:224] Handling call
0709 12:25:17.635348 (+   636us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:17.686944 (+ 51596us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:17.770934 (+ 83990us) log.cc:844] Serialized 14 byte log entry
0709 12:25:18.192722 (+421788us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:18.201864 (+  9142us) log.cc:844] Serialized 52 byte log entry
0709 12:25:18.202062 (+   198us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:18.202111 (+    49us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.202208 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.101321 (+899113us) raft_consensus.cc:1685] finished
0709 12:25:19.101342 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.101508 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":237447,"prepare.queue_time_us":84658,"prepare.run_cpu_time_us":60,"prepare.run_wall_time_us":62,"raft.queue_time_us":63722,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":46219,"thread_start_us":421304,"threads_started":1,"wal-append.queue_time_us":551070}
I20260709 12:25:19.089389  9942 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.102140  9940 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:19.130981  9180 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:19.093595  9593 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.131606  9180 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.100255  9757 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.127990  9771 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:19.125905  9675 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:19.171003 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 327557us
I20260709 12:25:19.175105 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.176085 32378 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:25:19.179273 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.180280 32378 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:19.163003  9500 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.132843  9751 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:19.168248  9945 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.097s	user 0.000s	sys 0.001s
I20260709 12:25:19.193150 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.201277  9180 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:19.162710  9939 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:19.253341  9823 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.255479  9593 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:19.168529  9881 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:19.256358  9881 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.258699  9751 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.259804 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:19.261065  9751 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.261848  9751 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:19.262871 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:19.264983  9819 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:19.267665  9721 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:19.268005  9868 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:19.268285  9721 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.269169  9520 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:19.269543  9520 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:19.270306  9747 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:19.270543  9684 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.270387  9520 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.271166  9684 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.204946  9500 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.273041 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4203) took 1014 ms. Trace:
I20260709 12:25:19.273288  9684 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:19.203099 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4187) took 1599 ms. Trace:
I20260709 12:25:19.284691 31749 rpcz_store.cc:276] 0709 12:25:17.603577 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.603707 (+   130us) service_pool.cc:224] Handling call
0709 12:25:17.604016 (+   309us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:17.807356 (+203340us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.202857 (+1395501us) spinlock_profiling.cc:243] Waited 195 ms on lock 0x7b54002f76d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:19.203035 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":447932672}
I20260709 12:25:19.286155 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:19.286757 31749 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:19.185062  9675 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.210232  9930 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.211159 31674 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 29 index: 16. (term mismatch)
I20260709 12:25:19.223423  9873 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:19.225684  9756 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:19.227152  9586 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.228253  9679 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.230152  9757 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:19.231642  9949 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.095s	user 0.000s	sys 0.000s
I20260709 12:25:19.233470  9907 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:19.235145  9621 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:19.238395  9942 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:19.243870  9616 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.244489  9772 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.245111  9812 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.246737  9851 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:19.249791  9943 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.274032  9939 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:19.259805  9753 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.274714  9500 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:19.274804  9747 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:19.186748  9834 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.275195 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4178) took 1717 ms. Trace:
I20260709 12:25:19.273448 31826 rpcz_store.cc:276] 0709 12:25:18.258574 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.438767 (+180193us) service_pool.cc:224] Handling call
0709 12:25:18.439200 (+   433us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.439667 (+   467us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.439709 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:18.507831 (+ 68122us) log.cc:844] Serialized 26 byte log entry
0709 12:25:18.999144 (+491313us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:18.999211 (+    67us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.999324 (+   113us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.272798 (+273474us) raft_consensus.cc:1685] finished
0709 12:25:19.272818 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.272980 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":68567,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":148689,"thread_start_us":215849,"threads_started":1,"wal-append.queue_time_us":567069}
W20260709 12:25:19.187529  9911 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.269s	user 0.002s	sys 0.008s
I20260709 12:25:19.283742 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4200) took 1026 ms. Trace:
W20260709 12:25:19.210142  9734 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.940s	user 0.009s	sys 0.002s
I20260709 12:25:19.296041 31827 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
I20260709 12:25:19.189291  9771 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.304772 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.302917  9930 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:19.284082  9919 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.122s	user 0.003s	sys 0.010s
I20260709 12:25:19.295233  9871 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:19.295753  9881 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:19.270392  9721 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:19.136453  9857 consensus_queue.cc:260] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.303774  9756 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:19.325181 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.423578 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 1
W20260709 12:25:19.425820 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:19.431376 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.237295 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.239400 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4181) took 1344 ms. Trace:
I20260709 12:25:19.469067 31670 rpcz_store.cc:276] 0709 12:25:17.895378 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.897870 (+  2492us) service_pool.cc:224] Handling call
0709 12:25:17.898238 (+   368us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:19.105426 (+1207188us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.239151 (+133725us) spinlock_profiling.cc:243] Waited 1.21 s on lock 0x7b54001b35d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:19.239311 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2774741888}
I20260709 12:25:19.471026 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.238212 31677 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:19.433177 31671 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 3113) took 1850 ms. Trace:
I20260709 12:25:19.275303 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.476287 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:19.477591 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.298707 31674 pending_rounds.cc:85] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Aborting all ops after (but not including) 15
I20260709 12:25:19.479084 31674 pending_rounds.cc:107] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Aborting uncommitted NO_OP operation due to leader change: 28.16
I20260709 12:25:19.479426 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.479532 31674 raft_consensus.cc:2889] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:19.235877 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.480953 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.481741 32369 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:25:19.482160 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:25:19.306008 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44123.6us
I20260709 12:25:19.280270 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4190) took 1675 ms. Trace:
I20260709 12:25:19.483266 31745 rpcz_store.cc:276] 0709 12:25:17.604912 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.621770 (+ 16858us) service_pool.cc:224] Handling call
0709 12:25:17.665302 (+ 43532us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:17.847216 (+181914us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:17.915525 (+ 68309us) log.cc:844] Serialized 14 byte log entry
0709 12:25:18.283759 (+368234us) log.cc:844] Serialized 14 byte log entry
0709 12:25:18.285472 (+  1713us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:18.285525 (+    53us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:18.285572 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.238575 (+953003us) spinlock_profiling.cc:243] Waited 4.19 ms on lock 0x7b1800104ce0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:19.260817 (+ 22242us) spinlock_profiling.cc:243] Waited 5.53 ms on lock 0x7b1800104ce0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:19.279402 (+ 18585us) spinlock_profiling.cc:243] Waited 1.57 ms on lock 0x7b1800104ce0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9cb62a6 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419
0709 12:25:19.279957 (+   555us) spinlock_profiling.cc:243] Waited 182 ms on lock 0x7b54002959d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:19.280012 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.280200 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":25769,"prepare.queue_time_us":98640,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":99,"raft.queue_time_us":18837,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":15120,"spinlock_wait_cycles":443738368,"thread_start_us":337486,"threads_started":1,"wal-append.queue_time_us":432334}
I20260709 12:25:19.242668 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:19.444572 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4194) took 1716 ms. Trace:
I20260709 12:25:19.488129 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.487206 31744 rpcz_store.cc:276] 0709 12:25:17.728221 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.276480 (+548259us) service_pool.cc:224] Handling call
0709 12:25:18.278738 (+  2258us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.279126 (+   388us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.279155 (+    29us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:18.285290 (+  6135us) log.cc:844] Serialized 26 byte log entry
0709 12:25:18.991493 (+706203us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:19.172463 (+180970us) log.cc:844] Serialized 14 byte log entry
0709 12:25:19.172749 (+   286us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.172857 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.444003 (+271146us) raft_consensus.cc:1685] finished
0709 12:25:19.444037 (+    34us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.444333 (+   296us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1740,"prepare.queue_time_us":269594,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":32,"raft.queue_time_us":909,"raft.run_cpu_time_us":261,"raft.run_wall_time_us":3977,"thread_start_us":802330,"threads_started":2,"wal-append.queue_time_us":804475}
I20260709 12:25:19.240350 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3280) took 1489 ms. Trace:
I20260709 12:25:19.491046 31747 rpcz_store.cc:276] 0709 12:25:17.750685 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.279866 (+529181us) service_pool.cc:224] Handling call
0709 12:25:18.280305 (+   439us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:19.107950 (+827645us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.240128 (+132178us) spinlock_profiling.cc:243] Waited 811 ms on lock 0x7b54001ed2d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:19.240294 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1865432960}
I20260709 12:25:19.237496 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:19.333788 31753 rpcz_store.cc:276] 0709 12:25:17.557406 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.580826 (+ 23420us) service_pool.cc:224] Handling call
0709 12:25:17.581111 (+   285us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:17.828788 (+247677us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.274965 (+1446177us) spinlock_profiling.cc:243] Waited 170 ms on lock 0x7b54002f62d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:19.275133 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":391287552}
I20260709 12:25:19.448168 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:19.329994 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:19.495505 31819 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:19.497572 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:19.319260 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:19.500232 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:19.501338 31822 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:19.336232 31824 rpcz_store.cc:276] 0709 12:25:18.257790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.265384 (+  7594us) service_pool.cc:224] Handling call
0709 12:25:18.266744 (+  1360us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.267533 (+   789us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.267578 (+    45us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:18.277206 (+  9628us) log.cc:844] Serialized 26 byte log entry
0709 12:25:18.942378 (+665172us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:18.942420 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:18.942494 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.283470 (+340976us) raft_consensus.cc:1685] finished
0709 12:25:19.283497 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.283671 (+   174us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":211,"raft.queue_time_us":347,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":7765,"thread_start_us":651503,"threads_started":1,"wal-append.queue_time_us":745210}
I20260709 12:25:19.500236 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:19.503791 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:19.504961 31824 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 35: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:19.506143 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:19.342013 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:19.459779 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:19.461129 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.461023 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:19.294415 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:19.318794 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:19.471220 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.236617 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.474843 31671 rpcz_store.cc:276] 0709 12:25:17.582900 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.643767 (+ 60867us) service_pool.cc:224] Handling call
0709 12:25:17.644112 (+   345us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:17.644464 (+   352us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:17.644495 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.666941 (+ 22446us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.969944 (+303003us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:17.970005 (+    61us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.970105 (+   100us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:18.968055 (+997950us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b4800080058. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe901cfeaa1 00007fe901cf5b37 00007fe901cf59ca 00007fe901ded26f 00007fe901decf37 00007fe901e0155c 00007fe901e01508 00007fe901e01488 00007fe901e01443 00007fe901e006ec 00007fe900d2aa96
0709 12:25:19.283351 (+315296us) spinlock_profiling.cc:243] Waited 4.95 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:19.432307 (+148956us) raft_consensus.cc:1685] finished
0709 12:25:19.432402 (+    95us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.432871 (+   469us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":7406,"raft.run_cpu_time_us":35096,"raft.run_wall_time_us":1316358,"spinlock_wait_cycles":48489600,"thread_start_us":246219,"threads_started":1,"wal-append.queue_time_us":397123}
I20260709 12:25:19.474483 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.349730 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4196) took 1093 ms. Trace:
I20260709 12:25:19.340621 32373 raft_consensus.cc:2368] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 36. Current term is 37.
I20260709 12:25:19.508769 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.508823 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.509166 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.446745 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1504) took 1554 ms. Trace:
I20260709 12:25:19.534042 31818 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 36: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:19.510550 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:19.535987 31675 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 35: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:19.536772 32376 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:19.535245 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:19.537597 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:19.539376 32374 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:19.538331 31752 rpcz_store.cc:276] 0709 12:25:17.892258 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.328688 (+436430us) service_pool.cc:224] Handling call
0709 12:25:18.328938 (+   250us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:18.329130 (+   192us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:25:18.345620 (+ 16490us) log.cc:844] Serialized 14 byte log entry
0709 12:25:19.318245 (+972625us) log.cc:844] Serialized 14 byte log entry
0709 12:25:19.445824 (+127579us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:19.445963 (+   139us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:19.446111 (+   148us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.446235 (+   124us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.446499 (+   264us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":202921,"prepare.run_cpu_time_us":138,"prepare.run_wall_time_us":1724,"raft.queue_time_us":1103,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":2711,"thread_start_us":971819,"threads_started":2,"wal-append.queue_time_us":960057}
I20260709 12:25:19.541265 31823 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:19.542292 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.543062 31824 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:19.544621 31824 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35. Ops: []
W20260709 12:25:19.535300 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.538336 31820 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
I20260709 12:25:19.538151 31825 rpcz_store.cc:276] 0709 12:25:18.256581 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:18.258768 (+  2187us) service_pool.cc:224] Handling call
0709 12:25:18.271289 (+ 12521us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:18.896533 (+625244us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:18.896583 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:18.897235 (+   652us) log.cc:844] Serialized 26 byte log entry
0709 12:25:19.171584 (+274349us) raft_consensus.cc:1645] Marking committed up to 15
0709 12:25:19.171621 (+    37us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.171690 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.349500 (+177810us) raft_consensus.cc:1685] finished
0709 12:25:19.349521 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.349663 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13644,"raft.run_cpu_time_us":126,"raft.run_wall_time_us":3990,"thread_start_us":898488,"threads_started":2,"wal-append.queue_time_us":426180}
I20260709 12:25:19.523692 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:19.537624 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.548621 31671 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 29 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:25:19.510694 31748 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:19.537391 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.545583 31677 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 34: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:19.535385 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:19.524305 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:19.556216 31818 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 29 because replica has last-logged OpId of term: 26 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 25 index: 14.
I20260709 12:25:19.563330 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:19.565199  9453 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:19.567046  9355 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.566910  9283 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:19.568849  9593 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:19.370965  9520 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:19.577656  9611 consensus_queue.cc:1059] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 24.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:19.348996  9675 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.616918  9851 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.366142  9747 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.617627  9851 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.619519  9755 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:19.619678  9862 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: INVALID_TERM, Last received: 23.13, Next index: 14, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:19.354425  9734 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.620824  9851 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:19.621027  9862 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Stepping down as leader of term 23
I20260709 12:25:19.526273 32342 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:19.580219  9621 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:19.300459  9684 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.367698  9756 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.343858  9803 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.347638  9819 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:19.317008  9823 consensus_queue.cc:1059] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 33.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.018s
I20260709 12:25:19.628178  9621 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.635958  9756 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:19.636237  9823 raft_consensus.cc:3055] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:25:19.636924  9823 raft_consensus.cc:740] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:19.321102  9824 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 1.380s	user 0.002s	sys 0.022s
I20260709 12:25:19.622748  9937 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:19.639091  9824 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 1.725s	user 0.003s	sys 0.022s
I20260709 12:25:19.624392  9593 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.630546  9942 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:19.640365  9611 raft_consensus.cc:3055] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:19.357116  9871 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.640877  9611 raft_consensus.cc:740] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 26 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:19.641013  9881 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:19.365795  9857 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:19.320936  9907 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:19.645196  9684 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.355293  9940 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.650236  9940 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:19.530462 31749 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 35.
I20260709 12:25:19.604633 31673 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
I20260709 12:25:19.609050  9753 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:19.609750  9355 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.579875  9656 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.031s
I20260709 12:25:19.610744  9745 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:19.612709  9751 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:19.614318 31748 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:19.615695  9771 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:19.607527 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 56412.9us
W20260709 12:25:19.589146 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 86881.1us
I20260709 12:25:19.628711 31823 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 37.
I20260709 12:25:19.621583  9862 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:19.630110 31825 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 29. Current term is 31. Ops: []
I20260709 12:25:19.628250  9748 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.025s
I20260709 12:25:19.636458 31669 raft_consensus.cc:1240] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:19.663024  9355 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.640108  9942 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.665179  9684 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:19.665622 31676 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43190 (request call id 3110) took 2130 ms. Trace:
I20260709 12:25:19.666461  9942 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:19.345329  9860 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.669410  9860 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.389114  9930 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.641914  9881 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.650604  9803 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.650831  9752 consensus_queue.cc:1059] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 30.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:19.651279 31670 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:19.653087 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.642338  9611 consensus_queue.cc:260] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.548542 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:19.653110 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.638514  9621 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.657387  9871 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.657574  9964 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:19.690057  9611 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:19.632879  9734 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:19.694617  9881 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:19.695658  9940 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:19.696924  9940 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:19.697553  9745 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:19.698015  9771 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:19.698177  9964 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:19.698716  9771 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:19.699726  9964 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.700119  9771 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 34.17, Last appended by leader: 17, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.700902 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.702052  9771 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:19.702662  9862 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 23.13, Last appended by leader: 13, Current term: 23, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.703394 31642 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:19.704559  9862 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:19.704742 31642 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:19.706832  9881 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:19.706995  9930 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:19.666014 31676 rpcz_store.cc:276] 0709 12:25:17.535369 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:17.591489 (+ 56120us) service_pool.cc:224] Handling call
0709 12:25:17.613220 (+ 21731us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:17.640178 (+ 26958us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:17.640211 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:17.646146 (+  5935us) log.cc:844] Serialized 26 byte log entry
0709 12:25:17.717073 (+ 70927us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:17.717136 (+    63us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:17.717244 (+   108us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:19.066694 (+1349450us) spinlock_profiling.cc:243] Waited 6.03 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:19.432395 (+365701us) raft_consensus.cc:1685] finished
0709 12:25:19.432476 (+    81us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:19.665282 (+232806us) spinlock_profiling.cc:243] Waited 3.86 ms on lock 0x7b54002e6b50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:19.665463 (+   181us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":26552,"raft.queue_time_us":27184,"raft.run_cpu_time_us":155,"raft.run_wall_time_us":24920,"spinlock_wait_cycles":22746752,"thread_start_us":70551,"threads_started":1,"wal-append.queue_time_us":137222}
I20260709 12:25:19.698745 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.698110  9745 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:19.713145  9860 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election won for term 35
W20260709 12:25:19.713364 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 165078us
I20260709 12:25:19.713970  9860 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 35 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:19.713864  9745 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.715391 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:19.715991  9745 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:19.715476  9860 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.716666  9824 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:19.717516  9824 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:19.717914  9824 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:19.718258  9824 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:19.719067  9593 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:19.719978  9930 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:19.721489  9930 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.724089 31675 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 31 index: 15. Preceding OpId from leader: term: 35 index: 17. (index mismatch)
W20260709 12:25:19.731807 31672 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.566s	user 0.012s	sys 0.010s
W20260709 12:25:19.732432 31672 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.696s	user 0.013s	sys 0.011s
I20260709 12:25:19.638271  9823 consensus_queue.cc:260] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.734135  9826 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:19.739274 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:19.740082 31753 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 35.
I20260709 12:25:19.741145 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:19.741958 31753 raft_consensus.cc:2393] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 33: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:19.743621 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:19.747545 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.748327 32373 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:19.750240 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.750735 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:19.750773 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.734935 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:19.735649  9593 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:19.735831  9979 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.109s	user 0.000s	sys 0.001s
I20260709 12:25:19.735000  9823 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:25:19.737752  9978 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
I20260709 12:25:19.707682  9881 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.721495 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:19.747473 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:19.765908 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54419.6us
I20260709 12:25:19.768162  9734 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.768450 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:19.769438  9537 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:19.770469 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.772338  9803 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:19.772879  9803 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:19.773300  9803 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:19.768924  9734 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.780063 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:19.790004  9942 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:19.790993  9942 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.793648  9942 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:19.795279 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.796446 31674 raft_consensus.cc:2393] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:25:19.786998 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 76836.4us
I20260709 12:25:19.804576  9537 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:19.808708  9734 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:19.809175  9611 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:19.805994 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.810844 31669 raft_consensus.cc:2410] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 37 because replica has last-logged OpId of term: 36 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:25:19.811833 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:19.808352  9537 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.806236  9593 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:19.820133 32373 raft_consensus.cc:2393] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 33: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:19.820713  9792 raft_consensus.cc:3055] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:19.821117  9593 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:19.821177  9792 raft_consensus.cc:740] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:19.823151  9881 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:19.822877  9792 consensus_queue.cc:260] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.825330 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.826014 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:19.826279  9537 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.830159 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.830874 31670 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 24.
W20260709 12:25:19.831866 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 306071us
I20260709 12:25:19.825362 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:19.836012 31669 raft_consensus.cc:2393] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 32: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:19.835547  9824 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:19.836887 31749 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:19.838886 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:19.840066  9734 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:19.840368 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:19.840778  9734 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:19.828866 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.842808  9824 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:19.841202 31749 raft_consensus.cc:2410] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 27 because replica has last-logged OpId of term: 26 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 24 index: 17.
I20260709 12:25:19.826167 31674 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:25:19.848407 32373 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:25:19.827219 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:19.873433 31748 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 31: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:19.874040 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:19.873986  9881 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:19.874888  9881 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:19.875643 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:19.880950  9792 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:19.881006 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 24 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:19.882730  9907 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:19.886483 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52453.9us
W20260709 12:25:19.890854 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.903929 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:19.904637 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:19.919000  9656 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:19.932763 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 211364us
W20260709 12:25:19.968286 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:19.872677 32343 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:20.001626 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 134936us
I20260709 12:25:20.001030 31751 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:20.005551  9656 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:20.011961  9656 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:20.012367  9734 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:20.015137 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:20.016754 31678 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:20.018553  9734 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:20.018759 31749 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:20.019253  9734 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:20.019837 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:20.020661 31749 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:25:20.021636  9734 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:20.027357  9675 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.028005 31747 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:25:20.028170  9675 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:20.038353  9675 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:20.044581 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:20.047252 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:20.051630 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:19.901604  9881 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:25:20.066008 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:20.066053 31576 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 157) took 30510 ms (client timeout 15000 ms). Trace:
W20260709 12:25:20.079618 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 297976us
I20260709 12:25:20.077718 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:20.086338  9881 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:20.087143  9881 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:20.087826 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58224.4us
I20260709 12:25:20.082343 32370 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:20.103921 31752 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:20.110363 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:20.110026  9865 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:20.113633  9752 raft_consensus.cc:2764] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election decision vote started in defunct term 34: won
W20260709 12:25:20.113888 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 98853.3us
I20260709 12:25:20.093720  9942 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.124306 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:20.089473  9881 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:20.123070 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 59900.3us
I20260709 12:25:20.095067  9803 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:20.137720  9803 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:20.139715  9520 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.140460  9520 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:20.142117 31821 raft_consensus.cc:1275] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:25:20.143615 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:20.144451 31826 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:20.144767 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:20.144460  9812 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.146081  9638 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:25:20.146708  9638 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 37 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:20.146404  9812 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:20.147761  9638 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:20.148772  9812 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:20.114137  9865 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.151849 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:20.153733 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.151113 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:20.150202 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:20.168097 31818 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:25:20.181156 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:20.181912 31824 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:20.181396 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:20.183876  9832 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:20.186364 31744 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 35.
I20260709 12:25:20.172506 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:20.127213  9857 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.192045 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:20.194895 31745 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:20.198175 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:20.149744  9747 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:20.190793 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:20.205359 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:20.206100 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53507us
I20260709 12:25:20.210830 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:20.212126 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 99329.8us
I20260709 12:25:20.213054 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:20.213328  9942 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:20.214349  9757 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:25:20.225548 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47003.1us
I20260709 12:25:20.221330 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:20.227267 32373 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:20.227769 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:20.169072  9865 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:20.228565 31678 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:20.257985 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:20.258755 31678 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
W20260709 12:25:20.284122 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 71219.5us
I20260709 12:25:20.168022  9755 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.091s
I20260709 12:25:20.230105  9520 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:20.297500 31670 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:25:20.285619 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:20.246043  9832 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:20.311640 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:20.304570 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:20.329109  9823 consensus_meta.cc:287] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.552s	user 0.017s	sys 0.003s
W20260709 12:25:20.330036 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 52917.9us
W20260709 12:25:20.330441  9984 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.115s	user 0.001s	sys 0.000s
I20260709 12:25:20.299979  9792 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:20.330852 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:20.332358 31827 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:20.350801  9939 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.351835  9939 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:20.354408 32342 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.194:46639 timed out after 1.713s (SENT) [suppressed 13 similar messages]
W20260709 12:25:20.356009 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 179619us
I20260709 12:25:20.357245 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:20.357976 31827 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:20.358397  9792 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:20.367686  9988 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.273s	user 0.000s	sys 0.000s
I20260709 12:25:20.381901 31746 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:20.383870 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 4210) took 1152 ms. Trace:
I20260709 12:25:20.384409 31746 rpcz_store.cc:276] 0709 12:25:19.231092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.242404 (+ 11312us) service_pool.cc:224] Handling call
0709 12:25:20.383554 (+1141150us) spinlock_profiling.cc:243] Waited 894 ms on lock 0x7b54004051d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:20.383766 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2056987392}
W20260709 12:25:20.386106 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 91505.7us
I20260709 12:25:20.388206 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:20.390156 31567 catalog_manager.cc:5697] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 26 to 29, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 29 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:20.067101 31576 rpcz_store.cc:269] 0709 12:24:49.555085 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:59.853549 (+10298464us) service_pool.cc:224] Handling call
0709 12:25:20.065643 (+20212094us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:10.879143 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:10.879829 (+   686us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:10.879879 (+    50us) write_op.cc:435] Acquired schema lock
0709 12:25:10.879899 (+    20us) tablet.cc:662] Decoding operations
0709 12:25:11.131478 (+251579us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:11.200985 (+ 69507us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:11.209260 (+  8275us) tablet.cc:685] Acquiring locks for 17 operations
0709 12:25:11.231910 (+ 22650us) tablet.cc:701] Row locks acquired
0709 12:25:11.231982 (+    72us) write_op.cc:260] PREPARE: finished
0709 12:25:11.232198 (+   216us) write_op.cc:270] Start()
0709 12:25:11.232390 (+   192us) write_op.cc:276] Timestamp: P: 1783599911232128 usec, L: 0
0709 12:25:11.232434 (+    44us) op_driver.cc:348] REPLICATION: starting
0709 12:25:11.233325 (+   891us) log.cc:844] Serialized 6716 byte log entry
0709 12:25:11.931149 (+697824us) op_driver.cc:464] REPLICATION: finished
0709 12:25:12.267072 (+335923us) write_op.cc:301] APPLY: starting
0709 12:25:12.270728 (+  3656us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:12.401796 (+131068us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:12.401948 (+   152us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:12.584575 (+182627us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:12.585029 (+   454us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=34,key_file_lookups=34,delta_file_lookups=0,mrs_lookups=0
0709 12:25:12.585098 (+    69us) write_op.cc:312] APPLY: finished
0709 12:25:12.585569 (+   471us) log.cc:844] Serialized 156 byte log entry
0709 12:25:12.585780 (+   211us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:12.586114 (+   334us) write_op.cc:454] Released schema lock
0709 12:25:12.586193 (+    79us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:12.621534 (+ 35341us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:12.623228 (+  1694us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":5613817,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":263181,"apply.run_cpu_time_us":26404,"apply.run_wall_time_us":340328,"cfile_cache_hit":70,"cfile_cache_hit_bytes":75153,"num_ops":17,"prepare.queue_time_us":207431,"prepare.run_cpu_time_us":23117,"prepare.run_wall_time_us":556769,"raft.queue_time_us":219547,"raft.run_cpu_time_us":29150,"raft.run_wall_time_us":310429,"replication_time_us":675245,"thread_start_us":599945,"threads_started":4,"wal-append.queue_time_us":231185,"wal-append.run_cpu_time_us":27689,"wal-append.run_wall_time_us":2273827}]]}
I20260709 12:25:20.392161 31567 catalog_manager.cc:5697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 14 to 16, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 16 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:20.394029 31576 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42072 (request call id 247) took 17380 ms (client timeout 15000 ms). Trace:
I20260709 12:25:20.394418 31567 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 19 to 27. New cstate: current_term: 27 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:20.329298  9986 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.119s	user 0.000s	sys 0.001s
I20260709 12:25:20.396411 31826 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:25:20.277491  9734 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:25:20.398272 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 66018.1us
W20260709 12:25:20.394388 31576 rpcz_store.cc:269] 0709 12:25:03.014072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.393779 (+17379707us) service_pool.cc:211] Skipping call since client already timed out
0709 12:25:20.393927 (+   148us) inbound_call.cc:177] Queueing failure response
Metrics: {}
I20260709 12:25:20.405277  9747 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:20.405977 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:20.409735 31824 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 35.
I20260709 12:25:20.318370  9832 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.414109  9832 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:20.404469  9734 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:20.422994  9734 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:20.423377 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:20.424185 31824 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 36.
I20260709 12:25:20.425385 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:20.428467 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:20.429553 31824 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:20.431175 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:20.433190 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:20.435230 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:20.437305 31820 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:25:20.411516  9747 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 35 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:20.444942  9747 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:20.446342 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.433300 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:20.447584 31822 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 35.
I20260709 12:25:20.321844  9862 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:25:20.412978  9826 consensus_meta.cc:287] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.672s	user 0.013s	sys 0.007s
W20260709 12:25:20.415364  9771 consensus_meta.cc:287] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.713s	user 0.009s	sys 0.009s
I20260709 12:25:20.307098  9857 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:20.462898  9942 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:20.469357 32375 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
W20260709 12:25:20.460948  9756 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.824s	user 0.014s	sys 0.009s
I20260709 12:25:20.476189 32373 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:20.478034 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:20.460316  9907 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:20.476280  9413 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.486593  9862 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:20.500979  9942 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:20.539880 31567 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 21 to 26, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 26 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:20.526237  9453 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:20.521087  9413 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:20.548190 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:20.527659  9939 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:20.556227 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 89085.8us
I20260709 12:25:20.558705 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:20.541599  9942 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:20.567497 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:20.564798 31753 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:25:20.534472  9755 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.034s
I20260709 12:25:20.573654 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:20.579290 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:20.569905  9942 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:20.582415 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 28 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:20.573467 31567 catalog_manager.cc:5697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 19 to 26. New cstate: current_term: 26 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:20.590641 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54520.4us
I20260709 12:25:20.593007 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:20.594592 31746 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 37
I20260709 12:25:20.592672  9771 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.597909 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:20.598470 31792 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:20.603870 31753 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:20.594657  9453 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:20.608177 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:20.610252 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 255878us
I20260709 12:25:20.606863 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:20.629894 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.610985  9453 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.597903  9771 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:20.638862 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 233041us
I20260709 12:25:20.611683  9857 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:20.605804  9907 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.027s
I20260709 12:25:20.636075 31567 catalog_manager.cc:5697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 19 to 29, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 29 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:20.650585 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:20.633796  9520 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:20.648100 31744 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:25:20.603199  9757 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:20.637580  9453 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:20.660118 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:20.658217 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 24 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:20.667225 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:20.667353 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:20.668413 31752 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
I20260709 12:25:20.668828 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:20.646080  9964 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.028s
I20260709 12:25:20.654917  9520 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.677107  9520 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:20.678758 31753 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 24: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:20.679697 31751 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:25:20.682940  9771 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:20.694123 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4220) took 1041 ms. Trace:
I20260709 12:25:20.694717 31748 rpcz_store.cc:276] 0709 12:25:19.652719 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.932389 (+279670us) service_pool.cc:224] Handling call
0709 12:25:20.028959 (+ 96570us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:20.120760 (+ 91801us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:20.120803 (+    43us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:20.149963 (+ 29160us) log.cc:844] Serialized 26 byte log entry
0709 12:25:20.466946 (+316983us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:20.467002 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.467087 (+    85us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:20.693855 (+226768us) raft_consensus.cc:1685] finished
0709 12:25:20.693884 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.694044 (+   160us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":139074,"raft.queue_time_us":85771,"raft.run_cpu_time_us":244,"raft.run_wall_time_us":12616,"thread_start_us":316477,"threads_started":1,"wal-append.queue_time_us":461406}
I20260709 12:25:20.697214  9998 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:20.697834  9998 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.697956  9537 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:20.698315  9998 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:20.698693  9537 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:20.690675 31746 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 37.
I20260709 12:25:20.701886  9537 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:20.702709  9748 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:20.654538  9862 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:20.711894 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:20.714911 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.718171 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4221) took 1065 ms. Trace:
I20260709 12:25:20.718786 31749 rpcz_store.cc:276] 0709 12:25:19.653035 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.021293 (+368258us) service_pool.cc:224] Handling call
0709 12:25:20.021683 (+   390us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:20.133586 (+111903us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:20.133625 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:20.188234 (+ 54609us) log.cc:844] Serialized 52 byte log entry
0709 12:25:20.469987 (+281753us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:20.470034 (+    47us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.470105 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:20.717866 (+247761us) raft_consensus.cc:1685] finished
0709 12:25:20.717895 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.718095 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":105892,"raft.queue_time_us":133652,"raft.run_cpu_time_us":145,"raft.run_wall_time_us":8350,"thread_start_us":281282,"threads_started":1,"wal-append.queue_time_us":427653}
I20260709 12:25:20.721127 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4224) took 1021 ms. Trace:
I20260709 12:25:20.721589 31747 rpcz_store.cc:276] 0709 12:25:19.699624 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.036907 (+337283us) service_pool.cc:224] Handling call
0709 12:25:20.037357 (+   450us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:20.189448 (+152091us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:20.189484 (+    36us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:20.285613 (+ 96129us) log.cc:844] Serialized 26 byte log entry
0709 12:25:20.652138 (+366525us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:20.652184 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.652267 (+    83us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:20.720872 (+ 68605us) raft_consensus.cc:1685] finished
0709 12:25:20.720893 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.721063 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":121288,"raft.queue_time_us":129456,"raft.run_cpu_time_us":237,"raft.run_wall_time_us":7704,"spinlock_wait_cycles":71552,"thread_start_us":239703,"threads_started":1,"wal-append.queue_time_us":320991}
W20260709 12:25:20.727269 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 77517.4us
I20260709 12:25:20.703225  9757 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:20.738698 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.741328  9832 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:20.742909 32377 raft_consensus.cc:2368] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30.
I20260709 12:25:20.743934  9413 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:20.745105 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4213) took 1311 ms. Trace:
I20260709 12:25:20.745282  9756 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:20.745697  9756 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:20.745517 31750 rpcz_store.cc:276] 0709 12:25:19.434323 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.616551 (+182228us) service_pool.cc:224] Handling call
0709 12:25:19.643501 (+ 26950us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:19.744405 (+100904us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:19.744444 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:19.745070 (+   626us) log.cc:844] Serialized 26 byte log entry
0709 12:25:19.801536 (+ 56466us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:19.801602 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:19.801705 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:20.734283 (+932578us) spinlock_profiling.cc:243] Waited 8.28 ms on lock 0x7b24002a9ae8. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9b6e9c4 00007fe8f9b6cacb 00007fe8f9d27dc9 00007fe8f9d27a55 00007fe8f9d26fb0 00007fe8fc99d488 00007fe8fc99dbe8 00007fe8fc975398 00007fe8fc97076a 00007fe8fc96fbb6
0709 12:25:20.744679 (+ 10396us) raft_consensus.cc:1685] finished
0709 12:25:20.744702 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.744936 (+   234us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":60668,"raft.queue_time_us":65376,"raft.run_cpu_time_us":86,"raft.run_wall_time_us":86,"spinlock_wait_cycles":19053952,"thread_start_us":55916,"threads_started":1,"wal-append.queue_time_us":693712}
I20260709 12:25:20.746713  9756 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:20.653932  9755 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.064s
I20260709 12:25:20.690158  9857 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:20.778180 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:20.783691 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 134079us
W20260709 12:25:20.786345 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 135740us
I20260709 12:25:20.771365  9756 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:20.789661  9453 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:20.791484 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:20.792630  9748 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:20.793170 31567 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 18 to 24, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 24 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:20.797294 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 86426.8us
I20260709 12:25:20.808140  9453 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:20.754580  9998 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:20.804751 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:20.815167  9823 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.820286  9823 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:20.826649  9907 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:20.827401  9907 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:20.828527  9823 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:20.829516  9907 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:20.830102  9907 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:20.796927  9748 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:20.838449 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:20.832394 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:20.831029 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:20.831399 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:20.831650 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:20.834785 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:20.801857  9755 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:20.805126 10001 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:20.846288 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:20.842038 31567 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 23 to 26, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 26 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:20.823318  9453 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:20.854015 31670 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:20.856539 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51088.7us
I20260709 12:25:20.858119  9748 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:20.803339  9862 consensus_queue.cc:1048] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.039s
I20260709 12:25:20.859711 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:20.866958 31671 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 36.
W20260709 12:25:20.862964 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 71600.1us
W20260709 12:25:20.819367  9865 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.533s	user 0.009s	sys 0.003s
I20260709 12:25:20.861755 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:20.878118 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 73460.6us
I20260709 12:25:20.846655 10001 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.830600  9907 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:20.901801 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.902448 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:20.903893 31718 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:20.897143  9586 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:20.903468 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:20.864882  9520 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:20.889056  9832 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:20.875901  9453 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:20.933961 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:20.901388 10001 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:20.915076  9857 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:20.915307  9586 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:20.945407 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 358961us
I20260709 12:25:20.928383  9823 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:20.949143 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4206) took 1054 ms. Trace:
I20260709 12:25:20.909312  9755 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.034s
I20260709 12:25:20.931299  9792 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:20.950793  9823 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:20.965613  9823 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:20.966926  9453 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:20.969051 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 72204.9us
I20260709 12:25:20.950901 31825 rpcz_store.cc:276] 0709 12:25:19.878830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.901182 (+ 22352us) service_pool.cc:224] Handling call
0709 12:25:20.000254 (+ 99072us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.123200 (+122946us) raft_consensus.cc:1534] Early marking committed up to index 13
0709 12:25:20.180194 (+ 56994us) log.cc:844] Serialized 14 byte log entry
0709 12:25:20.932983 (+752789us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:20.933229 (+   246us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:20.933300 (+    71us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.933411 (+   111us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:20.933579 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":45228,"prepare.run_cpu_time_us":65,"prepare.run_wall_time_us":85,"thread_start_us":732281,"threads_started":2,"wal-append.queue_time_us":578733}
I20260709 12:25:20.896214  9865 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:20.963749 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 59907.9us
I20260709 12:25:20.948884  9586 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:20.971558 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:20.972235 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 110592us
I20260709 12:25:20.955740  9771 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:20.973332  9991 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.382s	user 0.000s	sys 0.005s
I20260709 12:25:20.970221 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:20.956826  9748 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:20.984723 31747 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 33 index: 20. (term mismatch)
I20260709 12:25:20.985181 31747 pending_rounds.cc:85] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 19
I20260709 12:25:20.985417 31747 pending_rounds.cc:107] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 32.20
I20260709 12:25:20.985643 31747 raft_consensus.cc:2889] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:20.990823 31750 raft_consensus.cc:1240] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:20.982303  9792 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:20.988169  9998 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:21.005920 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:21.016402  9453 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:21.017053  9453 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:20.993005  9771 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.018039  9453 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.006489  9792 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.034534  9757 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.052946 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:21.037346  9998 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:25:21.064220 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.057034  9998 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.059844  9757 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.086227 31820 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 35 index: 23. (index mismatch)
I20260709 12:25:21.058000  9520 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:21.056612  9907 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:21.096076 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:21.101284 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:21.102125 31820 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 30
I20260709 12:25:21.090584  9998 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:21.099803 31821 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:25:21.093848  9862 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:21.117714 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 64878.2us
I20260709 12:25:21.104736  9865 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:21.091607  9823 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.135061 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:21.135875 32373 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:21.139797 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:21.124482  9881 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:21.145355  9792 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:21.148051 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:21.131678  9520 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:21.150563 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.153429 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:21.154322 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:21.154950  9755 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:21.161820 32375 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 1424) took 1689 ms (client timeout 1769 ms). Trace:
I20260709 12:25:21.166162 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:21.174592 31671 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
I20260709 12:25:21.171906 32373 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:21.176885 32372 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:25:21.174226 32375 rpcz_store.cc:269] 0709 12:25:19.472936 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:19.508055 (+ 35119us) service_pool.cc:224] Handling call
0709 12:25:21.161532 (+1653477us) spinlock_profiling.cc:243] Waited 935 ms on lock 0x7b540031d654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:21.161703 (+   171us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2151677824}
I20260709 12:25:21.179288  9755 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.181857  9755 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:21.185081 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 415387us
W20260709 12:25:21.185279 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 186810us
W20260709 12:25:21.167449  9942 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.552s	user 0.010s	sys 0.004s
I20260709 12:25:21.128131  9862 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.191452 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:21.192848 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:21.193718 31669 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:25:21.199513 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:21.200462 31671 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:25:21.203106 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:21.197901  9823 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:21.205421 31717 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:21.209810 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:21.210668 31670 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:25:21.205761  9734 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.779s	user 0.014s	sys 0.003s
I20260709 12:25:21.191107  9942 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.223392 32378 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1429) took 1030 ms. Trace:
I20260709 12:25:21.223913  9413 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:21.224478  9283 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:21.227661 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 257541us
W20260709 12:25:21.228360 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 148579us
W20260709 12:25:21.247462 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 273994us
I20260709 12:25:21.244670 32378 rpcz_store.cc:276] 0709 12:25:20.193343 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.193477 (+   134us) service_pool.cc:224] Handling call
0709 12:25:20.221030 (+ 27553us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:20.329920 (+108890us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:25:20.647817 (+317897us) log.cc:844] Serialized 14 byte log entry
0709 12:25:20.932411 (+284594us) log.cc:844] Serialized 14 byte log entry
0709 12:25:21.077815 (+145404us) spinlock_profiling.cc:243] Waited 14.8 ms on lock 0x7b18004b9f40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8522 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:21.222753 (+144938us) spinlock_profiling.cc:243] Waited 4.1 ms on lock 0x7b18004b9f40. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:25:21.222921 (+   168us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:21.222967 (+    46us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:25:21.223005 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.223074 (+    69us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:21.223191 (+   117us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":12118,"prepare.queue_time_us":455206,"prepare.run_cpu_time_us":119,"prepare.run_wall_time_us":2240,"raft.queue_time_us":25738,"raft.run_wall_time_us":4603,"spinlock_wait_cycles":71651968,"thread_start_us":532459,"threads_started":2,"wal-append.queue_time_us":448500}
I20260709 12:25:21.250083 10016 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:21.255985 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:21.253417  9865 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:21.254199 31820 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:21.191244  9520 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.258745 31821 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:21.261785 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:21.264986 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.253443  9930 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.283252 31820 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:21.280160 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:21.296049 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 26 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:21.297159 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:21.297791 31824 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:21.246248  9734 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.310155 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:21.319913 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:21.273239 10016 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:21.304251  9537 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:21.325940 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:21.336603 31820 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:21.302254  9862 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:21.338282 31752 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:25:21.338979 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:21.339636 31827 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:21.340891 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:21.262372  9283 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.344646 31746 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:25:21.346800 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:21.350010 31820 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:21.300989  9593 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.033s
I20260709 12:25:21.348217 31818 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:21.346951 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:21.359464 31748 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:21.341817 31747 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:21.313840  9865 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:25:21.363613 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 65329.8us
I20260709 12:25:21.351456 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:21.351684 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:21.352341 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:21.346975 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 217461us
I20260709 12:25:21.355192 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:21.367592 31820 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 31: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:21.365281 31824 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 36.
W20260709 12:25:21.361555 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 107668us
I20260709 12:25:21.323257  9757 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:21.322899  9771 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:21.371945 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:21.366537 31744 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
W20260709 12:25:21.377696 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.374069 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:21.336621  9537 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:21.338807 10013 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:21.392161 31820 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 33 index: 20. (index mismatch)
I20260709 12:25:21.378419 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:21.380074 32343 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:21.313409  9930 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:21.408154 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 128080us
I20260709 12:25:21.363335  9865 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.364099  9803 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:21.395294 31827 raft_consensus.cc:3055] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:21.412730 31827 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:21.409336 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:21.379607  9413 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.410600 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:21.420370 31717 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.423622 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:21.396390  9537 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:21.395798 10013 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.425649 31820 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:21.389705  9771 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:21.411489  9862 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:25:21.422214  9992 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.894s	user 0.000s	sys 0.005s
I20260709 12:25:21.412365  9865 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:21.456657 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 82696.6us
I20260709 12:25:21.408525  9860 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.023s
W20260709 12:25:21.440558  9756 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.645s	user 0.014s	sys 0.000s
W20260709 12:25:21.465245 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 48230.6us
I20260709 12:25:21.451071  9413 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.473326  9413 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:21.474484  9413 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:21.480181  9734 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:21.481786  9747 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:21.482331  9747 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:21.482698 10010 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:21.483325  9747 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.484558  9747 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:21.440068  9964 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.055s
I20260709 12:25:21.496328  9520 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:21.453301 10016 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.460768  9862 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:21.498844 10014 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:21.501914 10014 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.502216  9862 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.504540 10014 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:21.508147 10012 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.114s	user 0.002s	sys 0.000s
I20260709 12:25:21.434232  9771 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:21.511010 10017 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.103s	user 0.000s	sys 0.002s
I20260709 12:25:21.483855  9930 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:21.429749 10013 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:21.499464 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 203201us
I20260709 12:25:21.471361 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:21.517819 32343 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:21.519192 32373 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:25:21.480329 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4227) took 1054 ms. Trace:
W20260709 12:25:21.521494  9748 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.532s	user 0.011s	sys 0.002s
I20260709 12:25:21.460644 31827 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.516396 31670 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 37.
W20260709 12:25:21.521775 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 57011.8us
I20260709 12:25:21.452320  9942 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:21.520239 31818 rpcz_store.cc:276] 0709 12:25:20.425632 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.427597 (+  1965us) service_pool.cc:224] Handling call
0709 12:25:20.427919 (+   322us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.370900 (+942981us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:21.480070 (+109170us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x7b54004470d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:21.480263 (+   193us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2116008192}
I20260709 12:25:21.456562 10015 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:21.583772 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4209) took 1455 ms. Trace:
W20260709 12:25:21.524338 10001 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.555s	user 0.012s	sys 0.002s
I20260709 12:25:21.597366 10001 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.597990  9771 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:21.574821 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:21.600087 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 76570.3us
I20260709 12:25:21.601514 31825 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:21.584337 31823 rpcz_store.cc:276] 0709 12:25:20.130844 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.139011 (+  8167us) service_pool.cc:224] Handling call
0709 12:25:20.139478 (+   467us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:20.139956 (+   478us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:20.139991 (+    35us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:20.140631 (+   640us) log.cc:844] Serialized 26 byte log entry
0709 12:25:20.236836 (+ 96205us) raft_consensus.cc:1645] Marking committed up to 14
0709 12:25:20.236890 (+    54us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:20.236972 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:21.257244 (+1020272us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:21.568619 (+311375us) raft_consensus.cc:1685] finished
0709 12:25:21.568652 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:21.583699 (+ 15047us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3984,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"spinlock_wait_cycles":251965184,"thread_start_us":95711,"threads_started":1,"wal-append.queue_time_us":329064}
W20260709 12:25:21.521695 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 111159us
I20260709 12:25:21.501163  9283 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:21.452737  9586 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.522280  9939 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:21.628813 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:21.598073 10028 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:21.634572 10028 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.634975 10028 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:21.582507 10015 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.639504 31823 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:25:21.643671  9413 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:21.653618  9537 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.635004 10026 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:21.655218 10026 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:21.655628 10026 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:21.622411 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:21.629334 31827 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:21.660027 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:21.601704  9756 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.663616 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:21.664861 32378 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:21.629971  9930 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:21.672634 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50346.8us
I20260709 12:25:21.601274  9832 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:21.677004 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:21.678581 31670 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 27.
I20260709 12:25:21.678699 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:21.659176  9771 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:21.661407 10015 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:21.600853  9748 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:21.680498 32377 raft_consensus.cc:2368] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 33. Current term is 34.
I20260709 12:25:21.641388  9283 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.061s
I20260709 12:25:21.689904  9832 raft_consensus.cc:697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:21.664717  9638 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:21.619609  9593 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 19, Time since last communication: 0.051s
I20260709 12:25:21.684227  9586 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:21.667328  9860 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:21.635630  9734 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.044s
I20260709 12:25:21.690114 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:21.744697 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54670.4us
I20260709 12:25:21.745688 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:21.691195  9857 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 32.19, Next index: 20, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:21.691449 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:21.691946 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:21.692471 10010 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20260709 12:25:21.693612 10001 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:21.661703  9939 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.694084 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:21.694978  9771 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.690053  9812 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:25:21.696779  9998 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.579s	user 0.011s	sys 0.000s
I20260709 12:25:21.700688 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:21.713864 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:21.737854  9832 consensus_queue.cc:237] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.672245  9756 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.749243 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:21.781105 31670 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:21.753413 32374 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:25:21.758121 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:21.758474 32375 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:21.709161  9747 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.764029 31827 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:25:21.747181  9755 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.764917 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:21.763882 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:21.776194 31669 raft_consensus.cc:2393] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 30: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:21.714775  9638 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:21.797511 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.780613 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:21.798756 32377 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 31: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:21.747714  9871 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:21.786180 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:21.793825 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:21.800235 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:21.751363 10030 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:21.713304  9748 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.709609 10016 consensus_queue.cc:1059] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.069s
I20260709 12:25:21.801158  9638 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:25:21.815385 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 117608us
I20260709 12:25:21.822474 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:21.827992 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:21.833325 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 141440us
I20260709 12:25:21.835057 31791 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:21.835268 10043 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.072s	user 0.000s	sys 0.000s
W20260709 12:25:21.836426 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55903.9us
I20260709 12:25:21.786680  9998 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.810242 10014 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:21.849323 10014 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.835086  9939 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:21.851934 10014 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:21.852602 10014 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:21.852968  9939 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:21.853101 10014 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.853916  9939 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 36 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:21.805397 10001 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:21.855034 10001 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:21.855517 10001 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:21.812691  9823 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:21.858970  9823 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:21.859418  9823 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:21.860018  9748 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:21.862447  9939 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 34.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.824230  9860 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:25:21.840927 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 87707.5us
I20260709 12:25:21.817843  9871 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:21.846916 31821 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 34: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:21.848107  9998 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:21.840425 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:21.870112 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47745.1us
I20260709 12:25:21.800239  9747 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:21.883998 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52927.8us
I20260709 12:25:21.802453  9755 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.881721 10010 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:21.882176 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:21.885537 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 50577.1us
I20260709 12:25:21.887741 32372 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 34: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:21.888100 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:21.892135 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:21.876302  9860 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:21.878751  9586 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:21.813901 10030 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:21.916416 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:21.939154 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56391.4us
I20260709 12:25:21.946070 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:21.946098 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 53742.7us
I20260709 12:25:21.923074  9860 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:21.923435  9586 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:21.925709  9537 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:21.962999 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:21.969134 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 128797us
I20260709 12:25:21.960515  9586 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:21.973755 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:21.954092  9771 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:21.960841  9748 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:21.982110 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:21.974485 10047 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:21.979224  9865 consensus_meta.cc:287] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.518s	user 0.011s	sys 0.000s
I20260709 12:25:21.963680  9862 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.025s
I20260709 12:25:21.988273  9756 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:21.993971 10025 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.993880 10038 consensus_queue.cc:1059] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.011s
I20260709 12:25:21.984913  9771 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:21.985610  9792 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:21.983247  9748 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:22.098356  9748 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:22.098585  9871 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:22.000664 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 112710us
I20260709 12:25:22.100592 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 24 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:22.103179  9638 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.104413 31718 leader_election.cc:400] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 36. Current term is 37.
I20260709 12:25:22.104771 31718 leader_election.cc:403] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:22.105628  9638 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:22.106968  9792 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election lost for term 36. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 36. Current term is 37.
I20260709 12:25:22.108364 31718 leader_election.cc:400] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 33. Current term is 34.
I20260709 12:25:22.108698 31718 leader_election.cc:403] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Cancelling election due to peer responding with higher term
I20260709 12:25:21.989866 10010 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:21.983508 10050 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:21.990736 10004 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:21.999330 10046 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.004s
I20260709 12:25:22.000876 10014 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:21.979405 10028 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.118110  9747 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:22.001319  9865 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.122800 31672 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:25:22.096290 10047 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:22.114413 10010 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:22.103207  9803 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:22.163058 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62545.4us
I20260709 12:25:22.119513 10037 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:22.168689 32373 raft_consensus.cc:2368] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 32. Current term is 33.
W20260709 12:25:22.165074 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 184004us
I20260709 12:25:22.116192 10050 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:22.155668 10051 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:22.154264  9413 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:22.172801 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.175693 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:22.096804 10026 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.162822 10014 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:22.165786 10010 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:25:22.171833 10013 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.591s	user 0.007s	sys 0.005s
I20260709 12:25:22.169060 10037 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:22.112516  9771 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.170292 10050 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:22.109642 10025 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.188540 10048 raft_consensus.cc:3055] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:22.184386  9413 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.189065 10049 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:22.189355  9857 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:22.189608 10028 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:22.191020  9865 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:22.161509 10033 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.025s
I20260709 12:25:22.183271 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:22.162760 10063 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 23 FOLLOWER]: Advancing to term 24
I20260709 12:25:22.168320  9803 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:22.162411 10015 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.169468  9638 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.026s
I20260709 12:25:22.211401 32374 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:22.199193 10037 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:22.213796 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:22.214694 32375 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
W20260709 12:25:22.218736 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.204952  9755 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:22.204948 10048 raft_consensus.cc:740] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:22.191560 10066 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.057s
I20260709 12:25:22.209031  9857 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:22.199695 10013 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.212185 10030 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:22.222167 10046 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:22.237005 31750 raft_consensus.cc:2393] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 36: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:25:22.239461 10054 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.127s	user 0.000s	sys 0.001s
W20260709 12:25:22.248783 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55122.3us
I20260709 12:25:22.233208  9755 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:22.251365  9755 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.253506  9755 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:22.258543 10058 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.259183 10058 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.261462 10058 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:22.261540 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:22.263747  9771 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:22.264757 10048 consensus_queue.cc:260] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.265354  9734 consensus_queue.cc:1059] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:22.266152 10048 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:22.269119 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:22.269745 31670 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:22.272374 10037 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.273073 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4229) took 1059 ms. Trace:
I20260709 12:25:22.233345  9857 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:22.274776 10068 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:22.274819 10062 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.002s
I20260709 12:25:22.240824 10026 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:22.279834 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:22.280504 31669 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:22.280884 10014 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election lost for term 33. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 33. Current term is 34.
I20260709 12:25:22.282807  9413 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:22.285761 10015 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:22.232805  9638 consensus_queue.cc:1243] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 33.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:25:22.231204  9812 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:25:22.245316 10042 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.432s	user 0.002s	sys 0.000s
W20260709 12:25:22.241017 10044 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.353s	user 0.000s	sys 0.003s
W20260709 12:25:22.295683 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 182393us
I20260709 12:25:22.215394 10049 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.238399 10046 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:22.290787 10016 raft_consensus.cc:3055] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:25:22.297762 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 189012us
I20260709 12:25:22.273488 31819 rpcz_store.cc:276] 0709 12:25:21.213875 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.214012 (+   137us) service_pool.cc:224] Handling call
0709 12:25:21.254674 (+ 40662us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:21.282725 (+ 28051us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:21.563413 (+280688us) log.cc:844] Serialized 14 byte log entry
0709 12:25:21.693572 (+130159us) spinlock_profiling.cc:243] Waited 59.7 ms on lock 0x7b18004638a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:22.272523 (+578951us) spinlock_profiling.cc:243] Waited 1.07 ms on lock 0x7b18004638a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287
0709 12:25:22.272711 (+   188us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:22.272763 (+    52us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:22.272805 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:22.272897 (+    92us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:22.273013 (+   116us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":16876,"prepare.queue_time_us":181410,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":44,"raft.queue_time_us":37822,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":8789,"spinlock_wait_cycles":139870976,"thread_start_us":439381,"threads_started":3,"wal-append.queue_time_us":370423}
I20260709 12:25:22.338523 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:22.339783 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:22.302374 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:22.340953 31678 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:22.303249 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:22.315115 31670 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
W20260709 12:25:22.344527 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 362510us
I20260709 12:25:22.347723 31717 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:22.343328 32342 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:22.296473 10037 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:22.362776 31669 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:22.359464 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:22.370517 31678 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
I20260709 12:25:22.377264 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.262473  9823 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.378331 31822 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
I20260709 12:25:22.340529 10016 raft_consensus.cc:740] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:22.351133  9771 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:22.392591 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:22.340212 10046 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:22.394685 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:22.405015 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:22.396900 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:22.445808 31822 raft_consensus.cc:3060] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 33
W20260709 12:25:22.408577 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.412663 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:22.364635  9942 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:22.418301 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:22.428331 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:22.447502 31751 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:22.432631 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:22.386569 10062 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:22.435015 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:22.429318  9747 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.436404 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.436548 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:22.404660 10015 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:25:22.432024 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:22.459753 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:22.437433 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.460714 31820 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:25:22.461632 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.462276 31820 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:22.437536 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:22.408077  9756 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.441857 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:22.442039 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.416198 10014 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.450938 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:22.379415  9860 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.464907 31675 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:22.439317 10025 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:22.463861 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:22.464944 31823 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:22.460143 31747 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:22.436054 10073 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:22.482719 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:22.483584 31820 raft_consensus.cc:2368] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 33. Current term is 34.
I20260709 12:25:22.484506 32374 raft_consensus.cc:2368] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 33. Current term is 34.
I20260709 12:25:22.469336 32373 raft_consensus.cc:2393] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 35: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:22.463445 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:22.471125 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:22.471505 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:22.472424 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 113103us
I20260709 12:25:22.475344 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:22.476188 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:22.481303 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:22.486346 31674 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:22.486047 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:22.488765 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:22.490959 32378 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:25:22.487538 31748 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:22.488346 10053 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:22.439630  9862 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.033s
I20260709 12:25:22.495476 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:22.505548 31673 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:22.498240 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:22.439739  9748 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.448390 10058 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.043s
I20260709 12:25:22.507865 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:22.520331 32342 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:22.506441 31747 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:22.515518 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:22.524755 31823 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:22.497720  9586 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.513s	user 0.010s	sys 0.001s
I20260709 12:25:22.452364  9942 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.483289 10073 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:22.528728 31792 leader_election.cc:400] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 32. Current term is 33.
I20260709 12:25:22.536518 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:22.536892 31792 leader_election.cc:403] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:22.533701 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:22.527463 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:22.465730  9747 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.494910 10015 raft_consensus.cc:697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 37 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:22.473101  9756 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.549850 31746 raft_consensus.cc:2468] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:22.555413 31674 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:25:22.557181 31747 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:25:22.476328 10014 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:22.522450 10001 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.666s	user 0.013s	sys 0.001s
I20260709 12:25:22.568001 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.532397 10029 raft_consensus.cc:2804] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:22.461555  9812 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.459739 10016 consensus_queue.cc:260] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.534157 10013 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:22.573434 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:22.573658 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:22.499001  9857 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.547286  9865 raft_consensus.cc:2804] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:22.547106  9871 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:22.547379 10076 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:22.547398 10073 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:22.589939 31749 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
W20260709 12:25:22.607952 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 118773us
I20260709 12:25:22.619041 10016 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:22.579401 10067 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.281s	user 0.000s	sys 0.001s
I20260709 12:25:22.613112 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.628991 31824 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:22.630256 10076 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.633250 10076 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:22.577694 10060 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
I20260709 12:25:22.636523 10060 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:22.585459 10001 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.581259 10072 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:22.589934 31673 raft_consensus.cc:2468] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 36.
I20260709 12:25:22.588649 31823 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:22.525177 10053 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.550849  9586 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.567528 10071 consensus_queue.cc:1059] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.195s
I20260709 12:25:22.664746 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:22.664511 10015 consensus_queue.cc:237] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.665514 31676 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:22.669027 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:22.670183 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:22.649173 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:22.650162 31671 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:22.657264 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:22.663117 10072 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:22.627703 10083 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:25:22.671980 31747 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:22.680121 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 68909.5us
I20260709 12:25:22.694727 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:22.696348 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:22.698388 31823 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:22.699232 31744 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
I20260709 12:25:22.700948 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:22.702092 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:22.703596 31717 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:22.599257 10004 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:22.716997 31825 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: []
W20260709 12:25:22.695924 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 47868.7us
I20260709 12:25:22.647972  9823 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:22.723778 31675 raft_consensus.cc:2410] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 34 because replica has last-logged OpId of term: 32 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:25:22.601590 10029 raft_consensus.cc:697] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 33 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:22.659973 10060 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:22.651999  9865 raft_consensus.cc:697] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 35 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:22.647732  9871 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:22.726750 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:22.702976  9638 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:22.800424 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:22.714067 10074 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:22.802963 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 82189.3us
I20260709 12:25:22.715304 10025 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:22.834465 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:22.668437 10033 consensus_queue.cc:1059] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 28.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.126s
I20260709 12:25:22.880260 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:22.893419  9586 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:22.899947  9756 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:22.886761 10004 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:22.907701 10050 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.701s	user 0.005s	sys 0.005s
I20260709 12:25:22.908263 10050 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.914110 31822 raft_consensus.cc:2468] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
I20260709 12:25:22.887526 10013 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:22.915159 10013 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:22.916879 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:22.921140 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.922151 31822 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:22.923208 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:22.923669  9803 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.698s	user 0.002s	sys 0.009s
I20260709 12:25:22.923854 31822 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:22.924160  9803 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:22.947178 10029 consensus_queue.cc:237] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.817008 10074 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:22.951495 10074 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:22.953635  9865 consensus_queue.cc:237] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.24, Last appended by leader: 24, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:22.958341 10085 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
I20260709 12:25:22.960963 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.962092 31827 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:22.967070  9871 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:22.969558 31822 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:22.972046  9757 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 32. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 32. Current term is 33.
I20260709 12:25:22.973387 31672 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:22.974807 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.975483 31822 raft_consensus.cc:2376] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 36. Re-sending same reply.
I20260709 12:25:22.976601 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:22.979884 10077 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.386s	user 0.001s	sys 0.000s
I20260709 12:25:22.982468 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.984148 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.984560 10053 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:22.984807 31826 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:25:22.985009 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:22.917512 10013 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:25:22.919343 10004 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:22.767892 10076 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 14, Time since last communication: 0.145s
I20260709 12:25:22.853459 10025 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:23.028363 31751 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.578s	user 0.012s	sys 0.000s
W20260709 12:25:23.008872 31745 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.554s	user 0.010s	sys 0.000s
I20260709 12:25:23.033414 31745 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:23.017352 31676 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:25:23.032373 31751 raft_consensus.cc:2468] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:23.044227 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 4234) took 1735 ms. Trace:
I20260709 12:25:22.926523  9803 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:23.044760 31750 rpcz_store.cc:276] 0709 12:25:21.309109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:21.321157 (+ 12048us) service_pool.cc:224] Handling call
0709 12:25:23.044020 (+1722863us) spinlock_profiling.cc:243] Waited 868 ms on lock 0x7b54003e3ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:23.044162 (+   142us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1996010880}
I20260709 12:25:23.045526 31827 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:25:23.067741 31827 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:22.891306  9748 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:22.995991 10050 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:23.082185 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:23.100315 10072 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:23.104203 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.104522 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:23.009382 10028 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:23.047878 10025 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:23.133658  9857 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:23.137420 10082 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.752s	user 0.002s	sys 0.002s
I20260709 12:25:23.055837 10014 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:23.145146 31717 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:23.121289 10010 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.916s	user 0.011s	sys 0.001s
I20260709 12:25:23.150172 31717 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:23.077337 31823 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 37.
I20260709 12:25:23.154599 31747 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:23.115435 10028 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:23.115141  9942 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:23.262327 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:23.152187 10063 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.918s	user 0.012s	sys 0.000s
I20260709 12:25:23.270789 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:23.273635 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 562338us
I20260709 12:25:23.274552 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:23.151288 10072 raft_consensus.cc:2804] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:23.274663 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:23.156160 10014 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election won for term 28
W20260709 12:25:23.266278 10046 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.800s	user 0.006s	sys 0.006s
W20260709 12:25:23.266001 10016 consensus_meta.cc:287] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.636s	user 0.012s	sys 0.000s
I20260709 12:25:23.289116 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:23.283409 32375 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 36. Current term is 37. Ops: []
I20260709 12:25:23.261531  9998 raft_consensus.cc:3055] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:23.285028 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:23.297348 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:23.299118 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.628s	user 0.009s	sys 0.040s
I20260709 12:25:23.285248 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:23.300840 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.629s	user 0.009s	sys 0.041s
W20260709 12:25:23.271152 10073 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.645s	user 0.011s	sys 0.000s
W20260709 12:25:23.301349 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 630699us
I20260709 12:25:23.304591 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:23.154153 10010 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:23.311806 32373 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:23.296939  9998 raft_consensus.cc:740] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:23.308377 10072 raft_consensus.cc:697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:23.478458 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:23.479950 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:23.658787 31825 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:25:23.659273 31827 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 31: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:25:23.346737 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61788.8us
W20260709 12:25:23.541004 10113 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.069s	user 0.003s	sys 0.000s
I20260709 12:25:23.347469 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:23.662166 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:23.663496 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 316062us
I20260709 12:25:23.359223 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:25:23.668883 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:25:23.670035 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:23.664259 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:23.671392 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:23.671710 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:23.672209 32374 raft_consensus.cc:2393] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 29: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:23.672360 32342 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:23.658494 10025 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.529s	user 0.010s	sys 0.001s
I20260709 12:25:23.564690 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1888) took 1270 ms. Trace:
I20260709 12:25:23.576653 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:23.314782 10073 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:23.497784 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:23.519860 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:23.683488 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 185796us
I20260709 12:25:23.416095 10105 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:23.559934 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:23.517149 10106 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.060s	user 0.000s	sys 0.001s
W20260709 12:25:23.414410 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.466655 10058 raft_consensus.cc:3055] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:23.575264 31669 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 36. Current term is 37. Ops: []
I20260709 12:25:23.491029  9907 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:23.487738 10013 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.726464 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:23.726912 10013 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:23.727695 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:23.490433  9747 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:23.582988 10093 log.cc:927] Time spent T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.235s	user 0.001s	sys 0.001s
I20260709 12:25:23.509127 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:23.728571  9757 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:23.492137 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57176 (request call id 1522) took 1082 ms. Trace:
I20260709 12:25:23.579035 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.728991 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:23.729105 31746 rpcz_store.cc:276] 0709 12:25:22.410072 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.410208 (+   136us) service_pool.cc:224] Handling call
0709 12:25:23.491979 (+1081771us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x7b54002945d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:23.492080 (+   101us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":233407744}
I20260709 12:25:23.729701 31675 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:23.729936  9757 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:23.577688 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.546196  9860 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:23.579376 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.731101 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:23.315294 10014 raft_consensus.cc:697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:23.731827 31824 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:23.732920 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:23.733631 31824 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:23.732829 10014 consensus_queue.cc:237] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 26.18, Last appended by leader: 18, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:23.593941 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.502367 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:23.730392 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:23.743191 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 240868us
I20260709 12:25:23.348449 10097 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:23.399181 10080 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:23.752893 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.287051  9942 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:23.294896 10046 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:23.340235 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 65660.2us
I20260709 12:25:23.757241 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.523268 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:23.759310 31823 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:23.531642 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:23.293491 10001 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:23.367815 31821 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:23.509902 10089 raft_consensus.cc:3055] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:23.356877 10028 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:23.484993 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1683) took 1189 ms. Trace:
I20260709 12:25:23.643031 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 4030) took 1159 ms. Trace:
W20260709 12:25:23.556612 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:23.644644 10076 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:23.638595 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4256) took 1056 ms. Trace:
I20260709 12:25:23.771087 31752 rpcz_store.cc:276] 0709 12:25:22.581874 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.131202 (+549328us) service_pool.cc:224] Handling call
0709 12:25:23.131520 (+   318us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.131919 (+   399us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.140790 (+  8871us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.143422 (+  2632us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.637903 (+494481us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:23.637979 (+    76us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.638082 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.638152 (+    70us) raft_consensus.cc:1685] finished
0709 12:25:23.638170 (+    18us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.638350 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":12777,"raft.run_cpu_time_us":152,"raft.run_wall_time_us":5888,"thread_start_us":368343,"threads_started":1,"wal-append.queue_time_us":312821}
W20260709 12:25:23.277195 10015 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s	user 0.022s	sys 0.020s
I20260709 12:25:23.646608 10010 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:23.656387 10060 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.856s	user 0.009s	sys 0.003s
I20260709 12:25:23.672410 31825 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:23.673534 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:23.284600 10090 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:23.675297 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:23.578612 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:23.683691 31748 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.536s	user 0.056s	sys 0.079s
W20260709 12:25:23.789526 31748 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.642s	user 0.056s	sys 0.079s
I20260709 12:25:23.695995 10118 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:23.701007 31819 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:23.672310  9832 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:23.676884 31818 rpcz_store.cc:276] 0709 12:25:22.294478 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.294800 (+   322us) service_pool.cc:224] Handling call
0709 12:25:22.338397 (+ 43597us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.354412 (+ 16015us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:22.442956 (+ 88544us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.282005 (+839049us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.282097 (+    92us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:23.282140 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.564319 (+282179us) spinlock_profiling.cc:243] Waited 15.7 ms on lock 0x7b5400196ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:23.564391 (+    72us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.564601 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":205768,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":50,"raft.queue_time_us":92256,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"spinlock_wait_cycles":36156672,"thread_start_us":894718,"threads_started":3,"wal-append.queue_time_us":838398}
I20260709 12:25:23.710696 10105 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:23.717825 32374 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:23.718595  9998 consensus_queue.cc:260] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:23.829700 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:23.830735 31749 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
W20260709 12:25:23.778154 10015 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.110s	user 0.023s	sys 0.021s
I20260709 12:25:23.683460 10056 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.719816 10072 consensus_queue.cc:237] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 34.17, Last appended by leader: 17, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:23.683197  9792 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.676609  9638 raft_consensus.cc:2804] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:23.682935  9771 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.676419  9865 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:25:23.723708 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1526) took 1298 ms. Trace:
I20260709 12:25:23.724936 10111 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 28.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:23.725168 10079 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 36.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.000s
I20260709 12:25:23.727792 10062 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:23.548127 10049 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:23.281674 10063 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election lost for term 24. Reason: could not achieve majority
I20260709 12:25:23.716125  9907 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:23.821619  9832 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:23.735585 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:23.729033 10013 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:23.716266 10058 raft_consensus.cc:740] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:23.744846 31824 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:23.764110  9857 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.080s
I20260709 12:25:23.746049 32378 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:23.753211 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.836740 10105 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:23.756359 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.756669  9757 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:23.761013 31675 raft_consensus.cc:2368] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30.
I20260709 12:25:23.763079 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.763682 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:23.876246 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 112638us
I20260709 12:25:23.877224 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:23.758543 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:23.684176 10025 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:23.753283 10080 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:23.781256 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1896) took 1442 ms. Trace:
I20260709 12:25:23.765295 31672 rpcz_store.cc:276] 0709 12:25:22.295483 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.384778 (+ 89295us) service_pool.cc:224] Handling call
0709 12:25:22.385111 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.973951 (+588840us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.484698 (+510747us) spinlock_profiling.cc:243] Waited 547 ms on lock 0x7b540022b854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:23.484924 (+   226us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1258604928}
I20260709 12:25:23.760797 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:23.893105 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:23.893790 31823 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
I20260709 12:25:23.766395 31820 rpcz_store.cc:276] 0709 12:25:22.484201 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.485140 (+   939us) service_pool.cc:224] Handling call
0709 12:25:22.485514 (+   374us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:22.508123 (+ 22609us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:22.508173 (+    50us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:22.523463 (+ 15290us) log.cc:844] Serialized 26 byte log entry
0709 12:25:22.804028 (+280565us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:25:22.877535 (+ 73507us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.642615 (+765080us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.642732 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.642783 (+    51us) raft_consensus.cc:1685] finished
0709 12:25:23.642804 (+    21us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.642968 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":73309,"prepare.run_cpu_time_us":109,"prepare.run_wall_time_us":38707,"raft.queue_time_us":30427,"raft.run_cpu_time_us":228,"raft.run_wall_time_us":26488,"thread_start_us":611765,"threads_started":2,"wal-append.queue_time_us":528929}
W20260709 12:25:23.788138 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 114666us
I20260709 12:25:23.806285 10060 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:23.791180 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:23.799242 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4250) took 1325 ms. Trace:
I20260709 12:25:23.803634 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:23.804983 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:23.820490 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4249) took 1346 ms. Trace:
I20260709 12:25:23.891194 31821 rpcz_store.cc:276] 0709 12:25:22.339288 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.350926 (+ 11638us) service_pool.cc:224] Handling call
0709 12:25:22.387245 (+ 36319us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.764274 (+1377029us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.780737 (+ 16463us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x7b54003ca0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:23.781152 (+   415us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2254489472}
I20260709 12:25:23.903084 31748 rpcz_store.cc:276] 0709 12:25:22.473830 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.542989 (+ 69159us) service_pool.cc:224] Handling call
0709 12:25:22.543470 (+   481us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:22.758870 (+215400us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:23.147495 (+388625us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.790043 (+642548us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.790098 (+    55us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:23.790153 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.820103 (+ 29950us) spinlock_profiling.cc:243] Waited 215 ms on lock 0x7b54002f76d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:23.820178 (+    75us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.820395 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":219919,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":498,"spinlock_wait_cycles":512301312,"thread_start_us":980610,"threads_started":2,"wal-append.queue_time_us":614644}
I20260709 12:25:23.905778 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:23.823056 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:23.824607 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:23.719543 10073 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:23.838486 31670 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:23.841904 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 127699us
I20260709 12:25:23.907214 31751 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 31: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:23.857923 10062 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:23.843497 31819 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 4250) took 1311 ms (client timeout 1621 ms). Trace:
I20260709 12:25:23.845876 31674 raft_consensus.cc:2368] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35.
I20260709 12:25:23.824056 10028 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:23.852171 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:23.853021 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.833835 10037 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:23.935249 31749 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:25:23.751123 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.862676 31673 raft_consensus.cc:2393] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 37: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:23.756150 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.847779 31753 rpcz_store.cc:276] 0709 12:25:22.425529 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.425732 (+   203us) service_pool.cc:224] Handling call
0709 12:25:22.439469 (+ 13737us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:22.470689 (+ 31220us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:25:22.722963 (+252274us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.281035 (+558072us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:23.290502 (+  9467us) log.cc:844] Serialized 52 byte log entry
0709 12:25:23.290688 (+   186us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:25:23.291057 (+   369us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.303886 (+ 12829us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.303975 (+    89us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:23.499490 (+195515us) spinlock_profiling.cc:243] Waited 7.58 ms on lock 0x7b18002b6de0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:23.723449 (+223959us) raft_consensus.cc:1685] finished
0709 12:25:23.723475 (+    26us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.723625 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38315,"prepare.queue_time_us":249550,"prepare.run_cpu_time_us":67,"prepare.run_wall_time_us":68,"raft.queue_time_us":56158,"raft.run_cpu_time_us":546,"raft.run_wall_time_us":43444,"spinlock_wait_cycles":17427712,"thread_start_us":617850,"threads_started":2,"wal-append.queue_time_us":604684}
I20260709 12:25:23.879122 31678 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 29: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:23.881736 31822 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 24.
I20260709 12:25:23.888684 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:23.848425 10056 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:23.920167 10058 consensus_queue.cc:260] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:23.896904 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 138477us
I20260709 12:25:23.940945 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.850584  9792 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:23.941555 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:23.942229 31746 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:23.898363 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:23.902068 31745 rpcz_store.cc:276] 0709 12:25:22.474144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.080809 (+606665us) service_pool.cc:224] Handling call
0709 12:25:23.549723 (+468914us) spinlock_profiling.cc:243] Waited 3.64 ms on lock 0x7b5800187b98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0e9c 00007fe8f9cb0a5b 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901daefc0 00007fe901d99929 00007fe901d993d2 00007fe8fb310bd4
0709 12:25:23.549906 (+   183us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.550184 (+   278us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:23.550577 (+   393us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.792912 (+242335us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.798794 (+  5882us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:23.798866 (+    72us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:23.798924 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.799011 (+    87us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:23.799165 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3372,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":66,"raft.queue_time_us":10938,"raft.run_cpu_time_us":5797,"raft.run_wall_time_us":35449,"spinlock_wait_cycles":51270784,"thread_start_us":238257,"threads_started":1,"wal-append.queue_time_us":298969}
I20260709 12:25:23.763972 10089 raft_consensus.cc:740] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:23.911237 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.912716 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:23.849826  9771 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:23.914140 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:23.915362 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:23.916190 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:23.916800  9865 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:23.906562 31748 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:23.917585 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:23.919135  9907 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:23.881213 10121 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.083s	user 0.000s	sys 0.007s
I20260709 12:25:23.863019 10097 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:23.833380 10118 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:23.863379  9832 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:23.930431 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
W20260709 12:25:23.930967 31819 rpcz_store.cc:269] 0709 12:25:22.525485 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.646200 (+120715us) service_pool.cc:224] Handling call
0709 12:25:23.836126 (+1189926us) spinlock_profiling.cc:243] Waited 617 ms on lock 0x7b5400379454. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:23.836343 (+   217us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1420100864}
I20260709 12:25:23.929237 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.884522 10013 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:25:23.854512 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:23.753309 10066 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:23.988637 31826 raft_consensus.cc:2410] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 31 because replica has last-logged OpId of term: 29 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 18.
I20260709 12:25:23.890537  9998 raft_consensus.cc:3060] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:23.992111 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:23.994746 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:23.995694 31751 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 34: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:23.901351 10080 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:23.891084 10049 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:23.900596  9812 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:23.954927 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:23.954797 31672 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 34: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:24.005615 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:23.977185 10073 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:25:23.954839 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66322us
I20260709 12:25:23.962198 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:23.969264 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:24.011241 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:23.975548 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:23.976176 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:24.013240 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:23.978397 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:23.931497  9638 raft_consensus.cc:697] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:24.024576 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:24.024968  9638 consensus_queue.cc:237] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 29.19, Last appended by leader: 19, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.026229 31676 raft_consensus.cc:1240] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 27. Current term is 28. Ops: []
I20260709 12:25:23.984176 10097 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:23.981593 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:24.027772 31745 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:25:24.028978 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:24.029922 31819 raft_consensus.cc:2410] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 34 because replica has last-logged OpId of term: 32 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 18.
I20260709 12:25:23.937809 10062 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:23.985658 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:23.980417 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:24.036164 31821 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
W20260709 12:25:23.950990 10074 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.999s	user 0.015s	sys 0.000s
I20260709 12:25:23.913396 31825 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 36: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:23.906104  9930 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:24.041604 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:24.037972 10025 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:24.045534 10118 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:24.045822 10001 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:24.046758 10060 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:24.047201 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.047518 10079 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:24.048159 10079 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.000016 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:24.001055 31673 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 36 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:24.051698 31669 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:23.959338 10128 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:24.013933 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.015904 31823 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:23.984876 10013 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:24.017624 10124 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.096s	user 0.000s	sys 0.001s
I20260709 12:25:24.018714 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4258) took 1338 ms. Trace:
I20260709 12:25:24.022073 10014 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:25:24.087877 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 53230.5us
W20260709 12:25:24.092733 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 51218.4us
I20260709 12:25:23.994163  9907 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:24.023751 10010 raft_consensus.cc:2764] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election decision vote started in defunct term 28: won
I20260709 12:25:24.026958 10102 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:23.988788 10066 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.034471 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:23.989714 10058 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:24.039215 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:24.102557 31826 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:24.046327 10089 consensus_queue.cc:260] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:23.905918 10090 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:24.001705 10040 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:24.123533 10040 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.124927  9871 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:24.125612  9871 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.126077 10040 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:24.080104 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66440us
I20260709 12:25:24.127116 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.128058 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:24.128444 31718 leader_election.cc:400] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 33. Current term is 34.
I20260709 12:25:24.128728 31676 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:24.015316 10073 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.081115 31747 rpcz_store.cc:276] 0709 12:25:22.680947 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.261713 (+580766us) service_pool.cc:224] Handling call
0709 12:25:23.262073 (+   360us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:23.262474 (+   401us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:23.262507 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:23.264938 (+  2431us) log.cc:844] Serialized 26 byte log entry
0709 12:25:23.782211 (+517273us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:23.782275 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:23.782376 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.018458 (+236082us) raft_consensus.cc:1685] finished
0709 12:25:24.018482 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.018652 (+   170us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1453,"raft.queue_time_us":611,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":7766,"thread_start_us":479775,"threads_started":1,"wal-append.queue_time_us":619954}
I20260709 12:25:24.090481 31748 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:24.135351 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:24.136054 31749 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:23.999123 31746 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:24.012348 10111 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.011s
I20260709 12:25:24.094386 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:24.137694 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:24.138034 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 43770.6us
I20260709 12:25:23.980472  9865 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:24.138736 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46403.3us
I20260709 12:25:24.104529 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:24.144264 31751 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:25:24.114585 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 110283us
I20260709 12:25:24.117497 31675 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 28: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:24.108495 10015 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:24.100800 10102 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:24.103905 10068 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:24.012850 10049 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.130014 10090 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:24.192830 31750 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:24.141927 10073 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:24.027657 10134 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.108s
I20260709 12:25:24.091656 10014 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:24.087728  9930 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:24.108969 10066 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:24.166388 10089 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:24.233258 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 88996.2us
I20260709 12:25:24.098515 10025 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:24.143146  9747 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:24.192279 32377 raft_consensus.cc:1240] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30. Ops: []
W20260709 12:25:24.204157 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.247839 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:24.213768 10079 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:24.210803 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:24.162551 10124 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:25:24.149014 10001 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:25:24.224252 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.050458 10074 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.158169 10046 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:24.156456 10102 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:24.236622 31675 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
W20260709 12:25:24.234834 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 89483.1us
I20260709 12:25:24.202736 10090 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:24.222146  9638 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.057s
I20260709 12:25:24.210736 10037 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:24.219673  9413 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:24.223354 10014 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:24.304235 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:24.244639 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:24.282373 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:24.238217 10111 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:24.289608 10001 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:24.306272 10033 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.311324 10001 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:24.311570 10033 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.313851 10033 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:24.319289 10133 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.127s	user 0.000s	sys 0.000s
W20260709 12:25:24.320396 10138 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.104s	user 0.000s	sys 0.000s
I20260709 12:25:24.323622 31818 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:24.325457 31818 raft_consensus.cc:1275] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
W20260709 12:25:24.342795 10137 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.059s	user 0.001s	sys 0.000s
W20260709 12:25:24.345678  9942 consensus_meta.cc:287] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.589s	user 0.010s	sys 0.000s
I20260709 12:25:24.162048 10015 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.248773  9747 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:24.367416 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:24.420616 31677 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:24.373270 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:24.377135 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 132811us
I20260709 12:25:24.241523 10025 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:24.433322 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:24.433930 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:24.296245 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.243866  9860 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.046s
I20260709 12:25:24.300225 31791 leader_election.cc:400] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30.
I20260709 12:25:24.436722 31791 leader_election.cc:403] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:24.147097 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.302556 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4214) took 1939 ms. Trace:
I20260709 12:25:24.346891 10111 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:25:24.439708 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 146263us
I20260709 12:25:24.347281 31824 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 34. Current term is 35. Ops: [34.22-34.22]
I20260709 12:25:24.353220 10074 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:24.274540 10056 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:24.356289  9586 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:24.358672 31673 raft_consensus.cc:1240] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 35. Current term is 36. Ops: [32.16-35.17]
I20260709 12:25:24.302405 10140 consensus_queue.cc:1059] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 33.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.074s
I20260709 12:25:24.209142 31669 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:25:24.435891 31643 leader_election.cc:400] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 33. Current term is 34.
W20260709 12:25:24.437027 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.437459 31670 rpcz_store.cc:276] 0709 12:25:22.363526 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.365638 (+  2112us) service_pool.cc:224] Handling call
0709 12:25:22.406480 (+ 40842us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.907210 (+1500730us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.302232 (+395022us) spinlock_profiling.cc:243] Waited 1.43 s on lock 0x7b540017c0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:24.302489 (+   257us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3292783360}
W20260709 12:25:24.443132 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 296120us
I20260709 12:25:24.446570 31643 leader_election.cc:403] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Cancelling election due to peer responding with higher term
I20260709 12:25:24.449620 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.442296 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.450932 10076 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:24.451380 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.452276 10076 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:24.452728 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.453632 10076 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:24.453912 10079 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Leader pre-election won for term 25
I20260709 12:25:24.454519 10079 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:24.454892 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.455070 10079 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:24.407779 10013 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 36.24, Last appended by leader: 24, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.456261 32371 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:24.456339 10145 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30.
I20260709 12:25:24.456614 10013 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:24.458104 10128 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:25:24.459717  9586 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:24.460729  9586 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.462108 10140 raft_consensus.cc:3055] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:24.455937  9748 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:24.407648  9747 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:24.409885 31674 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:24.465938 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.380029  9453 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:24.419370 31751 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 36.
I20260709 12:25:24.470687 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:24.432046  9907 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.372066 10049 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:24.464664 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.373865 31744 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:24.402292 31749 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:24.462522 10140 raft_consensus.cc:740] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:24.437757 10025 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:24.436896 10046 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:24.436795  9283 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:24.481990 31749 raft_consensus.cc:1240] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 32. Current term is 33. Ops: [32.19-32.19]
I20260709 12:25:24.362928 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:24.501879 31718 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:24.483894 31716 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.484803 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:24.464905 10040 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election lost for term 33. Reason: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for earlier term 33. Current term is 34.
I20260709 12:25:24.490826 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.491869 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:24.494531 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:24.362556 10097 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.498258 31821 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 34 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:25:24.471535 10127 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:24.409075  9942 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:24.519582 31753 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 31 index: 20. (index mismatch)
I20260709 12:25:24.506539 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.367053 10037 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 32.16, Last appended by leader: 16, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.485920  9453 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:24.523130 32374 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 28: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:24.499850 10049 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:24.503329  9748 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:24.368466  9930 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:24.548815 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 99420.9us
I20260709 12:25:24.551029 32370 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
W20260709 12:25:24.535560 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44832.4us
I20260709 12:25:24.558131 31820 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:24.482817  9907 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.560631 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:24.506247 10128 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:24.562209 31821 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:25:24.560313 31791 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:24.488442 10070 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.061s
I20260709 12:25:24.566511 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:24.558298 10080 consensus_meta.cc:287] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.543s	user 0.011s	sys 0.002s
I20260709 12:25:24.526269  9871 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:24.549979 10145 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:24.557658  9638 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.507170 10046 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.584867 31821 raft_consensus.cc:1240] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31. Ops: []
W20260709 12:25:24.584385 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 102103us
I20260709 12:25:24.526894 10060 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.057s
W20260709 12:25:24.567936  9832 consensus_meta.cc:287] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.565s	user 0.009s	sys 0.003s
I20260709 12:25:24.594195 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:24.507521  9283 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.606132 31821 raft_consensus.cc:1275] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:25:24.577670 10049 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:24.597797 31753 raft_consensus.cc:3060] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:24.607160 31820 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:25:24.588553  9942 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:24.586974  9748 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:24.608886  9832 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.606915 10140 consensus_queue.cc:260] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.587468 10080 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:24.618171 10142 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.139s	user 0.001s	sys 0.000s
W20260709 12:25:24.619019 31643 leader_election.cc:336] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Timed out: RequestConsensusVote RPC to 127.29.201.195:44645 timed out after 1.621s (SENT)
I20260709 12:25:24.610077 10049 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:24.620292 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4263) took 1127 ms. Trace:
W20260709 12:25:24.634179 10062 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.548s	user 0.009s	sys 0.000s
I20260709 12:25:24.640411  9907 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:24.530164 10111 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.608258 10076 consensus_queue.cc:1059] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 31.15, Next index: 16, Last known committed idx: 14, Time since last communication: 0.040s
I20260709 12:25:24.604159  9638 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.692222 31819 rpcz_store.cc:276] 0709 12:25:23.492879 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.050221 (+557342us) service_pool.cc:224] Handling call
0709 12:25:24.056361 (+  6140us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.092512 (+ 36151us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:24.092542 (+    30us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:24.100644 (+  8102us) log.cc:844] Serialized 26 byte log entry
0709 12:25:24.505773 (+405129us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:24.505831 (+    58us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.505904 (+    73us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.620051 (+114147us) raft_consensus.cc:1685] finished
0709 12:25:24.620070 (+    19us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.620231 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":3012,"raft.run_cpu_time_us":184,"raft.run_wall_time_us":12263,"thread_start_us":293167,"threads_started":1,"wal-append.queue_time_us":422999}
I20260709 12:25:24.691742 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4262) took 1199 ms. Trace:
I20260709 12:25:24.711361 31678 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:24.715682 31670 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 32 index: 23. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:25:24.713476 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:24.661453 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:24.751549 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:24.660674 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:24.667191 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.664870 31717 leader_election.cc:400] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35.
I20260709 12:25:24.714037 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:24.758208 31717 leader_election.cc:403] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Cancelling election due to peer responding with higher term
I20260709 12:25:24.646553 10001 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:24.759349 10152 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.759747 10102 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.760058 10152 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.761853 10001 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:25:24.762081 10102 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:24.763129 10076 raft_consensus.cc:3055] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:25:24.763629 10076 raft_consensus.cc:740] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:24.763729 10025 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:24.764858 10066 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.520s	user 0.006s	sys 0.003s
I20260709 12:25:24.764616 10076 consensus_queue.cc:260] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 35.17, Last appended by leader: 17, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.765450 10066 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.766088 10076 raft_consensus.cc:3060] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:24.647461  9586 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:24.653138 10147 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.198s	user 0.001s	sys 0.000s
W20260709 12:25:24.767201 10073 consensus_meta.cc:287] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.544s	user 0.009s	sys 0.001s
I20260709 12:25:24.767947 10066 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:24.768760 31824 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:24.766188 10025 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:24.769367 31746 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:25:24.798422 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.754240 31825 rpcz_store.cc:276] 0709 12:25:23.492582 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.048147 (+555565us) service_pool.cc:224] Handling call
0709 12:25:24.048500 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.091669 (+ 43169us) raft_consensus.cc:1534] Early marking committed up to index 18
0709 12:25:24.091716 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:24.108240 (+ 16524us) log.cc:844] Serialized 26 byte log entry
0709 12:25:24.518944 (+410704us) spinlock_profiling.cc:243] Waited 4.73 ms on lock 0x7b180043a3e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc9750d1 00007fe8fcd90a0f
0709 12:25:24.608709 (+ 89765us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:24.608773 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.608890 (+   117us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.691425 (+ 82535us) raft_consensus.cc:1685] finished
0709 12:25:24.691447 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.691676 (+   229us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30478,"raft.queue_time_us":43431,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":12036,"spinlock_wait_cycles":10882816,"thread_start_us":89505,"threads_started":1,"wal-append.queue_time_us":533754}
I20260709 12:25:24.666188 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.679716 32373 raft_consensus.cc:1240] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:24.767035 10111 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:24.684894 31748 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:24.686936 31675 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:24.689438 31643 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:24.772985 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 108270us
W20260709 12:25:24.650465 10130 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.412s	user 0.002s	sys 0.000s
I20260709 12:25:24.679473  9832 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:24.768221 10014 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:24.777236 10105 consensus_meta.cc:287] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.875s	user 0.012s	sys 0.000s
I20260709 12:25:24.651239 10063 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:24.651365 10010 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:24.665601 10128 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.112s
I20260709 12:25:24.653472  9942 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:24.873744 31747 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 34: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:24.739451 10062 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.650938 10068 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 28.15, Next index: 16, Last known committed idx: 13, Time since last communication: 0.128s
I20260709 12:25:24.651109 10144 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:24.659260 10151 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
I20260709 12:25:24.669996  9771 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:24.778856 10073 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.892398 31752 raft_consensus.cc:2368] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31.
I20260709 12:25:24.762408 10152 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:24.780578  9756 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.673862 10056 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:25:24.676641  9283 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:24.671944  9748 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:24.894094 10056 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:24.671736 10015 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:24.894517  9748 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:24.783531 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 4037) took 2125 ms (client timeout 2475 ms). Trace:
W20260709 12:25:24.898873 31822 rpcz_store.cc:269] 0709 12:25:22.660247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.984738 (+324491us) service_pool.cc:224] Handling call
0709 12:25:24.783306 (+1798568us) spinlock_profiling.cc:243] Waited 893 ms on lock 0x7b540030edd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:24.783454 (+   148us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2053425024}
I20260709 12:25:24.900216 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:24.679692 10058 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.564s	user 0.010s	sys 0.000s
I20260709 12:25:24.788506 10001 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:24.818277 10066 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:24.681195 10046 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:24.788841 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:24.684621  9998 consensus_meta.cc:287] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.680s	user 0.005s	sys 0.004s
I20260709 12:25:24.790963 10127 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:24.791064 10097 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:24.686650 10080 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:24.646970  9453 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 27.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.788760 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3311) took 1019 ms. Trace:
I20260709 12:25:24.913854 31750 rpcz_store.cc:276] 0709 12:25:23.769438 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.787384 (+ 17946us) service_pool.cc:224] Handling call
0709 12:25:23.842026 (+ 54642us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.196439 (+354413us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.788494 (+592055us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b54003e3ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:24.788697 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":806267136}
I20260709 12:25:24.914515 31827 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:24.652231 10033 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:24.800521 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 134399us
I20260709 12:25:24.664501 10158 consensus_queue.cc:1059] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 27.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.137s
I20260709 12:25:24.771795 10090 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.811749 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4272) took 1209 ms. Trace:
I20260709 12:25:24.878551 10128 consensus_queue.cc:1243] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer b95aeeaf4c51442b8092cff1de674f2b log is divergent from this leader: its last log entry 36.21 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 20
I20260709 12:25:24.811538 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:24.812832 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 193835us
I20260709 12:25:24.813860 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:24.927285 31748 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:24.712352 10079 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.824271 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:24.840211 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4264) took 1347 ms. Trace:
I20260709 12:25:24.931876 31823 rpcz_store.cc:276] 0709 12:25:23.493184 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.057820 (+564636us) service_pool.cc:224] Handling call
0709 12:25:24.058173 (+   353us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.238119 (+179946us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:24.238178 (+    59us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:24.270490 (+ 32312us) log.cc:844] Serialized 26 byte log entry
0709 12:25:24.575296 (+304806us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:24.575344 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.575428 (+    84us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.839923 (+264495us) raft_consensus.cc:1685] finished
0709 12:25:24.839948 (+    25us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.840142 (+   194us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":36509,"raft.queue_time_us":64312,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"thread_start_us":411565,"threads_started":2,"wal-append.queue_time_us":493910}
I20260709 12:25:24.840611 31753 raft_consensus.cc:1275] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 33 index: 24. Preceding OpId from leader: term: 35 index: 25. (index mismatch)
I20260709 12:25:24.805850 10110 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
W20260709 12:25:24.843842 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 315572us
W20260709 12:25:24.859782 31744 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 4259) took 2114 ms (client timeout 1570 ms). Trace:
I20260709 12:25:24.943684 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:24.943638 31744 rpcz_store.cc:269] 0709 12:25:22.745787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:22.811387 (+ 65600us) service_pool.cc:224] Handling call
0709 12:25:24.859461 (+2048074us) spinlock_profiling.cc:243] Waited 1.49 s on lock 0x7b54002968d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:24.859677 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":3426903808}
I20260709 12:25:24.944133 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.863448 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 4052) took 1219 ms. Trace:
I20260709 12:25:24.945647 31820 rpcz_store.cc:276] 0709 12:25:23.643941 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.894995 (+251054us) service_pool.cc:224] Handling call
0709 12:25:24.367215 (+472220us) spinlock_profiling.cc:243] Waited 3.57 ms on lock 0x7b5800003f98. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9cb0b71 00007fe8f9cb0a22 00007fe902bc98d9 00007fe901d23ee0 00007fe901ded0ab 00007fe901ded131 00007fe901d9b0a0 00007fe901d9a984 00007fe8fb3156d4 00007fe8fb315662
0709 12:25:24.863170 (+495955us) spinlock_profiling.cc:243] Waited 138 ms on lock 0x7b54001c93d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:24.863358 (+   188us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":324920576}
I20260709 12:25:24.862403 10048 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 36 LEADER]: Stepping down as leader of term 36
W20260709 12:25:24.870735 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.876710 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:24.948418 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.880437 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:24.878340 10145 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:24.884569  9747 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.957183 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1535) took 1047 ms. Trace:
I20260709 12:25:24.886122  9638 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:24.689936 10049 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:24.919740 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:24.961055 31676 raft_consensus.cc:2368] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31.
I20260709 12:25:24.961997 31819 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:24.820534 10105 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.934962 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1536) took 1024 ms. Trace:
I20260709 12:25:24.916383 10033 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:24.921077 31826 rpcz_store.cc:276] 0709 12:25:23.602144 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.498149 (+896005us) service_pool.cc:224] Handling call
0709 12:25:24.545217 (+ 47068us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.545457 (+   240us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:24.569143 (+ 23686us) log.cc:844] Serialized 14 byte log entry
0709 12:25:24.811331 (+242188us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:24.811386 (+    55us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:24.811429 (+    43us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.811488 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.811689 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":15742,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":4968,"raft.queue_time_us":911,"raft.run_cpu_time_us":174,"raft.run_wall_time_us":6191,"thread_start_us":241702,"threads_started":1}
I20260709 12:25:24.971479 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:24.938603 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:24.977833 31744 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:24.978057 31823 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:24.939220 10068 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:24.939806 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:24.949963 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.939232 32375 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 36. Current term is 37. Ops: []
I20260709 12:25:24.873615 10010 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.881511  9942 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.947294 10066 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:24.894129  9756 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.907056 10144 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:24.952943 10110 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.000s
I20260709 12:25:24.953627 10127 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:24.954089 10128 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:24.954964 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:24.958432 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:24.950817 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:24.962486 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:24.997752 31820 raft_consensus.cc:2468] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:24.844378 10063 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.036473 10063 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:25.039319 31750 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:25:25.040561 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:24.957630 31745 rpcz_store.cc:276] 0709 12:25:23.910181 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.502477 (+592296us) service_pool.cc:224] Handling call
0709 12:25:24.526423 (+ 23946us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.526702 (+   279us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:25:24.526726 (+    24us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:24.527193 (+   467us) log.cc:844] Serialized 26 byte log entry
0709 12:25:24.841267 (+314074us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:24.841326 (+    59us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.841438 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:24.956947 (+115509us) raft_consensus.cc:1685] finished
0709 12:25:24.956975 (+    28us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:24.957119 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":264154,"threads_started":1,"wal-append.queue_time_us":397159}
I20260709 12:25:24.964342 31753 rpcz_store.cc:276] 0709 12:25:23.910448 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.584242 (+673794us) service_pool.cc:224] Handling call
0709 12:25:24.597634 (+ 13392us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.934516 (+336882us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.934880 (+   364us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:24.935539 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 111410us
I20260709 12:25:24.967881 10062 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:24.901880 10001 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:24.942968 10111 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.040s
I20260709 12:25:24.957613  9747 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:24.971395  9413 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:24.971930 10014 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:24.973773 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 3318) took 1186 ms. Trace:
I20260709 12:25:24.974323 10079 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:24.974225 31671 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:24.976389 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:24.983673 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:24.943434 10025 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election lost for term 34. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35.
I20260709 12:25:24.985718 10158 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:24.986083 10068 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:24.948709 10048 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 36 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:24.996008 10127 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:25.001262 10105 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:24.983186 10033 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:25.003692 10144 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:25.012966 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3324) took 1148 ms. Trace:
I20260709 12:25:25.019562 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:25.024057 10090 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:25.025820 10074 raft_consensus.cc:3055] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:25.026681 10073 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:25.033808 10066 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:25.042984 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:25.044507 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:25.097550 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 164890us
I20260709 12:25:25.047703 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:25.047993 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48894.3us
I20260709 12:25:25.050375 31746 rpcz_store.cc:276] 0709 12:25:23.787790 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.373005 (+585215us) service_pool.cc:224] Handling call
0709 12:25:24.973559 (+600554us) spinlock_profiling.cc:243] Waited 347 ms on lock 0x7b54001a4fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:24.973708 (+   149us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":797419776}
W20260709 12:25:25.075105 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 91510us
I20260709 12:25:25.099822 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.076673 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:25.083088 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1903) took 1356 ms. Trace:
I20260709 12:25:25.095392 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:25.050168  9413 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:25.091777 31819 raft_consensus.cc:2468] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:25.091403 31751 rpcz_store.cc:276] 0709 12:25:23.864967 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.478508 (+613541us) service_pool.cc:224] Handling call
0709 12:25:24.478886 (+   378us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:24.486796 (+  7910us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:24.486829 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:24.519719 (+ 32890us) log.cc:844] Serialized 26 byte log entry
0709 12:25:24.881863 (+362144us) raft_consensus.cc:1645] Marking committed up to 13
0709 12:25:24.881928 (+    65us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.882029 (+   101us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:25.012595 (+130566us) raft_consensus.cc:1685] finished
0709 12:25:25.012631 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.012872 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":23336,"raft.queue_time_us":7626,"raft.run_cpu_time_us":149,"raft.run_wall_time_us":12453,"thread_start_us":318436,"threads_started":1,"wal-append.queue_time_us":437168}
I20260709 12:25:25.104571 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:25.106529 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 63653us
I20260709 12:25:25.107384 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.098605 31825 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:25.109444 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.109792 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.111023 31820 raft_consensus.cc:1240] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:25.111757 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1904) took 1384 ms. Trace:
I20260709 12:25:25.112071 31820 rpcz_store.cc:276] 0709 12:25:23.727523 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.110462 (+1382939us) service_pool.cc:224] Handling call
0709 12:25:25.110841 (+   379us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.111477 (+   636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.111700 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.117852 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:25.083609 10127 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:25.126813 31825 raft_consensus.cc:1275] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 33 index: 21. (index mismatch)
I20260709 12:25:25.134441 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1906) took 1016 ms. Trace:
I20260709 12:25:25.134811 31825 rpcz_store.cc:276] 0709 12:25:24.118137 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.116149 (+998012us) service_pool.cc:224] Handling call
0709 12:25:25.116441 (+   292us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.134170 (+ 17729us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.134386 (+   216us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.050777 10040 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.053s
I20260709 12:25:25.092955 10074 raft_consensus.cc:740] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:25.147549 32373 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1445) took 1214 ms. Trace:
I20260709 12:25:25.129930 31791 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:25.136835 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:25.152485 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1905) took 1034 ms. Trace:
I20260709 12:25:25.128554 31744 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:25.137189 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:25.148634 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.168485 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:25.142966 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.136909 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:25.166884 31819 rpcz_store.cc:276] 0709 12:25:24.117751 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.111589 (+993838us) service_pool.cc:224] Handling call
0709 12:25:25.151785 (+ 40196us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.152035 (+   250us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:25.152073 (+    38us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:25.152129 (+    56us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:25.152175 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.152246 (+    71us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.152405 (+   159us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.174294 31744 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:25.169396 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 59805.6us
I20260709 12:25:25.082175 10161 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:25.178170 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48323.9us
I20260709 12:25:25.165547 32373 rpcz_store.cc:276] 0709 12:25:23.916277 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.918807 (+  2530us) service_pool.cc:224] Handling call
0709 12:25:23.919127 (+   320us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.800103 (+880976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.130154 (+330051us) spinlock_profiling.cc:243] Waited 760 ms on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:25.130435 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1748718464}
I20260709 12:25:25.179672 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.181351 31751 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 31: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:25.183521 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:25.184949 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:25.154045 31718 connection.cc:256] RPC call timeout handler was delayed by -1.4028480090000812s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1517s for a response
I20260709 12:25:25.187634 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1444) took 1271 ms. Trace:
W20260709 12:25:25.189800 31718 connection.cc:256] RPC call timeout handler was delayed by -1.3920480090000638s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1529s for a response
W20260709 12:25:25.190173 31718 connection.cc:256] RPC call timeout handler was delayed by -1.3092480089999299s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16210000000000002s for a response
I20260709 12:25:25.115778 10073 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:25.188010 32377 rpcz_store.cc:276] 0709 12:25:23.915959 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.919391 (+  3432us) service_pool.cc:224] Handling call
0709 12:25:23.919651 (+   260us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.243776 (+324125us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.187414 (+943638us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b5400427f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:25.187575 (+   161us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":626327424}
I20260709 12:25:25.115053 10062 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:25.097038  9748 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:25.101059 31821 rpcz_store.cc:276] 0709 12:25:23.727147 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.610478 (+883331us) service_pool.cc:224] Handling call
0709 12:25:24.612313 (+  1835us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.612525 (+   212us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:25:24.743457 (+130932us) log.cc:844] Serialized 14 byte log entry
0709 12:25:25.081209 (+337752us) log.cc:844] Serialized 14 byte log entry
0709 12:25:25.082573 (+  1364us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:25.082661 (+    88us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:25:25.082714 (+    53us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.082811 (+    97us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.082994 (+   183us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":153079,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":43,"raft.queue_time_us":2472,"raft.run_cpu_time_us":191,"raft.run_wall_time_us":14324,"spinlock_wait_cycles":35484928,"thread_start_us":331598,"threads_started":1}
I20260709 12:25:25.148717 10079 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:25:25.287492 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.271819 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:25.262930 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:25.307580 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.234205 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4270) took 1632 ms. Trace:
I20260709 12:25:25.239884 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.291635 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.251554 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:25.241652 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:25.229095 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 3314) took 1458 ms. Trace:
I20260709 12:25:25.312048 31747 rpcz_store.cc:276] 0709 12:25:23.775253 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.159848 (+384595us) service_pool.cc:224] Handling call
0709 12:25:25.228880 (+1069032us) spinlock_profiling.cc:243] Waited 591 ms on lock 0x7b54003e30d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:25.229030 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1359922432}
I20260709 12:25:25.237249 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.324131 31675 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:25.187388 10048 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.314227 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:25.147794 10144 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:25.314781 31825 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 35: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:25.315975 31674 raft_consensus.cc:2393] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 29: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:25.228188 10102 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:25.308501 31824 rpcz_store.cc:276] 0709 12:25:23.601428 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.440730 (+839302us) service_pool.cc:224] Handling call
0709 12:25:24.469631 (+ 28901us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.796678 (+327047us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.233871 (+437193us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b54001969d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:25.234104 (+   233us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":687491456}
I20260709 12:25:25.324940 31821 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:25:25.325433 31673 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:25.274412 31744 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
W20260709 12:25:25.338989 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.131620 10158 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:25.330134 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:25.347333 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:25.347992 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.348423 31751 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:25.182524 10111 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:25.182731 10161 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.347421 31746 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:25.332060 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.350742 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:25.360450 31825 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:25.354868 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:25.362057 31677 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 32: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:25:25.314694 10166 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.128s	user 0.000s	sys 0.000s
I20260709 12:25:25.191016  9757 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:25.355480 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:25.369485 31744 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:25.356710 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:25.370971 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:25.371520 31747 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:25.189303 10076 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:25.371716 31818 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:25.362804 31673 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 24 index: 17. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:25.377367 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:25.377748 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:25.252162 10068 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:25.364959 31675 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 36. Current term is 37. Ops: []
I20260709 12:25:25.365034 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:25.192894 10145 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.173s
I20260709 12:25:25.366074 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:25.226214  9998 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.224753 10164 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.373394 10048 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:25.193854  9586 consensus_queue.cc:1048] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 22, Time since last communication: 0.180s
W20260709 12:25:25.375005 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 184352us
I20260709 12:25:25.221990 10010 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:25.229547 10001 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:25.321805 10073 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:25.389395 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:25.390185 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:25.393507 31745 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:25.298130  9871 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:25.331130  9756 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:25.389468 10149 consensus_queue.cc:1048] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.043s
I20260709 12:25:25.332401 10102 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:25.189558 10143 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 35.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.186s
I20260709 12:25:25.425532 10144 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.446916 31746 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:25.420921 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:25.421975 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:25.473327 31821 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
W20260709 12:25:25.428718 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:25.430814 31718 connection.cc:256] RPC call timeout handler was delayed by -1.191348009000194s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17520000000000002s for a response
W20260709 12:25:25.475332 31718 connection.cc:256] RPC call timeout handler was delayed by -1.1760480090001693s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1769s for a response
I20260709 12:25:25.251932 10049 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.476413 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:25.477519 10049 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:25.435983 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:25.437227 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:25.438401 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.478968 31745 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:25.479427 31673 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:25.479797 31718 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.407301 10079 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.444823 31744 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:25.414202  9757 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:25.417238 31819 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 31: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:25.419788 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:25.427345 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:25.368099 10161 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:25.375444 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:25.481060 31822 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:25.392457 10174 consensus_queue.cc:1048] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.081s
I20260709 12:25:25.484741 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.436514 10073 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:25.494424 31747 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:25.491441 31717 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:25.495590 31673 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 31 index: 15. Preceding OpId from leader: term: 36 index: 18. (index mismatch)
I20260709 12:25:25.513036 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:25.499987 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 72742.7us
I20260709 12:25:25.442411 10062 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 36 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:25.511745 31744 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:25.512892 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:25.514657 31821 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 24.
I20260709 12:25:25.520792 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.428660  9998 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:25.521149 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3327) took 1617 ms. Trace:
I20260709 12:25:25.524036 31751 raft_consensus.cc:1240] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:25.632617 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.480717 10158 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:25.633383 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.389225 10074 consensus_queue.cc:260] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.522346 10014 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:25.530830 31747 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
W20260709 12:25:25.601924 31791 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.193:41935 timed out after 1.688s (SENT) [suppressed 8 similar messages]
I20260709 12:25:25.432564 10164 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:25.500110 10157 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:25.444626  9871 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.442975 10171 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.093s
I20260709 12:25:25.657107 31824 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:25.634044 10143 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 24 FOLLOWER]: Advancing to term 25
W20260709 12:25:25.528973 10170 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.122s	user 0.000s	sys 0.000s
I20260709 12:25:25.633456 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3328) took 1695 ms. Trace:
I20260709 12:25:25.525219  9638 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:25.523563  9747 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:25.669718  9638 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.670162  9638 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:25.529760 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1538) took 1258 ms. Trace:
I20260709 12:25:25.675100 31744 rpcz_store.cc:276] 0709 12:25:24.269350 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.511159 (+1241809us) service_pool.cc:224] Handling call
0709 12:25:25.511565 (+   406us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.527696 (+ 16131us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.528105 (+   409us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.632575 31749 rpcz_store.cc:276] 0709 12:25:23.903975 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.501691 (+597716us) service_pool.cc:224] Handling call
0709 12:25:24.527586 (+ 25895us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:24.584902 (+ 57316us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:24.584941 (+    39us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:24.586318 (+  1377us) log.cc:844] Serialized 52 byte log entry
0709 12:25:24.978834 (+392516us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:25:25.177226 (+198392us) log.cc:844] Serialized 14 byte log entry
0709 12:25:25.227705 (+ 50479us) log.cc:844] Serialized 14 byte log entry
0709 12:25:25.227979 (+   274us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.228061 (+    82us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:25.520922 (+292861us) raft_consensus.cc:1685] finished
0709 12:25:25.520945 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.521088 (+   143us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":42928,"prepare.queue_time_us":727958,"prepare.run_cpu_time_us":83,"prepare.run_wall_time_us":84,"raft.queue_time_us":104480,"raft.run_cpu_time_us":307,"raft.run_wall_time_us":10180,"thread_start_us":508340,"threads_started":3,"wal-append.queue_time_us":509730}
I20260709 12:25:25.677273 31749 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:25.677992 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1541) took 1029 ms. Trace:
I20260709 12:25:25.526247  9792 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:25.641494 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:25.679270 31819 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:25.643389 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1540) took 1028 ms. Trace:
I20260709 12:25:25.682819 31747 rpcz_store.cc:276] 0709 12:25:24.615413 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.527481 (+912068us) service_pool.cc:224] Handling call
0709 12:25:25.527820 (+   339us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.643019 (+115199us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.643308 (+   289us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.485308 10144 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.687697  9998 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:25.688836  9748 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:25.688552 10056 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:25.646833 10014 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.540060 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:25.533751 10145 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:25.500926 10079 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.482543 10001 consensus_queue.cc:1059] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.158s
I20260709 12:25:25.659364 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1909) took 1294 ms. Trace:
W20260709 12:25:25.641698 10147 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.126s	user 0.001s	sys 0.000s
I20260709 12:25:25.664609 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.640657 10158 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:25.667783 31751 rpcz_store.cc:276] 0709 12:25:23.938462 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.495976 (+1557514us) service_pool.cc:224] Handling call
0709 12:25:25.523861 (+ 27885us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.633180 (+109319us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.633394 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:25.668771 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:25.669294 10184 log.cc:927] Time spent T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.183s	user 0.001s	sys 0.000s
W20260709 12:25:25.654219 31750 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.536s	user 0.007s	sys 0.000s
I20260709 12:25:25.678308 31749 rpcz_store.cc:276] 0709 12:25:24.648787 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.676683 (+1027896us) service_pool.cc:224] Handling call
0709 12:25:25.677084 (+   401us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.677731 (+   647us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.677932 (+   201us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.711061 31750 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 37.
I20260709 12:25:25.711620 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 4264) took 1017 ms. Trace:
I20260709 12:25:25.711928 31750 rpcz_store.cc:276] 0709 12:25:24.693748 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.117610 (+423862us) service_pool.cc:224] Handling call
0709 12:25:25.711557 (+593947us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:25.712460 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 60197.8us
I20260709 12:25:25.713194 31750 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:25.713877 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4262) took 1020 ms. Trace:
I20260709 12:25:25.715160 31750 rpcz_store.cc:276] 0709 12:25:24.693441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.712673 (+1019232us) service_pool.cc:224] Handling call
0709 12:25:25.713027 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.713605 (+   578us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.713819 (+   214us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.697175 31825 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:25.694104  9865 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:25.716966 31750 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:25.641295 10186 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:25.686503 10144 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:25.706393 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 166462us
I20260709 12:25:25.724452 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.707619 31821 rpcz_store.cc:276] 0709 12:25:24.365067 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.643078 (+1278011us) service_pool.cc:224] Handling call
0709 12:25:25.658712 (+ 15634us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.658946 (+   234us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:25.658986 (+    40us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:25.659043 (+    57us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:25.659088 (+    45us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.659162 (+    74us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.659302 (+   140us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.645313 10059 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:25:25.716742 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:25.648790 10185 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.721374 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:25.691289 10157 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:25.705464 31819 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:25:25.732163 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1910) took 1356 ms. Trace:
I20260709 12:25:25.732492 31819 rpcz_store.cc:276] 0709 12:25:24.376017 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.731416 (+1355399us) service_pool.cc:224] Handling call
0709 12:25:25.731687 (+   271us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.731837 (+   150us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:25.731873 (+    36us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:25.731932 (+    59us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:25.731962 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.732011 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.732106 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.725710 31825 raft_consensus.cc:2393] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:25:25.366739 10058 consensus_queue.cc:1175] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5206fd0087404d2da964a5d985a04ff4" responder_term: 37 status { last_received { term: 36 index: 21 } last_committed_idx: 20 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 36. Current term is 37. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260709 12:25:25.735044 10173 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:25.737711 10164 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:25.750391 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.730373 10059 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:25.752547 10059 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.753656 10059 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:25.754891 10185 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.757470 10185 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:25.729888 10186 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.759925  9871 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:25.760186 10186 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:25.761672 31825 raft_consensus.cc:1275] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:25:25.714051 10145 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:25.764644 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.719719 10136 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.765520 10136 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.765935 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.632067 10102 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 28.15, Last appended by leader: 15, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:25.767896 10136 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:25.768877 31827 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 4043) took 2241 ms (client timeout 1621 ms). Trace:
W20260709 12:25:25.769306 31827 rpcz_store.cc:269] 0709 12:25:23.527710 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.674947 (+147237us) service_pool.cc:224] Handling call
0709 12:25:25.768670 (+2093723us) spinlock_profiling.cc:243] Waited 1.13 s on lock 0x7b54000fcb54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:25.768775 (+   105us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2587817856}
I20260709 12:25:25.770951 31827 raft_consensus.cc:1275] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
W20260709 12:25:25.773941  9413 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.627s	user 0.010s	sys 0.000s
I20260709 12:25:25.774370  9413 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.774966 10073 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:25.775740 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:25.776739 31670 raft_consensus.cc:2368] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32.
I20260709 12:25:25.777108 10149 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:25.777792 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.705830 10014 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:25.781105  9413 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:25.715420 10037 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:25:25.774243 31672 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.709368 10074 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:25.766541 31675 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
W20260709 12:25:25.779646 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 266820us
I20260709 12:25:25.787475 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:25.725759 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4267) took 1031 ms. Trace:
I20260709 12:25:25.788326 31821 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 31: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:25.780943 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1664) took 1355 ms. Trace:
I20260709 12:25:25.788546 31750 rpcz_store.cc:276] 0709 12:25:24.694301 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.716481 (+1022180us) service_pool.cc:224] Handling call
0709 12:25:25.716794 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.718727 (+  1933us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.725699 (+  6972us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.667428 10010 consensus_queue.cc:1059] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 35.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.056s
I20260709 12:25:25.782239 31747 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 32 index: 21. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:25:25.764825 10145 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:25.777679 10149 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:25.790864 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:25.789184 32370 rpcz_store.cc:276] 0709 12:25:24.425508 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.425640 (+   132us) service_pool.cc:224] Handling call
0709 12:25:24.425919 (+   279us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:24.562812 (+136893us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.780706 (+1217894us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b54004cfed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:25.780888 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":287155968}
W20260709 12:25:25.782182 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 180330us
I20260709 12:25:25.660024  9757 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 28.19, Last appended by leader: 19, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.729259  9998 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:25.778417 10076 raft_consensus.cc:3055] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:25.808219 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:25.780717 10180 log.cc:927] Time spent T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.051s	user 0.000s	sys 0.005s
W20260709 12:25:25.825004 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.784770  9907 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:25.776892 10073 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:25.820556 31675 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 36 LEADER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 35. Current term is 36. Ops: [35.21-35.21]
W20260709 12:25:25.797909 10175 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.360s	user 0.001s	sys 0.000s
I20260709 12:25:25.827426 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:25.817538 31821 raft_consensus.cc:1240] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:25.833088 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4283) took 1037 ms. Trace:
I20260709 12:25:25.833449 31821 rpcz_store.cc:276] 0709 12:25:24.795476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.788928 (+993452us) service_pool.cc:224] Handling call
0709 12:25:25.817283 (+ 28355us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.832834 (+ 15551us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.833030 (+   196us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.704928 10062 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 35.20, Last appended by leader: 20, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:25.830761 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.853971 10066 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.854671 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:25.854578 10066 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.858026 10066 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:25.819211 10076 raft_consensus.cc:740] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:25.841365 10079 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:25.843990 10161 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:25.859494 10076 consensus_queue.cc:260] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:25.860798 10076 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:25.861876 10161 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:25.827574  9998 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:25.862991 10149 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.864560  9998 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:25.847291 10127 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.679s	user 0.011s	sys 0.000s
I20260709 12:25:25.865465 10127 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:25.868101 10127 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:25.869089 10033 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.784s	user 0.012s	sys 0.000s
I20260709 12:25:25.852425 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:25.869554 10033 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.828475 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:25.871814 10033 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:25.872896 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:25.875169 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:25.877187 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:25.879060 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:25.881157 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:25.887171 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.888263 31675 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 32: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:25.891486 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:25.893303 31752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 3330) took 1910 ms (client timeout 2226 ms). Trace:
W20260709 12:25:25.893770 31752 rpcz_store.cc:269] 0709 12:25:23.983711 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.097016 (+113305us) service_pool.cc:224] Handling call
0709 12:25:25.893077 (+1796061us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b5400294ad4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:25.893239 (+   162us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1733676416}
I20260709 12:25:25.783813  9638 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.894919 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:25.896232 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:25.824873 10157 raft_consensus.cc:2764] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election decision vote started in defunct term 29: won
I20260709 12:25:25.896909 31752 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:25:25.897780 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:25.900031 31791 leader_election.cc:400] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32.
I20260709 12:25:25.900521 31791 leader_election.cc:403] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Cancelling election due to peer responding with higher term
W20260709 12:25:25.901311 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:25.901417 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:25.828814 10037 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.903628 31752 raft_consensus.cc:1240] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:25.904297 10140 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:25.904263 10037 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.904956 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4269) took 1210 ms. Trace:
I20260709 12:25:25.905740 31752 rpcz_store.cc:276] 0709 12:25:24.694617 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.902671 (+1208054us) service_pool.cc:224] Handling call
0709 12:25:25.903276 (+   605us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.904442 (+  1166us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.904864 (+   422us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.907003 32377 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
I20260709 12:25:25.902535  9998 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:25.908170 31752 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:25.908795 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4270) took 1213 ms. Trace:
I20260709 12:25:25.909116 31752 rpcz_store.cc:276] 0709 12:25:24.694927 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.907652 (+1212725us) service_pool.cc:224] Handling call
0709 12:25:25.908006 (+   354us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.908570 (+   564us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.908745 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.910280 31752 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:25.910840 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4271) took 1215 ms. Trace:
I20260709 12:25:25.911018 10037 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:25.914531 10187 consensus_queue.cc:1048] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:25.903469 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.916170 10186 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:25.916608 10186 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:25.916991 10186 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:25.852028  9907 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.918124 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1912) took 1269 ms. Trace:
I20260709 12:25:25.918531 31825 rpcz_store.cc:276] 0709 12:25:24.648351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.734242 (+1085891us) service_pool.cc:224] Handling call
0709 12:25:25.734487 (+   245us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.762079 (+ 27592us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.917861 (+155782us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b54000fa854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:25.918059 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":61672832}
I20260709 12:25:25.919813 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:25.920204  9907 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:25.920426 31825 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:25.913062 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
W20260709 12:25:25.913933 10182 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.254s	user 0.000s	sys 0.001s
I20260709 12:25:25.921545 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.922652 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.896534  9638 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:25.924623 31676 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 3146) took 1703 ms (client timeout 2226 ms). Trace:
W20260709 12:25:25.925132 31676 rpcz_store.cc:269] 0709 12:25:24.221052 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.262496 (+ 41444us) service_pool.cc:224] Handling call
0709 12:25:25.924408 (+1661912us) spinlock_profiling.cc:243] Waited 650 ms on lock 0x7b540036f6d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:25.924542 (+   134us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1495081728}
I20260709 12:25:25.927127 31822 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 4278) took 1139 ms. Trace:
I20260709 12:25:25.927547 31822 rpcz_store.cc:276] 0709 12:25:24.787434 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:24.899875 (+112441us) service_pool.cc:224] Handling call
0709 12:25:25.926897 (+1027022us) spinlock_profiling.cc:243] Waited 578 ms on lock 0x7b5400196ed4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:25.927065 (+   168us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1328742272}
I20260709 12:25:25.927966  9998 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election lost for term 31. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32.
I20260709 12:25:25.928766 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 25 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:25.929064 10143 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:25.929356 31822 raft_consensus.cc:2376] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25. Re-sending same reply.
I20260709 12:25:25.931046 31791 leader_election.cc:400] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31.
I20260709 12:25:25.931473 31791 leader_election.cc:403] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:25.933683 31818 raft_consensus.cc:1275] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 35 index: 17. Preceding OpId from leader: term: 36 index: 18. (index mismatch)
I20260709 12:25:25.935922 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4272) took 1240 ms. Trace:
I20260709 12:25:25.936332 31753 rpcz_store.cc:276] 0709 12:25:24.695537 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.935133 (+1239596us) service_pool.cc:224] Handling call
0709 12:25:25.935401 (+   268us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.935584 (+   183us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:25.935616 (+    32us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:25.935664 (+    48us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:25.935702 (+    38us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.935757 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.935852 (+    95us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.937930 31753 raft_consensus.cc:1240] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: [31.18-31.18]
I20260709 12:25:25.938045 31670 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:25.938683 31670 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:25.940281  9942 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:25.940876  9942 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 30. Reason: Vote denied by peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935) with higher term. Message: Invalid argument: T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 30. Current term is 31.
I20260709 12:25:25.944144 10056 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:25.947018  9998 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:25.947618  9998 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:25.911080 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:25.950713 31678 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.698s	user 0.023s	sys 0.010s
W20260709 12:25:25.951563 31678 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.699s	user 0.023s	sys 0.010s
I20260709 12:25:25.951731 31670 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:25.911161 31752 rpcz_store.cc:276] 0709 12:25:24.695228 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.909791 (+1214563us) service_pool.cc:224] Handling call
0709 12:25:25.910124 (+   333us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.910603 (+   479us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.910780 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.930928 31820 raft_consensus.cc:1240] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:25.952680 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.951737 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.954922 31818 raft_consensus.cc:1275] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 26 index: 18. Preceding OpId from leader: term: 28 index: 19. (index mismatch)
I20260709 12:25:25.955010 31821 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:25.955626 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:25.938535 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4273) took 1238 ms. Trace:
I20260709 12:25:25.956537 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:25.957386 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:25.959759 31753 rpcz_store.cc:276] 0709 12:25:24.700469 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.937358 (+1236889us) service_pool.cc:224] Handling call
0709 12:25:25.937693 (+   335us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:25.938295 (+   602us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.938484 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.960608 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.959650 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4285) took 1030 ms. Trace:
I20260709 12:25:25.961560 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:25.961763 31821 rpcz_store.cc:276] 0709 12:25:24.928868 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.834166 (+905298us) service_pool.cc:224] Handling call
0709 12:25:25.954763 (+120597us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.959343 (+  4580us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:25.959584 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:25.962597 31825 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:25:25.964452 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.966418 32372 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33954 (request call id 1513) took 2420 ms. Trace:
I20260709 12:25:25.969033 31673 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:25.970466 31818 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 32 index: 16. Preceding OpId from leader: term: 34 index: 17. (index mismatch)
I20260709 12:25:25.971619 32370 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:25:25.972898 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:25.977185 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.980782 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:25.982342 31752 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:25.985447 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:25.986313 31821 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:25.983049 32342 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:25.994266 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:25.966969 32372 rpcz_store.cc:276] 0709 12:25:23.530531 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.549579 (+ 19048us) service_pool.cc:224] Handling call
0709 12:25:23.657662 (+108083us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:23.659966 (+  2304us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:25:23.785285 (+125319us) log.cc:844] Serialized 14 byte log entry
0709 12:25:23.847908 (+ 62623us) log.cc:844] Serialized 14 byte log entry
0709 12:25:24.360920 (+513012us) spinlock_profiling.cc:243] Waited 2.3 ms on lock 0x7b18004efdc0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:24.617113 (+256193us) spinlock_profiling.cc:243] Waited 4.61 ms on lock 0x7b18004efdc0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ccae07 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:25:24.874097 (+256984us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:24.874159 (+    62us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:25:24.874211 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:24.874279 (+    68us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:25.930572 (+1056293us) spinlock_profiling.cc:243] Waited 108 ms on lock 0x7b5400544f50. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:25.951265 (+ 20693us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":129883,"prepare.run_cpu_time_us":59,"prepare.run_wall_time_us":62,"raft.queue_time_us":21226,"raft.run_cpu_time_us":288,"raft.run_wall_time_us":26537,"spinlock_wait_cycles":263646464,"thread_start_us":430864,"threads_started":3,"wal-append.queue_time_us":190937,"wal-append.run_cpu_time_us":1672,"wal-append.run_wall_time_us":1038054}
I20260709 12:25:25.996120 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.996368 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:25.997865 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.999969 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:25.965195 10025 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:25.970520 10013 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:26.003245 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:25.965307 10157 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election lost for term 25. Reason: could not achieve majority
I20260709 12:25:26.005285 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:26.002844 32372 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 36: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:25:26.015694 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.015810 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 25 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:25.984408  9942 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:26.017582 32371 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 37: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:25.935922 10186 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:25.997510 10127 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:25.997246 10187 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 14, Time since last communication: 0.030s
I20260709 12:25:26.027904 10025 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:26.049453 10025 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:26.005028 10171 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:26.010565 10145 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:26.002300 10079 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:26.002432 10161 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:26.006677 10152 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:26.045085 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1908) took 1915 ms. Trace:
I20260709 12:25:26.026882  9748 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader election won for term 25
I20260709 12:25:26.006923 31792 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:26.009275 10013 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:25.974742 10158 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:26.056686 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:26.020112 31821 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:26.041736 10127 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:26.041043 10048 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.607s	user 0.007s	sys 0.003s
I20260709 12:25:26.075596 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4276) took 1301 ms. Trace:
I20260709 12:25:26.010214 10010 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:26.078763 31820 rpcz_store.cc:276] 0709 12:25:24.129178 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.113879 (+984701us) service_pool.cc:224] Handling call
0709 12:25:25.930737 (+816858us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.952846 (+ 22109us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.044796 (+ 91950us) spinlock_profiling.cc:243] Waited 816 ms on lock 0x7b5400196ed0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:26.044999 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1877887360}
I20260709 12:25:26.056099  9998 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:26.022032  9942 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.082208 31825 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 36 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:26.053146 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:26.101032 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.101159 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 94324.6us
I20260709 12:25:26.115752 31674 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 37.
I20260709 12:25:26.116624 31792 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:26.117900 10144 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:26.102919  9413 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.118978  9413 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:26.121560  9413 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:26.109588 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1542) took 1454 ms. Trace:
I20260709 12:25:26.108603 31753 rpcz_store.cc:276] 0709 12:25:24.774551 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.960847 (+1186296us) service_pool.cc:224] Handling call
0709 12:25:25.979958 (+ 19111us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.980207 (+   249us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:25:26.005803 (+ 25596us) log.cc:844] Serialized 14 byte log entry
0709 12:25:26.075046 (+ 69243us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:26.075129 (+    83us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:25:26.075198 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.075299 (+   101us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.075505 (+   206us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17275,"prepare.queue_time_us":29360,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":54,"raft.queue_time_us":17496,"raft.run_cpu_time_us":170,"raft.run_wall_time_us":19682,"thread_start_us":51417,"threads_started":1}
I20260709 12:25:26.110698 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45166 (request call id 3334) took 1238 ms. Trace:
I20260709 12:25:26.122931 31747 rpcz_store.cc:276] 0709 12:25:24.655533 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.683770 (+1028237us) service_pool.cc:224] Handling call
0709 12:25:25.684159 (+   389us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:25.792855 (+108696us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.109257 (+316402us) spinlock_profiling.cc:243] Waited 46 ms on lock 0x7b54004051d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:26.109504 (+   247us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":105751808}
I20260709 12:25:26.123404 31746 rpcz_store.cc:276] 0709 12:25:24.872560 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.108551 (+235991us) service_pool.cc:224] Handling call
0709 12:25:26.110420 (+1001869us) spinlock_profiling.cc:243] Waited 181 ms on lock 0x7b5400212fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:26.110595 (+   175us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":415648128}
W20260709 12:25:26.102756 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46187.8us
W20260709 12:25:26.102962 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.118161 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:26.096448 31752 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:26.126374 31750 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 34. Ops: []
I20260709 12:25:26.078778 10068 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:26.127732 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:26.056598  9756 consensus_queue.cc:1048] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.055s
W20260709 12:25:26.108855 10195 log.cc:927] Time spent T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.071s	user 0.000s	sys 0.001s
I20260709 12:25:26.104789 10164 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:26.102219 10013 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:26.129742 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:26.131079 31746 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 24. Current term is 25. Ops: []
I20260709 12:25:26.076210 10145 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.129091 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:26.133213 31752 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 35. Current term is 36. Ops: []
W20260709 12:25:26.129274 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 103219us
I20260709 12:25:26.139343 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:26.143664 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:26.145955 32369 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:26.122750  9942 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:26.132462 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:26.143030 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4279) took 1365 ms. Trace:
I20260709 12:25:26.147061 31746 rpcz_store.cc:276] 0709 12:25:24.777342 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.130581 (+1353239us) service_pool.cc:224] Handling call
0709 12:25:26.130894 (+   313us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.142670 (+ 11776us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.142953 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:26.131342 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 78271us
I20260709 12:25:26.108304 10079 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:26.102648 10062 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.081414  9792 consensus_queue.cc:1048] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.047s
I20260709 12:25:26.104822 10152 raft_consensus.cc:697] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:26.162976 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:26.163664 31676 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:26.157171 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:26.114645  9748 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:26.144718 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.176122 31753 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:26.140345 10157 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.149900 31792 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:26.145366 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4280) took 1367 ms. Trace:
I20260709 12:25:26.152210 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 28 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:26.082964 10025 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.140756  9756 consensus_queue.cc:1243] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 35.17 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 16
I20260709 12:25:26.156651 31827 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4282) took 1361 ms. Trace:
I20260709 12:25:26.182878 31827 rpcz_store.cc:276] 0709 12:25:24.795193 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.771664 (+976471us) service_pool.cc:224] Handling call
0709 12:25:25.771947 (+   283us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:25.797142 (+ 25195us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:25.797175 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:25.821291 (+ 24116us) log.cc:844] Serialized 26 byte log entry
0709 12:25:26.003460 (+182169us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:26.003500 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.003569 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:26.156426 (+152857us) raft_consensus.cc:1685] finished
0709 12:25:26.156448 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.156586 (+   138us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":24756,"raft.queue_time_us":24928,"raft.run_cpu_time_us":175,"raft.run_wall_time_us":13743,"thread_start_us":172883,"threads_started":1,"wal-append.queue_time_us":186261}
I20260709 12:25:26.185205 31827 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 35. Current term is 36. Ops: [35.21-35.21]
I20260709 12:25:26.185560 31753 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:26.148393 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:26.146836 31818 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:26.132427  9638 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.134150 10191 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:26.127499 10010 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.194182  9638 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:26.195876 10010 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:26.132761  9998 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:26.166266 10014 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.199643 10014 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:26.171101 10199 log.cc:927] Time spent T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:25:26.174695 31676 raft_consensus.cc:2468] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:25:26.148890 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:26.180631 31823 raft_consensus.cc:2376] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28. Re-sending same reply.
I20260709 12:25:26.182945 10171 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.178040 10074 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:26.203135 10177 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.869s	user 0.001s	sys 0.002s
I20260709 12:25:26.186357 10001 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:26.205232 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4261) took 1511 ms. Trace:
I20260709 12:25:26.205840 31751 rpcz_store.cc:276] 0709 12:25:24.693271 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.710571 (+1017300us) service_pool.cc:224] Handling call
0709 12:25:25.722534 (+ 11963us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:25.724448 (+  1914us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:25.724483 (+    35us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:25.734877 (+ 10394us) log.cc:844] Serialized 52 byte log entry
0709 12:25:26.001202 (+266325us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:26.001250 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.001315 (+    65us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:26.204896 (+203581us) raft_consensus.cc:1685] finished
0709 12:25:26.204936 (+    40us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.205149 (+   213us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":557,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":20493,"thread_start_us":265927,"threads_started":1,"wal-append.queue_time_us":369944}
I20260709 12:25:26.176489 10164 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.181299 31752 rpcz_store.cc:276] 0709 12:25:24.777633 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.131725 (+1354092us) service_pool.cc:224] Handling call
0709 12:25:26.132982 (+  1257us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.145129 (+ 12147us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.145315 (+   186us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:26.186282 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4281) took 1408 ms. Trace:
I20260709 12:25:26.216346 31753 rpcz_store.cc:276] 0709 12:25:24.777950 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.176960 (+1399010us) service_pool.cc:224] Handling call
0709 12:25:26.185347 (+  8387us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.185983 (+   636us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.186224 (+   241us) inbound_call.cc:177] Queueing success response
Metrics: {}
W20260709 12:25:26.217284 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 68456.6us
I20260709 12:25:26.216986  9748 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 23.13, Last appended by leader: 13, Current term: 25, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:26.218053 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4284) took 1422 ms. Trace:
I20260709 12:25:26.157456 10079 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.218293 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:26.218845 10079 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:26.218621 31753 rpcz_store.cc:276] 0709 12:25:24.795338 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.217202 (+1421864us) service_pool.cc:224] Handling call
0709 12:25:26.217548 (+   346us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.217724 (+   176us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:26.217748 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:26.217789 (+    41us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:26.217819 (+    30us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.217874 (+    55us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.217984 (+   110us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:26.219588 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:26.220830 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4285) took 1419 ms. Trace:
I20260709 12:25:26.189797 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:26.221450  9998 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:26.221076 10127 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.222167 31642 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:26.157275 10144 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:26.223133 10144 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:26.223582 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:26.191596 10048 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election lost for term 28. Reason: could not achieve majority
I20260709 12:25:26.224400 10046 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:26.187722  9747 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:26.196600  9638 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:26.220793 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:26.223769 10144 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:25:26.196623 10194 log.cc:927] Time spent T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.064s	user 0.000s	sys 0.000s
I20260709 12:25:26.225555 31753 rpcz_store.cc:276] 0709 12:25:24.801408 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.220027 (+1418619us) service_pool.cc:224] Handling call
0709 12:25:26.220358 (+   331us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.220524 (+   166us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:26.220548 (+    24us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:26.220586 (+    38us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:25:26.220615 (+    29us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.220664 (+    49us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.220775 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:26.145705 10068 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.224709 10048 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:26.244714 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:26.224749 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:26.241899 31751 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
W20260709 12:25:26.249214 31671 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43190 (request call id 3142) took 2243 ms (client timeout 1688 ms). Trace:
I20260709 12:25:26.224907 10040 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:26.225056 10046 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:26.164433 10062 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:26.273895 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 49217.9us
I20260709 12:25:26.273492 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4283) took 1495 ms. Trace:
I20260709 12:25:26.284946 31751 rpcz_store.cc:276] 0709 12:25:24.778535 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.207895 (+1429360us) service_pool.cc:224] Handling call
0709 12:25:26.241630 (+ 33735us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.273172 (+ 31542us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.273426 (+   254us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:26.274443 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4286) took 1472 ms. Trace:
I20260709 12:25:26.288548 31752 rpcz_store.cc:276] 0709 12:25:24.801753 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.229311 (+1427558us) service_pool.cc:224] Handling call
0709 12:25:26.273911 (+ 44600us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.274101 (+   190us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:26.274123 (+    22us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:26.274166 (+    43us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:26.274202 (+    36us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.274261 (+    59us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.274389 (+   128us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:26.291639 10033 consensus_queue.cc:1059] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 30.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:26.293323 10033 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:26.294755 10066 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 29.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.001s
I20260709 12:25:26.296530  9907 raft_consensus.cc:3055] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:26.297027  9907 raft_consensus.cc:740] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:26.299688  9857 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:26.300293  9857 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:26.301522  9857 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.277335 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:26.298197  9907 consensus_queue.cc:260] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.308326  9907 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:26.314283  9857 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:26.275869 31791 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:26.323676 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47935.5us
I20260709 12:25:26.186743 10157 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:26.225240  9747 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:26.270397 10191 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:26.327684 31753 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:26.203140 10171 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:26.277477 31671 rpcz_store.cc:269] 0709 12:25:23.984756 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:23.985385 (+   629us) service_pool.cc:224] Handling call
0709 12:25:26.227917 (+2242532us) spinlock_profiling.cc:243] Waited 866 ms on lock 0x7b54002e7f54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:26.228072 (+   155us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1991551744}
I20260709 12:25:26.336160 10068 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:26.272646 10048 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.342036 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4288) took 1539 ms. Trace:
I20260709 12:25:26.331688 10046 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:26.346902 31753 rpcz_store.cc:276] 0709 12:25:24.802351 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.269981 (+1467630us) service_pool.cc:224] Handling call
0709 12:25:26.327479 (+ 57498us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.341651 (+ 14172us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.341954 (+   303us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:26.327966 10144 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:25:26.345417 10203 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.154s	user 0.001s	sys 0.000s
I20260709 12:25:26.238547 10164 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:26.352473 10172 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.929s	user 0.000s	sys 0.003s
I20260709 12:25:26.353255 10164 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:26.354416 10185 raft_consensus.cc:3055] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:25:26.354920 10185 raft_consensus.cc:740] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:26.355752 10185 consensus_queue.cc:260] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.23, Last appended by leader: 23, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:26.357342 10185 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:26.326009  9930 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Leader election won for term 37
W20260709 12:25:26.361053 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.369393 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.392625 31747 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 36 index: 21. (term mismatch)
I20260709 12:25:26.393255 31747 pending_rounds.cc:85] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 20
I20260709 12:25:26.393543 31747 pending_rounds.cc:107] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 35.21
I20260709 12:25:26.393767 31747 raft_consensus.cc:2889] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:26.339145  9747 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:26.402029 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:26.359548 10196 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.079s	user 0.000s	sys 0.000s
I20260709 12:25:26.367465 10048 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:26.367771 10001 raft_consensus.cc:2804] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:25:26.410274 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.361215 10062 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:26.387471 10144 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:26.362828 10157 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:26.423807 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.421319 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:26.433537 32369 raft_consensus.cc:2410] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 36 because replica has last-logged OpId of term: 34 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 22.
I20260709 12:25:26.400126  9930 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 37 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:26.365685 10118 consensus_queue.cc:1059] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 30.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.047s
I20260709 12:25:26.442091 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:26.358620 10079 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:26.472913 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.474360 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 162125us
I20260709 12:25:26.344125 10152 consensus_queue.cc:237] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 28.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:26.477761 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4291) took 1657 ms. Trace:
I20260709 12:25:26.437312 10001 raft_consensus.cc:697] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 32 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:26.484797 31748 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:26.486079 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:26.487227 31748 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:26.491341 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1547) took 1182 ms. Trace:
I20260709 12:25:26.483438 31751 rpcz_store.cc:276] 0709 12:25:24.820683 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.286124 (+1465441us) service_pool.cc:224] Handling call
0709 12:25:26.286598 (+   474us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.286917 (+   319us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:25:26.287323 (+   406us) log.cc:844] Serialized 14 byte log entry
0709 12:25:26.475069 (+187746us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:26.475127 (+    58us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:26.476791 (+  1664us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.477554 (+   763us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.477693 (+   139us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":3655,"prepare.run_cpu_time_us":56,"prepare.run_wall_time_us":58,"raft.queue_time_us":783,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":6901,"thread_start_us":186589,"threads_started":1}
I20260709 12:25:26.497280 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:26.498188 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:26.502419  9942 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.503911 10059 consensus_queue.cc:1048] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:26.504994  9942 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:26.505457 10059 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:26.505915 10059 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:26.506404 10059 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:26.506119 10149 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:26.505872 10066 raft_consensus.cc:3055] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:26.512367 10066 raft_consensus.cc:740] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:25:26.516395 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:26.518714 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.517604 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:26.519275 31671 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:26.520749  9907 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:26.523476 10171 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:26.525069 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.517642 10059 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:26.529160 31671 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:26.517987 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.473881 10145 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:26.491995 31752 rpcz_store.cc:276] 0709 12:25:25.309426 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.289984 (+980558us) service_pool.cc:224] Handling call
0709 12:25:26.290441 (+   457us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.305187 (+ 14746us) raft_consensus.cc:1534] Early marking committed up to index 24
0709 12:25:26.305615 (+   428us) log.cc:844] Serialized 14 byte log entry
0709 12:25:26.489526 (+183911us) log.cc:844] Serialized 14 byte log entry
0709 12:25:26.490997 (+  1471us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:26.491058 (+    61us) raft_consensus.cc:1645] Marking committed up to 24
0709 12:25:26.491106 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.491172 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.491287 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":38261,"prepare.queue_time_us":2283,"prepare.run_cpu_time_us":47,"prepare.run_wall_time_us":49,"raft.queue_time_us":45891,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":22903,"thread_start_us":133558,"threads_started":1}
I20260709 12:25:26.576344 31791 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:26.578876 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:26.581281 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:26.582849 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:26.583581 31827 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
W20260709 12:25:26.584020 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 72431.3us
I20260709 12:25:26.585693 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 33 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.586611 31751 raft_consensus.cc:2410] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 36 because replica has last-logged OpId of term: 34 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 22.
I20260709 12:25:26.490187 10074 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.597482 31672 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34010 (request call id 1704) took 1169 ms. Trace:
I20260709 12:25:26.575749  9998 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:26.611841 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:26.552861 10040 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:26.604537 31672 rpcz_store.cc:276] 0709 12:25:25.428269 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.805287 (+377018us) service_pool.cc:224] Handling call
0709 12:25:25.805759 (+   472us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:25.812681 (+  6922us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:25.812725 (+    44us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:25.828211 (+ 15486us) log.cc:844] Serialized 52 byte log entry
0709 12:25:26.105858 (+277647us) raft_consensus.cc:1645] Marking committed up to 22
0709 12:25:26.105932 (+    74us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.106050 (+   118us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:26.597191 (+491141us) raft_consensus.cc:1685] finished
0709 12:25:26.597220 (+    29us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:26.597420 (+   200us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":111,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":4505,"thread_start_us":209645,"threads_started":1,"wal-append.queue_time_us":332196}
I20260709 12:25:26.606417 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:26.577940 10062 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.606448 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:26.620736 10204 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 12:25:26.583528 10076 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.577591  9942 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:26.626489 31751 raft_consensus.cc:3060] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:26.584578 10164 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:26.652659 10211 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:25:26.681843  9942 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.685638 10164 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.605230 10074 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:26.687386 10210 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:25:26.689109 10074 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:26.597359 10213 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.028s
I20260709 12:25:26.619297 10215 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Stepping down as leader of term 24
I20260709 12:25:26.690413 10215 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:26.691478 10079 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:26.691735 10140 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.787s	user 0.013s	sys 0.000s
I20260709 12:25:26.691500 10215 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 24, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:26.659593 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 160162us
I20260709 12:25:26.597148 10049 raft_consensus.cc:2804] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:25:26.669842 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.577260 10013 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.693552 10215 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:26.694423 10152 consensus_queue.cc:1059] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:26.695405 10013 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:26.695996 10192 raft_consensus.cc:3055] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:26.696121 31746 raft_consensus.cc:1240] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:26.696444 10192 raft_consensus.cc:740] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:26.680548 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.697140 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:26.697415 10192 consensus_queue.cc:260] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:26.681144 10076 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.680195 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.699252 31752 raft_consensus.cc:2393] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:26.700165 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:26.700767 31752 raft_consensus.cc:2376] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29. Re-sending same reply.
I20260709 12:25:26.701663 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.703110 10192 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:26.697933 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.704738 31753 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:26.705969 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.709928 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.713887 10158 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:25:26.677228 10062 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.717756 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.718864 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 25 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.731501 31748 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:26.733438 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:24.671736 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:25:26.724067 10118 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:25:26.698609 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.731634 32369 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:25:26.731819 31752 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 25: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:25:26.719950 31567 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 156) took 57253 ms (client timeout 15000 ms). Trace:
I20260709 12:25:26.725303 10228 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.737742 31748 raft_consensus.cc:2368] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32.
W20260709 12:25:26.723271 10183 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 1.190s	user 0.003s	sys 0.000s
I20260709 12:25:26.700794 10224 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:26.747999 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:26.748996 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.756635 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.757714 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.758289 31752 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:26.759274 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.752913 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:26.762506 31751 raft_consensus.cc:2468] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:26.764909 31751 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:26.773731 31752 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 35 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:25:26.731573 10062 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:26.776777 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 174275us
I20260709 12:25:26.608268 10144 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:26.766588 10224 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:26.845652 10224 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:26.772877 10049 raft_consensus.cc:697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 34 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:25:26.851410 10218 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.174s	user 0.000s	sys 0.000s
I20260709 12:25:26.851895 10049 consensus_queue.cc:237] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.625379 10066 consensus_queue.cc:260] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.814360 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:26.634718  9930 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 36.24, Last appended by leader: 24, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.775269  9865 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.777072 10213 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.675623 10001 consensus_queue.cc:237] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:26.766281 10228 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.838577 10058 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.776712 10158 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:26.834014 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.818899 10079 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.821475 10059 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:26.745306 31567 rpcz_store.cc:269] 0709 12:24:29.466929 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:30.812996 (+1346067us) service_pool.cc:224] Handling call
0709 12:25:26.719823 (+55906827us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:21.429015 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:21.495322 (+ 66307us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:21.495379 (+    57us) write_op.cc:435] Acquired schema lock
0709 12:25:21.495399 (+    20us) tablet.cc:662] Decoding operations
0709 12:25:21.496354 (+   955us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:21.496456 (+   102us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:21.496483 (+    27us) tablet.cc:685] Acquiring locks for 8 operations
0709 12:25:21.497280 (+   797us) tablet.cc:701] Row locks acquired
0709 12:25:21.497334 (+    54us) write_op.cc:260] PREPARE: finished
0709 12:25:21.497526 (+   192us) write_op.cc:270] Start()
0709 12:25:21.497678 (+   152us) write_op.cc:276] Timestamp: P: 1783599921497455 usec, L: 0
0709 12:25:21.497709 (+    31us) op_driver.cc:348] REPLICATION: starting
0709 12:25:21.602793 (+105084us) log.cc:844] Serialized 3302 byte log entry
0709 12:25:21.862210 (+259417us) spinlock_profiling.cc:243] Waited 18.4 ms on lock 0x7b54002828d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd0e28 00007fe90080b581 00007fe90080ad58 00007fe90080f9f1 00007fe90080f9a9 00007fe90080f939 00007fe90080f901 00007fe90080eb2d 0000000000388634 0000000000388479
0709 12:25:22.566956 (+704746us) op_driver.cc:464] REPLICATION: finished
0709 12:25:23.313572 (+746616us) write_op.cc:301] APPLY: starting
0709 12:25:23.313686 (+   114us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:23.832692 (+519006us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:23.843040 (+ 10348us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:23.914526 (+ 71486us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:23.917787 (+  3261us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=16,key_file_lookups=16,delta_file_lookups=0,mrs_lookups=0
0709 12:25:23.917864 (+    77us) write_op.cc:312] APPLY: finished
0709 12:25:23.918287 (+   423us) log.cc:844] Serialized 81 byte log entry
0709 12:25:24.034153 (+115866us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:24.047893 (+ 13740us) write_op.cc:454] Released schema lock
0709 12:25:24.047971 (+    78us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:24.048476 (+   505us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:24.048508 (+    32us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":70424,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":746162,"apply.run_cpu_time_us":29413,"apply.run_wall_time_us":734885,"cfile_cache_hit":34,"cfile_cache_hit_bytes":37551,"num_ops":8,"prepare.queue_time_us":202903,"prepare.run_cpu_time_us":33099,"prepare.run_wall_time_us":477756,"raft.queue_time_us":146753,"raft.run_cpu_time_us":118374,"raft.run_wall_time_us":521317,"replication_time_us":1050434,"spinlock_wait_cycles":42373888,"thread_start_us":819832,"threads_started":5,"wal-append.queue_time_us":401870,"wal-append.run_cpu_time_us":45959,"wal-append.run_wall_time_us":2844478}]]}
I20260709 12:25:26.870070  9865 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:26.893600  9413 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:26.894558  9413 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:26.822642 10040 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:26.905216 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.867767 10219 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:25:26.914987 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.916543  9942 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:26.840302 10048 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.883073 10066 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:26.924767 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54265.1us
I20260709 12:25:26.934865 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:26.945406 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:26.948549 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.949450  9413 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:26.951689 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:26.952776  9413 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:26.873112 10213 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.880287 10058 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:26.956076 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:26.912935 10219 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:26.883948 10158 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.938812 10209 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:26.930051 10229 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:26.881791 10079 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:26.913861  9747 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:26.965611 10111 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:25:26.980073 31675 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:26.975099 31671 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:26.975574 31674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:26.977535 31673 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:26.978082 31678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:26.981695 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:26.995775 31669 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:26.954663 10230 consensus_queue.cc:1059] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.037s
I20260709 12:25:26.996588 31669 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:25:26.998044  9413 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:26.995716 31676 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b" is_pre_election: true
I20260709 12:25:26.998865 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:26.999900 31676 raft_consensus.cc:2410] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 35 because replica has last-logged OpId of term: 34 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 21.
I20260709 12:25:27.002463  9747 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:27.005093 10197 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.876s	user 0.002s	sys 0.001s
I20260709 12:25:27.006800 10228 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:27.007952 10213 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:27.008527 10079 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:27.008952  9857 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.693s	user 0.004s	sys 0.008s
I20260709 12:25:27.009079  9865 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:26.995316 10219 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:27.011687 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:27.009204 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.010473 10235 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:27.012387 31827 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:27.019524 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1915) took 1733 ms. Trace:
I20260709 12:25:27.020066 31825 rpcz_store.cc:276] 0709 12:25:25.286476 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.963766 (+677290us) service_pool.cc:224] Handling call
0709 12:25:25.964743 (+   977us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.115161 (+150418us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.019227 (+904066us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54003ca0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:27.019450 (+   223us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":269498112}
I20260709 12:25:27.020860 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:26.980015 10111 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.029173 10118 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:26.968390 10233 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.048s
I20260709 12:25:27.041147 31677 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:27.044823 31675 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:25:27.025949 10235 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:27.038563 10160 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:27.038316 10111 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:26.970238 10229 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:27.038806 10228 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.065295 31677 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:25:27.066543 31675 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "b95aeeaf4c51442b8092cff1de674f2b"
I20260709 12:25:27.067262 10059 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.067893 10059 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:27.060420 30503 tablet_server.cc:179] TabletServer@127.29.201.193:0 shutting down...
I20260709 12:25:27.072599 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:27.058260 10160 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.054373 10118 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:27.061281 10079 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:25:27.088076 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.090077 31717 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:27.073820 10059 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:27.077977 10160 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:27.083838 10079 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:27.099623 10164 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:27.100811 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4282) took 2322 ms. Trace:
I20260709 12:25:27.101416 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:27.102025 10074 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:27.102792 10013 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: INVALID_TERM, Last received: 31.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.009s
W20260709 12:25:27.103094 10236 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.107s	user 0.001s	sys 0.000s
I20260709 12:25:27.116147 31746 rpcz_store.cc:276] 0709 12:25:24.778239 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.177423 (+1399184us) service_pool.cc:224] Handling call
0709 12:25:26.189577 (+ 12154us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.696599 (+507022us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.100550 (+403951us) spinlock_profiling.cc:243] Waited 506 ms on lock 0x7b54001ed7d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:27.100747 (+   197us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1164467968}
I20260709 12:25:27.163545 31675 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 37.
I20260709 12:25:27.064934 10228 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:27.163236 10152 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.170352 10158 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:27.169494 10235 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:27.176018 10074 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:27.188341 10229 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:27.189200 10224 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:27.193642 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 242424us
I20260709 12:25:27.198776 32342 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:27.199254 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 97929.3us
W20260709 12:25:27.185132 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 250428us
I20260709 12:25:27.207741 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 35 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:27.209043 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:27.209689 31751 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:25:27.210997 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:27.211385 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:27.213371 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:27.215451 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:27.192942 10158 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.221115 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:27.221976 31744 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 37: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:27.194648 10074 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:27.226529 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.227396 10040 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.003s
I20260709 12:25:27.230605 10040 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:27.231264 10040 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:27.232564 31748 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:27.232302 10040 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:27.223033 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:27.186363 10152 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:27.215188 10164 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:27.260123 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:27.261085 31820 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:27.229163 10140 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.024s
I20260709 12:25:27.244985 10253 consensus_queue.cc:1059] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.016s
I20260709 12:25:27.234122 10040 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:27.265272 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:27.223472  9942 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:27.283813 31827 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:27.251641 10079 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:27.252068 10224 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:27.253657 10152 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:27.245472 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:27.297636 31744 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:27.258458 31674 raft_consensus.cc:2410] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 32 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:25:27.246956 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:27.299317 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52605.5us
W20260709 12:25:27.299846 10192 consensus_meta.cc:287] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.596s	user 0.013s	sys 0.000s
I20260709 12:25:27.300911 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 25 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:27.302229 31791 leader_election.cc:400] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32.
I20260709 12:25:27.302909 10191 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:27.303719 10191 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 24 FOLLOWER]: Advancing to term 25
I20260709 12:25:27.264552 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:27.307384 32377 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:25:27.308460 10063 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:27.309082 10063 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:27.310547 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.247624 31568 catalog_manager.cc:5697] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 16 to 18. New cstate: current_term: 18 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:27.263919 10215 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.570s	user 0.010s	sys 0.003s
I20260709 12:25:27.228296 10158 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:27.311266 31643 connection.cc:256] RPC call timeout handler was delayed by -1.2629475320004531s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.17420000000000002s for a response
I20260709 12:25:27.312409 10232 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:27.319304 10232 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:27.323098 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:27.315419 32369 raft_consensus.cc:1240] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:27.324203 10253 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:25:27.325160 10229 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:27.325554 10229 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:27.326112 10229 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:27.315155 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:27.328622 10200 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.140s	user 0.005s	sys 0.000s
I20260709 12:25:27.329298 10079 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:27.330260 10257 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:27.220439 10144 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:27.221190 10049 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:27.284905 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:27.285073 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:27.228550 10074 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:27.192312 10118 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 28.15, Last appended by leader: 15, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:27.220443 10149 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:25:27.295094 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:27.295215 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:27.362234 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.353193 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:27.363914 10252 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.067s	user 0.002s	sys 0.000s
W20260709 12:25:27.365619 10249 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.239s	user 0.000s	sys 0.001s
I20260709 12:25:27.368307 10144 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:27.369457 10206 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.175s	user 0.004s	sys 0.000s
I20260709 12:25:27.369362 10074 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:27.371179 10040 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:27.371678 10040 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:27.372045 10040 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:27.372196 31746 raft_consensus.cc:1240] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 33. Current term is 34. Ops: []
W20260709 12:25:27.347679 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45591.9us
I20260709 12:25:27.351653 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:27.375023 31827 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:27.375763 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4294) took 1725 ms. Trace:
I20260709 12:25:27.376071 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:27.377264 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:27.377821 31827 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:25:27.376302 31824 rpcz_store.cc:276] 0709 12:25:25.650758 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.048399 (+397641us) service_pool.cc:224] Handling call
0709 12:25:26.048846 (+   447us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:26.049276 (+   430us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:26.049318 (+    42us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:26.050049 (+   731us) log.cc:844] Serialized 26 byte log entry
0709 12:25:26.193237 (+143188us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:26.193310 (+    73us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.193417 (+   107us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:27.368584 (+1175167us) spinlock_profiling.cc:243] Waited 37 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:27.375314 (+  6730us) raft_consensus.cc:1685] finished
0709 12:25:27.375346 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.375689 (+   343us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":2303,"raft.run_cpu_time_us":196,"raft.run_wall_time_us":25963,"spinlock_wait_cycles":84995456,"thread_start_us":142687,"threads_started":1,"wal-append.queue_time_us":143209}
I20260709 12:25:27.353647 31752 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:25:27.354875 10228 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:27.355129 31749 raft_consensus.cc:1240] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
W20260709 12:25:27.296880 10244 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.219s	user 0.001s	sys 0.000s
I20260709 12:25:27.370824  9907 consensus_queue.cc:1059] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 35.21, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:27.381502 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:27.383101 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:27.317085 31568 catalog_manager.cc:5697] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 18 to 22. New cstate: current_term: 22 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:27.258586 10062 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:27.385836 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:27.389122 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:27.383910 10049 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:25:27.396166 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 73166.3us
I20260709 12:25:27.396785 32370 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:27.397886 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:27.416108 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:27.385465 10225 raft_consensus.cc:3055] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:25:27.419190 31827 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:27.431242 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:27.432937 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.445253 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:27.452190 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:27.425791 31820 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:25:27.461714 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:27.433964 10235 raft_consensus.cc:2804] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:27.434293 10144 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:27.434471 10225 raft_consensus.cc:740] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 35 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:27.474942 10250 consensus_queue.cc:1048] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:27.473210 10235 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:27.449523 10233 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:27.467554  9930 raft_consensus.cc:2764] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election decision vote started in defunct term 31: won
I20260709 12:25:27.504160 10058 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:27.504752 10063 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:27.504902 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.394225  9998 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.063s
I20260709 12:25:27.444553 10046 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:25:27.505153 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.474157  9638 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:27.517248 10033 raft_consensus.cc:3055] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:27.495308 10245 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:27.432610 10230 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 29.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.043s
I20260709 12:25:27.523113 10228 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:27.427454 10253 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.044s
I20260709 12:25:27.505330 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:27.508838 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:27.512419 31750 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4268) took 2817 ms. Trace:
I20260709 12:25:27.541378 31750 rpcz_store.cc:276] 0709 12:25:24.694461 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.811106 (+1116645us) service_pool.cc:224] Handling call
0709 12:25:26.110596 (+299490us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.126938 (+ 16342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:26.817175 (+690237us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b5400212fd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:27.489642 (+672467us) spinlock_profiling.cc:243] Waited 299 ms on lock 0x7b5400212fd0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:27.512356 (+ 22714us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":723387136}
I20260709 12:25:27.542395 31744 raft_consensus.cc:1240] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:27.519646 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1549) took 1636 ms. Trace:
I20260709 12:25:27.544227 31745 rpcz_store.cc:276] 0709 12:25:25.883175 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.360021 (+476846us) service_pool.cc:224] Handling call
0709 12:25:26.360413 (+   392us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:26.394678 (+ 34265us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:26.394731 (+    53us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:26.404150 (+  9419us) log.cc:844] Serialized 26 byte log entry
0709 12:25:26.609835 (+205685us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:26.609965 (+   130us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.487160 (+877195us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b54004033d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:27.508307 (+ 21147us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:27.508377 (+    70us) raft_consensus.cc:1685] finished
0709 12:25:27.508399 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.519564 (+ 11165us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":13130,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":11277,"spinlock_wait_cycles":31570176,"thread_start_us":169828,"threads_started":1,"wal-append.queue_time_us":271292}
I20260709 12:25:27.512210 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4278) took 2735 ms. Trace:
I20260709 12:25:27.546702 31747 rpcz_store.cc:276] 0709 12:25:24.777032 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.129387 (+1352355us) service_pool.cc:224] Handling call
0709 12:25:26.146508 (+ 17121us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.394256 (+247748us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.486610 (+1092354us) spinlock_profiling.cc:243] Waited 242 ms on lock 0x7b5400404cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:27.512128 (+ 25518us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":555923712}
I20260709 12:25:27.508239 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4296) took 1857 ms. Trace:
I20260709 12:25:27.547997 31826 rpcz_store.cc:276] 0709 12:25:25.651205 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.112282 (+461077us) service_pool.cc:224] Handling call
0709 12:25:26.129125 (+ 16843us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.329681 (+200556us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:26.329712 (+    31us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:26.329760 (+    48us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:26.329787 (+    27us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.488934 (+1159147us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54000fc154. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:27.507985 (+ 19051us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:27.508163 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":460767488}
I20260709 12:25:27.524410 31746 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
W20260709 12:25:27.528281 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.508925 10231 raft_consensus.cc:3055] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:25:27.507218 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 1.35242e+06us
I20260709 12:25:27.505692 10164 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:27.540604 31749 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:27.552744 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:27.506668 10158 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:27.505949 10232 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:27.517076 10066 raft_consensus.cc:3055] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:27.504797 10048 leader_election.cc:290] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:27.456240 10234 raft_consensus.cc:2804] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:27.551414 10186 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:27.531503 10251 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.128s	user 0.002s	sys 0.000s
I20260709 12:25:27.540915 31752 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:27.553814 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.555629 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:27.532953 10033 raft_consensus.cc:740] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:27.523587 10235 raft_consensus.cc:3060] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:27.551957 10225 consensus_queue.cc:260] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:27.576367 10225 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:27.577955 31752 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:27.560290 31819 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:27.536058  9638 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 36 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:27.561887 10234 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:27.579331 10234 raft_consensus.cc:3060] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:27.579736  9638 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.22, Last appended by leader: 22, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:27.554445 10164 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:27.540524 10076 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:27.569660 10242 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.566s	user 0.003s	sys 0.000s
I20260709 12:25:27.572829  9756 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:27.573577 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:27.562398 10213 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 32.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.016s
I20260709 12:25:27.574826 10158 consensus_queue.cc:1059] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 35.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:25:27.556495 10145 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.551331 10231 raft_consensus.cc:740] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:27.557662 10013 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:27.556983 10066 raft_consensus.cc:740] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:27.593502 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:27.590988 10111 consensus_meta.cc:287] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.526s	user 0.009s	sys 0.002s
I20260709 12:25:27.605894  9942 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:27.567880 10232 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.024s
I20260709 12:25:27.585134 10164 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:27.588750  9747 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.588534  9756 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:27.610740 10111 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:27.610785 10229 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:27.611833 10191 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election lost for term 25. Reason: could not achieve majority
I20260709 12:25:27.599674 10079 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:27.616704 10231 consensus_queue.cc:260] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:27.619362 10066 consensus_queue.cc:260] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:27.662065 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:27.662340 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:27.662966 31820 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:27.651081  9756 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:27.633836 10228 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:27.585402 10048 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.025s
W20260709 12:25:27.583048 10245 consensus_queue.cc:1175] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "039d66a6c85e43c3936bf5017501b344" responder_term: 36 status { last_received { term: 35 index: 21 } last_committed_idx: 20 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 35. Current term is 36. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260709 12:25:27.658473 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 433692us
I20260709 12:25:27.649749 10013 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:25:27.674870 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:27.689664 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:27.620085 10234 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:27.677446 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 456851us
I20260709 12:25:27.699218 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:27.674794 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 285702us
W20260709 12:25:27.695717 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 224801us
W20260709 12:25:27.636904 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.700544 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:27.701095 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:27.702647 32369 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:25:27.703313 32370 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:27.645917 10074 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:27.703889 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:27.653652 10209 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:27.706148 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:27.709942 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:27.710304 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:27.785192 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:27.789336 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.792529 32376 raft_consensus.cc:2368] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 37. Current term is 38.
W20260709 12:25:27.793182 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 82965.2us
W20260709 12:25:27.792932 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 86896us
I20260709 12:25:27.604022 10145 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:27.691210 10228 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:27.792516 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:27.682380  9998 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:27.683043 10079 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:27.709216 10191 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:27.703109 10140 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.714161 10231 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:27.651352  9747 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:27.717199  9757 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:27.851744 31746 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
W20260709 12:25:27.848726 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 149559us
I20260709 12:25:27.717864 10046 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:27.805505 10239 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:27.786937 10013 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.785049 10068 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.804044 10118 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:27.717422 10076 consensus_queue.cc:1059] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 20, Last known committed idx: 17, Time since last communication: 0.142s
I20260709 12:25:27.804375 10049 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.857158 31820 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:27.921859 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:27.873281 10040 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:27.854198  9907 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.914402 10234 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:27.915725 10230 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:27.849834 10074 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:25:27.932082 31820 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:27.939936 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:27.866796  9998 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:28.006259 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 66423.1us
I20260709 12:25:28.007377 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:27.999809 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:27.873749 10191 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:27.924427 10186 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:28.013024 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 77608.3us
W20260709 12:25:28.013561 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 129247us
I20260709 12:25:27.884405 10140 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:28.001232 10160 consensus_meta.cc:287] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.849s	user 0.011s	sys 0.005s
I20260709 12:25:28.031183 31820 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:28.036082 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:28.046443 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:27.920639 10068 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:27.920341 10013 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.022126 10076 raft_consensus.cc:3055] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:28.000599 10158 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.047744 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:28.071180 10265 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.051s	user 0.000s	sys 0.000s
I20260709 12:25:28.048765 10079 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.075677  9747 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:27.992671  9907 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.078076 10140 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:28.013165 10102 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:28.080263 32371 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 36: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:25:28.044490 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 238735us
I20260709 12:25:28.047452 10285 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:28.041610 10261 log.cc:927] Time spent T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.438s	user 0.000s	sys 0.001s
I20260709 12:25:28.078770 10234 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:28.163167 10234 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.163671 10234 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:28.164893  9747 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:28.165501  9747 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.166101 10118 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:27.939742 10049 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.167863 10068 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:28.168325 10049 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:28.168550 10068 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.167869  9907 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:28.169129 10068 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.171944 10191 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:28.095006 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.104543 31820 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 36.
I20260709 12:25:28.028719 10074 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 37 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:28.086429 10076 raft_consensus.cc:740] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:25:28.092808 10225 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.516s	user 0.005s	sys 0.005s
I20260709 12:25:28.056193 10239 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:28.080682 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.149413 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.023211  9942 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.088s
I20260709 12:25:28.085196 10048 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:28.034420 10186 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.120438 10158 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.230273 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:28.016826 10230 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.195218 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:28.079964 10160 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:28.161775 10246 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.073s	user 0.000s	sys 0.004s
I20260709 12:25:28.216147 10040 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:28.216287 32376 raft_consensus.cc:3055] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Stepping down as leader of term 31
W20260709 12:25:28.217236 10262 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.331s	user 0.000s	sys 0.001s
I20260709 12:25:28.134802 10111 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:28.086881 10277 consensus_queue.cc:1059] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 29.14, Next index: 15, Last known committed idx: 14, Time since last communication: 0.147s
W20260709 12:25:28.202951 10265 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.067s	user 0.000s	sys 0.000s
I20260709 12:25:28.174502 10229 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:28.184221  9857 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:28.292039  9857 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:28.253159 10286 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:28.293058  9857 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.293080 10160 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:28.299681 10266 log.cc:927] Time spent T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:25:28.294572  9857 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:28.258749 31752 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:28.260098 31820 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:25:28.303159 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.304144 10228 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.261795 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.304728 10219 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:28.263464 10269 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:28.307627 10269 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.310757 10145 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:28.306069 10074 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.311823  9747 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:28.312466  9747 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.227576 10239 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:28.314649 10240 log.cc:927] Time spent T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 1.311s	user 0.016s	sys 0.010s
I20260709 12:25:28.316900 10187 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:28.317554 10187 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:28.312922  9747 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.321568 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1930) took 1273 ms. Trace:
I20260709 12:25:28.323210  9998 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:28.322041 31825 rpcz_store.cc:276] 0709 12:25:27.048045 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.048198 (+   153us) service_pool.cc:224] Handling call
0709 12:25:27.048635 (+   437us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.049124 (+   489us) raft_consensus.cc:1534] Early marking committed up to index 16
0709 12:25:27.198002 (+148878us) log.cc:844] Serialized 14 byte log entry
0709 12:25:27.520205 (+322203us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.520833 (+   628us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.520996 (+   163us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:27.521048 (+    52us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.521151 (+   103us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.321309 (+800158us) raft_consensus.cc:1685] finished
0709 12:25:28.321339 (+    30us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.321489 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":640,"prepare.queue_time_us":144231,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":41,"raft.queue_time_us":2505,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":7885,"spinlock_wait_cycles":513920,"thread_start_us":412701,"threads_started":2,"wal-append.queue_time_us":465500}
I20260709 12:25:28.324643 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.325352 31825 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
W20260709 12:25:28.331769 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.184382 10285 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.256732 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.334306 31746 raft_consensus.cc:2393] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:28.334184 10229 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.334944 10229 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.336215 10066 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:28.285421 10076 consensus_queue.cc:260] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.20, Last appended by leader: 20, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.339427 10076 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:28.276394 32376 raft_consensus.cc:740] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:28.341421 32376 consensus_queue.cc:260] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.289292 32373 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 35: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:28.293678 10286 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.342813 10286 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.349110 10258 raft_consensus.cc:3055] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:25:28.349762 10258 raft_consensus.cc:740] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:28.350860 10258 consensus_queue.cc:260] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.352252 10258 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:28.326320 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.354220 31825 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:25:28.355201 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1911) took 3978 ms. Trace:
I20260709 12:25:28.355618 31819 rpcz_store.cc:276] 0709 12:25:24.376419 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.733317 (+1356898us) service_pool.cc:224] Handling call
0709 12:25:25.733580 (+   263us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:27.578881 (+1845301us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.354933 (+776052us) spinlock_profiling.cc:243] Waited 1.83 s on lock 0x7b54001973d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:28.355111 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":4200576384}
I20260709 12:25:28.356618 31825 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:25:28.360572 10068 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:28.257886 31753 raft_consensus.cc:1275] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:25:28.364665 31670 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43200 (request call id 4268) took 1388 ms. Trace:
W20260709 12:25:28.301812  9756 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.599s	user 0.009s	sys 0.001s
I20260709 12:25:28.304473 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:28.370277  9756 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:28.339551 32342 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:28.371851 10209 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.388067 10209 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.260689 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 26 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.373464 10282 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:28.389632  9756 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:28.390252 10282 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.393432 10187 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.394582 10187 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:28.395920 32376 raft_consensus.cc:3060] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:28.357089 31819 pending_rounds.cc:85] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 16
I20260709 12:25:28.408485 10286 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:28.414076  9757 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.527s	user 0.007s	sys 0.001s
W20260709 12:25:28.416435 10062 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 1.027s	user 0.014s	sys 0.000s
I20260709 12:25:28.428078 10062 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:28.428916 31824 raft_consensus.cc:1240] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:28.433810 31822 raft_consensus.cc:2410] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 29 because replica has last-logged OpId of term: 28 index: 15, which is greater than that of the candidate, which has last-logged OpId of term: 27 index: 14.
I20260709 12:25:28.434376 10001 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.414608  9757 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:28.434959 10001 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.435420 10269 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:28.250542 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.300168  9930 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:28.386417 31821 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4297) took 1126 ms. Trace:
I20260709 12:25:28.423954 32343 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:28.446949 31821 rpcz_store.cc:276] 0709 12:25:27.259644 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.275514 (+ 15870us) service_pool.cc:224] Handling call
0709 12:25:27.275921 (+   407us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:27.284014 (+  8093us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:27.284045 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:27.284513 (+   468us) log.cc:844] Serialized 26 byte log entry
0709 12:25:27.653324 (+368811us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:27.653384 (+    60us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.653496 (+   112us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.386074 (+732578us) raft_consensus.cc:1685] finished
0709 12:25:28.386106 (+    32us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.386326 (+   220us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":7599,"raft.queue_time_us":11101,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":31108,"thread_start_us":337248,"threads_started":1,"wal-append.queue_time_us":1068712}
W20260709 12:25:28.387476 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47978.5us
I20260709 12:25:28.419989 31819 pending_rounds.cc:107] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 35.17
I20260709 12:25:28.453728 31819 raft_consensus.cc:2889] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:28.439709 31818 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1925) took 1694 ms. Trace:
I20260709 12:25:28.440304 31642 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:28.377302 10257 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.234607 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.467363 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4301) took 1206 ms. Trace:
I20260709 12:25:28.365298 31670 rpcz_store.cc:276] 0709 12:25:26.976097 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.976223 (+   126us) service_pool.cc:224] Handling call
0709 12:25:26.976527 (+   304us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:26.976730 (+   203us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:25:27.197620 (+220890us) log.cc:844] Serialized 14 byte log entry
0709 12:25:28.363854 (+1166234us) spinlock_profiling.cc:243] Waited 16.1 ms on lock 0x7b18003ed2e0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b 00007fe8fc974d23 00007fe8fc97523a
0709 12:25:28.364175 (+   321us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:28.364249 (+    74us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:25:28.364317 (+    68us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.364433 (+   116us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.364589 (+   156us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":467197,"prepare.run_cpu_time_us":85,"prepare.run_wall_time_us":3696,"raft.queue_time_us":1672,"raft.run_cpu_time_us":123,"raft.run_wall_time_us":124,"spinlock_wait_cycles":37126016,"thread_start_us":469479,"threads_started":2,"wal-append.queue_time_us":369586}
W20260709 12:25:28.445022 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.424765 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.468400 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.468026 31824 rpcz_store.cc:276] 0709 12:25:27.261117 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.378594 (+117477us) service_pool.cc:224] Handling call
0709 12:25:27.383897 (+  5303us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:28.429297 (+1045400us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.467141 (+ 37844us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54000fc654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:28.467307 (+   166us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2402877824}
W20260709 12:25:28.445915 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 409938us
I20260709 12:25:28.470299  9757 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.471019 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.470882  9757 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.472069 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.472373 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.472863 31748 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 35.
I20260709 12:25:28.473059  9757 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:28.475581 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:28.475399 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51599.5us
I20260709 12:25:28.479418 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.480257 31753 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:28.483999 31821 raft_consensus.cc:1240] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 32 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:28.484400 31824 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 33 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
W20260709 12:25:28.417374 10164 consensus_meta.cc:287] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.744s	user 0.008s	sys 0.000s
I20260709 12:25:28.456589 31642 leader_election.cc:304] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:28.455768 31818 rpcz_store.cc:276] 0709 12:25:26.745827 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.746082 (+   255us) service_pool.cc:224] Handling call
0709 12:25:26.746598 (+   516us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:26.776351 (+ 29753us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:26.776383 (+    32us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:26.779450 (+  3067us) log.cc:844] Serialized 26 byte log entry
0709 12:25:26.918078 (+138628us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:27.072070 (+153992us) log.cc:844] Serialized 14 byte log entry
0709 12:25:27.072328 (+   258us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:27.072406 (+    78us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:28.313817 (+1241411us) spinlock_profiling.cc:243] Waited 10.7 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:28.439297 (+125480us) raft_consensus.cc:1685] finished
0709 12:25:28.439426 (+   129us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:28.439629 (+   203us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":29218,"prepare.queue_time_us":251804,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":56,"raft.queue_time_us":30479,"raft.run_cpu_time_us":342,"raft.run_wall_time_us":12205,"spinlock_wait_cycles":24567680,"thread_start_us":195594,"threads_started":2,"wal-append.queue_time_us":140704}
W20260709 12:25:28.465912 31822 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 4075) took 2553 ms (client timeout 3073 ms). Trace:
W20260709 12:25:28.462020 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.473151 31746 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 36: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:25:28.423239 10235 consensus_meta.cc:287] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.847s	user 0.008s	sys 0.004s
W20260709 12:25:28.495244 31822 rpcz_store.cc:269] 0709 12:25:25.912412 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:25.955357 (+ 42945us) service_pool.cc:224] Handling call
0709 12:25:28.465695 (+2510338us) spinlock_profiling.cc:243] Waited 2.47 s on lock 0x7b54002675d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:28.465849 (+   154us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":5688948096}
I20260709 12:25:28.487820 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.496609 31749 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:28.495901 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:28.498890 31827 raft_consensus.cc:3060] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:28.499454 31823 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:28.500118 31752 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 33 index: 23. (index mismatch)
I20260709 12:25:28.424157 10230 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:28.425758 10282 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:28.446806  9930 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:28.475404 10213 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:28.503834 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.425024 10186 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:28.417383 10285 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:28.502180 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.506390 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.424708 10239 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:28.502959 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.475481 10290 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.509931 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:28.516752 31753 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
I20260709 12:25:28.513000 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.513411 31746 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 32 index: 16. Preceding OpId from leader: term: 34 index: 17. (index mismatch)
I20260709 12:25:28.517313 31748 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:28.501605  9907 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:28.419909 10234 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.497447 10288 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.530146 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:28.540413 31753 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:28.500825 10229 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:28.533722 31752 raft_consensus.cc:1275] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 28 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:28.543970 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:28.534616 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:28.528795 32372 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:25:28.501026 10013 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:28.537230 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.508174 10158 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:28.543231 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.543746 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:28.544200 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:28.568890 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:28.570441  9930 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.20, Last appended by leader: 20, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:28.571858  9930 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:28.572260 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.572872 31749 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
I20260709 12:25:28.573632 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:28.575217 31827 raft_consensus.cc:2468] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
I20260709 12:25:28.513782 10213 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.573719 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:28.567759 31748 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 28.
I20260709 12:25:28.580276 31818 raft_consensus.cc:2368] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 35. Current term is 36.
I20260709 12:25:28.509285 10270 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:28.567759 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:28.543231 32377 raft_consensus.cc:1240] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:28.577469 32343 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:28.498872 10164 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.580755 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:28.501459 10235 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.601307 10235 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:28.544883  9907 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:28.568933 10190 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.071s
I20260709 12:25:28.602744  9907 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.603492 10190 consensus_queue.cc:1243] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer 4d48c83c761f48079bc6ea176649dda0 log is divergent from this leader: its last log entry 33.22 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 21
I20260709 12:25:28.606160 10076 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:28.606887 10076 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.607252 10076 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:28.587702 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:28.493472 10257 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.593048  9638 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:28.545174 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 35 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.615650 31818 pending_rounds.cc:85] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 21
I20260709 12:25:28.596942 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.616060 31818 pending_rounds.cc:107] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 33.22
I20260709 12:25:28.616400 31818 raft_consensus.cc:2889] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:28.597594 10234 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:28.539569 10290 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.562703 10288 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.599193 31753 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:25:28.614379 10174 raft_consensus.cc:2804] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:28.619663 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:28.617728  9638 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.624727  9638 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:28.626464 10164 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:28.628695 10257 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:28.584576 10270 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.584388 10213 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:28.616029 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.613926 10209 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:28.635890 31745 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:28.631306 10190 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:28.641276 10190 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:28.641690 10190 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:28.637590 10174 raft_consensus.cc:697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 35 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:28.643517 10174 consensus_queue.cc:237] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.644338 10290 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:28.629966 10263 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.015s
W20260709 12:25:28.649891 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.632937 10111 consensus_queue.cc:1048] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.004s
I20260709 12:25:28.636924 31749 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 36 index: 23. (index mismatch)
I20260709 12:25:28.652801 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:28.656458 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:28.651894 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 274793us
I20260709 12:25:28.633563 10270 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:25:28.659610 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.670755 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.671680 31823 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:28.673280 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.674203 31827 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 36.
I20260709 12:25:28.667155 10264 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 34.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.017s
I20260709 12:25:28.680147  9638 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.667121 10269 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:28.682026 10234 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.682382  9638 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:28.682677 10234 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.685247 10234 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:28.691514 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.692301 31821 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
W20260709 12:25:28.482813 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 475538us
I20260709 12:25:28.697327 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:28.698614 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:28.710930 31747 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 36 index: 24. Preceding OpId from leader: term: 37 index: 25. (index mismatch)
I20260709 12:25:28.696410 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:28.725018 32376 raft_consensus.cc:1275] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 32 index: 23. (index mismatch)
I20260709 12:25:28.663795 10263 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:28.720611 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:28.753507 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:28.754186 31747 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:28.756915 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:28.760121 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.761488 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.760430 31821 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 29 index: 17. (index mismatch)
I20260709 12:25:28.761617 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:28.762212 31823 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:28.676311 10140 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:28.765134 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:28.771936 31821 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:28.777220 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:28.772956 31827 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:28.785670 31718 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:28.770005 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:28.787446 31718 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:28.774385 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:28.731786 10234 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:28.791158 31823 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:28.794375 32378 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 33. Current term is 34. Ops: []
I20260709 12:25:28.795212 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.791714 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:28.795926 31824 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:28.783162 31791 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:28.753218 10263 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.798520 31823 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
W20260709 12:25:28.798839 31744 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.012s	sys 0.012s
I20260709 12:25:28.796195 31747 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
W20260709 12:25:28.800400 31744 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.543s	user 0.012s	sys 0.012s
I20260709 12:25:28.797180 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:28.756770 10228 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:28.803429 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:28.806383 32342 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:28.836069 32372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:28.837188 32342 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:28.836818 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:25:28.783684 10260 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.762s	user 0.010s	sys 0.018s
I20260709 12:25:28.761682 10257 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:28.791115 10239 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:28.826385 32378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:28.782330 10219 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.047s
I20260709 12:25:28.799836 10234 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.794044 10291 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:28.831234 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:28.845738 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:28.761277 10253 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.060s
I20260709 12:25:28.757179 10209 consensus_queue.cc:1048] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.065s
I20260709 12:25:28.832211 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:28.833222 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:28.838734 32342 leader_election.cc:336] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:28.832147 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.846175 31821 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 36 index: 24. Preceding OpId from leader: term: 37 index: 25. (index mismatch)
I20260709 12:25:28.834251 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.848546 31827 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:28.828243 10285 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:28.803465 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:28.803221 10263 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:28.822927 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:28.823468 10282 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:28.849648 31822 raft_consensus.cc:2393] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 35: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:28.756532 10118 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.848083 31823 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:25:28.793838  9757 consensus_queue.cc:1048] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.042s
I20260709 12:25:28.847775 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:28.830050 10102 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:25:28.844714 10278 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.514s	user 0.037s	sys 0.044s
I20260709 12:25:28.854620 31747 raft_consensus.cc:2393] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:28.790343 10062 consensus_queue.cc:1059] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.040s
I20260709 12:25:28.827968 10245 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:28.841205 10260 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.828s	user 0.010s	sys 0.018s
W20260709 12:25:28.891909 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.873543 10102 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.882751 10058 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.893488 10102 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:28.855650 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:28.893694 10225 consensus_queue.cc:1059] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.002s
I20260709 12:25:28.896311 10225 raft_consensus.cc:3055] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:28.896746 10225 raft_consensus.cc:740] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:28.897073 10033 consensus_queue.cc:260] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.14, Last appended by leader: 14, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.897746 10225 consensus_queue.cc:260] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.898665 10033 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:28.899323 10225 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:28.900471 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.852450 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:28.840211 10239 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.877072 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:28.879583 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:28.889758 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.857882 10292 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:28.860090 10164 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:28.912396 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60565.2us
I20260709 12:25:28.913640 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:25:28.905818 32342 leader_election.cc:336] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:28.916325 31643 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:25:28.917138 32342 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:28.918525 31643 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:28.923957 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:28.850000 10234 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:28.898324 31717 leader_election.cc:336] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:28.868494 10282 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.889330 10264 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:28.861317 10291 raft_consensus.cc:697] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:28.883244 10278 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.037s	sys 0.045s
I20260709 12:25:28.868680 10235 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:28.911159 10239 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:28.917884 10164 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:28.936720  9907 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:28.850235 10257 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:28.948951 10231 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 1.062s	user 0.014s	sys 0.002s
W20260709 12:25:28.940250 31717 leader_election.cc:336] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:28.983130 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:28.937058 32342 leader_election.cc:336] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:28.945722 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.939340 10290 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:28.900347  9857 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:28.869035 10118 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:28.984006 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47005.2us
W20260709 12:25:28.993316 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:28.960817 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.942078 10233 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:28.974618 10062 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:28.915242 10292 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:28.849274  9747 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:28.951396 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:28.902801 10190 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:28.977942 32373 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33962 (request call id 1680) took 1693 ms (client timeout 1559 ms). Trace:
I20260709 12:25:28.899938 10245 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.078876 10245 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:28.873502 10285 raft_consensus.cc:697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:28.976740 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:28.980361  9413 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:28.890302 10288 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:29.080202 10285 consensus_queue.cc:237] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.893823 10058 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:28.985016 31747 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 25.
I20260709 12:25:29.012794 10290 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 33 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:29.083432 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:29.084061 31747 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:29.084240 10290 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.949290 10232 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:28.948411 10158 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:25:28.995237 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:28.903584 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.995854 31751 raft_consensus.cc:2468] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:28.959702 10235 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:28.972205 10164 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:29.086452 31744 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:29.087061 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:29.087594 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:29.022984 10233 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:25:29.017383 31717 consensus_peers.cc:597] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:29.088235 31744 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
W20260709 12:25:29.088639 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 71350.7us
I20260709 12:25:28.944293 10079 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.039s
W20260709 12:25:29.089375 31717 consensus_peers.cc:597] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:29.089040 10233 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 31.23, Last appended by leader: 23, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:29.089660 31824 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:29.088300 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:28.957119 10049 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:29.091878 10049 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:29.092783 31717 consensus_peers.cc:597] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:29.022699 10292 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:29.029170 10213 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:29.093709 31717 leader_election.cc:336] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.003558  9857 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.030150 10190 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:29.043200 10302 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.187s	user 0.001s	sys 0.000s
I20260709 12:25:29.044745  9756 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:25:29.044922 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 93621.8us
I20260709 12:25:29.100417 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:29.102277 31718 leader_election.cc:400] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 35. Current term is 36.
I20260709 12:25:29.102847 31718 leader_election.cc:403] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Cancelling election due to peer responding with higher term
I20260709 12:25:29.047293 10291 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.049443 10270 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:29.053614 10076 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:29.105433 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:29.060505 10282 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:29.012244 10062 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:29.086198 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 91021.1us
I20260709 12:25:28.945824 10066 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.037s
I20260709 12:25:28.942992 10260 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
W20260709 12:25:29.020491 32373 rpcz_store.cc:269] 0709 12:25:27.284477 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:27.284653 (+   176us) service_pool.cc:224] Handling call
0709 12:25:28.977685 (+1693032us) spinlock_profiling.cc:243] Waited 936 ms on lock 0x7b54004984d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:28.977867 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2153085824}
I20260709 12:25:29.092941  9757 raft_consensus.cc:3055] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:29.035573 32374 raft_consensus.cc:2393] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 35: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:25:29.094659 31717 leader_election.cc:336] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.097410 10231 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.117415 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:29.035332 10304 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.082s	user 0.000s	sys 0.000s
W20260709 12:25:29.100625 10187 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.706s	user 0.008s	sys 0.004s
I20260709 12:25:29.097311 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:29.098800 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.046023 31819 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41204 (request call id 4098) took 1169 ms. Trace:
I20260709 12:25:29.011925 10253 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:29.121573 31744 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:29.112022 32371 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 37: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:25:29.114574 32342 leader_election.cc:336] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:29.138166 32342 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.138793 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:28.984244  9638 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:29.121618 31819 rpcz_store.cc:276] 0709 12:25:27.877058 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.356522 (+479464us) service_pool.cc:224] Handling call
0709 12:25:28.356878 (+   356us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:28.454703 (+ 97825us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:28.454749 (+    46us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:28.455445 (+   696us) log.cc:844] Serialized 52 byte log entry
0709 12:25:28.852338 (+396893us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:28.852404 (+    66us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:28.852525 (+   121us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:29.045747 (+193222us) raft_consensus.cc:1685] finished
0709 12:25:29.045770 (+    23us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:29.045961 (+   191us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":1092,"raft.run_cpu_time_us":167,"raft.run_wall_time_us":23599,"thread_start_us":349368,"threads_started":1,"wal-append.queue_time_us":501340}
I20260709 12:25:29.153625 31747 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 35.
I20260709 12:25:29.108758 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:25:29.123740 31717 consensus_peers.cc:597] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:29.155468 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 46910.6us
I20260709 12:25:29.107527  9747 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:29.122740 10213 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:29.109328 10295 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election lost for term 35. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 35. Current term is 36.
W20260709 12:25:29.131110 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 275504us
I20260709 12:25:29.113754 10079 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:29.116920  9757 raft_consensus.cc:740] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:25:29.161558 32342 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:29.156363 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:29.117035 10260 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:29.195079 31717 leader_election.cc:336] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.114315 10062 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:29.195693 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.196255 31744 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 28 index: 15. Preceding OpId from leader: term: 29 index: 16. (index mismatch)
W20260709 12:25:29.200982 32342 leader_election.cc:336] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.123364  9756 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:29.124919 10058 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:29.122041 10049 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:29.207366 31717 leader_election.cc:336] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:29.214182 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57933.4us
I20260709 12:25:29.153589  9638 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:29.162729 10066 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:29.218257 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:29.228106 31750 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:29.194851 10187 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:25:29.228909 10187 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.125305 10231 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.195016 10079 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.195672 10190 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:25:29.223259 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:29.204401 10260 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:29.153625 10253 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:29.242683 31791 leader_election.cc:336] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.245923 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:29.215595 10037 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.227277 10074 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:29.278826 10074 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.279268 10074 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:29.279997 10239 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.287163 10235 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:29.288512 10313 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.090s	user 0.000s	sys 0.000s
W20260709 12:25:29.268239 31791 leader_election.cc:336] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.289184 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:29.229336 10187 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:29.269790 10049 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:29.270314 10310 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.156s	user 0.000s	sys 0.000s
I20260709 12:25:29.271080 10263 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:29.227780 10066 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:29.271677  9857 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:29.272905 10258 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.920s	user 0.014s	sys 0.000s
I20260709 12:25:29.274899  9638 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 34.17, Last appended by leader: 17, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:29.275477 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:29.295693 10306 log.cc:927] Time spent T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:25:29.251369 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:29.290649 31791 leader_election.cc:336] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.329300 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:29.268924 10037 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:29.333606 31791 leader_election.cc:336] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:29.294610 31717 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:29.338202 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 43626.9us
I20260709 12:25:29.299460 10076 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:29.313365 31750 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
I20260709 12:25:29.242134 10079 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:29.317720 31819 raft_consensus.cc:1275] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 33 index: 21. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
I20260709 12:25:29.300283 10209 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:29.325284 31746 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:25:29.295794 10257 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:29.276300 10253 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:29.329712 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 78460.4us
I20260709 12:25:29.334348 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:29.353514 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:29.339195 31717 leader_election.cc:336] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.296650 10158 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:29.325444 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 320628us
I20260709 12:25:29.356837 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:29.357209 31718 leader_election.cc:400] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 37. Current term is 38.
I20260709 12:25:29.357935 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:29.289391 10239 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:29.349625 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.360035 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:29.361896 31717 leader_election.cc:336] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:29.372673 31791 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.372642 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:29.374254 31791 leader_election.cc:336] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.375138 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:29.377286 31791 leader_election.cc:336] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.340085 10299 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:29.332156  9857 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.344722 10049 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:29.387243 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:29.387873 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:29.401691 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:25:29.363477 31717 leader_election.cc:336] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.388542 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:29.379804 31746 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 34. Current term is 35. Ops: []
I20260709 12:25:29.347115  9638 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:29.246798  9757 consensus_queue.cc:260] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.21, Last appended by leader: 21, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:25:29.416150 31791 leader_election.cc:336] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.416775 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:29.418195 31791 consensus_peers.cc:597] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:29.322930 10102 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:29.410959 31717 leader_election.cc:336] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.432478 10263 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:29.444248 31748 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 34. Current term is 35. Ops: [34.17-34.17]
I20260709 12:25:29.460477 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:29.372310 10235 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:25:29.454491 31791 leader_election.cc:336] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.381496  9747 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:29.397508 10152 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:25:29.474587 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:29.458920 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51143us
I20260709 12:25:29.385998  9756 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
W20260709 12:25:29.468158 31791 leader_election.cc:336] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.510926  9756 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.510995 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:29.511543  9756 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:29.512061 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 43935.5us
W20260709 12:25:29.512820 31791 leader_election.cc:336] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.513207 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:29.454998 31747 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:29.391260 10257 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:25:29.458086 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47197.9us
I20260709 12:25:29.448136 10292 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:29.407332  9930 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:29.413362 10033 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:29.453605 10158 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:29.470278 10307 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.086s	user 0.001s	sys 0.000s
I20260709 12:25:29.414999 10316 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:29.526945 10316 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.527566 10316 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.396368 10066 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.23, Last appended by leader: 23, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:29.381846 10253 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.089s
I20260709 12:25:29.532823 10316 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:29.534191 10316 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.534716 10316 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.464323 10276 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.397853 10213 consensus_queue.cc:1059] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.088s
I20260709 12:25:29.464567 10245 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:25:29.461249 10186 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.504s	user 0.019s	sys 0.042s
I20260709 12:25:29.466398 10209 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:29.517369 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:29.470860  9757 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:29.470607 10076 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
W20260709 12:25:29.515719 31717 leader_election.cc:336] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.473512 10295 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:29.604803 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 89130.1us
W20260709 12:25:29.610340 31717 consensus_peers.cc:597] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:29.548609 10231 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:29.560027 10102 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:29.560460 10001 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:29.518967 10152 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:29.561116 10235 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.040s
I20260709 12:25:29.594660 10253 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:29.604159 10186 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.650s	user 0.019s	sys 0.042s
I20260709 12:25:29.600883 31751 raft_consensus.cc:1240] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:29.590740 10118 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:29.604599 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 87334.2us
W20260709 12:25:29.594208 10164 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.507s	user 0.002s	sys 0.010s
W20260709 12:25:29.591274 10234 consensus_meta.cc:287] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.637s	user 0.012s	sys 0.000s
W20260709 12:25:29.613780 31717 consensus_peers.cc:597] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:29.615490 10037 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:29.422766  9857 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.483985 10280 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:29.600410 10245 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.578788 10230 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.034s
I20260709 12:25:29.632370 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:29.725874 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:29.758988 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 126719us
I20260709 12:25:29.759917 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:29.588253 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:29.637214 31717 leader_election.cc:336] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.641866 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33976 (request call id 1465) took 1143 ms. Trace:
W20260709 12:25:29.761302 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 124138us
I20260709 12:25:29.618553 10033 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.757215 31823 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:25:29.600502 10276 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.620018 10288 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:29.761356 32371 rpcz_store.cc:276] 0709 12:25:28.498747 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.509358 (+ 10611us) service_pool.cc:224] Handling call
0709 12:25:29.641607 (+1132249us) spinlock_profiling.cc:243] Waited 544 ms on lock 0x7b5400593654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:29.641789 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1250085248}
I20260709 12:25:29.755074 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:29.616773 10315 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:25:29.762292 31717 consensus_peers.cc:597] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:29.630215 10213 raft_consensus.cc:3055] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:29.800707 32377 raft_consensus.cc:1240] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 34. Current term is 35. Ops: []
W20260709 12:25:29.771335 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 183389us
I20260709 12:25:29.812181 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:29.802461 31821 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:29.801365 32374 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:33954 (request call id 1535) took 1129 ms. Trace:
W20260709 12:25:29.805816 31717 leader_election.cc:336] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.616871 10295 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.695225 10102 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.817951 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:29.821341 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4317) took 1123 ms. Trace:
I20260709 12:25:29.813170 32374 rpcz_store.cc:276] 0709 12:25:28.671819 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.697057 (+ 25238us) service_pool.cc:224] Handling call
0709 12:25:29.800253 (+1103196us) spinlock_profiling.cc:243] Waited 337 ms on lock 0x7b54006dd654. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:29.801302 (+  1049us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":774974720}
I20260709 12:25:29.827602 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:29.828265 31821 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 25.
I20260709 12:25:29.826251 31752 rpcz_store.cc:276] 0709 12:25:28.698131 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.091141 (+393010us) service_pool.cc:224] Handling call
0709 12:25:29.091485 (+   344us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:29.112837 (+ 21352us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:29.112884 (+    47us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:29.113401 (+   517us) log.cc:844] Serialized 26 byte log entry
0709 12:25:29.362878 (+249477us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:29.362933 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:29.804938 (+442005us) spinlock_profiling.cc:243] Waited 17.7 ms on lock 0x7b5400404cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:29.804996 (+    58us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:29.805046 (+    50us) raft_consensus.cc:1685] finished
0709 12:25:29.805062 (+    16us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:29.821253 (+ 16191us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":3156,"raft.queue_time_us":3302,"raft.run_cpu_time_us":129,"raft.run_wall_time_us":6583,"spinlock_wait_cycles":40697216,"thread_start_us":249120,"threads_started":1,"wal-append.queue_time_us":437632}
I20260709 12:25:29.772325 10118 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:29.835697 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:29.771907 10245 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:29.798189 10232 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:29.836936 31823 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
I20260709 12:25:29.772264 10233 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:29.810412 10288 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.632370 10253 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.798316  9930 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.871042 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:29.846616 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:29.852173 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:29.854197 31749 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1570) took 1283 ms. Trace:
I20260709 12:25:29.872989 31749 rpcz_store.cc:276] 0709 12:25:28.570510 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.613839 (+ 43329us) service_pool.cc:224] Handling call
0709 12:25:28.621578 (+  7739us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:28.655672 (+ 34094us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:29.853982 (+1198310us) spinlock_profiling.cc:243] Waited 7.43 ms on lock 0x7b5400065250. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:29.854133 (+   151us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":17100160}
W20260709 12:25:29.855149 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:29.842216 10258 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:29.639669 10234 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:29.841907 10285 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.507s	user 0.010s	sys 0.018s
I20260709 12:25:29.696972 10074 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.757737 10280 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.811403 10213 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:29.798924 10033 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:29.876327 32374 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 25 index: 14. (index mismatch)
I20260709 12:25:29.877038 32371 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:29.640908 10164 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.884043 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:29.850384 10232 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.756873 10152 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 32.22, Last appended by leader: 22, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:29.863137 10288 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:29.856211 10235 raft_consensus.cc:2804] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:25:29.877744 10062 consensus_meta.cc:287] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.658s	user 0.009s	sys 0.005s
W20260709 12:25:29.901259 32342 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.519s	user 0.011s	sys 0.023s
I20260709 12:25:29.839026 10102 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:29.884681 10285 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.562s	user 0.011s	sys 0.019s
I20260709 12:25:29.874465 10317 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:25:29.906950 10260 consensus_meta.cc:287] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.639s	user 0.008s	sys 0.005s
I20260709 12:25:29.884348 10258 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.921005 10235 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:29.924256 10062 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.878091  9930 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.918504 10232 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 25 FOLLOWER]: Advancing to term 26
W20260709 12:25:29.926551 10325 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.073s	user 0.000s	sys 0.000s
I20260709 12:25:29.942602 10258 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:29.942207 10213 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.943300 10074 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:29.944880 10280 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:29.931208 10326 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.299s	user 0.001s	sys 0.000s
I20260709 12:25:29.936307 10316 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:29.946193 10270 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:29.941087 10260 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:29.952636 10253 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:29.948796 10235 raft_consensus.cc:3060] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:29.963393 10234 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:29.964907 10295 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:29.972633 10260 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:29.874491 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:30.006734 31749 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
W20260709 12:25:30.007102 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 167579us
W20260709 12:25:30.009135 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 196918us
I20260709 12:25:29.876681 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:30.010054 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:30.011044 31745 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.509s	user 0.014s	sys 0.016s
I20260709 12:25:30.012820 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:29.899982 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:30.014402 31792 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.541s	user 0.011s	sys 0.009s
W20260709 12:25:30.014952 31792 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.542s	user 0.011s	sys 0.009s
I20260709 12:25:30.016346 31748 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:30.016603 31744 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: [29.17-29.17]
W20260709 12:25:30.017324 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 137105us
I20260709 12:25:30.020009 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:29.894706 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 381885us
W20260709 12:25:30.024904 31793 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:30.024783 31791 leader_election.cc:336] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:29.908542 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.027875 31820 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:25:30.027936 31822 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:25:29.903703 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.032366 31825 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:30.033275 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.033957 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:29.892059 31826 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 34. Current term is 35. Ops: []
W20260709 12:25:29.921094 32342 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.011s	sys 0.023s
I20260709 12:25:29.944036 10033 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:29.937083  9857 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.044287 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:30.087224 10013 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:30.039578 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 128537us
I20260709 12:25:30.090005 10013 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:30.091259 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 709383us
I20260709 12:25:30.055526 10074 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:30.034407 31823 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:29.927390 10102 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:29.954373 10164 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:29.952641 10079 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.597s	user 0.010s	sys 0.001s
I20260709 12:25:30.064563 10076 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:30.098197 10213 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:30.099165 10074 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:30.030481 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.042689 10001 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:30.054265 10062 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:30.038028 31791 consensus_peers.cc:597] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:30.076743 10049 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:30.102789 10013 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:29.929953  9756 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.129515 31749 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 29: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:30.130870 31820 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 32: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:30.133163 10001 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:30.104261 10190 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.638s	user 0.008s	sys 0.009s
I20260709 12:25:30.106678 10079 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:29.930342 10187 consensus_meta.cc:287] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.607s	user 0.008s	sys 0.001s
I20260709 12:25:30.107947 10260 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:30.035899 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.099591 10111 consensus_queue.cc:1048] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.008s
I20260709 12:25:30.111267 10257 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:30.111539 10258 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:30.116314 10280 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.117408 10282 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:30.072821 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 50857.4us
I20260709 12:25:30.094192 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:30.135299 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:30.145684 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 107908us
W20260709 12:25:30.155701 10190 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.695s	user 0.008s	sys 0.010s
I20260709 12:25:30.108996  9930 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.162066 31825 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:30.157133 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.187765 10234 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.188683 10253 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.189563 10341 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.190519 10342 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:30.193688 10079 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.194753 10231 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:30.195061 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 101240us
I20260709 12:25:30.197782 10224 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:30.199548 31823 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
W20260709 12:25:30.207041 31791 leader_election.cc:336] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:30.200716  9756 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:30.201750 10258 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.203246 31824 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:30.174995 10257 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.136807 10049 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.219241 10001 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:30.219276 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 175309us
I20260709 12:25:30.219655 10187 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:30.248180 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.248584 10231 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:30.247479 10257 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:30.251008 10233 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.241443 10224 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.226744 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4321) took 1524 ms. Trace:
I20260709 12:25:30.251917 10224 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:30.252118 10245 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.252321 31747 rpcz_store.cc:276] 0709 12:25:28.702295 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.154652 (+452357us) service_pool.cc:224] Handling call
0709 12:25:29.454817 (+300165us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:29.514204 (+ 59387us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.226159 (+711955us) spinlock_profiling.cc:243] Waited 250 ms on lock 0x7b54001a63d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:30.226442 (+   283us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":574084480}
I20260709 12:25:30.253960 10232 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.136528 10074 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.232409 31744 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4327) took 1508 ms. Trace:
I20260709 12:25:30.207688 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:30.224819 10280 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.251160 31753 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 37. Current term is 38. Ops: [37.25-37.25]
W20260709 12:25:30.227063 10301 log.cc:927] Time spent T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.088s	user 0.000s	sys 0.001s
I20260709 12:25:30.209457 10076 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.230160 31748 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 33. Current term is 34. Ops: [33.23-33.23]
I20260709 12:25:30.255206 31744 rpcz_store.cc:276] 0709 12:25:28.724414 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.404293 (+679879us) service_pool.cc:224] Handling call
0709 12:25:29.404731 (+   438us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.017247 (+612516us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.231928 (+214681us) spinlock_profiling.cc:243] Waited 611 ms on lock 0x7b5400064354. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:30.232323 (+   395us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1405354368}
I20260709 12:25:30.269357 10118 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.270309 10118 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.271265 10074 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:30.250466 10187 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.273864 10013 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Leader pre-election won for term 26
I20260709 12:25:30.278620 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.261277  9756 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.283675 10118 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:30.284389 31745 raft_consensus.cc:1240] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 33. Current term is 34. Ops: [33.21-33.21]
I20260709 12:25:30.286466 10231 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:30.286830 10230 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.267416 10282 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:30.295521 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 94401.4us
I20260709 12:25:30.299386 10341 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.300608 10076 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.338670 10076 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:30.301743 10292 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:30.274248 10234 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.341897 10234 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.305248 10013 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.342286 10232 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.342685 10013 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:30.345050 31753 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4330) took 1394 ms. Trace:
I20260709 12:25:30.345566 31753 rpcz_store.cc:276] 0709 12:25:28.951183 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.438069 (+486886us) service_pool.cc:224] Handling call
0709 12:25:29.438671 (+   602us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.277808 (+839137us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.344692 (+ 66884us) spinlock_profiling.cc:243] Waited 812 ms on lock 0x7b54002134d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:30.344973 (+   281us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1867322112}
I20260709 12:25:30.347693 32377 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33962 (request call id 1686) took 1547 ms. Trace:
I20260709 12:25:30.348112 32377 rpcz_store.cc:276] 0709 12:25:28.800607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:28.801387 (+   780us) service_pool.cc:224] Handling call
0709 12:25:29.800416 (+999029us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:29.821851 (+ 21435us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.347267 (+525416us) spinlock_profiling.cc:243] Waited 999 ms on lock 0x7b54006dd650. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:30.347479 (+   212us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2296762752}
I20260709 12:25:30.348807 10341 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:30.349872 31791 leader_election.cc:336] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:30.350461 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:30.351074 10049 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:30.312132  9756 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:30.313424 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 112613us
I20260709 12:25:30.314253 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3380) took 1082 ms. Trace:
I20260709 12:25:30.314851 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4319) took 1616 ms. Trace:
W20260709 12:25:30.324019 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.324740 10033 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 13, Time since last communication: 0.001s
I20260709 12:25:30.325562 10230 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.328963 10280 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:30.329239 10282 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:30.307744 10342 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.329833 31823 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 37: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:30.356515 31745 rpcz_store.cc:276] 0709 12:25:29.232358 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.176755 (+944397us) service_pool.cc:224] Handling call
0709 12:25:30.246571 (+ 69816us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.313905 (+ 67334us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.314163 (+   258us) inbound_call.cc:177] Queueing success response
Metrics: {}
I20260709 12:25:30.332650 31824 raft_consensus.cc:1240] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: [29.17-29.17]
I20260709 12:25:30.251827 10233 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.336308 10258 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:30.338240 10347 log.cc:927] Time spent T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.151s	user 0.001s	sys 0.001s
I20260709 12:25:30.354063 10257 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.272711 10253 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.378252 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:30.380270 10253 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:30.381059 10342 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.383836 10342 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:30.384424  9756 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.384728 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:30.385862 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.388598 10235 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.390727 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:30.391551 32370 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:30.392073 10245 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.392671 10033 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:30.366261 10111 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:30.393369 10033 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.394006 10033 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:30.393693 10111 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:30.394711  9757 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.724s	user 0.008s	sys 0.012s
W20260709 12:25:30.395509  9638 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.972s	user 0.016s	sys 0.008s
I20260709 12:25:30.396623 10111 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.397181 10230 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.403496 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.405570 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.407141 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.407444 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.408136 31823 raft_consensus.cc:2468] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
W20260709 12:25:30.408874 32342 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:30.403591 10276 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.406535 31824 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:30.410542 10066 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:30.416365 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.356652 31751 rpcz_store.cc:276] 0709 12:25:28.701731 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:29.098101 (+396370us) service_pool.cc:224] Handling call
0709 12:25:29.098385 (+   284us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:29.620780 (+522395us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.314223 (+693443us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b54004056d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:30.314610 (+   387us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1155038592}
I20260709 12:25:30.424690 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:30.427000 10315 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.593s	user 0.007s	sys 0.012s
I20260709 12:25:30.443802 10315 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:30.444944 31825 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 36.
I20260709 12:25:30.451078 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:30.451555 10276 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:25:30.452322 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:30.455945 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 62949us
I20260709 12:25:30.453153 32377 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:30.452462 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:30.462136 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.467273 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:30.426106 10235 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:30.420636 10257 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.483738  9756 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.005s
I20260709 12:25:30.470975 10342 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:30.487646 10342 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.488134 10342 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:30.488569 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:30.489496 32371 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 38: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:30.491149 10225 consensus_queue.cc:1059] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 34.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:25:30.492551 10213 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:30.494954 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.495699 31823 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:30.482167 10224 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.499671 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.500699 31819 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:30.502935 10062 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:30.503795 10062 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.506837 10049 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:30.507081 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1575) took 1294 ms. Trace:
I20260709 12:25:30.508140 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.459570 10013 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.510411 31821 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:30.508960 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.513439 31744 raft_consensus.cc:3060] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:30.513585 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:30.514266 31822 raft_consensus.cc:2410] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 34 because replica has last-logged OpId of term: 33 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 20.
I20260709 12:25:30.516042 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:30.507562 31752 rpcz_store.cc:276] 0709 12:25:29.212642 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.019396 (+806754us) service_pool.cc:224] Handling call
0709 12:25:30.020375 (+   979us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:30.021192 (+   817us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:30.021240 (+    48us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:30.152963 (+131723us) log.cc:844] Serialized 26 byte log entry
0709 12:25:30.356281 (+203318us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:30.356332 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:30.357202 (+   870us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:30.506705 (+149503us) raft_consensus.cc:1685] finished
0709 12:25:30.506727 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:30.506890 (+   163us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":86746,"raft.run_cpu_time_us":667,"raft.run_wall_time_us":872,"thread_start_us":109229,"threads_started":1,"wal-append.queue_time_us":319364}
I20260709 12:25:30.519541 31819 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:25:30.418569 10288 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.525533 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:30.529660 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47947.3us
I20260709 12:25:30.528923 10276 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:30.528187 31820 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:30.498476 10224 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:30.533231 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 180570us
W20260709 12:25:30.533505 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 54421.4us
W20260709 12:25:30.537490 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 127794us
I20260709 12:25:30.537698 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.538892 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:30.525718 10280 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:30.539827 32342 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:30.552335 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.553256 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.550184 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:30.556591 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:30.557572 31751 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 31: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:30.555718 31824 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:30.547469  9756 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:30.562067 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:30.567301 10276 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:30.567713 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:30.568519 31750 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:25:30.568984 10233 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.568043 10276 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.569764 31753 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 23. Preceding OpId from leader: term: 32 index: 24. (index mismatch)
I20260709 12:25:30.570017 10276 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:30.570230 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:30.570708  9756 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:30.571064 31745 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:30.571218  9756 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:30.571309 10076 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.569787 10013 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.571981 10076 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
W20260709 12:25:30.575559 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 123233us
I20260709 12:25:30.567803 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:30.575992 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60082.6us
I20260709 12:25:30.577885 10288 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.583114 10314 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.583989 10190 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.588054 10190 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.583990 10314 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.594895 31746 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 36.
I20260709 12:25:30.596670 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:30.606876 31820 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:30.574517 10076 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:25:30.608786 32370 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:25:30.612752 10314 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.612912 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:30.613827 31752 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 34: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:30.624076 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:30.625334 10233 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:30.626549 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:30.628482 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:30.631778 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.635206 10288 consensus_queue.cc:1059] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 27.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:30.638336 10033 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.638921 10062 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:30.641057 10033 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.645860 10228 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:30.646118 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:30.634073 31826 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
I20260709 12:25:30.630259 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.651387 31824 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:30.645974 10253 raft_consensus.cc:3055] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:30.653898 10233 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:30.654134 10253 raft_consensus.cc:740] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:30.655253 31749 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:30.655400 10253 consensus_queue.cc:260] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.656833 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:30.658273 10058 raft_consensus.cc:2804] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:30.658402 10213 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:30.658823 10058 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.659128 10213 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.659399 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.659739 10213 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:30.660423 31823 raft_consensus.cc:2410] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 34 because replica has last-logged OpId of term: 33 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 19.
I20260709 12:25:30.665828 10288 raft_consensus.cc:3055] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:30.666394 10288 raft_consensus.cc:740] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:30.667958 10288 consensus_queue.cc:260] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.669099 10314 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
I20260709 12:25:30.669911 10288 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:30.657043 10253 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:30.659289 10058 raft_consensus.cc:3060] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:30.679414 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:30.679896  9756 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.609385 10190 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.683301 10233 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:30.699630 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:30.707165 10233 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.707811 10233 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:30.707927 31746 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 34: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:30.694453 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.708722 10276 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.709264 31822 raft_consensus.cc:2368] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 36. Current term is 37.
I20260709 12:25:30.712031 10276 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.683585 10342 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.714722  9756 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.717427 10342 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.682806 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:30.719221 31643 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:30.720276 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:30.721072 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 94730.4us
I20260709 12:25:30.721292 10033 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:30.700582 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:30.723811 31749 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 26: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:30.706310 10058 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.726871 10058 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.727473 10058 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election won for term 26
I20260709 12:25:30.721808 10033 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.729878 10033 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:30.735131 10058 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:30.736444 10058 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 24.16, Last appended by leader: 16, Current term: 26, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.736871 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.738386 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.734998 32343 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:30.703471 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:30.746598 31745 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:30.747081 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:30.735042 10245 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:30.747936 10245 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.748330 10245 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:30.745764 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:30.739866 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.740921 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.722119 31642 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:30.744326 31821 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 30: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:30.699887 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:30.741467 31820 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 35.
I20260709 12:25:30.741994 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.721488 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:30.744130 31822 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
W20260709 12:25:30.758442 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58733.1us
I20260709 12:25:30.756868 31823 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:30.763379 31744 raft_consensus.cc:2468] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 37.
I20260709 12:25:30.744874 10331 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:30.768859 31827 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 32: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:25:30.770183 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48801.8us
I20260709 12:25:30.779263 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.773228 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:30.780552 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:30.764688 10331 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.741073 10033 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:30.785049 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 240550us
I20260709 12:25:30.766147 10076 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:30.738603 10213 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.794327 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:30.796424 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 29 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:30.773609 10276 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:25:30.791878 32342 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 86 similar messages]
I20260709 12:25:30.801424 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:30.780524 10253 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:30.700176 10228 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.799532 31744 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 29: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:30.808452 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:30.791047 10331 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:30.819877 31824 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 37.
I20260709 12:25:30.820925 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:30.810571 32369 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:30.794425 10076 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.819324 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:25:30.825802 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 103782us
I20260709 12:25:30.821657 31753 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:25:30.806063 10288 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:25:30.829459 32342 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:30.806500  9756 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.830279 32370 raft_consensus.cc:2393] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 29: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:30.833987 10228 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:30.835306 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:30.809787 10253 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.858798 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:30.864089 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.871040 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 36 index: 24 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:30.865000 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 66693.4us
I20260709 12:25:30.831182 10076 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:30.864360 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:30.864003 32371 raft_consensus.cc:2368] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 36. Current term is 37.
I20260709 12:25:30.837453 10288 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.893339 31820 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:25:30.892304 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.824419 10233 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:30.860780  9756 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:30.864166 10253 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:30.864074 10282 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:30.903023 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:30.917135 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:30.884066 10288 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:30.903318 10225 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:30.929970 31642 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:30.920444 32370 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 23. Preceding OpId from leader: term: 32 index: 24. (index mismatch)
I20260709 12:25:30.921066 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.947472 31827 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:30.921788 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.922441 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.923554 31823 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 34. Current term is 35. Ops: [34.19-34.19]
I20260709 12:25:30.924885 10335 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:30.925612 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:30.949716 32377 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 37: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:30.932009 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:30.934909 10233 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.938016 10282 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:30.941395 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.942588 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:30.943665 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:30.883177 10314 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:30.945577 32369 raft_consensus.cc:1240] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 37. Current term is 38. Ops: []
I20260709 12:25:30.957001  9757 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:30.956897 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.956897 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:30.968791 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.969726 31826 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:30.967177 32374 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
W20260709 12:25:30.975162 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:30.977594  9757 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:30.981598 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:30.950050 10335 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:30.987986 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:30.988708 32377 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:30.989871 31823 raft_consensus.cc:2393] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 37: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:30.948810 10213 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.959090 10225 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 35 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:30.985816 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.908253 10013 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.071s
W20260709 12:25:30.994037 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 64220.1us
I20260709 12:25:30.960417 10342 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:30.999167 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 92328.4us
I20260709 12:25:31.001740 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:31.002928 10033 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:30.996413 10335 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:31.003979 10228 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:31.004582 10228 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:31.008495 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61404.8us
W20260709 12:25:31.009701 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 116879us
I20260709 12:25:31.010608 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:31.024183 10341 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:31.028476 31819 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:31.024562 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:31.035416 32375 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 37: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:31.001745 10037 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:25:31.037478 31717 consensus_peers.cc:597] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:31.011790 10228 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:31.042676 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:30.991166  9757 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 34.17, Last appended by leader: 17, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:30.986291 10314 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:31.041980 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:31.059363 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:31.060009 31819 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:31.065732 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:31.049328  9756 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
W20260709 12:25:31.065806 31717 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:31.071051 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:31.072619 31825 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:31.083125 31822 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 30 index: 17. (index mismatch)
I20260709 12:25:31.054113 10342 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:31.081805 31717 leader_election.cc:336] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:31.099179 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:31.099987 31822 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:31.107244 31575 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 152) took 158083 ms (client timeout 15000 ms). Trace:
W20260709 12:25:31.107810 31575 rpcz_store.cc:269] 0709 12:22:53.023441 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:53.039325 (+ 15884us) service_pool.cc:224] Handling call
0709 12:22:53.619807 (+580482us) spinlock_profiling.cc:243] Waited 190 ms on lock 0x7b54002828d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3f090 00007fe8fcdddf3a 00007fe902b93e9c 00007fe902c9204e 00007fe9011532a4 00007fe901153232 00007fe901153161 00007fe9011530dc 00007fe901152392 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:31.106927 (+157487120us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":94461,"reported_tablets":28,"spinlock_wait_cycles":436693376}
I20260709 12:25:31.109215 10331 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:31.068612 10213 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.141853 31827 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:31.061205 10037 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:31.077921 10033 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:31.143832 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 59847.2us
W20260709 12:25:31.145612 31717 leader_election.cc:336] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:31.157094 31717 leader_election.cc:336] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:31.121042 10224 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.158527 10258 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.159322 10258 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:31.158545 10224 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:31.161667 10258 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:31.162559 10224 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:31.163343 10033 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:31.165916 10033 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:31.170500 10037 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:31.150723 10213 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:31.175799 10213 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:31.147766 10225 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:31.182475 10362 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.157s	user 0.000s	sys 0.001s
I20260709 12:25:31.175625 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 26 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:31.187760 31827 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:31.189620 32377 raft_consensus.cc:2468] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:31.191991 31822 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:31.193038 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:31.193984 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:31.118248 10118 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.117950  9747 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:31.194860 32377 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 37: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:31.198185 31819 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 37.
I20260709 12:25:31.211730 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.212917 31744 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:31.213160 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:31.214721 31822 raft_consensus.cc:2393] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 37: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:25:31.211875 31717 leader_election.cc:336] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:31.217036 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.218657 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:31.230254 31745 raft_consensus.cc:2393] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 37: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:25:31.225503 31717 leader_election.cc:336] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:31.233505 31717 consensus_peers.cc:597] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:31.243443 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 47516.6us
I20260709 12:25:31.249007 10314 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:31.118065 10235 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.102s
I20260709 12:25:31.259138 10282 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:31.241737 10118 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:31.265188 10118 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:31.283095 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:31.284729 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:31.285631 31827 raft_consensus.cc:2410] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 35 because replica has last-logged OpId of term: 33 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 20.
I20260709 12:25:31.286765 32343 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:31.289580 31826 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:31.242292 32342 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:31.294086 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 53803us
I20260709 12:25:31.086256 10276 consensus_queue.cc:1059] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 36.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.131s
I20260709 12:25:31.111662 10342 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:31.313838 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 69445.1us
W20260709 12:25:31.316087 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 87492.3us
W20260709 12:25:31.299440 10245 consensus_meta.cc:287] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.551s	user 0.011s	sys 0.003s
I20260709 12:25:31.301195 10341 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:31.320094 10263 raft_consensus.cc:2804] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:25:31.286494 10335 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:31.327865 31825 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:31.333746 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:31.336768 31822 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 31. Current term is 32. Ops: [31.17-31.17]
I20260709 12:25:31.348109 10037 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:31.320585 10264 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:31.358464 31818 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 32. Current term is 33. Ops: [32.18-32.18]
I20260709 12:25:31.360626 31821 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 32: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:31.319383 10245 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:31.387831 31744 raft_consensus.cc:2468] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:31.359418 10263 raft_consensus.cc:697] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 37 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:31.390436 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:31.368681 10264 raft_consensus.cc:697] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 35 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:31.366909 10341 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.029s
I20260709 12:25:31.366840 10253 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:31.376016 10276 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.043s
I20260709 12:25:31.431512 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 28 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:31.436110 31568 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 155) took 88196 ms (client timeout 15000 ms). Trace:
I20260709 12:25:31.436844 10331 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:31.437525 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62079.9us
I20260709 12:25:31.439424 31642 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:31.439594 31576 catalog_manager.cc:5697] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 18 to 30, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 30 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } }
I20260709 12:25:31.440925 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:31.442282 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:31.442904 10033 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:31.441620 31576 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 26 to 28, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 28 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:31.428217 10118 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:31.468156 10118 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.468557 10253 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:31.468634 10118 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:31.469575 10263 consensus_queue.cc:237] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 35.22, Last appended by leader: 22, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:31.462723 31717 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:31.473830 10228 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:31.441646 10235 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:31.431649 10315 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:31.494999 10228 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:31.495071 31576 catalog_manager.cc:5697] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 17 to 28, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 28 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:31.496479 32342 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:31.497641 10245 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:31.499226 10076 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.596s	user 0.011s	sys 0.000s
I20260709 12:25:31.510569 31576 catalog_manager.cc:5697] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 22 to 28, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 28 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:31.503705 10118 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:31.468681 10342 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:31.532334 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:31.533336 31749 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:31.459992 10190 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 21, Time since last communication: 0.063s
I20260709 12:25:31.534942 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.536108 31745 raft_consensus.cc:2393] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:31.539912 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:31.540889 31744 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 37.
W20260709 12:25:31.519232 31717 leader_election.cc:336] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:31.519755 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 77135.8us
I20260709 12:25:31.525957 10118 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:31.544445 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:31.527380 10264 consensus_queue.cc:237] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 34.23, Last appended by leader: 23, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:31.551298 31717 leader_election.cc:336] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:31.553617 31717 leader_election.cc:336] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:31.555074 31642 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:31.555987 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 30 candidate_status { last_received { term: 27 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.549105 31576 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 26 to 28. New cstate: current_term: 28 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:31.561403 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:31.520323 10224 raft_consensus.cc:2804] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader election won for term 37
W20260709 12:25:31.563908 31717 leader_election.cc:336] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:31.565035 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:31.564411 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:31.566340 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:31.572258 31749 raft_consensus.cc:2393] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 30: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:25:31.564834 31717 leader_election.cc:336] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:31.573123 31717 leader_election.cc:336] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:31.572151 31752 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:31.574723 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:31.575582 31744 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
W20260709 12:25:31.563102 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 477661us
I20260709 12:25:31.511091 10076 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:31.575304 31717 leader_election.cc:336] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:31.601919 31576 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 20 to 29. New cstate: current_term: 29 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:31.576551 31753 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:25:31.556854 10228 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:31.610729 31717 leader_election.cc:336] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:31.602757 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:31.613171 31643 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:31.565809 10235 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:31.614107 31643 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:31.533696 10315 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:31.572075 10213 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:31.571429 10245 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:25:31.625394 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.568s	user 0.020s	sys 0.052s
W20260709 12:25:31.626597 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.569s	user 0.020s	sys 0.053s
W20260709 12:25:31.624555 32375 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.543s	user 0.007s	sys 0.010s
W20260709 12:25:31.629040 32375 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.548s	user 0.007s	sys 0.010s
I20260709 12:25:31.630518 31824 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 31: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:31.631212 31718 leader_election.cc:400] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 36. Current term is 37.
I20260709 12:25:31.638855 31718 leader_election.cc:403] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Cancelling election due to peer responding with higher term
I20260709 12:25:31.625562 31750 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 36.
I20260709 12:25:31.625304 31576 catalog_manager.cc:5697] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 20 to 28, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 28 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
W20260709 12:25:31.658341 31823 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.588s	user 0.016s	sys 0.019s
W20260709 12:25:31.658984 31823 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.589s	user 0.017s	sys 0.019s
I20260709 12:25:31.601204 10062 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:31.628355 31717 leader_election.cc:336] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:31.607244 10224 raft_consensus.cc:697] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 37 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:31.621341 10058 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:31.665323 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:31.665145 10062 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:31.665927  9756 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election lost for term 36. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 36. Current term is 37.
I20260709 12:25:31.620419 10225 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:31.666730 10224 consensus_queue.cc:237] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 35.23, Last appended by leader: 23, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:31.670074 32371 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.511s	user 0.003s	sys 0.020s
I20260709 12:25:31.627941 10335 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:31.669200 10213 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:25:31.684008 31576 catalog_manager.cc:5697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 21 to 30, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 30 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:25:31.686758 32371 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.539s	user 0.003s	sys 0.021s
I20260709 12:25:31.688689 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:31.690111 31717 leader_election.cc:336] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:31.699121 31717 consensus_peers.cc:597] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:31.701028 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.700465 31576 catalog_manager.cc:5697] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 21 to 28, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 28 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:31.701972 31750 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:31.690465 10013 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:31.702477 10315 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:31.703457 10013 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:31.703063 10062 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:31.703881 10013 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:31.702454 31576 catalog_manager.cc:5697] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 22 to 29. New cstate: current_term: 29 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:31.705660 31576 catalog_manager.cc:5697] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 23 to 30, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 30 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:31.707427 31576 catalog_manager.cc:5697] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 21 to 27, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 27 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:31.709429 31576 catalog_manager.cc:5697] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 25 to 31. New cstate: current_term: 31 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:25:31.436674 31568 rpcz_store.cc:269] 0709 12:24:03.239246 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:03.928256 (+689010us) service_pool.cc:224] Handling call
0709 12:25:31.435882 (+87507626us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:28.275819 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:28.327812 (+ 51993us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:28.327866 (+    54us) write_op.cc:435] Acquired schema lock
0709 12:25:28.327891 (+    25us) tablet.cc:662] Decoding operations
0709 12:25:28.328420 (+   529us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:28.328486 (+    66us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:28.328522 (+    36us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:25:28.328881 (+   359us) tablet.cc:701] Row locks acquired
0709 12:25:28.328902 (+    21us) write_op.cc:260] PREPARE: finished
0709 12:25:28.329097 (+   195us) write_op.cc:270] Start()
0709 12:25:28.329216 (+   119us) write_op.cc:276] Timestamp: P: 1783599928329031 usec, L: 0
0709 12:25:28.329246 (+    30us) op_driver.cc:348] REPLICATION: starting
0709 12:25:28.330051 (+   805us) log.cc:844] Serialized 925 byte log entry
0709 12:25:29.544918 (+1214867us) spinlock_profiling.cc:243] Waited 1.46 ms on lock 0x7b18004c5700. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:29.947070 (+402152us) spinlock_profiling.cc:243] Waited 4.01 ms on lock 0x7b18004c5700. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:30.323354 (+376284us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b5400281fa8. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe9031919e1 00007fe90319064c 00007fe90080c3bb 00007fe90080e71a 00007fe90080e6a8 00007fe90080e628 00007fe90080e5e3 00007fe90080d7fc 00007fe902c402f6 00007fe902c3aaa8 00007fe8fcde1310
0709 12:25:30.323656 (+   302us) op_driver.cc:464] REPLICATION: finished
0709 12:25:30.624545 (+300889us) spinlock_profiling.cc:243] Waited 16 ms on lock 0x7b18004c5700. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:30.624817 (+   272us) write_op.cc:301] APPLY: starting
0709 12:25:30.625006 (+   189us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:30.663095 (+ 38089us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:30.663139 (+    44us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:30.680766 (+ 17627us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:30.681039 (+   273us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:25:30.681102 (+    63us) write_op.cc:312] APPLY: finished
0709 12:25:30.681483 (+   381us) log.cc:844] Serialized 33 byte log entry
0709 12:25:30.682945 (+  1462us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:30.683290 (+   345us) write_op.cc:454] Released schema lock
0709 12:25:30.683418 (+   128us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:30.684027 (+   609us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:30.684060 (+    33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":31085,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":189249,"apply.run_cpu_time_us":6605,"apply.run_wall_time_us":59164,"cfile_cache_hit":10,"cfile_cache_hit_bytes":12483,"num_ops":2,"prepare.queue_time_us":755841,"prepare.run_cpu_time_us":83533,"prepare.run_wall_time_us":686345,"raft.queue_time_us":314918,"raft.run_cpu_time_us":17914,"raft.run_wall_time_us":529043,"replication_time_us":1618008,"spinlock_wait_cycles":426189568,"thread_start_us":1440243,"threads_started":4,"wal-append.queue_time_us":647045}]]}
I20260709 12:25:31.713462 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:31.713696 31576 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 24 to 26, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 26 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:31.716032 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.715472 31576 catalog_manager.cc:5697] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 20 to 32, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 32 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:31.716958 31749 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 31: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:31.717489 31576 catalog_manager.cc:5697] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 18 to 24, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 24 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:31.719058 31752 raft_consensus.cc:1240] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:31.720163 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.758756 31745 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:31.721010 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.759690 31750 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:31.722239 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:31.692503 10288 consensus_meta.cc:287] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.729s	user 0.009s	sys 0.005s
I20260709 12:25:31.723057 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.636142 10228 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:31.727116 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.703063 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:31.683249 10058 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:31.737552  9756 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:31.772166 31744 raft_consensus.cc:2410] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 35 because replica has last-logged OpId of term: 34 index: 20, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 19.
I20260709 12:25:31.758105 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:31.761204 10066 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.547s	user 0.013s	sys 0.042s
W20260709 12:25:31.783553 10066 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.575s	user 0.013s	sys 0.043s
I20260709 12:25:31.783933 10058 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:31.784574 31576 catalog_manager.cc:5697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 20 to 28, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 28 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:31.770287  9756 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:31.771835 31717 consensus_peers.cc:597] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:31.809957 31717 leader_election.cc:336] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:31.811851 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.812757 31752 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:31.791143 32370 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.525s	user 0.005s	sys 0.022s
I20260709 12:25:31.786829 31576 catalog_manager.cc:5697] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 16 to 23, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 23 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
W20260709 12:25:31.817013 32370 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.006s	sys 0.022s
W20260709 12:25:31.823035 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 65038.5us
W20260709 12:25:31.817150 31717 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:31.827411 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 319149us
I20260709 12:25:31.837285 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.772823 10225 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:31.809201  9756 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:31.846081 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:31.851641 31576 catalog_manager.cc:5697] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 26 to 29, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 29 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:31.814564 10282 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:31.809595 10213 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 37 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:25:31.854821 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 378234us
I20260709 12:25:31.857542 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:31.863404 31744 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:31.771675 10288 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:31.890306 31750 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:31.843827 10276 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:31.922499 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 98327.5us
I20260709 12:25:31.923729 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:31.853925 10375 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:31.894323 10152 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.502s	user 0.000s	sys 0.017s
I20260709 12:25:31.867202 10341 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.922343 10234 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:31.941284 31576 catalog_manager.cc:5697] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 19 to 28, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 28 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:31.939133 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.951881 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.979380 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:31.980377 31750 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
I20260709 12:25:31.895982 10378 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:25:31.989184 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 375112us
I20260709 12:25:31.956269 10233 raft_consensus.cc:3055] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 37 LEADER]: Stepping down as leader of term 37
W20260709 12:25:31.973829 10152 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.887s	user 0.001s	sys 0.017s
I20260709 12:25:32.002530 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:31.962493 10076 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:32.004413 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:31.945133 10288 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:32.009549 31753 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:32.012441 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:31.985848 31576 catalog_manager.cc:5697] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 24 to 31. New cstate: current_term: 31 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:32.005959 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:32.005234 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:32.014021 31753 raft_consensus.cc:3060] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:32.015633 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:32.033344 31576 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 26 to 27, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 27 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:32.023463 31643 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:31.942560 10276 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:31.953693 10375 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:32.061821 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 211481us
I20260709 12:25:32.003192 10233 raft_consensus.cc:740] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 37 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:31.978027 10341 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:31.953631 10234 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:32.005916 10342 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:32.103915 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4362) took 1066 ms. Trace:
I20260709 12:25:32.075681 32371 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1479) took 1002 ms. Trace:
I20260709 12:25:32.095737 32375 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1477) took 1056 ms. Trace:
I20260709 12:25:32.075681 31576 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 24 to 27, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 27 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:32.108466 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 240414us
I20260709 12:25:32.120393 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:32.073063 10245 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:25:32.129774 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 206173us
W20260709 12:25:32.106729 10033 consensus_meta.cc:287] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.653s	user 0.010s	sys 0.004s
I20260709 12:25:32.104401 31823 rpcz_store.cc:276] 0709 12:25:31.021593 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.021833 (+   240us) service_pool.cc:224] Handling call
0709 12:25:31.036686 (+ 14853us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.040268 (+  3582us) raft_consensus.cc:1534] Early marking committed up to index 25
0709 12:25:31.668539 (+628271us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.087178 (+418639us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:32.087252 (+    74us) raft_consensus.cc:1645] Marking committed up to 25
0709 12:25:32.087321 (+    69us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.087417 (+    96us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.087606 (+   189us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":17135,"prepare.queue_time_us":50771,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":36,"raft.queue_time_us":134,"thread_start_us":1016231,"threads_started":2}
I20260709 12:25:32.108546 32371 rpcz_store.cc:276] 0709 12:25:31.073802 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.082054 (+  8252us) service_pool.cc:224] Handling call
0709 12:25:31.119728 (+ 37674us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.119971 (+   243us) raft_consensus.cc:1534] Early marking committed up to index 23
0709 12:25:31.695847 (+575876us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.075295 (+379448us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:32.075351 (+    56us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:25:32.075399 (+    48us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.075481 (+    82us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.075611 (+   130us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":182360,"prepare.run_cpu_time_us":71,"prepare.run_wall_time_us":17890,"thread_start_us":889743,"threads_started":3}
W20260709 12:25:32.119727 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 104181us
I20260709 12:25:32.108597 32375 rpcz_store.cc:276] 0709 12:25:31.039501 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.039642 (+   141us) service_pool.cc:224] Handling call
0709 12:25:31.042580 (+  2938us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.050914 (+  8334us) raft_consensus.cc:1534] Early marking committed up to index 21
0709 12:25:31.638995 (+588081us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.071647 (+432652us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:32.071737 (+    90us) raft_consensus.cc:1645] Marking committed up to 21
0709 12:25:32.095358 (+ 23621us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.095468 (+   110us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.095653 (+   185us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":259662,"prepare.run_cpu_time_us":52,"prepare.run_wall_time_us":52,"thread_start_us":942375,"threads_started":2}
W20260709 12:25:32.137989 10235 consensus_meta.cc:287] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.510s	user 0.006s	sys 0.004s
I20260709 12:25:32.145756 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:32.153285 31576 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 27 to 31. New cstate: current_term: 31 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:32.007583 10213 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 35.21, Last appended by leader: 21, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.130887 10258 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:32.180529 31791 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:32.171999 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:32.159278 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:32.097822 10288 consensus_queue.cc:1059] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 30.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.045s
I20260709 12:25:32.183967 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:32.187506 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:32.188742 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 195346us
I20260709 12:25:32.140026 10076 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.190078 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:32.140311 10314 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:32.191063 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 48771.7us
I20260709 12:25:32.034289 10225 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.146320 10033 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:32.145763 10058 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:25:32.227222 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 96469.6us
I20260709 12:25:32.152992 10235 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:32.243456 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:32.171880 10342 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:32.201646 10245 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:32.265540 31751 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 31: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:32.265792 31576 catalog_manager.cc:5697] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 18 to 28, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 28 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:32.170611 10049 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:32.259918 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:32.291898 31747 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:32.281505 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:25:32.292905 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 121002us
I20260709 12:25:32.250061  9757 raft_consensus.cc:2804] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:32.267107 10378 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:25:32.228829 10076 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.179924 10233 consensus_queue.cc:260] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.25, Last appended by leader: 25, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:32.344611 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 77353.2us
I20260709 12:25:32.294723 10033 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:32.295310 10342 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:32.241078 10314 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:32.345597 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:32.280364 10058 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:32.324777 10245 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:32.356784 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:32.372004 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:32.338806 10379 log.cc:927] Time spent T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.377s	user 0.000s	sys 0.002s
W20260709 12:25:32.353711 10371 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.459s	user 0.000s	sys 0.002s
W20260709 12:25:32.388912 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 169338us
I20260709 12:25:32.330729 10049 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:32.392055 31751 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:32.404284 10397 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.204s	user 0.000s	sys 0.001s
W20260709 12:25:32.406610 10385 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.286s	user 0.000s	sys 0.001s
W20260709 12:25:32.407559 10394 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.147s	user 0.000s	sys 0.000s
W20260709 12:25:32.421932 31745 consensus_meta.cc:287] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.650s	user 0.009s	sys 0.005s
I20260709 12:25:32.433465 31745 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 37.
W20260709 12:25:32.420825 31752 consensus_meta.cc:287] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.607s	user 0.011s	sys 0.000s
I20260709 12:25:32.364202 10033 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:32.431787 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 86292.4us
I20260709 12:25:32.377983 10342 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:32.434897 31752 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:32.436249 10335 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.461453 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:32.389299 10233 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:32.466660 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:32.467559 31752 raft_consensus.cc:2393] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:25:32.422297 10376 log.cc:927] Time spent T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.406s	user 0.001s	sys 0.001s
I20260709 12:25:32.464008 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:25:32.436398 10377 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.394s	user 0.001s	sys 0.000s
I20260709 12:25:32.455214 10033 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:32.459095 10342 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:32.512588 32375 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 35. Current term is 36. Ops: []
W20260709 12:25:32.517498 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.519584 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:32.402094 10058 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.479800 10288 raft_consensus.cc:3055] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Stepping down as leader of term 30
W20260709 12:25:32.529124 31717 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.573s	user 0.016s	sys 0.026s
W20260709 12:25:32.536245 31717 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.580s	user 0.016s	sys 0.026s
I20260709 12:25:32.492290 10378 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:32.543234 10253 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:32.543432 10383 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.355s	user 0.001s	sys 0.000s
I20260709 12:25:32.543927 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 26 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:32.535751 10288 raft_consensus.cc:740] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:25:32.547381 10399 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.096s	user 0.000s	sys 0.000s
I20260709 12:25:32.555047 10391 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:32.584481 10391 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:32.584802  9756 consensus_meta.cc:287] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.691s	user 0.008s	sys 0.004s
I20260709 12:25:32.584913 10391 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:32.585389  9756 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.585410 31751 raft_consensus.cc:2468] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
I20260709 12:25:32.586824 31744 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 33: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:32.555691 10417 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 33.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:32.587657 10245 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.463181 10335 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.593321 10335 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:32.594031 10335 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:32.522516 10234 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:32.556763 31753 consensus_meta.cc:287] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.516s	user 0.010s	sys 0.001s
I20260709 12:25:32.495822 10066 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:32.596624 31642 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:32.586727 10380 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.385s	user 0.001s	sys 0.001s
I20260709 12:25:32.580125 10058 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:32.580809 10398 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.077s	user 0.000s	sys 0.001s
I20260709 12:25:32.556985 10378 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:32.582921 10314 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:32.583266 10288 consensus_queue.cc:260] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.556149 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4354) took 1654 ms. Trace:
I20260709 12:25:32.596591 31748 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:32.556177 10389 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:25:32.601082 31747 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 34 index: 23. (index mismatch)
I20260709 12:25:32.570734 32371 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 26: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:32.619778 31753 raft_consensus.cc:2468] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 37.
I20260709 12:25:32.580209 10375 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:32.680752 31753 raft_consensus.cc:1240] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 37. Current term is 38. Ops: [37.25-37.25]
I20260709 12:25:32.655792 31820 rpcz_store.cc:276] 0709 12:25:30.901974 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:30.919411 (+ 17437us) service_pool.cc:224] Handling call
0709 12:25:30.959940 (+ 40529us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.046749 (+ 86809us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.046786 (+    37us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:31.083797 (+ 37011us) log.cc:844] Serialized 26 byte log entry
0709 12:25:31.572935 (+489138us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:31.572977 (+    42us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.573046 (+    69us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:32.555917 (+982871us) raft_consensus.cc:1685] finished
0709 12:25:32.555939 (+    22us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.556089 (+   150us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9731,"raft.run_cpu_time_us":168,"raft.run_wall_time_us":7583,"thread_start_us":429550,"threads_started":1,"wal-append.queue_time_us":836772}
I20260709 12:25:32.655092 10378 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:32.594589 10335 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.596415 10253 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.689589 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:32.646888 10234 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:25:32.692523 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 155684us
I20260709 12:25:32.655308 10427 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:32.665329 10389 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:32.707234 32342 leader_election.cc:336] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:32.679276 10066 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:32.730962 10429 raft_consensus.cc:2764] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election decision vote started in defunct term 33: won
I20260709 12:25:32.731590 10066 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.732681 10253 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:32.703794 10341 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:32.734892  9756 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:32.716586 10058 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:32.736215 10076 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:32.736688 10058 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:32.719830 10234 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:32.738561 10335 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:32.708808 10419 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.747068 10428 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:32.747726 10428 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.739931 10234 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:32.750154 10428 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:32.750875 10405 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:32.751447 10058 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:32.701581 10314 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:32.747112 10419 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.751385 10405 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:32.742751 10427 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.753523 10405 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.19, Last appended by leader: 19, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.754061 10419 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:32.752143 10314 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:32.755098 10427 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:32.742019 10013 consensus_meta.cc:287] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 1.038s	user 0.013s	sys 0.001s
W20260709 12:25:32.766139 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58917.9us
I20260709 12:25:32.716866 10389 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:25:32.736616 10384 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.081s	user 0.000s	sys 0.000s
I20260709 12:25:32.796353 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3387) took 1374 ms. Trace:
I20260709 12:25:32.795881 10391 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.796835 31746 rpcz_store.cc:276] 0709 12:25:31.421917 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.422063 (+   146us) service_pool.cc:224] Handling call
0709 12:25:31.435694 (+ 13631us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:31.436052 (+   358us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:31.436091 (+    39us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:31.436631 (+   540us) log.cc:844] Serialized 26 byte log entry
0709 12:25:31.859568 (+422937us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:31.859632 (+    64us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:31.859734 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:32.796058 (+936324us) raft_consensus.cc:1685] finished
0709 12:25:32.796085 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.796265 (+   180us) inbound_call.cc:177] Queueing success response
Metrics: {"thread_start_us":422216,"threads_started":1,"wal-append.queue_time_us":1119726}
I20260709 12:25:32.746345 10342 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:32.769302 32342 leader_election.cc:336] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:32.801702 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:32.801707 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:32.783845 10314 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:32.802704 32343 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.557s	user 0.006s	sys 0.009s
W20260709 12:25:32.820246 32343 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.574s	user 0.007s	sys 0.009s
I20260709 12:25:32.827942 10402 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:32.828629  9757 raft_consensus.cc:697] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:25:32.820639 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 574755us
I20260709 12:25:32.777695 10013 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:32.843209 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 379268us
I20260709 12:25:32.847304 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:32.798594 10391 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:32.851258 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:32.852236 31791 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:32.852308 10391 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:25:32.846683 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 77391.7us
I20260709 12:25:32.860101 31749 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 34: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:32.840862 10342 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:32.861687 31823 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:25:32.860775 31791 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:32.841048 10428 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:32.887040 31718 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.550s	user 0.000s	sys 0.056s
W20260709 12:25:32.888093 31718 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.551s	user 0.000s	sys 0.057s
W20260709 12:25:32.888474 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 552880us
I20260709 12:25:32.889449 31718 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:32.846920 10335 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:32.850339 10049 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:32.894348 31751 raft_consensus.cc:1240] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 30. Current term is 31. Ops: [30.21-30.21]
W20260709 12:25:32.862365 31643 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.619s	user 0.000s	sys 0.033s
I20260709 12:25:32.850421 10429 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:32.853271 10391 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:32.895118 31643 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.653s	user 0.000s	sys 0.034s
W20260709 12:25:32.900279 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 710243us
I20260709 12:25:32.901376 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:32.895572 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:32.924399 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 34 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:32.925246 31753 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:32.932070 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:32.934080 10423 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:32.934798 10276 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:32.935001 10426 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:32.935760 10426 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:32.936723 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1475) took 1935 ms. Trace:
I20260709 12:25:32.936863 10434 raft_consensus.cc:2804] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:32.937594 10434 raft_consensus.cc:697] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:32.935854 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:32.960282 31718 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 26 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:32.904501 10391 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:32.942555 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4376) took 1217 ms. Trace:
I20260709 12:25:32.965199 32342 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:32.898774 10335 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:32.938225 10426 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:32.967016 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:32.967621 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:32.968209 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:32.942108 10228 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:32.941146 10288 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:32.969785 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:32.918944  9756 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
W20260709 12:25:32.973426 10413 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.070s	user 0.000s	sys 0.002s
I20260709 12:25:32.976325 10436 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:32.937322 32370 rpcz_store.cc:276] 0709 12:25:31.001078 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.023313 (+ 22235us) service_pool.cc:224] Handling call
0709 12:25:31.036012 (+ 12699us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:31.214900 (+178888us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:25:31.822370 (+607470us) log.cc:844] Serialized 14 byte log entry
0709 12:25:32.189783 (+367413us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:32.189853 (+    70us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:32.189909 (+    56us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.936447 (+746538us) spinlock_profiling.cc:243] Waited 150 ms on lock 0x7b540074e854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:32.936513 (+    66us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.936657 (+   144us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":9044,"prepare.queue_time_us":144281,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":61,"raft.queue_time_us":263038,"raft.run_cpu_time_us":119,"raft.run_wall_time_us":119,"spinlock_wait_cycles":345494784,"thread_start_us":868708,"threads_started":3,"wal-append.queue_time_us":502333}
I20260709 12:25:32.918346 10342 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:32.915422 10420 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:32.979660 31752 rpcz_store.cc:276] 0709 12:25:31.725327 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.493778 (+768451us) service_pool.cc:224] Handling call
0709 12:25:32.494180 (+   402us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:32.494574 (+   394us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:32.494612 (+    38us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:32.527541 (+ 32929us) log.cc:844] Serialized 26 byte log entry
0709 12:25:32.788053 (+260512us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:25:32.788108 (+    55us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:32.788205 (+    97us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:32.942341 (+154136us) raft_consensus.cc:1685] finished
0709 12:25:32.942361 (+    20us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:32.942494 (+   133us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":436,"raft.queue_time_us":169962,"raft.run_cpu_time_us":160,"raft.run_wall_time_us":11301,"thread_start_us":142332,"threads_started":1,"wal-append.queue_time_us":249175}
I20260709 12:25:32.991762 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:33.000277 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 139556us
I20260709 12:25:32.938951 10414 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.990239 32377 raft_consensus.cc:2376] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31. Re-sending same reply.
I20260709 12:25:32.947412 10402 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:32.903589 10429 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:32.938766 10390 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:25:33.007836 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47651.7us
I20260709 12:25:33.027613 31718 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:33.030043 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:32.896924 10428 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.018854 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:33.020133 31792 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:32.997335 10152 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:32.941854 10387 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.077s
I20260709 12:25:33.015619 10436 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:33.035588 31791 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:33.037181 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 78809.7us
I20260709 12:25:32.965673 10423 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.061s
I20260709 12:25:33.042241 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:33.045753 31753 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:25:32.915885  9757 consensus_queue.cc:237] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:33.025100 10224 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:33.047713 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:33.052992 31718 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:33.056907 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.032677 10118 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.027213 10390 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.058857 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.059073 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:33.034608 10435 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.034505 10422 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader election lost for term 26. Reason: could not achieve majority
I20260709 12:25:33.058458 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:33.068495 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:33.069070 31818 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:33.062023 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:33.061015 10439 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.148s	user 0.001s	sys 0.000s
I20260709 12:25:33.065733 10430 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:33.047379  9756 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:33.040207 10049 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:33.070611 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:33.048089 10436 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:33.071841 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:32.938966 10434 consensus_queue.cc:237] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:33.022977 10228 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.027s
I20260709 12:25:33.059108 10426 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:33.066731 10118 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:33.061719 10342 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:33.080183 10388 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.045s
I20260709 12:25:33.084800 31750 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
I20260709 12:25:33.067759 10390 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:33.027487 10414 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.029594 10420 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.097015 10433 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:33.096880 10401 consensus_queue.cc:1059] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.059s
I20260709 12:25:33.097790 10410 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:33.097934 10396 log.cc:927] Time spent T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.082s	user 0.000s	sys 0.001s
I20260709 12:25:33.099838 10228 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:25:33.186678 10228 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:33.188016 10228 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 35.18, Last appended by leader: 18, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.189309 10228 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:33.103762 10429 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:33.074054 10013 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:33.104691 10342 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.191809 10428 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:33.192658 10335 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:33.090531 10435 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.193334 10342 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:33.075423 10282 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.030s
I20260709 12:25:33.085563 10432 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 16, Time since last communication: 0.013s
I20260709 12:25:33.107702 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:33.087787 10076 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:33.112131 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:33.199976 32371 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:33.113125 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:33.032498 10402 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:33.094700 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 59132.3us
I20260709 12:25:33.115688 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:33.202064 31821 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 33
I20260709 12:25:33.117683 10118 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:33.125684 10391 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:33.089902 10408 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.132460 31818 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
W20260709 12:25:33.133502 31749 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 4371) took 1409 ms (client timeout 1802 ms). Trace:
I20260709 12:25:33.137056 31744 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:25:33.136566 10378 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:33.137593  9756 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
W20260709 12:25:33.207764 31749 rpcz_store.cc:269] 0709 12:25:31.723850 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.951549 (+227699us) service_pool.cc:224] Handling call
0709 12:25:33.133241 (+1181692us) spinlock_profiling.cc:243] Waited 896 ms on lock 0x7b54002f76d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:33.133405 (+   164us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2060080384}
I20260709 12:25:33.146952 10049 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election lost for term 26. Reason: could not achieve majority
I20260709 12:25:33.147218 31748 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4378) took 1421 ms. Trace:
I20260709 12:25:33.175508 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:33.211900 31753 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:33.176568 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:33.185922 10401 raft_consensus.cc:3055] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:33.220762 10401 raft_consensus.cc:740] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:25:33.221683 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 114062us
I20260709 12:25:33.221752 10401 consensus_queue.cc:260] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.222726 31718 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:33.222890 10402 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:33.224087 31718 leader_election.cc:400] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Vote denied by peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855) with higher term. Message: Invalid argument: T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 36. Current term is 37.
I20260709 12:25:33.224638 10423 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:33.190711 10410 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.224969 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:33.105273 10033 consensus_meta.cc:287] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.571s	user 0.009s	sys 0.001s
I20260709 12:25:33.227058 10432 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:33.227735 10432 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:33.227478 10033 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.228183 10414 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:33.228444 10417 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Stepping down as leader of term 32
W20260709 12:25:33.202066 31791 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:33.229521 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.229871 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:33.210701 31748 rpcz_store.cc:276] 0709 12:25:31.725887 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.521545 (+795658us) service_pool.cc:224] Handling call
0709 12:25:32.596410 (+ 74865us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:32.664814 (+ 68404us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.146951 (+482137us) spinlock_profiling.cc:243] Waited 53.6 ms on lock 0x7b54000b86d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:33.147138 (+   187us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":123371264}
I20260709 12:25:33.230268 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.205230 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:33.205579 32373 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 31: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:33.219468 10282 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:33.226315 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:33.237143 32371 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
W20260709 12:25:33.225056 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 190680us
I20260709 12:25:33.238868 31821 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:33.078455 10430 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.259676 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.269604 31818 raft_consensus.cc:2393] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 33: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:33.267855 31745 raft_consensus.cc:1240] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:33.247063 31718 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:33.247803 32342 consensus_peers.cc:597] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:33.272564 31820 raft_consensus.cc:2410] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:25:33.228982 10417 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:33.279794 10417 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.18, Last appended by leader: 18, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:33.282155 10417 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:33.288527 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.289083 10378 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:33.289229 31818 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:33.290010 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:33.280860 32342 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:33.283778 10405 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:33.290742 31753 raft_consensus.cc:2468] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 35.
I20260709 12:25:33.283901 10420 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:33.284987 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 82955.4us
I20260709 12:25:33.291805 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:33.292470 31791 leader_election.cc:336] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:33.293013 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:33.271446 10013 consensus_queue.cc:1059] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.023s
I20260709 12:25:33.295346 10013 raft_consensus.cc:3055] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:25:33.295845 10013 raft_consensus.cc:740] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:33.296895 10013 consensus_queue.cc:260] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 37.22, Last appended by leader: 22, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.298272 10013 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:33.225060 10423 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:33.298874 10423 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:33.271906 10435 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:33.302598 10033 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:33.305035 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.305172 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:33.306123 10414 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:25:33.306604 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.266342 10404 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:33.290302 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.307490 10404 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:33.307983 10404 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:33.308076 31818 raft_consensus.cc:2468] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:33.241771 10408 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.309573 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.238910 10282 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:33.313102 10282 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 31.17, Last appended by leader: 17, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:33.313092 31792 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:33.318864 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.307474 31821 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:33.320356 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.320809 31752 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:25:33.320924 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.321020 31827 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:33.322074 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.322738 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.323829 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:33.324954 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:33.286002 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:33.321115 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.330571 31744 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:33.327814 31819 raft_consensus.cc:1275] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 35 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
I20260709 12:25:33.332314 31820 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 26 index: 17. (index mismatch)
I20260709 12:25:33.333104 31827 raft_consensus.cc:2468] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:33.335033 31820 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
W20260709 12:25:33.338498 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.311406 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.340385 31818 raft_consensus.cc:2468] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 37.
I20260709 12:25:33.247543 31642 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:33.354564 10408 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:33.357019 31821 raft_consensus.cc:2393] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 31: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:25:33.358771 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 77953.2us
W20260709 12:25:33.373471 32342 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:33.359230 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 66799.5us
I20260709 12:25:33.359912 10390 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:33.375686 31791 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:33.376161 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:33.379990 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 132527us
W20260709 12:25:33.380764 31717 leader_election.cc:336] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:33.382459 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 135466us
I20260709 12:25:33.383168 31749 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:25:33.390146 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:33.381633 31717 leader_election.cc:336] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:33.393143 31745 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4380) took 1666 ms. Trace:
I20260709 12:25:33.393527 31745 rpcz_store.cc:276] 0709 12:25:31.726498 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.538197 (+811699us) service_pool.cc:224] Handling call
0709 12:25:32.596042 (+ 57845us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.270804 (+674762us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.392902 (+122098us) spinlock_profiling.cc:243] Waited 671 ms on lock 0x7b54002f6cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:33.393084 (+   182us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1544320256}
W20260709 12:25:33.374307 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:33.402829 10422 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:33.403555 10422 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.405829 10391 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:25:33.408392 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.409132 31819 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 36.
I20260709 12:25:33.410095 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.379631 10401 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:33.410806 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 97807.6us
I20260709 12:25:33.411265 31827 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:25:33.411648 31792 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.412266 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:33.415380 10430 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:33.415992 10408 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:33.416406 10408 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:33.416457 10389 consensus_meta.cc:287] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.624s	user 0.013s	sys 0.005s
I20260709 12:25:33.423446 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.423858 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.417577 32377 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: [25.14-25.14]
W20260709 12:25:33.417661 31717 leader_election.cc:336] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:33.419395 32373 raft_consensus.cc:2393] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 36: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:25:33.425308 31717 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:33.426229 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:33.427006 31744 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:33.428313 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:33.431530 31822 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:33.431023 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.434759 31821 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 36 because replica has last-logged OpId of term: 35 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 16.
I20260709 12:25:33.439085 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:33.437155 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:33.438541 10282 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:33.426007 10435 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.441197 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:33.441335 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.471985 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.442232 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:33.442395 10390 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:33.442492 10233 raft_consensus.cc:3060] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:33.443120 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 67461.6us
I20260709 12:25:33.444698 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:33.445111 31818 raft_consensus.cc:1240] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 25. Current term is 26. Ops: []
I20260709 12:25:33.445691 31748 raft_consensus.cc:1275] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:25:33.417035  9756 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.391702 10378 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:33.460911 10387 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.461519 10331 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:25:33.466339 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:33.474128 31821 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 34: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:33.422765 10408 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:33.477571 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.448405 10228 raft_consensus.cc:2804] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:33.488531 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.486912 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:33.447302 10258 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:33.468839 10426 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:25:33.541049 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 54256.3us
I20260709 12:25:33.513125 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:33.542150 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.481735 10235 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:33.501544 10258 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:33.428701 10389 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.500489 10276 raft_consensus.cc:2804] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:33.558406 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:33.562322 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:33.562762 31818 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 33. Current term is 34. Ops: [33.19-33.19]
I20260709 12:25:33.480583 10435 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:33.481140 10066 consensus_queue.cc:1059] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 28.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.068s
I20260709 12:25:33.576169 31824 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 37. Current term is 38. Ops: []
W20260709 12:25:33.568638 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55322.3us
I20260709 12:25:33.568531 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:33.486016 10058 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.067s
I20260709 12:25:33.515347 10228 raft_consensus.cc:697] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:33.486755  9756 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.550595 10245 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:25:33.554854 10190 consensus_queue.cc:1048] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.028s
I20260709 12:25:33.553706 10258 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:33.560374 10276 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:33.585471 31791 leader_election.cc:336] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:33.593035 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51003.6us
I20260709 12:25:33.591621 31744 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 35.
I20260709 12:25:33.593971 31643 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:33.595209 31753 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 33 index: 18. (term mismatch)
I20260709 12:25:33.595613 31753 pending_rounds.cc:85] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 17
I20260709 12:25:33.596114 31749 raft_consensus.cc:1240] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 34. Current term is 35. Ops: [34.23-34.23]
I20260709 12:25:33.497131 10387 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.592476 10213 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:33.547726 10224 consensus_queue.cc:1048] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.040s
I20260709 12:25:33.583326 10409 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:33.601197 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:33.602644 10424 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:33.607307 31745 raft_consensus.cc:1275] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 35 index: 23. Preceding OpId from leader: term: 37 index: 24. (index mismatch)
I20260709 12:25:33.595868 31753 pending_rounds.cc:107] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 32.18
I20260709 12:25:33.612975 31753 raft_consensus.cc:2889] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:33.597133 10276 raft_consensus.cc:3060] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:33.627873 10118 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.610550 10033 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election won for term 33
W20260709 12:25:33.646310 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45298.3us
I20260709 12:25:33.647581 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:33.658089 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 207688us
I20260709 12:25:33.660095 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:33.622067 10404 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.672820 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.642536 10424 raft_consensus.cc:2804] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:33.681565 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.642091 10409 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 36 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:33.682760 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:33.686990 31752 raft_consensus.cc:1240] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:33.688299 10225 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:33.676373 10013 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:33.689309 10464 consensus_queue.cc:1059] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 25.14, Next index: 15, Last known committed idx: 12, Time since last communication: 0.001s
I20260709 12:25:33.689908 10013 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
I20260709 12:25:33.690690 10013 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Stepping down as leader of term 25
I20260709 12:25:33.691072 10013 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 25 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:33.691736 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:33.691991 10013 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.14, Last appended by leader: 14, Current term: 25, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.692742 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.693619 10013 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 25 FOLLOWER]: Advancing to term 26
I20260709 12:25:33.694211 10245 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:33.694859 10245 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.695303 10245 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:33.694281 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:33.698513 10066 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:33.610183 10228 consensus_queue.cc:237] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.705370 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:33.679298 10033 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:33.710033 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.680683 10422 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:33.684206 10424 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:33.682413 10403 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:33.678260 10465 raft_consensus.cc:2804] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:33.663084 10118 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:33.702618 10455 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:25:33.722667 10465 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:33.728464 10288 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.710s	user 0.008s	sys 0.012s
I20260709 12:25:33.733721 10404 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:33.736428 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:33.715559 10424 raft_consensus.cc:3060] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:33.738448 10233 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:33.715255 10066 raft_consensus.cc:2804] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:25:33.743723 10468 raft_consensus.cc:3060] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:33.743933 10387 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:33.730422 10462 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.109s	user 0.002s	sys 0.000s
I20260709 12:25:33.711622 10058 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.047s
I20260709 12:25:33.763078 10342 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:33.763399 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:33.765725 10410 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:33.738363 10436 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.644s	user 0.009s	sys 0.005s
I20260709 12:25:33.766081 31750 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 17. (term mismatch)
I20260709 12:25:33.770849 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:33.799166 31750 pending_rounds.cc:85] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 16
W20260709 12:25:33.783615 10461 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.105s	user 0.001s	sys 0.000s
I20260709 12:25:33.714183 10213 raft_consensus.cc:2804] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:33.789312 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:33.789971 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:33.796857 31749 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
I20260709 12:25:33.807027 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:33.807922 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:33.766813  9756 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:33.701941 10234 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.072s
W20260709 12:25:33.682286 31562 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4d48c83c761f48079bc6ea176649dda0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 26 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:33.710417 10225 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:33.799539 31750 pending_rounds.cc:107] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 31.17
I20260709 12:25:33.818588 31750 raft_consensus.cc:2889] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:33.802614 31747 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 34 index: 19. Preceding OpId from leader: term: 35 index: 19. (term mismatch)
I20260709 12:25:33.819300 31747 pending_rounds.cc:85] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 18
I20260709 12:25:33.819626 31747 pending_rounds.cc:107] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 34.19
I20260709 12:25:33.819923 31747 raft_consensus.cc:2889] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:33.790202 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:33.783087 10465 raft_consensus.cc:3060] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:33.796128 10066 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:33.815851 31562 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 173164us
I20260709 12:25:33.796511 10389 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:33.799535 31752 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:25:33.827970 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:33.795536 10335 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:33.810303 10213 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:33.835412 10213 raft_consensus.cc:3060] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:33.835497 10258 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:33.837944 10258 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:33.838738 10033 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.843415 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:33.848485 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:33.828899 10066 raft_consensus.cc:3060] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:33.824321 10435 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:33.805081 10436 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.847671 10331 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:33.810310 10409 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 34.17, Last appended by leader: 17, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.894487 31826 raft_consensus.cc:1240] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 36. Current term is 37. Ops: []
I20260709 12:25:33.847303 10402 consensus_queue.cc:1059] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 32.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.034s
I20260709 12:25:33.844262 10386 consensus_queue.cc:1048] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.049s
W20260709 12:25:33.921051 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 126329us
I20260709 12:25:33.844874 10335 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.876855 10263 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election won for term 37
W20260709 12:25:33.924911 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 235711us
W20260709 12:25:33.928140 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 439705us
I20260709 12:25:33.935670 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:33.946504 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:33.941244 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 223857us
I20260709 12:25:33.948426 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:33.949468 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:33.953496 31718 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:33.934553 10076 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:25:33.961113 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:33.961858 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.947394 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:33.963594 31718 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:33.964834 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:33.962194 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 35 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:33.941788 10403 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:33.939755 10263 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:33.967499 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.968217 31791 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.923242 10276 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:33.970671 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:33.969682 31643 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:33.972074 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:33.975803 31744 raft_consensus.cc:2393] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:33.964322 10282 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:33.958343 10470 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 31.18, Next index: 19, Last known committed idx: 18, Time since last communication: 0.021s
I20260709 12:25:33.989235 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:33.972234 10258 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:33.969635 10013 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:33.975616 10118 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:33.985744 10390 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:34.040140 10390 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:34.040553 10390 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:33.992203 10423 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.693s	user 0.012s	sys 0.004s
I20260709 12:25:34.043565 10423 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.047269 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:34.048415 31749 raft_consensus.cc:2393] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 38: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:34.048480 10401 raft_consensus.cc:3055] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:34.049006 10401 raft_consensus.cc:740] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:34.049887 10401 consensus_queue.cc:260] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.23, Last appended by leader: 23, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.069569 10401 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:34.073657 31824 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1989) took 1227 ms. Trace:
I20260709 12:25:34.074235 31824 rpcz_store.cc:276] 0709 12:25:32.845996 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.846174 (+   178us) service_pool.cc:224] Handling call
0709 12:25:32.859054 (+ 12880us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.576689 (+717635us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.073373 (+496684us) spinlock_profiling.cc:243] Waited 717 ms on lock 0x7b5400446bd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:34.073583 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1648701184}
I20260709 12:25:33.968452 10428 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:33.993870 10435 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:33.994596 10433 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:33.997149 10470 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:33.997874 10477 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:33.999238 10235 raft_consensus.cc:3055] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
W20260709 12:25:34.096470 10473 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.068s	user 0.004s	sys 0.000s
I20260709 12:25:34.006805 10234 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:34.034276 10245 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.007483 10378 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.008800 10426 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:34.012521 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:34.012923 32373 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 21. (term mismatch)
I20260709 12:25:34.013176 10479 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.129743 32373 pending_rounds.cc:85] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 20
I20260709 12:25:34.130244 32373 pending_rounds.cc:107] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 30.21
I20260709 12:25:34.130121 10479 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.019311 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:34.023319 10436 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:34.023824 31717 leader_election.cc:336] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.022806 10233 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.133381 10233 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.136080 10233 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:34.103427 10235 raft_consensus.cc:740] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
W20260709 12:25:34.147048 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 176457us
I20260709 12:25:34.033289 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:34.034744 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:34.035434 31744 raft_consensus.cc:1275] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 29 index: 17. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:25:34.148799 31745 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:34.051548 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45178 (request call id 4385) took 1417 ms. Trace:
I20260709 12:25:34.012961 10408 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.149466 31747 rpcz_store.cc:276] 0709 12:25:32.634541 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.726977 (+ 92436us) service_pool.cc:224] Handling call
0709 12:25:32.727483 (+   506us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:33.825446 (+1097963us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.051230 (+225784us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b54003e30d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:34.051469 (+   239us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2460350336}
I20260709 12:25:34.067724 31750 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32. Ops: []
W20260709 12:25:34.096208 10478 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.097s	user 0.001s	sys 0.003s
W20260709 12:25:34.102039 10466 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.137s	user 0.000s	sys 0.004s
W20260709 12:25:34.096176 10459 log.cc:927] Time spent T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.247s	user 0.003s	sys 0.001s
W20260709 12:25:34.102761 10469 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.120s	user 0.004s	sys 0.000s
I20260709 12:25:33.971627  9756 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:34.103353 10458 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.280s	user 0.001s	sys 0.004s
W20260709 12:25:34.103760 10456 log.cc:927] Time spent T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.553s	user 0.007s	sys 0.000s
I20260709 12:25:33.970541 10263 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:34.124714 10213 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.102480 10424 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.130769 32373 raft_consensus.cc:2889] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:34.131706 31717 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:34.189400 31746 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1591) took 1296 ms. Trace:
I20260709 12:25:34.169909 10435 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.104033 10470 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 33 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:33.998802 10403 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.148844 31752 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:34.036379 10276 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:34.172032 31825 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:34.178160 31821 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 33 index: 18. (index mismatch)
I20260709 12:25:34.178041 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:34.178798 10234 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:34.179167 31642 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:34.179033 10288 consensus_queue.cc:1059] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 28.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:34.183564 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:34.184528 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:34.182708 31745 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 35.
I20260709 12:25:34.185456 32376 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 35 because replica has last-logged OpId of term: 34 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 33 index: 23.
I20260709 12:25:34.185376 10235 consensus_queue.cc:260] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 16, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.186357 10464 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:34.225692 10464 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:34.192723 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 168920us
I20260709 12:25:34.227855 10263 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.190116 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:34.166815 10426 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.171784 10423 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:34.145648 10378 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.172936 10472 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.204075 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:34.204901 31824 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
W20260709 12:25:34.253298 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.208459 10436 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.212831 10245 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:34.213097 10408 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:34.213657 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:34.202379 10386 consensus_queue.cc:1059] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 36.23, Next index: 24, Last known committed idx: 23, Time since last communication: 0.045s
I20260709 12:25:34.215121 10235 raft_consensus.cc:3060] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
W20260709 12:25:34.222721 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.223588 10228 consensus_queue.cc:1048] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:34.224329 10213 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:34.243294 10342 consensus_queue.cc:1059] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 32.24, Next index: 25, Last known committed idx: 23, Time since last communication: 0.018s
I20260709 12:25:34.190807 31826 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:25:34.226783 31791 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:34.202826 10435 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.170387 10390 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.027880 10335 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:34.233212 10288 raft_consensus.cc:3055] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Stepping down as leader of term 28
I20260709 12:25:34.233596 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 1996) took 1027 ms. Trace:
I20260709 12:25:34.192327 31746 rpcz_store.cc:276] 0709 12:25:32.892767 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:32.892918 (+   151us) service_pool.cc:224] Handling call
0709 12:25:32.914888 (+ 21970us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:32.917392 (+  2504us) raft_consensus.cc:1534] Early marking committed up to index 22
0709 12:25:33.145994 (+228602us) spinlock_profiling.cc:243] Waited 1.55 ms on lock 0x7b1800004f60. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8fc97d4bc 00007fe8f9ccb48b 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce5455 00007fe9007cd3c7 00007fe8fcddf9a7 00007fe8fcdef0bf
0709 12:25:33.146198 (+   204us) log.cc:844] Serialized 14 byte log entry
0709 12:25:33.384063 (+237865us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:33.418299 (+ 34236us) log.cc:844] Serialized 26 byte log entry
0709 12:25:33.418438 (+   139us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:25:33.425052 (+  6614us) log.cc:844] Serialized 14 byte log entry
0709 12:25:33.425332 (+   280us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.425434 (+   102us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:33.732518 (+307084us) spinlock_profiling.cc:243] Waited 4.34 ms on lock 0x7b1800004f60. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:34.008931 (+276413us) spinlock_profiling.cc:243] Waited 3.9 ms on lock 0x7b1800004f60. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce83f3 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:34.024343 (+ 15412us) spinlock_profiling.cc:243] Waited 4.77 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:34.189161 (+164818us) raft_consensus.cc:1685] finished
0709 12:25:34.189194 (+    33us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:34.189339 (+   145us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":15570,"prepare.queue_time_us":128109,"prepare.run_cpu_time_us":103,"prepare.run_wall_time_us":4666,"raft.queue_time_us":37672,"raft.run_cpu_time_us":441,"raft.run_wall_time_us":60263,"spinlock_wait_cycles":33480448,"thread_start_us":310661,"threads_started":2,"wal-append.queue_time_us":322784}
I20260709 12:25:34.228696 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:34.209123 31819 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 37.
I20260709 12:25:34.228750 10414 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 37.24, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:25:34.228118 10464 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:34.170253 10233 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.294350 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:34.243698 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 53663.5us
I20260709 12:25:34.295892 31823 rpcz_store.cc:276] 0709 12:25:33.206099 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.234718 (+ 28619us) service_pool.cc:224] Handling call
0709 12:25:33.269039 (+ 34321us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:33.418219 (+149180us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:33.418252 (+    33us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:33.418714 (+   462us) log.cc:844] Serialized 26 byte log entry
0709 12:25:33.932083 (+513369us) raft_consensus.cc:1645] Marking committed up to 23
0709 12:25:33.932134 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:33.932224 (+    90us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:34.233319 (+301095us) raft_consensus.cc:1685] finished
0709 12:25:34.233346 (+    27us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:34.233523 (+   177us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":9316,"raft.run_cpu_time_us":131,"raft.run_wall_time_us":9966,"thread_start_us":455159,"threads_started":1,"wal-append.queue_time_us":537798}
I20260709 12:25:34.298946 31826 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:34.235510 10118 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:34.200448 10282 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:34.249389 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:34.215198 10470 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.19, Last appended by leader: 19, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.267484 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:34.305614 31819 raft_consensus.cc:1240] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 26. Current term is 27. Ops: [26.17-26.17]
I20260709 12:25:34.268417 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:34.242149 10468 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:34.256821 10475 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.087s	user 0.000s	sys 0.001s
I20260709 12:25:34.291764 10477 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:34.273597 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:34.225718  9757 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 31.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:25:34.284839 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 58130.8us
I20260709 12:25:34.282335 10190 raft_consensus.cc:3055] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:25:34.285943 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:34.286470 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:34.287705 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:34.288877 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:34.292901 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60441.1us
I20260709 12:25:34.293578 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:34.295877 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:34.251361 10058 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.036s
I20260709 12:25:34.247967 10033 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.039s
W20260709 12:25:34.247742 10481 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.114s	user 0.003s	sys 0.000s
I20260709 12:25:34.260284 10471 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.038s
I20260709 12:25:34.276005 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:34.311741 32371 raft_consensus.cc:1275] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
W20260709 12:25:34.379778 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 103967us
I20260709 12:25:34.303867 10479 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:34.350811 31824 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:34.317109 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:34.382803 10429 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.383139 31750 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:34.382771 10479 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:34.314965 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:34.358219 31752 raft_consensus.cc:2468] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 35.
I20260709 12:25:34.384296 10479 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.384943 10479 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.385246 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 29 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:34.370590 31753 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:34.386657 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:34.387462 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:34.271411 10436 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.371398 31820 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:34.372484 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:34.371683 32369 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 38.
I20260709 12:25:34.295661 10288 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:34.371171 31749 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 38.
I20260709 12:25:34.383526 10429 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.388758 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:34.423727 31752 raft_consensus.cc:2410] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 39 because replica has last-logged OpId of term: 37 index: 22, which is greater than that of the candidate, which has last-logged OpId of term: 36 index: 21.
I20260709 12:25:34.388069 31753 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:34.425953 31824 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:34.309821 10276 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.282956 10474 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 16, Time since last communication: 0.087s
I20260709 12:25:34.308878 10433 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:34.454849 31745 raft_consensus.cc:1240] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 28. Current term is 29. Ops: []
I20260709 12:25:34.394851 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4371) took 1344 ms. Trace:
I20260709 12:25:34.387475 32371 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:34.295580 10066 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.372959 10228 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.463645 31825 rpcz_store.cc:276] 0709 12:25:33.050805 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:33.050989 (+   184us) service_pool.cc:224] Handling call
0709 12:25:33.071646 (+ 20657us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.206988 (+1135342us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:34.394537 (+187549us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400379e54. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:34.394775 (+   238us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2530160000}
I20260709 12:25:34.357425 10234 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.468976 31749 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:25:34.262912 10472 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.395097 10013 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:34.373517 10190 raft_consensus.cc:740] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:34.378032 10470 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:34.423789 10342 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.308310 10233 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.372541 10477 raft_consensus.cc:697] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:34.476853 10228 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:34.486197 10485 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.091s	user 0.000s	sys 0.000s
I20260709 12:25:34.486733 10436 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:34.487378 10472 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:34.487846 10486 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.109s	user 0.000s	sys 0.000s
I20260709 12:25:34.490854 31818 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 35 index: 21. Preceding OpId from leader: term: 37 index: 22. (index mismatch)
I20260709 12:25:34.420120 10424 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:34.503170 31750 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
W20260709 12:25:34.504570 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 122795us
I20260709 12:25:34.465741 10423 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.515990 10390 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:34.394820 10225 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.111s
I20260709 12:25:34.521600 31820 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:34.468169 10468 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.042s
I20260709 12:25:34.455925 10276 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.519560 31643 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:34.519438 10484 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.273s	user 0.001s	sys 0.001s
I20260709 12:25:34.504405 10118 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 35 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:34.547591 31643 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:34.553264 31753 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:34.480409 10234 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.559273 10118 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 33.23, Last appended by leader: 23, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.612066 10276 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:34.612776 10234 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:34.560577 10426 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:34.561203 10263 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:34.555832 10190 consensus_queue.cc:260] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:34.571359 10487 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.054s	user 0.000s	sys 0.000s
W20260709 12:25:34.575376 10465 consensus_meta.cc:287] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.743s	user 0.008s	sys 0.003s
I20260709 12:25:34.578253 10066 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:34.588122 10429 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:34.588788 10288 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 28, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:34.589819 10483 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.311s	user 0.000s	sys 0.001s
I20260709 12:25:34.524569 10423 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.592582 10480 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Stepping down as leader of term 31
I20260709 12:25:34.595280 10378 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:34.595707 10474 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:34.471071 10433 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.596792 10402 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 37 LEADER]: Stepping down as leader of term 37
I20260709 12:25:34.696134 10402 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 37 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:34.697232 10402 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 37, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.698014 10433 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:34.597921 30503 thread.cc:527] Waited for 1000ms trying to join with heartbeat (tid 31703)
I20260709 12:25:34.701056 10228 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:34.698395 10402 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:34.703028 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:34.704279 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.554526 10424 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:25:34.704982 10424 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:34.705348 10424 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:25:34.707834 10496 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:34.600489 10408 raft_consensus.cc:3055] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:34.710815 10408 raft_consensus.cc:740] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:34.711849 10408 consensus_queue.cc:260] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 32.24, Last appended by leader: 24, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.712347  9756 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:34.713302 10408 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:34.713192  9756 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.604569 10233 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:34.549780 10435 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:34.642172 31717 leader_election.cc:336] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.714758 10497 consensus_queue.cc:1059] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 24.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:34.715525 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:34.716394 10497 raft_consensus.cc:3055] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Stepping down as leader of term 26
I20260709 12:25:34.717077 10497 raft_consensus.cc:740] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:34.719274 10477 consensus_queue.cc:237] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.21, Last appended by leader: 21, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:34.725978 10423 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:34.640677 10426 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:34.727072 10426 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.730185 10426 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:34.672585 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:34.674683 31562 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4d48c83c761f48079bc6ea176649dda0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 26 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:34.735361 31562 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 60772.1us
I20260709 12:25:34.678941 31818 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 37. Current term is 38. Ops: []
I20260709 12:25:34.679634 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:34.680663 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:34.686754 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:34.688387 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:34.660197 10465 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.691283 31825 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 33. Current term is 34. Ops: [33.19-33.19]
I20260709 12:25:34.691540 10479 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:34.695279 10474 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:34.716455 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 74342.1us
W20260709 12:25:34.743291 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54970.6us
I20260709 12:25:34.717150 10432 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:25:34.755282 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 71801.1us
I20260709 12:25:34.756075 31792 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:34.681282 10480 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:25:34.731712 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:34.648504 10190 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:34.649042 10428 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.648670 10013 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.665014 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:34.669072 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:34.739486 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:34.774456 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 105541us
I20260709 12:25:34.776932 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:34.779211 32377 raft_consensus.cc:1240] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 36. Current term is 37. Ops: []
I20260709 12:25:34.741245 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 29 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:34.743395 31717 leader_election.cc:336] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.784395 31717 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:34.507645 10342 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:34.774204 32376 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:25:34.788566 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:34.752187 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:34.785317 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:34.756209 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:34.761683 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:34.791469 31818 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 29.
I20260709 12:25:34.770344 31792 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:34.771397 32375 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:25:34.792089 31820 raft_consensus.cc:2410] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 34 because replica has last-logged OpId of term: 31 index: 18, which is greater than that of the candidate, which has last-logged OpId of term: 30 index: 17.
I20260709 12:25:34.788682 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:34.763080 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:34.789068 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47957.4us
I20260709 12:25:34.789368 31748 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:34.797444 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 24 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:34.751197 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 176570us
I20260709 12:25:34.803574 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:34.753674 10474 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:34.805271 31822 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:25:34.799201 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:34.807053 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:34.807400 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64051.7us
I20260709 12:25:34.808635 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:34.809510 31824 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:34.809994 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 35 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:34.806993 31751 raft_consensus.cc:2410] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 32 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 20.
I20260709 12:25:34.807094 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:34.768707 10435 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:25:34.806181 31643 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:34.778084 10476 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:25:34.809125 31717 leader_election.cc:336] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:34.818588 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:34.819326 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:34.818702 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:34.818128 31718 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:34.790096 10464 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.786763 10404 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:34.820128 31752 raft_consensus.cc:2410] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:25:34.793983 10233 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader election lost for term 29. Reason: could not achieve majority
I20260709 12:25:34.785672 10228 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:25:34.822220 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:34.822554 31749 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 35: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:34.789746 10470 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.718318 10497 consensus_queue.cc:260] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 26, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.837109 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:34.803957 10423 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:34.835129 31826 raft_consensus.cc:2410] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 27 because replica has last-logged OpId of term: 25 index: 14, which is greater than that of the candidate, which has last-logged OpId of term: 23 index: 13.
I20260709 12:25:34.837895 31752 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:25:34.828393 10403 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:34.790552 10013 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.874357 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:34.791049 10428 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.871886 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:34.816092 10258 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:25:34.871964 31643 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:34.916936 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 45084.9us
I20260709 12:25:34.798256 10076 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:34.880599 10433 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:34.928051 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:34.899488  9756 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:34.816478 10049 consensus_queue.cc:1048] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.060s
I20260709 12:25:34.903749 10342 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:34.906121 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:34.884783 10473 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.051s	user 0.001s	sys 0.000s
I20260709 12:25:34.841490 10470 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.818670 10410 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.072s
I20260709 12:25:34.904775 10465 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:34.872557 10335 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:34.920225 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 102195us
I20260709 12:25:34.833657 10464 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:34.935711 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:34.907998 31821 raft_consensus.cc:2393] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 35: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:34.943186 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:34.915827 10497 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:34.947834 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:34.895691 10423 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.915104 10403 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:34.930394 10118 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:34.950939 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:34.955317 31750 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
I20260709 12:25:34.939299 10234 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:34.963745 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:34.957417 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:34.968319 10426 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:34.968837 10426 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:34.969242 10426 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:34.970762 10234 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:34.971302 10234 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:34.971855 31745 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 31. Current term is 32. Ops: []
I20260709 12:25:34.972028 10433 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:25:34.972276 10235 consensus_meta.cc:287] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.694s	user 0.011s	sys 0.001s
I20260709 12:25:34.972586 10433 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:34.972967 10433 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:34.971179 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:34.974558 31821 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 38: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:34.840399 10480 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 31, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:34.934309 10496 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.021s
I20260709 12:25:34.957473 10335 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:35.018157 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 203189us
I20260709 12:25:35.019809 31821 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:25:35.022306 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:34.991629 10118 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:34.991351 10493 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Leader pre-election won for term 39
W20260709 12:25:35.018947 32342 leader_election.cc:336] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.024176 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:35.006877 10066 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:35.022718 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:35.035461 32342 leader_election.cc:336] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.042963 32342 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:35.044579 32342 leader_election.cc:336] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.045888 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:35.025005 10499 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.150s	user 0.000s	sys 0.001s
W20260709 12:25:35.059314 10503 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.113s	user 0.001s	sys 0.001s
I20260709 12:25:35.060952 10013 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:35.063548 10464 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:35.066395 10470 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:35.043256 10403 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.17, Last appended by leader: 17, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:35.027422 10498 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.086s	user 0.000s	sys 0.001s
I20260709 12:25:35.063110 10118 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:35.026804 10493 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:35.082283 32342 leader_election.cc:336] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:35.071966 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 259559us
I20260709 12:25:35.087419 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:35.040804 10066 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:35.090518 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:35.081831 10506 log.cc:927] Time spent T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.087s	user 0.001s	sys 0.001s
I20260709 12:25:35.049738 10190 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:35.049522 10420 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:35.097043 10424 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 39 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:35.088649 10493 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:25:35.087376 10477 consensus_queue.cc:1059] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: INVALID_TERM, Last received: 36.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.011s
I20260709 12:25:35.028223 10213 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.056s
I20260709 12:25:35.116552 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:35.093251 10066 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:35.111194 10435 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:25:35.131220 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:35.135403 31827 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:35.137030 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:35.141033 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:35.158578 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:35.119319 10496 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:35.124838 10428 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:35.134874 10479 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.174686 10470 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:35.222608 10390 raft_consensus.cc:3055] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:25:35.223278 10390 raft_consensus.cc:740] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:35.226542 31791 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:35.229476 10228 raft_consensus.cc:2804] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:35.230365 10228 raft_consensus.cc:697] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:35.231727 10228 consensus_queue.cc:237] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.20, Last appended by leader: 20, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:35.234685 31791 leader_election.cc:336] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:35.236063 31791 leader_election.cc:336] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.242378 10390 consensus_queue.cc:260] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:35.243093 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:35.244834 10390 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:25:35.260982 10514 log.cc:927] Time spent T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.226s	user 0.002s	sys 0.000s
I20260709 12:25:35.284174 10335 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:35.292040 10480 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:35.192775 10470 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:35.316946 31747 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:45166 (request call id 3418) took 1134 ms. Trace:
I20260709 12:25:35.323367 32370 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 34 index: 23. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:25:35.325397 10528 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:25:35.328830 10424 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:35.300141 10487 log.cc:927] Time spent T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.108s	user 0.000s	sys 0.001s
I20260709 12:25:35.184926 10479 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:35.368724 10470 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:35.361685 31747 rpcz_store.cc:276] 0709 12:25:34.182712 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:34.182841 (+   129us) service_pool.cc:224] Handling call
0709 12:25:34.187164 (+  4323us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:34.270372 (+ 83208us) raft_consensus.cc:1534] Early marking committed up to index 20
0709 12:25:34.465919 (+195547us) log.cc:844] Serialized 14 byte log entry
0709 12:25:35.086368 (+620449us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:35.086428 (+    60us) raft_consensus.cc:1645] Marking committed up to 20
0709 12:25:35.090919 (+  4491us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.316546 (+225627us) spinlock_profiling.cc:243] Waited 82.8 ms on lock 0x7b54000f3cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:35.316630 (+    84us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:35.316840 (+   210us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20574,"prepare.queue_time_us":78232,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":33548,"raft.run_cpu_time_us":157,"raft.run_wall_time_us":6451,"spinlock_wait_cycles":190422272,"thread_start_us":776841,"threads_started":3,"wal-append.queue_time_us":351596}
I20260709 12:25:35.321863 10233 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.058s
I20260709 12:25:35.477492 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:35.479067 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:35.481626 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:35.477543 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 241509us
I20260709 12:25:35.496151 31752 raft_consensus.cc:1240] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 36. Current term is 37. Ops: []
W20260709 12:25:35.498967 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 278977us
W20260709 12:25:35.500456 31791 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.506953 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:35.508008 10477 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:35.508407 31791 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.508980 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:35.511646 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:35.512172 10477 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:35.512660 31752 raft_consensus.cc:2393] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 35: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:25:35.513468 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 338849us
W20260709 12:25:35.520339 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 277398us
I20260709 12:25:35.518016 31642 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:35.519093 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:35.523142 31791 leader_election.cc:336] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.529155 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:35.529212 31642 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:35.536574 32343 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:35.530789 32375 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 38.
I20260709 12:25:35.541447 31744 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:25:35.541646 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:35.498478 10288 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:35.542884 10335 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.557319 10335 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:35.563355 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 52772.1us
W20260709 12:25:35.564316 31791 leader_election.cc:336] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:35.565251 31791 leader_election.cc:336] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.565639 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:35.567028 31744 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 32 index: 22. Preceding OpId from leader: term: 35 index: 24. (index mismatch)
I20260709 12:25:35.502333 10233 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:35.552627 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:35.553332 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:35.575389 31753 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:25:35.555070 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:35.497956 10511 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.551740 31822 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 35 index: 18. Preceding OpId from leader: term: 36 index: 18. (term mismatch)
I20260709 12:25:35.515219 10464 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:35.578347 31822 pending_rounds.cc:85] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Aborting all ops after (but not including) 17
I20260709 12:25:35.578682 31822 pending_rounds.cc:107] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Aborting uncommitted NO_OP operation due to leader change: 35.18
I20260709 12:25:35.578929 31822 raft_consensus.cc:2889] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:35.578956 10464 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.579140 10509 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:35.579542 10464 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:35.579993 10509 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:35.540647 10471 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:35.514961 10013 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:35.590967 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:35.591993 31820 raft_consensus.cc:2410] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 37 because replica has last-logged OpId of term: 34 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 16.
W20260709 12:25:35.586302 31791 leader_election.cc:336] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.573939 10049 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:35.574167 10235 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:35.515146 10516 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:35.568869 10472 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:35.588644 10213 raft_consensus.cc:3060] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:35.594067 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:35.603506 10493 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 39 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:35.594475 10408 consensus_meta.cc:287] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.881s	user 0.018s	sys 0.019s
W20260709 12:25:35.606556 10234 consensus_meta.cc:287] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.635s	user 0.008s	sys 0.013s
I20260709 12:25:35.623219 10234 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:35.630118 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 52204.5us
I20260709 12:25:35.568509 10532 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:35.574525 10276 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:35.640481 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 88658.7us
W20260709 12:25:35.651747 10433 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.668s	user 0.024s	sys 0.036s
W20260709 12:25:35.668036 10066 consensus_meta.cc:287] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.534s	user 0.010s	sys 0.002s
I20260709 12:25:35.684350 10433 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:35.691497 10509 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:35.640705 10470 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:35.628674 10013 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:35.707376 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:35.692394 31746 raft_consensus.cc:1240] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 33. Current term is 34. Ops: [33.18-33.18]
W20260709 12:25:35.616014 31791 leader_election.cc:336] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.766003 31791 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:35.698975 30503 thread.cc:527] Waited for 2000ms trying to join with heartbeat (tid 31703)
I20260709 12:25:35.709133 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:35.764173 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 122212us
I20260709 12:25:35.626148 10511 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:35.739876 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 162394us
I20260709 12:25:35.771771 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 37 index: 25 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:35.772617 31818 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Advancing to term 39
I20260709 12:25:35.773087 10391 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:35.744889 31753 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 39.
I20260709 12:25:35.776124 10190 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
W20260709 12:25:35.776355 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 160369us
I20260709 12:25:35.776836 10190 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:35.777215 31791 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:35.778224 31791 leader_election.cc:336] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:35.779151 31791 leader_election.cc:336] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.779551 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:35.767642 32375 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 38.
I20260709 12:25:35.795537 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:35.828410 32376 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:25:35.794479 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:35.658799 10276 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:35.722188 10423 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:35.827963 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:35.784145 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:35.784758 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:35.791332 31821 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
W20260709 12:25:35.833123 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48568.9us
I20260709 12:25:35.834888 31819 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
W20260709 12:25:35.684391 10497 consensus_meta.cc:287] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.720s	user 0.027s	sys 0.006s
W20260709 12:25:35.830240 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51107.6us
I20260709 12:25:35.843843 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 32 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:35.850205 32373 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 26.
W20260709 12:25:35.858968 31791 leader_election.cc:336] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.865639 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:35.866221 31791 leader_election.cc:336] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:35.866710 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:35.836581 31818 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 39.
I20260709 12:25:35.758328 10493 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:35.849824 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 81740us
I20260709 12:25:35.686247 10258 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.616042 10522 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:25:35.863576 32342 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 109 similar messages]
I20260709 12:25:35.686031 10213 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:25:35.606706 10474 consensus_meta.cc:287] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.785s	user 0.016s	sys 0.019s
I20260709 12:25:35.835534 10428 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:35.697186 10479 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:35.767735 10335 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:35.608135 10472 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:35.700531 10066 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:35.964075 10479 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.872018 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:35.782912 10464 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:35.621387 10049 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
W20260709 12:25:35.879551 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 383933us
I20260709 12:25:35.860133 10509 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:25:35.780360 10530 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.098s
W20260709 12:25:35.932775 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 45997.1us
W20260709 12:25:35.934216 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 56485.1us
I20260709 12:25:35.888051 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:35.844169 10435 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:35.992303 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 120375us
I20260709 12:25:35.993935 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:35.942020 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 75825.1us
I20260709 12:25:35.943141 10213 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.040s
I20260709 12:25:35.879515 10522 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:35.889535 10420 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:35.962941 10524 log.cc:927] Time spent T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:25:35.963757 10390 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.701s	user 0.008s	sys 0.024s
I20260709 12:25:35.942876 10234 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.032080 10546 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:36.032891 10511 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.032717 10546 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.033483 10511 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.033960 10511 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:35.940439 10532 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:35.871028 10058 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.882289 10497 consensus_queue.cc:1059] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 36.21, Next index: 22, Last known committed idx: 21, Time since last communication: 0.017s
I20260709 12:25:35.963581 10228 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:35.820472 10190 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:35.947034 10433 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:35.968653 10409 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:35.887216 10474 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:35.886106 10258 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:35.859974 10423 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:35.995184 31791 leader_election.cc:336] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.051937 10245 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.077096 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:35.996169 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 108342us
W20260709 12:25:35.991869 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 125607us
W20260709 12:25:35.996685 31717 leader_election.cc:336] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.084474 31717 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:36.085678 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 89007.2us
I20260709 12:25:36.071383 10058 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:35.992630 10479 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.091234 10058 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.092581 10479 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:36.093356 10542 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.067s	user 0.001s	sys 0.000s
W20260709 12:25:36.095260 10531 log.cc:927] Time spent T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.152s	user 0.001s	sys 0.000s
I20260709 12:25:36.057201 10118 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:36.019292 10435 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:36.049686 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 55838.7us
I20260709 12:25:36.101436 10435 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.101577 31642 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:35.993860 10464 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:36.020257 10288 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:35.994673 10509 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.102833 10433 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:36.102761 10464 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.103791 10433 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:36.104334 10468 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:36.104449 10433 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.024215 10522 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:36.020460 10213 consensus_queue.cc:1243] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 5206fd0087404d2da964a5d985a04ff4 log is divergent from this leader: its last log entry 34.23 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 22
I20260709 12:25:36.065532 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
W20260709 12:25:35.877600 10426 consensus_meta.cc:287] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.908s	user 0.014s	sys 0.008s
I20260709 12:25:36.107206 10033 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:25:36.107633 10426 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.109042 31749 raft_consensus.cc:1240] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:36.109983 10235 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:25:36.110558 10235 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 39 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:36.111397 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.111476 10235 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 37.25, Last appended by leader: 25, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.113416 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.105187 10468 consensus_queue.cc:1243] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 33.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:25:36.102876 10509 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:25:36.115801 10468 consensus_queue.cc:1059] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 31.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:36.116703 10468 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:36.117643 10468 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.119129 10118 raft_consensus.cc:3055] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Stepping down as leader of term 33
I20260709 12:25:36.118588 10468 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
W20260709 12:25:36.066689 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 203136us
I20260709 12:25:36.057716 10544 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:36.140514 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.145298 32374 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:25:36.065425 10276 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.028952 10420 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.155812 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.119750 10118 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:36.083564 10545 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:36.171090 10118 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 33, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.172823 10118 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:36.077836 10409 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.181205 10409 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.121272 10423 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:36.184463 10545 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.189144 10511 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.134141 10245 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.193768 10245 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.195292 10225 raft_consensus.cc:3055] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:25:36.195794 10225 raft_consensus.cc:740] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 36 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:36.198084 10545 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.199451 10472 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.118456 10233 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.039s
I20260709 12:25:36.200402 10225 consensus_queue.cc:260] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:36.199563 10539 log.cc:927] Time spent T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.225s	user 0.002s	sys 0.000s
I20260709 12:25:36.203186 10474 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.204712 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.204965 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:36.205551 31818 raft_consensus.cc:2393] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 37: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:36.054780 10532 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.206184 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.206874 31825 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 26.
I20260709 12:25:36.207782 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.207824 10532 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:36.208568 10480 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.838s	user 0.013s	sys 0.003s
I20260709 12:25:36.209414 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:36.210083 31819 raft_consensus.cc:3060] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 32
I20260709 12:25:36.215372 10410 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.216151 10410 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.219641 10013 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.150701 10546 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.204741 10066 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:36.227103 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64911.9us
I20260709 12:25:36.230170 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:36.230914 32371 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:36.198432 10386 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:36.234401 32376 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 39.
I20260709 12:25:36.236877 31819 raft_consensus.cc:2468] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:36.239668 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.255652 10386 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:36.254621 10342 thread.cc:641] prepare [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.013s	sys 0.014s
I20260709 12:25:36.239063 10511 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.279778 31643 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:36.293145 31752 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 32 index: 16. Preceding OpId from leader: term: 36 index: 18. (index mismatch)
W20260709 12:25:36.281678 10342 thread.cc:608] prepare [worker] (thread pool) Time spent starting thread: real 0.560s	user 0.013s	sys 0.015s
I20260709 12:25:36.276265 10066 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.300319 32371 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:36.301900 31643 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:36.303560 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:36.304229 31752 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 29.
I20260709 12:25:36.291466 10245 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 39 FOLLOWER]: Leader election won for term 39
I20260709 12:25:36.319101 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:36.319901 31752 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:25:36.277016 10546 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:36.300640 10403 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.292419 10511 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.355572 10403 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.357947 10511 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.359273 10509 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.308583 10408 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:36.360700 10408 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:36.361653 10509 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 39 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.302585 10066 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:36.403874 10556 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.098s	user 0.000s	sys 0.001s
I20260709 12:25:36.404048 10386 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.348809 10245 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 39 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:36.421427 10470 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.432323 10408 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 27.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.392067 10403 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.439955 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.440814 32373 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 37.
I20260709 12:25:36.441851 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.443277 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:36.445595 31819 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:36.447448 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.450261 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 32 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.449396 31718 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:36.451380 32373 raft_consensus.cc:2410] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 36 because replica has last-logged OpId of term: 34 index: 23, which is greater than that of the candidate, which has last-logged OpId of term: 32 index: 22.
I20260709 12:25:36.453754 10245 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 39, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.458495 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.462817 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:36.466017 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.452190 10288 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:36.470696 10414 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.013s
I20260709 12:25:36.472491 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.473461 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:36.473512 31825 raft_consensus.cc:2393] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 27: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:25:36.499161 31644 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.417806 10522 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.517926 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:36.519234 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.364460 10546 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.484546 10562 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:36.520190 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 39 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:36.533725 32371 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 39: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
I20260709 12:25:36.458119 10511 raft_consensus.cc:2804] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:36.525125 31819 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:36.526594 10509 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:36.535049 10509 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.535482 10509 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:36.532536 10546 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.533406 10562 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.534531 10511 raft_consensus.cc:697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:36.539606 10066 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.458823 10470 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election won for term 27
I20260709 12:25:36.540400 10470 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.540773 10470 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:36.543026 10420 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:36.543495 10562 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.544656 10511 consensus_queue.cc:237] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 30.16, Last appended by leader: 16, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.548815 10468 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.523600 10118 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.550930 10118 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.552505 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:36.553053 10118 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.553741 10562 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:36.554261 10562 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.554080 10509 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.554867 10562 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:36.555658 10522 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.557440 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.559821 10509 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.560642 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:36.561408 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.562184 10522 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.562698 10522 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.563499 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:36.564242 31825 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 38: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:36.565089 10522 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.567454 10288 consensus_queue.cc:1059] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 29.16, Next index: 17, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:36.569096 10342 raft_consensus.cc:3055] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:36.569585 10342 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 29 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:36.570559 10342 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.571174 32371 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 38.
I20260709 12:25:36.572151 10342 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:36.573283 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:36.574111 31750 raft_consensus.cc:2393] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 39: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:36.573900 10468 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.565138 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.577878 31752 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:25:36.579469 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:36.581842 10386 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:36.582170 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:36.583343 31747 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:25:36.583415 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:36.584072 31744 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:36.584468 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:36.584885 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:36.585246 31747 raft_consensus.cc:2410] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 30 because replica has last-logged OpId of term: 29 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 16.
I20260709 12:25:36.585440 31744 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:36.586037 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:36.586259 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:36.582365 10386 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:36.588330 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:36.594219 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:36.589421 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:36.600618 31819 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:36.602144 31747 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
W20260709 12:25:36.602921 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 140291us
I20260709 12:25:36.599196 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:36.599730 31744 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:36.609051 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:36.610800 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:36.600611 31751 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 26.
I20260709 12:25:36.630703 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.632287 31825 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 28.
I20260709 12:25:36.636435 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:36.641863 31819 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
I20260709 12:25:36.604430 10386 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:36.629757 10288 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:36.654026 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:36.629141 10213 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:36.633880 10066 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:36.592087 10468 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.055s
I20260709 12:25:36.667801 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.670655 32371 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:36.661384 10213 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.666343 10410 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.671741 10213 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:36.676465 32373 pending_rounds.cc:85] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Aborting all ops after (but not including) 22
I20260709 12:25:36.677034 32373 pending_rounds.cc:107] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Aborting uncommitted NO_OP operation due to leader change: 34.23
I20260709 12:25:36.677340 32373 raft_consensus.cc:2889] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:25:36.653103 10276 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:36.624455 10245 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.046s
I20260709 12:25:36.657666 10288 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.691378 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:36.692277 31744 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:36.689194 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.693744 10118 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.694300 10118 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.669409 10433 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:36.697871 32374 raft_consensus.cc:2410] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 32 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260709 12:25:36.686647 10276 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.697357 10288 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:36.722386 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:36.714150 10276 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:36.730437 10470 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.733691 10470 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.735387 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:36.736119 32374 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:36.742976 10493 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.743827 10493 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.745502 32374 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:25:36.748296 10493 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.749127 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.750787 10470 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:36.751557 10470 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.755470 10470 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.756031 10118 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.757156 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.760170 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.761147 32374 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 37: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
W20260709 12:25:36.767181 30503 thread.cc:527] Waited for 3000ms trying to join with heartbeat (tid 31703)
I20260709 12:25:36.767405 31744 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
I20260709 12:25:36.769263 31744 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 34. Current term is 35. Ops: [32.19-34.20]
I20260709 12:25:36.770798 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:36.793394 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:36.801988 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:36.803547 31642 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:36.804467 10118 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election won for term 28
W20260709 12:25:36.804674 10571 log.cc:927] Time spent T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.078s	user 0.000s	sys 0.000s
W20260709 12:25:36.804930 10548 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.746s	user 0.002s	sys 0.004s
I20260709 12:25:36.805110 10468 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:36.805791 10468 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:36.807224 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4430) took 1035 ms. Trace:
I20260709 12:25:36.807874 10493 consensus_queue.cc:1059] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:36.808847 10493 consensus_queue.cc:1243] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer 039d66a6c85e43c3936bf5017501b344 log is divergent from this leader: its last log entry 33.19 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 18
I20260709 12:25:36.809388 10234 raft_consensus.cc:3055] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:36.809794 10234 raft_consensus.cc:740] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:36.806893 10468 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 29.14, Last appended by leader: 14, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.810804 10234 consensus_queue.cc:260] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.20, Last appended by leader: 20, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:36.807803 31826 rpcz_store.cc:276] 0709 12:25:35.771969 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.772122 (+   153us) service_pool.cc:224] Handling call
0709 12:25:35.772449 (+   327us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:35.774127 (+  1678us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:35.774158 (+    31us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:35.774752 (+   594us) log.cc:844] Serialized 26 byte log entry
0709 12:25:35.945245 (+170493us) raft_consensus.cc:1645] Marking committed up to 18
0709 12:25:35.945296 (+    51us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:35.945370 (+    74us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:36.804305 (+858935us) spinlock_profiling.cc:243] Waited 5.02 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:36.806773 (+  2468us) raft_consensus.cc:1685] finished
0709 12:25:36.806809 (+    36us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.807137 (+   328us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":1231,"raft.queue_time_us":2856,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":120,"spinlock_wait_cycles":11548416,"thread_start_us":166511,"threads_started":1,"wal-append.queue_time_us":210240}
I20260709 12:25:36.804936 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:36.818166 31744 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 36 index: 24. Preceding OpId from leader: term: 39 index: 26. (index mismatch)
I20260709 12:25:36.822780 10234 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:36.809818 10464 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.823830 10258 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:36.804961 10118 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:36.817562 10423 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935)
I20260709 12:25:36.833542 10118 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:36.827181 10066 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:36.855417 31747 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:36.842648 10497 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.025s
I20260709 12:25:36.868212 10493 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.889902 10493 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:36.938766 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 616951us
I20260709 12:25:36.951088 31823 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4436) took 1158 ms. Trace:
W20260709 12:25:36.951156 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 866479us
W20260709 12:25:36.952445 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 957198us
W20260709 12:25:36.953132 31791 leader_election.cc:336] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 26 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:36.953737 31562 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4d48c83c761f48079bc6ea176649dda0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 26 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:36.954170 31791 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.968786 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.959165 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:36.982470 31791 leader_election.cc:336] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:36.954783 31792 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:36.983176 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:36.984346 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:36.986941 31821 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 16. Preceding OpId from leader: term: 31 index: 18. (index mismatch)
I20260709 12:25:36.962698 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:36.989377 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:36.991148 31717 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.991597 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:36.992587 31717 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:36.993789 31717 leader_election.cc:336] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:36.994287 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.995296 31819 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:36.996467 31824 raft_consensus.cc:2393] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:36.996879 31826 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:25:36.995208 31718 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:36.961694 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.951632 31823 rpcz_store.cc:276] 0709 12:25:35.793206 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:35.793410 (+   204us) service_pool.cc:224] Handling call
0709 12:25:35.793846 (+   436us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:35.794141 (+   295us) raft_consensus.cc:1534] Early marking committed up to index 19
0709 12:25:35.875613 (+ 81472us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.027760 (+152147us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.069936 (+ 42176us) log.cc:844] Serialized 14 byte log entry
0709 12:25:36.419220 (+349284us) spinlock_profiling.cc:243] Waited 7.16 ms on lock 0x7b18004341a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9ce8557 00007fe8f9cea7c1 00007fe8f9cea779 00007fe8f9cea709 00007fe8f9cea6d1 00007fe8f9ce99cd 0000000000388634 0000000000388479 00007fe8f9ccb83a
0709 12:25:36.950502 (+531282us) spinlock_profiling.cc:243] Waited 905 us on lock 0x7b18004341a0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902bb0ab1 00007fe902bbd281 00007fe8f9c62c0a 00007fe9033a2219 00007fe8f9ceaa40 00007fe8f9ce54e4 00007fe8f9ce5455 00007fe8fcdcd060 00007fe8fcddf716 00007fe8fcdf0287 00007fe8fcdf0208
0709 12:25:36.950708 (+   206us) raft_consensus.cc:1539] Triggering prepare for 0 ops
0709 12:25:36.950762 (+    54us) raft_consensus.cc:1645] Marking committed up to 19
0709 12:25:36.950808 (+    46us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:36.950886 (+    78us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:36.951021 (+   135us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":966389,"prepare.queue_time_us":835936,"prepare.run_cpu_time_us":100,"prepare.run_wall_time_us":103,"raft.queue_time_us":978221,"raft.run_cpu_time_us":389,"raft.run_wall_time_us":27836,"spinlock_wait_cycles":18551936,"thread_start_us":144248,"threads_started":1}
I20260709 12:25:37.006494 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:37.006806 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:37.009611 31819 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 26 FOLLOWER]: Advancing to term 27
I20260709 12:25:36.955008 10480 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:37.007732 31818 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 37.
I20260709 12:25:36.956741 10049 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.994321 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:37.015544 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:36.956940 10391 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.957410 10552 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:36.957136 10335 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:36.977254 10426 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:37.000007 10477 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:37.039471 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:37.024950 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:37.006016 10433 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election won for term 30
I20260709 12:25:37.033226 31821 raft_consensus.cc:1240] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 34. Current term is 35. Ops: [32.19-34.20]
I20260709 12:25:37.016574 10228 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:37.024025 10049 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:36.957036 10562 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:37.035378 10578 log.cc:927] Time spent T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.141s	user 0.000s	sys 0.001s
I20260709 12:25:37.036090 31823 raft_consensus.cc:1275] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 16. Preceding OpId from leader: term: 32 index: 17. (index mismatch)
I20260709 12:25:37.013912 10470 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:37.014441 10234 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.089754 10234 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:37.092334 10234 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:37.036986 31717 consensus_peers.cc:597] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:37.093266 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56317.5us
W20260709 12:25:37.093955 31717 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.093988 10049 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:37.094377 31717 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:37.044092 10552 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:37.042332 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 14 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
W20260709 12:25:37.039664 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 57223.3us
I20260709 12:25:37.045290 31827 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:37.047183 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:37.096547 10552 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:37.097549 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:37.051357 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:37.056268 31819 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 27.
I20260709 12:25:37.039000 10258 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:37.038687 10480 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:37.099859 10480 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:37.102417 10480 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:37.115485 10585 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.098s	user 0.000s	sys 0.000s
W20260709 12:25:37.118412 10586 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.083s	user 0.000s	sys 0.000s
I20260709 12:25:37.122476 10562 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:37.123615 31752 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 33: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:37.130096 10532 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.130860 10532 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:37.046705 10335 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:37.136157 10335 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:37.137123 31824 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 37.
I20260709 12:25:37.041153 10391 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:37.140542 10391 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:37.141403 31820 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
W20260709 12:25:37.085750 10580 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.190s	user 0.001s	sys 0.000s
I20260709 12:25:37.035923 10118 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:37.157249 10470 raft_consensus.cc:697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 35 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:37.158754 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 64835.9us
W20260709 12:25:37.159559 31717 leader_election.cc:336] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.159968 31717 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:37.048044 10433 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.050971 10465 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.167572 32373 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 39 index: 22. (index mismatch)
I20260709 12:25:37.073341 10464 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:37.163563 10258 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:25:37.172807 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 125696us
I20260709 12:25:37.175055 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:37.176007 31643 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.176038 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:37.176908 31821 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 27.
I20260709 12:25:37.179682 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:37.182686 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:37.188737 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:37.189563 31827 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:37.191861 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:37.170068 10433 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:37.197924 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:37.181143 10426 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:37.181494 10511 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:37.198426 10423 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:37.164572 10276 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:37.177740 10465 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:37.203822 10258 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:25:37.203594 10426 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:37.150058 10288 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:37.184418 10496 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.029s
I20260709 12:25:37.224179 31822 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 39 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 37 index: 25. Preceding OpId from leader: term: 39 index: 26. (index mismatch)
I20260709 12:25:37.158325 10470 consensus_queue.cc:237] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.19, Last appended by leader: 19, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:37.234023 10587 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.062s	user 0.000s	sys 0.001s
I20260709 12:25:37.232959 31827 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:37.212941 10493 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:37.249202 10258 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 38 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:37.236482 10426 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:37.279296 10408 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.280115 10408 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:37.288959 10386 consensus_meta.cc:287] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Time spent flushing consensus metadata: real 0.623s	user 0.013s	sys 0.001s
W20260709 12:25:37.320573 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 245174us
W20260709 12:25:37.322805 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.336043 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:37.339843 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:37.343353 31750 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 39 index: 22. (term mismatch)
I20260709 12:25:37.343991 31750 pending_rounds.cc:85] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Aborting all ops after (but not including) 21
I20260709 12:25:37.344291 31750 pending_rounds.cc:107] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: Aborting uncommitted NO_OP operation due to leader change: 37.22
I20260709 12:25:37.344620 31750 raft_consensus.cc:2889] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
W20260709 12:25:37.322760 10213 consensus_meta.cc:287] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.650s	user 0.007s	sys 0.007s
I20260709 12:25:37.348907 31747 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:37.354806 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:37.355793 31745 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 26 FOLLOWER]: Advancing to term 28
I20260709 12:25:37.363873 10276 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:37.371003 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:37.372109 31744 raft_consensus.cc:2410] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 34 because replica has last-logged OpId of term: 33 index: 19, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 17.
I20260709 12:25:37.382269 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:37.493891 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 860842us
W20260709 12:25:37.501363 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 116884us
I20260709 12:25:37.326004 10386 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:37.516453 10408 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:37.515293 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:37.517728 10589 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.268s	user 0.001s	sys 0.000s
I20260709 12:25:37.490839 10258 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 37.24, Last appended by leader: 24, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:37.520265 10225 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:37.362241 10213 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:37.524628 10583 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.130s	user 0.001s	sys 0.000s
I20260709 12:25:37.527021 10386 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:37.542717 10403 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.028s
W20260709 12:25:37.513841 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 402276us
W20260709 12:25:37.570529 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 56562.8us
W20260709 12:25:37.573073 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57868.1us
W20260709 12:25:37.581806 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 422244us
I20260709 12:25:37.582396 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:37.587716 31745 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:37.595842 10288 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:37.596670 10288 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:37.597577 31752 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:45178 (request call id 4447) took 1018 ms. Trace:
I20260709 12:25:37.599798 10594 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:37.602335 10213 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:37.590088 10408 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:37.598129 31752 rpcz_store.cc:276] 0709 12:25:36.579113 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.579217 (+   104us) service_pool.cc:224] Handling call
0709 12:25:37.597291 (+1018074us) spinlock_profiling.cc:243] Waited 543 ms on lock 0x7b54001a59d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:37.597489 (+   198us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1249221888}
I20260709 12:25:37.607018 10433 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:37.610097 10595 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:37.610430 10433 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:37.610857 10595 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:37.598595 10288 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.612071 10288 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:37.613237 10595 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:37.614939 10288 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:37.620952 10465 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:37.625823 31717 leader_election.cc:336] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:37.671761 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 348228us
I20260709 12:25:37.662369 10535 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.675932 10433 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:37.687479 10433 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:37.688760 10579 log.cc:927] Time spent T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.793s	user 0.005s	sys 0.000s
I20260709 12:25:37.691316 10433 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:37.691807 10532 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:37.693295 10433 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.693924 10433 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:37.695094 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:37.695907 10593 log.cc:927] Time spent T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.179s	user 0.001s	sys 0.000s
I20260709 12:25:37.699513 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:37.712610 32373 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 34
W20260709 12:25:37.700068 31791 connection.cc:256] RPC call timeout handler was delayed by -1.2779702510001698s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2014s for a response
W20260709 12:25:37.714382 31791 connection.cc:256] RPC call timeout handler was delayed by -1.0242017370001122s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19510000000000002s for a response
I20260709 12:25:37.698665 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:37.713567 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:37.719854 31752 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:37.723412 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 35 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:37.726046 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:37.707607 31717 consensus_peers.cc:597] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:37.724411 31749 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 36.
I20260709 12:25:37.715200 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:37.685693 10562 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:37.728839 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:37.731094 31717 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.733821 31748 raft_consensus.cc:1275] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 32 index: 20. Preceding OpId from leader: term: 35 index: 22. (index mismatch)
W20260709 12:25:37.733951 31717 leader_election.cc:336] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.734556 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:37.738351 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:37.740404 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:37.741192 31744 raft_consensus.cc:2368] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30.
W20260709 12:25:37.746301 31717 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.747624 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:37.748303 31749 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 36.
I20260709 12:25:37.749240 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:37.749461 31717 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:37.757009 31752 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:37.758585 31752 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 14. Preceding OpId from leader: term: 32 index: 15. (index mismatch)
I20260709 12:25:37.765061 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:37.765718 31748 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:37.767815 30503 thread.cc:527] Waited for 4000ms trying to join with heartbeat (tid 31703)
I20260709 12:25:37.697019 10535 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:37.732894 10591 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:37.776113 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 27 candidate_status { last_received { term: 25 index: 14 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:37.789651 31744 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:25:37.774391 31717 leader_election.cc:336] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:37.791484 31717 leader_election.cc:336] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:37.792513 31717 leader_election.cc:336] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.792896 31717 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:37.793902 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:37.789923 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:37.797435 31749 raft_consensus.cc:2410] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 37 because replica has last-logged OpId of term: 35 index: 24, which is greater than that of the candidate, which has last-logged OpId of term: 34 index: 23.
I20260709 12:25:37.790227 31642 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:37.799116 10058 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:37.799633 10058 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:37.800495 10058 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 26.17, Last appended by leader: 17, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:37.794854 10591 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:37.811308 10591 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:37.740854 10562 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:37.824194 31642 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:37.825707 10562 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:37.766575 10465 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.060s
I20260709 12:25:37.764606 10607 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.081s
I20260709 12:25:37.746989 10511 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:37.830538 31745 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 32: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:37.846161 32373 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
W20260709 12:25:37.850782 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 262899us
I20260709 12:25:37.859501 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:37.862849 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:37.778877 10598 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:25:37.884011 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 91474.4us
W20260709 12:25:37.886207 31717 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.888131 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:37.889691 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:37.892135 31750 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:37.892513 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:37.831635 10423 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election lost for term 34. Reason: could not achieve majority
W20260709 12:25:37.864161 10548 log.cc:927] Time spent T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.088s	user 0.000s	sys 0.000s
I20260709 12:25:37.907063 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:37.908595 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:37.913055 31747 raft_consensus.cc:2393] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 30: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:37.909603 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:37.917845 31717 leader_election.cc:336] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.918408 31717 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:37.864625 10511 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:37.925837 31643 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:37.926033 31717 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.926584 31717 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:37.867869 10598 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:37.867591 10465 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader election lost for term 32. Reason: could not achieve majority
I20260709 12:25:37.866982 10410 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:37.932859 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 203497us
W20260709 12:25:37.933626 31717 leader_election.cc:336] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:37.934237 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:37.935127 32377 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 38.
I20260709 12:25:37.938894 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:37.939747 31744 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:37.929881 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:37.949779 31750 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:37.953806 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:37.954958 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:37.909422 10464 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:37.911729 10606 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:37.917645 10263 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:37.988389 10263 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:37.928962 10276 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:37.933076 10562 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:37.928896 10245 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:37.933772 10598 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:37.967139 32377 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 37.
I20260709 12:25:37.944947 10423 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:37.980206 10617 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:37.980244 10603 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:37.980587 10596 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.049s
W20260709 12:25:37.983408 10610 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.152s	user 0.000s	sys 0.000s
I20260709 12:25:37.989688 31826 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:37.999300 10433 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:37.977820 10511 raft_consensus.cc:493] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:38.000685 10535 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:38.004055 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48551.2us
I20260709 12:25:37.975955 10213 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:38.011830 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 32 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:38.012076 31643 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:38.032558 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:37.975811 10599 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:38.022827 31642 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:38.028129 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:38.036015 31791 leader_election.cc:400] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30.
I20260709 12:25:38.036448 31791 leader_election.cc:403] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:38.027143 10511 raft_consensus.cc:515] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:38.040825 31791 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:38.025058 10603 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:38.049176 10118 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:38.049273 10591 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:38.018707 10276 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.025s
I20260709 12:25:38.058418 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:38.057966 10058 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader election won for term 33
I20260709 12:25:38.042834 10599 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:38.065647 10426 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.788s	user 0.010s	sys 0.000s
W20260709 12:25:38.064215 10600 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.546s	user 0.004s	sys 0.000s
I20260709 12:25:38.028784 10617 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:38.079461 10426 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:38.080116 10058 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:38.081473 10617 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:38.081270 10058 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 33, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:38.081976 10426 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:38.059127 10213 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:38.086404 10622 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:38.057403 10603 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:38.092518 10263 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:38.066205 10511 leader_election.cc:290] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:38.059437 10468 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:38.096371 31748 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:38.104275 10622 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:38.117161 31642 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:38.121553 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:38.066383 10591 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:38.121618 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 30 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:38.098863 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:38.102855 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44547.4us
I20260709 12:25:37.996456 31718 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:38.113072 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:38.107616 10118 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.109675 10612 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:38.097884 31826 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 32.
W20260709 12:25:38.113423 31717 leader_election.cc:336] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.170372 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:38.135561 10599 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:38.352178 31751 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.522s	user 0.023s	sys 0.056s
W20260709 12:25:38.353503 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 226436us
I20260709 12:25:38.108455 10617 consensus_queue.cc:1048] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.027s
I20260709 12:25:38.185438 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:38.188117 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:38.189904 10598 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:38.192448 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 196148us
I20260709 12:25:38.193037 10511 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Leader election won for term 27
I20260709 12:25:38.198035 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:38.203788 31824 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 4200) took 1188 ms (client timeout 1512 ms). Trace:
I20260709 12:25:38.123096 10622 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:25:38.390781 31824 rpcz_store.cc:269] 0709 12:25:37.015110 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.015284 (+   174us) service_pool.cc:224] Handling call
0709 12:25:38.199319 (+1184035us) spinlock_profiling.cc:243] Waited 106 ms on lock 0x7b54003ca0d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:38.203680 (+  4361us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":243151744}
I20260709 12:25:38.230281 10626 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:25:38.235311 31825 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34642 (request call id 2034) took 1277 ms. Trace:
I20260709 12:25:38.278231 10118 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:38.163985 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 181968us
I20260709 12:25:38.401767 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
I20260709 12:25:38.396317 31825 rpcz_store.cc:276] 0709 12:25:36.957620 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:36.960412 (+  2792us) service_pool.cc:224] Handling call
0709 12:25:36.975944 (+ 15532us) raft_consensus.cc:1471] Updating replica for 1 ops
0709 12:25:36.976370 (+   426us) raft_consensus.cc:1534] Early marking committed up to index 17
0709 12:25:37.210274 (+233904us) log.cc:844] Serialized 14 byte log entry
0709 12:25:37.316278 (+106004us) raft_consensus.cc:1539] Triggering prepare for 1 ops
0709 12:25:37.316762 (+   484us) log.cc:844] Serialized 26 byte log entry
0709 12:25:37.316894 (+   132us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:37.316925 (+    31us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:37.317002 (+    77us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:38.063431 (+746429us) spinlock_profiling.cc:243] Waited 32.3 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:38.234869 (+171438us) raft_consensus.cc:1685] finished
0709 12:25:38.235025 (+   156us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:38.235227 (+   202us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":30584,"prepare.queue_time_us":311402,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":1927,"raft.queue_time_us":7075,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":19134,"spinlock_wait_cycles":74239872,"thread_start_us":339041,"threads_started":2,"wal-append.queue_time_us":307051}
I20260709 12:25:38.404393 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:38.340277 31717 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:38.352064 31818 raft_consensus.cc:2393] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 33: Already voted for candidate b95aeeaf4c51442b8092cff1de674f2b in this term.
W20260709 12:25:38.353809 31751 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.523s	user 0.023s	sys 0.056s
I20260709 12:25:38.363345 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41210 (request call id 4450) took 1323 ms. Trace:
I20260709 12:25:38.405596 31826 rpcz_store.cc:276] 0709 12:25:37.040339 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.042043 (+  1704us) service_pool.cc:224] Handling call
0709 12:25:38.363165 (+1321122us) spinlock_profiling.cc:243] Waited 894 ms on lock 0x7b5400166cd4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:38.363280 (+   115us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2055360896}
W20260709 12:25:38.363474 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 360847us
I20260709 12:25:38.406822 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:38.407455 31822 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:38.390813 31823 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:38.394433 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:38.409283 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:38.411036 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:38.407435 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:38.229372 10591 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 30
W20260709 12:25:38.416368 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 302962us
I20260709 12:25:38.417273 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:38.418108 31821 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:38.421965 32371 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 27 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:38.424422 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 26 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:38.425272 31825 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
W20260709 12:25:38.417586 31717 leader_election.cc:336] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:38.427114 31717 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 39 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.352267 10468 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:38.438591 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 27 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:38.438912 31826 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 14. Preceding OpId from leader: term: 32 index: 15. (index mismatch)
I20260709 12:25:38.439637 32374 raft_consensus.cc:2393] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 27: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:38.427609 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:38.322000 10603 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:38.438671 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:38.390772 10532 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:38.395964 10626 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:38.457875 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 417093us
I20260709 12:25:38.415868 10511 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:38.480824 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 85107.3us
W20260709 12:25:38.490618 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 63536.4us
I20260709 12:25:38.491369 32370 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
W20260709 12:25:38.492077 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 48962.8us
I20260709 12:25:38.493053 31718 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:38.491496 31717 leader_election.cc:336] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.494240 31717 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:38.496665 31822 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:38.446177 10258 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:25:38.507555 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 69617.5us
I20260709 12:25:38.506561 31825 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 28.
I20260709 12:25:38.441696 10621 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:38.535483 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:38.488587 10288 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.476835 10616 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader pre-election won for term 39
I20260709 12:25:38.545303 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:38.441460 10423 consensus_queue.cc:1048] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.068s
I20260709 12:25:38.499780 10496 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.548249 31748 raft_consensus.cc:2393] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 34: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:38.498342 10276 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.502909 10626 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:38.498136 10562 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:25:38.498060 10213 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.498342 10596 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:38.583590 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 90592.8us
I20260709 12:25:38.527781 10620 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:38.546783 10599 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:38.548861 10532 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:38.548938 10616 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:38.668864 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 175534us
I20260709 12:25:38.579282 10477 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:38.665145 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 122567us
W20260709 12:25:38.672988 31576 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42070 (request call id 229) took 33154 ms (client timeout 15000 ms). Trace:
W20260709 12:25:38.676726 31791 leader_election.cc:336] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.664029 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:25:38.677982 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 186505us
I20260709 12:25:38.556722 10288 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:38.697413 31717 leader_election.cc:336] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:38.698580 31717 leader_election.cc:336] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:38.699541 31717 leader_election.cc:336] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:38.701128 31717 leader_election.cc:336] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.701661 31717 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:38.665899 10620 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:25:38.590162 10496 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:38.658811 10276 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:38.673795 31576 rpcz_store.cc:269] 0709 12:25:05.511750 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:20.479436 (+14967686us) service_pool.cc:224] Handling call
0709 12:25:38.666507 (+18187071us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:33.026745 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:33.085934 (+ 59189us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:33.085999 (+    65us) write_op.cc:435] Acquired schema lock
0709 12:25:33.086025 (+    26us) tablet.cc:662] Decoding operations
0709 12:25:33.305257 (+219232us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:33.305372 (+   115us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:33.305404 (+    32us) tablet.cc:685] Acquiring locks for 25 operations
0709 12:25:33.406027 (+100623us) tablet.cc:701] Row locks acquired
0709 12:25:33.406099 (+    72us) write_op.cc:260] PREPARE: finished
0709 12:25:33.406308 (+   209us) write_op.cc:270] Start()
0709 12:25:33.406480 (+   172us) write_op.cc:276] Timestamp: P: 1783599933406238 usec, L: 0
0709 12:25:33.406509 (+    29us) op_driver.cc:348] REPLICATION: starting
0709 12:25:33.407370 (+   861us) log.cc:844] Serialized 9886 byte log entry
0709 12:25:34.234784 (+827414us) op_driver.cc:464] REPLICATION: finished
0709 12:25:34.477142 (+242358us) write_op.cc:301] APPLY: starting
0709 12:25:34.488297 (+ 11155us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:35.606747 (+1118450us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:35.606879 (+   132us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:36.575337 (+968458us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:36.575851 (+   514us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=50,key_file_lookups=50,delta_file_lookups=0,mrs_lookups=0
0709 12:25:36.575928 (+    77us) write_op.cc:312] APPLY: finished
0709 12:25:36.630993 (+ 55065us) log.cc:844] Serialized 220 byte log entry
0709 12:25:36.730398 (+ 99405us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:36.730676 (+   278us) write_op.cc:454] Released schema lock
0709 12:25:36.730746 (+    70us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:36.731200 (+   454us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:36.731234 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":257505,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":218289,"apply.run_cpu_time_us":64776,"apply.run_wall_time_us":2264003,"cfile_cache_hit":102,"cfile_cache_hit_bytes":108577,"num_ops":25,"prepare.queue_time_us":240805,"prepare.run_cpu_time_us":20015,"prepare.run_wall_time_us":564966,"raft.queue_time_us":222850,"raft.run_cpu_time_us":9732,"raft.run_wall_time_us":175470,"replication_time_us":796954,"thread_start_us":588849,"threads_started":5,"wal-append.queue_time_us":322703,"wal-append.run_cpu_time_us":35328,"wal-append.run_wall_time_us":2593878}]]}
I20260709 12:25:38.722179 10619 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:38.722983 10619 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:38.725934 10619 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:38.727698 10628 log.cc:927] Time spent T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.068s	user 0.000s	sys 0.000s
W20260709 12:25:38.730204 10636 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
W20260709 12:25:38.731820 10630 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.176s	user 0.000s	sys 0.001s
I20260709 12:25:38.732596 10496 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:38.732787 10641 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.071s	user 0.000s	sys 0.000s
I20260709 12:25:38.710886 10562 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:38.706295 10599 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:38.734387 10599 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:38.734763 10634 log.cc:927] Time spent T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.054s	user 0.000s	sys 0.001s
I20260709 12:25:38.735059 10276 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:38.713804 10627 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:38.743378 10627 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:38.744383 10288 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:38.676007 10616 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:25:38.731839 10633 log.cc:927] Time spent T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.223s	user 0.001s	sys 0.000s
I20260709 12:25:38.747145 10645 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.747802 10645 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:38.734835 10599 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:38.679824 10058 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.746272 10627 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:38.680058 10033 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:38.718377 31791 leader_election.cc:336] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.757368 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:38.750353 10645 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:38.761462 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 47827.3us
I20260709 12:25:38.766660 10532 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 32.24, Last appended by leader: 24, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:38.768390 30503 thread.cc:527] Waited for 5000ms trying to join with heartbeat (tid 31703)
I20260709 12:25:38.771289 31751 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57176 (request call id 1623) took 1078 ms. Trace:
I20260709 12:25:38.771852 31751 rpcz_store.cc:276] 0709 12:25:37.693109 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:37.696735 (+  3626us) service_pool.cc:224] Handling call
0709 12:25:37.731293 (+ 34558us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:37.743131 (+ 11838us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:37.743176 (+    45us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:37.755266 (+ 12090us) log.cc:844] Serialized 52 byte log entry
0709 12:25:38.421870 (+666604us) raft_consensus.cc:1645] Marking committed up to 17
0709 12:25:38.490049 (+ 68179us) log.cc:844] Serialized 14 byte log entry
0709 12:25:38.493025 (+  2976us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:38.493111 (+    86us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:38.730913 (+237802us) raft_consensus.cc:1685] finished
0709 12:25:38.730937 (+    24us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:38.771036 (+ 40099us) spinlock_profiling.cc:243] Waited 3.36 ms on lock 0x7b54002f71d0. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd48ad 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:38.771214 (+   178us) inbound_call.cc:177] Queueing success response
Metrics: {"prepare.queue_time_us":67993,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":51,"raft.queue_time_us":18048,"raft.run_cpu_time_us":290,"raft.run_wall_time_us":24242,"spinlock_wait_cycles":7737216,"thread_start_us":574857,"threads_started":1,"wal-append.queue_time_us":668902}
W20260709 12:25:38.773679 10638 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.055s	user 0.000s	sys 0.000s
I20260709 12:25:38.659422 10468 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:38.778537 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 355382us
W20260709 12:25:38.779903 10639 log.cc:927] Time spent T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
I20260709 12:25:38.668257 10213 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:38.712577 10620 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:38.669227 10596 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:38.684173 10477 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:38.820675 10591 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 29. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30.
W20260709 12:25:38.830298 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 64874.9us
I20260709 12:25:38.831635 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:38.672425 10511 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 25.14, Last appended by leader: 14, Current term: 27, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:38.840121 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:38.805313 10288 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.761533 10058 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:38.863346 32342 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.863904 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:38.854080 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:38.865376 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:38.854497 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:38.869555 31748 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 37.
I20260709 12:25:38.783066 10033 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:38.828600 10603 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:38.844069 10595 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.874519 10663 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:38.855765 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:38.875882 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:38.828198 10118 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:38.875331 10225 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:38.904400 10225 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:38.881630 10629 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:38.882248 10596 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:38.882344 10652 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:25:38.882616 10647 log.cc:927] Time spent T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.101s	user 0.001s	sys 0.000s
I20260709 12:25:38.847955 10423 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:38.885478 31744 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
I20260709 12:25:38.885504 31751 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:38.892094 10058 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:38.893956 10033 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:38.862999 10288 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:38.853229 31820 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
W20260709 12:25:38.916800 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 51516.4us
I20260709 12:25:38.922044 10598 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:38.935204 10616 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 39 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:38.936352 10656 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.072s	user 0.001s	sys 0.000s
I20260709 12:25:38.937896 10616 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 39 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:38.847389 10645 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.083s
I20260709 12:25:38.938994 10225 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:38.916061 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 197828us
I20260709 12:25:38.906263 10477 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:38.921725 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:38.942436 31827 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:25:38.943277 31791 leader_election.cc:336] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.944103 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:38.952199 10288 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:38.930613 10423 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:38.955089 10423 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:38.957448 31791 leader_election.cc:336] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.958055 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:38.920362 10066 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 26 FOLLOWER]: Advancing to term 27
W20260709 12:25:38.979136 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 392962us
I20260709 12:25:38.979449 31827 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 38.
I20260709 12:25:38.985174 32343 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:38.986176 31791 leader_election.cc:336] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.986716 31791 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:38.992167 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:38.994611 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:38.997912 31791 leader_election.cc:336] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:38.950165 10626 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.012410 10626 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:38.917785 10595 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:39.003554 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 62520.5us
I20260709 12:25:38.994482 10263 raft_consensus.cc:2804] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:25:39.016258 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:39.017143 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:39.017590 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:39.017951 31822 raft_consensus.cc:2368] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 37. Current term is 38.
I20260709 12:25:39.019172 31823 raft_consensus.cc:1240] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 39 FOLLOWER]: Rejecting Update request from peer 039d66a6c85e43c3936bf5017501b344 for earlier term 38. Current term is 39. Ops: []
W20260709 12:25:39.019351 31791 leader_election.cc:336] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.019855 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.019968 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:39.020870 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:39.022177 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:39.028441 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:38.979918 10477 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:39.034045 32376 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 38 index: 25. (index mismatch)
I20260709 12:25:38.994452 10616 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:39.040126 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:39.015573 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:39.043097 31642 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:39.045792 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1e27b968c91a4906aec537421f481269" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:38.943245 31825 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:25:39.047134 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:39.055575 31818 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
I20260709 12:25:39.057991 31819 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 36.
I20260709 12:25:39.058966 31642 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:39.014809 10591 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:39.061558 31791 leader_election.cc:336] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.062091 31791 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.063838 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:39.070827 31818 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 24 index: 16. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:25:39.072000 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:39.072499 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:39.072976 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344
W20260709 12:25:39.050141 32342 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:39.074254 31718 leader_election.cc:400] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 37. Current term is 38.
I20260709 12:25:39.075300 31718 leader_election.cc:403] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Cancelling election due to peer responding with higher term
I20260709 12:25:39.035007 10619 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:39.043447 10627 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 39 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:39.076023 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:39.072650 31818 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 29.
I20260709 12:25:39.091972 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:39.090152 31791 leader_election.cc:336] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:39.093189 31791 leader_election.cc:336] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.093595 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.048460 10649 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader election lost for term 30. Reason: could not achieve majority
I20260709 12:25:39.091041 31643 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:39.045074 10608 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:39.048182 10595 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:39.047998 10263 raft_consensus.cc:697] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 38 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:39.095736 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:39.100585 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:39.102842 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:39.075675 10497 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:39.119551 31747 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:39.104233 31748 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 26 index: 17. Preceding OpId from leader: term: 28 index: 18. (index mismatch)
I20260709 12:25:39.105037 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:39.121816 31751 raft_consensus.cc:2468] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 35.
I20260709 12:25:39.123433 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:39.106787 31827 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:39.109591 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:39.111683 31824 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
W20260709 12:25:39.113106 31791 consensus_peers.cc:597] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.130872 31749 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 33 index: 19. (index mismatch)
I20260709 12:25:39.114995 10477 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:39.115073 10669 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:25:39.105082 10464 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.094775 10626 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:39.139838 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:39.119324 10276 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:39.115337 10410 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:39.131212 31791 consensus_peers.cc:597] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.132771 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:39.094352 10213 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:39.116916 10245 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:39.110477 10591 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:39.140558 31827 raft_consensus.cc:2468] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 35.
I20260709 12:25:39.126873 10497 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.075829 10470 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.054s
W20260709 12:25:39.142891 31562 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4d48c83c761f48079bc6ea176649dda0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 27 committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:39.106226 10493 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.032s
I20260709 12:25:39.151473 10245 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.156622 10567 raft_consensus.cc:3060] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:39.161020 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:39.162333 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:39.135141 10477 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:39.145854 31791 leader_election.cc:336] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.149649 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:39.150147 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:39.146883 31643 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:39.128640 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 425782us
I20260709 12:25:39.156751 10245 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:39.098198 10619 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.143749 10669 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:25:39.164335 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 47074.6us
W20260709 12:25:39.173768 31717 consensus_peers.cc:597] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.173686 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.165963 10258 consensus_queue.cc:1059] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.021s
I20260709 12:25:39.142795 10464 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:39.188742 31791 leader_election.cc:336] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.190939 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.164117 10497 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:39.192281 31791 leader_election.cc:336] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:39.193251 31791 leader_election.cc:336] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.193660 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.196089 10616 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:39.189276 31718 leader_election.cc:304] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:39.176098 10477 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:39.200052 31746 raft_consensus.cc:2410] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 32 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 28 index: 20.
I20260709 12:25:39.174681 10213 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:39.201284 31643 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:39.178925 10669 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:39.153252 10410 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.190996 10608 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:25:39.189842 10591 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:39.176846 10225 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.041s
I20260709 12:25:39.252743 10276 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 16, Time since last communication: 0.001s
I20260709 12:25:39.199343 10626 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election lost for term 27. Reason: could not achieve majority
I20260709 12:25:39.222774 10213 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.256503 10213 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:39.258745 10619 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:39.259538 10672 raft_consensus.cc:3055] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 38 LEADER]: Stepping down as leader of term 38
I20260709 12:25:39.260092 10672 raft_consensus.cc:740] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 38 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:39.260943 10410 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:39.261173 10672 consensus_queue.cc:260] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.25, Last appended by leader: 25, Current term: 38, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:39.239621 10667 log.cc:927] Time spent T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.060s	user 0.000s	sys 0.000s
I20260709 12:25:39.217926 10652 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.022s
I20260709 12:25:39.240134 10493 raft_consensus.cc:2804] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:39.165380 10621 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:39.177171 10263 consensus_queue.cc:237] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 36.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.262603 10672 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Advancing to term 39
I20260709 12:25:39.229755 10660 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:39.341675 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:39.350208 32342 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:39.350809 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 259987us
I20260709 12:25:39.350929 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:39.352448 10673 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:39.353055 10673 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.360262 10066 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:39.334020 10562 raft_consensus.cc:493] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:39.351480 32342 leader_election.cc:336] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.274744 10470 raft_consensus.cc:2749] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:39.317993 10493 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.366340 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.381853 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:39.383239 10658 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.211s	user 0.003s	sys 0.001s
I20260709 12:25:39.384758 10464 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:39.385957 10673 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:39.389745 31820 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:41204 (request call id 4208) took 1206 ms. Trace:
I20260709 12:25:39.405354 31820 rpcz_store.cc:276] 0709 12:25:38.183766 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.185126 (+  1360us) service_pool.cc:224] Handling call
0709 12:25:39.389566 (+1204440us) spinlock_profiling.cc:243] Waited 460 ms on lock 0x7b54001982d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd92fa 00007fe901d9ab17 00007fe8fb3156d4 00007fe8fb315662 00007fe8fb315591 00007fe8fb31550c 00007fe8fb3147c2 00007fe8fb11048c 00007fe8fb10f8c6 00007fe8fb10f26f
0709 12:25:39.389677 (+   111us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":1058025216}
W20260709 12:25:39.344485 10622 consensus_meta.cc:287] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.928s	user 0.011s	sys 0.002s
I20260709 12:25:39.391017 10621 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.357004 31575 catalog_manager.cc:5697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:39.399276 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
W20260709 12:25:39.403403 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 51947.9us
I20260709 12:25:39.406867 31825 raft_consensus.cc:1275] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 35 index: 20. (index mismatch)
I20260709 12:25:39.364080 10562 raft_consensus.cc:515] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:39.416818 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 188002us
I20260709 12:25:39.401548 10493 raft_consensus.cc:3060] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:25:39.412428 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:39.408459 10662 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:39.434971 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:39.438982 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:39.439590 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:39.439913 31748 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 35.
I20260709 12:25:39.441382 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:39.442970 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:39.380260 10660 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.447007 31752 raft_consensus.cc:1275] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 35 index: 21. (index mismatch)
I20260709 12:25:39.438228 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:39.443044 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:39.452920 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:39.453639 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:39.447165 31791 consensus_peers.cc:597] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.455862 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:39.459362 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:39.464262 31791 leader_election.cc:336] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:39.465444 31791 consensus_peers.cc:597] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.424346 10665 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election won for term 32
W20260709 12:25:39.466773 31791 leader_election.cc:336] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.467211 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 27 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.468139 10066 raft_consensus.cc:2804] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:39.468600 10066 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.468859 10616 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 27. Reason: could not achieve majority
I20260709 12:25:39.469143 10066 raft_consensus.cc:3060] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:39.473272 32375 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 23 index: 13. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:25:39.430600 31575 catalog_manager.cc:5697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 29 to 37, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 37 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:39.487351 31575 catalog_manager.cc:5697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 23 to 30. New cstate: current_term: 30 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
W20260709 12:25:39.471527 31791 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.489018 31819 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
W20260709 12:25:39.501180 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48360.4us
I20260709 12:25:39.620141 31642 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:39.508303 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:39.510128 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 38 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:39.511018 32374 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 32 index: 24. Preceding OpId from leader: term: 34 index: 25. (index mismatch)
I20260709 12:25:39.500336 10665 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:39.503808 10465 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:39.519444 10213 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.520243 31745 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 39 because replica has last-logged OpId of term: 38 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 24.
I20260709 12:25:39.449832 10464 raft_consensus.cc:493] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:39.411420 10622 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.522028 31748 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
I20260709 12:25:39.535959 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:39.557155 31820 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 25 index: 14. Preceding OpId from leader: term: 27 index: 15. (index mismatch)
I20260709 12:25:39.594224 10342 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.619575 31575 catalog_manager.cc:5697] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 27 to 31, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 31 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:39.608426 10562 leader_election.cc:290] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:39.609041 10599 consensus_meta.cc:287] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.860s	user 0.011s	sys 0.003s
W20260709 12:25:39.615473 10620 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.809s	user 0.010s	sys 0.002s
I20260709 12:25:39.567027 10660 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:39.618419 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:39.659999 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 188480us
I20260709 12:25:39.628566 10403 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
W20260709 12:25:39.651984 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 116132us
W20260709 12:25:39.655087 31717 leader_election.cc:336] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.655380 31746 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 32 index: 24. Preceding OpId from leader: term: 34 index: 25. (index mismatch)
I20260709 12:25:39.618000 10477 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.627087 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:25:39.692011 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 65032.3us
W20260709 12:25:39.677940 31719 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:39.661468 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 229150us
I20260709 12:25:39.691160 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:39.707989 31791 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.708634 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.717792 31823 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 36. Current term is 37. Ops: []
W20260709 12:25:39.718130 31791 leader_election.cc:336] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.720809 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.711942 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:25:39.723642 31791 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.724279 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.690852 31717 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.690310 31575 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 26 to 34, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 34 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:39.687795 10567 consensus_meta.cc:287] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.530s	user 0.006s	sys 0.008s
I20260709 12:25:39.660480 10649 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:25:39.734802 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 79749us
I20260709 12:25:39.739853 31575 catalog_manager.cc:5697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 31 to 35, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 35 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:39.631338 10066 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:39.750375 31748 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30. Ops: []
I20260709 12:25:39.642740 10118 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.075s
I20260709 12:25:39.663271 10620 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:39.720865 10468 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s	user 0.139s	sys 0.203s
W20260709 12:25:39.777076 10468 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.596s	user 0.140s	sys 0.203s
I20260709 12:25:39.707428 10645 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:39.777307 10213 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:39.777311 31751 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 37.
I20260709 12:25:39.748425 31718 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:39.778595 10620 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:39.779512 10620 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:39.738351 10644 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election won for term 38
W20260709 12:25:39.804579 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 45711.8us
I20260709 12:25:39.805156 10644 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:39.805728 10644 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:39.806974 31642 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:39.649112 10497 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:39.659346 10342 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:39.824810 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 101214us
I20260709 12:25:39.764209 10066 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:39.734704 10637 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:39.750115 10567 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader election lost for term 37. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 37. Current term is 38.
W20260709 12:25:39.770711 30503 thread.cc:527] Waited for 6000ms trying to join with heartbeat (tid 31703)
I20260709 12:25:39.717782 10616 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:39.660126 10464 raft_consensus.cc:515] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.775456 10618 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:39.755389 10511 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 12, Time since last communication: 0.033s
I20260709 12:25:39.788317 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 30 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:39.852952 31819 raft_consensus.cc:2376] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30. Re-sending same reply.
I20260709 12:25:39.789371 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:39.717970 10562 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:39.790297 10477 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:39.791219 10665 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 31.20, Last appended by leader: 20, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.794369 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:39.795729 10622 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:39.747435 10423 raft_consensus.cc:2749] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:39.717430 10591 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:39.736006 10617 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:39.782724 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "945fd223ac334bce88917d1b9b4c72e1" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:39.661123 10599 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.747181 10470 consensus_queue.cc:1048] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 18, Time since last communication: 0.058s
I20260709 12:25:39.823153 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:39.886343 31820 raft_consensus.cc:2368] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 37. Current term is 38.
W20260709 12:25:39.826092 31791 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.827885 10066 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:25:39.839133 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 90102.4us
W20260709 12:25:39.837544 10669 consensus_meta.cc:287] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.597s	user 0.011s	sys 0.000s
I20260709 12:25:39.873603 31744 raft_consensus.cc:2468] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 38.
I20260709 12:25:39.876200 32375 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 37: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:39.755681 31575 catalog_manager.cc:5697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 31 to 34, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 34 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:39.894201 31575 catalog_manager.cc:5697] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 22 to 34. New cstate: current_term: 34 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:39.896153 31575 catalog_manager.cc:5697] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 29 to 37, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 37 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:39.889235 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 151424us
W20260709 12:25:39.887866 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61817.4us
I20260709 12:25:39.905445 31822 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:39.907804 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:39.886356 10591 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:39.911201 10497 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:39.863729 10562 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:39.913646 10562 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:39.845274 10594 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.043s
I20260709 12:25:39.911350 10591 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:39.916661 10599 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:39.867465 10245 consensus_meta.cc:287] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: Time spent flushing consensus metadata: real 0.688s	user 0.009s	sys 0.004s
I20260709 12:25:39.857004 10615 consensus_queue.cc:1059] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 36.18, Next index: 19, Last known committed idx: 17, Time since last communication: 0.062s
I20260709 12:25:39.918696 10245 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:39.919536 10615 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:39.920048 10615 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:39.920974 10245 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:39.834324 10493 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:39.923480 10493 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:39.885717 10258 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 13, Time since last communication: 0.050s
I20260709 12:25:39.926044 31747 raft_consensus.cc:2368] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 32. Current term is 33.
I20260709 12:25:39.927779 10420 raft_consensus.cc:3055] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 36 LEADER]: Stepping down as leader of term 36
I20260709 12:25:39.928153 10420 raft_consensus.cc:740] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 36 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:25:39.891012 31717 leader_election.cc:336] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:39.902506 31791 leader_election.cc:336] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:39.938267 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47294.7us
W20260709 12:25:39.941758 31791 consensus_peers.cc:597] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:39.945582 32370 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Rejecting Update request from peer 4d48c83c761f48079bc6ea176649dda0 for earlier term 29. Current term is 30. Ops: []
W20260709 12:25:39.942849 31717 leader_election.cc:336] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:39.952041 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:39.946676 31717 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.953131 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.953301 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:39.897989 31575 catalog_manager.cc:5697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 30 to 33. New cstate: current_term: 33 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:39.954805 10403 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.955417 10403 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:39.956363 10675 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.066s	user 0.000s	sys 0.000s
I20260709 12:25:39.957790 10403 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:39.959347 10464 leader_election.cc:290] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:39.961790 10342 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:39.963092 10567 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:39.966254 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:39.967015 31746 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:39.970136 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 27 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:39.977917 31746 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 37.
I20260709 12:25:39.981209 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:39.985821 31575 catalog_manager.cc:5697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 21 to 28. New cstate: current_term: 28 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:39.939366 10678 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:39.996412 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 143907us
I20260709 12:25:39.999538 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:40.004173 31718 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 30 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:39.995020 31575 catalog_manager.cc:5697] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 24 to 28, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 28 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:39.995672 31717 leader_election.cc:336] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:39.999465 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 51875.3us
I20260709 12:25:40.014470 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:40.016758 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:40.018824 10477 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:40.021060 31575 catalog_manager.cc:5697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 30 to 32. New cstate: current_term: 32 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:40.021584 31791 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:40.025866 31823 raft_consensus.cc:2393] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:25:40.023535 31717 leader_election.cc:336] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.027413 31717 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:40.026108 31718 leader_election.cc:400] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 37. Current term is 38.
I20260709 12:25:40.028970 31718 leader_election.cc:403] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Cancelling election due to peer responding with higher term
I20260709 12:25:40.028096 31575 catalog_manager.cc:5697] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 24 to 37, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 37 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:39.940644 10669 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.031992 31575 catalog_manager.cc:5697] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 31 to 36, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 36 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:40.039413 31575 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 29 to 31, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 31 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:40.013007 10535 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.000435 10619 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:39.999163 10637 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:40.041400 31575 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 31, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 31 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:39.947847 10644 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.023910 10672 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader election won for term 30
I20260709 12:25:40.053611 31575 catalog_manager.cc:5697] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 18 to 30, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 30 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:40.036911 31717 leader_election.cc:336] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.057013 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:40.057309 31575 catalog_manager.cc:5697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 32, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 32 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:40.058668 31717 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:40.059795 31717 leader_election.cc:336] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.060153 31717 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:40.030611 10626 raft_consensus.cc:2804] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:25:40.040691 10567 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:40.019950 10591 consensus_queue.cc:1048] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 17, Time since last communication: 0.031s
I20260709 12:25:40.040633 10620 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:40.074848 10676 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:40.077612 10493 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:25:40.079290 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:40.105926 31823 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 31
I20260709 12:25:40.056596 10637 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:40.054824 10535 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:40.024405 10622 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 29.20, Next index: 21, Last known committed idx: 20, Time since last communication: 0.031s
I20260709 12:25:40.084764 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:40.084762 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:40.116868 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:40.087554 10468 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.088205 10033 raft_consensus.cc:493] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.121704 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:40.087998 10562 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.089996 10644 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:40.013975 10678 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:40.100041 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:40.140071 31819 raft_consensus.cc:3060] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Advancing to term 38
I20260709 12:25:40.101130 32370 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:33976 (request call id 1511) took 1294 ms. Trace:
I20260709 12:25:40.058082 10464 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:40.049944 10342 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.055s
I20260709 12:25:40.129940 31718 leader_election.cc:304] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:40.140916 32370 rpcz_store.cc:276] 0709 12:25:38.806409 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.806596 (+   187us) service_pool.cc:224] Handling call
0709 12:25:38.835531 (+ 28935us) raft_consensus.cc:1471] Updating replica for 0 ops
0709 12:25:39.946341 (+1110810us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:40.100802 (+154461us) spinlock_profiling.cc:243] Waited 1.11 s on lock 0x7b540074e854. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3e571 00007fe8fcdd5e01 00007fe8fcdd483e 00007fe901d99419 00007fe8fb310bd4 00007fe8fb310b62 00007fe8fb310a91 00007fe8fb310a0c 00007fe8fb30fcc2 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:40.101054 (+   252us) inbound_call.cc:177] Queueing success response
Metrics: {"spinlock_wait_cycles":2552251136}
W20260709 12:25:40.152972 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 93195.4us
W20260709 12:25:40.153949 31717 consensus_peers.cc:597] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:40.151067 10468 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.157858 10468 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:40.154688 31717 leader_election.cc:336] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.061167 10619 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:40.128768 31575 catalog_manager.cc:5697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 25 to 28, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 28 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:40.159495 31717 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:40.106340 10626 raft_consensus.cc:697] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 37 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:40.111605 10672 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:40.111083 10637 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:25:40.173645 31717 consensus_peers.cc:597] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:40.164338 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 147823us
I20260709 12:25:40.175717 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:40.176023 31819 raft_consensus.cc:2468] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 38.
I20260709 12:25:40.177929 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:40.171655 31575 catalog_manager.cc:5697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 15 to 28, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 28 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:40.135733 10618 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:40.184360 31718 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:40.159097 10567 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:40.192165 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:40.191179 31575 catalog_manager.cc:5697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 23 to 24, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 24 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:40.202544 31575 catalog_manager.cc:5697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:40.173147 10497 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.175997 10644 raft_consensus.cc:2749] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:40.234169 31820 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:40.177177 10464 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:40.181887 10622 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Stepping down as leader of term 29
I20260709 12:25:40.138684 10033 raft_consensus.cc:515] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.273566 31820 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 35.
I20260709 12:25:40.210160 10691 raft_consensus.cc:2804] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:25:40.263331 10622 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Follower/Learner. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:25:40.316125 10685 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.250s	user 0.000s	sys 0.001s
I20260709 12:25:40.326913 31822 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
W20260709 12:25:40.333597 31717 consensus_peers.cc:597] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:40.259922 10497 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.318344 10665 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:25:40.365146 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:40.366137 31827 raft_consensus.cc:2393] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 38: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:40.308702 10691 raft_consensus.cc:697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 38 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:40.350215 10684 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.550s	user 0.002s	sys 0.002s
I20260709 12:25:40.354893 10604 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:25:40.361102 10619 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:40.379711 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:40.399981 10535 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:40.343511 10669 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:40.363063 10665 raft_consensus.cc:493] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.439980 31823 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
W20260709 12:25:40.543571 31642 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.734s	user 0.019s	sys 0.015s
W20260709 12:25:40.549654 31717 leader_election.cc:336] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:40.571995 31717 consensus_peers.cc:597] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:40.573776 31717 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:40.574864 31642 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.765s	user 0.019s	sys 0.015s
W20260709 12:25:40.575402 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 765928us
I20260709 12:25:40.576370 31823 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:40.577108 31643 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:40.577939 31826 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:41210 (request call id 4494) took 1127 ms. Trace:
I20260709 12:25:40.578552 31826 rpcz_store.cc:276] 0709 12:25:39.450092 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:39.450288 (+   196us) service_pool.cc:224] Handling call
0709 12:25:39.450678 (+   390us) raft_consensus.cc:1471] Updating replica for 2 ops
0709 12:25:39.451143 (+   465us) raft_consensus.cc:1534] Early marking committed up to index 0
0709 12:25:39.451173 (+    30us) raft_consensus.cc:1539] Triggering prepare for 2 ops
0709 12:25:39.451889 (+   716us) log.cc:844] Serialized 52 byte log entry
0709 12:25:39.749808 (+297919us) raft_consensus.cc:1645] Marking committed up to 16
0709 12:25:39.749848 (+    40us) raft_consensus.cc:1697] Filling consensus response to leader.
0709 12:25:39.749919 (+    71us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0709 12:25:40.349514 (+599595us) spinlock_profiling.cc:243] Waited 10.2 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:25:40.577272 (+227758us) raft_consensus.cc:1685] finished
0709 12:25:40.577311 (+    39us) raft_consensus.cc:1691] UpdateReplicas() finished
0709 12:25:40.577735 (+   424us) inbound_call.cc:177] Queueing success response
Metrics: {"raft.queue_time_us":983,"raft.run_cpu_time_us":121,"raft.run_wall_time_us":10617,"spinlock_wait_cycles":23390464,"thread_start_us":228628,"threads_started":1,"wal-append.queue_time_us":299431}
I20260709 12:25:40.599505 31643 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:40.384298 10258 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:40.436208 10562 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:40.613624 10616 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:40.307140 10626 consensus_queue.cc:237] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 36.18, Last appended by leader: 18, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:40.650123 10342 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:40.318586 10672 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 28.17, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.659186 10342 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 35 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:40.660403 10493 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:25:40.661311 10342 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.476728 10678 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:40.471032 10691 consensus_queue.cc:237] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.428982 10622 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 29.20, Last appended by leader: 20, Current term: 29, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:40.663789 10717 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.187s	user 0.000s	sys 0.001s
I20260709 12:25:40.665642 10622 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Advancing to term 30
I20260709 12:25:40.667188 10612 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.645429 10496 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:40.668149 10612 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:40.669080 10496 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:40.670470 10616 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.671149 10496 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 29.16, Last appended by leader: 16, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.672669 10615 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:40.673487 10616 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:40.630780 10033 leader_election.cc:290] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:40.649891 10258 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.653251 10652 consensus_queue.cc:1059] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 32.17, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:40.436941 10665 raft_consensus.cc:3060] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:40.687095 10497 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:40.690834 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 310118us
I20260709 12:25:40.695360 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:40.705523 10403 raft_consensus.cc:3055] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:40.706212 10403 raft_consensus.cc:740] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:40.707317 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:40.714125 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2c162dbe81714319bfeccd9d7cb7bfea" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:40.715065 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:40.731284 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:40.733368 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:40.729903 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 114651us
I20260709 12:25:40.738397 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:40.743469 31827 raft_consensus.cc:1240] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 32. Current term is 33. Ops: []
W20260709 12:25:40.754493 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 55679.2us
I20260709 12:25:40.766170 10662 consensus_queue.cc:1059] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.007s
I20260709 12:25:40.767853 10612 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:40.769613 10662 raft_consensus.cc:3055] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Stepping down as leader of term 32
I20260709 12:25:40.770198 10662 raft_consensus.cc:740] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:40.774847 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:40.775527 10420 consensus_queue.cc:260] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 36.18, Last appended by leader: 18, Current term: 36, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:40.776983 10420 raft_consensus.cc:3060] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Advancing to term 37
W20260709 12:25:40.779990 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 604244us
I20260709 12:25:40.781641 32342 leader_election.cc:400] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 32. Current term is 33.
I20260709 12:25:40.785598 10662 consensus_queue.cc:260] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.21, Last appended by leader: 21, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:40.813169 31791 leader_election.cc:336] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.814050 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:40.815459 10687 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:25:40.815898 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 66638.2us
I20260709 12:25:40.825958 31718 leader_election.cc:304] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:40.828665 31718 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:40.830401 10562 raft_consensus.cc:2749] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:25:40.831203 10712 log.cc:927] Time spent T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.355s	user 0.002s	sys 0.004s
I20260709 12:25:40.831462 10562 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:40.755878 10665 raft_consensus.cc:515] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:40.847852 30503 thread.cc:527] Waited for 7000ms trying to join with heartbeat (tid 31703)
I20260709 12:25:40.851888 10646 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:40.852852 10646 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:40.853381 10258 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:40.857021 10720 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.364s	user 0.000s	sys 0.001s
W20260709 12:25:40.860921 10719 log.cc:927] Time spent T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.367s	user 0.002s	sys 0.000s
I20260709 12:25:40.866868 10615 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:25:40.867272 10701 log.cc:927] Time spent T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.380s	user 0.000s	sys 0.003s
I20260709 12:25:40.867801 10615 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:40.870800 10620 consensus_meta.cc:287] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Time spent flushing consensus metadata: real 0.766s	user 0.009s	sys 0.014s
I20260709 12:25:40.871554 10620 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader election lost for term 37. Reason: Vote denied by peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) with higher term. Message: Invalid argument: T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for earlier term 37. Current term is 38.
I20260709 12:25:40.872762 10620 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:40.875046 31792 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.201.196:44855 timed out after 1.892s (SENT) [suppressed 105 similar messages]
I20260709 12:25:40.875226 10646 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:40.877641 31827 raft_consensus.cc:1240] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 32. Current term is 33. Ops: []
I20260709 12:25:40.884775 10403 consensus_queue.cc:260] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 32, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.885458 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:40.886153 10403 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:40.886495 32373 raft_consensus.cc:2410] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b for term 32 because replica has last-logged OpId of term: 31 index: 17, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 16.
I20260709 12:25:40.889565 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2c281519748458cb11517e3303d0a17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:40.889930 31822 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 30.
I20260709 12:25:40.892259 10615 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:40.893965 31825 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:40.899130 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:40.842268 10662 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:40.917907 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:40.920517 32373 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 38.
W20260709 12:25:40.929239 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 93602.4us
I20260709 12:25:40.937592 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:40.937687 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:40.939357 10599 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:40.939870 31825 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 39.
I20260709 12:25:40.940129 10599 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:40.940860 31717 leader_election.cc:336] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.941195 10646 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:40.941612 31717 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:40.941900 10646 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.943992 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:40.944399 10646 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:40.945243 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:40.946012 10619 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:40.947269 10599 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:40.947911 10619 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:25:40.937601 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:40.948469 10619 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:40.948869 31749 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 37
I20260709 12:25:40.948925 10619 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:25:40.949980 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:40.951152 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:40.951941 31745 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:40.912173 10258 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:40.955149 10258 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:40.957491 10258 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:40.958761 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 29 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:40.959525 31746 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:40.961010 31826 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 37 LEADER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 36. Current term is 37. Ops: []
I20260709 12:25:40.899195 10620 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:40.984920 31791 leader_election.cc:336] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.991621 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:40.990350 31717 leader_election.cc:336] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:40.966820 10729 raft_consensus.cc:493] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:41.003263 31749 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 37.
I20260709 12:25:41.006064 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:40.988345 10258 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:41.013407 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.014032 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:41.014604 31824 raft_consensus.cc:2393] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 39 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 39: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:40.998245 10622 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:41.018173 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:41.018340 10263 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.012s
I20260709 12:25:41.021848 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:41.018196 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:41.027464 10626 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:41.027236 10258 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:41.028178 10626 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:41.030052 10258 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:41.027318 10646 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:41.030817 10470 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:41.031705 10729 raft_consensus.cc:515] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:41.031935 10470 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:41.031271 10646 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.030355 10626 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:41.037660 10732 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.115s	user 0.000s	sys 0.001s
I20260709 12:25:41.038247 10470 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:41.039292 10646 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:41.041414 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.050295 31746 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 31.
W20260709 12:25:41.054929 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 72683.4us
I20260709 12:25:41.058091 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:41.058858 32375 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Advancing to term 39
W20260709 12:25:41.065918 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 52590.1us
I20260709 12:25:41.068189 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.072604 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.073947 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:41.074601 32373 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 38: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:41.074894 10620 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:41.085217 31717 leader_election.cc:336] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.087716 31745 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 35.
I20260709 12:25:41.088460 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.091773 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:41.091753 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f2f85d943e24cfdb47fe2ad3185097f" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.048022 10665 leader_election.cc:290] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:41.098351 31717 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.091756 10691 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:41.099395 31717 leader_election.cc:336] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.099442 10691 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:41.100406 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.103678 10567 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:41.055977 10729 leader_election.cc:290] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:41.106966 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:41.108708 31718 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.086807 10258 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:41.117573 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.101787 10691 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:41.107292 10470 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:41.122730 10225 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.135037 10423 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:41.119465 10567 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:25:41.128480 31791 consensus_peers.cc:597] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:41.140447 31643 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:41.148134 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 31 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:41.142174 10225 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:41.170145 10637 consensus_meta.cc:287] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Time spent flushing consensus metadata: real 0.945s	user 0.006s	sys 0.018s
W20260709 12:25:41.176153 10734 log.cc:927] Time spent T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.261s	user 0.002s	sys 0.000s
I20260709 12:25:41.176604 10687 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.177659 10652 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:41.142536 10470 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:41.177362 10687 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:41.175784 10637 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:41.191046 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7523cff6f9fb44b5ba8fc04726d85b65" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 35 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:41.191592 10741 log.cc:927] Time spent T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.097s	user 0.002s	sys 0.000s
I20260709 12:25:41.192677 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:41.197163 10637 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:41.198040 31643 leader_election.cc:304] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:41.183095 31717 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:41.199223 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55784.3us
W20260709 12:25:41.200080 31717 leader_election.cc:336] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:41.206409 31790 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.214136 32375 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 39 because replica has last-logged OpId of term: 38 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 24.
I20260709 12:25:41.187554 10225 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:41.217846 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.219334 31822 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:25:41.219518 10258 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:41.220072 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:41.248744 31717 leader_election.cc:336] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.220862 10669 raft_consensus.cc:2749] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:41.221292 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 30 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.221603 10496 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.229317 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:41.231202 31823 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 36: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:41.228096 32371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 39 candidate_status { last_received { term: 37 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:41.251986 31717 leader_election.cc:336] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.234622 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.229221 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:41.229909 32369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:41.254431 32371 raft_consensus.cc:2410] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 39 because replica has last-logged OpId of term: 38 index: 25, which is greater than that of the candidate, which has last-logged OpId of term: 37 index: 24.
I20260709 12:25:41.265267 32376 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 38: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:41.226935 10619 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:41.274180 31819 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:41.262410 10258 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:41.275940 31819 raft_consensus.cc:2393] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 38: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:25:41.279524 31562 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4d48c83c761f48079bc6ea176649dda0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 27 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:41.280066 10496 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:41.280918 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 33 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:41.287221 32370 raft_consensus.cc:2410] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 for term 33 because replica has last-logged OpId of term: 32 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 31 index: 20.
W20260709 12:25:41.289701 10738 log.cc:927] Time spent T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: Append to log took a long time: real 0.091s	user 0.000s	sys 0.002s
W20260709 12:25:41.287410 10736 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.226s	user 0.005s	sys 0.000s
I20260709 12:25:41.282601 10496 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:41.302723 31717 consensus_peers.cc:597] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:41.303527 10225 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:41.319727 31717 consensus_peers.cc:597] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:41.325608 10423 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:41.351589 31749 raft_consensus.cc:1240] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 36. Current term is 37. Ops: []
I20260709 12:25:41.330370 10718 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:41.355932 32373 raft_consensus.cc:2468] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 37.
I20260709 12:25:41.333350 10225 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:41.365451 31643 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:41.365780 10423 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:41.354952 10619 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:41.368353 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.377162 10652 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:41.361482 10718 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:41.380334 10652 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:41.378897 31751 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:41.378087 10225 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:41.381526 10718 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:41.368479 10423 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:41.393950 31752 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:25:41.395207 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.396052 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:41.382423 10258 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:41.396042 31822 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 30.
W20260709 12:25:41.439523 31717 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.446019 31825 raft_consensus.cc:1240] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Rejecting Update request from peer b95aeeaf4c51442b8092cff1de674f2b for earlier term 35. Current term is 36. Ops: []
I20260709 12:25:41.454170 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:41.448655 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:41.450963 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 32 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:41.460625 31643 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:41.471555 10725 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Leader pre-election won for term 31
W20260709 12:25:41.503079 32342 leader_election.cc:336] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.503882 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.505051 10263 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:41.506898 10288 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:41.494598 10725 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:41.516471 10725 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:41.521549 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 39 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:41.524273 31753 raft_consensus.cc:2368] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 39. Current term is 40.
I20260709 12:25:41.473853 10646 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:41.526409 31752 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 30. Current term is 31. Ops: []
I20260709 12:25:41.528774 10646 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Requested vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:41.534487 31750 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:25:41.535895 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:41.537360 31750 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d30338be26ae44ff8e627f7b8a99036c" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 38 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:41.564854 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 59114.4us
I20260709 12:25:41.568437 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:41.569766 31749 raft_consensus.cc:1275] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 36 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:25:41.571157 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:41.572158 10288 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:25:41.573648 31642 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:41.575173 10423 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:41.578190 10470 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:41.575197 10225 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:25:41.585531 31750 raft_consensus.cc:2468] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 38.
I20260709 12:25:41.598690 31821 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 29 index: 16. Preceding OpId from leader: term: 31 index: 17. (index mismatch)
I20260709 12:25:41.609248 31718 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.620920 31746 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 36 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
I20260709 12:25:41.624303 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:41.624850 10622 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.625229 31747 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 36: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:41.625456 10622 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:41.634189 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:41.634953 32370 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:25:41.640139 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 40 candidate_status { last_received { term: 38 index: 25 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:41.644794 31818 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:25:41.649601 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:41.656811 10622 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:41.659106 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.663515 31825 raft_consensus.cc:1275] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:25:41.665349 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:41.666733 31823 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:41.651024 10288 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.019s
I20260709 12:25:41.660910 10225 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Leader election won for term 37
I20260709 12:25:41.675335 10225 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 37 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:41.676582 10225 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 34.22, Last appended by leader: 22, Current term: 37, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:41.678159 10678 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:41.663733 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:41.660809 10637 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:41.680048 10637 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:41.649119 10616 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.034s
I20260709 12:25:41.684636 10617 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:25:41.685359 10617 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:41.688755 10617 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:41.690128 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:41.636121 10423 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:41.703415 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 49735.3us
W20260709 12:25:41.707484 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 133952us
I20260709 12:25:41.682550 10637 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:41.709928 31824 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 28 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:41.710201 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:41.662786 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.719415 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.720732 31824 raft_consensus.cc:2393] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:41.720551 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.720630 10691 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:41.723999 10691 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:41.724706 10470 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:41.726286 10691 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:41.727032 32342 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:41.726052 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.718972 31642 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:41.731868 31718 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.739204 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.732372 10470 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:41.736174 10691 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:25:41.747745 10620 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:25:41.757104 32342 leader_election.cc:400] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 39 pre-election: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 39. Current term is 40.
I20260709 12:25:41.757474 32342 leader_election.cc:403] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 39 pre-election: Cancelling election due to peer responding with higher term
I20260709 12:25:41.745446 10423 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:41.752403 10652 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:41.764568 10687 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:41.765223 10604 consensus_queue.cc:1048] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.001s
I20260709 12:25:41.767585 32342 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:41.768447 10604 raft_consensus.cc:2804] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:25:41.769192 10604 raft_consensus.cc:697] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 38 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:25:41.767611 10746 log.cc:927] Time spent T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.078s	user 0.002s	sys 0.002s
I20260709 12:25:41.770270 10604 consensus_queue.cc:237] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:41.757890 31717 leader_election.cc:336] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:41.782348 31791 leader_election.cc:336] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:41.791030 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 78098.7us
W20260709 12:25:41.792512 31641 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:25:41.790022 31717 leader_election.cc:336] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.798362 10602 consensus_queue.cc:1059] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:41.798285 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:41.809478 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 75215.9us
I20260709 12:25:41.811318 32374 raft_consensus.cc:1275] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 36 index: 18. Preceding OpId from leader: term: 37 index: 19. (index mismatch)
W20260709 12:25:41.813449 32342 leader_election.cc:336] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.814093 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.826877 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:41.835526 32342 leader_election.cc:336] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.836166 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:41.837723 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:41.842121 10690 consensus_queue.cc:1059] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: INVALID_TERM, Last received: 35.19, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:41.842832 10691 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:41.843691 10263 raft_consensus.cc:2749] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:41.843822 10665 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:41.845575 10665 raft_consensus.cc:3055] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Stepping down as leader of term 30
I20260709 12:25:41.845830 31818 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 40.
W20260709 12:25:41.848495 30503 thread.cc:527] Waited for 8000ms trying to join with heartbeat (tid 31703)
I20260709 12:25:41.852267 31792 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:41.812248 10752 log.cc:927] Time spent T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.058s	user 0.000s	sys 0.001s
I20260709 12:25:41.853111 10637 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:41.854485 31792 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 39 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:41.855255 10594 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
I20260709 12:25:41.855876 10637 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 40 FOLLOWER]: Leader election lost for term 39. Reason: could not achieve majority
I20260709 12:25:41.856130 31792 leader_election.cc:304] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:41.845976 10665 raft_consensus.cc:740] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:41.857788 10637 raft_consensus.cc:2749] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election lost for term 38. Reason: could not achieve majority
W20260709 12:25:41.859076 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 66988.3us
W20260709 12:25:41.817523 31717 leader_election.cc:336] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:41.867892 32342 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.868554 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:41.870929 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 59956.4us
I20260709 12:25:41.873577 31718 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:41.887871 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 76730.3us
W20260709 12:25:41.878897 31791 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.878111 31642 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:41.895936 32342 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:41.897727 31718 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.876452 10263 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:25:41.858083 10725 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:41.904874 31792 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:41.892210 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:41.905745 32342 leader_election.cc:336] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.910255 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:41.908771 31717 leader_election.cc:336] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.913295 31718 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:25:41.917661 31791 leader_election.cc:336] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.918332 31791 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.906767 10464 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:41.876179 10691 consensus_queue.cc:1048] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.034s
I20260709 12:25:41.907035 10567 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:41.921912 31718 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:41.923120 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.924136 31744 raft_consensus.cc:1240] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 34. Current term is 37. Ops: []
I20260709 12:25:41.924467 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 39 candidate_status { last_received { term: 36 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:41.927661 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:41.888285 10423 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.035s
I20260709 12:25:41.967478 32370 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 40.
W20260709 12:25:41.958174 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 61288.8us
W20260709 12:25:41.970311 32342 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:41.971283 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:41.915047 10617 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election lost for term 36. Reason: could not achieve majority
I20260709 12:25:41.976141 10420 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:41.972549 10662 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Leader pre-election lost for term 39. Reason: Vote denied by peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639) with higher term. Message: Invalid argument: T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 40 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 for earlier term 39. Current term is 40.
I20260709 12:25:41.905637 10665 consensus_queue.cc:260] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 30, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:41.988327 31791 leader_election.cc:336] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:41.959546 10616 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:41.959525 10687 raft_consensus.cc:2749] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
W20260709 12:25:41.982748 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 122497us
I20260709 12:25:41.938879 10718 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:25:41.994210 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:41.995323 32342 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:42.004647 31791 leader_election.cc:336] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:42.006215 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 84408.9us
I20260709 12:25:42.007490 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:42.005599 31791 leader_election.cc:336] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:42.009346 31791 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:42.011644 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:42.016824 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:42.014801 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59f73cf80df4c20b6c17715783f2121" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 36 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:42.003101 31717 consensus_peers.cc:597] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:42.017107 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 97515.3us
I20260709 12:25:42.014351 10616 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:42.018524 32374 raft_consensus.cc:2376] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 37. Re-sending same reply.
W20260709 12:25:42.049382 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 176850us
I20260709 12:25:42.015435 10594 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:42.056034 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:42.063090 10612 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.064023 10612 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:42.018683 10464 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:42.042248 10665 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:42.047243 31642 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:42.015342 10718 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 40 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:42.054817 10470 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:42.040589 10626 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:42.083743 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:42.085134 10620 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:42.090245 10620 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:25:42.092751 10758 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:25:42.095916 10612 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:42.099702 10718 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 38.25, Last appended by leader: 25, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:42.100543 31792 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:42.101403 10764 consensus_queue.cc:1059] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: INVALID_TERM, Last received: 34.22, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:25:42.102524 10764 raft_consensus.cc:3055] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Stepping down as leader of term 34
I20260709 12:25:42.103013 10764 raft_consensus.cc:740] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:42.103536 10626 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:25:42.104123 10764 consensus_queue.cc:260] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 34.22, Last appended by leader: 22, Current term: 34, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:42.105690 10764 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Advancing to term 37
I20260709 12:25:42.102811 10616 consensus_queue.cc:1048] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:42.109270 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:42.112799 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:42.116887 10725 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:42.159134 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 143169us
I20260709 12:25:42.160660 31752 raft_consensus.cc:1240] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Rejecting Update request from peer 5206fd0087404d2da964a5d985a04ff4 for earlier term 30. Current term is 31. Ops: [30.21-30.21]
I20260709 12:25:42.142825 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:42.134580 10342 raft_consensus.cc:3055] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 35 LEADER]: Stepping down as leader of term 35
I20260709 12:25:42.118021 10690 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:42.173316 31643 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 28 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:42.185510 31642 leader_election.cc:304] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:42.196007 10567 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:42.196722 10761 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Leader pre-election lost for term 28. Reason: could not achieve majority
I20260709 12:25:42.197702 10761 raft_consensus.cc:2749] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:42.200101 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:42.201160 10594 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:42.181705 10342 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 35 LEADER]: Becoming Follower/Learner. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:42.209115 10342 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 35, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:42.214939 10342 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:42.222111 10594 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 17, Time since last communication: 0.001s
I20260709 12:25:42.224512 31827 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:25:42.227407 10662 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260709 12:25:42.231681 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 44804.6us
I20260709 12:25:42.232458 10594 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260709 12:25:42.241062 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 54087.6us
W20260709 12:25:42.251197 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 168860us
I20260709 12:25:42.273949 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 28 candidate_status { last_received { term: 23 index: 13 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:42.277012 31746 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
W20260709 12:25:42.260136 31717 consensus_peers.cc:597] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:42.274331 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 31 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:42.286281 32341 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.293633 10690 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:42.294646 31791 consensus_peers.cc:597] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:42.294461 10690 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:42.295176 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 49639.5us
I20260709 12:25:42.296949 10690 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:42.332872 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:42.358703 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:42.372434 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 91433.2us
W20260709 12:25:42.367571 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 67403.6us
I20260709 12:25:42.361956 10496 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:42.432106 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 120879us
W20260709 12:25:42.429107 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:42.479833 32370 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 34 index: 22. Preceding OpId from leader: term: 37 index: 23. (index mismatch)
W20260709 12:25:42.496309 10771 log.cc:927] Time spent T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: Append to log took a long time: real 0.134s	user 0.001s	sys 0.000s
W20260709 12:25:42.517100 32344 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:25:42.456539 10496 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:42.536438 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 97762.2us
I20260709 12:25:42.550557 10726 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.551286 10726 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:42.553512 10726 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:42.540083 10567 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:42.554577 10567 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:42.556670 10786 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.556844 10567 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:42.557260 10786 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:42.534621 10496 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:42.564284 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:42.578773 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 179638us
I20260709 12:25:42.578847 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:42.591756 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 31 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:42.592506 10782 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:25:42.600867 10786 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:42.605605 10687 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:42.607790 10602 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:42.608242 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:42.610321 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:42.621594 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:42.618746 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:42.614889 10785 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader election lost for term 31. Reason: could not achieve majority
I20260709 12:25:42.622537 31822 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:42.642019 10784 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:25:42.628383 10602 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:42.652374 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:42.657490 10602 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:42.625419 10687 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:42.662321 10687 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:42.639683 10785 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:42.658547 10342 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:42.664086 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:42.664830 31821 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
W20260709 12:25:42.668159 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 98573.4us
I20260709 12:25:42.674324 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:42.679841 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:42.681354 31791 leader_election.cc:336] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:42.683334 31791 consensus_peers.cc:597] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:42.685632 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:42.691658 31752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:42.683944 31791 consensus_peers.cc:597] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:42.692394 31752 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 33.
I20260709 12:25:42.693733 31643 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:42.692901 10496 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:42.707501 10496 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:42.713487 10669 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:42.713989 10669 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.714383 10669 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:42.692847 10604 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:42.719012 10604 raft_consensus.cc:493] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:42.719389 10604 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:42.723683 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:42.729439 31744 raft_consensus.cc:1275] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:25:42.729933 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:42.737722 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:42.746402 31745 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
I20260709 12:25:42.755021 31642 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:42.757571 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:42.760869 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:42.736765 10496 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:42.771673 10776 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:42.766304 10786 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:42.786545 31827 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 40 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 37 index: 24. Preceding OpId from leader: term: 40 index: 26. (index mismatch)
I20260709 12:25:42.792047 32373 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 38 index: 25. Preceding OpId from leader: term: 40 index: 26. (index mismatch)
I20260709 12:25:42.797760 10618 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:42.775779 10766 consensus_queue.cc:1048] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.024s
I20260709 12:25:42.805047 10604 raft_consensus.cc:515] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:42.824319 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:42.825102 31818 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:42.821928 10669 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:42.840543 31818 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:42.841687 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c34755b3b9d040f69be353e35d7431e7" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 32 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:42.842325 31745 raft_consensus.cc:3060] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:42.847306 32343 leader_election.cc:304] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:42.848520 10760 raft_consensus.cc:2804] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:42.849092 10760 raft_consensus.cc:697] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:25:42.850703 30503 thread.cc:527] Waited for 9000ms trying to join with heartbeat (tid 31703)
I20260709 12:25:42.874985 31745 raft_consensus.cc:2468] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
W20260709 12:25:42.871158 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 48617.5us
I20260709 12:25:42.872015 10423 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.008s
I20260709 12:25:42.850019 10760 consensus_queue.cc:237] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 32.21, Last appended by leader: 21, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:42.895227 10718 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 24, Time since last communication: 0.001s
I20260709 12:25:42.901639 10678 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:42.902369 10678 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:42.906471 31717 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:42.907088 31717 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:42.916383 10678 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:42.910251 10619 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
W20260709 12:25:42.942142 32342 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:42.914219 31717 consensus_peers.cc:597] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:42.876982 10604 leader_election.cc:290] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:42.965039 32373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:42.966835 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 52785.5us
W20260709 12:25:42.972587 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 74283.8us
W20260709 12:25:43.113726 31717 consensus_peers.cc:597] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:43.116217 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:43.119884 31744 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:43.069411 10770 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.075s	user 0.004s	sys 0.000s
I20260709 12:25:43.148339 10669 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:43.161114 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 32 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:43.189292 31827 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:43.199668 31744 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:43.216411 31642 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:43.217466 10690 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:43.218056 10690 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Becoming Leader. State: Replica: b95aeeaf4c51442b8092cff1de674f2b, State: Running, Role: LEADER
I20260709 12:25:43.219015 10690 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 32.17, Last appended by leader: 17, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:43.284071 10619 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.284724 10619 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:43.292768 31827 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 34.
I20260709 12:25:43.303741 10622 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
W20260709 12:25:43.342007 31717 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:43.348838 31717 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:43.348654 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 133097us
W20260709 12:25:43.470232 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 122964us
I20260709 12:25:43.428964 10619 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:43.495486 10797 log.cc:927] Time spent T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.065s	user 0.000s	sys 0.000s
I20260709 12:25:43.466782 10567 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:43.347479 10622 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:43.491755 10782 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:43.552541 10803 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.206s	user 0.000s	sys 0.001s
W20260709 12:25:43.564435 10800 log.cc:927] Time spent T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
W20260709 12:25:43.564679 10802 log.cc:927] Time spent T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.075s	user 0.000s	sys 0.000s
I20260709 12:25:43.662182 31746 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:25:43.563102 10782 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:43.716741 10782 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:43.718741 10622 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:43.726125 10567 raft_consensus.cc:493] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:43.726823 10567 raft_consensus.cc:515] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:43.728982 10567 leader_election.cc:290] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:43.746583 10811 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.747262 10811 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:43.749368 10811 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:43.767226 10718 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:43.768026 10718 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:43.770821 10718 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:43.797557 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:43.815701 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:43.819408 31749 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 31: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:43.816486 32370 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:43.831496 10820 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.832181 10820 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:43.834724 10820 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:43.841960 10691 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:43.811105 10769 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.842654 10691 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:43.851154 30503 thread.cc:527] Waited for 10000ms trying to join with heartbeat (tid 31703)
I20260709 12:25:43.854311 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:43.862766 32376 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:43.828325 10420 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.027s
I20260709 12:25:43.874850 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:43.891516 10618 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:43.901314 10618 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:43.901852 10618 raft_consensus.cc:3060] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:25:43.907169 31717 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:43.908241 31717 consensus_peers.cc:597] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:43.848060 10769 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:43.915735 10769 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:43.919793 10691 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:43.922801 10826 raft_consensus.cc:493] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.923983 10826 raft_consensus.cc:515] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:43.926148 10777 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.926805 10777 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:43.928414 10826 leader_election.cc:290] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:43.928939 10777 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:43.929562 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54176.2us
I20260709 12:25:43.936012 31569 catalog_manager.cc:5697] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 25 to 28, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 28 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:43.963853 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:43.964617 31749 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 36.
I20260709 12:25:43.975986 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:43.977972 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:43.978989 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:43.979707 31820 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 36.
I20260709 12:25:43.983214 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "45588632b8d64f9fb9c2e2ba18b20546" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 31 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:43.967909 10777 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.986433 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:43.987649 10777 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:43.994601 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:43.995579 31822 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 36.
I20260709 12:25:43.997557 10825 raft_consensus.cc:493] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:43.990438 10777 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:43.998152 10825 raft_consensus.cc:515] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:44.000110 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "812ed9ab41214308b368b53417e6e9f3" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:44.000480 10825 leader_election.cc:290] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:43.940522 31575 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42072 (request call id 248) took 13917 ms (client timeout 15000 ms). Trace:
I20260709 12:25:44.002279 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:44.018769 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:44.021481 31569 catalog_manager.cc:5697] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 20 to 25, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 25 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
W20260709 12:25:44.025334 31791 consensus_peers.cc:597] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:44.032128 10824 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.037935 10824 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:44.043111 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:44.043839 31749 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b95aeeaf4c51442b8092cff1de674f2b in term 36.
I20260709 12:25:44.049738 10618 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:44.040225 10824 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:44.076784 10618 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:44.002089 31575 rpcz_store.cc:269] 0709 12:25:30.023003 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.139118 (+1116115us) service_pool.cc:224] Handling call
0709 12:25:43.940207 (+12801089us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:40.477364 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:40.478168 (+   804us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:40.478214 (+    46us) write_op.cc:435] Acquired schema lock
0709 12:25:40.478241 (+    27us) tablet.cc:662] Decoding operations
0709 12:25:40.487296 (+  9055us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:40.487411 (+   115us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:40.487458 (+    47us) tablet.cc:685] Acquiring locks for 23 operations
0709 12:25:40.489942 (+  2484us) tablet.cc:701] Row locks acquired
0709 12:25:40.489974 (+    32us) write_op.cc:260] PREPARE: finished
0709 12:25:40.490206 (+   232us) write_op.cc:270] Start()
0709 12:25:40.490386 (+   180us) write_op.cc:276] Timestamp: P: 1783599940490123 usec, L: 0
0709 12:25:40.490419 (+    33us) op_driver.cc:348] REPLICATION: starting
0709 12:25:40.491489 (+  1070us) log.cc:844] Serialized 9097 byte log entry
0709 12:25:40.831934 (+340445us) spinlock_profiling.cc:243] Waited 1.69 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8f9ccb0ac 00007fe902b259a9 00007fe8f9ce769b 00007fe8f9ce593f 00007fe8f9ce7c02 00007fe8fc96f20b
0709 12:25:41.745050 (+913116us) op_driver.cc:464] REPLICATION: finished
0709 12:25:41.898754 (+153704us) write_op.cc:301] APPLY: starting
0709 12:25:41.898900 (+   146us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:42.159894 (+260994us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:42.159948 (+    54us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:42.610686 (+450738us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:42.637216 (+ 26530us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=46,key_file_lookups=46,delta_file_lookups=0,mrs_lookups=0
0709 12:25:42.637288 (+    72us) write_op.cc:312] APPLY: finished
0709 12:25:42.637886 (+   598us) log.cc:844] Serialized 204 byte log entry
0709 12:25:42.638099 (+   213us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:42.638583 (+   484us) write_op.cc:454] Released schema lock
0709 12:25:42.649153 (+ 10570us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:42.658119 (+  8966us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:42.658155 (+    36us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":51648,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":121468,"apply.run_cpu_time_us":42519,"apply.run_wall_time_us":759416,"cfile_cache_hit":94,"cfile_cache_hit_bytes":100221,"num_ops":23,"prepare.queue_time_us":152857,"prepare.run_cpu_time_us":35339,"prepare.run_wall_time_us":566259,"raft.queue_time_us":280025,"raft.run_cpu_time_us":15091,"raft.run_wall_time_us":95546,"replication_time_us":1230361,"spinlock_wait_cycles":3883392,"thread_start_us":923820,"threads_started":4,"wal-append.queue_time_us":603374,"wal-append.run_cpu_time_us":105593,"wal-append.run_wall_time_us":2563996}]]}
I20260709 12:25:44.058915 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:44.059648 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:44.027527 31791 consensus_peers.cc:597] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:44.088155 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 60696.8us
I20260709 12:25:44.068523 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35c247463bef4f77990becbf6089be7d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 39 candidate_status { last_received { term: 38 index: 24 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:44.073382 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:44.077785 31717 consensus_peers.cc:597] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:44.089608 31717 consensus_peers.cc:597] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:44.078469 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:44.090579 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:44.090570 31717 consensus_peers.cc:597] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:44.091662 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:44.092511 31717 leader_election.cc:336] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.092911 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:44.094839 10423 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:44.095273 10423 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.095645 10423 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:44.099342 10470 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:44.113605 32374 raft_consensus.cc:2468] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:44.055879 31569 catalog_manager.cc:5697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 19 to 25, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 25 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:44.083684 10823 raft_consensus.cc:493] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:44.119143 31792 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:44.121735 31569 catalog_manager.cc:5697] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 17 to 20, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 20 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:44.118880 10823 raft_consensus.cc:515] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:44.124418 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:44.079232 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b; no voters: 
I20260709 12:25:44.123122 10618 raft_consensus.cc:2804] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:44.125231 10824 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:44.125535 10823 leader_election.cc:290] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:44.117055 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:44.125945 10618 raft_consensus.cc:697] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:44.126516 31746 raft_consensus.cc:2393] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 31: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:44.127102 10618 consensus_queue.cc:237] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:44.128440 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b08faba9c13b4f0f85baf8b16c97d85b" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 35 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:44.129133 10825 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:44.129573 10825 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.129914 10825 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:44.110340 10423 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:44.136140 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 56986.4us
I20260709 12:25:44.099401 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:44.137089 31718 leader_election.cc:304] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:44.137583 31717 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
W20260709 12:25:44.141968 31791 consensus_peers.cc:597] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:44.144691 10809 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election won for term 37
I20260709 12:25:44.145208 10809 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:44.145578 10809 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Advancing to term 37
I20260709 12:25:44.146466 10825 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:44.148788 10825 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:44.151590 31827 raft_consensus.cc:1275] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 31 index: 20. Preceding OpId from leader: term: 34 index: 22. (index mismatch)
I20260709 12:25:44.156797 10612 raft_consensus.cc:493] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.157435 10612 raft_consensus.cc:515] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:44.159726 10612 leader_election.cc:290] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:44.155059 10470 raft_consensus.cc:2749] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:25:44.165519 31643 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
W20260709 12:25:44.167147 31717 consensus_peers.cc:597] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:44.167819 31717 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.168009 10826 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:44.169291 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "181ddae071d549aab328e3c1d8553734" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 28 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:44.173738 10809 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:44.142397 10423 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:44.175045 31822 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
W20260709 12:25:44.150280 31791 consensus_peers.cc:597] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:44.176249 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:44.178174 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "292851e482024b3a8dcd05c5dab04534" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:44.180186 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:44.189324 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:44.191294 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:44.191644 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:44.192744 31744 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 37: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:44.194331 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f2e7a775ae0d44108be0965c85245ecf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 28 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:44.196162 31746 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer b95aeeaf4c51442b8092cff1de674f2b: Log matching property violated. Preceding OpId in replica: term: 32 index: 17. Preceding OpId from leader: term: 34 index: 18. (index mismatch)
W20260709 12:25:44.176291 31791 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.200842 31643 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:44.209005 31820 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 37: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:44.211845 31643 leader_election.cc:304] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:44.215788 31643 leader_election.cc:304] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:44.217752 10729 raft_consensus.cc:2749] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:44.220817 10811 raft_consensus.cc:2749] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:44.222368 10822 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:44.225229 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:44.226179 10604 consensus_queue.cc:1048] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:44.226583 10423 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:44.233539 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:44.244124 31825 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 37: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:25:44.238979 31791 leader_election.cc:336] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.251529 31791 leader_election.cc:304] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:44.254769 31643 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:44.260098 31718 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:44.264616 10815 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
I20260709 12:25:44.267153 10718 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:25:44.265748 10784 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.009s
W20260709 12:25:44.384799 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 111061us
I20260709 12:25:44.267719 10618 raft_consensus.cc:2749] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:25:44.400220 31643 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 145566us
W20260709 12:25:44.403827 31642 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 148547us
I20260709 12:25:44.403555 10423 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.430244 31643 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:44.431504 10423 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:44.443204 10612 raft_consensus.cc:493] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.443804 10612 raft_consensus.cc:515] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:44.444975 10809 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:44.434288 10423 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:44.449309 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:44.451789 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:44.452714 31744 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 37: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:44.406776 10782 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:44.453714 32370 raft_consensus.cc:2410] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 for term 32 because replica has last-logged OpId of term: 30 index: 21, which is greater than that of the candidate, which has last-logged OpId of term: 29 index: 20.
I20260709 12:25:44.453316 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:44.414817 10817 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.456189 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "804df7046b7545d9890885896a333a3e" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 29 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:44.459489 10817 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:44.461395 10817 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:44.464325 31643 leader_election.cc:304] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:44.464989 31718 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:44.465637 31643 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:44.467020 10815 raft_consensus.cc:2749] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:44.467743 10815 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:25:44.476146 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:44.477509 31791 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:44.478006 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:44.463967 10777 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:25:44.481845 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:44.470342 10470 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:44.489545 10809 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader election lost for term 37. Reason: could not achieve majority
I20260709 12:25:44.495617 10825 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.503312 10825 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:44.496353 10612 leader_election.cc:290] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:44.522329 10726 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.522953 10726 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:44.523509 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81c14d7d4fc14386b173a86e9623eb25" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 32 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:44.525094 10726 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:44.528591 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:44.533385 10824 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:44.532737 31643 leader_election.cc:304] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:44.537456 10825 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:44.550685 10825 raft_consensus.cc:2749] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:44.553807 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:44.560730 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:44.561589 10825 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.563515 10825 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:44.565829 10825 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:44.599486 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:44.601226 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:44.606606 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 28 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:44.615865 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:44.622323 31642 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:44.623309 10820 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:44.624778 31642 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:44.626154 10825 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:44.633522 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:44.641410 10669 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:25:44.640761 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:44.655107 32342 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:44.646497 10777 raft_consensus.cc:493] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:44.666131 10777 raft_consensus.cc:515] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:44.671806 10777 leader_election.cc:290] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:44.681665 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:44.684731 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00854b2a70f24ef3a7b43d17477a0ccf" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 30 candidate_status { last_received { term: 29 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:44.696640 31643 leader_election.cc:304] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 30 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:44.701062 10784 raft_consensus.cc:2749] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Leader pre-election lost for term 30. Reason: could not achieve majority
I20260709 12:25:44.717712 32375 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:44.721397 10618 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:25:44.740684 31746 raft_consensus.cc:1275] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 29 index: 20. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:44.768606 10618 consensus_queue.cc:1048] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 20, Time since last communication: 0.005s
W20260709 12:25:44.803859 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:44.851073 10784 raft_consensus.cc:493] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:44.851662 10784 raft_consensus.cc:515] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:44.852065 30503 thread.cc:527] Waited for 11000ms trying to join with heartbeat (tid 31703)
I20260709 12:25:44.853931 10784 leader_election.cc:290] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:44.858099 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:44.862218 32342 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:44.894959 31717 consensus_peers.cc:597] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:44.898603 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6821bb8779e64160887e3195aa5078a1" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:44.901859 31643 leader_election.cc:304] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:44.916947 10784 raft_consensus.cc:2749] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:25:45.005527 31569 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 242) took 69448 ms (client timeout 15000 ms). Trace:
W20260709 12:25:45.043059 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:45.006183 31569 rpcz_store.cc:269] 0709 12:24:35.547120 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:48.787538 (+13240418us) service_pool.cc:224] Handling call
0709 12:25:44.995207 (+56207669us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:44.271379 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:44.433840 (+162461us) spinlock_profiling.cc:243] Waited 1.32 ms on lock 0x7b5c0021fd0c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe902fc7331 00007fe902fc7221 00007fe902fc1f6c 00007fe90081fd69 00007fe90080b0a6 00007fe90080ad58 00007fe90080f9f1 00007fe90080f9a9 00007fe90080f939 00007fe90080f901 00007fe90080eb2d
0709 12:25:44.433954 (+   114us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:44.434010 (+    56us) write_op.cc:435] Acquired schema lock
0709 12:25:44.434040 (+    30us) tablet.cc:662] Decoding operations
0709 12:25:44.434836 (+   796us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:44.434952 (+   116us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:44.434991 (+    39us) tablet.cc:685] Acquiring locks for 4 operations
0709 12:25:44.442261 (+  7270us) tablet.cc:701] Row locks acquired
0709 12:25:44.442308 (+    47us) write_op.cc:260] PREPARE: finished
0709 12:25:44.442494 (+   186us) write_op.cc:270] Start()
0709 12:25:44.442772 (+   278us) write_op.cc:276] Timestamp: P: 1783599944442426 usec, L: 0
0709 12:25:44.442800 (+    28us) op_driver.cc:348] REPLICATION: starting
0709 12:25:44.487618 (+ 44818us) log.cc:844] Serialized 1718 byte log entry
0709 12:25:44.720690 (+233072us) op_driver.cc:464] REPLICATION: finished
0709 12:25:44.803083 (+ 82393us) write_op.cc:301] APPLY: starting
0709 12:25:44.803171 (+    88us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:44.806293 (+  3122us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:44.806320 (+    27us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:44.857282 (+ 50962us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:44.857503 (+   221us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=8,key_file_lookups=8,delta_file_lookups=0,mrs_lookups=0
0709 12:25:44.857546 (+    43us) write_op.cc:312] APPLY: finished
0709 12:25:44.857791 (+   245us) log.cc:844] Serialized 49 byte log entry
0709 12:25:44.857975 (+   184us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:44.858072 (+    97us) write_op.cc:454] Released schema lock
0709 12:25:44.858119 (+    47us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:44.858582 (+   463us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:44.858617 (+    35us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":41878,"reported_tablets":51,"child_traces":[["op",{"apply.queue_time_us":82141,"apply.run_cpu_time_us":7752,"apply.run_wall_time_us":55474,"cfile_cache_hit":18,"cfile_cache_hit_bytes":20839,"num_ops":4,"prepare.queue_time_us":92877,"prepare.run_cpu_time_us":34584,"prepare.run_wall_time_us":344539,"raft.queue_time_us":74945,"raft.run_cpu_time_us":32346,"raft.run_wall_time_us":66014,"replication_time_us":277708,"spinlock_wait_cycles":3995264,"thread_start_us":325503,"threads_started":4,"wal-append.queue_time_us":143809}]]}
W20260709 12:25:45.023154 31571 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 153) took 149034 ms (client timeout 15000 ms). Trace:
W20260709 12:25:45.068528 31571 rpcz_store.cc:269] 0709 12:23:15.988607 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:23:15.989042 (+   435us) service_pool.cc:224] Handling call
0709 12:25:45.022924 (+149033882us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":140772,"reported_tablets":28}
W20260709 12:25:45.130522 31573 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 150) took 214533 ms (client timeout 15000 ms). Trace:
W20260709 12:25:45.133982 31573 rpcz_store.cc:269] 0709 12:22:10.597057 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:10.597275 (+   218us) service_pool.cc:224] Handling call
0709 12:25:45.130303 (+214533028us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":22766,"reported_tablets":28,"spinlock_wait_cycles":8448}
W20260709 12:25:45.141974 31717 consensus_peers.cc:597] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:45.133438 31572 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 151) took 194083 ms (client timeout 15000 ms). Trace:
W20260709 12:25:45.151185 31572 rpcz_store.cc:269] 0709 12:22:31.050292 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:22:31.050613 (+   321us) service_pool.cc:224] Handling call
0709 12:25:45.133221 (+194082608us) inbound_call.cc:177] Queueing success response
Metrics: {"mutex_wait_us":20256,"reported_tablets":28}
I20260709 12:25:45.179756 31574 catalog_manager.cc:5697] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 24 to 26, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 26 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:45.187017 31717 consensus_peers.cc:597] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:45.192489 31574 catalog_manager.cc:5697] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 18 to 21, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 21 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
W20260709 12:25:45.196308 32342 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:45.284483 32342 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:45.332600 31569 catalog_manager.cc:5697] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 30 to 37, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 37 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:45.334283 31569 catalog_manager.cc:5697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 37 to 39, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 39 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:45.335778 31569 catalog_manager.cc:5697] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 30 to 32, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 32 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:45.337339 31569 catalog_manager.cc:5697] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 29 to 33, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 33 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.350468 31569 catalog_manager.cc:5697] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 33. New cstate: current_term: 33 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
W20260709 12:25:45.370266 32342 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:45.370750 31569 catalog_manager.cc:5697] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 25 to 34. New cstate: current_term: 34 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.372200 31569 catalog_manager.cc:5697] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 31 to 34, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 34 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:45.412130 31791 consensus_peers.cc:597] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:45.420558 31852 heartbeater.cc:646] Failed to heartbeat to 127.29.201.254:37495 (9 consecutive failures): Timed out: Failed to send heartbeat to master: TSHeartbeat RPC to 127.29.201.254:37495 timed out after 15.000s (SENT)
W20260709 12:25:45.423372 31574 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42070 (request call id 227) took 100077 ms (client timeout 15000 ms). Trace:
I20260709 12:25:45.373668 31569 catalog_manager.cc:5697] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 25 to 30, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 30 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:45.424906 31562 catalog_manager.cc:5227] ChangeConfig:REMOVE_PEER RPC for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645) due to error UNKNOWN_ERROR; will retry: Illegal state: Replica 4d48c83c761f48079bc6ea176649dda0 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 27 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:45.425865 31569 catalog_manager.cc:5697] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 34 to 38. New cstate: current_term: 38 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:45.424065 31574 rpcz_store.cc:269] 0709 12:24:05.346337 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:24:08.294412 (+2948075us) service_pool.cc:224] Handling call
0709 12:25:45.423285 (+97128873us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:45.251523 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:45.252138 (+   615us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:45.252177 (+    39us) write_op.cc:435] Acquired schema lock
0709 12:25:45.252198 (+    21us) tablet.cc:662] Decoding operations
0709 12:25:45.252617 (+   419us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:45.252704 (+    87us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:45.252732 (+    28us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:25:45.253123 (+   391us) tablet.cc:701] Row locks acquired
0709 12:25:45.253164 (+    41us) write_op.cc:260] PREPARE: finished
0709 12:25:45.253328 (+   164us) write_op.cc:270] Start()
0709 12:25:45.253463 (+   135us) write_op.cc:276] Timestamp: P: 1783599945253278 usec, L: 0
0709 12:25:45.253489 (+    26us) op_driver.cc:348] REPLICATION: starting
0709 12:25:45.256120 (+  2631us) log.cc:844] Serialized 935 byte log entry
0709 12:25:45.259041 (+  2921us) op_driver.cc:464] REPLICATION: finished
0709 12:25:45.259547 (+   506us) write_op.cc:301] APPLY: starting
0709 12:25:45.259642 (+    95us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:45.261400 (+  1758us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:45.261500 (+   100us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:45.262988 (+  1488us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:45.263200 (+   212us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:25:45.263252 (+    52us) write_op.cc:312] APPLY: finished
0709 12:25:45.263455 (+   203us) log.cc:844] Serialized 33 byte log entry
0709 12:25:45.263630 (+   175us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:45.263722 (+    92us) write_op.cc:454] Released schema lock
0709 12:25:45.263765 (+    43us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:45.264232 (+   467us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:45.264279 (+    47us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":6464,"reported_tablets":50,"spinlock_wait_cycles":7296,"child_traces":[["op",{"apply.queue_time_us":263,"apply.run_cpu_time_us":4617,"apply.run_wall_time_us":4616,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8356,"num_ops":2,"prepare.queue_time_us":31658,"prepare.run_cpu_time_us":3137,"prepare.run_wall_time_us":4955,"replication_time_us":5391,"thread_start_us":21568,"threads_started":1}]]}
I20260709 12:25:45.431466 31569 catalog_manager.cc:5697] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 35 to 37, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 37 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:45.437444 31569 catalog_manager.cc:5697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 27 to 31. New cstate: current_term: 31 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.440687 31569 catalog_manager.cc:5697] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 37. New cstate: current_term: 37 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.467151 31569 catalog_manager.cc:5697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 25 to 35. New cstate: current_term: 35 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.468650 31569 catalog_manager.cc:5697] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 34. New cstate: current_term: 34 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.469900 31569 catalog_manager.cc:5697] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 19 to 30, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 30 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:45.471199 31569 catalog_manager.cc:5697] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 34 to 35, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 35 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:45.472518 31569 catalog_manager.cc:5697] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 21 to 35. New cstate: current_term: 35 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.473764 31569 catalog_manager.cc:5697] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 37 to 40, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 40 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:45.492365 31569 catalog_manager.cc:5697] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 26 to 39. New cstate: current_term: 39 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.493746 31569 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 27 to 31. New cstate: current_term: 31 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:45.497027 31569 catalog_manager.cc:5697] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 30 to 35, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 35 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } }
I20260709 12:25:45.500649 31569 catalog_manager.cc:5697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 30, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 30 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:45.502655 31569 catalog_manager.cc:5697] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 32 to 38, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 38 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:45.509779 31569 catalog_manager.cc:5697] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 30. New cstate: current_term: 30 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:45.511253 31569 catalog_manager.cc:5697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 32 to 34. New cstate: current_term: 34 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.513295 31569 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 31 to 34, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 34 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:45.514719 31569 catalog_manager.cc:5697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 21 to 32, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 32 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.521397 31569 catalog_manager.cc:5697] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 37, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 37 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:45.522818 31569 catalog_manager.cc:5697] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 31 to 38, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 38 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:45.542889 31569 catalog_manager.cc:5697] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 30 to 35. New cstate: current_term: 35 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:45.544127 31569 catalog_manager.cc:5697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 32 to 35, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 35 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:45.545619 31569 catalog_manager.cc:5697] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 33, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 33 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.546917 31569 catalog_manager.cc:5697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 20 to 31, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 31 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.548357 31569 catalog_manager.cc:5697] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 32. New cstate: current_term: 32 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:45.549705 31569 catalog_manager.cc:5697] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 21 to 35, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 35 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:45.551290 31569 catalog_manager.cc:5697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 16 to 27, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 27 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:45.558120 31569 catalog_manager.cc:5697] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 29, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 29 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.569867 31569 catalog_manager.cc:5697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 18 to 33. New cstate: current_term: 33 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.571532 31569 catalog_manager.cc:5697] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 24 to 28. New cstate: current_term: 28 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.572999 31569 catalog_manager.cc:5697] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 29 to 37. New cstate: current_term: 37 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:45.574368 31569 catalog_manager.cc:5697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 30 to 33. New cstate: current_term: 33 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:45.588917 10669 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.589663 10669 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:45.595170 10669 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:45.596231 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:45.602258 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ef31b84f462f424fa6162b810ea8c8a9" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 31 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:45.604014 31642 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:45.614560 10726 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:25:45.686228 31791 consensus_peers.cc:597] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:45.712301 31717 consensus_peers.cc:597] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:45.725142 10726 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.725791 10726 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:45.729094 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:45.730253 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "49cc564ccd0d411fa79831efa24f87bb" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:45.733217 31642 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:45.734833 10669 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:45.740780 10726 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:45.754719 10726 raft_consensus.cc:493] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:45.756213 10669 raft_consensus.cc:493] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 5206fd0087404d2da964a5d985a04ff4)
I20260709 12:25:45.756908 10669 raft_consensus.cc:515] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:45.762921 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:45.763139 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70ada7cfaa154c8c9cc927936b46c30a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 36 candidate_status { last_received { term: 35 index: 25 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:45.764773 10669 leader_election.cc:290] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:45.766116 31643 leader_election.cc:304] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:45.768843 10669 raft_consensus.cc:2749] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:45.755352 10726 raft_consensus.cc:515] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:45.786408 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:45.792811 32374 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5dfbfce13484fc0b1feba45daaf8247" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:45.794648 31643 leader_election.cc:304] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:45.802171 10784 raft_consensus.cc:2749] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:45.801779 10726 leader_election.cc:290] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:45.821059 31791 consensus_peers.cc:597] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:45.852696 30503 thread.cc:527] Waited for 12000ms trying to join with heartbeat (tid 31703)
W20260709 12:25:45.879163 32342 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 239 similar messages]
W20260709 12:25:45.887243 32342 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:46.052699 31717 consensus_peers.cc:597] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:46.057322 31569 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42072 (request call id 249) took 1597 ms. Trace:
I20260709 12:25:46.066370 31569 rpcz_store.cc:276] 0709 12:25:44.459646 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.062220 (+602574us) service_pool.cc:224] Handling call
0709 12:25:46.057206 (+994986us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:45.615200 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:45.615808 (+   608us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:45.615852 (+    44us) write_op.cc:435] Acquired schema lock
0709 12:25:45.615873 (+    21us) tablet.cc:662] Decoding operations
0709 12:25:45.620807 (+  4934us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:45.620871 (+    64us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:45.620901 (+    30us) tablet.cc:685] Acquiring locks for 41 operations
0709 12:25:45.625970 (+  5069us) tablet.cc:701] Row locks acquired
0709 12:25:45.625991 (+    21us) write_op.cc:260] PREPARE: finished
0709 12:25:45.626164 (+   173us) write_op.cc:270] Start()
0709 12:25:45.626292 (+   128us) write_op.cc:276] Timestamp: P: 1783599945626117 usec, L: 0
0709 12:25:45.626317 (+    25us) op_driver.cc:348] REPLICATION: starting
0709 12:25:45.627156 (+   839us) log.cc:844] Serialized 16160 byte log entry
0709 12:25:45.633995 (+  6839us) op_driver.cc:464] REPLICATION: finished
0709 12:25:45.634489 (+   494us) write_op.cc:301] APPLY: starting
0709 12:25:45.634580 (+    91us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:45.717685 (+ 83105us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:45.717815 (+   130us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:45.868106 (+150291us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:45.868618 (+   512us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=82,key_file_lookups=82,delta_file_lookups=0,mrs_lookups=0
0709 12:25:45.868673 (+    55us) write_op.cc:312] APPLY: finished
0709 12:25:45.913391 (+ 44718us) log.cc:844] Serialized 348 byte log entry
0709 12:25:45.913604 (+   213us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:45.913936 (+   332us) write_op.cc:454] Released schema lock
0709 12:25:45.914025 (+    89us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:45.926420 (+ 12395us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:45.926465 (+    45us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":23496,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":276,"apply.run_cpu_time_us":59110,"apply.run_wall_time_us":291841,"cfile_cache_hit":164,"cfile_cache_hit_bytes":171298,"num_ops":41,"prepare.queue_time_us":126,"prepare.run_cpu_time_us":8198,"prepare.run_wall_time_us":12312,"replication_time_us":7539,"spinlock_wait_cycles":71552}]]}
I20260709 12:25:46.094009 31751 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37588
I20260709 12:25:46.094900 31751 raft_consensus.cc:606] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Received request to transfer leadership
I20260709 12:25:46.101583 31751 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37588
I20260709 12:25:46.102255 31751 raft_consensus.cc:606] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Received request to transfer leadership
I20260709 12:25:46.120289 31569 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 31 to 34, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 34 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
I20260709 12:25:46.128563 31569 catalog_manager.cc:5697] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 28 to 31, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 31 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:46.140091 31751 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17"
dest_uuid: "039d66a6c85e43c3936bf5017501b344"
mode: GRACEFUL
 from {username='slave'} at 127.0.0.1:37588
I20260709 12:25:46.140795 31751 raft_consensus.cc:606] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Received request to transfer leadership
I20260709 12:25:46.156445 10618 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.157078 10618 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:46.159693 10618 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:46.164611 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:46.165418 31751 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 37.
W20260709 12:25:46.165565 31791 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:46.165660 10784 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.166307 10784 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:46.166780 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:46.168032 10618 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:25:46.168519 10618 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:46.168924 10618 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:46.168299 10669 raft_consensus.cc:493] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.170392 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:46.170413 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8725620974f7465eb0371f48761a1625" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 36 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:46.170572 10669 raft_consensus.cc:515] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:46.172333 10784 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:46.174319 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:46.174569 31642 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:46.175796 10784 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:25:46.176570 10669 leader_election.cc:290] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:46.179464 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "daa53c7009fe4881956dcfb5a20febde" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 34 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:46.181452 31642 leader_election.cc:304] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:46.182375 10669 raft_consensus.cc:2749] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
W20260709 12:25:46.192919 31570 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 243) took 46519 ms (client timeout 15000 ms). Trace:
W20260709 12:25:46.193512 31570 rpcz_store.cc:269] 0709 12:24:59.793843 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:10.529403 (+10735560us) service_pool.cc:224] Handling call
0709 12:25:46.192708 (+35663305us) inbound_call.cc:177] Queueing success response
Metrics: {"reported_tablets":51,"spinlock_wait_cycles":1920}
I20260709 12:25:46.203015 10618 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:46.227139 31568 catalog_manager.cc:5697] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 29 to 33, leader changed from 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 33 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:46.234680 31717 consensus_peers.cc:597] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:46.236505 31624 auto_rebalancer.cc:893] Removing replace marker failed: Service unavailable: leader transfer in progress
W20260709 12:25:46.237030 31624 auto_rebalancer.cc:901] Could not move replica: Remote error: Service unavailable: leadership transfer for b25009db0c5a47a9ab9f6f8b2e5c1b17 already in progress
W20260709 12:25:46.237313 31624 auto_rebalancer.cc:330] scheduled replica move failed to complete: Remote error: Service unavailable: leadership transfer for b25009db0c5a47a9ab9f6f8b2e5c1b17 already in progress
I20260709 12:25:46.242660 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
W20260709 12:25:46.243057 31791 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:46.243491 31746 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:46.243795 10618 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:46.250053 31746 raft_consensus.cc:2468] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 38.
I20260709 12:25:46.251796 31791 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:46.252883 10809 raft_consensus.cc:2804] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:25:46.251643 31568 catalog_manager.cc:5697] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 27 to 30, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 30 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:46.268554 31568 catalog_manager.cc:5697] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 28 to 30, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 30 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:46.273300 31568 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 27 to 31. New cstate: current_term: 31 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:46.276149 31568 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 31 to 33. New cstate: current_term: 33 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:46.278616 10809 raft_consensus.cc:697] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 38 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:25:46.277349 31791 consensus_peers.cc:597] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:46.280655 31568 catalog_manager.cc:5697] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 26 to 29. New cstate: current_term: 29 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:46.279691 10809 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:46.297458 10784 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.298105 10784 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:46.300647 10784 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:46.302825 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:46.307004 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab8ea7b957bd41e0a2e7967cbd101f36" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 33 candidate_status { last_received { term: 30 index: 16 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:46.310276 31643 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:46.311968 10726 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:25:46.348124 10423 raft_consensus.cc:993] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: : Instructing follower 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:25:46.359968 10669 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.360643 10669 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:46.363054 10669 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:46.363845 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:46.367112 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 29 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:46.372771 31643 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:46.374768 10669 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:46.385380 10423 raft_consensus.cc:1081] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Signalling peer 4d48c83c761f48079bc6ea176649dda0 to start an election
I20260709 12:25:46.388680 31822 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17"
dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
 from {username='slave'} at 127.0.0.1:41204
I20260709 12:25:46.389510 31822 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:25:46.389879 31822 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:46.408607 31567 catalog_manager.cc:5697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 26 to 28, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 28 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:46.418300 31822 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:46.422394 10784 raft_consensus.cc:493] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:46.422673 31717 consensus_peers.cc:597] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:46.433837 10784 raft_consensus.cc:515] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } }
I20260709 12:25:46.437343 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:46.441850 31567 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 27 to 29. New cstate: current_term: 29 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:46.443549 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: true dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:46.444180 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b25009db0c5a47a9ab9f6f8b2e5c1b17" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 28 candidate_status { last_received { term: 27 index: 15 } } ignore_live_leader: true dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:46.444931 32375 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:46.445564 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
W20260709 12:25:46.445592 31791 leader_election.cc:336] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260709 12:25:46.444242 31751 raft_consensus.cc:3055] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Stepping down as leader of term 27
I20260709 12:25:46.446988 31751 raft_consensus.cc:740] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 27 LEADER]: Becoming Follower/Learner. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:46.447007 31822 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:46.449493 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f84a67c6dba04300a5240cc80c94718a" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:46.449023 31751 consensus_queue.cc:260] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 27, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:46.451138 31751 raft_consensus.cc:3060] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 27 FOLLOWER]: Advancing to term 28
I20260709 12:25:46.448592 10784 leader_election.cc:290] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:46.457082 30503 tablet_replica.cc:333] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.459277 30503 raft_consensus.cc:2243] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:46.466188 30503 raft_consensus.cc:2272] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P b95aeeaf4c51442b8092cff1de674f2b [term 27 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.477860 30503 tablet_replica.cc:333] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.488582 30503 raft_consensus.cc:2243] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Raft consensus shutting down.
W20260709 12:25:46.451102 31568 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 244) took 16034 ms (client timeout 15000 ms). Trace:
I20260709 12:25:46.460758 31642 leader_election.cc:304] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:46.489601 10669 raft_consensus.cc:2749] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:46.489341 30503 pending_rounds.cc:70] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Trying to abort 1 pending ops.
I20260709 12:25:46.496362 30503 pending_rounds.cc:77] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b: Aborting op as it isn't in flight: id { term: 33 index: 21 } timestamp: 7305625285030445056 op_type: NO_OP noop_request { }
I20260709 12:25:46.497128 30503 raft_consensus.cc:2889] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:46.497586 30503 raft_consensus.cc:2272] T 2f2f85d943e24cfdb47fe2ad3185097f P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:46.490617 31568 rpcz_store.cc:269] 0709 12:25:30.416717 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:31.714017 (+1297300us) service_pool.cc:224] Handling call
0709 12:25:31.955677 (+241660us) spinlock_profiling.cc:243] Waited 8.39 ms on lock 0x7b54002828d4. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8fcd3f090 00007fe8fcdddf3a 00007fe902b93e9c 00007fe902c9204e 00007fe9011532a4 00007fe901153232 00007fe901153161 00007fe9011530dc 00007fe901152392 00007fe8fb11048c 00007fe8fb10f8c6
0709 12:25:46.451026 (+14495349us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:46.289250 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:46.290093 (+   843us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:46.290148 (+    55us) write_op.cc:435] Acquired schema lock
0709 12:25:46.290169 (+    21us) tablet.cc:662] Decoding operations
0709 12:25:46.295362 (+  5193us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:46.295423 (+    61us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:46.295450 (+    27us) tablet.cc:685] Acquiring locks for 6 operations
0709 12:25:46.296010 (+   560us) tablet.cc:701] Row locks acquired
0709 12:25:46.296026 (+    16us) write_op.cc:260] PREPARE: finished
0709 12:25:46.296177 (+   151us) write_op.cc:270] Start()
0709 12:25:46.296288 (+   111us) write_op.cc:276] Timestamp: P: 1783599946296133 usec, L: 0
0709 12:25:46.296309 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 12:25:46.296947 (+   638us) log.cc:844] Serialized 2466 byte log entry
0709 12:25:46.301795 (+  4848us) op_driver.cc:464] REPLICATION: finished
0709 12:25:46.302754 (+   959us) write_op.cc:301] APPLY: starting
0709 12:25:46.302837 (+    83us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:46.306590 (+  3753us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:46.323334 (+ 16744us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:46.329736 (+  6402us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:46.335525 (+  5789us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=12,key_file_lookups=12,delta_file_lookups=0,mrs_lookups=0
0709 12:25:46.335589 (+    64us) write_op.cc:312] APPLY: finished
0709 12:25:46.335905 (+   316us) log.cc:844] Serialized 65 byte log entry
0709 12:25:46.336776 (+   871us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:46.336910 (+   134us) write_op.cc:454] Released schema lock
0709 12:25:46.336963 (+    53us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:46.370213 (+ 33250us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:46.370251 (+    38us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":51,"spinlock_wait_cycles":19306880,"child_traces":[["op",{"apply.queue_time_us":690,"apply.run_cpu_time_us":10100,"apply.run_wall_time_us":67374,"cfile_cache_hit":24,"cfile_cache_hit_bytes":25068,"num_ops":6,"prepare.queue_time_us":151,"prepare.run_cpu_time_us":3837,"prepare.run_wall_time_us":17877,"replication_time_us":5355}]]}
I20260709 12:25:46.502506 31751 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
W20260709 12:25:46.503063 31791 consensus_peers.cc:597] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:46.503060 31717 consensus_peers.cc:597] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:46.509861 30503 tablet_replica.cc:333] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.511140 30503 raft_consensus.cc:2243] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:25:46.512705 30503 raft_consensus.cc:2272] T f79ad11703094c998ce1b3905e5c9ff6 P b95aeeaf4c51442b8092cff1de674f2b [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.521787 30503 tablet_replica.cc:333] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.495828 32375 raft_consensus.cc:2468] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:25:46.523902 30503 raft_consensus.cc:2243] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:25:46.527921 30503 raft_consensus.cc:2272] T 436783e8e1cc49edbad793a31645e2d6 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.529135 31792 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 28 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:46.530772 10856 raft_consensus.cc:2804] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader election won for term 28
I20260709 12:25:46.531447 10856 raft_consensus.cc:697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:25:46.531814 31567 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 158) took 34426 ms (client timeout 15000 ms). Trace:
I20260709 12:25:46.532742 30503 tablet_replica.cc:333] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.534018 30503 raft_consensus.cc:2243] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:46.532498 10856 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 27.15, Last appended by leader: 15, Current term: 28, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:46.539444 31576 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 26 to 29. New cstate: current_term: 29 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:46.543262 10669 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:25:46.532306 31567 rpcz_store.cc:269] 0709 12:25:12.105203 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:26.898669 (+14793466us) service_pool.cc:224] Handling call
0709 12:25:46.492128 (+19593459us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:46.492158 (+    30us) write_op.cc:454] Released schema lock
0709 12:25:46.531708 (+ 39550us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:46.463305 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:46.463874 (+   569us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:46.463918 (+    44us) write_op.cc:435] Acquired schema lock
0709 12:25:46.463936 (+    18us) tablet.cc:662] Decoding operations
0709 12:25:46.464417 (+   481us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:46.464479 (+    62us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:46.464513 (+    34us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:25:46.464855 (+   342us) tablet.cc:701] Row locks acquired
0709 12:25:46.464870 (+    15us) write_op.cc:260] PREPARE: finished
0709 12:25:46.465028 (+   158us) write_op.cc:270] Start()
0709 12:25:46.465170 (+   142us) write_op.cc:276] Timestamp: P: 1783599946464974 usec, L: 0
0709 12:25:46.465188 (+    18us) op_driver.cc:348] REPLICATION: starting
0709 12:25:46.465869 (+   681us) log.cc:844] Serialized 925 byte log entry
0709 12:25:46.468142 (+  2273us) op_driver.cc:464] REPLICATION: finished
0709 12:25:46.469261 (+  1119us) write_op.cc:301] APPLY: starting
0709 12:25:46.469352 (+    91us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:46.470907 (+  1555us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:46.471015 (+   108us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:46.472432 (+  1417us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:46.472711 (+   279us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:25:46.472759 (+    48us) write_op.cc:312] APPLY: finished
0709 12:25:46.472981 (+   222us) log.cc:844] Serialized 33 byte log entry
0709 12:25:46.473164 (+   183us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:46.473257 (+    93us) write_op.cc:454] Released schema lock
0709 12:25:46.473300 (+    43us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":14511,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":865,"apply.run_cpu_time_us":4339,"apply.run_wall_time_us":4387,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8356,"num_ops":2,"prepare.queue_time_us":4426,"prepare.run_cpu_time_us":2896,"prepare.run_wall_time_us":2897,"replication_time_us":2794}]]}
I20260709 12:25:46.543874 10669 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:46.546170 31576 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b reported cstate change: term changed from 29 to 31, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193). New cstate: current_term: 31 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:46.548768 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:46.551088 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "992b9a77f3124e49a9bb783e08a9e856" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 37 candidate_status { last_received { term: 34 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:46.553185 31642 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:46.554095 10784 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:25:46.551724 10669 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:46.559973 30503 pending_rounds.cc:70] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Trying to abort 2 pending ops.
I20260709 12:25:46.561589 30503 pending_rounds.cc:77] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Aborting op as it isn't in flight: id { term: 28 index: 19 } timestamp: 7305625287626903552 op_type: NO_OP noop_request { }
I20260709 12:25:46.561971 30503 raft_consensus.cc:2889] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:46.562282 30503 pending_rounds.cc:77] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b: Aborting op as it isn't in flight: id { term: 26 index: 18 } timestamp: 7305625240666902528 op_type: NO_OP noop_request { }
I20260709 12:25:46.562686 30503 raft_consensus.cc:2889] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:46.563378 30503 raft_consensus.cc:2272] T 292851e482024b3a8dcd05c5dab04534 P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.589958 30503 tablet_replica.cc:333] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.613515 30503 raft_consensus.cc:2243] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:46.614144 30503 raft_consensus.cc:2272] T 8725620974f7465eb0371f48761a1625 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.638809 30503 tablet_replica.cc:333] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.644258 10726 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.653065 10726 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:46.653461 31575 catalog_manager.cc:5697] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 33 to 35. New cstate: current_term: 35 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:46.656924 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:46.659435 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b043bbe90410471da70e6b00856b5c8d" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 39 candidate_status { last_received { term: 37 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:46.660681 31575 catalog_manager.cc:5697] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 28 to 30, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 30 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:46.661398 31642 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:46.662642 10784 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:25:46.663169 10726 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:46.665733 30503 raft_consensus.cc:2243] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:25:46.667594 30503 raft_consensus.cc:2272] T c22bf85ce45841fb80e303e2746080da P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.677183 30503 tablet_replica.cc:333] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.690668 30503 raft_consensus.cc:2243] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:25:46.701771 30503 raft_consensus.cc:2272] T fc47871ef692418388113c1d6c88b262 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.697664 10726 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:46.706472 10726 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:46.709219 10726 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:46.665057 31575 catalog_manager.cc:5697] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 31 to 32, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 32 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:46.710642 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:46.712216 30503 tablet_replica.cc:333] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.712775 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3a0925b3d6e9448684fd239b27453b69" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 38 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:46.731029 31751 raft_consensus.cc:2393] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b95aeeaf4c51442b8092cff1de674f2b in current term 38: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
W20260709 12:25:46.728636 31791 consensus_peers.cc:597] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:46.727236 31576 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42070 (request call id 230) took 15095 ms (client timeout 15000 ms). Trace:
I20260709 12:25:46.732005 31642 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:25:46.733188 10726 raft_consensus.cc:3060] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:25:46.739734 31791 consensus_peers.cc:597] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:46.743355 10726 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:25:46.732285 31576 rpcz_store.cc:269] 0709 12:25:31.632305 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:38.721349 (+7089044us) service_pool.cc:224] Handling call
0709 12:25:46.727128 (+8005779us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:46.553430 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:46.554007 (+   577us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:46.554046 (+    39us) write_op.cc:435] Acquired schema lock
0709 12:25:46.554063 (+    17us) tablet.cc:662] Decoding operations
0709 12:25:46.554494 (+   431us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:46.554568 (+    74us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:46.554596 (+    28us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:25:46.555001 (+   405us) tablet.cc:701] Row locks acquired
0709 12:25:46.555022 (+    21us) write_op.cc:260] PREPARE: finished
0709 12:25:46.555190 (+   168us) write_op.cc:270] Start()
0709 12:25:46.555304 (+   114us) write_op.cc:276] Timestamp: P: 1783599946555143 usec, L: 0
0709 12:25:46.555325 (+    21us) op_driver.cc:348] REPLICATION: starting
0709 12:25:46.556485 (+  1160us) log.cc:844] Serialized 925 byte log entry
0709 12:25:46.562928 (+  6443us) op_driver.cc:464] REPLICATION: finished
0709 12:25:46.565160 (+  2232us) write_op.cc:301] APPLY: starting
0709 12:25:46.565271 (+   111us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:46.567116 (+  1845us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:46.567222 (+   106us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:46.571444 (+  4222us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:46.571695 (+   251us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:25:46.571754 (+    59us) write_op.cc:312] APPLY: finished
0709 12:25:46.572056 (+   302us) log.cc:844] Serialized 33 byte log entry
0709 12:25:46.572237 (+   181us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:46.572335 (+    98us) write_op.cc:454] Released schema lock
0709 12:25:46.572383 (+    48us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:46.589705 (+ 17322us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:46.589739 (+    34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":18570,"reported_tablets":50,"child_traces":[["op",{"apply.queue_time_us":239,"apply.run_cpu_time_us":5164,"apply.run_wall_time_us":24511,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8356,"num_ops":2,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":2993,"prepare.run_wall_time_us":3370,"replication_time_us":6200}]]}
I20260709 12:25:46.733455 30503 raft_consensus.cc:2243] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:25:46.765547 30503 raft_consensus.cc:2272] T f6404cb9c2174326b06cbbb0d27d43f2 P b95aeeaf4c51442b8092cff1de674f2b [term 39 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:46.775297 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:46.776779 30503 tablet_replica.cc:333] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.778662 31751 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 38 index: 20. (index mismatch)
I20260709 12:25:46.789513 30503 raft_consensus.cc:2243] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:25:46.790035 10809 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:46.792572 30503 raft_consensus.cc:2272] T ffab0b886efd41099d99c20339a3c4ae P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:46.793009 31791 consensus_peers.cc:597] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:46.787354 31575 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:33936 (request call id 159) took 12952 ms (client timeout 15000 ms). Trace:
W20260709 12:25:46.800237 31575 rpcz_store.cc:269] 0709 12:25:33.835247 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:44.128292 (+10293045us) service_pool.cc:224] Handling call
0709 12:25:46.787257 (+2658965us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:46.725724 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:46.726313 (+   589us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:46.726353 (+    40us) write_op.cc:435] Acquired schema lock
0709 12:25:46.726371 (+    18us) tablet.cc:662] Decoding operations
0709 12:25:46.727081 (+   710us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:46.727142 (+    61us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:46.727173 (+    31us) tablet.cc:685] Acquiring locks for 3 operations
0709 12:25:46.727541 (+   368us) tablet.cc:701] Row locks acquired
0709 12:25:46.727561 (+    20us) write_op.cc:260] PREPARE: finished
0709 12:25:46.727728 (+   167us) write_op.cc:270] Start()
0709 12:25:46.727848 (+   120us) write_op.cc:276] Timestamp: P: 1783599946727676 usec, L: 0
0709 12:25:46.727871 (+    23us) op_driver.cc:348] REPLICATION: starting
0709 12:25:46.728553 (+   682us) log.cc:844] Serialized 1311 byte log entry
0709 12:25:46.731228 (+  2675us) op_driver.cc:464] REPLICATION: finished
0709 12:25:46.732904 (+  1676us) write_op.cc:301] APPLY: starting
0709 12:25:46.733228 (+   324us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:46.735217 (+  1989us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:46.735313 (+    96us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:46.737215 (+  1902us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:46.737407 (+   192us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=6,key_file_lookups=6,delta_file_lookups=0,mrs_lookups=0
0709 12:25:46.737449 (+    42us) write_op.cc:312] APPLY: finished
0709 12:25:46.737681 (+   232us) log.cc:844] Serialized 41 byte log entry
0709 12:25:46.737841 (+   160us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:46.737941 (+   100us) write_op.cc:454] Released schema lock
0709 12:25:46.737980 (+    39us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:46.738383 (+   403us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:46.738413 (+    30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":26610,"reported_tablets":28,"child_traces":[["op",{"apply.queue_time_us":144,"apply.run_cpu_time_us":5256,"apply.run_wall_time_us":6120,"cfile_cache_hit":12,"cfile_cache_hit_bytes":12534,"num_ops":3,"prepare.queue_time_us":109,"prepare.run_cpu_time_us":3136,"prepare.run_wall_time_us":5289,"replication_time_us":3233}]]}
I20260709 12:25:46.815341 30503 tablet_replica.cc:333] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.818662 30503 raft_consensus.cc:2243] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:46.819476 30503 pending_rounds.cc:70] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Trying to abort 1 pending ops.
I20260709 12:25:46.819810 30503 pending_rounds.cc:77] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b: Aborting op as it isn't in flight: id { term: 33 index: 21 } timestamp: 7305625264797364224 op_type: NO_OP noop_request { }
I20260709 12:25:46.820181 30503 raft_consensus.cc:2889] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:46.820617 30503 raft_consensus.cc:2272] T b08faba9c13b4f0f85baf8b16c97d85b P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.822587 31574 catalog_manager.cc:5697] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 29 to 31, leader changed from 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 31 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:46.837884 31574 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 31 to 33. New cstate: current_term: 33 leader_uuid: "b95aeeaf4c51442b8092cff1de674f2b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:46.853227 30503 tablet_replica.cc:333] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.854437 30503 raft_consensus.cc:2243] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:46.855389 30503 raft_consensus.cc:2272] T 45588632b8d64f9fb9c2e2ba18b20546 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.858705 30503 tablet_replica.cc:333] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.859724 30503 raft_consensus.cc:2243] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:46.860365 30503 raft_consensus.cc:2272] T 20650b4ef2514a05bde5dfd38a379f32 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.871801 31575 catalog_manager.cc:5697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 34 to 38, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 38 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } }
W20260709 12:25:46.913383 31717 consensus_peers.cc:597] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:46.944936 30503 tablet_replica.cc:333] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.951422 30503 raft_consensus.cc:2243] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:25:46.962414 30503 raft_consensus.cc:2272] T bc0ece2345cd46edb250e06e8f5431d6 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.965548 30503 tablet_replica.cc:333] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.966840 30503 raft_consensus.cc:2243] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:25:46.968137 30503 raft_consensus.cc:2272] T a22e63b9ca824c92ae9ba374d2f3242a P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.970582 31574 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:42080 (request call id 245) took 1394 ms. Trace:
I20260709 12:25:46.973489 30503 tablet_replica.cc:333] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.971079 31574 rpcz_store.cc:276] 0709 12:25:45.576401 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:25:45.576601 (+   200us) service_pool.cc:224] Handling call
0709 12:25:46.970482 (+1393881us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:25:46.851779 (+     0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0709 12:25:46.852324 (+   545us) write_op.cc:432] Acquiring schema lock in shared mode
0709 12:25:46.852357 (+    33us) write_op.cc:435] Acquired schema lock
0709 12:25:46.852377 (+    20us) tablet.cc:662] Decoding operations
0709 12:25:46.852828 (+   451us) write_op.cc:620] Acquiring the partition lock for write op
0709 12:25:46.852892 (+    64us) write_op.cc:641] Partition lock acquired for write op
0709 12:25:46.852929 (+    37us) tablet.cc:685] Acquiring locks for 2 operations
0709 12:25:46.853263 (+   334us) tablet.cc:701] Row locks acquired
0709 12:25:46.853276 (+    13us) write_op.cc:260] PREPARE: finished
0709 12:25:46.853418 (+   142us) write_op.cc:270] Start()
0709 12:25:46.853529 (+   111us) write_op.cc:276] Timestamp: P: 1783599946853367 usec, L: 0
0709 12:25:46.853545 (+    16us) op_driver.cc:348] REPLICATION: starting
0709 12:25:46.854186 (+   641us) log.cc:844] Serialized 925 byte log entry
0709 12:25:46.856523 (+  2337us) op_driver.cc:464] REPLICATION: finished
0709 12:25:46.857076 (+   553us) write_op.cc:301] APPLY: starting
0709 12:25:46.857170 (+    94us) tablet.cc:1367] starting BulkCheckPresence
0709 12:25:46.858986 (+  1816us) tablet.cc:1370] finished BulkCheckPresence
0709 12:25:46.859103 (+   117us) tablet.cc:1372] starting ApplyRowOperation cycle
0709 12:25:46.860728 (+  1625us) tablet.cc:1383] finished ApplyRowOperation cycle
0709 12:25:46.860943 (+   215us) tablet_metrics.cc:581] ProbeStats: bloom_lookups=4,key_file_lookups=4,delta_file_lookups=0,mrs_lookups=0
0709 12:25:46.860989 (+    46us) write_op.cc:312] APPLY: finished
0709 12:25:46.861234 (+   245us) log.cc:844] Serialized 33 byte log entry
0709 12:25:46.861448 (+   214us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:46.862474 (+  1026us) write_op.cc:454] Released schema lock
0709 12:25:46.862527 (+    53us) write_op.cc:341] FINISH: Updating metrics
0709 12:25:46.862996 (+   469us) write_op.cc:489] Releasing partition, row and schema locks
0709 12:25:46.863034 (+    38us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":51,"spinlock_wait_cycles":4352,"child_traces":[["op",{"apply.queue_time_us":302,"apply.run_cpu_time_us":4991,"apply.run_wall_time_us":5877,"cfile_cache_hit":8,"cfile_cache_hit_bytes":8356,"num_ops":2,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":2690,"prepare.run_wall_time_us":2689,"replication_time_us":2816}]]}
I20260709 12:25:46.974821 30503 raft_consensus.cc:2243] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:46.975515 30503 raft_consensus.cc:2272] T 181ddae071d549aab328e3c1d8553734 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.989902 30503 tablet_replica.cc:333] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:46.991282 30503 raft_consensus.cc:2243] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:46.991976 30503 raft_consensus.cc:2272] T 6821bb8779e64160887e3195aa5078a1 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:46.995208 30503 tablet_replica.cc:333] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
W20260709 12:25:47.003149 32342 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:47.004797 30503 raft_consensus.cc:2243] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.005805 30503 raft_consensus.cc:2272] T 00854b2a70f24ef3a7b43d17477a0ccf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.003929 31574 catalog_manager.cc:5697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 27 to 28, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 28 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:47.038074 30503 tablet_replica.cc:333] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.041558 30503 raft_consensus.cc:2243] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.048487 30503 raft_consensus.cc:2272] T f84a67c6dba04300a5240cc80c94718a P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.053766 30503 tablet_replica.cc:333] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.062779 30503 raft_consensus.cc:2243] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.063576 30503 raft_consensus.cc:2272] T 812ed9ab41214308b368b53417e6e9f3 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.065418 31746 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:25:47.070578 30503 tablet_replica.cc:333] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.071759 30503 raft_consensus.cc:2243] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.072356 10809 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
I20260709 12:25:47.091995 30503 raft_consensus.cc:2272] T bc55086b81374990afb970c972568882 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.105072 30503 tablet_replica.cc:333] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.106189 30503 raft_consensus.cc:2243] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 30 LEADER]: Raft consensus shutting down.
W20260709 12:25:47.106349 32342 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:47.108191 31791 consensus_peers.cc:597] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:47.107532 30503 raft_consensus.cc:2272] T 6e7ad549735d4871af75658a91bb98c4 P b95aeeaf4c51442b8092cff1de674f2b [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.125128 30503 tablet_replica.cc:333] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.126308 30503 raft_consensus.cc:2243] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.132294 30503 pending_rounds.cc:70] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Trying to abort 1 pending ops.
I20260709 12:25:47.137310 30503 pending_rounds.cc:77] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b: Aborting op as it isn't in flight: id { term: 32 index: 19 } timestamp: 7305625250535174144 op_type: NO_OP noop_request { }
I20260709 12:25:47.141615 30503 raft_consensus.cc:2889] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:47.142194 30503 raft_consensus.cc:2272] T 7523cff6f9fb44b5ba8fc04726d85b65 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.161479 10784 raft_consensus.cc:493] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:47.162226 10784 raft_consensus.cc:515] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:47.171123 30503 tablet_replica.cc:333] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.171563 32375 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 27 index: 15. Preceding OpId from leader: term: 28 index: 16. (index mismatch)
I20260709 12:25:47.172228 30503 raft_consensus.cc:2243] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.173498 30503 raft_consensus.cc:2272] T b416bf5304744b14b72501e697b8c951 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.173718 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:47.176030 10856 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
I20260709 12:25:47.177268 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "11075d6cbd1a40ccaf381a4eb70cbf24" candidate_uuid: "b95aeeaf4c51442b8092cff1de674f2b" candidate_term: 32 candidate_status { last_received { term: 30 index: 18 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:47.178833 30503 tablet_replica.cc:333] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.179194 31643 leader_election.cc:304] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b95aeeaf4c51442b8092cff1de674f2b; no voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:47.187877 10726 raft_consensus.cc:2749] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:47.179720 10784 leader_election.cc:290] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:47.190557 30503 raft_consensus.cc:2243] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.195942 30503 raft_consensus.cc:2272] T 354c519f070b453dadcdef3f0d835c69 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.224195 30503 tablet_replica.cc:333] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.233320 30503 raft_consensus.cc:2243] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.234705 30503 raft_consensus.cc:2272] T d0a3795ad4d543f0998c3834aac7be45 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.240311 30503 tablet_replica.cc:333] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.242435 30503 raft_consensus.cc:2243] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.243211 30503 raft_consensus.cc:2272] T daa53c7009fe4881956dcfb5a20febde P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.251030 30503 tablet_replica.cc:333] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.252168 30503 raft_consensus.cc:2243] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.252867 30503 pending_rounds.cc:70] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Trying to abort 2 pending ops.
I20260709 12:25:47.253211 30503 pending_rounds.cc:77] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Aborting op as it isn't in flight: id { term: 34 index: 20 } timestamp: 7305625332091760640 op_type: NO_OP noop_request { }
I20260709 12:25:47.253636 30503 raft_consensus.cc:2889] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:47.254005 30503 pending_rounds.cc:77] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b: Aborting op as it isn't in flight: id { term: 32 index: 19 } timestamp: 7305625292274483200 op_type: NO_OP noop_request { }
I20260709 12:25:47.254383 30503 raft_consensus.cc:2889] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:47.254801 30503 raft_consensus.cc:2272] T 992b9a77f3124e49a9bb783e08a9e856 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.258322 30503 tablet_replica.cc:333] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.259244 30503 raft_consensus.cc:2243] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.260354 30503 raft_consensus.cc:2272] T e59926725a06436e9899dbbbcae3f622 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.273497 30503 tablet_replica.cc:333] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.274554 30503 raft_consensus.cc:2243] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.275302 30503 pending_rounds.cc:70] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Trying to abort 1 pending ops.
I20260709 12:25:47.275664 30503 pending_rounds.cc:77] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b: Aborting op as it isn't in flight: id { term: 31 index: 18 } timestamp: 7305625286680899584 op_type: NO_OP noop_request { }
I20260709 12:25:47.276103 30503 raft_consensus.cc:2889] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:47.276532 30503 raft_consensus.cc:2272] T 49cc564ccd0d411fa79831efa24f87bb P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.284632 30503 tablet_replica.cc:333] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.285621 30503 raft_consensus.cc:2243] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.286834 30503 raft_consensus.cc:2272] T 6499e78dd87c4e05be32f5681957a6b0 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.290059 30503 tablet_replica.cc:333] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.291002 30503 raft_consensus.cc:2243] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.291615 30503 pending_rounds.cc:70] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Trying to abort 1 pending ops.
I20260709 12:25:47.291901 30503 pending_rounds.cc:77] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b: Aborting op as it isn't in flight: id { term: 30 index: 19 } timestamp: 7305625280934371328 op_type: NO_OP noop_request { }
I20260709 12:25:47.292331 30503 raft_consensus.cc:2889] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:47.292773 30503 raft_consensus.cc:2272] T a5dfbfce13484fc0b1feba45daaf8247 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.295964 30503 tablet_replica.cc:333] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.296877 30503 raft_consensus.cc:2243] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.297613 30503 raft_consensus.cc:2272] T 11075d6cbd1a40ccaf381a4eb70cbf24 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.300773 30503 tablet_replica.cc:333] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.301669 30503 raft_consensus.cc:2243] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.302516 30503 raft_consensus.cc:2272] T b2c281519748458cb11517e3303d0a17 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.305925 30503 tablet_replica.cc:333] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.306906 30503 raft_consensus.cc:2243] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.307570 30503 raft_consensus.cc:2272] T f2e7a775ae0d44108be0965c85245ecf P b95aeeaf4c51442b8092cff1de674f2b [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.310920 30503 tablet_replica.cc:333] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.311786 30503 raft_consensus.cc:2243] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.312662 30503 raft_consensus.cc:2272] T 81c14d7d4fc14386b173a86e9623eb25 P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.316380 30503 tablet_replica.cc:333] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.317243 30503 raft_consensus.cc:2243] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.318476 30503 raft_consensus.cc:2272] T 9c849da11b414111b8461d6ae06c15b3 P b95aeeaf4c51442b8092cff1de674f2b [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.321776 30503 tablet_replica.cc:333] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.322947 30503 raft_consensus.cc:2243] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 39 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.324169 30503 raft_consensus.cc:2272] T 504cb3d29e9347fe98f010a3707a55cc P b95aeeaf4c51442b8092cff1de674f2b [term 39 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.327289 30503 tablet_replica.cc:333] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.328256 30503 raft_consensus.cc:2243] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.329450 30503 raft_consensus.cc:2272] T 5d0d0402170d41b79dc30f93bee98766 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:47.329697 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:47.333702 30503 tablet_replica.cc:333] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.334751 30503 raft_consensus.cc:2243] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.335749 30503 pending_rounds.cc:70] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Trying to abort 2 pending ops.
I20260709 12:25:47.336071 30503 pending_rounds.cc:77] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Aborting op as it isn't in flight: id { term: 35 index: 25 } timestamp: 7305625285144858624 op_type: NO_OP noop_request { }
I20260709 12:25:47.336495 30503 raft_consensus.cc:2889] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:47.336863 30503 pending_rounds.cc:77] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b: Aborting op as it isn't in flight: id { term: 33 index: 24 } timestamp: 7305625245618696192 op_type: NO_OP noop_request { }
I20260709 12:25:47.337332 30503 raft_consensus.cc:2889] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:47.339133 30503 raft_consensus.cc:2272] T 70ada7cfaa154c8c9cc927936b46c30a P b95aeeaf4c51442b8092cff1de674f2b [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.342468 30503 tablet_replica.cc:333] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.343475 30503 raft_consensus.cc:2243] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.344640 30503 raft_consensus.cc:2272] T 55f4b2e07ada4e06a285803fcbfd6344 P b95aeeaf4c51442b8092cff1de674f2b [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.347594 30503 tablet_replica.cc:333] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.348374 30503 raft_consensus.cc:2243] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.348930 30503 raft_consensus.cc:2272] T 3a0925b3d6e9448684fd239b27453b69 P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.351739 30503 tablet_replica.cc:333] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.352610 30503 raft_consensus.cc:2243] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.353842 30503 raft_consensus.cc:2272] T 5bb2dc5276714f75b6e3e61ca9275f84 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.357241 30503 tablet_replica.cc:333] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.358098 30503 raft_consensus.cc:2243] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.359220 30503 raft_consensus.cc:2272] T 553d0b2dddba47779b0219c71ed9ce83 P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.362087 30503 tablet_replica.cc:333] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.362910 30503 raft_consensus.cc:2243] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.363538 30503 raft_consensus.cc:2272] T ab8ea7b957bd41e0a2e7967cbd101f36 P b95aeeaf4c51442b8092cff1de674f2b [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.366618 30503 tablet_replica.cc:333] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.367482 30503 raft_consensus.cc:2243] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.368636 30503 raft_consensus.cc:2272] T f933d4319cf444afb9f67bd265576a31 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.371917 30503 tablet_replica.cc:333] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.372762 30503 raft_consensus.cc:2243] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.373481 30503 raft_consensus.cc:2272] T ef31b84f462f424fa6162b810ea8c8a9 P b95aeeaf4c51442b8092cff1de674f2b [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.376789 30503 tablet_replica.cc:333] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.377614 30503 raft_consensus.cc:2243] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.378180 30503 pending_rounds.cc:70] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Trying to abort 1 pending ops.
I20260709 12:25:47.378505 30503 pending_rounds.cc:77] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b: Aborting op as it isn't in flight: id { term: 29 index: 17 } timestamp: 7305625301116088320 op_type: NO_OP noop_request { }
I20260709 12:25:47.378937 30503 raft_consensus.cc:2889] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:25:47.379418 30503 raft_consensus.cc:2272] T d11aa4e638cd4a1dad543baff3b3f965 P b95aeeaf4c51442b8092cff1de674f2b [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.382181 30503 tablet_replica.cc:333] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.383015 30503 raft_consensus.cc:2243] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:25:47.384755 30503 raft_consensus.cc:2272] T 76c4ab1f56e44ff7b6adf7cfe850e01d P b95aeeaf4c51442b8092cff1de674f2b [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:47.387797 30503 tablet_replica.cc:333] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b: stopping tablet replica
I20260709 12:25:47.388612 30503 raft_consensus.cc:2243] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:47.389424 30503 raft_consensus.cc:2272] T b043bbe90410471da70e6b00856b5c8d P b95aeeaf4c51442b8092cff1de674f2b [term 38 FOLLOWER]: Raft consensus is shut down!
W20260709 12:25:47.493870 31717 consensus_peers.cc:597] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:47.527191 31624 auto_rebalancer.cc:629] Failed to schedule move for tablet f79ad11703094c998ce1b3905e5c9ff6: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:47.547829 10423 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:47.551149 10423 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:47.556571 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:47.558823 10423 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:25:47.560437 31624 auto_rebalancer.cc:629] Failed to schedule move for tablet 504cb3d29e9347fe98f010a3707a55cc: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:47.579481 31717 consensus_peers.cc:597] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:47.580134 31717 leader_election.cc:336] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:47.580641 31717 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:47.581627 10423 raft_consensus.cc:2749] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:47.610460 10423 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:47.611039 10423 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:47.615594 10423 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:47.618570 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:47.623632 31717 leader_election.cc:336] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:47.624354 31717 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:47.626420 10423 raft_consensus.cc:2749] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:47.652431 31820 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 38.20, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
W20260709 12:25:47.672648 31792 consensus_peers.cc:597] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a0925b3d6e9448684fd239b27453b69. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:47.682189 32342 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:47.682947 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:47.703562 32342 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:47.705313 31717 consensus_peers.cc:597] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:47.711737 10423 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:47.712489 10423 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:47.715708 10423 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:47.716379 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:47.725854 31744 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 38 index: 20. Preceding OpId from leader: term: 38 index: 21. (index mismatch)
W20260709 12:25:47.735320 31717 leader_election.cc:336] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:47.736115 10866 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:47.737498 10866 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:47.745630 32342 leader_election.cc:336] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:47.747102 10866 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:47.749209 31717 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:47.752830 10856 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.000s
I20260709 12:25:47.757786 10423 raft_consensus.cc:2749] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:25:47.780023 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:47.780715 31751 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
W20260709 12:25:47.786746 31791 consensus_peers.cc:597] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:47.793618 10809 raft_consensus.cc:2955] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 38 LEADER]: Committing config change with OpId 38.21: config changed from index -1 to 21, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:25:47.798666 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:47.799674 10819 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:47.800076 10819 raft_consensus.cc:493] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:47.800414 10819 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:47.808369 10819 raft_consensus.cc:515] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:47.811661 10819 leader_election.cc:290] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:47.811789 31744 raft_consensus.cc:2955] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Committing config change with OpId 38.21: config changed from index -1 to 21, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
W20260709 12:25:47.819024 32342 leader_election.cc:336] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:47.819694 31751 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:25:47.827351 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c22bf85ce45841fb80e303e2746080da" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 32 candidate_status { last_received { term: 31 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:47.828112 31746 raft_consensus.cc:3060] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:47.837646 31746 raft_consensus.cc:2468] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:25:47.838023 31574 catalog_manager.cc:5697] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 21, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 38 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:25:47.848608 32342 leader_election.cc:304] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:47.849857 10866 raft_consensus.cc:2804] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:47.851567 10866 raft_consensus.cc:697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:47.852842 10866 consensus_queue.cc:237] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 21, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:47.862825 10856 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:47.863684 10856 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:47.887207 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 44806.2us
I20260709 12:25:47.896754 10856 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:47.902118 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:47.903090 31746 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 28.
I20260709 12:25:47.904027 31575 catalog_manager.cc:5697] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 31 to 32, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 32 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:47.906118 32342 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:47.908668 31791 consensus_peers.cc:597] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:47.916169 31820 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:25:47.917230 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:47.918512 10880 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election won for term 29
I20260709 12:25:47.919090 10880 raft_consensus.cc:493] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:47.919461 10860 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:47.925576 31717 consensus_peers.cc:597] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:47.925910 10860 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:47.928917 10860 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:47.929701 10860 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:47.924999 31791 leader_election.cc:336] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:47.929745 10880 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:47.930451 10860 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:47.935235 10860 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:47.935566 10856 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:47.936287 10856 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:47.935974 31717 leader_election.cc:336] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:47.940328 10856 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:47.940558 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:25:47.916052 31718 consensus_peers.cc:597] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20650b4ef2514a05bde5dfd38a379f32. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:47.944619 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:47.950440 31791 leader_election.cc:336] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:47.951104 31791 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:47.950017 10880 raft_consensus.cc:515] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:47.953752 10880 leader_election.cc:290] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:47.954401 10856 raft_consensus.cc:2749] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:47.960299 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f79ad11703094c998ce1b3905e5c9ff6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 29 candidate_status { last_received { term: 28 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:47.961091 31744 raft_consensus.cc:3060] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Advancing to term 29
I20260709 12:25:47.967795 10883 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:47.969137 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:47.970172 31718 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:47.970610 31744 raft_consensus.cc:2468] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 29.
W20260709 12:25:47.977775 31791 leader_election.cc:336] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:47.979106 31717 leader_election.cc:336] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:47.979239 31791 leader_election.cc:304] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:47.980767 10880 raft_consensus.cc:2804] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Leader election won for term 29
I20260709 12:25:47.981603 10880 raft_consensus.cc:697] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:47.982648 10880 consensus_queue.cc:237] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 28.18, Last appended by leader: 18, Current term: 29, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:47.986501 10819 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:47.986878 10883 raft_consensus.cc:2749] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:47.987224 10819 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:47.989392 31718 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:47.994589 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:47.995394 31820 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 33.
I20260709 12:25:47.996858 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:47.997766 10866 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:47.998171 10866 raft_consensus.cc:493] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:47.998555 10866 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:47.994858 10819 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:47.999491 10883 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:25:47.999624 32342 leader_election.cc:336] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.013995 10819 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.014708 10819 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.017313 10819 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.017925 10819 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:48.023330 32342 leader_election.cc:336] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.027597 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.028752 31744 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 37.
I20260709 12:25:48.030602 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.018478 10819 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.044122 10819 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.044759 10819 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:25:48.045117 10819 raft_consensus.cc:493] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.045500 10819 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Advancing to term 38
W20260709 12:25:48.051447 32342 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.035401 10866 raft_consensus.cc:515] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.056998 10866 leader_election.cc:290] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:48.054610 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:48.062844 32342 leader_election.cc:336] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.063874 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.065846 10866 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:25:48.073099 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc47871ef692418388113c1d6c88b262" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:48.073941 31820 raft_consensus.cc:3060] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:48.078088 10880 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.078826 10880 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.081377 10880 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.086596 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.087436 31744 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:48.093580 31820 raft_consensus.cc:2468] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:48.100795 10856 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.101465 10856 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.103536 32343 leader_election.cc:304] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.105222 10866 raft_consensus.cc:2804] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:48.105868 10866 raft_consensus.cc:697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:48.068905 10819 raft_consensus.cc:515] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.107239 10856 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.106886 10866 consensus_queue.cc:237] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:48.111847 31791 leader_election.cc:336] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.112517 10819 leader_election.cc:290] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.115430 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "436783e8e1cc49edbad793a31645e2d6" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 38 candidate_status { last_received { term: 37 index: 23 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:48.115665 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.115718 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.116137 31744 raft_consensus.cc:3060] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:48.116289 31751 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:48.120227 10882 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.120960 10882 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.123838 10882 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:48.123219 10860 raft_consensus.cc:2955] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Committing config change with OpId 30.19: config changed from index -1 to 19, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:25:48.129302 10856 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:48.129894 10856 raft_consensus.cc:493] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.130336 10856 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:48.133394 31744 raft_consensus.cc:2468] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 38.
I20260709 12:25:48.135320 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:48.141017 32370 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 39.
W20260709 12:25:48.138154 31717 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:48.133246 32342 leader_election.cc:336] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.097249 31574 catalog_manager.cc:5697] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 28 to 29, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 29 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.147625 31820 raft_consensus.cc:2955] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Committing config change with OpId 30.19: config changed from index -1 to 19, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:25:48.151203 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.153208 10877 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.153833 10877 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.156216 10877 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.159228 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:48.159924 32342 leader_election.cc:304] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.160274 10882 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:25:48.169770 10882 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.169296 31576 catalog_manager.cc:5697] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 33 to 34, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 34 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.170271 10882 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:25:48.178797 10889 raft_consensus.cc:2804] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:25:48.183332 10889 raft_consensus.cc:697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 38 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:48.185020 10889 consensus_queue.cc:237] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 23, Committed index: 23, Last appended: 37.23, Last appended by leader: 23, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.192377 10856 raft_consensus.cc:515] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.196528 10877 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:48.200661 10877 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.201174 10877 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:48.202696 10856 leader_election.cc:290] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.205070 10882 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 40 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.213325 10877 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:48.217324 31717 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.227628 10882 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:48.229346 31574 catalog_manager.cc:5697] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index -1 to 19, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 30 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:25:48.225744 31576 catalog_manager.cc:5697] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 37 to 38, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 38 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.237108 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.253396 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f6404cb9c2174326b06cbbb0d27d43f2" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 40 candidate_status { last_received { term: 39 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:48.254048 32370 raft_consensus.cc:3060] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 39 FOLLOWER]: Advancing to term 40
W20260709 12:25:48.258343 31791 leader_election.cc:336] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.265743 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ffab0b886efd41099d99c20339a3c4ae" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:48.266480 31744 raft_consensus.cc:3060] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:48.273880 10819 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.274499 10819 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.276762 10819 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.277477 31744 raft_consensus.cc:2468] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:48.279450 10893 ts_tablet_manager.cc:933] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:48.282536 32342 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:48.286059 32342 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.288242 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.293998 10877 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:48.297428 31717 consensus_peers.cc:597] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:48.299557 31746 raft_consensus.cc:1275] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 32 index: 22. (index mismatch)
I20260709 12:25:48.300575 32370 raft_consensus.cc:2468] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 40.
I20260709 12:25:48.299568 31791 leader_election.cc:304] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:48.302970 10856 raft_consensus.cc:2804] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:48.303544 10856 raft_consensus.cc:697] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 34 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:48.304531 10856 consensus_queue.cc:237] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.314579 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:48.315285 31744 raft_consensus.cc:3060] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:25:48.318075 31791 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.318591 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.319267 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.320163 10819 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:48.320786 10809 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:48.320855 10889 consensus_queue.cc:1048] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:25:48.324157 31744 raft_consensus.cc:2468] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 35.
W20260709 12:25:48.322124 31791 leader_election.cc:336] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.331434 31718 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.334509 31791 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:48.340775 10880 raft_consensus.cc:2804] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:48.341403 10880 raft_consensus.cc:697] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:48.342336 10880 consensus_queue.cc:237] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:48.346850 31791 consensus_peers.cc:597] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:48.347484 31791 leader_election.cc:336] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.368966 10809 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.369895 10809 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.372423 10809 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.364382 10860 raft_consensus.cc:2804] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:25:48.390033 10809 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.387678 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.383255 10882 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.390746 10809 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:48.401129 31791 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.397095 10876 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.401732 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.402743 10877 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:48.402393 10876 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.404470 10882 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.405068 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.406872 10882 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:48.401211 10809 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.408303 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:48.412860 10876 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.417073 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.395176 10860 raft_consensus.cc:697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 40 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:48.422427 31717 leader_election.cc:336] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.424008 10882 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.428228 31717 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.428571 10882 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.429080 10883 raft_consensus.cc:2749] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Leader pre-election lost for term 40. Reason: could not achieve majority
I20260709 12:25:48.429726 10423 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:48.430033 31791 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.430298 10423 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.430512 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.432507 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:48.435240 10860 consensus_queue.cc:237] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 39.22, Last appended by leader: 22, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.436195 10423 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:48.437883 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:48.440081 10882 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:48.443306 31791 consensus_peers.cc:597] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:48.447257 10883 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.447880 10883 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.450026 10883 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:48.451597 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:48.453567 31575 catalog_manager.cc:5697] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 33 to 34, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 34 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.443037 10876 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:25:48.443320 10893 tablet_copy_client.cc:323] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.195:44645
I20260709 12:25:48.464824 31837 tablet_copy_service.cc:140] P 4d48c83c761f48079bc6ea176649dda0: Received BeginTabletCopySession request for tablet 3a0925b3d6e9448684fd239b27453b69 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34642)
I20260709 12:25:48.465471 31837 tablet_copy_service.cc:161] P 4d48c83c761f48079bc6ea176649dda0: Beginning new tablet copy session on tablet 3a0925b3d6e9448684fd239b27453b69 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34642: session id = 5206fd0087404d2da964a5d985a04ff4-3a0925b3d6e9448684fd239b27453b69
W20260709 12:25:48.469129 31717 leader_election.cc:336] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.469666 31717 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.479213 10883 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:48.481401 31791 leader_election.cc:336] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:48.487836 31717 leader_election.cc:336] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.490042 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.490540 31717 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.491509 10423 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:25:48.492609 31717 leader_election.cc:336] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.493175 31717 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.493978 10423 raft_consensus.cc:2749] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
I20260709 12:25:48.494468 10900 ts_tablet_manager.cc:933] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.496368 31837 tablet_copy_source_session.cc:215] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:25:48.493472 10876 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:48.490005 10882 raft_consensus.cc:2749] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:25:48.491405 10883 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.508981 10893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a0925b3d6e9448684fd239b27453b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:48.509570 10883 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:48.516350 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:48.517076 32370 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
I20260709 12:25:48.520255 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:48.521347 10423 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:48.521714 10423 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.522034 10423 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:25:48.526755 32342 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:48.528774 10882 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.529595 10882 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.532759 10882 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:48.544023 10876 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.544669 10876 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.545882 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:48.546669 10876 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.548486 31575 catalog_manager.cc:5697] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 34 to 35, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 35 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:48.552616 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.553516 31746 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
W20260709 12:25:48.560220 30503 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 31699)
I20260709 12:25:48.555408 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:48.576759 10423 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.580788 10856 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:48.586423 10856 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.586885 10856 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:48.581789 10423 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:48.596120 31746 raft_consensus.cc:1275] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 28 index: 18. Preceding OpId from leader: term: 29 index: 19. (index mismatch)
I20260709 12:25:48.566794 10860 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.601158 10890 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.601460 10860 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.601814 10890 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.603785 10860 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:48.603968 10890 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:48.613415 31791 consensus_peers.cc:597] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:48.614212 31791 consensus_peers.cc:597] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:48.614809 31791 leader_election.cc:336] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:48.619038 32342 leader_election.cc:336] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.624544 10423 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.625159 10423 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.627168 10809 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.628041 10809 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.627403 10882 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.628611 10883 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.629659 10423 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:48.632038 31575 catalog_manager.cc:5697] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 39 to 40, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 40 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:48.630782 10877 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.635953 10880 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.642333 10880 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.644694 10880 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.642251 10877 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.649145 10809 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.650382 10877 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.651177 10860 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.651768 10860 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.654214 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.657269 31744 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 30.
I20260709 12:25:48.659724 10860 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:48.636147 10882 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.663702 10903 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.664283 10903 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.665544 10882 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:48.667320 10880 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.671741 10877 consensus_queue.cc:1048] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20260709 12:25:48.673650 32342 consensus_peers.cc:597] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:48.675294 31717 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:48.678220 31717 leader_election.cc:336] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.678676 31717 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.678803 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "354c519f070b453dadcdef3f0d835c69" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 24 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:25:48.679351 32370 raft_consensus.cc:3060] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:48.676991 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.666476 10903 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:48.681645 10819 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:25:48.683380 10819 raft_consensus.cc:493] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.683790 10819 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:48.681942 10423 raft_consensus.cc:2749] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:25:48.693819 32370 raft_consensus.cc:2468] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 36.
I20260709 12:25:48.694347 10893 tablet_copy_client.cc:806] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:25:48.662760 10883 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.736814 10893 tablet_copy_client.cc:670] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:25:48.738217 10883 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:48.667865 10880 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.740588 10880 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.700021 31718 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:48.700627 10876 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.741514 10876 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:48.742733 31717 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.744016 10876 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.707139 10900 tablet_copy_client.cc:323] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.194:46639
I20260709 12:25:48.707892 10890 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:48.745493 31717 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.745646 10890 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.748008 10890 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.712499 10809 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 39 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.748688 10809 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 39 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.749061 10882 raft_consensus.cc:2804] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:48.749701 10882 raft_consensus.cc:697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:48.750828 10809 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 40 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.716293 10856 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.753167 10882 consensus_queue.cc:237] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 24, Committed index: 24, Last appended: 35.24, Last appended by leader: 24, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.735872 10819 raft_consensus.cc:515] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:48.746178 31717 consensus_peers.cc:597] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:48.733422 10860 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:48.758476 31717 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.758802 10860 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.761059 10819 leader_election.cc:290] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:48.759337 31717 leader_election.cc:336] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:48.733403 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 61054.1us
I20260709 12:25:48.763676 32370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:25:48.764501 32370 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:48.765795 31746 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.768218 31746 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:48.767416 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.770047 31762 tablet_copy_service.cc:140] P 039d66a6c85e43c3936bf5017501b344: Received BeginTabletCopySession request for tablet 20650b4ef2514a05bde5dfd38a379f32 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:57176)
I20260709 12:25:48.770419 10860 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:48.770962 31762 tablet_copy_service.cc:161] P 039d66a6c85e43c3936bf5017501b344: Beginning new tablet copy session on tablet 20650b4ef2514a05bde5dfd38a379f32 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:57176: session id = 5206fd0087404d2da964a5d985a04ff4-20650b4ef2514a05bde5dfd38a379f32
W20260709 12:25:48.772341 32342 leader_election.cc:336] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.762669 10856 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:48.775377 31717 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:48.775964 31717 leader_election.cc:336] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.776316 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
W20260709 12:25:48.776798 32342 leader_election.cc:336] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.777345 10860 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.777878 10860 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.777364 31751 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 34.
I20260709 12:25:48.781771 10860 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:48.785631 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:48.784256 10883 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.787837 10883 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.790329 10423 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:48.790997 10423 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.791465 10423 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:48.792840 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.795060 31751 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 37.
I20260709 12:25:48.795318 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6e7ad549735d4871af75658a91bb98c4" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:48.796016 31753 raft_consensus.cc:3060] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:48.798277 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.810685 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.811450 31751 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 33.
I20260709 12:25:48.815819 10423 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:48.912734 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:48.938882 31825 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
W20260709 12:25:48.834388 31717 leader_election.cc:336] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:48.862356 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:48.939934 32375 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
W20260709 12:25:48.940320 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 78088.4us
I20260709 12:25:48.941016 10423 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:25:48.843665 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:48.840036 31753 raft_consensus.cc:2468] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 31.
I20260709 12:25:48.818611 31751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.943620 31751 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:48.823019 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:48.944708 31744 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 39.
I20260709 12:25:48.944849 10856 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:48.945358 10856 raft_consensus.cc:493] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.945740 10856 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:48.817580 10883 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:48.893082 31762 tablet_copy_source_session.cc:215] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: Tablet Copy: opened 0 blocks and 1 log segments
W20260709 12:25:48.940054 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 105735us
W20260709 12:25:48.949137 31717 consensus_peers.cc:597] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:48.949568 31717 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.951819 31751 raft_consensus.cc:1275] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 37 index: 23. Preceding OpId from leader: term: 38 index: 24. (index mismatch)
I20260709 12:25:48.959590 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:48.960464 31824 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:48.970054 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:48.971849 31825 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:48.972571 31825 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:25:48.974777 31821 raft_consensus.cc:1275] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:25:48.838847 31748 raft_consensus.cc:1275] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:25:48.887637 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:48.977869 31822 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 31.
I20260709 12:25:48.829308 31745 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc0ece2345cd46edb250e06e8f5431d6" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 20 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:48.979156 31745 raft_consensus.cc:3060] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Advancing to term 31
I20260709 12:25:48.830001 10889 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:48.865252 31575 catalog_manager.cc:5697] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 35 to 36, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 36 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:48.982956 32342 leader_election.cc:304] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:48.935343 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:48.983913 31820 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
I20260709 12:25:48.936194 31827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:48.817392 10809 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.899751 31826 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:48.983122 10889 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.984905 31827 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:25:48.985047 10809 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.985188 10889 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Advancing to term 35
W20260709 12:25:48.943008 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 99438us
I20260709 12:25:48.985241 31826 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:25:48.987214 10809 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:48.987996 31717 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.988106 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
W20260709 12:25:48.988978 31717 leader_election.cc:336] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:48.989080 10809 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election won for term 38
I20260709 12:25:48.989464 10809 raft_consensus.cc:493] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.990386 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:48.990983 10809 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:48.992027 10880 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:48.992543 10880 raft_consensus.cc:493] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:48.992986 10880 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:48.995316 31824 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
I20260709 12:25:48.995270 10856 raft_consensus.cc:515] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:48.997781 10856 leader_election.cc:290] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:48.998765 10890 raft_consensus.cc:2804] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:25:48.999452 10890 raft_consensus.cc:697] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:25:49.000396 10900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20650b4ef2514a05bde5dfd38a379f32. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:49.000593 10890 consensus_queue.cc:237] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:49.002190 10809 raft_consensus.cc:515] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.016191 10888 consensus_queue.cc:1048] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 24, Last known committed idx: 23, Time since last communication: 0.000s
W20260709 12:25:48.989518 31717 consensus_peers.cc:597] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:49.030778 10809 leader_election.cc:290] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:49.032054 31717 leader_election.cc:336] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:49.036314 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:49.037297 31745 raft_consensus.cc:2468] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:49.039343 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 40 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:49.039904 32370 raft_consensus.cc:1275] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 39 index: 22. Preceding OpId from leader: term: 40 index: 23. (index mismatch)
I20260709 12:25:49.041337 10809 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election won for term 35
W20260709 12:25:49.044688 31717 consensus_peers.cc:597] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:49.041767 10809 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:49.046996 10809 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:49.041896 10883 consensus_queue.cc:1048] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.007s
I20260709 12:25:49.048120 10877 consensus_queue.cc:1048] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:49.048653 10856 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 39 FOLLOWER]: Leader pre-election won for term 40
I20260709 12:25:49.049129 10856 raft_consensus.cc:493] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 39 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:49.049491 10856 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:25:49.053690 10880 raft_consensus.cc:515] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.043311 31791 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:49.055312 10876 raft_consensus.cc:2804] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election won for term 31
W20260709 12:25:49.055423 31791 leader_election.cc:336] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:49.055927 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:49.057027 31791 leader_election.cc:336] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:49.057754 10907 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election lost for term 34. Reason: could not achieve majority
I20260709 12:25:49.055982 10876 raft_consensus.cc:697] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:49.059603 10876 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:49.061218 31791 leader_election.cc:336] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:49.084404 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:49.085109 31749 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
I20260709 12:25:49.087306 31749 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "553d0b2dddba47779b0219c71ed9ce83" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:49.087888 31749 raft_consensus.cc:3060] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:49.098719 10888 consensus_queue.cc:1048] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:49.100175 31575 catalog_manager.cc:5697] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 30 to 31, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 31 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:49.110024 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.116585 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "55f4b2e07ada4e06a285803fcbfd6344" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 38 candidate_status { last_received { term: 37 index: 22 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:49.117398 31744 raft_consensus.cc:3060] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Advancing to term 38
I20260709 12:25:49.119163 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.120334 10423 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:49.120949 10423 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:49.121452 10423 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:49.133695 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.135885 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.103104 10877 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:49.147693 31744 raft_consensus.cc:2468] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 38.
I20260709 12:25:49.132447 10860 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election won for term 35
I20260709 12:25:49.138276 10882 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:49.156425 10423 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.168681 31749 raft_consensus.cc:2468] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 35.
I20260709 12:25:49.179226 10423 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:49.156903 10860 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:49.174001 10893 tablet_copy_client.cc:538] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:25:49.171177 10882 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:49.143004 10877 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.174093 10904 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:49.222455 10877 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:25:49.224383 10915 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.116s	user 0.002s	sys 0.000s
W20260709 12:25:49.225365 10911 log.cc:927] Time spent T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.131s	user 0.000s	sys 0.001s
W20260709 12:25:49.225971 31717 leader_election.cc:336] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:49.164215 10889 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:49.228193 10889 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:49.228829 32375 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 35: Already voted for candidate 5206fd0087404d2da964a5d985a04ff4 in this term.
I20260709 12:25:49.229908 10809 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.232071 10809 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:49.234596 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 35 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:49.205543 10860 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Advancing to term 35
I20260709 12:25:49.235592 31748 raft_consensus.cc:2393] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 5206fd0087404d2da964a5d985a04ff4 in current term 35: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:25:49.236550 10909 log.cc:927] Time spent T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.052s	user 0.000s	sys 0.000s
I20260709 12:25:49.215006 10882 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:49.235069 10900 tablet_copy_client.cc:806] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:25:49.218750 10904 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:49.219553 10880 leader_election.cc:290] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:49.276043 10893 tablet_bootstrap.cc:492] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
W20260709 12:25:49.281492 31791 consensus_peers.cc:597] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:49.269268 10900 tablet_copy_client.cc:670] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
W20260709 12:25:49.282591 31791 leader_election.cc:336] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:49.285451 31791 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:49.287956 31791 leader_election.cc:336] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 40 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:49.289079 31791 leader_election.cc:336] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:49.291076 10900 tablet_copy_client.cc:538] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 12:25:49.295683 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 111051us
I20260709 12:25:49.259861 10904 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:49.313429 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d0a3795ad4d543f0998c3834aac7be45" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:49.314254 31748 raft_consensus.cc:3060] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:49.352686 31717 consensus_peers.cc:597] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:49.354380 10882 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.356037 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:49.356798 31753 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:25:49.359153 31748 raft_consensus.cc:2468] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 34.
I20260709 12:25:49.359619 10856 raft_consensus.cc:515] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 40 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.361361 10860 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:49.367417 10913 log.cc:927] Time spent T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.070s	user 0.000s	sys 0.000s
W20260709 12:25:49.374737 32342 leader_election.cc:336] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:49.376878 10912 log.cc:927] Time spent T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.199s	user 0.005s	sys 0.000s
W20260709 12:25:49.384581 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 248765us
I20260709 12:25:49.404265 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.405139 10923 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:49.405495 31718 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.405666 10860 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:49.405555 10923 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:49.395735 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:49.395068 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:49.407122 31753 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 35: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
W20260709 12:25:49.400926 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55270.4us
I20260709 12:25:49.402499 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.409610 31791 leader_election.cc:304] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:49.408595 10923 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:49.412330 10920 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:49.422577 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.406405 10860 raft_consensus.cc:2764] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader pre-election decision vote started in defunct term 33: won
I20260709 12:25:49.427503 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.421904 10900 tablet_bootstrap.cc:492] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:25:49.436641 10856 leader_election.cc:290] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 40 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:49.434044 10925 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
I20260709 12:25:49.443912 10925 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:49.444352 10925 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:49.431385 10809 raft_consensus.cc:2804] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader election won for term 38
I20260709 12:25:49.457929 10809 raft_consensus.cc:697] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 38 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:25:49.458585 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:49.458959 10809 consensus_queue.cc:237] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 37.22, Last appended by leader: 22, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.461673 31748 raft_consensus.cc:1275] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:49.466730 10882 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:49.473873 10925 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.411273 10907 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election won for term 33
I20260709 12:25:49.477618 10907 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
I20260709 12:25:49.478113 10907 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:49.482817 10888 consensus_queue.cc:1048] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
W20260709 12:25:49.484802 31717 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:49.485447 31717 consensus_peers.cc:597] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:49.474319 10904 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:49.487675 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 55826.9us
I20260709 12:25:49.541474 10907 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.544839 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b416bf5304744b14b72501e697b8c951" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:49.545590 31822 raft_consensus.cc:3060] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:49.547160 32342 consensus_peers.cc:597] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:49.552215 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9c849da11b414111b8461d6ae06c15b3" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:49.502180 10903 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader pre-election won for term 33
W20260709 12:25:49.502465 31717 leader_election.cc:336] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:49.559551 31717 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 57136.6us
W20260709 12:25:49.532294 10924 log.cc:927] Time spent T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.106s	user 0.001s	sys 0.000s
I20260709 12:25:49.588549 31820 raft_consensus.cc:3060] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:49.533604 31575 catalog_manager.cc:5697] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 30 to 31, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 31 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:49.555094 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 35 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:49.544097 10907 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:49.557142 32370 raft_consensus.cc:1275] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 35 index: 24. Preceding OpId from leader: term: 36 index: 25. (index mismatch)
I20260709 12:25:49.562144 10903 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:49.566710 30503 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 31699)
I20260709 12:25:49.599391 10925 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:49.598510 31824 raft_consensus.cc:2393] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 35: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:49.573709 31718 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.599499 10903 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Advancing to term 33
I20260709 12:25:49.608791 10860 raft_consensus.cc:2749] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
W20260709 12:25:49.613610 31717 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:49.614311 31717 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:49.623986 31820 raft_consensus.cc:2468] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:49.427100 31791 leader_election.cc:304] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:49.643817 10907 raft_consensus.cc:2804] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election won for term 35
I20260709 12:25:49.644452 10907 raft_consensus.cc:697] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:49.645444 10907 consensus_queue.cc:237] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 35, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:49.624892 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 51300.6us
I20260709 12:25:49.533955 10923 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.656318 31822 raft_consensus.cc:2468] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
W20260709 12:25:49.659807 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 232771us
I20260709 12:25:49.592996 10904 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:49.623675 10903 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.676534 31824 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 32 candidate_status { last_received { term: 31 index: 17 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:49.679185 31824 raft_consensus.cc:3060] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:49.678449 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc55086b81374990afb970c972568882" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 33 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:49.680971 31822 raft_consensus.cc:3060] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:49.677448 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:49.688629 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.690912 31791 leader_election.cc:304] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:49.691900 10907 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader pre-election won for term 32
I20260709 12:25:49.679068 10925 consensus_queue.cc:1048] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 25, Last known committed idx: 24, Time since last communication: 0.014s
I20260709 12:25:49.692911 10907 raft_consensus.cc:493] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election (detected failure of leader b95aeeaf4c51442b8092cff1de674f2b)
W20260709 12:25:49.697677 31717 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:49.699386 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:49.701989 10809 raft_consensus.cc:2804] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:49.708567 10809 raft_consensus.cc:697] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 34 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:49.709482 10809 consensus_queue.cc:237] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.711028 31824 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 32.
W20260709 12:25:49.715080 31717 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:49.702420 31575 catalog_manager.cc:5697] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 37 to 38, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 38 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:49.720117 31718 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.723613 31718 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:49.725548 31791 consensus_peers.cc:597] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:49.726889 10925 raft_consensus.cc:2804] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:49.727504 10925 raft_consensus.cc:697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
W20260709 12:25:49.726239 31791 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:49.728924 10423 raft_consensus.cc:2804] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election won for term 34
W20260709 12:25:49.729677 31791 leader_election.cc:336] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:49.730355 31791 leader_election.cc:336] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:49.728480 10925 consensus_queue.cc:237] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 33.20, Last appended by leader: 20, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.731957 10423 raft_consensus.cc:697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:49.724327 10860 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:49.732931 10423 consensus_queue.cc:237] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 33.19, Last appended by leader: 19, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.737126 31822 raft_consensus.cc:2468] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:49.741259 31718 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:49.754578 32342 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:49.756510 31718 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.759166 10882 raft_consensus.cc:2804] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:49.759774 10882 raft_consensus.cc:697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:49.760737 10882 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 31.17, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.723706 10907 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:49.771844 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 35 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:49.772800 31820 raft_consensus.cc:3060] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:49.777040 31753 raft_consensus.cc:1275] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:25:49.741163 10923 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:49.788767 31820 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 36.
I20260709 12:25:49.793839 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:49.799345 31822 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 039d66a6c85e43c3936bf5017501b344 in current term 33: Already voted for candidate 4d48c83c761f48079bc6ea176649dda0 in this term.
I20260709 12:25:49.801608 31718 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.802848 10423 raft_consensus.cc:2804] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:49.803282 10423 raft_consensus.cc:697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:49.788908 10860 raft_consensus.cc:2804] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:49.816726 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:49.818156 31748 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "504cb3d29e9347fe98f010a3707a55cc" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 40 candidate_status { last_received { term: 39 index: 26 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:49.821691 31748 raft_consensus.cc:3060] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 39 FOLLOWER]: Advancing to term 40
I20260709 12:25:49.788457 10903 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 33 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:49.824175 10423 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 35.19, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:49.834089 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:49.835490 31753 raft_consensus.cc:2393] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election vote request: Denying vote to candidate 4d48c83c761f48079bc6ea176649dda0 in current term 33: Already voted for candidate 039d66a6c85e43c3936bf5017501b344 in this term.
I20260709 12:25:49.840413 31745 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 31 index: 21. (index mismatch)
I20260709 12:25:49.841280 31748 raft_consensus.cc:2468] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 40 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 40.
I20260709 12:25:49.850842 10860 raft_consensus.cc:697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:49.856261 10860 consensus_queue.cc:237] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 33.18, Last appended by leader: 18, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:49.861876 10939 log.cc:927] Time spent T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.117s	user 0.002s	sys 0.000s
I20260709 12:25:49.872017 31575 catalog_manager.cc:5697] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 34 to 35, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 35 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:49.885586 31575 catalog_manager.cc:5697] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 33 to 34, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 34 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:49.844569 10940 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:49.843290 10876 consensus_queue.cc:1048] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.043s
W20260709 12:25:49.922664 32342 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:49.955868 31576 catalog_manager.cc:5697] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 33 to 34, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 34 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:49.963626 31576 catalog_manager.cc:5697] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 33 to 34, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 34 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:50.018708 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 135409us
I20260709 12:25:50.028879 31791 leader_election.cc:304] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 40 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:50.051278 10921 raft_consensus.cc:493] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:50.051957 10921 raft_consensus.cc:515] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:50.054400 10921 leader_election.cc:290] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:50.056545 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a22e63b9ca824c92ae9ba374d2f3242a" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:25:50.075028 31791 leader_election.cc:336] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:50.092028 31791 consensus_peers.cc:597] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:50.092769 31791 leader_election.cc:336] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:50.093546 31791 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:50.094007 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:50.058710 10905 raft_consensus.cc:2804] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 40 FOLLOWER]: Leader election won for term 40
I20260709 12:25:50.045305 10876 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.057s
I20260709 12:25:50.102871 10905 raft_consensus.cc:697] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 40 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
W20260709 12:25:50.108078 31717 leader_election.cc:336] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:50.108714 31717 leader_election.cc:304] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:50.104028 10905 consensus_queue.cc:237] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 39.26, Last appended by leader: 26, Current term: 40, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:50.128158 10876 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader election lost for term 35. Reason: could not achieve majority
I20260709 12:25:50.094383 10907 raft_consensus.cc:515] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:50.147668 10907 leader_election.cc:290] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:50.127996 10925 raft_consensus.cc:2749] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:25:50.164045 31717 consensus_peers.cc:597] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:50.164806 31717 consensus_peers.cc:597] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:50.181030 31717 consensus_peers.cc:597] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:50.196028 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 68314.4us
I20260709 12:25:50.198771 31748 raft_consensus.cc:1275] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 37 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:25:50.237967 31745 raft_consensus.cc:1275] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
W20260709 12:25:50.195956 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 51117.1us
I20260709 12:25:50.228020 31574 catalog_manager.cc:5697] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 31 to 32, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 32 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:50.269052 31744 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5bb2dc5276714f75b6e3e61ca9275f84" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 32 candidate_status { last_received { term: 31 index: 18 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:50.252621 31574 catalog_manager.cc:5697] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 35 to 36, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 36 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:50.279364 31744 raft_consensus.cc:3060] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Advancing to term 32
I20260709 12:25:50.280172 31574 catalog_manager.cc:5697] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 33 to 34, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 34 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:50.266587 31753 raft_consensus.cc:1275] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 35 index: 19. (index mismatch)
I20260709 12:25:50.311849 31744 raft_consensus.cc:2468] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
W20260709 12:25:50.338389 31717 consensus_peers.cc:597] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:50.339216 31717 consensus_peers.cc:597] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:50.341346 31826 raft_consensus.cc:1275] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 33 index: 19. Preceding OpId from leader: term: 34 index: 20. (index mismatch)
I20260709 12:25:50.343135 10907 consensus_queue.cc:1048] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:25:50.349799 31820 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 35 index: 19. Preceding OpId from leader: term: 36 index: 20. (index mismatch)
I20260709 12:25:50.346755 31576 catalog_manager.cc:5697] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 39 to 40, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 40 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
I20260709 12:25:50.367354 10876 consensus_queue.cc:1048] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:25:50.378883 31717 consensus_peers.cc:597] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:50.387192 10876 consensus_queue.cc:1048] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
I20260709 12:25:50.387633 10904 consensus_queue.cc:1048] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:50.390591 31820 raft_consensus.cc:1275] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 33 index: 20. Preceding OpId from leader: term: 34 index: 21. (index mismatch)
I20260709 12:25:50.397450 31791 leader_election.cc:304] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:50.400601 10940 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:50.404206 10907 raft_consensus.cc:2804] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader election won for term 32
I20260709 12:25:50.405088 10907 raft_consensus.cc:697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 32 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:25:50.405006 10940 consensus_queue.cc:1048] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:50.407703 10907 consensus_queue.cc:237] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 31.18, Last appended by leader: 18, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:50.410966 31791 consensus_peers.cc:597] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:50.412199 31791 leader_election.cc:336] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 40 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:50.422816 32342 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:50.455063 32342 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:50.438537 31717 consensus_peers.cc:597] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:50.439213 31824 raft_consensus.cc:1275] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 33 index: 18. Preceding OpId from leader: term: 34 index: 19. (index mismatch)
I20260709 12:25:50.460201 10904 consensus_queue.cc:1048] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:25:50.430301 31791 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:50.464948 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:50.465437 31824 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 31 index: 17. Preceding OpId from leader: term: 32 index: 18. (index mismatch)
W20260709 12:25:50.466300 31791 consensus_peers.cc:597] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:50.466977 10916 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader election lost for term 33. Reason: could not achieve majority
I20260709 12:25:50.471215 10940 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.000s
W20260709 12:25:50.500478 31791 consensus_peers.cc:597] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:50.501583 31791 leader_election.cc:336] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 32 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:50.502467 31791 consensus_peers.cc:597] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:50.522408 32342 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:50.599879 30503 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 31699)
W20260709 12:25:50.625496 31791 consensus_peers.cc:597] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:50.574302 31576 catalog_manager.cc:5697] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 31 to 32, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 32 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } }
W20260709 12:25:50.616683 10900 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.062s	user 0.001s	sys 0.000s
W20260709 12:25:50.635924 31792 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:44855, user_credentials={real_user=slave}} blocked reactor thread for 120349us
W20260709 12:25:50.647173 31791 consensus_peers.cc:597] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:50.678194 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:50.746338 10968 log.cc:927] Time spent T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.071s	user 0.001s	sys 0.000s
W20260709 12:25:50.754683 31791 consensus_peers.cc:597] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:50.755481 10971 log.cc:927] Time spent T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:25:50.760949 31745 raft_consensus.cc:1275] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 40 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 39 index: 26. Preceding OpId from leader: term: 40 index: 27. (index mismatch)
I20260709 12:25:50.789027 10945 consensus_queue.cc:1048] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.000s
I20260709 12:25:50.792508 10893 tablet_bootstrap.cc:492] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=23 overwritten=2 applied=21 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:50.814953 10893 tablet_bootstrap.cc:492] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:25:50.815990 10893 ts_tablet_manager.cc:1403] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 1.540s	user 0.090s	sys 0.001s
W20260709 12:25:50.915390 31717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111) [suppressed 357 similar messages]
I20260709 12:25:50.926833 31744 raft_consensus.cc:1275] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 31 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
W20260709 12:25:50.931277 31717 consensus_peers.cc:597] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:50.935988 31717 consensus_peers.cc:597] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:50.983412 10893 raft_consensus.cc:359] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [term 38 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:25:50.985277 10893 raft_consensus.cc:740] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [term 38 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
W20260709 12:25:50.997799 10900 log.cc:927] Time spent T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4: Append to log took a long time: real 0.179s	user 0.001s	sys 0.000s
I20260709 12:25:50.986435 10893 consensus_queue.cc:260] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 21, Last appended: 38.21, Last appended by leader: 21, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:25:51.019299 10888 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.020107 10888 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:51.023218 10888 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:51.034705 10974 consensus_queue.cc:1048] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:25:51.041667 31791 consensus_peers.cc:597] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:51.042366 31791 consensus_peers.cc:597] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:51.083710 10893 ts_tablet_manager.cc:1434] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.267s	user 0.007s	sys 0.005s
W20260709 12:25:51.106762 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 108760us
W20260709 12:25:51.123522 10979 log.cc:927] Time spent T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: Append to log took a long time: real 0.067s	user 0.000s	sys 0.001s
I20260709 12:25:51.145660 31837 tablet_copy_service.cc:342] P 4d48c83c761f48079bc6ea176649dda0: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-3a0925b3d6e9448684fd239b27453b69 received from {username='slave'} at 127.0.0.1:34642
I20260709 12:25:51.146457 31837 tablet_copy_service.cc:434] P 4d48c83c761f48079bc6ea176649dda0: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-3a0925b3d6e9448684fd239b27453b69 on tablet 3a0925b3d6e9448684fd239b27453b69 with peer 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:51.171509 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 55151.1us
I20260709 12:25:51.178928 31747 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344" is_pre_election: true
I20260709 12:25:51.179857 31747 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 35.
W20260709 12:25:51.251528 31717 consensus_peers.cc:597] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:51.255867 31791 consensus_peers.cc:597] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:51.301779 10900 tablet_bootstrap.cc:492] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=19 overwritten=0 applied=19 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:51.321135 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:25:51.322137 10888 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:51.322573 10888 raft_consensus.cc:493] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.322968 10888 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:25:51.341869 32342 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:51.343372 10900 tablet_bootstrap.cc:492] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:25:51.346057 10900 ts_tablet_manager.cc:1403] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 1.943s	user 0.076s	sys 0.026s
W20260709 12:25:51.342586 32342 leader_election.cc:336] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:51.378098 32370 raft_consensus.cc:1217] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [term 38 LEARNER]: Deduplicated request from leader. Original: 38.20->[38.21-38.21]   Dedup: 38.21->[]
W20260709 12:25:51.390719 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 72630.4us
I20260709 12:25:51.334127 10888 raft_consensus.cc:515] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:51.396569 10888 leader_election.cc:290] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:51.422971 31753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6499e78dd87c4e05be32f5681957a6b0" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 36 candidate_status { last_received { term: 34 index: 25 } } ignore_live_leader: false dest_uuid: "039d66a6c85e43c3936bf5017501b344"
I20260709 12:25:51.423755 31753 raft_consensus.cc:3060] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
W20260709 12:25:51.427090 32342 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:51.442440 10900 raft_consensus.cc:359] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:25:51.443809 10900 raft_consensus.cc:740] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:25:51.444727 10900 consensus_queue.cc:260] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 30.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
W20260709 12:25:51.451618 32342 leader_election.cc:336] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:51.452383 10900 ts_tablet_manager.cc:1434] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.106s	user 0.007s	sys 0.003s
I20260709 12:25:51.454671 31762 tablet_copy_service.cc:342] P 039d66a6c85e43c3936bf5017501b344: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-20650b4ef2514a05bde5dfd38a379f32 received from {username='slave'} at 127.0.0.1:57176
I20260709 12:25:51.455216 31762 tablet_copy_service.cc:434] P 039d66a6c85e43c3936bf5017501b344: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-20650b4ef2514a05bde5dfd38a379f32 on tablet 20650b4ef2514a05bde5dfd38a379f32 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:51.470368 31753 raft_consensus.cc:2468] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:25:51.481262 32342 leader_election.cc:304] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4; no voters: b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:25:51.480861 31791 consensus_peers.cc:597] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:51.483530 10890 raft_consensus.cc:2804] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:51.486274 10890 raft_consensus.cc:697] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 36 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
W20260709 12:25:51.502491 31717 consensus_peers.cc:597] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:51.502573 10890 consensus_queue.cc:237] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 25, Committed index: 25, Last appended: 34.25, Last appended by leader: 25, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:51.506311 10860 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.507012 10860 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:51.510491 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:51.511221 31821 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 35.
I20260709 12:25:51.513372 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:51.514946 10423 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Leader pre-election won for term 36
I20260709 12:25:51.515303 10423 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.515605 10423 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:51.518424 10860 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:51.519249 31717 consensus_peers.cc:597] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:51.527417 10423 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:51.527645 31717 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:51.529661 10423 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:51.530846 31821 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "76c4ab1f56e44ff7b6adf7cfe850e01d" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 36 candidate_status { last_received { term: 34 index: 18 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:51.531482 31821 raft_consensus.cc:3060] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Advancing to term 36
I20260709 12:25:51.545940 31574 catalog_manager.cc:5697] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 34 to 36, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 36 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:51.549841 31821 raft_consensus.cc:2468] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 36.
W20260709 12:25:51.551203 32342 consensus_peers.cc:597] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:51.554071 31717 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:51.562306 31717 consensus_peers.cc:597] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:51.567802 31718 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 36 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:51.569906 10860 raft_consensus.cc:2804] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Leader election won for term 36
I20260709 12:25:51.570467 10860 raft_consensus.cc:697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:51.571465 10860 consensus_queue.cc:237] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 34.18, Last appended by leader: 18, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:51.595281 31576 catalog_manager.cc:5697] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 34 to 36, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 36 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:51.600921 30503 thread.cc:527] Waited for 4000ms trying to join with diag-logger (tid 31699)
I20260709 12:25:51.639863 32370 raft_consensus.cc:1217] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEARNER]: Deduplicated request from leader. Original: 30.18->[30.19-30.19]   Dedup: 30.19->[]
W20260709 12:25:51.665071 31717 consensus_peers.cc:597] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:51.688925 10906 raft_consensus.cc:1064] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:25:51.699834 10906 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 38.21, Last appended by leader: 19, Current term: 38, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:51.732439 31717 consensus_peers.cc:597] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:51.747839 31791 consensus_peers.cc:597] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:51.803965 31753 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:25:51.807140 32370 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [term 38 LEARNER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 38 index: 21. Preceding OpId from leader: term: 38 index: 22. (index mismatch)
I20260709 12:25:51.801035 10860 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:25:51.808772 10860 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:51.809289 10906 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:25:51.812077 10860 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:51.816383 31791 consensus_peers.cc:597] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:51.819096 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:25:51.820951 31820 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 33.
I20260709 12:25:51.823963 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: 
I20260709 12:25:51.824896 10423 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Leader pre-election won for term 34
W20260709 12:25:51.824975 31717 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:51.825471 10423 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:25:51.825876 10423 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Advancing to term 34
W20260709 12:25:51.837618 31717 consensus_peers.cc:597] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:51.843587 10972 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:25:51.860076 10423 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:25:51.873857 31820 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d0d0402170d41b79dc30f93bee98766" candidate_uuid: "039d66a6c85e43c3936bf5017501b344" candidate_term: 34 candidate_status { last_received { term: 32 index: 15 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:25:51.874872 31820 raft_consensus.cc:3060] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Advancing to term 34
I20260709 12:25:51.876259 10423 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Requested vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:25:51.912470 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:51.919598 31717 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:51.924827 31820 raft_consensus.cc:2468] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 039d66a6c85e43c3936bf5017501b344 in term 34.
I20260709 12:25:51.929409 31718 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [CANDIDATE]: Term 34 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0; no voters: b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:25:51.930912 10423 raft_consensus.cc:2804] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Leader election won for term 34
I20260709 12:25:51.934779 10423 raft_consensus.cc:697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Becoming Leader. State: Replica: 039d66a6c85e43c3936bf5017501b344, State: Running, Role: LEADER
I20260709 12:25:51.935849 10423 consensus_queue.cc:237] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 32.15, Last appended by leader: 15, Current term: 34, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:25:51.949195 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:51.964700 10919 raft_consensus.cc:2955] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 38 LEADER]: Committing config change with OpId 38.22: config changed from index 21 to 22, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:51.970558 32370 raft_consensus.cc:2955] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Committing config change with OpId 38.22: config changed from index 21 to 22, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:51.979652 31574 catalog_manager.cc:5697] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: term changed from 32 to 34, leader changed from b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) to 039d66a6c85e43c3936bf5017501b344 (127.29.201.194). New cstate: current_term: 34 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } }
W20260709 12:25:51.988582 31791 consensus_peers.cc:597] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:52.003782 32342 consensus_peers.cc:597] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:52.005858 32342 consensus_peers.cc:597] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:52.055565 31574 catalog_manager.cc:5697] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 21 to 22, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 38 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:52.072858 31747 raft_consensus.cc:1275] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 34 index: 25. Preceding OpId from leader: term: 36 index: 26. (index mismatch)
I20260709 12:25:52.071066 31753 raft_consensus.cc:2955] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Committing config change with OpId 38.22: config changed from index 21 to 22, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:52.074927 10888 consensus_queue.cc:1048] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 26, Last known committed idx: 25, Time since last communication: 0.000s
W20260709 12:25:52.087152 31791 consensus_peers.cc:597] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:52.111227 31791 consensus_peers.cc:597] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:52.135495 31822 consensus_queue.cc:237] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 38.22, Last appended by leader: 19, Current term: 38, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:52.152029 32370 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:25:52.154773 10972 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
I20260709 12:25:52.165225 10984 raft_consensus.cc:1064] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:25:52.165426 10972 raft_consensus.cc:2955] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 38 LEADER]: Committing config change with OpId 38.23: config changed from index 22 to 23, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:52.170683 10984 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 30.19, Last appended by leader: 17, Current term: 30, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:52.173245 32374 raft_consensus.cc:2955] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Committing config change with OpId 38.23: config changed from index 22 to 23, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:52.197350 31820 raft_consensus.cc:1275] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 34 index: 18. Preceding OpId from leader: term: 36 index: 19. (index mismatch)
I20260709 12:25:52.201128 10860 consensus_queue.cc:1048] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
I20260709 12:25:52.219275 31822 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:25:52.222844 32374 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEARNER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 19. Preceding OpId from leader: term: 30 index: 20. (index mismatch)
I20260709 12:25:52.225561 31749 raft_consensus.cc:1275] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 38 index: 22. Preceding OpId from leader: term: 38 index: 23. (index mismatch)
I20260709 12:25:52.228350 10423 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.000s
I20260709 12:25:52.229463 31562 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 3a0925b3d6e9448684fd239b27453b69 with cas_config_opid_index 22: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260709 12:25:52.240548 31576 catalog_manager.cc:5697] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 22 to 23, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New cstate: current_term: 38 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:52.249850 31717 consensus_peers.cc:597] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:52.261672 31717 consensus_peers.cc:597] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:52.275856 10973 consensus_queue.cc:1048] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.001s
W20260709 12:25:52.290023 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 67647.8us
I20260709 12:25:52.323875 10860 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
I20260709 12:25:52.368703 31749 raft_consensus.cc:2955] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Committing config change with OpId 38.23: config changed from index 22 to 23, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:52.379036 31791 consensus_peers.cc:597] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:52.442229 32342 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:52.456851 31791 consensus_peers.cc:597] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:52.466689 31561 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 115517us
I20260709 12:25:52.484506 31825 raft_consensus.cc:1275] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 32 index: 15. Preceding OpId from leader: term: 34 index: 16. (index mismatch)
I20260709 12:25:52.496836 10860 consensus_queue.cc:1048] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.001s
W20260709 12:25:52.500964 31561 catalog_manager.cc:4755] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3a0925b3d6e9448684fd239b27453b69: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:25:52.550215 32342 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:52.583129 10423 raft_consensus.cc:2955] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Committing config change with OpId 30.20: config changed from index 19 to 20, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:52.589097 32374 raft_consensus.cc:2955] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Committing config change with OpId 30.20: config changed from index 19 to 20, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:52.591969 31791 consensus_peers.cc:597] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:52.601492 30503 thread.cc:527] Waited for 5000ms trying to join with diag-logger (tid 31699)
W20260709 12:25:52.605765 31717 consensus_peers.cc:597] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:52.606431 31717 consensus_peers.cc:597] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:25:52.651448 31820 raft_consensus.cc:2955] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Committing config change with OpId 30.20: config changed from index 19 to 20, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:52.671015 32342 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:52.731560 31574 catalog_manager.cc:5697] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index 19 to 20, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 30 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:25:52.762979 32342 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:52.763090 31717 consensus_peers.cc:597] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:52.774572 31747 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 30.20, Last appended by leader: 17, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:52.787074 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 61647.1us
I20260709 12:25:52.810379 32377 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:52.823799 31822 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 20. Preceding OpId from leader: term: 30 index: 21. (index mismatch)
I20260709 12:25:52.844967 10984 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:25:52.851989 32342 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:52.854941 10423 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
I20260709 12:25:52.866029 10984 raft_consensus.cc:2955] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Committing config change with OpId 30.21: config changed from index 20 to 21, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:52.874366 31717 consensus_peers.cc:597] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:52.886044 32374 raft_consensus.cc:2955] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Committing config change with OpId 30.21: config changed from index 20 to 21, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:52.898258 31717 consensus_peers.cc:597] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:52.913949 31822 raft_consensus.cc:2955] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Committing config change with OpId 30.21: config changed from index 20 to 21, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:52.938328 31791 consensus_peers.cc:597] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:52.951864 31574 catalog_manager.cc:5697] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 20 to 21, VOTER b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193) evicted. New cstate: current_term: 30 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:52.983614 31717 consensus_peers.cc:597] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:52.987792 31561 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet 20650b4ef2514a05bde5dfd38a379f32 with cas_config_opid_index 20: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20260709 12:25:52.994110 31717 consensus_peers.cc:597] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:52.983820 31791 consensus_peers.cc:597] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:53.031715 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 47955.4us
W20260709 12:25:53.042676 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:53.042909 31717 consensus_peers.cc:597] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:53.077337 31791 consensus_peers.cc:597] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:53.078198 31791 consensus_peers.cc:597] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:53.165197 31791 consensus_peers.cc:597] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:53.217330 31791 consensus_peers.cc:597] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:53.233702 31791 consensus_peers.cc:597] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:53.337373 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:53.457433 31791 consensus_peers.cc:597] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:53.527180 31717 consensus_peers.cc:597] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:53.528807 32342 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:53.575444 31791 consensus_peers.cc:597] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:53.583009 31791 consensus_peers.cc:597] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:53.602070 30503 thread.cc:527] Waited for 6000ms trying to join with diag-logger (tid 31699)
I20260709 12:25:53.627521 31820 consensus_queue.cc:237] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 16, Committed index: 16, Last appended: 28.16, Last appended by leader: 15, Current term: 28, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:53.654431 32374 raft_consensus.cc:1275] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 28 index: 16. Preceding OpId from leader: term: 28 index: 17. (index mismatch)
W20260709 12:25:53.655143 31561 catalog_manager.cc:4755] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 20650b4ef2514a05bde5dfd38a379f32: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:53.656874 10973 consensus_queue.cc:1048] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 17, Last known committed idx: 16, Time since last communication: 0.000s
W20260709 12:25:53.658825 31791 consensus_peers.cc:597] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:53.714991 10919 raft_consensus.cc:2955] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Committing config change with OpId 28.17: config changed from index 3 to 17, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:53.717571 31717 consensus_peers.cc:597] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:25:53.736920 32374 raft_consensus.cc:2955] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Committing config change with OpId 28.17: config changed from index 3 to 17, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New config: { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:53.746801 31562 catalog_manager.cc:5210] ChangeConfig:REMOVE_PEER RPC for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 11)
I20260709 12:25:53.820600 31574 catalog_manager.cc:5697] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 3 to 17, VOTER 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) evicted. New cstate: current_term: 28 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:54.093086 31735 tablet_service.cc:1558] Processing DeleteTablet for tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 with delete_type TABLET_DATA_TOMBSTONED (TS 039d66a6c85e43c3936bf5017501b344 not found in new config with opid_index 17) from {username='slave'} at 127.0.0.1:37588
W20260709 12:25:54.099924 32342 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:54.115370 11023 tablet_replica.cc:333] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
W20260709 12:25:54.125535 31717 consensus_peers.cc:597] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:54.129823 11023 raft_consensus.cc:2243] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:25:54.135288 11023 raft_consensus.cc:2272] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:25:54.139415 11023 ts_tablet_manager.cc:1916] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260709 12:25:54.149619 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:46639, user_credentials={real_user=slave}} blocked reactor thread for 54526.8us
W20260709 12:25:54.150029 31717 consensus_peers.cc:597] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:54.201184 11023 ts_tablet_manager.cc:1929] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 28.16
I20260709 12:25:54.201887 11023 log.cc:1199] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 039d66a6c85e43c3936bf5017501b344: Deleting WAL directory at /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/wals/b25009db0c5a47a9ab9f6f8b2e5c1b17
W20260709 12:25:54.213927 32342 consensus_peers.cc:597] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:54.220329 31561 catalog_manager.cc:5028] TS 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): tablet b25009db0c5a47a9ab9f6f8b2e5c1b17 (table test-workload [id=d81b7fed68a84c348a74895b5b4310a8]) successfully deleted
W20260709 12:25:54.230821 32342 consensus_peers.cc:597] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:54.246938 31717 consensus_peers.cc:597] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:54.247649 31717 consensus_peers.cc:597] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:54.250349 31753 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 32.18, Last appended by leader: 17, Current term: 32, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
W20260709 12:25:54.255954 31717 consensus_peers.cc:597] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:54.258389 31791 consensus_peers.cc:597] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:54.267446 31827 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 32 index: 18. Preceding OpId from leader: term: 32 index: 19. (index mismatch)
I20260709 12:25:54.269167 11016 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.000s
W20260709 12:25:54.270102 31718 consensus_peers.cc:597] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f933d4319cf444afb9f67bd265576a31. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:54.280207 31717 consensus_peers.cc:597] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:54.285136 11021 raft_consensus.cc:2955] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Committing config change with OpId 32.19: config changed from index -1 to 19, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:25:54.292078 31827 raft_consensus.cc:2955] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Committing config change with OpId 32.19: config changed from index -1 to 19, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:25:54.335222 31753 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 20, Committed index: 20, Last appended: 36.20, Last appended by leader: 19, Current term: 36, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:25:54.342779 31575 catalog_manager.cc:5697] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index -1 to 19, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 32 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
W20260709 12:25:54.370960 31718 consensus_peers.cc:597] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e59926725a06436e9899dbbbcae3f622. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:54.376148 31827 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 36 index: 20. Preceding OpId from leader: term: 36 index: 21. (index mismatch)
I20260709 12:25:54.381758 11021 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 21, Last known committed idx: 20, Time since last communication: 0.001s
W20260709 12:25:54.382881 31717 consensus_peers.cc:597] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:54.448056 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:54.466747 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 57403.1us
W20260709 12:25:54.478153 31791 consensus_peers.cc:597] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:54.499727 11021 raft_consensus.cc:2955] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Committing config change with OpId 36.21: config changed from index -1 to 21, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:25:54.539553 31820 raft_consensus.cc:2955] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Committing config change with OpId 36.21: config changed from index -1 to 21, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
W20260709 12:25:54.575860 32342 consensus_peers.cc:597] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:54.583837 31574 catalog_manager.cc:5697] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index -1 to 21, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 36 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
W20260709 12:25:54.586381 31717 consensus_peers.cc:597] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:54.602656 30503 thread.cc:527] Waited for 7000ms trying to join with diag-logger (tid 31699)
W20260709 12:25:54.629266 32342 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:54.639483 31791 consensus_peers.cc:597] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:54.639866 31820 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 31.21, Last appended by leader: 20, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
W20260709 12:25:54.643993 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:54.656229 31792 consensus_peers.cc:597] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc0ece2345cd46edb250e06e8f5431d6. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:54.666646 31747 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 31 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:25:54.678841 10973 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.000s
W20260709 12:25:54.682118 31791 consensus_peers.cc:597] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:54.707049 31717 consensus_peers.cc:597] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:54.723639 10973 raft_consensus.cc:2955] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Committing config change with OpId 31.22: config changed from index -1 to 22, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:25:54.733630 31747 raft_consensus.cc:2955] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Committing config change with OpId 31.22: config changed from index -1 to 22, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:25:54.728801 11035 ts_tablet_manager.cc:933] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:54.761785 31574 catalog_manager.cc:5697] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 22, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 31 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:25:54.769057 31753 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 17, Committed index: 17, Last appended: 30.17, Last appended by leader: 16, Current term: 30, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:25:54.773536 11035 tablet_copy_client.cc:323] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.194:46639
I20260709 12:25:54.786465 31762 tablet_copy_service.cc:140] P 039d66a6c85e43c3936bf5017501b344: Received BeginTabletCopySession request for tablet f933d4319cf444afb9f67bd265576a31 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:57176)
I20260709 12:25:54.787241 31762 tablet_copy_service.cc:161] P 039d66a6c85e43c3936bf5017501b344: Beginning new tablet copy session on tablet f933d4319cf444afb9f67bd265576a31 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:57176: session id = 5206fd0087404d2da964a5d985a04ff4-f933d4319cf444afb9f67bd265576a31
I20260709 12:25:54.797458 31762 tablet_copy_source_session.cc:215] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:25:54.817545 11035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f933d4319cf444afb9f67bd265576a31. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:25:54.834973 31718 consensus_peers.cc:597] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 -> Peer 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855): Couldn't send request to peer 5206fd0087404d2da964a5d985a04ff4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d11aa4e638cd4a1dad543baff3b3f965. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:54.840307 31825 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 17. Preceding OpId from leader: term: 30 index: 18. (index mismatch)
I20260709 12:25:54.842870 11031 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 18, Last known committed idx: 17, Time since last communication: 0.001s
W20260709 12:25:54.859896 31717 consensus_peers.cc:597] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:54.864804 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800218440 after lost signal to thread 30506
W20260709 12:25:54.866129 31607 debug-util.cc:398] Leaking SignalData structure 0x7b0800351760 after lost signal to thread 31610
W20260709 12:25:54.867625 31607 debug-util.cc:398] Leaking SignalData structure 0x7b080031e500 after lost signal to thread 31777
W20260709 12:25:54.868880 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08002249e0 after lost signal to thread 31851
W20260709 12:25:54.870249 31607 debug-util.cc:398] Leaking SignalData structure 0x7b080022eae0 after lost signal to thread 32402
W20260709 12:25:54.870724 31607 debug-util.cc:398] Leaking SignalData structure 0x7b08000d17c0 after lost signal to thread 10871
I20260709 12:25:54.912379 11031 raft_consensus.cc:2955] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Committing config change with OpId 30.18: config changed from index -1 to 18, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:25:54.906111 11035 tablet_copy_client.cc:806] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:25:54.914680 11035 tablet_copy_client.cc:670] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:25:54.916343 31825 raft_consensus.cc:2955] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Committing config change with OpId 30.18: config changed from index -1 to 18, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New config: { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } } }
I20260709 12:25:54.920089 11035 tablet_copy_client.cc:538] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 12:25:54.921157 31561 catalog_manager.cc:4755] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3a0925b3d6e9448684fd239b27453b69: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:54.960124 11035 tablet_bootstrap.cc:492] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:25:54.998301 31575 catalog_manager.cc:5697] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 reported cstate change: config changed from index -1 to 18, NON_VOTER 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) added. New cstate: current_term: 30 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260709 12:25:55.062801 11042 ts_tablet_manager.cc:933] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:25:55.084419 11041 ts_tablet_manager.cc:933] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:55.145038 11042 tablet_copy_client.cc:323] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.195:44645
I20260709 12:25:55.177783 11041 tablet_copy_client.cc:323] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.194:46639
I20260709 12:25:55.185714 31837 tablet_copy_service.cc:140] P 4d48c83c761f48079bc6ea176649dda0: Received BeginTabletCopySession request for tablet bc0ece2345cd46edb250e06e8f5431d6 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:34642)
I20260709 12:25:55.186715 31837 tablet_copy_service.cc:161] P 4d48c83c761f48079bc6ea176649dda0: Beginning new tablet copy session on tablet bc0ece2345cd46edb250e06e8f5431d6 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:34642: session id = 5206fd0087404d2da964a5d985a04ff4-bc0ece2345cd46edb250e06e8f5431d6
I20260709 12:25:55.210850 31762 tablet_copy_service.cc:140] P 039d66a6c85e43c3936bf5017501b344: Received BeginTabletCopySession request for tablet e59926725a06436e9899dbbbcae3f622 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:57176)
I20260709 12:25:55.211565 31762 tablet_copy_service.cc:161] P 039d66a6c85e43c3936bf5017501b344: Beginning new tablet copy session on tablet e59926725a06436e9899dbbbcae3f622 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:57176: session id = 5206fd0087404d2da964a5d985a04ff4-e59926725a06436e9899dbbbcae3f622
I20260709 12:25:55.221982 31837 tablet_copy_source_session.cc:215] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: Tablet Copy: opened 0 blocks and 1 log segments
W20260709 12:25:55.217155 32342 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:55.242534 31717 consensus_peers.cc:597] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:55.247217 31762 tablet_copy_source_session.cc:215] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:25:55.257300 11042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc0ece2345cd46edb250e06e8f5431d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:25:55.280488 11041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e59926725a06436e9899dbbbcae3f622. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:25:55.285929 31717 consensus_peers.cc:597] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:55.333766 31717 consensus_peers.cc:597] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:55.331914 11041 tablet_copy_client.cc:806] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:25:55.343153 11041 tablet_copy_client.cc:670] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:25:55.353888 11054 ts_tablet_manager.cc:933] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4: Initiating tablet copy from peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:25:55.357681 11041 tablet_copy_client.cc:538] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:25:55.374014 11042 tablet_copy_client.cc:806] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
I20260709 12:25:55.377229 11041 tablet_bootstrap.cc:492] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:25:55.385795 11054 tablet_copy_client.cc:323] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.201.194:46639
W20260709 12:25:55.387725 31717 consensus_peers.cc:597] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:55.389782 31762 tablet_copy_service.cc:140] P 039d66a6c85e43c3936bf5017501b344: Received BeginTabletCopySession request for tablet d11aa4e638cd4a1dad543baff3b3f965 from peer 5206fd0087404d2da964a5d985a04ff4 ({username='slave'} at 127.0.0.1:57176)
I20260709 12:25:55.390472 31762 tablet_copy_service.cc:161] P 039d66a6c85e43c3936bf5017501b344: Beginning new tablet copy session on tablet d11aa4e638cd4a1dad543baff3b3f965 from peer 5206fd0087404d2da964a5d985a04ff4 at {username='slave'} at 127.0.0.1:57176: session id = 5206fd0087404d2da964a5d985a04ff4-d11aa4e638cd4a1dad543baff3b3f965
I20260709 12:25:55.389346 11042 tablet_copy_client.cc:670] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:25:55.414004 31762 tablet_copy_source_session.cc:215] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: Tablet Copy: opened 0 blocks and 1 log segments
I20260709 12:25:55.435231 11054 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d11aa4e638cd4a1dad543baff3b3f965. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:25:55.442953 32342 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:25:55.477799 11042 tablet_copy_client.cc:538] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20260709 12:25:55.541942 31717 consensus_peers.cc:597] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:25:55.552682 11042 tablet_bootstrap.cc:492] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:25:55.565716 11035 tablet_bootstrap.cc:492] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=21 overwritten=2 applied=19 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:55.570431 11035 tablet_bootstrap.cc:492] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:25:55.572439 11035 ts_tablet_manager.cc:1403] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.613s	user 0.066s	sys 0.027s
W20260709 12:25:55.578548 31717 consensus_peers.cc:597] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:55.581967 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:25:55.573812 11054 tablet_copy_client.cc:806] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 0 data blocks...
W20260709 12:25:55.604086 30503 thread.cc:527] Waited for 8000ms trying to join with diag-logger (tid 31699)
I20260709 12:25:55.614981 11054 tablet_copy_client.cc:670] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Starting download of 1 WAL segments...
I20260709 12:25:55.664692 11035 raft_consensus.cc:359] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
W20260709 12:25:55.685369 32342 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:25:55.684340 11054 tablet_copy_client.cc:538] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260709 12:25:55.717396 11054 tablet_bootstrap.cc:492] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap starting.
I20260709 12:25:55.691692 11035 raft_consensus.cc:740] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
W20260709 12:25:55.724340 32342 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:25:55.755379 11035 consensus_queue.cc:260] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 19, Last appended: 32.19, Last appended by leader: 19, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:25:55.778863 11035 ts_tablet_manager.cc:1434] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.205s	user 0.007s	sys 0.003s
W20260709 12:25:55.792732 31791 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 77396us
I20260709 12:25:55.808394 31762 tablet_copy_service.cc:342] P 039d66a6c85e43c3936bf5017501b344: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-f933d4319cf444afb9f67bd265576a31 received from {username='slave'} at 127.0.0.1:57176
I20260709 12:25:55.808905 31762 tablet_copy_service.cc:434] P 039d66a6c85e43c3936bf5017501b344: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-f933d4319cf444afb9f67bd265576a31 on tablet f933d4319cf444afb9f67bd265576a31 with peer 5206fd0087404d2da964a5d985a04ff4
W20260709 12:25:55.823390 31791 consensus_peers.cc:597] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:55.824002 31791 consensus_peers.cc:597] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:55.835824 31791 consensus_peers.cc:597] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:25:55.846939 31717 consensus_peers.cc:597] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:25:55.885555 11041 tablet_bootstrap.cc:492] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=23 overwritten=2 applied=21 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260709 12:25:55.900609 31791 consensus_peers.cc:597] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:55.903812 11041 tablet_bootstrap.cc:492] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:25:55.911798 11041 ts_tablet_manager.cc:1403] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.535s	user 0.077s	sys 0.010s
I20260709 12:25:55.915290 11041 raft_consensus.cc:359] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:25:55.916184 11041 raft_consensus.cc:740] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:25:55.916868 11041 consensus_queue.cc:260] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 21, Last appended: 36.21, Last appended by leader: 21, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:25:55.931661 32377 raft_consensus.cc:1217] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 LEARNER]: Deduplicated request from leader. Original: 36.20->[36.21-36.21]   Dedup: 36.21->[]
W20260709 12:25:55.938534 31717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111) [suppressed 478 similar messages]
I20260709 12:25:55.953980 11041 ts_tablet_manager.cc:1434] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.042s	user 0.008s	sys 0.000s
I20260709 12:25:55.986914 31762 tablet_copy_service.cc:342] P 039d66a6c85e43c3936bf5017501b344: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-e59926725a06436e9899dbbbcae3f622 received from {username='slave'} at 127.0.0.1:57176
I20260709 12:25:55.987453 31762 tablet_copy_service.cc:434] P 039d66a6c85e43c3936bf5017501b344: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-e59926725a06436e9899dbbbcae3f622 on tablet e59926725a06436e9899dbbbcae3f622 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:55.997499 11042 tablet_bootstrap.cc:492] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=24 overwritten=2 applied=22 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:55.998591 11042 tablet_bootstrap.cc:492] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:25:55.999356 11042 ts_tablet_manager.cc:1403] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.447s	user 0.053s	sys 0.033s
W20260709 12:25:56.004908 31791 consensus_peers.cc:597] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:56.004752 11042 raft_consensus.cc:359] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:25:56.005779 11042 raft_consensus.cc:740] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:25:56.006573 11042 consensus_queue.cc:260] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 22, Last appended: 31.22, Last appended by leader: 22, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
W20260709 12:25:56.047808 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:56.076678 31791 consensus_peers.cc:597] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:56.085323 32342 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:56.099874 11042 ts_tablet_manager.cc:1434] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.100s	user 0.006s	sys 0.002s
I20260709 12:25:56.112905 31837 tablet_copy_service.cc:342] P 4d48c83c761f48079bc6ea176649dda0: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-bc0ece2345cd46edb250e06e8f5431d6 received from {username='slave'} at 127.0.0.1:34642
I20260709 12:25:56.113463 31837 tablet_copy_service.cc:434] P 4d48c83c761f48079bc6ea176649dda0: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-bc0ece2345cd46edb250e06e8f5431d6 on tablet bc0ece2345cd46edb250e06e8f5431d6 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:56.151974 11054 tablet_bootstrap.cc:492] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:25:56.153122 11054 tablet_bootstrap.cc:492] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4: Bootstrap complete.
I20260709 12:25:56.153983 11054 ts_tablet_manager.cc:1403] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4: Time spent bootstrapping tablet: real 0.437s	user 0.060s	sys 0.007s
I20260709 12:25:56.160362 11054 raft_consensus.cc:359] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:25:56.161242 11054 raft_consensus.cc:740] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEARNER]: Becoming Follower/Learner. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Initialized, Role: LEARNER
I20260709 12:25:56.161918 11054 consensus_queue.cc:260] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 30.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 18 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: NON_VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: true } }
I20260709 12:25:56.177863 11054 ts_tablet_manager.cc:1434] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
W20260709 12:25:56.193959 31791 consensus_peers.cc:597] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:25:56.201478 32374 raft_consensus.cc:1217] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEARNER]: Deduplicated request from leader. Original: 30.17->[30.18-30.18]   Dedup: 30.18->[]
I20260709 12:25:56.220240 31762 tablet_copy_service.cc:342] P 039d66a6c85e43c3936bf5017501b344: Request end of tablet copy session 5206fd0087404d2da964a5d985a04ff4-d11aa4e638cd4a1dad543baff3b3f965 received from {username='slave'} at 127.0.0.1:57176
I20260709 12:25:56.220803 31762 tablet_copy_service.cc:434] P 039d66a6c85e43c3936bf5017501b344: ending tablet copy session 5206fd0087404d2da964a5d985a04ff4-d11aa4e638cd4a1dad543baff3b3f965 on tablet d11aa4e638cd4a1dad543baff3b3f965 with peer 5206fd0087404d2da964a5d985a04ff4
I20260709 12:25:56.225587 32377 raft_consensus.cc:1217] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEARNER]: Deduplicated request from leader. Original: 31.21->[31.22-31.22]   Dedup: 31.22->[]
W20260709 12:25:56.259037 31791 consensus_peers.cc:597] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:56.317489 32377 raft_consensus.cc:1217] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEARNER]: Deduplicated request from leader. Original: 32.18->[32.19-32.19]   Dedup: 32.19->[]
W20260709 12:25:56.421427 31791 consensus_peers.cc:597] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:56.473541 32342 consensus_peers.cc:597] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:56.487270 31717 consensus_peers.cc:597] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:25:56.522748 11048 raft_consensus.cc:1064] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
W20260709 12:25:56.527138 31791 consensus_peers.cc:597] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:56.525913 11048 consensus_queue.cc:237] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 36.21, Last appended by leader: 19, Current term: 36, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:56.563998 32342 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:25:56.567690 32377 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 LEARNER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:25:56.569554 11031 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
I20260709 12:25:56.581648 31825 raft_consensus.cc:1275] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 36 index: 21. Preceding OpId from leader: term: 36 index: 22. (index mismatch)
I20260709 12:25:56.583555 11048 consensus_queue.cc:1048] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
W20260709 12:25:56.593562 31791 consensus_peers.cc:597] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:56.598523 31717 consensus_peers.cc:597] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:56.604588 30503 thread.cc:527] Waited for 9000ms trying to join with diag-logger (tid 31699)
W20260709 12:25:56.614034 31717 consensus_peers.cc:597] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:56.645182 11031 raft_consensus.cc:2955] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Committing config change with OpId 36.22: config changed from index 21 to 22, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:56.647217 32374 raft_consensus.cc:2955] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Committing config change with OpId 36.22: config changed from index 21 to 22, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:56.656260 31791 consensus_peers.cc:597] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:56.670316 11063 raft_consensus.cc:1064] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:25:56.672394 31820 raft_consensus.cc:2955] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Committing config change with OpId 36.22: config changed from index 21 to 22, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:56.690105 11063 consensus_queue.cc:237] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 22, Committed index: 22, Last appended: 31.22, Last appended by leader: 20, Current term: 31, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:56.734479 31717 consensus_peers.cc:597] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:25:56.737123 32374 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEARNER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:25:56.739486 31747 raft_consensus.cc:1275] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 31 index: 22. Preceding OpId from leader: term: 31 index: 23. (index mismatch)
I20260709 12:25:56.750674 10973 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:25:56.751588 10906 consensus_queue.cc:1048] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 23, Last known committed idx: 22, Time since last communication: 0.000s
I20260709 12:25:56.755185 31576 catalog_manager.cc:5697] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 21 to 22, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 36 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:56.770202 31791 consensus_peers.cc:597] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:56.776247 11031 raft_consensus.cc:1064] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:25:56.780223 11031 consensus_queue.cc:237] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 19, Committed index: 19, Last appended: 32.19, Last appended by leader: 17, Current term: 32, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:25:56.786809 31717 consensus_peers.cc:597] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:25:56.786972 11047 raft_consensus.cc:1064] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: attempting to promote NON_VOTER 5206fd0087404d2da964a5d985a04ff4 to VOTER
I20260709 12:25:56.793411 11047 consensus_queue.cc:237] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 30.18, Last appended by leader: 16, Current term: 30, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:25:56.808530 10906 raft_consensus.cc:2955] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Committing config change with OpId 31.23: config changed from index 22 to 23, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:56.819948 31745 raft_consensus.cc:2955] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Committing config change with OpId 31.23: config changed from index 22 to 23, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:56.828754 31717 consensus_peers.cc:597] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:25:56.850768 32374 raft_consensus.cc:2955] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Committing config change with OpId 31.23: config changed from index 22 to 23, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:56.878943 32374 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEARNER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:25:56.901393 31825 raft_consensus.cc:1275] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 30 index: 18. Preceding OpId from leader: term: 30 index: 19. (index mismatch)
I20260709 12:25:56.901597 31820 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:25:56.902491 11048 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:25:56.906411 31717 consensus_peers.cc:597] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:25:56.908550 32374 raft_consensus.cc:1275] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEARNER]: Refusing update from remote peer 039d66a6c85e43c3936bf5017501b344: Log matching property violated. Preceding OpId in replica: term: 32 index: 19. Preceding OpId from leader: term: 32 index: 20. (index mismatch)
I20260709 12:25:56.921485 11048 consensus_queue.cc:1048] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 19, Last known committed idx: 18, Time since last communication: 0.001s
W20260709 12:25:56.926812 31791 consensus_peers.cc:597] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:25:56.923489 11047 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.003s
I20260709 12:25:56.927601 31576 catalog_manager.cc:5697] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 22 to 23, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 31 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260709 12:25:56.946475 10860 consensus_queue.cc:1048] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 20, Last known committed idx: 19, Time since last communication: 0.001s
W20260709 12:25:56.952122 31717 consensus_peers.cc:597] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:25:56.969187 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:57.039372 32342 consensus_peers.cc:597] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:25:57.060889 11031 raft_consensus.cc:2955] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Committing config change with OpId 32.20: config changed from index 19 to 20, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:57.063649 32377 raft_consensus.cc:2955] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Committing config change with OpId 32.20: config changed from index 19 to 20, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:57.081453 31717 consensus_peers.cc:597] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:57.093214 11086 log.cc:927] Time spent T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: Append to log took a long time: real 0.061s	user 0.000s	sys 0.000s
I20260709 12:25:57.101627 11064 raft_consensus.cc:2955] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Committing config change with OpId 30.19: config changed from index 18 to 19, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:57.117043 31820 raft_consensus.cc:2955] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Committing config change with OpId 32.20: config changed from index 19 to 20, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:57.121248 31567 catalog_manager.cc:5697] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: config changed from index 19 to 20, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 32 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
I20260709 12:25:57.109048 31825 raft_consensus.cc:2955] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Committing config change with OpId 30.19: config changed from index 18 to 19, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:57.134853 32342 consensus_peers.cc:597] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:57.139629 31561 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 54455.7us
W20260709 12:25:57.148469 32342 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:25:57.151537 31574 catalog_manager.cc:5697] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: config changed from index 18 to 19, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New cstate: current_term: 30 leader_uuid: "039d66a6c85e43c3936bf5017501b344" committed_config { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:57.159235 31561 catalog_manager.cc:4755] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3a0925b3d6e9448684fd239b27453b69: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:25:57.168603 32377 raft_consensus.cc:2955] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Committing config change with OpId 30.19: config changed from index 18 to 19, 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196) changed from NON_VOTER to VOTER. New config: { opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } } }
W20260709 12:25:57.235555 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:57.246685 31717 consensus_peers.cc:597] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:57.259807 32342 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:57.290897 31791 consensus_peers.cc:597] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:57.383751 31717 consensus_peers.cc:597] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:57.606137 30503 thread.cc:527] Waited for 10000ms trying to join with diag-logger (tid 31699)
W20260709 12:25:57.665359 31717 consensus_peers.cc:597] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:57.733372 31717 consensus_peers.cc:597] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:57.773396 31717 consensus_peers.cc:597] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:57.784333 32342 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:57.858896 31791 consensus_peers.cc:597] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:57.893669 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:57.922289 31791 consensus_peers.cc:597] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:57.954047 31717 consensus_peers.cc:597] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:25:57.967754 31717 consensus_peers.cc:597] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:58.080058 31791 consensus_peers.cc:597] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:58.086341 31791 consensus_peers.cc:597] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:58.123625 31791 consensus_peers.cc:597] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:58.142885 32342 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:58.179181 31717 consensus_peers.cc:597] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:58.297164 32342 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:58.374799 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:58.422436 31791 consensus_peers.cc:597] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:58.487982 31791 consensus_peers.cc:597] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:58.568900 31717 consensus_peers.cc:597] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:58.599284 32342 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:58.606706 30503 thread.cc:527] Waited for 11000ms trying to join with diag-logger (tid 31699)
W20260709 12:25:58.721515 31791 consensus_peers.cc:597] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:58.960302 31717 consensus_peers.cc:597] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:25:59.025918 31717 consensus_peers.cc:597] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:59.042680 31791 consensus_peers.cc:597] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:25:59.080168 32342 consensus_peers.cc:597] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:59.079443 31718 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44707.7us
W20260709 12:25:59.102965 31717 consensus_peers.cc:597] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:59.122325 31791 consensus_peers.cc:597] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:59.123238 31791 consensus_peers.cc:597] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:59.168856 31717 consensus_peers.cc:597] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:59.169427 31717 consensus_peers.cc:597] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:59.170107 31791 consensus_peers.cc:597] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:59.170747 31791 consensus_peers.cc:597] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:25:59.171214 31791 consensus_peers.cc:597] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:59.261839 31717 consensus_peers.cc:597] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:59.283725 31791 consensus_peers.cc:597] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:25:59.300406 31791 consensus_peers.cc:597] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:59.301251 32342 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:59.514281 31717 consensus_peers.cc:597] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:25:59.558643 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:59.607956 30503 thread.cc:527] Waited for 12000ms trying to join with diag-logger (tid 31699)
W20260709 12:25:59.623790 32342 consensus_peers.cc:597] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:59.693431 32342 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:25:59.801626 32342 consensus_peers.cc:597] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:59.869637 31717 consensus_peers.cc:597] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:25:59.870410 31791 consensus_peers.cc:597] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:25:59.886687 31717 consensus_peers.cc:597] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:25:59.888511 32342 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:25:59.901516 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:00.016494 31717 consensus_peers.cc:597] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:00.035246 31717 consensus_peers.cc:597] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:00.058075 31717 consensus_peers.cc:597] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:00.252764 31717 consensus_peers.cc:597] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:00.321558 32342 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:00.457101 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:00.478101 31791 consensus_peers.cc:597] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:00.498904 31717 consensus_peers.cc:597] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:00.609205 30503 thread.cc:527] Waited for 13000ms trying to join with diag-logger (tid 31699)
W20260709 12:26:00.613715 31717 consensus_peers.cc:597] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:00.634832 31791 consensus_peers.cc:597] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:00.649550 32342 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:00.650722 31791 consensus_peers.cc:597] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:00.771301 31791 consensus_peers.cc:597] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:00.774950 31717 consensus_peers.cc:597] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:00.873258 31791 consensus_peers.cc:597] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:00.965308 31717 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111) [suppressed 476 similar messages]
W20260709 12:26:00.974174 31791 consensus_peers.cc:597] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:00.984589 31791 consensus_peers.cc:597] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:01.031019 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:01.053074 31717 consensus_peers.cc:597] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:01.238256 31791 consensus_peers.cc:597] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:01.273643 32342 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:01.364164 32342 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:01.376474 31791 consensus_peers.cc:597] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:01.386782 31717 consensus_peers.cc:597] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:01.425093 31717 consensus_peers.cc:597] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:01.446911 31791 consensus_peers.cc:597] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:01.463851 31561 catalog_manager.cc:4755] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3a0925b3d6e9448684fd239b27453b69: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:01.610054 30503 thread.cc:527] Waited for 14000ms trying to join with diag-logger (tid 31699)
W20260709 12:26:02.005431 31717 consensus_peers.cc:597] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:02.190117 31717 consensus_peers.cc:597] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:02.205721 31717 consensus_peers.cc:597] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:02.207013 31717 consensus_peers.cc:597] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:26:02.211968 30503 tablet_server.cc:196] TabletServer@127.29.201.193:0 shutdown complete.
W20260709 12:26:02.299274 31791 consensus_peers.cc:597] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:02.304795 31791 consensus_peers.cc:597] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:02.305497 31791 consensus_peers.cc:597] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:02.306017 31791 consensus_peers.cc:597] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:02.306542 31791 consensus_peers.cc:597] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:02.307077 31791 consensus_peers.cc:597] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:02.337031 32342 consensus_peers.cc:597] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:02.383744 32342 consensus_peers.cc:597] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:02.390301 32342 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:02.501675 31717 consensus_peers.cc:597] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:02.513523 31791 consensus_peers.cc:597] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:02.593083 31717 consensus_peers.cc:597] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:02.608937 32342 consensus_peers.cc:597] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:02.617863 30503 tablet_server.cc:179] TabletServer@127.29.201.194:0 shutting down...
W20260709 12:26:02.651546 31791 consensus_peers.cc:597] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:02.656735 31791 consensus_peers.cc:597] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:02.685412 31791 consensus_peers.cc:597] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:02.689205 31717 consensus_peers.cc:597] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:02.702595 32342 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:02.710698 32342 consensus_peers.cc:597] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:02.711464 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:02.722847 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:02.723773 32342 consensus_peers.cc:597] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:02.724331 32342 consensus_peers.cc:597] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:02.724905 32342 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:02.727059 32342 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:02.779371 32342 consensus_peers.cc:597] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:02.785400 31774 debug-util.cc:398] Leaking SignalData structure 0x7b0800354940 after lost signal to thread 30506
W20260709 12:26:02.786773 31774 debug-util.cc:398] Leaking SignalData structure 0x7b0800354b80 after lost signal to thread 31610
W20260709 12:26:02.788188 31774 debug-util.cc:398] Leaking SignalData structure 0x7b0800165180 after lost signal to thread 31777
W20260709 12:26:02.789387 31774 debug-util.cc:398] Leaking SignalData structure 0x7b08001021a0 after lost signal to thread 31851
W20260709 12:26:02.790799 31774 debug-util.cc:398] Leaking SignalData structure 0x7b080032a2c0 after lost signal to thread 32402
W20260709 12:26:02.796350 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:02.797420 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:02.796433 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:02.805938 30503 tablet_replica.cc:333] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.807225 30503 raft_consensus.cc:2243] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:02.808425 30503 raft_consensus.cc:2272] T 35c247463bef4f77990becbf6089be7d P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:02.819958 30503 tablet_replica.cc:333] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.821055 30503 raft_consensus.cc:2243] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:02.822096 30503 raft_consensus.cc:2272] T f79ad11703094c998ce1b3905e5c9ff6 P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:02.825476 30503 tablet_replica.cc:333] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
W20260709 12:26:02.825914 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:02.830793 30503 raft_consensus.cc:2243] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:02.831740 30503 raft_consensus.cc:2272] T c22bf85ce45841fb80e303e2746080da P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:02.834712 30503 tablet_replica.cc:333] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.839452 30503 raft_consensus.cc:2243] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:02.848130 30503 raft_consensus.cc:2272] T 436783e8e1cc49edbad793a31645e2d6 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:02.834884 32342 consensus_peers.cc:597] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:02.852046 30503 tablet_replica.cc:333] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
W20260709 12:26:02.852413 32342 consensus_peers.cc:597] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:02.853096 30503 raft_consensus.cc:2243] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 40 LEADER]: Raft consensus shutting down.
I20260709 12:26:02.854540 30503 raft_consensus.cc:2272] T f6404cb9c2174326b06cbbb0d27d43f2 P 039d66a6c85e43c3936bf5017501b344 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:02.857661 30503 tablet_replica.cc:333] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.860843 30503 raft_consensus.cc:2243] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:02.862754 30503 raft_consensus.cc:2272] T 2c162dbe81714319bfeccd9d7cb7bfea P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:02.880640 30503 tablet_replica.cc:333] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.881665 30503 raft_consensus.cc:2243] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:02.882548 30503 raft_consensus.cc:2272] T ffab0b886efd41099d99c20339a3c4ae P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:02.885579 30503 tablet_replica.cc:333] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.886597 30503 raft_consensus.cc:2243] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:02.887912 30503 raft_consensus.cc:2272] T 20650b4ef2514a05bde5dfd38a379f32 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:02.888600 31717 consensus_peers.cc:597] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:26:02.896436 30503 tablet_replica.cc:333] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.897655 30503 raft_consensus.cc:2243] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:02.898874 30503 raft_consensus.cc:2272] T 7523cff6f9fb44b5ba8fc04726d85b65 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:02.902238 30503 tablet_replica.cc:333] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.907436 30503 raft_consensus.cc:2243] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:02.908321 30503 raft_consensus.cc:2272] T c34755b3b9d040f69be353e35d7431e7 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:02.917732 30503 tablet_replica.cc:333] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.918752 30503 raft_consensus.cc:2243] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:02.919531 30503 raft_consensus.cc:2272] T 6e7ad549735d4871af75658a91bb98c4 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:02.931015 30503 tablet_replica.cc:333] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.931979 30503 raft_consensus.cc:2243] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:02.932963 30503 raft_consensus.cc:2272] T f84a67c6dba04300a5240cc80c94718a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:02.938856 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:26:02.939213 30503 tablet_replica.cc:333] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.948347 30503 raft_consensus.cc:2243] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:26:02.949867 30503 raft_consensus.cc:2272] T bc55086b81374990afb970c972568882 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:02.950393 31791 consensus_peers.cc:597] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:02.952777 30503 tablet_replica.cc:333] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.953797 30503 raft_consensus.cc:2243] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:02.955312 30503 raft_consensus.cc:2272] T bc0ece2345cd46edb250e06e8f5431d6 P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:02.958384 30503 tablet_replica.cc:333] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.972345 30503 raft_consensus.cc:2243] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:02.973524 30503 raft_consensus.cc:2272] T a22e63b9ca824c92ae9ba374d2f3242a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:02.976609 30503 tablet_replica.cc:333] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.983422 30503 raft_consensus.cc:2243] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:02.982329 31791 consensus_peers.cc:597] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:02.985824 30503 raft_consensus.cc:2272] T 181ddae071d549aab328e3c1d8553734 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:02.988991 30503 tablet_replica.cc:333] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:02.989912 30503 raft_consensus.cc:2243] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:02.992844 30503 raft_consensus.cc:2272] T 6821bb8779e64160887e3195aa5078a1 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:02.991369 31791 consensus_peers.cc:597] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:03.002882 30503 tablet_replica.cc:333] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.003847 30503 raft_consensus.cc:2243] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.005331 30503 raft_consensus.cc:2272] T 354c519f070b453dadcdef3f0d835c69 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.008531 30503 tablet_replica.cc:333] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
W20260709 12:26:03.018473 31791 consensus_peers.cc:597] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:03.021281 30503 raft_consensus.cc:2243] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.022369 30503 raft_consensus.cc:2272] T d0a3795ad4d543f0998c3834aac7be45 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:03.036980 31791 consensus_peers.cc:597] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:03.047334 31717 consensus_peers.cc:597] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:03.063160 31791 consensus_peers.cc:597] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:03.073834 30503 tablet_replica.cc:333] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.075142 30503 raft_consensus.cc:2243] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.079931 30503 raft_consensus.cc:2272] T daa53c7009fe4881956dcfb5a20febde P 039d66a6c85e43c3936bf5017501b344 [term 33 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:03.083978 31791 consensus_peers.cc:597] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:03.090330 30503 tablet_replica.cc:333] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
W20260709 12:26:03.091607 32342 consensus_peers.cc:597] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:03.097944 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:03.097124 30503 raft_consensus.cc:2243] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.110989 30503 raft_consensus.cc:2272] T b416bf5304744b14b72501e697b8c951 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:03.112746 31791 consensus_peers.cc:597] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:03.115228 31791 consensus_peers.cc:597] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:03.122239 31791 consensus_peers.cc:597] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:03.123139 30503 tablet_replica.cc:333] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.125646 30503 raft_consensus.cc:2243] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 40 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.129963 30503 raft_consensus.cc:2272] T 04bdf51be7b94825a28028a5cfb96245 P 039d66a6c85e43c3936bf5017501b344 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.141912 30503 tablet_replica.cc:333] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.146878 30503 raft_consensus.cc:2243] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.149072 30503 raft_consensus.cc:2272] T e59926725a06436e9899dbbbcae3f622 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:03.145789 31791 consensus_peers.cc:597] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:03.149932 31791 consensus_peers.cc:597] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:03.150583 31791 consensus_peers.cc:597] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:03.171833 31791 consensus_peers.cc:597] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:03.174069 30503 tablet_replica.cc:333] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.178669 30503 raft_consensus.cc:2243] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.180765 30503 raft_consensus.cc:2272] T 49cc564ccd0d411fa79831efa24f87bb P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:03.203397 31717 consensus_peers.cc:597] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:03.189301 30503 tablet_replica.cc:333] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.222548 30503 raft_consensus.cc:2243] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.228359 30503 raft_consensus.cc:2272] T 804df7046b7545d9890885896a333a3e P 039d66a6c85e43c3936bf5017501b344 [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:03.225885 31791 consensus_peers.cc:597] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:03.237376 32342 consensus_peers.cc:597] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:03.257521 30503 tablet_replica.cc:333] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.258769 30503 raft_consensus.cc:2243] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.260533 30503 raft_consensus.cc:2272] T 992b9a77f3124e49a9bb783e08a9e856 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.263983 30503 tablet_replica.cc:333] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.264878 30503 raft_consensus.cc:2243] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.265791 30503 raft_consensus.cc:2272] T f2e7a775ae0d44108be0965c85245ecf P 039d66a6c85e43c3936bf5017501b344 [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:03.272533 32342 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:03.282341 31791 consensus_peers.cc:597] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:03.291904 30503 tablet_replica.cc:333] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.293192 30503 raft_consensus.cc:2243] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.294049 30503 raft_consensus.cc:2272] T 6499e78dd87c4e05be32f5681957a6b0 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:03.294307 31717 consensus_peers.cc:597] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:26:03.308593 30503 tablet_replica.cc:333] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.309810 30503 raft_consensus.cc:2243] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.320022 30503 raft_consensus.cc:2272] T 55f4b2e07ada4e06a285803fcbfd6344 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.323009 30503 tablet_replica.cc:333] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.323796 30503 raft_consensus.cc:2243] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.325776 30503 raft_consensus.cc:2272] T 3a0925b3d6e9448684fd239b27453b69 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.329663 30503 tablet_replica.cc:333] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.333778 30503 raft_consensus.cc:2243] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.338783 30503 raft_consensus.cc:2272] T 5bb2dc5276714f75b6e3e61ca9275f84 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.342005 30503 tablet_replica.cc:333] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.342941 30503 raft_consensus.cc:2243] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.344292 30503 raft_consensus.cc:2272] T 292851e482024b3a8dcd05c5dab04534 P 039d66a6c85e43c3936bf5017501b344 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.353468 30503 tablet_replica.cc:333] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.354300 30503 raft_consensus.cc:2243] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.355813 30503 raft_consensus.cc:2272] T 8725620974f7465eb0371f48761a1625 P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.358827 30503 tablet_replica.cc:333] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.359700 30503 raft_consensus.cc:2243] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.360363 30503 raft_consensus.cc:2272] T e59f73cf80df4c20b6c17715783f2121 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.363394 30503 tablet_replica.cc:333] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.364617 30503 raft_consensus.cc:2243] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.365816 30503 raft_consensus.cc:2272] T 5d0d0402170d41b79dc30f93bee98766 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.368925 30503 tablet_replica.cc:333] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.369684 30503 raft_consensus.cc:2243] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.370666 30503 raft_consensus.cc:2272] T d30338be26ae44ff8e627f7b8a99036c P 039d66a6c85e43c3936bf5017501b344 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.374620 30503 tablet_replica.cc:333] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.375407 30503 raft_consensus.cc:2243] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.376200 30503 raft_consensus.cc:2272] T 945fd223ac334bce88917d1b9b4c72e1 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.379201 30503 tablet_replica.cc:333] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.379997 30503 raft_consensus.cc:2243] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.381058 30503 raft_consensus.cc:2272] T 70ada7cfaa154c8c9cc927936b46c30a P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.383901 30503 tablet_replica.cc:333] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.384676 30503 raft_consensus.cc:2243] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.385414 30503 raft_consensus.cc:2272] T 81c14d7d4fc14386b173a86e9623eb25 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.388088 30503 tablet_replica.cc:333] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.388778 30503 raft_consensus.cc:2243] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 40 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.389462 30503 raft_consensus.cc:2272] T 504cb3d29e9347fe98f010a3707a55cc P 039d66a6c85e43c3936bf5017501b344 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.392230 30503 tablet_replica.cc:333] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.392957 30503 raft_consensus.cc:2243] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.394308 30503 raft_consensus.cc:2272] T 9c849da11b414111b8461d6ae06c15b3 P 039d66a6c85e43c3936bf5017501b344 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.397316 30503 tablet_replica.cc:333] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.399264 30503 raft_consensus.cc:2243] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.399994 30503 raft_consensus.cc:2272] T 1e27b968c91a4906aec537421f481269 P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.405958 30503 tablet_replica.cc:333] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.406908 30503 raft_consensus.cc:2243] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.408387 30503 raft_consensus.cc:2272] T ab8ea7b957bd41e0a2e7967cbd101f36 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.411723 30503 tablet_replica.cc:333] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.412623 30503 raft_consensus.cc:2243] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.414125 30503 raft_consensus.cc:2272] T ef31b84f462f424fa6162b810ea8c8a9 P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.417392 30503 tablet_replica.cc:333] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.418179 30503 raft_consensus.cc:2243] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.419785 30503 raft_consensus.cc:2272] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 039d66a6c85e43c3936bf5017501b344 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.423007 30503 tablet_replica.cc:333] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.423748 30503 raft_consensus.cc:2243] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.425693 30503 raft_consensus.cc:2272] T d11aa4e638cd4a1dad543baff3b3f965 P 039d66a6c85e43c3936bf5017501b344 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.428792 30503 tablet_replica.cc:333] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.429565 30503 raft_consensus.cc:2243] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.431336 30503 raft_consensus.cc:2272] T b043bbe90410471da70e6b00856b5c8d P 039d66a6c85e43c3936bf5017501b344 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.436703 30503 tablet_replica.cc:333] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.437388 30503 raft_consensus.cc:2243] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:03.439268 30503 raft_consensus.cc:2272] T f933d4319cf444afb9f67bd265576a31 P 039d66a6c85e43c3936bf5017501b344 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:03.442430 30503 tablet_replica.cc:333] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344: stopping tablet replica
I20260709 12:26:03.443216 30503 raft_consensus.cc:2243] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:03.444006 30503 raft_consensus.cc:2272] T 553d0b2dddba47779b0219c71ed9ce83 P 039d66a6c85e43c3936bf5017501b344 [term 35 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:03.606325 31791 consensus_peers.cc:597] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:03.617120 31791 consensus_peers.cc:597] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:03.735565 31791 consensus_peers.cc:597] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:03.781458 31791 consensus_peers.cc:597] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:03.809561 32342 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:03.885223 31791 consensus_peers.cc:597] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20260709 12:26:03.885901 31791 consensus_peers.cc:597] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:03.999377 32342 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:04.003825 11103 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.004621 11103 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:04.019253 11103 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:04.021018 31791 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:04.032851 31791 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.036487 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.037703 11103 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:04.047878 11117 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.048053 11131 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.048646 11117 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:04.048813 11131 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:04.051327 11131 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:26:04.052407 11117 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:26:04.052735 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
W20260709 12:26:04.069306 32342 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:04.070194 32342 leader_election.cc:336] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.072522 32342 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:26:04.075106 11131 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:04.076416 32342 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.083729 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.086768 11131 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
I20260709 12:26:04.097074 11103 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.097800 11103 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:04.107923 11103 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:26:04.111188 32342 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:26:04.114296 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:26:04.115031 32377 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:26:04.116721 31792 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:26:04.117925 11131 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.118538 11131 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:04.120900 11131 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:26:04.121556 11103 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election won for term 31
I20260709 12:26:04.122051 11103 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.122448 11103 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:04.190253 31791 leader_election.cc:336] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.138409 11103 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Starting leader election with config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:26:04.200678 32342 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:04.208599 11103 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:26:04.230273 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20650b4ef2514a05bde5dfd38a379f32" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 21 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4"
I20260709 12:26:04.231040 32377 raft_consensus.cc:3060] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Advancing to term 31
W20260709 12:26:04.232654 32342 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.233165 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.234947 11131 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:04.236860 11103 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.237447 11103 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:04.240366 32377 raft_consensus.cc:2468] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 31.
I20260709 12:26:04.247426 11103 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:26:04.251984 31791 consensus_peers.cc:597] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:26:04.275707 31792 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:26:04.276707 11103 raft_consensus.cc:2804] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Leader election won for term 31
I20260709 12:26:04.277475 11103 raft_consensus.cc:697] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Becoming Leader. State: Replica: 4d48c83c761f48079bc6ea176649dda0, State: Running, Role: LEADER
I20260709 12:26:04.278517 11103 consensus_queue.cc:237] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 21, Committed index: 21, Last appended: 30.21, Last appended by leader: 21, Current term: 31, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:04.289557 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
W20260709 12:26:04.307371 31791 consensus_peers.cc:597] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:04.316524 11134 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.317302 11134 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:04.320676 11134 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
W20260709 12:26:04.323488 31791 leader_election.cc:336] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.338990 31576 catalog_manager.cc:5697] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 reported cstate change: term changed from 30 to 31, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195). New cstate: current_term: 31 leader_uuid: "4d48c83c761f48079bc6ea176649dda0" committed_config { opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:04.356473 31791 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:04.357087 31791 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.360051 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:26:04.362339 11103 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:04.363523 11103 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.364086 11103 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:04.366962 11103 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:04.369091 31791 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
W20260709 12:26:04.387357 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:26:04.397004 11103 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.397686 11103 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:26:04.399012 31791 leader_election.cc:336] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.399626 31791 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, 5206fd0087404d2da964a5d985a04ff4
I20260709 12:26:04.400424 11134 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.400967 11134 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:04.404747 11134 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:26:04.400055 11103 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:26:04.405627 11127 raft_consensus.cc:2749] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:26:04.410140 31791 leader_election.cc:336] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:04.411504 31791 leader_election.cc:336] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.411919 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.414000 11134 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:04.422847 31791 leader_election.cc:336] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:04.423425 31791 leader_election.cc:336] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:04.424263 31791 leader_election.cc:336] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.424676 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.425884 11127 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:04.425925 31791 leader_election.cc:336] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.426729 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.427582 11127 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:26:04.431720 31791 consensus_peers.cc:597] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:04.458801 11101 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.459412 11101 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:04.461549 11101 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:26:04.463268 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:26:04.464047 31818 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 40 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 40.
I20260709 12:26:04.466166 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:26:04.467248 11101 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Leader pre-election won for term 41
I20260709 12:26:04.467746 11101 raft_consensus.cc:493] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Starting leader election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.468217 11101 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Advancing to term 41
W20260709 12:26:04.473995 32342 leader_election.cc:336] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.477200 11127 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.477967 11127 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:04.480602 11127 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:26:04.485392 11101 raft_consensus.cc:515] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 41 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:04.489102 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04bdf51be7b94825a28028a5cfb96245" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 41 candidate_status { last_received { term: 40 index: 26 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0"
I20260709 12:26:04.489713 31818 raft_consensus.cc:3060] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 40 FOLLOWER]: Advancing to term 41
I20260709 12:26:04.492484 11101 leader_election.cc:290] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 election: Requested vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:26:04.497293 31818 raft_consensus.cc:2468] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 41 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 41.
I20260709 12:26:04.498963 32343 leader_election.cc:304] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 
I20260709 12:26:04.500046 11101 raft_consensus.cc:2804] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 41 FOLLOWER]: Leader election won for term 41
I20260709 12:26:04.500643 11101 raft_consensus.cc:697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 41 LEADER]: Becoming Leader. State: Replica: 5206fd0087404d2da964a5d985a04ff4, State: Running, Role: LEADER
I20260709 12:26:04.504796 11131 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:26:04.509107 31791 leader_election.cc:336] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:04.505471 11131 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:26:04.511507 31791 leader_election.cc:336] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.512033 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.512447 11131 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:26:04.514391 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:26:04.515094 31818 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:26:04.515307 11127 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:04.503245 11101 consensus_queue.cc:237] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 26, Committed index: 26, Last appended: 40.26, Last appended by leader: 26, Current term: 41, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:26:04.521256 32342 leader_election.cc:336] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
W20260709 12:26:04.532056 31791 consensus_peers.cc:597] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:04.549974 31576 catalog_manager.cc:5697] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 reported cstate change: term changed from 40 to 41, leader changed from 039d66a6c85e43c3936bf5017501b344 (127.29.201.194) to 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196). New cstate: current_term: 41 leader_uuid: "5206fd0087404d2da964a5d985a04ff4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260709 12:26:04.551697 32342 consensus_peers.cc:597] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20260709 12:26:04.552384 32342 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:04.553613 32342 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.554073 32342 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.555154 11101 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:04.564491 11127 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.565166 11127 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:04.567699 11127 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:26:04.575165 11127 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.575806 11127 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:04.578217 11127 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:04.586732 30503 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 31774)
W20260709 12:26:04.589099 31791 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:04.590057 31791 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.590468 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.591317 11103 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:26:04.597466 31791 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:04.598461 31791 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.598877 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.599666 11103 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:04.605171 11127 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.605809 11127 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:26:04.607616 31791 consensus_peers.cc:597] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:04.608210 31791 consensus_peers.cc:597] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:04.608263 11127 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:04.618567 31791 leader_election.cc:336] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:04.619422 31791 leader_election.cc:336] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.619858 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.620700 11127 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
W20260709 12:26:04.641569 32342 consensus_peers.cc:597] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:04.642228 11103 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.642865 11103 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:04.645359 11103 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:04.651103 31791 leader_election.cc:336] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:04.652098 31791 leader_election.cc:336] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.652537 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.653791 11103 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:04.657232 11103 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.657785 11103 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:04.662132 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:26:04.663666 11101 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.664283 11101 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:04.668510 11103 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:26:04.669433 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:26:04.669487 11101 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:26:04.673473 31791 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:04.673978 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 5206fd0087404d2da964a5d985a04ff4, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:26:04.674201 32342 leader_election.cc:336] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:04.674671 32342 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:26:04.674850 31791 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.675414 11101 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:04.676087 11103 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:04.676726 11127 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.682868 11127 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:04.684934 11127 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:04.675489 32342 leader_election.cc:336] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.694248 11127 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.694875 11127 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:04.697146 11127 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:04.700634 31791 leader_election.cc:336] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
W20260709 12:26:04.704438 31791 leader_election.cc:336] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:04.704924 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.708560 11127 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:26:04.718291 31791 leader_election.cc:336] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:04.719201 31791 leader_election.cc:336] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.719615 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.720391 11127 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:26:04.788569 31791 consensus_peers.cc:597] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:04.794301 31791 consensus_peers.cc:597] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:26:04.812068 11127 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.812740 11127 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:04.815081 11127 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:04.823737 31791 leader_election.cc:336] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:04.831157 32342 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:04.833771 31791 leader_election.cc:336] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.834329 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.836126 11127 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:26:04.860044 31791 consensus_peers.cc:597] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:04.870648 11127 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.871311 11127 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:04.876084 11127 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:26:04.876629 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:26:04.877408 32377 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
W20260709 12:26:04.886184 31791 leader_election.cc:336] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:04.887163 31791 leader_election.cc:336] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:04.887619 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.888429 11127 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:04.900321 11101 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:04.900996 11101 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:04.904449 11101 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:26:04.906960 32342 consensus_peers.cc:597] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:04.909803 31822 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:26:04.910712 31822 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 32.
I20260709 12:26:04.912328 32377 raft_consensus.cc:1275] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Refusing update from remote peer 4d48c83c761f48079bc6ea176649dda0: Log matching property violated. Preceding OpId in replica: term: 30 index: 21. Preceding OpId from leader: term: 31 index: 22. (index mismatch)
I20260709 12:26:04.915062 11127 consensus_queue.cc:1048] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 22, Last known committed idx: 21, Time since last communication: 0.001s
W20260709 12:26:04.918751 32342 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
W20260709 12:26:04.919661 32342 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:04.920120 32342 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:04.920946 11101 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:04.989058 31822 raft_consensus.cc:1275] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 41 FOLLOWER]: Refusing update from remote peer 5206fd0087404d2da964a5d985a04ff4: Log matching property violated. Preceding OpId in replica: term: 40 index: 26. Preceding OpId from leader: term: 41 index: 27. (index mismatch)
I20260709 12:26:04.990669 11101 consensus_queue.cc:1048] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 27, Last known committed idx: 26, Time since last communication: 0.001s
W20260709 12:26:05.007732 31791 consensus_peers.cc:597] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.050791 31791 consensus_peers.cc:597] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.070986 32342 consensus_peers.cc:597] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:05.077308 32342 consensus_peers.cc:597] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.120213 31791 consensus_peers.cc:597] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:05.126873 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:05.127434 31791 consensus_peers.cc:597] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.180207 32342 consensus_peers.cc:597] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.224633 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.236330 31791 consensus_peers.cc:597] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.284089 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:05.355612 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.359711 32342 consensus_peers.cc:597] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.367537 32342 consensus_peers.cc:597] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.389907 32342 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.444300 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.464843 31791 consensus_peers.cc:597] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.467202 32342 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:05.476388 32342 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:05.498986 32342 consensus_peers.cc:597] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.505906 31791 consensus_peers.cc:597] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.516486 32342 consensus_peers.cc:597] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.519691 31791 consensus_peers.cc:597] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.532732 31791 consensus_peers.cc:597] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.545873 31791 consensus_peers.cc:597] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.553851 32342 consensus_peers.cc:597] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.576073 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:05.587302 30503 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 31774)
W20260709 12:26:05.640347 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.668506 31791 consensus_peers.cc:597] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.674986 31791 consensus_peers.cc:597] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.694414 31791 consensus_peers.cc:597] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:05.702227 11127 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:05.702919 11127 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:05.705176 11127 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:05.713819 31791 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
W20260709 12:26:05.715173 31791 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:05.716289 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:05.718415 11127 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:26:05.732653 31791 consensus_peers.cc:597] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.733361 31791 consensus_peers.cc:597] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.744673 31791 consensus_peers.cc:597] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:05.766495 31791 consensus_peers.cc:597] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.818221 31791 consensus_peers.cc:597] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:05.828244 31791 consensus_peers.cc:597] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.904003 32342 consensus_peers.cc:597] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:05.970554 31791 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111) [suppressed 600 similar messages]
I20260709 12:26:05.988799 11127 raft_consensus.cc:493] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:05.989568 11127 raft_consensus.cc:515] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:26:06.002172 31791 leader_election.cc:336] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:05.999845 11127 leader_election.cc:290] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:06.013336 31791 leader_election.cc:336] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:06.013983 31791 leader_election.cc:304] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.014911 11127 raft_consensus.cc:2749] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:26:06.038087 31791 consensus_peers.cc:597] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:06.071473 32342 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:06.108846 31791 consensus_peers.cc:597] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:06.128743 31791 consensus_peers.cc:597] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:06.173213 11127 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:06.173789 11127 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:06.176024 11127 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:06.185395 31791 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:06.185599 11127 raft_consensus.cc:493] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:26:06.186411 31791 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:06.186271 11127 raft_consensus.cc:515] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:06.186929 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.188781 11127 leader_election.cc:290] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:26:06.192291 11127 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:26:06.214947 31791 leader_election.cc:336] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:06.216480 31791 consensus_peers.cc:597] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:06.222507 31791 leader_election.cc:336] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:06.223068 31791 leader_election.cc:304] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.223862 11149 raft_consensus.cc:2749] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:26:06.231685 31791 consensus_peers.cc:597] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20260709 12:26:06.245227 11149 raft_consensus.cc:493] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:06.245899 11149 raft_consensus.cc:515] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:06.248237 11149 leader_election.cc:290] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:06.257174 31791 leader_election.cc:336] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:06.258881 31791 leader_election.cc:336] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:06.259400 31791 leader_election.cc:304] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 36 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.260231 11127 raft_consensus.cc:2749] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Leader pre-election lost for term 36. Reason: could not achieve majority
I20260709 12:26:06.265681 11127 raft_consensus.cc:493] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:06.266211 11127 raft_consensus.cc:515] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:06.268229 11127 leader_election.cc:290] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:06.287446 31791 leader_election.cc:336] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:06.288511 31791 leader_election.cc:336] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:06.289047 31791 leader_election.cc:304] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.292652 11150 raft_consensus.cc:2749] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
W20260709 12:26:06.314975 31791 consensus_peers.cc:597] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:06.323714 11148 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:06.324342 11148 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:06.337178 11147 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:06.337754 11147 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:06.339937 11148 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:26:06.348649 11147 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
I20260709 12:26:06.362440 11127 raft_consensus.cc:493] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:06.363138 11127 raft_consensus.cc:515] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:06.365366 11127 leader_election.cc:290] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:06.367883 32342 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:06.369724 11150 raft_consensus.cc:493] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
W20260709 12:26:06.373320 31791 leader_election.cc:336] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:06.375097 11150 raft_consensus.cc:515] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:26:06.376649 31791 leader_election.cc:336] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:06.377117 31791 leader_election.cc:304] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.388621 11127 raft_consensus.cc:2749] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
I20260709 12:26:06.377280 11150 leader_election.cc:290] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:06.379385 32342 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:06.390846 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:26:06.381299 32343 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:44645, user_credentials={real_user=slave}} blocked reactor thread for 44981.1us
I20260709 12:26:06.398839 31818 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e59926725a06436e9899dbbbcae3f622" candidate_uuid: "5206fd0087404d2da964a5d985a04ff4" candidate_term: 37 candidate_status { last_received { term: 36 index: 22 } } ignore_live_leader: false dest_uuid: "4d48c83c761f48079bc6ea176649dda0" is_pre_election: true
I20260709 12:26:06.399520 31818 raft_consensus.cc:2468] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5206fd0087404d2da964a5d985a04ff4 in term 36.
I20260709 12:26:06.398833 11147 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:26:06.402137 31791 leader_election.cc:336] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
W20260709 12:26:06.402786 32342 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
W20260709 12:26:06.404021 32342 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:06.404572 32342 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.405457 11152 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:06.407886 11150 raft_consensus.cc:493] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:06.408509 11150 raft_consensus.cc:515] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:26:06.402858 31791 leader_election.cc:336] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:06.410739 11150 leader_election.cc:290] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:26:06.417613 11152 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:06.418246 11152 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:06.423776 11152 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:26:06.410899 31791 leader_election.cc:304] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.427546 11150 raft_consensus.cc:2749] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
W20260709 12:26:06.438340 32342 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:06.441857 31791 leader_election.cc:336] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:06.448792 31791 leader_election.cc:336] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:06.449321 31791 leader_election.cc:304] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.452819 11150 raft_consensus.cc:2749] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:26:06.457180 32342 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
W20260709 12:26:06.458518 32342 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:06.463799 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.468189 11152 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
I20260709 12:26:06.501266 11150 raft_consensus.cc:493] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:06.501853 11150 raft_consensus.cc:515] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
W20260709 12:26:06.502188 31791 consensus_peers.cc:597] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260709 12:26:06.504155 11150 leader_election.cc:290] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:06.539361 31791 leader_election.cc:336] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:06.543488 31791 leader_election.cc:336] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:06.544039 31791 leader_election.cc:304] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.545753 11150 raft_consensus.cc:2749] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:06.550067 11150 raft_consensus.cc:493] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:06.550686 11150 raft_consensus.cc:515] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:06.551115 11127 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:06.551620 11127 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:06.552799 11150 leader_election.cc:290] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:26:06.553992 11127 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:26:06.560639 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d11aa4e638cd4a1dad543baff3b3f965" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 31 candidate_status { last_received { term: 30 index: 19 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:26:06.561404 32377 raft_consensus.cc:2468] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 30.
I20260709 12:26:06.582610 11127 raft_consensus.cc:493] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:06.583191 11127 raft_consensus.cc:515] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:06.585110 11127 leader_election.cc:290] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:06.601461 31791 leader_election.cc:336] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:06.611104 31791 leader_election.cc:336] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:06.618935 31791 leader_election.cc:336] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:06.619421 31791 leader_election.cc:304] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.629382 11127 raft_consensus.cc:2749] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:06.630753 31791 leader_election.cc:336] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:06.631317 31791 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.632055 11127 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
W20260709 12:26:06.655097 31791 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
W20260709 12:26:06.664430 31791 leader_election.cc:336] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:06.664927 31791 leader_election.cc:304] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.683439 11127 raft_consensus.cc:2749] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:26:06.728622 31791 consensus_peers.cc:597] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:06.743064 32342 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20260709 12:26:06.782740 11153 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:06.785027 11153 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:26:06.782876 32342 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:26:06.794991 11153 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 5206fd0087404d2da964a5d985a04ff4 (127.29.201.196:44855)
I20260709 12:26:06.799360 32377 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f933d4319cf444afb9f67bd265576a31" candidate_uuid: "4d48c83c761f48079bc6ea176649dda0" candidate_term: 33 candidate_status { last_received { term: 32 index: 20 } } ignore_live_leader: false dest_uuid: "5206fd0087404d2da964a5d985a04ff4" is_pre_election: true
I20260709 12:26:06.800175 32377 raft_consensus.cc:2468] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d48c83c761f48079bc6ea176649dda0 in term 32.
W20260709 12:26:06.819207 31791 consensus_peers.cc:597] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:06.825994 32342 consensus_peers.cc:597] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:06.854322 31791 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
W20260709 12:26:06.863173 31791 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:06.863680 31791 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0, 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:06.867079 11153 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:06.891136 31791 consensus_peers.cc:597] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:06.895993 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:06.979233 31791 consensus_peers.cc:597] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:07.041985 32342 consensus_peers.cc:597] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:07.094007 31791 consensus_peers.cc:597] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20260709 12:26:07.122669 31791 consensus_peers.cc:597] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20260709 12:26:07.151283 30503 tablet_server.cc:196] TabletServer@127.29.201.194:0 shutdown complete.
W20260709 12:26:07.151019 31791 consensus_peers.cc:597] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20260709 12:26:07.170557 31791 consensus_peers.cc:597] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260709 12:26:07.329051 11153 raft_consensus.cc:493] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:07.329655 11153 raft_consensus.cc:515] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:07.334573 11153 leader_election.cc:290] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:07.347613 31791 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:07.348527 31791 consensus_peers.cc:597] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:07.348982 31791 leader_election.cc:336] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:07.350857 31791 leader_election.cc:304] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:07.352694 11153 raft_consensus.cc:2749] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:07.438757 30503 tablet_server.cc:179] TabletServer@127.29.201.195:0 shutting down...
W20260709 12:26:07.454414 32342 consensus_peers.cc:597] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:07.471204 32343 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.504420 32343 consensus_peers.cc:597] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.553586 32343 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.553856 31791 consensus_peers.cc:597] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:07.563210 32343 consensus_peers.cc:597] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.572499 31791 consensus_peers.cc:597] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:07.576620 32342 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20260709 12:26:07.637537 32342 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41935, user_credentials={real_user=slave}} blocked reactor thread for 58741.4us
W20260709 12:26:07.641248 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:26:07.672207 31791 consensus_peers.cc:597] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:07.672955 31791 consensus_peers.cc:597] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:07.755398 31791 consensus_peers.cc:597] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:07.755703 11127 raft_consensus.cc:493] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:07.756233 11127 raft_consensus.cc:515] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } }
I20260709 12:26:07.758540 11127 leader_election.cc:290] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
I20260709 12:26:07.798302 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:07.799842 30503 tablet_replica.cc:333] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.802191 30503 raft_consensus.cc:2243] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.803773 30503 raft_consensus.cc:2272] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.808189 30503 tablet_replica.cc:333] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.809419 30503 raft_consensus.cc:2243] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.810350 30503 raft_consensus.cc:2272] T 35c247463bef4f77990becbf6089be7d P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.814015 30503 tablet_replica.cc:333] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.815119 30503 raft_consensus.cc:2243] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.816354 30503 raft_consensus.cc:2272] T 2c162dbe81714319bfeccd9d7cb7bfea P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.819604 30503 tablet_replica.cc:333] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.820583 30503 raft_consensus.cc:2243] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.821348 30503 raft_consensus.cc:2272] T 2f2f85d943e24cfdb47fe2ad3185097f P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.824515 30503 tablet_replica.cc:333] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.825628 30503 raft_consensus.cc:2243] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.826540 30503 raft_consensus.cc:2272] T f79ad11703094c998ce1b3905e5c9ff6 P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.829607 30503 tablet_replica.cc:333] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
W20260709 12:26:07.830101 31791 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:07.830506 30503 raft_consensus.cc:2243] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.832801 30503 raft_consensus.cc:2272] T 20650b4ef2514a05bde5dfd38a379f32 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.832986 31791 leader_election.cc:336] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:07.833444 31791 leader_election.cc:304] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [CANDIDATE]: Term 35 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4d48c83c761f48079bc6ea176649dda0; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:07.835623 11156 raft_consensus.cc:2749] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Leader pre-election lost for term 35. Reason: could not achieve majority
I20260709 12:26:07.836282 30503 tablet_replica.cc:333] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.837224 30503 raft_consensus.cc:2243] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.838743 30503 raft_consensus.cc:2272] T ffab0b886efd41099d99c20339a3c4ae P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.842156 30503 tablet_replica.cc:333] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.843233 30503 raft_consensus.cc:2243] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.844264 30503 raft_consensus.cc:2272] T bc55086b81374990afb970c972568882 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.846658 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:07.848971 30503 tablet_replica.cc:333] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
W20260709 12:26:07.855158 32343 consensus_peers.cc:597] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.858162 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:07.858668 32343 consensus_peers.cc:597] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.859627 32343 consensus_peers.cc:597] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.860373 32343 consensus_peers.cc:597] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.860976 32343 consensus_peers.cc:597] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:07.858662 30503 raft_consensus.cc:2243] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:07.861912 32342 consensus_peers.cc:597] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20260709 12:26:07.863833 30503 raft_consensus.cc:2272] T fc47871ef692418388113c1d6c88b262 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.867776 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:07.870088 32342 consensus_peers.cc:597] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:07.870615 32342 consensus_peers.cc:597] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:07.874941 32342 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:07.875896 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:07.873327 32343 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.882513 32343 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.887048 32342 consensus_peers.cc:597] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:07.888893 32343 consensus_peers.cc:597] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:07.892026 32342 consensus_peers.cc:597] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:07.893450 30503 tablet_replica.cc:333] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.895700 30503 raft_consensus.cc:2243] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.897766 30503 raft_consensus.cc:2272] T 7523cff6f9fb44b5ba8fc04726d85b65 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.898691 32343 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:07.901373 30503 tablet_replica.cc:333] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.902410 30503 raft_consensus.cc:2243] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.903129 30503 raft_consensus.cc:2272] T c34755b3b9d040f69be353e35d7431e7 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.906303 30503 tablet_replica.cc:333] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.907251 30503 raft_consensus.cc:2243] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.908051 30503 raft_consensus.cc:2272] T b08faba9c13b4f0f85baf8b16c97d85b P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.911367 30503 tablet_replica.cc:333] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.912295 30503 raft_consensus.cc:2243] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.913712 30503 raft_consensus.cc:2272] T 45588632b8d64f9fb9c2e2ba18b20546 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.916885 30503 tablet_replica.cc:333] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.917744 30503 raft_consensus.cc:2243] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.919063 30503 raft_consensus.cc:2272] T a22e63b9ca824c92ae9ba374d2f3242a P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.922120 30503 tablet_replica.cc:333] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.923013 30503 raft_consensus.cc:2243] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.924932 30503 raft_consensus.cc:2272] T bc0ece2345cd46edb250e06e8f5431d6 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.929258 30503 tablet_replica.cc:333] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.930159 30503 raft_consensus.cc:2243] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.931404 30503 raft_consensus.cc:2272] T 00854b2a70f24ef3a7b43d17477a0ccf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.932509 31791 consensus_peers.cc:597] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:07.932904 32342 consensus_peers.cc:597] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:07.934346 30503 tablet_replica.cc:333] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.935338 30503 raft_consensus.cc:2243] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.935957 30503 raft_consensus.cc:2272] T b416bf5304744b14b72501e697b8c951 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.937700 32343 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:07.938848 30503 tablet_replica.cc:333] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.939812 30503 raft_consensus.cc:2243] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.941069 30503 raft_consensus.cc:2272] T d0a3795ad4d543f0998c3834aac7be45 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.944151 30503 tablet_replica.cc:333] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.945040 30503 raft_consensus.cc:2243] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.946413 30503 raft_consensus.cc:2272] T daa53c7009fe4881956dcfb5a20febde P 4d48c83c761f48079bc6ea176649dda0 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.949471 30503 tablet_replica.cc:333] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.950325 30503 raft_consensus.cc:2243] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.951162 30503 raft_consensus.cc:2272] T 992b9a77f3124e49a9bb783e08a9e856 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.954041 30503 tablet_replica.cc:333] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.954921 30503 raft_consensus.cc:2243] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 41 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.955497 30503 raft_consensus.cc:2272] T 04bdf51be7b94825a28028a5cfb96245 P 4d48c83c761f48079bc6ea176649dda0 [term 41 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.959193 30503 tablet_replica.cc:333] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.960057 30503 raft_consensus.cc:2243] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.961114 30503 raft_consensus.cc:2272] T d11aa4e638cd4a1dad543baff3b3f965 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.964125 30503 tablet_replica.cc:333] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.964951 30503 raft_consensus.cc:2243] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.965833 30503 raft_consensus.cc:2272] T 76c4ab1f56e44ff7b6adf7cfe850e01d P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.968952 30503 tablet_replica.cc:333] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.969828 30503 raft_consensus.cc:2243] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.970858 30503 raft_consensus.cc:2272] T b043bbe90410471da70e6b00856b5c8d P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.973901 30503 tablet_replica.cc:333] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.974735 30503 raft_consensus.cc:2243] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.975512 30503 raft_consensus.cc:2272] T ef31b84f462f424fa6162b810ea8c8a9 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.978546 30503 tablet_replica.cc:333] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.979452 30503 raft_consensus.cc:2243] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.980162 30503 raft_consensus.cc:2272] T 49cc564ccd0d411fa79831efa24f87bb P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.983093 30503 tablet_replica.cc:333] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.983924 30503 raft_consensus.cc:2243] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.985116 30503 raft_consensus.cc:2272] T 804df7046b7545d9890885896a333a3e P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.988145 30503 tablet_replica.cc:333] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.989079 30503 raft_consensus.cc:2243] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:07.990528 30503 raft_consensus.cc:2272] T b2c281519748458cb11517e3303d0a17 P 4d48c83c761f48079bc6ea176649dda0 [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:07.990890 32342 consensus_peers.cc:597] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:07.993655 30503 tablet_replica.cc:333] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.994591 30503 raft_consensus.cc:2243] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:07.995458 30503 raft_consensus.cc:2272] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:07.998477 30503 tablet_replica.cc:333] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:07.999353 30503 raft_consensus.cc:2243] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:08.000428 30503 raft_consensus.cc:2272] T e59926725a06436e9899dbbbcae3f622 P 4d48c83c761f48079bc6ea176649dda0 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.003455 30503 tablet_replica.cc:333] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.004268 30503 raft_consensus.cc:2243] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:26:08.005579 30503 raft_consensus.cc:2272] T 553d0b2dddba47779b0219c71ed9ce83 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.008605 30503 tablet_replica.cc:333] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.009467 30503 raft_consensus.cc:2243] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:08.010892 30503 raft_consensus.cc:2272] T 5bb2dc5276714f75b6e3e61ca9275f84 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.014038 30503 tablet_replica.cc:333] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.014840 30503 raft_consensus.cc:2243] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:26:08.016314 30503 raft_consensus.cc:2272] T 55f4b2e07ada4e06a285803fcbfd6344 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.019416 30503 tablet_replica.cc:333] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.020273 30503 raft_consensus.cc:2243] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:26:08.021461 30503 raft_consensus.cc:2272] T 3a0925b3d6e9448684fd239b27453b69 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.024435 30503 tablet_replica.cc:333] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.025367 30503 raft_consensus.cc:2243] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:08.026384 30503 raft_consensus.cc:2272] T a5dfbfce13484fc0b1feba45daaf8247 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.029383 30503 tablet_replica.cc:333] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.030215 30503 raft_consensus.cc:2243] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:08.032140 30503 raft_consensus.cc:2272] T 1e27b968c91a4906aec537421f481269 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.035261 30503 tablet_replica.cc:333] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.036105 30503 raft_consensus.cc:2243] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:08.036783 30503 raft_consensus.cc:2272] T 5d0d0402170d41b79dc30f93bee98766 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.040127 30503 tablet_replica.cc:333] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.040930 30503 raft_consensus.cc:2243] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:26:08.042307 30503 raft_consensus.cc:2272] T e59f73cf80df4c20b6c17715783f2121 P 4d48c83c761f48079bc6ea176649dda0 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.045552 30503 tablet_replica.cc:333] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.046403 30503 raft_consensus.cc:2243] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:08.047079 30503 raft_consensus.cc:2272] T d30338be26ae44ff8e627f7b8a99036c P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.050122 30503 tablet_replica.cc:333] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.050997 30503 raft_consensus.cc:2243] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:08.051638 30503 raft_consensus.cc:2272] T 945fd223ac334bce88917d1b9b4c72e1 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.058869 30503 tablet_replica.cc:333] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.059762 30503 raft_consensus.cc:2243] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 40 LEADER]: Raft consensus shutting down.
I20260709 12:26:08.066368 30503 raft_consensus.cc:2272] T 504cb3d29e9347fe98f010a3707a55cc P 4d48c83c761f48079bc6ea176649dda0 [term 40 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.069639 30503 tablet_replica.cc:333] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.070439 30503 raft_consensus.cc:2243] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:08.071532 30503 raft_consensus.cc:2272] T 9c849da11b414111b8461d6ae06c15b3 P 4d48c83c761f48079bc6ea176649dda0 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.074556 30503 tablet_replica.cc:333] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.075336 30503 raft_consensus.cc:2243] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:08.075966 30503 raft_consensus.cc:2272] T 812ed9ab41214308b368b53417e6e9f3 P 4d48c83c761f48079bc6ea176649dda0 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.078908 30503 tablet_replica.cc:333] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.079679 30503 raft_consensus.cc:2243] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:26:08.081048 30503 raft_consensus.cc:2272] T f84a67c6dba04300a5240cc80c94718a P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.100502 30503 tablet_replica.cc:333] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.101305 30503 raft_consensus.cc:2243] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:08.102562 30503 raft_consensus.cc:2272] T f2e7a775ae0d44108be0965c85245ecf P 4d48c83c761f48079bc6ea176649dda0 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.105350 30503 tablet_replica.cc:333] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.106130 30503 raft_consensus.cc:2243] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:26:08.107316 30503 raft_consensus.cc:2272] T 81c14d7d4fc14386b173a86e9623eb25 P 4d48c83c761f48079bc6ea176649dda0 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.110041 30503 tablet_replica.cc:333] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.110792 30503 raft_consensus.cc:2243] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:08.111418 30503 raft_consensus.cc:2272] T 292851e482024b3a8dcd05c5dab04534 P 4d48c83c761f48079bc6ea176649dda0 [term 28 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:08.112874 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:26:08.114212 30503 tablet_replica.cc:333] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.115048 30503 raft_consensus.cc:2243] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:08.115792 30503 raft_consensus.cc:2272] T 8725620974f7465eb0371f48761a1625 P 4d48c83c761f48079bc6ea176649dda0 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.118539 30503 tablet_replica.cc:333] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.119237 30503 raft_consensus.cc:2243] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:08.119894 30503 raft_consensus.cc:2272] T ab8ea7b957bd41e0a2e7967cbd101f36 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:08.122681 30503 tablet_replica.cc:333] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0: stopping tablet replica
I20260709 12:26:08.123322 30503 raft_consensus.cc:2243] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:08.124289 30503 raft_consensus.cc:2272] T f933d4319cf444afb9f67bd265576a31 P 4d48c83c761f48079bc6ea176649dda0 [term 32 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:08.151916 32342 consensus_peers.cc:597] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20260709 12:26:08.162914 11158 raft_consensus.cc:493] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:08.163662 11158 raft_consensus.cc:515] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:08.166154 11158 leader_election.cc:290] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:08.180570 32342 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:08.181423 32342 leader_election.cc:336] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:08.181902 32342 leader_election.cc:304] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:08.182778 11158 raft_consensus.cc:2749] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:26:08.202829 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:26:08.233877 32342 consensus_peers.cc:597] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:08.254410 11158 raft_consensus.cc:493] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:08.255147 11158 raft_consensus.cc:515] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:08.257501 11158 leader_election.cc:290] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639)
W20260709 12:26:08.263413 32342 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:08.264264 32342 leader_election.cc:336] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:08.264684 32342 leader_election.cc:304] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 41 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:08.265458 11158 raft_consensus.cc:2749] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Leader pre-election lost for term 41. Reason: could not achieve majority
W20260709 12:26:08.335011 32342 consensus_peers.cc:597] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:26:08.368268 32342 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:26:08.432320 32342 consensus_peers.cc:597] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:08.637879 11159 raft_consensus.cc:493] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:08.638581 11159 raft_consensus.cc:515] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 20 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:26:08.648612 32343 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111)
I20260709 12:26:08.646291 11159 leader_election.cc:290] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:26:08.654793 32342 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:08.655296 32342 leader_election.cc:304] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:26:08.656072 11159 raft_consensus.cc:2749] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
W20260709 12:26:08.656471 32342 leader_election.cc:336] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:08.719098 11159 raft_consensus.cc:493] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:26:08.719796 11159 raft_consensus.cc:515] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Starting pre-election with config: opid_index: 17 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:08.722255 11159 leader_election.cc:290] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:26:08.729338 32343 leader_election.cc:336] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111)
W20260709 12:26:08.730664 32342 leader_election.cc:336] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:08.731441 32342 leader_election.cc:304] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 29 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:08.732261 11159 raft_consensus.cc:2749] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Leader pre-election lost for term 29. Reason: could not achieve majority
W20260709 12:26:08.771597 32399 debug-util.cc:398] Leaking SignalData structure 0x7b08001c9d60 after lost signal to thread 30506
W20260709 12:26:08.772898 32399 debug-util.cc:398] Leaking SignalData structure 0x7b08003de780 after lost signal to thread 31610
W20260709 12:26:08.773581 32399 debug-util.cc:398] Leaking SignalData structure 0x7b0800382580 after lost signal to thread 31851
W20260709 12:26:08.774946 32399 debug-util.cc:398] Leaking SignalData structure 0x7b08002a1e40 after lost signal to thread 32402
I20260709 12:26:08.869942 11159 raft_consensus.cc:493] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:08.870748 11159 raft_consensus.cc:515] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Starting pre-election with config: opid_index: 19 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:08.874855 11159 leader_election.cc:290] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:26:08.909933 32343 leader_election.cc:336] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111)
W20260709 12:26:08.913726 32342 leader_election.cc:336] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:08.915668 32342 leader_election.cc:304] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
W20260709 12:26:08.918344 32342 leader_election.cc:336] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 31 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:08.918922 11159 raft_consensus.cc:2749] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Leader pre-election lost for term 31. Reason: could not achieve majority
I20260709 12:26:08.932487 11159 raft_consensus.cc:493] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:26:08.933141 11159 raft_consensus.cc:515] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:08.936375 11159 leader_election.cc:290] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:26:08.943540 32343 leader_election.cc:336] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111)
W20260709 12:26:08.965008 32342 consensus_peers.cc:597] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:26:08.966883 32342 leader_election.cc:336] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:08.967470 32342 leader_election.cc:304] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:26:08.970080 11159 raft_consensus.cc:2749] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
I20260709 12:26:08.990172 11159 raft_consensus.cc:493] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:26:08.991091 11159 raft_consensus.cc:515] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } attrs { replace: true } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:26:09.000939 32343 leader_election.cc:336] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111)
I20260709 12:26:09.004552 11159 leader_election.cc:290] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:26:09.021453 32342 leader_election.cc:336] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:09.022042 32342 leader_election.cc:304] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:26:09.022873 11159 raft_consensus.cc:2749] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:09.026490 32342 leader_election.cc:336] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:09.050261 11159 raft_consensus.cc:493] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:26:09.050896 11159 raft_consensus.cc:515] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Starting pre-election with config: opid_index: 23 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:09.053124 11159 leader_election.cc:290] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 39 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:26:09.060168 32343 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111)
W20260709 12:26:09.075104 32342 consensus_peers.cc:597] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20260709 12:26:09.076735 32342 leader_election.cc:336] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 39 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:09.077246 32342 leader_election.cc:304] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 39 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:26:09.078032 11159 raft_consensus.cc:2749] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Leader pre-election lost for term 39. Reason: could not achieve majority
I20260709 12:26:09.150802 11159 raft_consensus.cc:493] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:26:09.151330 11159 raft_consensus.cc:515] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:09.153487 11159 leader_election.cc:290] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:26:09.164431 32343 leader_election.cc:336] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111)
I20260709 12:26:09.175268 11159 raft_consensus.cc:493] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:26:09.175926 11159 raft_consensus.cc:515] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:09.178023 11159 leader_election.cc:290] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:26:09.185676 32343 leader_election.cc:336] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111)
W20260709 12:26:09.210422 32342 leader_election.cc:336] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:09.211026 32342 leader_election.cc:304] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:09.211805 11159 raft_consensus.cc:2749] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:09.217816 32342 leader_election.cc:336] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:09.218359 32342 leader_election.cc:304] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 38 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:26:09.219153 11159 raft_consensus.cc:2749] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Leader pre-election lost for term 38. Reason: could not achieve majority
W20260709 12:26:09.229871 30503 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 31848)
I20260709 12:26:09.305645 11159 raft_consensus.cc:493] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:26:09.306557 11159 raft_consensus.cc:515] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:09.308998 11159 leader_election.cc:290] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:26:09.315543 32343 leader_election.cc:336] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111)
W20260709 12:26:09.346323 32342 leader_election.cc:336] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:09.352835 32342 leader_election.cc:304] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 33 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:09.353762 11159 raft_consensus.cc:2749] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Leader pre-election lost for term 33. Reason: could not achieve majority
I20260709 12:26:09.362854 11159 raft_consensus.cc:493] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election (detected failure of leader 4d48c83c761f48079bc6ea176649dda0)
I20260709 12:26:09.363422 11159 raft_consensus.cc:515] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Starting pre-election with config: opid_index: 21 OBSOLETE_local: false peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
I20260709 12:26:09.365636 11159 leader_election.cc:290] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Requested pre-vote from peers 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:26:09.371289 32343 leader_election.cc:336] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111)
W20260709 12:26:09.384701 32342 leader_election.cc:336] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
I20260709 12:26:09.385273 32342 leader_election.cc:304] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 32 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 039d66a6c85e43c3936bf5017501b344, 4d48c83c761f48079bc6ea176649dda0
I20260709 12:26:09.386070 11159 raft_consensus.cc:2749] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Leader pre-election lost for term 32. Reason: could not achieve majority
W20260709 12:26:09.418187 32342 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:26:09.471603 32342 consensus_peers.cc:597] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:09.510489 32342 consensus_peers.cc:597] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:26:09.560030 11159 raft_consensus.cc:493] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election (detected failure of leader 039d66a6c85e43c3936bf5017501b344)
I20260709 12:26:09.560637 11159 raft_consensus.cc:515] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Starting pre-election with config: opid_index: 22 OBSOLETE_local: false peers { permanent_uuid: "b95aeeaf4c51442b8092cff1de674f2b" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41935 } } peers { permanent_uuid: "039d66a6c85e43c3936bf5017501b344" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 46639 } } peers { permanent_uuid: "4d48c83c761f48079bc6ea176649dda0" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44645 } attrs { replace: true } } peers { permanent_uuid: "5206fd0087404d2da964a5d985a04ff4" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 44855 } attrs { promote: false } }
W20260709 12:26:09.568877 32343 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111)
I20260709 12:26:09.569736 11159 leader_election.cc:290] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Requested pre-vote from peers b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935), 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639), 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645)
W20260709 12:26:09.578883 32342 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:09.579692 32342 leader_election.cc:304] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 5206fd0087404d2da964a5d985a04ff4; no voters: 4d48c83c761f48079bc6ea176649dda0, b95aeeaf4c51442b8092cff1de674f2b
I20260709 12:26:09.580552 11159 raft_consensus.cc:2749] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Leader pre-election lost for term 37. Reason: could not achieve majority
W20260709 12:26:09.583504 32342 leader_election.cc:336] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [CANDIDATE]: Term 37 pre-election: RPC error from VoteRequest() call to peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111)
W20260709 12:26:09.739677 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:26:09.762674 32343 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:09.782750 31561 catalog_manager.cc:4755] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 3a0925b3d6e9448684fd239b27453b69: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
W20260709 12:26:09.845497 32343 consensus_peers.cc:597] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:09.988518 32342 consensus_peers.cc:597] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20260709 12:26:10.017694 32343 consensus_peers.cc:597] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:10.045729 32342 consensus_peers.cc:597] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:26:10.089495 30503 tablet_server.cc:196] TabletServer@127.29.201.195:0 shutdown complete.
W20260709 12:26:10.113706 32343 consensus_peers.cc:597] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:10.141269 32342 consensus_peers.cc:597] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20260709 12:26:10.201825 32342 consensus_peers.cc:597] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:10.218063 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:10.246814 32343 consensus_peers.cc:597] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:10.254501 30503 tablet_server.cc:179] TabletServer@127.29.201.196:0 shutting down...
W20260709 12:26:10.273048 32342 consensus_peers.cc:597] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20260709 12:26:10.296612 32343 consensus_peers.cc:597] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:10.316880 32342 consensus_peers.cc:597] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20260709 12:26:10.344049 32343 consensus_peers.cc:597] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:10.349532 32343 consensus_peers.cc:597] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:10.350293 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:10.351357 30503 tablet_replica.cc:333] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.352628 30503 raft_consensus.cc:2243] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.353421 30503 raft_consensus.cc:2272] T d11aa4e638cd4a1dad543baff3b3f965 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.357100 30503 tablet_replica.cc:333] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.358059 30503 raft_consensus.cc:2243] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.358871 30503 raft_consensus.cc:2272] T f933d4319cf444afb9f67bd265576a31 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.361889 30503 tablet_replica.cc:333] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.362828 30503 raft_consensus.cc:2243] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.363462 30503 raft_consensus.cc:2272] T 20650b4ef2514a05bde5dfd38a379f32 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.366669 30503 tablet_replica.cc:333] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.367669 30503 raft_consensus.cc:2243] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.368541 30503 raft_consensus.cc:2272] T 3a0925b3d6e9448684fd239b27453b69 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.371935 30503 tablet_replica.cc:333] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.372917 30503 raft_consensus.cc:2243] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.373909 30503 raft_consensus.cc:2272] T b25009db0c5a47a9ab9f6f8b2e5c1b17 P 5206fd0087404d2da964a5d985a04ff4 [term 28 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.377158 30503 tablet_replica.cc:333] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.380100 30503 raft_consensus.cc:2243] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:10.384594 32342 consensus_peers.cc:597] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:10.381493 30503 raft_consensus.cc:2272] T f6404cb9c2174326b06cbbb0d27d43f2 P 5206fd0087404d2da964a5d985a04ff4 [term 40 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:10.387596 32343 consensus_peers.cc:597] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:10.388890 30503 tablet_replica.cc:333] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.389945 30503 raft_consensus.cc:2243] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.391222 30503 raft_consensus.cc:2272] T 2c162dbe81714319bfeccd9d7cb7bfea P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.395449 30503 tablet_replica.cc:333] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.396365 30503 raft_consensus.cc:2243] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.397564 30503 raft_consensus.cc:2272] T 812ed9ab41214308b368b53417e6e9f3 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:10.400452 32342 consensus_peers.cc:597] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 -> Peer b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): Couldn't send request to peer b95aeeaf4c51442b8092cff1de674f2b. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20260709 12:26:10.400751 30503 tablet_replica.cc:333] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.401906 30503 raft_consensus.cc:2243] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.402812 30503 raft_consensus.cc:2272] T bc0ece2345cd46edb250e06e8f5431d6 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.406396 30503 tablet_replica.cc:333] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.407414 30503 raft_consensus.cc:2243] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 29 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.408792 30503 raft_consensus.cc:2272] T 00854b2a70f24ef3a7b43d17477a0ccf P 5206fd0087404d2da964a5d985a04ff4 [term 29 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.411924 30503 tablet_replica.cc:333] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.412904 30503 raft_consensus.cc:2243] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.414265 30503 raft_consensus.cc:2272] T 6821bb8779e64160887e3195aa5078a1 P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.417443 30503 tablet_replica.cc:333] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.418367 30503 raft_consensus.cc:2243] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.419055 30503 raft_consensus.cc:2272] T b2c281519748458cb11517e3303d0a17 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.421974 30503 tablet_replica.cc:333] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.422822 30503 raft_consensus.cc:2243] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.423992 30503 raft_consensus.cc:2272] T 11075d6cbd1a40ccaf381a4eb70cbf24 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.427201 30503 tablet_replica.cc:333] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.428103 30503 raft_consensus.cc:2243] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 32 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.429294 30503 raft_consensus.cc:2272] T c22bf85ce45841fb80e303e2746080da P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.432722 30503 tablet_replica.cc:333] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
W20260709 12:26:10.433003 32342 consensus_peers.cc:597] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:10.433771 30503 raft_consensus.cc:2243] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.435299 30503 raft_consensus.cc:2272] T fc47871ef692418388113c1d6c88b262 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.438257 30503 tablet_replica.cc:333] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.439213 30503 raft_consensus.cc:2243] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 33 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.440407 30503 raft_consensus.cc:2272] T 2f2f85d943e24cfdb47fe2ad3185097f P 5206fd0087404d2da964a5d985a04ff4 [term 33 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.443555 30503 tablet_replica.cc:333] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.444465 30503 raft_consensus.cc:2243] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 41 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.445775 30503 raft_consensus.cc:2272] T 04bdf51be7b94825a28028a5cfb96245 P 5206fd0087404d2da964a5d985a04ff4 [term 41 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:10.445936 32343 consensus_peers.cc:597] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 -> Peer 4d48c83c761f48079bc6ea176649dda0 (127.29.201.195:44645): Couldn't send request to peer 4d48c83c761f48079bc6ea176649dda0. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.195:44645: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:10.448635 30503 tablet_replica.cc:333] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.449592 30503 raft_consensus.cc:2243] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.450320 30503 raft_consensus.cc:2272] T 804df7046b7545d9890885896a333a3e P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.453459 30503 tablet_replica.cc:333] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.454347 30503 raft_consensus.cc:2243] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 31 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.455476 30503 raft_consensus.cc:2272] T 6e7ad549735d4871af75658a91bb98c4 P 5206fd0087404d2da964a5d985a04ff4 [term 31 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:10.457701 32342 consensus_peers.cc:597] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 -> Peer 039d66a6c85e43c3936bf5017501b344 (127.29.201.194:46639): Couldn't send request to peer 039d66a6c85e43c3936bf5017501b344. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:46639: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260709 12:26:10.458510 30503 tablet_replica.cc:333] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.459514 30503 raft_consensus.cc:2243] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 34 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.460644 30503 raft_consensus.cc:2272] T c34755b3b9d040f69be353e35d7431e7 P 5206fd0087404d2da964a5d985a04ff4 [term 34 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.463758 30503 tablet_replica.cc:333] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.464618 30503 raft_consensus.cc:2243] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 37 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.465612 30503 raft_consensus.cc:2272] T 1e27b968c91a4906aec537421f481269 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.468699 30503 tablet_replica.cc:333] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.469575 30503 raft_consensus.cc:2243] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.470450 30503 raft_consensus.cc:2272] T e59f73cf80df4c20b6c17715783f2121 P 5206fd0087404d2da964a5d985a04ff4 [term 37 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.473423 30503 tablet_replica.cc:333] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.474234 30503 raft_consensus.cc:2243] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.475446 30503 raft_consensus.cc:2272] T d30338be26ae44ff8e627f7b8a99036c P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.478535 30503 tablet_replica.cc:333] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.479424 30503 raft_consensus.cc:2243] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.480643 30503 raft_consensus.cc:2272] T 70ada7cfaa154c8c9cc927936b46c30a P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.483659 30503 tablet_replica.cc:333] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.484529 30503 raft_consensus.cc:2243] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.485776 30503 raft_consensus.cc:2272] T 945fd223ac334bce88917d1b9b4c72e1 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.488852 30503 tablet_replica.cc:333] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.489727 30503 raft_consensus.cc:2243] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.490943 30503 raft_consensus.cc:2272] T a5dfbfce13484fc0b1feba45daaf8247 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.494227 30503 tablet_replica.cc:333] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.495184 30503 raft_consensus.cc:2243] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.496644 30503 raft_consensus.cc:2272] T 35c247463bef4f77990becbf6089be7d P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.499998 30503 tablet_replica.cc:333] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.500944 30503 raft_consensus.cc:2243] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.501709 30503 raft_consensus.cc:2272] T 354c519f070b453dadcdef3f0d835c69 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.504837 30503 tablet_replica.cc:333] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.505661 30503 raft_consensus.cc:2243] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 36 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.507076 30503 raft_consensus.cc:2272] T 6499e78dd87c4e05be32f5681957a6b0 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.510296 30503 tablet_replica.cc:333] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.511166 30503 raft_consensus.cc:2243] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 35 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.512483 30503 raft_consensus.cc:2272] T b08faba9c13b4f0f85baf8b16c97d85b P 5206fd0087404d2da964a5d985a04ff4 [term 35 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.515755 30503 tablet_replica.cc:333] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.516564 30503 raft_consensus.cc:2243] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 38 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.517843 30503 raft_consensus.cc:2272] T 436783e8e1cc49edbad793a31645e2d6 P 5206fd0087404d2da964a5d985a04ff4 [term 38 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.520735 30503 tablet_replica.cc:333] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.521530 30503 raft_consensus.cc:2243] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.522172 30503 raft_consensus.cc:2272] T 45588632b8d64f9fb9c2e2ba18b20546 P 5206fd0087404d2da964a5d985a04ff4 [term 32 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.525383 30503 tablet_replica.cc:333] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.526216 30503 raft_consensus.cc:2243] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:10.526990 30503 raft_consensus.cc:2272] T e59926725a06436e9899dbbbcae3f622 P 5206fd0087404d2da964a5d985a04ff4 [term 36 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:10.530160 30503 tablet_replica.cc:333] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4: stopping tablet replica
I20260709 12:26:10.530959 30503 raft_consensus.cc:2243] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 30 LEADER]: Raft consensus shutting down.
I20260709 12:26:10.532404 30503 raft_consensus.cc:2272] T 181ddae071d549aab328e3c1d8553734 P 5206fd0087404d2da964a5d985a04ff4 [term 30 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:11.181789 30503 tablet_server.cc:196] TabletServer@127.29.201.196:0 shutdown complete.
I20260709 12:26:11.325619 30503 master.cc:562] Master@127.29.201.254:37495 shutting down...
W20260709 12:26:11.371061 31624 auto_rebalancer.cc:873] Removing replace marker failed: Catalog manager is not initialized. State: Closing
W20260709 12:26:11.371400 31624 auto_rebalancer.cc:901] Could not move replica: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:26:11.371626 31624 auto_rebalancer.cc:330] scheduled replica move failed to complete: Service unavailable: Catalog manager is not initialized. State: Closing
W20260709 12:26:26.188584 31561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111) [suppressed 540 similar messages]
W20260709 12:26:26.192878 31561 catalog_manager.cc:4755] TS b95aeeaf4c51442b8092cff1de674f2b (127.29.201.193:41935): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 20650b4ef2514a05bde5dfd38a379f32: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41935: connect: Connection refused (error 111)
I20260709 12:26:28.387208 30503 raft_consensus.cc:2243] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:28.388501 30503 raft_consensus.cc:2272] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:28.388989 30503 tablet_replica.cc:333] T 00000000000000000000000000000000 P ac4fa664fb214bda87cd96abb2116a4b: stopping tablet replica
I20260709 12:26:28.545440 30503 master.cc:584] Master@127.29.201.254:37495 shutdown complete.
[       OK ] LeaderRebalancerTest.AddTserver (374893 ms)
[ RUN      ] LeaderRebalancerTest.RestartTserver
I20260709 12:26:28.671859 30503 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.201.254:38597
I20260709 12:26:28.673034 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:28.679286 11172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:28.679927 11171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:28.679934 11174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:28.681571 30503 server_base.cc:1061] running on GCE node
I20260709 12:26:28.686363 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:28.686672 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:28.686879 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599988686847 us; error 0 us; skew 500 ppm
I20260709 12:26:28.687438 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:28.690179 30503 webserver.cc:533] Webserver started at http://127.29.201.254:39287/ using document root <none> and password file <none>
I20260709 12:26:28.690785 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:28.690984 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:28.691277 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:28.692662 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/master-0-root/instance:
uuid: "160751aa6dc745d783d91c4f9039e337"
format_stamp: "Formatted at 2026-07-09 12:26:28 on dist-test-slave-qgf5"
I20260709 12:26:28.697615 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.002s	sys 0.004s
I20260709 12:26:28.701414 11179 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:28.702294 30503 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:26:28.702713 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/master-0-root
uuid: "160751aa6dc745d783d91c4f9039e337"
format_stamp: "Formatted at 2026-07-09 12:26:28 on dist-test-slave-qgf5"
I20260709 12:26:28.703037 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:28.714493 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:28.715869 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:28.757426 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.254:38597
I20260709 12:26:28.757529 11230 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.254:38597 every 8 connection(s)
I20260709 12:26:28.761720 11231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:28.774318 11231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337: Bootstrap starting.
I20260709 12:26:28.779465 11231 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:28.784523 11231 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337: No bootstrap required, opened a new log
I20260709 12:26:28.787249 11231 raft_consensus.cc:359] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "160751aa6dc745d783d91c4f9039e337" member_type: VOTER }
I20260709 12:26:28.787811 11231 raft_consensus.cc:385] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:28.788117 11231 raft_consensus.cc:740] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 160751aa6dc745d783d91c4f9039e337, State: Initialized, Role: FOLLOWER
I20260709 12:26:28.788805 11231 consensus_queue.cc:260] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "160751aa6dc745d783d91c4f9039e337" member_type: VOTER }
I20260709 12:26:28.789351 11231 raft_consensus.cc:399] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:26:28.789651 11231 raft_consensus.cc:493] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:26:28.789980 11231 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:28.796599 11231 raft_consensus.cc:515] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "160751aa6dc745d783d91c4f9039e337" member_type: VOTER }
I20260709 12:26:28.797453 11231 leader_election.cc:304] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 160751aa6dc745d783d91c4f9039e337; no voters: 
I20260709 12:26:28.799139 11231 leader_election.cc:290] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:26:28.799515 11234 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:28.801164 11234 raft_consensus.cc:697] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [term 1 LEADER]: Becoming Leader. State: Replica: 160751aa6dc745d783d91c4f9039e337, State: Running, Role: LEADER
I20260709 12:26:28.802017 11234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "160751aa6dc745d783d91c4f9039e337" member_type: VOTER }
I20260709 12:26:28.802678 11231 sys_catalog.cc:565] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:26:28.809578 11235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "160751aa6dc745d783d91c4f9039e337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "160751aa6dc745d783d91c4f9039e337" member_type: VOTER } }
I20260709 12:26:28.810348 11235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [sys.catalog]: This master's current role is: LEADER
I20260709 12:26:28.811937 11236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 160751aa6dc745d783d91c4f9039e337. Latest consensus state: current_term: 1 leader_uuid: "160751aa6dc745d783d91c4f9039e337" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "160751aa6dc745d783d91c4f9039e337" member_type: VOTER } }
I20260709 12:26:28.812908 11236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [sys.catalog]: This master's current role is: LEADER
I20260709 12:26:28.814813 11241 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:26:28.820369 11241 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:26:28.822899 30503 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:26:28.830996 11241 catalog_manager.cc:1383] Generated new cluster ID: cab7cebac2e640b5ad0e42a2c210ae17
I20260709 12:26:28.831429 11241 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:26:28.857133 11241 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:26:28.858521 11241 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:26:28.873332 11241 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337: Generated new TSK 0
I20260709 12:26:28.874074 11241 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:26:28.890097 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:28.897615 11252 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:28.899116 11253 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:28.900806 11255 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:28.901700 30503 server_base.cc:1061] running on GCE node
I20260709 12:26:28.904742 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:28.905011 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:28.905224 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599988905203 us; error 0 us; skew 500 ppm
I20260709 12:26:28.905871 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:28.909039 30503 webserver.cc:533] Webserver started at http://127.29.201.193:35155/ using document root <none> and password file <none>
I20260709 12:26:28.909601 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:28.909799 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:28.910099 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:28.911474 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/instance:
uuid: "663d88d9634e4720a695c7cd69eaa25e"
format_stamp: "Formatted at 2026-07-09 12:26:28 on dist-test-slave-qgf5"
I20260709 12:26:28.916529 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:26:28.920282 11260 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:28.921152 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:26:28.921518 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-0-root
uuid: "663d88d9634e4720a695c7cd69eaa25e"
format_stamp: "Formatted at 2026-07-09 12:26:28 on dist-test-slave-qgf5"
I20260709 12:26:28.921839 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:28.937086 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:28.938452 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:28.940178 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:26:28.942893 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:26:28.943092 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:28.943419 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:26:28.943593 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:28.991328 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.193:37401
I20260709 12:26:28.991442 11322 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.193:37401 every 8 connection(s)
I20260709 12:26:28.997419 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:29.007763 11328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:29.008917 11327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:29.010830 30503 server_base.cc:1061] running on GCE node
W20260709 12:26:29.012212 11330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:29.014357 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:29.014618 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:29.014856 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599989014809 us; error 0 us; skew 500 ppm
I20260709 12:26:29.015517 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:29.019057 30503 webserver.cc:533] Webserver started at http://127.29.201.194:33013/ using document root <none> and password file <none>
I20260709 12:26:29.019685 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:29.019968 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:29.020334 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:29.020346 11323 heartbeater.cc:344] Connected to a master server at 127.29.201.254:38597
I20260709 12:26:29.021021 11323 heartbeater.cc:461] Registering TS with master...
I20260709 12:26:29.021911 11323 heartbeater.cc:507] Master 127.29.201.254:38597 requested a full tablet report, sending...
I20260709 12:26:29.021926 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/instance:
uuid: "05b93ba4435f40fb97b54343c4fdd71e"
format_stamp: "Formatted at 2026-07-09 12:26:29 on dist-test-slave-qgf5"
I20260709 12:26:29.024766 11196 ts_manager.cc:194] Registered new tserver with Master: 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:29.027652 11196 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45208
I20260709 12:26:29.029069 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:26:29.033159 11335 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:29.033974 30503 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.000s
I20260709 12:26:29.034332 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-1-root
uuid: "05b93ba4435f40fb97b54343c4fdd71e"
format_stamp: "Formatted at 2026-07-09 12:26:29 on dist-test-slave-qgf5"
I20260709 12:26:29.034772 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:29.058818 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:29.060174 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:29.061867 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:26:29.064507 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:26:29.064775 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:29.065023 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:26:29.065196 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:29.109416 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.194:44567
I20260709 12:26:29.109565 11397 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.194:44567 every 8 connection(s)
I20260709 12:26:29.114565 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:29.121734 11401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:29.124768 11402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:29.125847 11404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:29.126127 30503 server_base.cc:1061] running on GCE node
I20260709 12:26:29.127400 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:29.127682 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:29.127898 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599989127880 us; error 0 us; skew 500 ppm
I20260709 12:26:29.128609 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:29.129316 11398 heartbeater.cc:344] Connected to a master server at 127.29.201.254:38597
I20260709 12:26:29.129743 11398 heartbeater.cc:461] Registering TS with master...
I20260709 12:26:29.130617 11398 heartbeater.cc:507] Master 127.29.201.254:38597 requested a full tablet report, sending...
I20260709 12:26:29.132472 30503 webserver.cc:533] Webserver started at http://127.29.201.195:35277/ using document root <none> and password file <none>
I20260709 12:26:29.133078 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:29.133111 11196 ts_manager.cc:194] Registered new tserver with Master: 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:29.133312 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:29.133736 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:29.134846 11196 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45216
I20260709 12:26:29.135203 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-2-root/instance:
uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
format_stamp: "Formatted at 2026-07-09 12:26:29 on dist-test-slave-qgf5"
I20260709 12:26:29.140304 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:26:29.144132 11409 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:29.145022 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:26:29.145360 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-2-root
uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
format_stamp: "Formatted at 2026-07-09 12:26:29 on dist-test-slave-qgf5"
I20260709 12:26:29.145680 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:29.184777 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:29.186167 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:29.187839 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:26:29.190274 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:26:29.190536 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:29.190811 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:26:29.191000 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:29.235894 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.195:36253
I20260709 12:26:29.236032 11471 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.195:36253 every 8 connection(s)
I20260709 12:26:29.241016 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:29.249406 11475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:29.250275 11476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:29.253832 11478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:29.254359 30503 server_base.cc:1061] running on GCE node
I20260709 12:26:29.255365 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:29.255630 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:29.255846 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783599989255826 us; error 0 us; skew 500 ppm
I20260709 12:26:29.256471 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:29.257125 11472 heartbeater.cc:344] Connected to a master server at 127.29.201.254:38597
I20260709 12:26:29.257597 11472 heartbeater.cc:461] Registering TS with master...
I20260709 12:26:29.258648 11472 heartbeater.cc:507] Master 127.29.201.254:38597 requested a full tablet report, sending...
I20260709 12:26:29.259459 30503 webserver.cc:533] Webserver started at http://127.29.201.196:39055/ using document root <none> and password file <none>
I20260709 12:26:29.260121 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:29.260356 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:26:29.260655 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:26:29.261059 11196 ts_manager.cc:194] Registered new tserver with Master: 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:29.261952 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-3-root/instance:
uuid: "9968c0ed1974454d828996824f859433"
format_stamp: "Formatted at 2026-07-09 12:26:29 on dist-test-slave-qgf5"
I20260709 12:26:29.262542 11196 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45230
I20260709 12:26:29.267716 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.004s	sys 0.003s
I20260709 12:26:29.271740 11483 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:29.272625 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:26:29.272964 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-3-root
uuid: "9968c0ed1974454d828996824f859433"
format_stamp: "Formatted at 2026-07-09 12:26:29 on dist-test-slave-qgf5"
I20260709 12:26:29.273306 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:29.298730 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:29.300098 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:29.301498 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:26:29.304613 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:26:29.304917 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:29.305199 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:26:29.305405 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:29.349769 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.196:34569
I20260709 12:26:29.349866 11545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.196:34569 every 8 connection(s)
I20260709 12:26:29.364228 11546 heartbeater.cc:344] Connected to a master server at 127.29.201.254:38597
I20260709 12:26:29.364629 11546 heartbeater.cc:461] Registering TS with master...
I20260709 12:26:29.365602 11546 heartbeater.cc:507] Master 127.29.201.254:38597 requested a full tablet report, sending...
I20260709 12:26:29.367712 11196 ts_manager.cc:194] Registered new tserver with Master: 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:29.368013 30503 internal_mini_cluster.cc:371] 4 TS(s) registered with all masters after 0.01515163s
I20260709 12:26:29.369685 11196 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:45236
I20260709 12:26:29.396814 11196 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45240:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
I20260709 12:26:29.668272 11436 tablet_service.cc:1511] Processing CreateTablet for tablet 5734e27937894275a18ba504ffedd114 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:26:29.669262 11437 tablet_service.cc:1511] Processing CreateTablet for tablet 8236893668af4eaa93bc53b6715f6cf0 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:26:29.670058 11436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5734e27937894275a18ba504ffedd114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.671056 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8236893668af4eaa93bc53b6715f6cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.674327 11435 tablet_service.cc:1511] Processing CreateTablet for tablet 43ed653326ff4ae6b3c7daa594b3228b (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:26:29.676112 11435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43ed653326ff4ae6b3c7daa594b3228b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.678835 11510 tablet_service.cc:1511] Processing CreateTablet for tablet 5734e27937894275a18ba504ffedd114 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:26:29.680552 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5734e27937894275a18ba504ffedd114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.680181 11434 tablet_service.cc:1511] Processing CreateTablet for tablet bd76933be5f240a1a5c069775527f562 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:26:29.680474 11506 tablet_service.cc:1511] Processing CreateTablet for tablet 393b5b835714456ab06eeecb9b4a116d (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:26:29.680932 11507 tablet_service.cc:1511] Processing CreateTablet for tablet c91074436b5b44779742b3478249f283 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:26:29.681902 11434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd76933be5f240a1a5c069775527f562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.681994 11506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 393b5b835714456ab06eeecb9b4a116d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.678711 11511 tablet_service.cc:1511] Processing CreateTablet for tablet d8586c90fc9848248b387776dc1f7db6 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:26:29.683884 11509 tablet_service.cc:1511] Processing CreateTablet for tablet 43ed653326ff4ae6b3c7daa594b3228b (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:26:29.685379 11288 tablet_service.cc:1511] Processing CreateTablet for tablet 8236893668af4eaa93bc53b6715f6cf0 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:26:29.685544 11508 tablet_service.cc:1511] Processing CreateTablet for tablet db58e810d4a04064bbf12daef58026f3 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:26:29.686767 11507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91074436b5b44779742b3478249f283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.686674 11285 tablet_service.cc:1511] Processing CreateTablet for tablet bd76933be5f240a1a5c069775527f562 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:26:29.687758 11508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db58e810d4a04064bbf12daef58026f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.688270 11285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd76933be5f240a1a5c069775527f562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.688061 11287 tablet_service.cc:1511] Processing CreateTablet for tablet d8586c90fc9848248b387776dc1f7db6 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:26:29.688882 11511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8586c90fc9848248b387776dc1f7db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.689749 11287 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8586c90fc9848248b387776dc1f7db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.689462 11282 tablet_service.cc:1511] Processing CreateTablet for tablet 6788efa5ae124a478e06ef927cb631ff (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:26:29.691107 11282 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6788efa5ae124a478e06ef927cb631ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.692106 11286 tablet_service.cc:1511] Processing CreateTablet for tablet 5734e27937894275a18ba504ffedd114 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260709 12:26:29.693813 11286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5734e27937894275a18ba504ffedd114. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.695135 11509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43ed653326ff4ae6b3c7daa594b3228b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.696884 11283 tablet_service.cc:1511] Processing CreateTablet for tablet c91074436b5b44779742b3478249f283 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:26:29.699766 11284 tablet_service.cc:1511] Processing CreateTablet for tablet db58e810d4a04064bbf12daef58026f3 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:26:29.704663 11432 tablet_service.cc:1511] Processing CreateTablet for tablet 393b5b835714456ab06eeecb9b4a116d (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260709 12:26:29.706988 11433 tablet_service.cc:1511] Processing CreateTablet for tablet c91074436b5b44779742b3478249f283 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260709 12:26:29.708851 11283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91074436b5b44779742b3478249f283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.712605 11288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8236893668af4eaa93bc53b6715f6cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.715137 11284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db58e810d4a04064bbf12daef58026f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.706449 11432 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 393b5b835714456ab06eeecb9b4a116d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.720711 11433 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91074436b5b44779742b3478249f283. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.726123 11505 tablet_service.cc:1511] Processing CreateTablet for tablet e57eb43303724eae812867b69fa94d9d (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:26:29.727682 11505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57eb43303724eae812867b69fa94d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.745390 11431 tablet_service.cc:1511] Processing CreateTablet for tablet 6788efa5ae124a478e06ef927cb631ff (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:26:29.747097 11431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6788efa5ae124a478e06ef927cb631ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.754124 11281 tablet_service.cc:1511] Processing CreateTablet for tablet e57eb43303724eae812867b69fa94d9d (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:26:29.755795 11281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57eb43303724eae812867b69fa94d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.796252 11566 tablet_bootstrap.cc:492] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:29.810792 11363 tablet_service.cc:1511] Processing CreateTablet for tablet 8236893668af4eaa93bc53b6715f6cf0 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260709 12:26:29.812358 11363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8236893668af4eaa93bc53b6715f6cf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.816061 11362 tablet_service.cc:1511] Processing CreateTablet for tablet d8586c90fc9848248b387776dc1f7db6 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260709 12:26:29.817683 11362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8586c90fc9848248b387776dc1f7db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.820115 11361 tablet_service.cc:1511] Processing CreateTablet for tablet 43ed653326ff4ae6b3c7daa594b3228b (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260709 12:26:29.821760 11361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43ed653326ff4ae6b3c7daa594b3228b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.822350 11566 tablet_bootstrap.cc:654] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:29.823969 11360 tablet_service.cc:1511] Processing CreateTablet for tablet bd76933be5f240a1a5c069775527f562 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260709 12:26:29.825470 11360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd76933be5f240a1a5c069775527f562. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.827903 11359 tablet_service.cc:1511] Processing CreateTablet for tablet db58e810d4a04064bbf12daef58026f3 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260709 12:26:29.829384 11359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db58e810d4a04064bbf12daef58026f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.831997 11358 tablet_service.cc:1511] Processing CreateTablet for tablet 393b5b835714456ab06eeecb9b4a116d (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
W20260709 12:26:29.839004 11244 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:29.841395 11357 tablet_service.cc:1511] Processing CreateTablet for tablet 6788efa5ae124a478e06ef927cb631ff (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260709 12:26:29.842981 11358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 393b5b835714456ab06eeecb9b4a116d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.843881 11510 tablet_service.cc:1511] Processing CreateTablet for tablet 6c4cc7481ed24aa6991531d818969460 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:26:29.845333 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4cc7481ed24aa6991531d818969460. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.844985 11356 tablet_service.cc:1511] Processing CreateTablet for tablet e57eb43303724eae812867b69fa94d9d (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260709 12:26:29.847240 11356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e57eb43303724eae812867b69fa94d9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.848506 11437 tablet_service.cc:1511] Processing CreateTablet for tablet 6c4cc7481ed24aa6991531d818969460 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:26:29.849943 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4cc7481ed24aa6991531d818969460. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.850720 11357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6788efa5ae124a478e06ef927cb631ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.858089 11568 tablet_bootstrap.cc:492] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:29.867810 11568 tablet_bootstrap.cc:654] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:29.874715 11285 tablet_service.cc:1511] Processing CreateTablet for tablet 6c4cc7481ed24aa6991531d818969460 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260709 12:26:29.876292 11285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4cc7481ed24aa6991531d818969460. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.880028 11567 tablet_bootstrap.cc:492] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:29.901697 11355 tablet_service.cc:1511] Processing CreateTablet for tablet f32b983bac2949418ec7a6bf8a0ecdea (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:26:29.903299 11355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f32b983bac2949418ec7a6bf8a0ecdea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.909003 11508 tablet_service.cc:1511] Processing CreateTablet for tablet f32b983bac2949418ec7a6bf8a0ecdea (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:26:29.910809 11508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f32b983bac2949418ec7a6bf8a0ecdea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.904135 11434 tablet_service.cc:1511] Processing CreateTablet for tablet f32b983bac2949418ec7a6bf8a0ecdea (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260709 12:26:29.917364 11434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f32b983bac2949418ec7a6bf8a0ecdea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.920258 11567 tablet_bootstrap.cc:654] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:29.920643 11566 tablet_bootstrap.cc:492] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:29.921212 11566 ts_tablet_manager.cc:1403] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.125s	user 0.017s	sys 0.023s
I20260709 12:26:29.923273 11284 tablet_service.cc:1511] Processing CreateTablet for tablet 57d6a333ba744c8389730bd74362cb43 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:26:29.925278 11284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57d6a333ba744c8389730bd74362cb43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.938129 11354 tablet_service.cc:1511] Processing CreateTablet for tablet 57d6a333ba744c8389730bd74362cb43 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:26:29.941342 11431 tablet_service.cc:1511] Processing CreateTablet for tablet 57d6a333ba744c8389730bd74362cb43 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260709 12:26:29.946602 11511 tablet_service.cc:1511] Processing CreateTablet for tablet fa781c13298b41e2ba513a74a2455086 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:26:29.948295 11511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa781c13298b41e2ba513a74a2455086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.959384 11568 tablet_bootstrap.cc:492] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:29.960057 11568 ts_tablet_manager.cc:1403] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.102s	user 0.014s	sys 0.003s
I20260709 12:26:29.964148 11568 raft_consensus.cc:359] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:29.964890 11568 raft_consensus.cc:385] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:29.965199 11568 raft_consensus.cc:740] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:29.966187 11568 consensus_queue.cc:260] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:29.971551 11566 raft_consensus.cc:359] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:29.972600 11566 raft_consensus.cc:385] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:29.973016 11566 raft_consensus.cc:740] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:29.973978 11566 consensus_queue.cc:260] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:29.982092 11354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57d6a333ba744c8389730bd74362cb43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.996806 11431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57d6a333ba744c8389730bd74362cb43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:29.997366 11571 tablet_bootstrap.cc:492] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:30.007757 11472 heartbeater.cc:499] Master 127.29.201.254:38597 was elected leader, sending a full tablet report...
I20260709 12:26:30.008363 11568 ts_tablet_manager.cc:1434] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.048s	user 0.014s	sys 0.011s
I20260709 12:26:30.009588 11568 tablet_bootstrap.cc:492] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:30.014324 11437 tablet_service.cc:1511] Processing CreateTablet for tablet bfa4ec6799b5454a9d750ba5a79fafa3 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:26:30.015789 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa4ec6799b5454a9d750ba5a79fafa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.015892 11567 tablet_bootstrap.cc:492] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:30.016705 11568 tablet_bootstrap.cc:654] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.016544 11567 ts_tablet_manager.cc:1403] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.137s	user 0.017s	sys 0.021s
I20260709 12:26:30.017047 11508 tablet_service.cc:1511] Processing CreateTablet for tablet bfa4ec6799b5454a9d750ba5a79fafa3 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:26:30.018488 11508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa4ec6799b5454a9d750ba5a79fafa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.020466 11567 raft_consensus.cc:359] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:30.021297 11567 raft_consensus.cc:385] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:30.021600 11567 raft_consensus.cc:740] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:30.016510 11281 tablet_service.cc:1511] Processing CreateTablet for tablet fa781c13298b41e2ba513a74a2455086 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:26:30.022598 11567 consensus_queue.cc:260] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:30.024629 11571 tablet_bootstrap.cc:654] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.024866 11281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa781c13298b41e2ba513a74a2455086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.031219 11323 heartbeater.cc:499] Master 127.29.201.254:38597 was elected leader, sending a full tablet report...
I20260709 12:26:30.031586 11363 tablet_service.cc:1511] Processing CreateTablet for tablet fa781c13298b41e2ba513a74a2455086 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260709 12:26:30.034612 11363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa781c13298b41e2ba513a74a2455086. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.043378 11566 ts_tablet_manager.cc:1434] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.122s	user 0.010s	sys 0.005s
I20260709 12:26:30.044703 11566 tablet_bootstrap.cc:492] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:30.058110 11567 ts_tablet_manager.cc:1434] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.041s	user 0.009s	sys 0.012s
I20260709 12:26:30.059013 11546 heartbeater.cc:499] Master 127.29.201.254:38597 was elected leader, sending a full tablet report...
I20260709 12:26:30.064101 11284 tablet_service.cc:1511] Processing CreateTablet for tablet bfa4ec6799b5454a9d750ba5a79fafa3 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260709 12:26:30.065670 11566 tablet_bootstrap.cc:654] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.065698 11284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfa4ec6799b5454a9d750ba5a79fafa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.073082 11567 tablet_bootstrap.cc:492] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:30.090116 11354 tablet_service.cc:1511] Processing CreateTablet for tablet e313f6102e9941248ba9069990945017 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:26:30.091694 11354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e313f6102e9941248ba9069990945017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.097636 11568 tablet_bootstrap.cc:492] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:30.098272 11568 ts_tablet_manager.cc:1403] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.089s	user 0.010s	sys 0.004s
I20260709 12:26:30.101362 11568 raft_consensus.cc:359] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:30.102109 11568 raft_consensus.cc:385] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:30.102401 11568 raft_consensus.cc:740] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
W20260709 12:26:30.105677 11547 tablet.cc:2406] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:26:30.107524 11571 tablet_bootstrap.cc:492] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:30.108825 11571 ts_tablet_manager.cc:1403] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.112s	user 0.014s	sys 0.005s
I20260709 12:26:30.108798 11568 consensus_queue.cc:260] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:30.111800 11568 ts_tablet_manager.cc:1434] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 12:26:30.113206 11568 tablet_bootstrap.cc:492] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:30.113412 11571 raft_consensus.cc:359] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.114441 11571 raft_consensus.cc:385] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:30.114907 11571 raft_consensus.cc:740] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:30.120038 11571 consensus_queue.cc:260] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.125111 11573 raft_consensus.cc:493] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.125910 11573 raft_consensus.cc:515] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:30.127153 11281 tablet_service.cc:1511] Processing CreateTablet for tablet e9836ae3a7d04d53ab2d78656c19c8f2 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:26:30.128803 11281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9836ae3a7d04d53ab2d78656c19c8f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.136212 11567 tablet_bootstrap.cc:654] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.136662 11508 tablet_service.cc:1511] Processing CreateTablet for tablet e313f6102e9941248ba9069990945017 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:26:30.137851 11398 heartbeater.cc:499] Master 127.29.201.254:38597 was elected leader, sending a full tablet report...
I20260709 12:26:30.138352 11508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e313f6102e9941248ba9069990945017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.141995 11573 leader_election.cc:290] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:30.144712 11437 tablet_service.cc:1511] Processing CreateTablet for tablet e313f6102e9941248ba9069990945017 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260709 12:26:30.148731 11354 tablet_service.cc:1511] Processing CreateTablet for tablet e9836ae3a7d04d53ab2d78656c19c8f2 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:26:30.150507 11354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9836ae3a7d04d53ab2d78656c19c8f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.151788 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e313f6102e9941248ba9069990945017. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.153997 11431 tablet_service.cc:1511] Processing CreateTablet for tablet e9836ae3a7d04d53ab2d78656c19c8f2 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260709 12:26:30.161665 11511 tablet_service.cc:1511] Processing CreateTablet for tablet 6d4384a4da6846e39db4bcd0330cb822 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:26:30.162549 11571 ts_tablet_manager.cc:1434] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.053s	user 0.009s	sys 0.004s
I20260709 12:26:30.163358 11511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d4384a4da6846e39db4bcd0330cb822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.163894 11571 tablet_bootstrap.cc:492] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:30.164149 11431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9836ae3a7d04d53ab2d78656c19c8f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.142158 11568 tablet_bootstrap.cc:654] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.163394 11284 tablet_service.cc:1511] Processing CreateTablet for tablet 6d4384a4da6846e39db4bcd0330cb822 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:26:30.173998 11284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d4384a4da6846e39db4bcd0330cb822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.175181 11573 raft_consensus.cc:493] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.175923 11573 raft_consensus.cc:515] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:30.215726 11434 tablet_service.cc:1511] Processing CreateTablet for tablet 0081c481615e480ea8dfafe6a806ee7f (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:26:30.217289 11434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0081c481615e480ea8dfafe6a806ee7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.219251 11285 tablet_service.cc:1511] Processing CreateTablet for tablet 0081c481615e480ea8dfafe6a806ee7f (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:26:30.220656 11510 tablet_service.cc:1511] Processing CreateTablet for tablet 0081c481615e480ea8dfafe6a806ee7f (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260709 12:26:30.221298 11285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0081c481615e480ea8dfafe6a806ee7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.222007 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0081c481615e480ea8dfafe6a806ee7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.223706 11356 tablet_service.cc:1511] Processing CreateTablet for tablet 6d4384a4da6846e39db4bcd0330cb822 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260709 12:26:30.228535 11436 tablet_service.cc:1511] Processing CreateTablet for tablet b0c4260f7779463dab087f51bf433ab8 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:26:30.230568 11436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0c4260f7779463dab087f51bf433ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.193312 11573 leader_election.cc:290] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:30.247690 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8236893668af4eaa93bc53b6715f6cf0" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:30.238412 11566 tablet_bootstrap.cc:492] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:30.251518 11566 ts_tablet_manager.cc:1403] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.207s	user 0.016s	sys 0.023s
W20260709 12:26:30.252629 11411 leader_election.cc:343] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.226429 11356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d4384a4da6846e39db4bcd0330cb822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.226300 11571 tablet_bootstrap.cc:654] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.235047 11360 tablet_service.cc:1511] Processing CreateTablet for tablet b0c4260f7779463dab087f51bf433ab8 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:26:30.260128 11360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0c4260f7779463dab087f51bf433ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.276403 11506 tablet_service.cc:1511] Processing CreateTablet for tablet b0c4260f7779463dab087f51bf433ab8 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260709 12:26:30.277958 11506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0c4260f7779463dab087f51bf433ab8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.283072 11566 raft_consensus.cc:359] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:30.283982 11566 raft_consensus.cc:385] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:30.284319 11566 raft_consensus.cc:740] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:30.284381 11298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8236893668af4eaa93bc53b6715f6cf0" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:30.285399 11566 consensus_queue.cc:260] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:30.286413 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5734e27937894275a18ba504ffedd114" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
W20260709 12:26:30.288926 11412 leader_election.cc:343] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.289470 11412 leader_election.cc:304] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:30.290359 11573 raft_consensus.cc:2749] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260709 12:26:30.290863 11412 leader_election.cc:343] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.294879 11568 tablet_bootstrap.cc:492] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:30.299800 11568 ts_tablet_manager.cc:1403] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.187s	user 0.016s	sys 0.045s
I20260709 12:26:30.308130 11571 tablet_bootstrap.cc:492] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:30.308779 11571 ts_tablet_manager.cc:1403] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.145s	user 0.010s	sys 0.006s
I20260709 12:26:30.311384 11568 raft_consensus.cc:359] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.312280 11568 raft_consensus.cc:385] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:30.312662 11567 tablet_bootstrap.cc:492] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:30.313380 11567 ts_tablet_manager.cc:1403] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.241s	user 0.012s	sys 0.009s
I20260709 12:26:30.313958 11574 raft_consensus.cc:493] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.314529 11574 raft_consensus.cc:515] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:30.312615 11568 raft_consensus.cc:740] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:30.326712 11363 tablet_service.cc:1511] Processing CreateTablet for tablet 3701a37fd75248f6b8b0ca0930791a9d (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:26:30.348479 11363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3701a37fd75248f6b8b0ca0930791a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.318522 11567 raft_consensus.cc:359] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:30.349134 11567 raft_consensus.cc:385] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:30.354460 11567 raft_consensus.cc:740] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:30.328373 11284 tablet_service.cc:1511] Processing CreateTablet for tablet 3701a37fd75248f6b8b0ca0930791a9d (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:26:30.359377 11567 consensus_queue.cc:260] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:30.365226 11436 tablet_service.cc:1511] Processing CreateTablet for tablet 2925eddaf8944bf5bf72cd650ed1e287 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:26:30.366671 11436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2925eddaf8944bf5bf72cd650ed1e287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.325248 11574 leader_election.cc:290] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:30.368835 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5734e27937894275a18ba504ffedd114" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:30.369630 11521 raft_consensus.cc:2468] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:30.371605 11410 leader_election.cc:304] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:30.372555 11567 ts_tablet_manager.cc:1434] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.059s	user 0.006s	sys 0.002s
I20260709 12:26:30.360289 11284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3701a37fd75248f6b8b0ca0930791a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.330804 11505 tablet_service.cc:1511] Processing CreateTablet for tablet fee758b4d91541148427a240c8f1db4e (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:26:30.351583 11579 raft_consensus.cc:493] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.331399 11568 consensus_queue.cc:260] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.340778 11571 raft_consensus.cc:359] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.332775 11281 tablet_service.cc:1511] Processing CreateTablet for tablet fee758b4d91541148427a240c8f1db4e (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:26:30.377070 11281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee758b4d91541148427a240c8f1db4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.372507 11573 raft_consensus.cc:2804] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:30.382838 11573 raft_consensus.cc:493] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:30.383363 11573 raft_consensus.cc:3060] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:30.385638 11510 tablet_service.cc:1511] Processing CreateTablet for tablet 2925eddaf8944bf5bf72cd650ed1e287 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:26:30.386202 11285 tablet_service.cc:1511] Processing CreateTablet for tablet 2925eddaf8944bf5bf72cd650ed1e287 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260709 12:26:30.387816 11285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2925eddaf8944bf5bf72cd650ed1e287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.391197 11354 tablet_service.cc:1511] Processing CreateTablet for tablet fee758b4d91541148427a240c8f1db4e (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260709 12:26:30.355170 11431 tablet_service.cc:1511] Processing CreateTablet for tablet 3701a37fd75248f6b8b0ca0930791a9d (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260709 12:26:30.392765 11354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee758b4d91541148427a240c8f1db4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.383411 11567 tablet_bootstrap.cc:492] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:30.375149 11579 raft_consensus.cc:515] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.381449 11505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee758b4d91541148427a240c8f1db4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.397519 11579 leader_election.cc:290] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:30.328071 11566 ts_tablet_manager.cc:1434] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.076s	user 0.007s	sys 0.001s
I20260709 12:26:30.376113 11571 raft_consensus.cc:385] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:30.392838 11431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3701a37fd75248f6b8b0ca0930791a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.415095 11437 tablet_service.cc:1511] Processing CreateTablet for tablet 4a5d23b0089f4d37a0c63d127537ec93 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:26:30.415323 11356 tablet_service.cc:1511] Processing CreateTablet for tablet fea9a20aab6840c1afb32c29bac2f4a4 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:26:30.411218 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2925eddaf8944bf5bf72cd650ed1e287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.420601 11356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fea9a20aab6840c1afb32c29bac2f4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.416391 11288 tablet_service.cc:1511] Processing CreateTablet for tablet fea9a20aab6840c1afb32c29bac2f4a4 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:26:30.427665 11288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fea9a20aab6840c1afb32c29bac2f4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.428910 11574 raft_consensus.cc:493] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.429509 11574 raft_consensus.cc:515] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:30.430233 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6788efa5ae124a478e06ef927cb631ff" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:30.420603 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a5d23b0089f4d37a0c63d127537ec93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.427390 11506 tablet_service.cc:1511] Processing CreateTablet for tablet 4a5d23b0089f4d37a0c63d127537ec93 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:26:30.398882 11571 raft_consensus.cc:740] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:30.438660 11571 consensus_queue.cc:260] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.442389 11571 ts_tablet_manager.cc:1434] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.133s	user 0.005s	sys 0.004s
I20260709 12:26:30.433302 11574 leader_election.cc:290] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
W20260709 12:26:30.434427 11262 leader_election.cc:343] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.435039 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd76933be5f240a1a5c069775527f562" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:30.419137 11566 tablet_bootstrap.cc:492] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:30.417239 11568 ts_tablet_manager.cc:1434] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.117s	user 0.012s	sys 0.007s
I20260709 12:26:30.449539 11575 raft_consensus.cc:493] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.401073 11360 tablet_service.cc:1511] Processing CreateTablet for tablet 4a5d23b0089f4d37a0c63d127537ec93 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260709 12:26:30.461846 11571 tablet_bootstrap.cc:492] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
W20260709 12:26:30.462342 11262 leader_election.cc:343] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.467583 11568 tablet_bootstrap.cc:492] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:30.457583 11575 raft_consensus.cc:515] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:30.469482 11508 tablet_service.cc:1511] Processing CreateTablet for tablet 578fd826b687468ca8be54244c3b4bc0 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:26:30.469482 11286 tablet_service.cc:1511] Processing CreateTablet for tablet 578fd826b687468ca8be54244c3b4bc0 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:26:30.471140 11362 tablet_service.cc:1511] Processing CreateTablet for tablet 578fd826b687468ca8be54244c3b4bc0 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260709 12:26:30.472131 11286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578fd826b687468ca8be54244c3b4bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.472390 11575 leader_election.cc:290] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:30.444734 11573 raft_consensus.cc:515] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:30.487463 11434 tablet_service.cc:1511] Processing CreateTablet for tablet fea9a20aab6840c1afb32c29bac2f4a4 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260709 12:26:30.489024 11434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fea9a20aab6840c1afb32c29bac2f4a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.495007 11567 tablet_bootstrap.cc:654] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.522280 11360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a5d23b0089f4d37a0c63d127537ec93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.528237 11571 tablet_bootstrap.cc:654] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.529099 11362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578fd826b687468ca8be54244c3b4bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.534560 11435 tablet_service.cc:1511] Processing CreateTablet for tablet 827a79573ac94525a9ff30c744c45164 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:26:30.536149 11435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827a79573ac94525a9ff30c744c45164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.502017 11506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a5d23b0089f4d37a0c63d127537ec93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.546166 11508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578fd826b687468ca8be54244c3b4bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.549109 11566 tablet_bootstrap.cc:654] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.549640 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6788efa5ae124a478e06ef927cb631ff" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:30.551885 11511 tablet_service.cc:1511] Processing CreateTablet for tablet 827a79573ac94525a9ff30c744c45164 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:26:30.552897 11573 leader_election.cc:290] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:30.553326 11511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827a79573ac94525a9ff30c744c45164. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.554448 11281 tablet_service.cc:1511] Processing CreateTablet for tablet 827a79573ac94525a9ff30c744c45164 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260709 12:26:30.547041 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd76933be5f240a1a5c069775527f562" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:30.555836 11281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 827a79573ac94525a9ff30c744c45164. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:26:30.557036 11261 leader_election.cc:343] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.557607 11261 leader_election.cc:304] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1
W20260709 12:26:30.559062 11261 leader_election.cc:343] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.559535 11261 leader_election.cc:304] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:30.560843 11574 raft_consensus.cc:2749] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:30.563429 11574 raft_consensus.cc:2749] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:30.555014 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5734e27937894275a18ba504ffedd114" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
W20260709 12:26:30.565359 11412 leader_election.cc:343] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.589963 11363 tablet_service.cc:1511] Processing CreateTablet for tablet ee281c78e97848f59386b1d97f96bcfe (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:26:30.606001 11568 tablet_bootstrap.cc:654] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.607331 11363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee281c78e97848f59386b1d97f96bcfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.630312 11566 tablet_bootstrap.cc:492] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:30.633761 11567 tablet_bootstrap.cc:492] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:30.635149 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5734e27937894275a18ba504ffedd114" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:30.672266 11567 ts_tablet_manager.cc:1403] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.289s	user 0.016s	sys 0.006s
I20260709 12:26:30.672380 11521 raft_consensus.cc:3060] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:30.639621 11437 tablet_service.cc:1511] Processing CreateTablet for tablet ee281c78e97848f59386b1d97f96bcfe (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:26:30.650012 11505 tablet_service.cc:1511] Processing CreateTablet for tablet ee281c78e97848f59386b1d97f96bcfe (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260709 12:26:30.675877 11505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee281c78e97848f59386b1d97f96bcfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.651101 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8586c90fc9848248b387776dc1f7db6" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:30.654297 11356 tablet_service.cc:1511] Processing CreateTablet for tablet 0175551ed6284d01b80e6a34e24a07ed (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
W20260709 12:26:30.655038 11261 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:36253, user_credentials={real_user=slave}} blocked reactor thread for 96007.5us
I20260709 12:26:30.655963 11285 tablet_service.cc:1511] Processing CreateTablet for tablet 0175551ed6284d01b80e6a34e24a07ed (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:26:30.656633 11435 tablet_service.cc:1511] Processing CreateTablet for tablet 0175551ed6284d01b80e6a34e24a07ed (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260709 12:26:30.675877 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ee281c78e97848f59386b1d97f96bcfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.683669 11285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0175551ed6284d01b80e6a34e24a07ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.684767 11510 tablet_service.cc:1511] Processing CreateTablet for tablet 1cf22a8b3fa4491c92c735f8151982d3 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:26:30.686389 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cf22a8b3fa4491c92c735f8151982d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.687230 11356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0175551ed6284d01b80e6a34e24a07ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.681799 11566 ts_tablet_manager.cc:1403] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.263s	user 0.010s	sys 0.013s
I20260709 12:26:30.691592 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8586c90fc9848248b387776dc1f7db6" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:30.692292 11566 raft_consensus.cc:359] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:30.693123 11566 raft_consensus.cc:385] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:26:30.693194 11485 leader_election.cc:343] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.693518 11566 raft_consensus.cc:740] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:30.696475 11288 tablet_service.cc:1511] Processing CreateTablet for tablet 1cf22a8b3fa4491c92c735f8151982d3 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:26:30.697945 11288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cf22a8b3fa4491c92c735f8151982d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.687873 11567 raft_consensus.cc:359] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.715010 11568 tablet_bootstrap.cc:492] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:30.715711 11568 ts_tablet_manager.cc:1403] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.248s	user 0.033s	sys 0.027s
I20260709 12:26:30.716274 11510 tablet_service.cc:1511] Processing CreateTablet for tablet 47908558560f437a92445e6ab3a30a57 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:26:30.727936 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47908558560f437a92445e6ab3a30a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.730098 11434 tablet_service.cc:1511] Processing CreateTablet for tablet 47908558560f437a92445e6ab3a30a57 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:26:30.730217 11360 tablet_service.cc:1511] Processing CreateTablet for tablet 267085844c6943b08c0f42b220e7e8ab (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:26:30.731813 11360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267085844c6943b08c0f42b220e7e8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.718888 11435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0175551ed6284d01b80e6a34e24a07ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.720795 11571 tablet_bootstrap.cc:492] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:30.720677 11568 raft_consensus.cc:359] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:30.737437 11568 raft_consensus.cc:385] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:30.737630 11571 ts_tablet_manager.cc:1403] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.276s	user 0.014s	sys 0.008s
I20260709 12:26:30.737879 11568 raft_consensus.cc:740] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:30.721661 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43ed653326ff4ae6b3c7daa594b3228b" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:30.739425 11568 consensus_queue.cc:260] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:30.743157 11568 ts_tablet_manager.cc:1434] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.027s	user 0.007s	sys 0.002s
I20260709 12:26:30.744499 11568 tablet_bootstrap.cc:492] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:30.745918 11434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47908558560f437a92445e6ab3a30a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.748963 11579 raft_consensus.cc:493] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.749612 11579 raft_consensus.cc:515] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.722721 11354 tablet_service.cc:1511] Processing CreateTablet for tablet 1cf22a8b3fa4491c92c735f8151982d3 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260709 12:26:30.722879 11281 tablet_service.cc:1511] Processing CreateTablet for tablet 47908558560f437a92445e6ab3a30a57 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260709 12:26:30.723917 11521 raft_consensus.cc:2468] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:30.751133 11354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cf22a8b3fa4491c92c735f8151982d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.751521 11281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47908558560f437a92445e6ab3a30a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.751884 11410 leader_election.cc:304] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:30.752782 11573 raft_consensus.cc:2804] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:30.717636 11566 consensus_queue.cc:260] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:30.753249 11573 raft_consensus.cc:697] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:30.754144 11573 consensus_queue.cc:237] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:30.754840 11566 ts_tablet_manager.cc:1434] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.066s	user 0.007s	sys 0.001s
I20260709 12:26:30.725127 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43ed653326ff4ae6b3c7daa594b3228b" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:30.711731 11567 raft_consensus.cc:385] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:30.756191 11579 leader_election.cc:290] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:30.756453 11567 raft_consensus.cc:740] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:30.757436 11567 consensus_queue.cc:260] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.750146 11571 raft_consensus.cc:359] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:30.759019 11571 raft_consensus.cc:385] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:30.759315 11571 raft_consensus.cc:740] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:30.759924 11567 ts_tablet_manager.cc:1434] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.087s	user 0.007s	sys 0.007s
I20260709 12:26:30.760445 11520 raft_consensus.cc:2468] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:30.761381 11567 tablet_bootstrap.cc:492] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:30.756309 11297 raft_consensus.cc:2468] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:30.766894 11571 consensus_queue.cc:260] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:30.768819 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f32b983bac2949418ec7a6bf8a0ecdea" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
W20260709 12:26:30.769824 11336 leader_election.cc:343] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.771164 11566 tablet_bootstrap.cc:492] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:30.772188 11567 tablet_bootstrap.cc:654] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.773311 11336 leader_election.cc:304] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:30.776639 11437 tablet_service.cc:1511] Processing CreateTablet for tablet 267085844c6943b08c0f42b220e7e8ab (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260709 12:26:30.778867 11579 raft_consensus.cc:2804] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:30.781281 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f32b983bac2949418ec7a6bf8a0ecdea" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:30.783083 11486 leader_election.cc:304] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e
I20260709 12:26:30.782742 11590 raft_consensus.cc:493] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260709 12:26:30.776681 11336 leader_election.cc:343] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.784385 11590 raft_consensus.cc:515] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.785059 11575 raft_consensus.cc:2804] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:26:30.785730 11336 leader_election.cc:343] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.786514 11336 leader_election.cc:304] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433
I20260709 12:26:30.787807 11590 leader_election.cc:290] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:30.789276 11579 raft_consensus.cc:493] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:30.789088 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "393b5b835714456ab06eeecb9b4a116d" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:30.789964 11579 raft_consensus.cc:3060] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:30.785892 11575 raft_consensus.cc:493] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:30.790846 11575 raft_consensus.cc:3060] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:30.791610 11411 leader_election.cc:343] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.797740 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267085844c6943b08c0f42b220e7e8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.802299 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "393b5b835714456ab06eeecb9b4a116d" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:30.801399 11510 tablet_service.cc:1511] Processing CreateTablet for tablet 267085844c6943b08c0f42b220e7e8ab (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
W20260709 12:26:30.803927 11410 leader_election.cc:343] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.804478 11410 leader_election.cc:304] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433
I20260709 12:26:30.804651 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 267085844c6943b08c0f42b220e7e8ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.806047 11590 raft_consensus.cc:2749] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:30.814966 11566 tablet_bootstrap.cc:654] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.819453 11603 raft_consensus.cc:2749] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:30.828604 11567 tablet_bootstrap.cc:492] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:30.829352 11567 ts_tablet_manager.cc:1403] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.068s	user 0.011s	sys 0.020s
I20260709 12:26:30.833328 11568 tablet_bootstrap.cc:654] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.833675 11288 tablet_service.cc:1511] Processing CreateTablet for tablet 8b0f568d1ffe4160afdf9d1bb4ba7c42 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:26:30.833480 11567 raft_consensus.cc:359] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:30.840286 11567 raft_consensus.cc:385] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:30.840710 11567 raft_consensus.cc:740] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:30.840977 11288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0f568d1ffe4160afdf9d1bb4ba7c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.834684 11579 raft_consensus.cc:515] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.844647 11579 leader_election.cc:290] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:30.835904 11571 ts_tablet_manager.cc:1434] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.098s	user 0.007s	sys 0.000s
I20260709 12:26:30.846335 11571 tablet_bootstrap.cc:492] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:30.846524 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43ed653326ff4ae6b3c7daa594b3228b" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:30.841854 11567 consensus_queue.cc:260] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
W20260709 12:26:30.848471 11336 leader_election.cc:343] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:30.842340 11244 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:30.849228 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43ed653326ff4ae6b3c7daa594b3228b" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:30.849983 11520 raft_consensus.cc:3060] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:30.857470 11566 tablet_bootstrap.cc:492] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:30.858253 11566 ts_tablet_manager.cc:1403] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.087s	user 0.012s	sys 0.000s
I20260709 12:26:30.869699 11571 tablet_bootstrap.cc:654] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:30.862323 11566 raft_consensus.cc:359] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:30.875383 11566 raft_consensus.cc:385] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:30.875720 11566 raft_consensus.cc:740] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:30.884387 11568 tablet_bootstrap.cc:492] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:30.885116 11568 ts_tablet_manager.cc:1403] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.141s	user 0.008s	sys 0.006s
I20260709 12:26:30.885555 11575 raft_consensus.cc:515] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:30.890107 11568 raft_consensus.cc:359] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.910992 11568 raft_consensus.cc:385] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:30.911463 11568 raft_consensus.cc:740] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:30.910480 11194 catalog_manager.cc:5697] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:30.912568 11568 consensus_queue.cc:260] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:30.893762 11566 consensus_queue.cc:260] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:30.895259 11360 tablet_service.cc:1511] Processing CreateTablet for tablet 8b0f568d1ffe4160afdf9d1bb4ba7c42 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:26:30.914914 11579 raft_consensus.cc:493] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:30.895301 11435 tablet_service.cc:1511] Processing CreateTablet for tablet 8b0f568d1ffe4160afdf9d1bb4ba7c42 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260709 12:26:30.915455 11579 raft_consensus.cc:515] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:30.916301 11435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0f568d1ffe4160afdf9d1bb4ba7c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.917135 11566 ts_tablet_manager.cc:1434] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.058s	user 0.010s	sys 0.000s
I20260709 12:26:30.919114 11360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b0f568d1ffe4160afdf9d1bb4ba7c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.922256 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db58e810d4a04064bbf12daef58026f3" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:30.923519 11566 tablet_bootstrap.cc:492] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:30.899123 11510 tablet_service.cc:1511] Processing CreateTablet for tablet b4438988b178419385f7ec82ef7c99d2 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
W20260709 12:26:30.925031 11336 leader_election.cc:343] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.928164 11568 ts_tablet_manager.cc:1434] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.043s	user 0.011s	sys 0.000s
I20260709 12:26:30.930351 11579 leader_election.cc:290] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:30.929809 11568 tablet_bootstrap.cc:492] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:30.931950 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4438988b178419385f7ec82ef7c99d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:30.906654 11520 raft_consensus.cc:2468] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:30.941535 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db58e810d4a04064bbf12daef58026f3" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:30.942354 11297 raft_consensus.cc:2468] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:30.943914 11338 leader_election.cc:304] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:30.944103 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8586c90fc9848248b387776dc1f7db6" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:30.945035 11297 raft_consensus.cc:3060] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:30.945218 11579 raft_consensus.cc:2804] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:30.945741 11579 raft_consensus.cc:493] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:30.946094 11579 raft_consensus.cc:3060] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:30.954272 11579 raft_consensus.cc:515] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:30.957759 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db58e810d4a04064bbf12daef58026f3" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
W20260709 12:26:30.959340 11336 leader_election.cc:343] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.962200 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8586c90fc9848248b387776dc1f7db6" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
W20260709 12:26:30.964654 11485 leader_election.cc:343] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:30.970045 11298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db58e810d4a04064bbf12daef58026f3" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:30.970666 11336 leader_election.cc:304] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:30.971067 11298 raft_consensus.cc:3060] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:30.979012 11603 raft_consensus.cc:2804] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:30.965545 11579 leader_election.cc:290] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:30.971851 11575 leader_election.cc:290] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:30.992657 11288 tablet_service.cc:1511] Processing CreateTablet for tablet b4438988b178419385f7ec82ef7c99d2 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:26:30.993304 11362 tablet_service.cc:1511] Processing CreateTablet for tablet b4438988b178419385f7ec82ef7c99d2 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260709 12:26:31.021768 11567 ts_tablet_manager.cc:1434] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.192s	user 0.046s	sys 0.063s
I20260709 12:26:31.000370 11505 tablet_service.cc:1511] Processing CreateTablet for tablet 8471ace194cf46238a8cd2862aa8cd5d (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:26:31.005215 11281 tablet_service.cc:1511] Processing CreateTablet for tablet 8471ace194cf46238a8cd2862aa8cd5d (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:26:31.008141 11356 tablet_service.cc:1511] Processing CreateTablet for tablet 5eda5b4099d041c9aef56884ca4e594a (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:26:31.008774 11298 raft_consensus.cc:2468] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:31.023818 11571 tablet_bootstrap.cc:492] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:31.024370 11571 ts_tablet_manager.cc:1403] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.178s	user 0.022s	sys 0.022s
I20260709 12:26:31.021915 11362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4438988b178419385f7ec82ef7c99d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.027678 11571 raft_consensus.cc:359] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:31.028409 11571 raft_consensus.cc:385] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:31.028936 11571 raft_consensus.cc:740] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:31.029044 11431 tablet_service.cc:1511] Processing CreateTablet for tablet 16773253264a4f8282ea1ea5b59a1ec6 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:26:31.013523 11566 tablet_bootstrap.cc:654] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:31.013266 11434 tablet_service.cc:1511] Processing CreateTablet for tablet 5eda5b4099d041c9aef56884ca4e594a (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:26:31.016252 11506 tablet_service.cc:1511] Processing CreateTablet for tablet 5eda5b4099d041c9aef56884ca4e594a (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260709 12:26:31.020283 11285 tablet_service.cc:1511] Processing CreateTablet for tablet 16773253264a4f8282ea1ea5b59a1ec6 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:26:30.995343 11437 tablet_service.cc:1511] Processing CreateTablet for tablet 8471ace194cf46238a8cd2862aa8cd5d (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260709 12:26:31.033417 11434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eda5b4099d041c9aef56884ca4e594a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.034209 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8471ace194cf46238a8cd2862aa8cd5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.035732 11505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8471ace194cf46238a8cd2862aa8cd5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.035936 11571 consensus_queue.cc:260] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:31.039806 11506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eda5b4099d041c9aef56884ca4e594a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.040228 11297 raft_consensus.cc:2468] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:31.041519 11486 leader_election.cc:304] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e
I20260709 12:26:31.022876 11603 raft_consensus.cc:697] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:31.042815 11603 consensus_queue.cc:237] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:31.032768 11356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5eda5b4099d041c9aef56884ca4e594a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.049311 11286 tablet_service.cc:1511] Processing CreateTablet for tablet 1ce080ae4c98415a868195fb1df17f18 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:26:31.050734 11286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ce080ae4c98415a868195fb1df17f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.023993 11338 leader_election.cc:304] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:31.054455 11609 raft_consensus.cc:2804] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:31.055020 11609 raft_consensus.cc:697] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:31.055866 11609 consensus_queue.cc:237] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:31.058010 11285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16773253264a4f8282ea1ea5b59a1ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.059285 11281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8471ace194cf46238a8cd2862aa8cd5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.061152 11431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16773253264a4f8282ea1ea5b59a1ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.060956 11363 tablet_service.cc:1511] Processing CreateTablet for tablet 1ce080ae4c98415a868195fb1df17f18 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:26:31.062438 11363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ce080ae4c98415a868195fb1df17f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.012985 11568 tablet_bootstrap.cc:654] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:31.045869 11510 tablet_service.cc:1511] Processing CreateTablet for tablet 1ce080ae4c98415a868195fb1df17f18 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260709 12:26:31.042809 11575 raft_consensus.cc:2804] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:31.038473 11571 ts_tablet_manager.cc:1434] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.001s
I20260709 12:26:31.068897 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1ce080ae4c98415a868195fb1df17f18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.072784 11571 tablet_bootstrap.cc:492] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:31.074216 11579 raft_consensus.cc:493] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.074824 11579 raft_consensus.cc:515] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:31.078032 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "393b5b835714456ab06eeecb9b4a116d" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:31.078780 11446 raft_consensus.cc:2468] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:31.070847 11575 raft_consensus.cc:697] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:31.078658 11288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4438988b178419385f7ec82ef7c99d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.080603 11336 leader_election.cc:304] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:31.087939 11567 tablet_bootstrap.cc:492] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:31.088071 11579 leader_election.cc:290] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:31.097118 11579 raft_consensus.cc:2804] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:31.107491 11437 tablet_service.cc:1511] Processing CreateTablet for tablet b8da681b4f1a47ff8327c26e796ad3c3 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:26:31.080443 11575 consensus_queue.cc:237] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:31.113354 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8da681b4f1a47ff8327c26e796ad3c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.109524 11579 raft_consensus.cc:493] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:31.123157 11579 raft_consensus.cc:3060] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:31.129782 11566 tablet_bootstrap.cc:492] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:31.134397 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "393b5b835714456ab06eeecb9b4a116d" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:31.134330 11360 tablet_service.cc:1511] Processing CreateTablet for tablet 16773253264a4f8282ea1ea5b59a1ec6 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260709 12:26:31.137769 11360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16773253264a4f8282ea1ea5b59a1ec6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.153225 11284 tablet_service.cc:1511] Processing CreateTablet for tablet b8da681b4f1a47ff8327c26e796ad3c3 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
W20260709 12:26:31.153891 11336 leader_election.cc:343] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:31.157836 11510 tablet_service.cc:1511] Processing CreateTablet for tablet b8da681b4f1a47ff8327c26e796ad3c3 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260709 12:26:31.159534 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8da681b4f1a47ff8327c26e796ad3c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.157439 11566 ts_tablet_manager.cc:1403] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.208s	user 0.025s	sys 0.009s
I20260709 12:26:31.166286 11284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8da681b4f1a47ff8327c26e796ad3c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.187974 11196 catalog_manager.cc:5697] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:31.198526 11362 tablet_service.cc:1511] Processing CreateTablet for tablet 91b161cce9ce49c58571f631c6810266 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:26:31.210269 11571 tablet_bootstrap.cc:654] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:31.210772 11362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91b161cce9ce49c58571f631c6810266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.202124 11568 tablet_bootstrap.cc:492] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:31.205608 11573 raft_consensus.cc:493] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.206993 11195 catalog_manager.cc:5697] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:31.209239 11566 raft_consensus.cc:359] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:31.215987 11573 raft_consensus.cc:515] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:31.221791 11573 leader_election.cc:290] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:31.216428 11566 raft_consensus.cc:385] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:31.227317 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91074436b5b44779742b3478249f283" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:31.228091 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91074436b5b44779742b3478249f283" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:31.235570 11520 raft_consensus.cc:2468] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:31.236749 11566 raft_consensus.cc:740] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:31.237049 11410 leader_election.cc:304] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:31.237838 11566 consensus_queue.cc:260] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:31.238448 11573 raft_consensus.cc:2804] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:31.239079 11297 raft_consensus.cc:2468] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:31.230095 11437 tablet_service.cc:1511] Processing CreateTablet for tablet 91b161cce9ce49c58571f631c6810266 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:26:31.241082 11567 tablet_bootstrap.cc:654] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:31.241626 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91b161cce9ce49c58571f631c6810266. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.194310 11579 raft_consensus.cc:515] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:31.257381 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "393b5b835714456ab06eeecb9b4a116d" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:31.258075 11446 raft_consensus.cc:3060] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:31.239115 11573 raft_consensus.cc:493] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:31.217736 11568 ts_tablet_manager.cc:1403] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.288s	user 0.010s	sys 0.011s
I20260709 12:26:31.259907 11579 leader_election.cc:290] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:31.267885 11568 raft_consensus.cc:359] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:31.271077 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "393b5b835714456ab06eeecb9b4a116d" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:31.268684 11568 raft_consensus.cc:385] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260709 12:26:31.280719 11336 leader_election.cc:343] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:31.281421 11506 tablet_service.cc:1511] Processing CreateTablet for tablet 91b161cce9ce49c58571f631c6810266 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260709 12:26:31.283676 11506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91b161cce9ce49c58571f631c6810266. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:26:31.284889 11412 consensus_peers.cc:597] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:31.282131 11567 tablet_bootstrap.cc:492] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:31.293002 11446 raft_consensus.cc:2468] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:31.299366 11567 ts_tablet_manager.cc:1403] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.232s	user 0.005s	sys 0.010s
I20260709 12:26:31.299188 11285 tablet_service.cc:1511] Processing CreateTablet for tablet 50c3c1a7849646fc95a663b6cd0427e8 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:26:31.300294 11336 leader_election.cc:304] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:31.275197 11573 raft_consensus.cc:3060] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:31.302145 11356 tablet_service.cc:1511] Processing CreateTablet for tablet 50c3c1a7849646fc95a663b6cd0427e8 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:26:31.296330 11566 ts_tablet_manager.cc:1434] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.125s	user 0.008s	sys 0.000s
I20260709 12:26:31.305543 11566 tablet_bootstrap.cc:492] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:31.305370 11567 raft_consensus.cc:359] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:31.306236 11567 raft_consensus.cc:385] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:31.298077 11434 tablet_service.cc:1511] Processing CreateTablet for tablet 50c3c1a7849646fc95a663b6cd0427e8 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260709 12:26:31.300599 11285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c3c1a7849646fc95a663b6cd0427e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.288776 11568 raft_consensus.cc:740] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:31.302902 11579 raft_consensus.cc:2804] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:31.316658 11579 raft_consensus.cc:697] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:31.317703 11579 consensus_queue.cc:237] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:31.317395 11568 consensus_queue.cc:260] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:31.320937 11568 ts_tablet_manager.cc:1434] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.059s	user 0.008s	sys 0.000s
I20260709 12:26:31.306555 11567 raft_consensus.cc:740] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:31.327656 11356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c3c1a7849646fc95a663b6cd0427e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.329282 11590 raft_consensus.cc:493] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.330267 11505 tablet_service.cc:1511] Processing CreateTablet for tablet 1d019dd2bd864166abac89cc3fb1be76 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:26:31.331856 11505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d019dd2bd864166abac89cc3fb1be76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.332513 11568 tablet_bootstrap.cc:492] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:31.330092 11590 raft_consensus.cc:515] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:31.348301 11288 tablet_service.cc:1511] Processing CreateTablet for tablet c8f2372474d94a11bb1e044fc69c60e5 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:26:31.349069 11571 tablet_bootstrap.cc:492] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:31.349634 11571 ts_tablet_manager.cc:1403] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.277s	user 0.021s	sys 0.008s
I20260709 12:26:31.349749 11288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8f2372474d94a11bb1e044fc69c60e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.351109 11567 consensus_queue.cc:260] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:31.351635 11510 tablet_service.cc:1511] Processing CreateTablet for tablet c8f2372474d94a11bb1e044fc69c60e5 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:26:31.353286 11590 leader_election.cc:290] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:31.354451 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43ed653326ff4ae6b3c7daa594b3228b" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:31.355337 11520 raft_consensus.cc:2393] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a1ce6e048ab47d69c453b5218cb64c1 in current term 1: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:31.357355 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8f2372474d94a11bb1e044fc69c60e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.361657 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43ed653326ff4ae6b3c7daa594b3228b" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:31.367411 11411 leader_election.cc:304] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433
I20260709 12:26:31.313484 11281 tablet_service.cc:1511] Processing CreateTablet for tablet 1d019dd2bd864166abac89cc3fb1be76 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:26:31.370867 11590 raft_consensus.cc:3060] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:31.371614 11281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d019dd2bd864166abac89cc3fb1be76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.373903 11567 ts_tablet_manager.cc:1434] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.074s	user 0.002s	sys 0.006s
I20260709 12:26:31.375290 11567 tablet_bootstrap.cc:492] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:31.376034 11571 raft_consensus.cc:359] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:31.376988 11571 raft_consensus.cc:385] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:31.377393 11571 raft_consensus.cc:740] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:31.382381 11571 consensus_queue.cc:260] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:31.384238 11434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50c3c1a7849646fc95a663b6cd0427e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.383421 11363 tablet_service.cc:1511] Processing CreateTablet for tablet 1d019dd2bd864166abac89cc3fb1be76 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260709 12:26:31.385408 11363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d019dd2bd864166abac89cc3fb1be76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.392728 11573 raft_consensus.cc:515] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:31.395416 11573 leader_election.cc:290] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:31.396198 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91074436b5b44779742b3478249f283" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:31.396898 11521 raft_consensus.cc:3060] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:31.399809 11590 raft_consensus.cc:2749] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:31.400893 11590 consensus_queue.cc:1048] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:31.404865 11571 ts_tablet_manager.cc:1434] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.053s	user 0.009s	sys 0.000s
I20260709 12:26:31.407599 11521 raft_consensus.cc:2468] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:31.417487 11410 leader_election.cc:304] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:31.418565 11624 raft_consensus.cc:2804] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:31.419195 11624 raft_consensus.cc:697] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:31.420094 11437 tablet_service.cc:1511] Processing CreateTablet for tablet c8f2372474d94a11bb1e044fc69c60e5 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260709 12:26:31.420373 11571 tablet_bootstrap.cc:492] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:31.420105 11624 consensus_queue.cc:237] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:31.421308 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8f2372474d94a11bb1e044fc69c60e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.423144 11566 tablet_bootstrap.cc:654] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:31.429684 11571 tablet_bootstrap.cc:654] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:31.411402 11568 tablet_bootstrap.cc:654] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:31.432360 11567 tablet_bootstrap.cc:654] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:31.478219 11283 tablet_service.cc:1511] Processing CreateTablet for tablet cc185f43f658441ba207bdfdc0e79a7f (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:26:31.484834 11283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc185f43f658441ba207bdfdc0e79a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.479418 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c91074436b5b44779742b3478249f283" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:31.507261 11297 raft_consensus.cc:3060] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:31.478221 11284 tablet_service.cc:1511] Processing CreateTablet for tablet 7179cfd71f5c4bcf8beac36a18e4c94e (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:26:31.478220 11286 tablet_service.cc:1511] Processing CreateTablet for tablet 610a732cba674c24bb228d619f52fce7 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:26:31.487936 11436 tablet_service.cc:1511] Processing CreateTablet for tablet cc185f43f658441ba207bdfdc0e79a7f (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:26:31.514313 11436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc185f43f658441ba207bdfdc0e79a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.488229 11431 tablet_service.cc:1511] Processing CreateTablet for tablet 019a6ed4536442de96c44bde2d713c8b (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:26:31.494019 11360 tablet_service.cc:1511] Processing CreateTablet for tablet 019a6ed4536442de96c44bde2d713c8b (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:26:31.516721 11431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019a6ed4536442de96c44bde2d713c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.494496 11510 tablet_service.cc:1511] Processing CreateTablet for tablet 019a6ed4536442de96c44bde2d713c8b (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260709 12:26:31.496155 11511 tablet_service.cc:1511] Processing CreateTablet for tablet cc185f43f658441ba207bdfdc0e79a7f (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260709 12:26:31.496610 11508 tablet_service.cc:1511] Processing CreateTablet for tablet 610a732cba674c24bb228d619f52fce7 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:26:31.519523 11508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 610a732cba674c24bb228d619f52fce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.521564 11573 raft_consensus.cc:493] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.496156 11362 tablet_service.cc:1511] Processing CreateTablet for tablet 7179cfd71f5c4bcf8beac36a18e4c94e (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:26:31.497619 11435 tablet_service.cc:1511] Processing CreateTablet for tablet 7179cfd71f5c4bcf8beac36a18e4c94e (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260709 12:26:31.523823 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019a6ed4536442de96c44bde2d713c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.522621 11566 tablet_bootstrap.cc:492] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:31.522348 11567 tablet_bootstrap.cc:492] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:31.523036 11360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019a6ed4536442de96c44bde2d713c8b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:26:31.546969 11626 log.cc:927] Time spent T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1: Append to log took a long time: real 0.093s	user 0.001s	sys 0.000s
I20260709 12:26:31.547968 11362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7179cfd71f5c4bcf8beac36a18e4c94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.525978 11575 raft_consensus.cc:493] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.559614 11575 raft_consensus.cc:515] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:31.560694 11284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7179cfd71f5c4bcf8beac36a18e4c94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.563740 11567 ts_tablet_manager.cc:1403] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.189s	user 0.014s	sys 0.003s
I20260709 12:26:31.522527 11573 raft_consensus.cc:515] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:31.567404 11298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57eb43303724eae812867b69fa94d9d" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:31.569144 11511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc185f43f658441ba207bdfdc0e79a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.569846 11573 leader_election.cc:290] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
W20260709 12:26:31.571520 11486 leader_election.cc:343] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:31.575907 11435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7179cfd71f5c4bcf8beac36a18e4c94e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.581584 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6788efa5ae124a478e06ef927cb631ff" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:31.582931 11373 raft_consensus.cc:2468] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:31.584942 11411 leader_election.cc:304] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:31.584950 11193 catalog_manager.cc:5697] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:31.525609 11354 tablet_service.cc:1511] Processing CreateTablet for tablet 610a732cba674c24bb228d619f52fce7 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260709 12:26:31.581527 11298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6788efa5ae124a478e06ef927cb631ff" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:31.596992 11437 tablet_service.cc:1511] Processing CreateTablet for tablet 75195e7d58a548658a538f4628a056d8 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:26:31.588459 11193 catalog_manager.cc:5697] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:31.603138 11575 leader_election.cc:290] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:31.602280 11506 tablet_service.cc:1511] Processing CreateTablet for tablet 75195e7d58a548658a538f4628a056d8 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:26:31.590973 11298 raft_consensus.cc:2468] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:31.655078 11506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75195e7d58a548658a538f4628a056d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.601336 11357 tablet_service.cc:1511] Processing CreateTablet for tablet 75195e7d58a548658a538f4628a056d8 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260709 12:26:31.658255 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75195e7d58a548658a538f4628a056d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.605707 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57eb43303724eae812867b69fa94d9d" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
W20260709 12:26:31.662171 11485 leader_election.cc:343] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:31.662731 11485 leader_election.cc:304] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:31.663575 11575 raft_consensus.cc:2749] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:31.590500 11354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 610a732cba674c24bb228d619f52fce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.589500 11568 tablet_bootstrap.cc:492] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:31.608070 11571 tablet_bootstrap.cc:492] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:31.587379 11573 raft_consensus.cc:2804] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:31.587450 11286 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 610a732cba674c24bb228d619f52fce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.587054 11566 ts_tablet_manager.cc:1403] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.282s	user 0.016s	sys 0.010s
I20260709 12:26:31.589788 11567 raft_consensus.cc:359] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:31.674878 11571 ts_tablet_manager.cc:1403] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.255s	user 0.016s	sys 0.019s
I20260709 12:26:31.586596 11297 raft_consensus.cc:2468] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:31.682819 11609 consensus_queue.cc:1048] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:26:31.687829 11485 consensus_peers.cc:597] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:31.689949 11357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75195e7d58a548658a538f4628a056d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.692775 11285 tablet_service.cc:1511] Processing CreateTablet for tablet f74457716f0d46568488137f1cb73fa8 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:26:31.606691 11579 consensus_queue.cc:1048] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.089s
I20260709 12:26:31.739688 11579 consensus_queue.cc:1048] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:31.715955 11573 raft_consensus.cc:493] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:31.736603 11485 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:44567, user_credentials={real_user=slave}} blocked reactor thread for 48523.2us
I20260709 12:26:31.738226 11567 raft_consensus.cc:385] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:31.753154 11434 tablet_service.cc:1511] Processing CreateTablet for tablet f74457716f0d46568488137f1cb73fa8 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:26:31.760919 11573 raft_consensus.cc:3060] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:31.769975 11285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f74457716f0d46568488137f1cb73fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.761874 11567 raft_consensus.cc:740] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:31.768059 11568 ts_tablet_manager.cc:1403] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.436s	user 0.020s	sys 0.019s
I20260709 12:26:31.782147 11508 tablet_service.cc:1511] Processing CreateTablet for tablet 32c7abdd125b4f38b6398a9b0e92fedd (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:26:31.783049 11434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f74457716f0d46568488137f1cb73fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.784032 11288 tablet_service.cc:1511] Processing CreateTablet for tablet 32c7abdd125b4f38b6398a9b0e92fedd (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:26:31.773372 11575 consensus_queue.cc:1048] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260709 12:26:31.786583 11508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32c7abdd125b4f38b6398a9b0e92fedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.787577 11568 raft_consensus.cc:359] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:31.788408 11568 raft_consensus.cc:385] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:31.789006 11568 raft_consensus.cc:740] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:31.789660 11288 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32c7abdd125b4f38b6398a9b0e92fedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.613008 11194 catalog_manager.cc:5697] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:31.793020 11336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:36253, user_credentials={real_user=slave}} blocked reactor thread for 76514.9us
I20260709 12:26:31.761547 11571 raft_consensus.cc:359] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:31.800755 11571 raft_consensus.cc:385] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:31.801165 11571 raft_consensus.cc:740] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:31.811424 11567 consensus_queue.cc:260] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:31.801985 11571 consensus_queue.cc:260] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:31.816780 11567 ts_tablet_manager.cc:1434] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.252s	user 0.017s	sys 0.008s
I20260709 12:26:31.816836 11568 consensus_queue.cc:260] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
W20260709 12:26:31.818506 11486 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 46757.2us
I20260709 12:26:31.825110 11573 raft_consensus.cc:515] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:31.839110 11360 tablet_service.cc:1511] Processing CreateTablet for tablet f74457716f0d46568488137f1cb73fa8 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260709 12:26:31.854570 11284 tablet_service.cc:1511] Processing CreateTablet for tablet 0c068ba8213a415b9df4e5380d0ad37e (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:26:31.856475 11284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c068ba8213a415b9df4e5380d0ad37e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:26:31.857272 11244 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:31.861184 11435 tablet_service.cc:1511] Processing CreateTablet for tablet e637809da62b408a91639b907355cad2 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:26:31.863255 11435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e637809da62b408a91639b907355cad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.863476 11505 tablet_service.cc:1511] Processing CreateTablet for tablet e637809da62b408a91639b907355cad2 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:26:31.864650 11573 leader_election.cc:290] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:31.864912 11505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e637809da62b408a91639b907355cad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.867060 11567 tablet_bootstrap.cc:492] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:31.869356 11571 ts_tablet_manager.cc:1434] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.133s	user 0.016s	sys 0.018s
I20260709 12:26:31.872378 11296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6788efa5ae124a478e06ef927cb631ff" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:31.873162 11296 raft_consensus.cc:3060] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:31.876348 11436 tablet_service.cc:1511] Processing CreateTablet for tablet 86bb3bcc0a6449a9adc6c21b722f6602 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:26:31.876694 11281 tablet_service.cc:1511] Processing CreateTablet for tablet 86bb3bcc0a6449a9adc6c21b722f6602 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:26:31.877839 11436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86bb3bcc0a6449a9adc6c21b722f6602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.879969 11507 tablet_service.cc:1511] Processing CreateTablet for tablet 359eb877601e4d519d6e7d6f3835b9a2 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:26:31.881464 11507 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 359eb877601e4d519d6e7d6f3835b9a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.884832 11590 consensus_queue.cc:1048] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:31.885098 11283 tablet_service.cc:1511] Processing CreateTablet for tablet 359eb877601e4d519d6e7d6f3835b9a2 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:26:31.886512 11510 tablet_service.cc:1511] Processing CreateTablet for tablet 0c068ba8213a415b9df4e5380d0ad37e (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:26:31.887159 11362 tablet_service.cc:1511] Processing CreateTablet for tablet 32c7abdd125b4f38b6398a9b0e92fedd (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260709 12:26:31.887816 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c068ba8213a415b9df4e5380d0ad37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.887642 11358 tablet_service.cc:1511] Processing CreateTablet for tablet 86bb3bcc0a6449a9adc6c21b722f6602 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260709 12:26:31.889178 11356 tablet_service.cc:1511] Processing CreateTablet for tablet e637809da62b408a91639b907355cad2 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260709 12:26:31.891937 11568 ts_tablet_manager.cc:1434] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.115s	user 0.006s	sys 0.001s
I20260709 12:26:31.891607 11355 tablet_service.cc:1511] Processing CreateTablet for tablet 359eb877601e4d519d6e7d6f3835b9a2 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260709 12:26:31.893466 11568 tablet_bootstrap.cc:492] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:31.877874 11281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86bb3bcc0a6449a9adc6c21b722f6602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.907492 11283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 359eb877601e4d519d6e7d6f3835b9a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.909776 11434 tablet_service.cc:1511] Processing CreateTablet for tablet bccc5d3239f3484eb991213e177b66cc (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:26:31.917915 11437 tablet_service.cc:1511] Processing CreateTablet for tablet 0c068ba8213a415b9df4e5380d0ad37e (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260709 12:26:31.898700 11590 raft_consensus.cc:493] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.927691 11434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bccc5d3239f3484eb991213e177b66cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.931282 11573 raft_consensus.cc:493] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:31.936393 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c068ba8213a415b9df4e5380d0ad37e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.944972 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6788efa5ae124a478e06ef927cb631ff" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:31.945662 11373 raft_consensus.cc:3060] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:31.949013 11571 tablet_bootstrap.cc:492] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:31.931955 11573 raft_consensus.cc:515] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:31.952971 11573 leader_election.cc:290] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:31.953972 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8236893668af4eaa93bc53b6715f6cf0" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:31.956161 11358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86bb3bcc0a6449a9adc6c21b722f6602. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.955847 11566 raft_consensus.cc:359] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:31.956683 11566 raft_consensus.cc:385] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:31.956885 11355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 359eb877601e4d519d6e7d6f3835b9a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.957021 11566 raft_consensus.cc:740] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:31.958371 11356 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e637809da62b408a91639b907355cad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.958246 11566 consensus_queue.cc:260] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:31.966882 11362 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32c7abdd125b4f38b6398a9b0e92fedd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.925823 11590 raft_consensus.cc:515] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
W20260709 12:26:31.970798 11412 leader_election.cc:343] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:31.957187 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8236893668af4eaa93bc53b6715f6cf0" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:31.978055 11372 raft_consensus.cc:2468] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:31.978679 11284 tablet_service.cc:1511] Processing CreateTablet for tablet bccc5d3239f3484eb991213e177b66cc (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:26:31.979480 11521 raft_consensus.cc:3060] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:31.979930 11590 leader_election.cc:290] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:31.984369 11511 tablet_service.cc:1511] Processing CreateTablet for tablet bccc5d3239f3484eb991213e177b66cc (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260709 12:26:31.986657 11509 tablet_service.cc:1511] Processing CreateTablet for tablet 431c66d13d0f4904a42b33665a9c09e1 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:26:31.990062 11511 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bccc5d3239f3484eb991213e177b66cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.992815 11509 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431c66d13d0f4904a42b33665a9c09e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.994094 11363 tablet_service.cc:1511] Processing CreateTablet for tablet 431c66d13d0f4904a42b33665a9c09e1 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:26:31.995666 11363 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431c66d13d0f4904a42b33665a9c09e1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:26:31.997004 11336 consensus_peers.cc:597] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Couldn't send request to peer 9968c0ed1974454d828996824f859433. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:31.997817 11609 consensus_queue.cc:1048] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:31.995611 11437 tablet_service.cc:1511] Processing CreateTablet for tablet 431c66d13d0f4904a42b33665a9c09e1 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260709 12:26:32.004117 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 431c66d13d0f4904a42b33665a9c09e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:32.011041 11285 tablet_service.cc:1511] Processing CreateTablet for tablet b3c0ce8bbe4f4392bdac15d4d35f84dd (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:26:32.013550 11435 tablet_service.cc:1511] Processing CreateTablet for tablet b3c0ce8bbe4f4392bdac15d4d35f84dd (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:26:32.014998 11360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f74457716f0d46568488137f1cb73fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:31.978072 11571 tablet_bootstrap.cc:654] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.023716 11284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bccc5d3239f3484eb991213e177b66cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:32.024811 11566 ts_tablet_manager.cc:1434] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.278s	user 0.010s	sys 0.000s
I20260709 12:26:32.024926 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd76933be5f240a1a5c069775527f562" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:32.025712 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd76933be5f240a1a5c069775527f562" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:32.042536 11294 raft_consensus.cc:2468] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:32.031759 11411 leader_election.cc:304] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:32.035640 11296 raft_consensus.cc:2468] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:32.037111 11285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c0ce8bbe4f4392bdac15d4d35f84dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:32.047899 11412 leader_election.cc:304] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:32.014992 11435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c0ce8bbe4f4392bdac15d4d35f84dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:26:32.075661 11411 leader_election.cc:343] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:32.098939 11624 raft_consensus.cc:2804] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:32.099444 11624 raft_consensus.cc:493] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.099793 11624 raft_consensus.cc:3060] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.077293 11573 raft_consensus.cc:2804] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:32.111013 11573 raft_consensus.cc:493] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:32.094877 11412 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 47076.3us
I20260709 12:26:32.135577 11357 tablet_service.cc:1511] Processing CreateTablet for tablet ff00b27947404e52acea283e9381d837 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:26:32.135582 11506 tablet_service.cc:1511] Processing CreateTablet for tablet ff00b27947404e52acea283e9381d837 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:26:32.142328 11294 raft_consensus.cc:3060] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.131664 11566 tablet_bootstrap.cc:492] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:32.151490 11354 tablet_service.cc:1511] Processing CreateTablet for tablet b3c0ce8bbe4f4392bdac15d4d35f84dd (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260709 12:26:32.155761 11567 tablet_bootstrap.cc:654] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.123373 11634 consensus_queue.cc:1048] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.051s
I20260709 12:26:32.158918 11412 leader_election.cc:304] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:32.160068 11632 raft_consensus.cc:2804] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:32.160578 11632 raft_consensus.cc:697] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:32.161550 11632 consensus_queue.cc:237] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.165881 11373 raft_consensus.cc:2468] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:32.168519 11506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff00b27947404e52acea283e9381d837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:32.168342 11510 tablet_service.cc:1511] Processing CreateTablet for tablet 70f75f882db94790be4eaf2994c44f61 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:26:32.168843 11284 tablet_service.cc:1511] Processing CreateTablet for tablet 70f75f882db94790be4eaf2994c44f61 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:26:32.169678 11510 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f75f882db94790be4eaf2994c44f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:32.170528 11281 tablet_service.cc:1511] Processing CreateTablet for tablet ff00b27947404e52acea283e9381d837 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260709 12:26:32.132331 11573 raft_consensus.cc:3060] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.174863 11357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff00b27947404e52acea283e9381d837. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:26:32.145336 11410 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 65240.5us
I20260709 12:26:32.193588 11437 tablet_service.cc:1511] Processing CreateTablet for tablet 70f75f882db94790be4eaf2994c44f61 (DEFAULT_TABLE table=test-workload [id=aa8eb9fe3d414fd2b12a837922689ec2]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260709 12:26:32.195122 11437 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f75f882db94790be4eaf2994c44f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:32.195775 11284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70f75f882db94790be4eaf2994c44f61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:32.197504 11354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3c0ce8bbe4f4392bdac15d4d35f84dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:32.208618 11281 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff00b27947404e52acea283e9381d837. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:26:32.231616 11568 tablet_bootstrap.cc:654] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.252494 11637 raft_consensus.cc:493] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.253101 11637 raft_consensus.cc:515] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:32.254041 11571 tablet_bootstrap.cc:492] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:32.254699 11571 ts_tablet_manager.cc:1403] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.306s	user 0.015s	sys 0.004s
I20260709 12:26:32.258003 11571 raft_consensus.cc:359] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.258832 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "393b5b835714456ab06eeecb9b4a116d" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:32.259805 11637 leader_election.cc:290] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:32.261185 11573 raft_consensus.cc:515] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.266463 11571 raft_consensus.cc:385] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.297781 11566 tablet_bootstrap.cc:654] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.292168 11571 raft_consensus.cc:740] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.325028 11573 leader_election.cc:290] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:32.310494 11624 raft_consensus.cc:515] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.368932 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8236893668af4eaa93bc53b6715f6cf0" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
W20260709 12:26:32.369838 11651 log.cc:927] Time spent T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1: Append to log took a long time: real 0.050s	user 0.001s	sys 0.000s
I20260709 12:26:32.369974 11373 raft_consensus.cc:3060] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.372856 11296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8236893668af4eaa93bc53b6715f6cf0" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
W20260709 12:26:32.375190 11412 leader_election.cc:343] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:32.362330 11571 consensus_queue.cc:260] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.381011 11567 tablet_bootstrap.cc:492] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:32.381564 11373 raft_consensus.cc:2468] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:32.383648 11411 leader_election.cc:304] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:32.384995 11573 raft_consensus.cc:2804] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:32.385661 11573 raft_consensus.cc:697] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:32.387096 11573 consensus_queue.cc:237] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.391584 11624 leader_election.cc:290] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:32.393481 11296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd76933be5f240a1a5c069775527f562" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:32.394155 11571 ts_tablet_manager.cc:1434] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.139s	user 0.005s	sys 0.005s
I20260709 12:26:32.394469 11296 raft_consensus.cc:3060] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.381839 11567 ts_tablet_manager.cc:1403] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.515s	user 0.016s	sys 0.028s
I20260709 12:26:32.412510 11571 tablet_bootstrap.cc:492] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:32.424022 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd76933be5f240a1a5c069775527f562" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:32.424912 11373 raft_consensus.cc:3060] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.427767 11567 raft_consensus.cc:359] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.429008 11296 raft_consensus.cc:2468] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:32.433624 11579 consensus_queue.cc:1048] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:32.436374 11568 tablet_bootstrap.cc:492] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:32.436940 11568 ts_tablet_manager.cc:1403] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.544s	user 0.027s	sys 0.026s
I20260709 12:26:32.437908 11571 tablet_bootstrap.cc:654] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.430032 11567 raft_consensus.cc:385] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.443361 11412 leader_election.cc:304] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:32.444669 11624 raft_consensus.cc:2804] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:32.445537 11624 raft_consensus.cc:697] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:32.446571 11624 consensus_queue.cc:237] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.448735 11568 raft_consensus.cc:359] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.451965 11568 raft_consensus.cc:385] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.452309 11568 raft_consensus.cc:740] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.457304 11566 tablet_bootstrap.cc:492] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:32.438892 11567 raft_consensus.cc:740] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.457986 11566 ts_tablet_manager.cc:1403] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.370s	user 0.025s	sys 0.006s
I20260709 12:26:32.467928 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "393b5b835714456ab06eeecb9b4a116d" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:32.469580 11566 raft_consensus.cc:359] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.472605 11484 leader_election.cc:304] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:32.470621 11566 raft_consensus.cc:385] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.474224 11652 raft_consensus.cc:2749] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:32.475298 11567 consensus_queue.cc:260] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.478817 11567 ts_tablet_manager.cc:1434] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.083s	user 0.005s	sys 0.004s
I20260709 12:26:32.487254 11373 raft_consensus.cc:2468] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:32.480577 11568 consensus_queue.cc:260] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.481048 11566 raft_consensus.cc:740] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.489641 11638 raft_consensus.cc:493] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5a1ce6e048ab47d69c453b5218cb64c1)
I20260709 12:26:32.490270 11638 raft_consensus.cc:515] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.490222 11566 consensus_queue.cc:260] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.493920 11568 ts_tablet_manager.cc:1434] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.057s	user 0.006s	sys 0.005s
I20260709 12:26:32.495616 11568 tablet_bootstrap.cc:492] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:32.496515 11566 ts_tablet_manager.cc:1434] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.034s	user 0.008s	sys 0.000s
I20260709 12:26:32.497927 11566 tablet_bootstrap.cc:492] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:32.498621 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5734e27937894275a18ba504ffedd114" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:32.501259 11571 tablet_bootstrap.cc:492] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:32.501914 11571 ts_tablet_manager.cc:1403] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.104s	user 0.014s	sys 0.001s
I20260709 12:26:32.503377 11568 tablet_bootstrap.cc:654] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.505250 11571 raft_consensus.cc:359] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.506126 11571 raft_consensus.cc:385] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.506496 11571 raft_consensus.cc:740] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.507332 11571 consensus_queue.cc:260] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.509496 11571 ts_tablet_manager.cc:1434] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:26:32.510166 11566 tablet_bootstrap.cc:654] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.510814 11571 tablet_bootstrap.cc:492] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:32.517915 11571 tablet_bootstrap.cc:654] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.524405 11566 tablet_bootstrap.cc:492] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:32.525214 11566 ts_tablet_manager.cc:1403] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.028s	user 0.008s	sys 0.011s
I20260709 12:26:32.525079 11638 leader_election.cc:290] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:32.530846 11571 tablet_bootstrap.cc:492] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:32.530074 11566 raft_consensus.cc:359] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.531476 11571 ts_tablet_manager.cc:1403] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.021s	user 0.009s	sys 0.011s
I20260709 12:26:32.531692 11566 raft_consensus.cc:385] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.532286 11566 raft_consensus.cc:740] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.533360 11566 consensus_queue.cc:260] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.535113 11571 raft_consensus.cc:359] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.535903 11571 raft_consensus.cc:385] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.536346 11571 raft_consensus.cc:740] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.537240 11566 ts_tablet_manager.cc:1434] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.011s	user 0.010s	sys 0.000s
I20260709 12:26:32.537217 11571 consensus_queue.cc:260] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.539059 11566 tablet_bootstrap.cc:492] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:32.540254 11571 ts_tablet_manager.cc:1434] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:26:32.541585 11571 tablet_bootstrap.cc:492] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:32.551585 11195 catalog_manager.cc:5697] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:32.555918 11566 tablet_bootstrap.cc:654] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.560012 11567 tablet_bootstrap.cc:492] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:32.561519 11571 tablet_bootstrap.cc:654] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.553711 11195 catalog_manager.cc:5697] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:32.567391 11195 catalog_manager.cc:5697] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:32.570405 11609 raft_consensus.cc:493] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.570433 11566 tablet_bootstrap.cc:492] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:32.571187 11609 raft_consensus.cc:515] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.571722 11566 ts_tablet_manager.cc:1403] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.033s	user 0.025s	sys 0.000s
I20260709 12:26:32.575085 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8586c90fc9848248b387776dc1f7db6" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:32.576830 11566 raft_consensus.cc:359] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.577953 11566 raft_consensus.cc:385] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.578779 11609 leader_election.cc:290] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:32.579105 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8586c90fc9848248b387776dc1f7db6" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:32.582021 11338 leader_election.cc:304] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:32.583370 11571 tablet_bootstrap.cc:492] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:32.584024 11571 ts_tablet_manager.cc:1403] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.043s	user 0.013s	sys 0.006s
I20260709 12:26:32.583777 11609 raft_consensus.cc:2749] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:32.578344 11566 raft_consensus.cc:740] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.585310 11566 consensus_queue.cc:260] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.587689 11571 raft_consensus.cc:359] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.588554 11571 raft_consensus.cc:385] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.588886 11571 raft_consensus.cc:740] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.589114 11566 ts_tablet_manager.cc:1434] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.017s	user 0.009s	sys 0.000s
I20260709 12:26:32.589759 11571 consensus_queue.cc:260] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.591109 11652 raft_consensus.cc:493] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.591922 11652 raft_consensus.cc:515] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.592790 11571 ts_tablet_manager.cc:1434] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:26:32.594064 11571 tablet_bootstrap.cc:492] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:32.596212 11652 leader_election.cc:290] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:32.597537 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db58e810d4a04064bbf12daef58026f3" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:32.591053 11566 tablet_bootstrap.cc:492] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:32.599112 11624 consensus_queue.cc:1048] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:32.597534 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db58e810d4a04064bbf12daef58026f3" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:32.601267 11486 leader_election.cc:304] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:32.603183 11652 raft_consensus.cc:2749] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:32.603449 11571 tablet_bootstrap.cc:654] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.603880 11567 tablet_bootstrap.cc:654] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.609477 11566 tablet_bootstrap.cc:654] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.610514 11571 tablet_bootstrap.cc:492] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:32.611131 11571 ts_tablet_manager.cc:1403] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.017s	user 0.012s	sys 0.001s
I20260709 12:26:32.614253 11568 tablet_bootstrap.cc:492] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:32.614923 11568 ts_tablet_manager.cc:1403] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.120s	user 0.042s	sys 0.059s
I20260709 12:26:32.617799 11571 raft_consensus.cc:359] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:32.618716 11571 raft_consensus.cc:385] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.619131 11571 raft_consensus.cc:740] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.618322 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5734e27937894275a18ba504ffedd114" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:32.620000 11571 consensus_queue.cc:260] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:32.621169 11261 leader_election.cc:304] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433
I20260709 12:26:32.618322 11568 raft_consensus.cc:359] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:32.622176 11568 raft_consensus.cc:385] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.622568 11568 raft_consensus.cc:740] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.622663 11638 raft_consensus.cc:2749] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:32.624770 11624 raft_consensus.cc:493] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.625516 11624 raft_consensus.cc:515] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.626168 11638 raft_consensus.cc:493] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.627018 11638 raft_consensus.cc:515] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.629686 11638 leader_election.cc:290] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:32.630472 11624 leader_election.cc:290] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:32.631826 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c4cc7481ed24aa6991531d818969460" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:32.631657 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c4cc7481ed24aa6991531d818969460" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:32.632616 11294 raft_consensus.cc:2468] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:32.631565 11568 consensus_queue.cc:260] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:32.634087 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c4cc7481ed24aa6991531d818969460" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:32.634233 11412 leader_election.cc:304] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:32.632591 11446 raft_consensus.cc:2468] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
W20260709 12:26:32.635836 11261 leader_election.cc:343] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:32.636417 11573 raft_consensus.cc:2804] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:32.636720 11261 leader_election.cc:304] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:32.629701 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c4cc7481ed24aa6991531d818969460" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:32.638398 11573 raft_consensus.cc:493] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.635823 11568 ts_tablet_manager.cc:1434] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.020s	user 0.007s	sys 0.001s
I20260709 12:26:32.638964 11573 raft_consensus.cc:3060] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:32.640554 11410 leader_election.cc:343] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:32.639066 11638 raft_consensus.cc:2804] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:32.644187 11638 raft_consensus.cc:493] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.644654 11638 raft_consensus.cc:3060] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.640727 11568 tablet_bootstrap.cc:492] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:32.653285 11632 consensus_queue.cc:1048] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:32.654394 11571 ts_tablet_manager.cc:1434] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.043s	user 0.008s	sys 0.000s
I20260709 12:26:32.655359 11568 tablet_bootstrap.cc:654] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.655829 11571 tablet_bootstrap.cc:492] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:32.664572 11567 tablet_bootstrap.cc:492] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:32.665232 11567 ts_tablet_manager.cc:1403] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.106s	user 0.012s	sys 0.003s
I20260709 12:26:32.668751 11571 tablet_bootstrap.cc:654] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.673759 11638 raft_consensus.cc:515] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.677134 11568 tablet_bootstrap.cc:492] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:32.677929 11568 ts_tablet_manager.cc:1403] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.038s	user 0.018s	sys 0.009s
I20260709 12:26:32.678946 11573 raft_consensus.cc:515] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.681732 11568 raft_consensus.cc:359] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.682493 11568 raft_consensus.cc:385] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.682808 11568 raft_consensus.cc:740] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.683061 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c4cc7481ed24aa6991531d818969460" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:32.682992 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c4cc7481ed24aa6991531d818969460" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:32.684007 11294 raft_consensus.cc:2393] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a1ce6e048ab47d69c453b5218cb64c1 in current term 1: Already voted for candidate 663d88d9634e4720a695c7cd69eaa25e in this term.
I20260709 12:26:32.684051 11568 consensus_queue.cc:260] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.687817 11568 ts_tablet_manager.cc:1434] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.004s
I20260709 12:26:32.689337 11568 tablet_bootstrap.cc:492] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:32.689590 11638 leader_election.cc:290] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:32.691517 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c4cc7481ed24aa6991531d818969460" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:32.678474 11567 raft_consensus.cc:359] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.692135 11567 raft_consensus.cc:385] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.692430 11567 raft_consensus.cc:740] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.693033 11566 tablet_bootstrap.cc:492] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:32.693653 11566 ts_tablet_manager.cc:1403] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.103s	user 0.037s	sys 0.033s
I20260709 12:26:32.693895 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c4cc7481ed24aa6991531d818969460" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:32.695079 11446 raft_consensus.cc:2393] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 663d88d9634e4720a695c7cd69eaa25e in current term 1: Already voted for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in this term.
I20260709 12:26:32.695741 11568 tablet_bootstrap.cc:654] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.696615 11566 raft_consensus.cc:359] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.696861 11261 leader_election.cc:304] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433
I20260709 12:26:32.697358 11566 raft_consensus.cc:385] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.697700 11566 raft_consensus.cc:740] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.698016 11638 raft_consensus.cc:2749] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:26:32.698578 11566 consensus_queue.cc:260] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.700055 11571 tablet_bootstrap.cc:492] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:32.700887 11571 ts_tablet_manager.cc:1403] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.045s	user 0.015s	sys 0.007s
I20260709 12:26:32.701685 11566 ts_tablet_manager.cc:1434] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.008s	user 0.002s	sys 0.003s
I20260709 12:26:32.693163 11567 consensus_queue.cc:260] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.702927 11521 raft_consensus.cc:3060] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.705627 11571 raft_consensus.cc:359] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.693953 11573 leader_election.cc:290] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:32.707233 11568 tablet_bootstrap.cc:492] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:32.708676 11568 ts_tablet_manager.cc:1403] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.020s	user 0.014s	sys 0.003s
I20260709 12:26:32.709223 11566 tablet_bootstrap.cc:492] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:32.711774 11568 raft_consensus.cc:359] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.712502 11568 raft_consensus.cc:385] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.712814 11568 raft_consensus.cc:740] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.713644 11568 consensus_queue.cc:260] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.708082 11571 raft_consensus.cc:385] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.714730 11571 raft_consensus.cc:740] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.715401 11521 raft_consensus.cc:2468] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:32.715678 11566 tablet_bootstrap.cc:654] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.717599 11568 ts_tablet_manager.cc:1434] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:26:32.715690 11571 consensus_queue.cc:260] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.719660 11567 ts_tablet_manager.cc:1434] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.054s	user 0.010s	sys 0.000s
I20260709 12:26:32.728793 11410 leader_election.cc:304] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:32.730051 11567 tablet_bootstrap.cc:492] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:32.730327 11573 raft_consensus.cc:2804] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:32.737082 11573 raft_consensus.cc:697] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:32.738729 11567 tablet_bootstrap.cc:654] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.738306 11573 consensus_queue.cc:237] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:32.739403 11624 raft_consensus.cc:493] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.739944 11624 raft_consensus.cc:515] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:32.742123 11624 leader_election.cc:290] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:32.743279 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f32b983bac2949418ec7a6bf8a0ecdea" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:32.744400 11372 raft_consensus.cc:2468] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:32.743481 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f32b983bac2949418ec7a6bf8a0ecdea" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:32.737108 11568 tablet_bootstrap.cc:492] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:32.746965 11411 leader_election.cc:304] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:32.728552 11571 ts_tablet_manager.cc:1434] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.027s	user 0.005s	sys 0.002s
W20260709 12:26:32.748266 11410 leader_election.cc:343] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:32.749342 11624 raft_consensus.cc:2804] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:32.749902 11624 raft_consensus.cc:493] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.750388 11624 raft_consensus.cc:3060] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.753120 11568 tablet_bootstrap.cc:654] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.756932 11571 tablet_bootstrap.cc:492] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:32.764757 11571 tablet_bootstrap.cc:654] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.762216 11624 raft_consensus.cc:515] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:32.782356 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f32b983bac2949418ec7a6bf8a0ecdea" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:32.783425 11372 raft_consensus.cc:3060] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.789705 11373 raft_consensus.cc:3060] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.795141 11638 raft_consensus.cc:493] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.795892 11638 raft_consensus.cc:515] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.801221 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8236893668af4eaa93bc53b6715f6cf0" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:32.803107 11638 leader_election.cc:290] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:32.804867 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8236893668af4eaa93bc53b6715f6cf0" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:32.805902 11371 raft_consensus.cc:2393] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 663d88d9634e4720a695c7cd69eaa25e in current term 1: Already voted for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in this term.
I20260709 12:26:32.808801 11262 leader_election.cc:304] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:32.810801 11638 raft_consensus.cc:3060] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.811690 11609 raft_consensus.cc:493] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.812518 11609 raft_consensus.cc:515] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.816180 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57eb43303724eae812867b69fa94d9d" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:32.816942 11521 raft_consensus.cc:2468] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:32.823604 11336 leader_election.cc:304] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:32.824844 11635 raft_consensus.cc:2804] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:32.825417 11635 raft_consensus.cc:493] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.825843 11635 raft_consensus.cc:3060] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.837987 11566 tablet_bootstrap.cc:492] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:32.838675 11566 ts_tablet_manager.cc:1403] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.130s	user 0.062s	sys 0.050s
I20260709 12:26:32.840408 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57eb43303724eae812867b69fa94d9d" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:32.841199 11294 raft_consensus.cc:2468] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:32.842072 11566 raft_consensus.cc:359] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.842831 11566 raft_consensus.cc:385] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.843353 11566 raft_consensus.cc:740] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.843293 11193 catalog_manager.cc:5697] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:32.844420 11566 consensus_queue.cc:260] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.842787 11609 leader_election.cc:290] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:32.777771 11567 tablet_bootstrap.cc:492] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:32.843319 11638 raft_consensus.cc:2749] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:32.847450 11567 ts_tablet_manager.cc:1403] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.118s	user 0.014s	sys 0.000s
I20260709 12:26:32.854566 11566 ts_tablet_manager.cc:1434] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.002s
I20260709 12:26:32.855937 11566 tablet_bootstrap.cc:492] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:32.858892 11571 tablet_bootstrap.cc:492] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:32.859513 11571 ts_tablet_manager.cc:1403] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.103s	user 0.023s	sys 0.017s
I20260709 12:26:32.864219 11624 leader_election.cc:290] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:32.870862 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f32b983bac2949418ec7a6bf8a0ecdea" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:32.871322 11567 raft_consensus.cc:359] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:32.873217 11567 raft_consensus.cc:385] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.873618 11567 raft_consensus.cc:740] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.873376 11635 raft_consensus.cc:515] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.874536 11567 consensus_queue.cc:260] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:32.876123 11521 raft_consensus.cc:3060] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.877126 11567 ts_tablet_manager.cc:1434] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.028s	user 0.007s	sys 0.000s
I20260709 12:26:32.877518 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57eb43303724eae812867b69fa94d9d" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:32.878199 11294 raft_consensus.cc:3060] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.878321 11567 tablet_bootstrap.cc:492] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:32.885816 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e57eb43303724eae812867b69fa94d9d" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:32.887125 11520 raft_consensus.cc:3060] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:32.888396 11244 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:32.890836 11566 tablet_bootstrap.cc:654] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.891706 11573 consensus_queue.cc:1048] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:32.885051 11635 leader_election.cc:290] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:32.893608 11571 raft_consensus.cc:359] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.894708 11571 raft_consensus.cc:385] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.895362 11571 raft_consensus.cc:740] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.895457 11521 raft_consensus.cc:2468] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:32.898252 11520 raft_consensus.cc:2468] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:32.899649 11336 leader_election.cc:304] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:32.901109 11579 raft_consensus.cc:2804] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:32.901881 11579 raft_consensus.cc:697] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:32.903419 11519 raft_consensus.cc:3060] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.903147 11579 consensus_queue.cc:237] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.905018 11294 raft_consensus.cc:2468] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:32.914222 11410 leader_election.cc:304] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:32.913661 11571 consensus_queue.cc:260] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.915418 11573 raft_consensus.cc:2804] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:32.916575 11573 raft_consensus.cc:697] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:32.917785 11567 tablet_bootstrap.cc:654] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.917805 11573 consensus_queue.cc:237] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:32.912464 11624 raft_consensus.cc:493] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.920874 11624 raft_consensus.cc:515] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.924424 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57d6a333ba744c8389730bd74362cb43" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:32.925130 11294 raft_consensus.cc:2468] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:32.926467 11370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57d6a333ba744c8389730bd74362cb43" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:32.928288 11370 raft_consensus.cc:2468] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:32.928819 11412 leader_election.cc:304] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:32.929841 11632 raft_consensus.cc:2804] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:32.930392 11632 raft_consensus.cc:493] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:32.930856 11632 raft_consensus.cc:3060] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.930742 11624 leader_election.cc:290] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:32.916877 11568 tablet_bootstrap.cc:492] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:32.933037 11568 ts_tablet_manager.cc:1403] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.196s	user 0.031s	sys 0.027s
I20260709 12:26:32.954257 11632 raft_consensus.cc:515] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.955083 11571 ts_tablet_manager.cc:1434] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.095s	user 0.009s	sys 0.000s
I20260709 12:26:32.969376 11571 tablet_bootstrap.cc:492] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:32.969974 11632 leader_election.cc:290] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:32.971665 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57d6a333ba744c8389730bd74362cb43" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:32.972453 11371 raft_consensus.cc:3060] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.973744 11567 tablet_bootstrap.cc:492] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:32.974306 11567 ts_tablet_manager.cc:1403] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.096s	user 0.018s	sys 0.004s
I20260709 12:26:32.974144 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57d6a333ba744c8389730bd74362cb43" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:32.973762 11195 catalog_manager.cc:5697] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:32.977310 11567 raft_consensus.cc:359] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:32.959967 11568 raft_consensus.cc:359] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:32.978080 11567 raft_consensus.cc:385] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.978296 11568 raft_consensus.cc:385] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:32.978585 11567 raft_consensus.cc:740] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.954218 11372 raft_consensus.cc:2468] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:32.978518 11193 catalog_manager.cc:5697] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:32.978801 11294 raft_consensus.cc:3060] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:32.981990 11371 raft_consensus.cc:2468] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:32.982846 11571 tablet_bootstrap.cc:654] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:32.983585 11411 leader_election.cc:304] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:32.984930 11624 raft_consensus.cc:2804] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:32.985406 11624 raft_consensus.cc:697] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:32.986505 11624 consensus_queue.cc:237] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:32.990926 11590 consensus_queue.cc:1048] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:32.992826 11568 raft_consensus.cc:740] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:32.993925 11568 consensus_queue.cc:260] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:32.996515 11568 ts_tablet_manager.cc:1434] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.063s	user 0.008s	sys 0.000s
I20260709 12:26:32.997565 11568 tablet_bootstrap.cc:492] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:32.999150 11609 raft_consensus.cc:493] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:32.962747 11566 tablet_bootstrap.cc:492] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:33.000622 11566 ts_tablet_manager.cc:1403] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.145s	user 0.026s	sys 0.039s
I20260709 12:26:32.999696 11609 raft_consensus.cc:515] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.002586 11590 raft_consensus.cc:493] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.017469 11590 raft_consensus.cc:515] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.019887 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:33.020183 11609 leader_election.cc:290] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:33.019954 11296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:33.021564 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfa4ec6799b5454a9d750ba5a79fafa3" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:33.005283 11638 raft_consensus.cc:493] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.024070 11638 raft_consensus.cc:515] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
W20260709 12:26:33.024456 11410 leader_election.cc:343] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.014016 11566 raft_consensus.cc:359] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.026031 11566 raft_consensus.cc:385] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.026388 11566 raft_consensus.cc:740] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.032838 11568 tablet_bootstrap.cc:654] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.033102 11566 consensus_queue.cc:260] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.028818 11590 leader_election.cc:290] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:33.036644 11632 consensus_queue.cc:1048] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260709 12:26:33.042593 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:33.042891 11660 raft_consensus.cc:493] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.028982 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfa4ec6799b5454a9d750ba5a79fafa3" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:33.029261 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:33.029775 11638 leader_election.cc:290] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:33.048379 11635 raft_consensus.cc:493] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.043651 11660 raft_consensus.cc:515] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.048980 11635 raft_consensus.cc:515] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.051010 11295 raft_consensus.cc:2468] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:33.051323 11635 leader_election.cc:290] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:33.054219 11660 leader_election.cc:290] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:33.055613 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e313f6102e9941248ba9069990945017" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:33.056332 11412 leader_election.cc:304] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:33.057641 11632 raft_consensus.cc:2804] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:26:33.057837 11336 leader_election.cc:343] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.043493 11371 raft_consensus.cc:2468] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:33.058036 11632 raft_consensus.cc:493] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.060184 11262 leader_election.cc:304] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:33.060734 11632 raft_consensus.cc:3060] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.068116 11567 consensus_queue.cc:260] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.070870 11567 ts_tablet_manager.cc:1434] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.096s	user 0.010s	sys 0.000s
I20260709 12:26:33.038038 11296 raft_consensus.cc:2468] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:33.080142 11573 consensus_queue.cc:1048] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:33.080615 11338 leader_election.cc:304] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:33.081914 11635 raft_consensus.cc:2804] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:33.082417 11635 raft_consensus.cc:493] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.082896 11635 raft_consensus.cc:3060] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.075641 11519 raft_consensus.cc:2468] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:33.094584 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e313f6102e9941248ba9069990945017" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
W20260709 12:26:33.096390 11336 leader_election.cc:343] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.096880 11336 leader_election.cc:304] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433
I20260709 12:26:33.098096 11579 raft_consensus.cc:2749] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:33.103688 11566 ts_tablet_manager.cc:1434] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.103s	user 0.009s	sys 0.001s
I20260709 12:26:33.097946 11635 raft_consensus.cc:515] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.077414 11660 raft_consensus.cc:2804] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:33.119181 11567 tablet_bootstrap.cc:492] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:33.121706 11566 tablet_bootstrap.cc:492] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:33.124588 11635 leader_election.cc:290] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:33.136040 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfa4ec6799b5454a9d750ba5a79fafa3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:33.125284 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:33.139573 11519 raft_consensus.cc:3060] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.147161 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfa4ec6799b5454a9d750ba5a79fafa3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:33.152280 11571 tablet_bootstrap.cc:492] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:33.153285 11571 ts_tablet_manager.cc:1403] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.184s	user 0.037s	sys 0.085s
I20260709 12:26:33.144799 11632 raft_consensus.cc:515] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.157212 11296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
W20260709 12:26:33.159883 11261 leader_election.cc:343] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.160115 11296 raft_consensus.cc:3060] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.161793 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfa4ec6799b5454a9d750ba5a79fafa3" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
W20260709 12:26:33.163386 11410 leader_election.cc:343] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.167541 11632 leader_election.cc:290] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:33.156482 11590 consensus_queue.cc:1048] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:26:33.171841 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfa4ec6799b5454a9d750ba5a79fafa3" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:33.172734 11297 raft_consensus.cc:3060] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.174018 11446 raft_consensus.cc:2393] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 663d88d9634e4720a695c7cd69eaa25e in current term 1: Already voted for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in this term.
I20260709 12:26:33.177769 11571 raft_consensus.cc:359] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.179433 11571 raft_consensus.cc:385] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.179764 11571 raft_consensus.cc:740] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.180547 11571 consensus_queue.cc:260] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.182186 11294 raft_consensus.cc:2468] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:33.183069 11571 ts_tablet_manager.cc:1434] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.029s	user 0.005s	sys 0.003s
I20260709 12:26:33.184198 11571 tablet_bootstrap.cc:492] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:33.184665 11635 raft_consensus.cc:493] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.185194 11635 raft_consensus.cc:515] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.188252 11635 leader_election.cc:290] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:33.188488 11293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9836ae3a7d04d53ab2d78656c19c8f2" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:33.190594 11568 tablet_bootstrap.cc:492] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:33.191166 11568 ts_tablet_manager.cc:1403] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.194s	user 0.027s	sys 0.012s
I20260709 12:26:33.194252 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9836ae3a7d04d53ab2d78656c19c8f2" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:33.194559 11568 raft_consensus.cc:359] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.195326 11568 raft_consensus.cc:385] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.195657 11568 raft_consensus.cc:740] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.197247 11296 raft_consensus.cc:2468] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
W20260709 12:26:33.199708 11338 leader_election.cc:343] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.200592 11567 tablet_bootstrap.cc:654] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.187922 11519 raft_consensus.cc:2468] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:33.201033 11660 raft_consensus.cc:493] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.200456 11338 leader_election.cc:304] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:33.202843 11566 tablet_bootstrap.cc:654] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.203409 11660 raft_consensus.cc:3060] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:33.204775 11261 leader_election.cc:304] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433
I20260709 12:26:33.203342 11568 consensus_queue.cc:260] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.208943 11447 raft_consensus.cc:2468] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:33.210803 11571 tablet_bootstrap.cc:654] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.212396 11635 raft_consensus.cc:2804] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:33.214597 11635 raft_consensus.cc:697] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:33.217648 11568 ts_tablet_manager.cc:1434] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.001s
I20260709 12:26:33.218852 11568 tablet_bootstrap.cc:492] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:33.216418 11635 consensus_queue.cc:237] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.232911 11632 consensus_queue.cc:1048] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:33.253448 11660 raft_consensus.cc:515] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.263092 11567 tablet_bootstrap.cc:492] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:33.263867 11567 ts_tablet_manager.cc:1403] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.156s	user 0.019s	sys 0.003s
I20260709 12:26:33.264292 11297 raft_consensus.cc:2468] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:33.268744 11703 raft_consensus.cc:2749] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:33.270771 11412 leader_election.cc:304] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:33.272505 11707 raft_consensus.cc:2804] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:33.273123 11707 raft_consensus.cc:697] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:33.274086 11707 consensus_queue.cc:237] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.275739 11571 tablet_bootstrap.cc:492] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:33.276366 11571 ts_tablet_manager.cc:1403] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.092s	user 0.015s	sys 0.028s
I20260709 12:26:33.271847 11195 catalog_manager.cc:5697] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:33.288832 11336 leader_election.cc:304] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:33.302105 11635 raft_consensus.cc:2804] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:33.302834 11635 raft_consensus.cc:493] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.303373 11635 raft_consensus.cc:3060] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.313316 11568 tablet_bootstrap.cc:654] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.320638 11706 consensus_queue.cc:1048] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:33.322412 11571 raft_consensus.cc:359] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.323287 11571 raft_consensus.cc:385] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.323632 11571 raft_consensus.cc:740] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.324676 11571 consensus_queue.cc:260] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.327790 11571 ts_tablet_manager.cc:1434] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.051s	user 0.009s	sys 0.000s
I20260709 12:26:33.328126 11635 raft_consensus.cc:515] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.338217 11567 raft_consensus.cc:359] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.339144 11567 raft_consensus.cc:385] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.339435 11567 raft_consensus.cc:740] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.328962 11571 tablet_bootstrap.cc:492] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
W20260709 12:26:33.341945 11486 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 76041.5us
I20260709 12:26:33.346861 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9836ae3a7d04d53ab2d78656c19c8f2" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:33.351554 11567 consensus_queue.cc:260] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.360350 11635 leader_election.cc:290] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:33.366460 11660 leader_election.cc:290] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:33.381261 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:33.382256 11518 raft_consensus.cc:3060] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:26:33.373281 11338 leader_election.cc:343] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.395066 11715 raft_consensus.cc:493] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.375638 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:33.396622 11567 ts_tablet_manager.cc:1434] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.132s	user 0.007s	sys 0.003s
I20260709 12:26:33.395654 11571 tablet_bootstrap.cc:654] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.408730 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9836ae3a7d04d53ab2d78656c19c8f2" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:33.411393 11567 tablet_bootstrap.cc:492] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:33.414907 11713 raft_consensus.cc:493] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.406764 11715 raft_consensus.cc:515] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.415534 11713 raft_consensus.cc:515] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.417689 11715 leader_election.cc:290] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:33.418882 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfa4ec6799b5454a9d750ba5a79fafa3" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:33.419785 11295 raft_consensus.cc:2393] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 1: Already voted for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in this term.
I20260709 12:26:33.419589 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bfa4ec6799b5454a9d750ba5a79fafa3" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:33.411252 11447 raft_consensus.cc:3060] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.419623 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0c4260f7779463dab087f51bf433ab8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:33.423017 11484 leader_election.cc:304] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:33.423473 11372 raft_consensus.cc:2468] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:33.424446 11568 tablet_bootstrap.cc:492] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:33.424990 11568 ts_tablet_manager.cc:1403] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.206s	user 0.013s	sys 0.011s
I20260709 12:26:33.429883 11713 leader_election.cc:290] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:33.429682 11635 consensus_queue.cc:1048] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:33.433369 11411 leader_election.cc:304] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:33.435101 11707 raft_consensus.cc:2804] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:33.439735 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0c4260f7779463dab087f51bf433ab8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
W20260709 12:26:33.441485 11410 leader_election.cc:343] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.427127 11715 raft_consensus.cc:3060] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.439769 11707 raft_consensus.cc:493] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.443184 11707 raft_consensus.cc:3060] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.444479 11518 raft_consensus.cc:2468] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:33.445843 11261 leader_election.cc:304] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e
I20260709 12:26:33.447189 11660 raft_consensus.cc:2804] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:33.447718 11660 raft_consensus.cc:697] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:33.448629 11660 consensus_queue.cc:237] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.449766 11567 tablet_bootstrap.cc:654] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.448514 11193 catalog_manager.cc:5697] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:33.451407 11635 raft_consensus.cc:493] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.452010 11635 raft_consensus.cc:515] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.455490 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d4384a4da6846e39db4bcd0330cb822" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:33.456346 11635 leader_election.cc:290] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
W20260709 12:26:33.458999 11338 leader_election.cc:343] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.459942 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d4384a4da6846e39db4bcd0330cb822" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
W20260709 12:26:33.461910 11336 leader_election.cc:343] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.462533 11336 leader_election.cc:304] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:33.463380 11609 raft_consensus.cc:2749] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:33.464875 11707 raft_consensus.cc:515] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.473621 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0c4260f7779463dab087f51bf433ab8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:33.474351 11372 raft_consensus.cc:3060] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.473637 11566 tablet_bootstrap.cc:492] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:33.476238 11566 ts_tablet_manager.cc:1403] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.359s	user 0.015s	sys 0.014s
I20260709 12:26:33.461963 11715 raft_consensus.cc:2749] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:33.482702 11372 raft_consensus.cc:2468] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:33.483126 11568 raft_consensus.cc:359] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.443198 11447 raft_consensus.cc:2468] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:33.484062 11568 raft_consensus.cc:385] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.484304 11411 leader_election.cc:304] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:33.486800 11567 tablet_bootstrap.cc:492] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:33.487468 11567 ts_tablet_manager.cc:1403] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.082s	user 0.013s	sys 0.001s
I20260709 12:26:33.489598 11590 raft_consensus.cc:2804] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:33.490213 11590 raft_consensus.cc:697] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:33.492244 11590 consensus_queue.cc:237] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.494611 11571 tablet_bootstrap.cc:492] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:33.495208 11571 ts_tablet_manager.cc:1403] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.166s	user 0.013s	sys 0.007s
I20260709 12:26:33.484503 11568 raft_consensus.cc:740] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.497613 11566 raft_consensus.cc:359] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.499444 11336 leader_election.cc:304] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:33.500658 11635 raft_consensus.cc:2804] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:33.498450 11566 raft_consensus.cc:385] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.501304 11571 raft_consensus.cc:359] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.501678 11566 raft_consensus.cc:740] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.503515 11566 consensus_queue.cc:260] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.501183 11635 raft_consensus.cc:697] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:33.505692 11635 consensus_queue.cc:237] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.499053 11568 consensus_queue.cc:260] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.502139 11571 raft_consensus.cc:385] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.511668 11571 raft_consensus.cc:740] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.516716 11568 ts_tablet_manager.cc:1434] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.091s	user 0.010s	sys 0.000s
I20260709 12:26:33.516717 11567 raft_consensus.cc:359] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.517724 11567 raft_consensus.cc:385] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.518025 11567 raft_consensus.cc:740] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.518150 11568 tablet_bootstrap.cc:492] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:33.518839 11567 consensus_queue.cc:260] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.518939 11193 catalog_manager.cc:5697] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:33.520987 11567 ts_tablet_manager.cc:1434] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.033s	user 0.006s	sys 0.000s
I20260709 12:26:33.513652 11196 catalog_manager.cc:5697] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 1 to 2, leader changed from 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 2 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:33.531273 11568 tablet_bootstrap.cc:654] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.535846 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0c4260f7779463dab087f51bf433ab8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:33.538338 11566 ts_tablet_manager.cc:1434] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.062s	user 0.009s	sys 0.000s
I20260709 12:26:33.540117 11571 consensus_queue.cc:260] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.532232 11590 consensus_queue.cc:1048] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
W20260709 12:26:33.545555 11410 leader_election.cc:343] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.545888 11566 tablet_bootstrap.cc:492] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:33.548580 11707 leader_election.cc:290] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:33.553542 11571 ts_tablet_manager.cc:1434] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.058s	user 0.004s	sys 0.003s
I20260709 12:26:33.556196 11713 consensus_queue.cc:1048] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:33.559727 11194 catalog_manager.cc:5697] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:33.561093 11713 consensus_queue.cc:1048] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:33.564555 11566 tablet_bootstrap.cc:654] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.568553 11567 tablet_bootstrap.cc:492] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:33.578210 11707 consensus_queue.cc:1048] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20260709 12:26:33.583922 11568 tablet_bootstrap.cc:492] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:33.584496 11568 ts_tablet_manager.cc:1403] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.067s	user 0.006s	sys 0.015s
I20260709 12:26:33.588289 11568 raft_consensus.cc:359] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.589778 11571 tablet_bootstrap.cc:492] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:33.590664 11568 raft_consensus.cc:385] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.593497 11568 raft_consensus.cc:740] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.594473 11568 consensus_queue.cc:260] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.589201 11567 tablet_bootstrap.cc:654] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.599462 11568 ts_tablet_manager.cc:1434] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.015s	user 0.010s	sys 0.000s
I20260709 12:26:33.601182 11568 tablet_bootstrap.cc:492] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:33.597981 11193 catalog_manager.cc:5697] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:33.626034 11571 tablet_bootstrap.cc:654] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.631136 11715 raft_consensus.cc:493] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.632781 11715 raft_consensus.cc:515] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.627988 11566 tablet_bootstrap.cc:492] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:33.637602 11566 ts_tablet_manager.cc:1403] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.092s	user 0.017s	sys 0.004s
I20260709 12:26:33.640695 11566 raft_consensus.cc:359] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.635584 11715 leader_election.cc:290] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:33.636778 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d4384a4da6846e39db4bcd0330cb822" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:33.642833 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d4384a4da6846e39db4bcd0330cb822" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:33.635353 11568 tablet_bootstrap.cc:654] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.650641 11371 raft_consensus.cc:2468] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:33.652526 11485 leader_election.cc:304] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:33.646402 11567 tablet_bootstrap.cc:492] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:33.654027 11715 raft_consensus.cc:2804] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:33.654202 11567 ts_tablet_manager.cc:1403] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.113s	user 0.012s	sys 0.004s
I20260709 12:26:33.654785 11715 raft_consensus.cc:493] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.655347 11715 raft_consensus.cc:3060] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.657298 11567 raft_consensus.cc:359] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.658013 11567 raft_consensus.cc:385] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.658285 11567 raft_consensus.cc:740] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.663453 11566 raft_consensus.cc:385] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.663949 11566 raft_consensus.cc:740] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.664918 11566 consensus_queue.cc:260] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.665921 11295 raft_consensus.cc:2468] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:33.666218 11715 raft_consensus.cc:515] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.669664 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d4384a4da6846e39db4bcd0330cb822" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:33.671550 11295 raft_consensus.cc:3060] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.666183 11567 consensus_queue.cc:260] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.673949 11715 leader_election.cc:290] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:33.676846 11567 ts_tablet_manager.cc:1434] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.000s
I20260709 12:26:33.679885 11567 tablet_bootstrap.cc:492] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:33.685621 11715 raft_consensus.cc:493] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.686295 11715 raft_consensus.cc:515] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.689595 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e313f6102e9941248ba9069990945017" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:33.689853 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e313f6102e9941248ba9069990945017" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:33.690524 11447 raft_consensus.cc:2468] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:33.692116 11567 tablet_bootstrap.cc:654] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.692036 11484 leader_election.cc:304] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:33.692973 11715 leader_election.cc:290] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:33.693015 11720 raft_consensus.cc:2804] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:33.690524 11371 raft_consensus.cc:2468] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:33.693617 11720 raft_consensus.cc:493] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.696126 11720 raft_consensus.cc:3060] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.685966 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6d4384a4da6846e39db4bcd0330cb822" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:33.697834 11372 raft_consensus.cc:3060] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.701061 11295 raft_consensus.cc:2468] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:33.702418 11486 leader_election.cc:304] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:33.702832 11566 ts_tablet_manager.cc:1434] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.065s	user 0.006s	sys 0.001s
I20260709 12:26:33.703657 11715 raft_consensus.cc:2804] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:33.704305 11715 raft_consensus.cc:697] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:33.705435 11715 consensus_queue.cc:237] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.706610 11566 tablet_bootstrap.cc:492] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:33.710335 11720 raft_consensus.cc:515] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.717095 11720 leader_election.cc:290] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:33.712471 11568 tablet_bootstrap.cc:492] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:33.718178 11568 ts_tablet_manager.cc:1403] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.117s	user 0.016s	sys 0.009s
I20260709 12:26:33.718532 11567 tablet_bootstrap.cc:492] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:33.719089 11567 ts_tablet_manager.cc:1403] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.039s	user 0.012s	sys 0.000s
I20260709 12:26:33.718935 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e313f6102e9941248ba9069990945017" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:33.719569 11371 raft_consensus.cc:3060] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.721194 11568 raft_consensus.cc:359] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.721925 11568 raft_consensus.cc:385] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.722268 11568 raft_consensus.cc:740] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.721993 11567 raft_consensus.cc:359] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.722847 11567 raft_consensus.cc:385] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.706982 11571 tablet_bootstrap.cc:492] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:33.723160 11567 raft_consensus.cc:740] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.723706 11571 ts_tablet_manager.cc:1403] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.150s	user 0.025s	sys 0.020s
I20260709 12:26:33.724125 11567 consensus_queue.cc:260] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.725302 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e313f6102e9941248ba9069990945017" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:33.725941 11447 raft_consensus.cc:3060] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.706426 11372 raft_consensus.cc:2468] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:33.726802 11567 ts_tablet_manager.cc:1434] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:26:33.726673 11571 raft_consensus.cc:359] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.727252 11571 raft_consensus.cc:385] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.727437 11571 raft_consensus.cc:740] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.727926 11567 tablet_bootstrap.cc:492] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:33.727931 11571 consensus_queue.cc:260] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.729875 11571 ts_tablet_manager.cc:1434] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.006s	user 0.002s	sys 0.001s
I20260709 12:26:33.731161 11568 consensus_queue.cc:260] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.733409 11568 ts_tablet_manager.cc:1434] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.000s
I20260709 12:26:33.736403 11571 tablet_bootstrap.cc:492] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:33.737576 11567 tablet_bootstrap.cc:654] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.739604 11568 tablet_bootstrap.cc:492] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:33.743149 11195 catalog_manager.cc:5697] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:33.745924 11568 tablet_bootstrap.cc:654] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.754048 11447 raft_consensus.cc:2468] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:33.755228 11571 tablet_bootstrap.cc:654] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.756819 11567 tablet_bootstrap.cc:492] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:33.757350 11567 ts_tablet_manager.cc:1403] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.030s	user 0.012s	sys 0.005s
I20260709 12:26:33.755244 11566 tablet_bootstrap.cc:654] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.760339 11567 raft_consensus.cc:359] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.761101 11567 raft_consensus.cc:385] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.761372 11567 raft_consensus.cc:740] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.762131 11567 consensus_queue.cc:260] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.764835 11567 ts_tablet_manager.cc:1434] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:26:33.765975 11567 tablet_bootstrap.cc:492] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:33.767783 11371 raft_consensus.cc:2468] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:33.768492 11566 tablet_bootstrap.cc:492] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:33.769008 11566 ts_tablet_manager.cc:1403] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.063s	user 0.012s	sys 0.000s
I20260709 12:26:33.769054 11485 leader_election.cc:304] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:33.770052 11715 raft_consensus.cc:2804] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:33.770578 11715 raft_consensus.cc:697] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:33.771463 11715 consensus_queue.cc:237] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.771991 11566 raft_consensus.cc:359] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.772730 11566 raft_consensus.cc:385] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.772994 11566 raft_consensus.cc:740] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.776381 11571 tablet_bootstrap.cc:492] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:33.776938 11571 ts_tablet_manager.cc:1403] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.041s	user 0.010s	sys 0.002s
I20260709 12:26:33.777683 11567 tablet_bootstrap.cc:654] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.779768 11571 raft_consensus.cc:359] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.780520 11571 raft_consensus.cc:385] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.780792 11571 raft_consensus.cc:740] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.781535 11571 consensus_queue.cc:260] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.784041 11566 consensus_queue.cc:260] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.786353 11566 ts_tablet_manager.cc:1434] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.001s
I20260709 12:26:33.788786 11568 tablet_bootstrap.cc:492] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:33.789372 11568 ts_tablet_manager.cc:1403] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.050s	user 0.008s	sys 0.005s
I20260709 12:26:33.792533 11568 raft_consensus.cc:359] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.793262 11568 raft_consensus.cc:385] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.793550 11568 raft_consensus.cc:740] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.794252 11568 consensus_queue.cc:260] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.796047 11568 ts_tablet_manager.cc:1434] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.001s
I20260709 12:26:33.798846 11297 raft_consensus.cc:1240] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Rejecting Update request from peer 05b93ba4435f40fb97b54343c4fdd71e for earlier term 1. Current term is 2. Ops: []
I20260709 12:26:33.800792 11635 consensus_queue.cc:1059] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:33.801568 11635 consensus_queue.cc:1243] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Peer 663d88d9634e4720a695c7cd69eaa25e log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260709 12:26:33.802093 11635 raft_consensus.cc:3055] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:33.802420 11635 raft_consensus.cc:740] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:33.803058 11566 tablet_bootstrap.cc:492] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:33.803368 11635 consensus_queue.cc:260] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.807711 11635 raft_consensus.cc:3060] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:33.810361 11568 tablet_bootstrap.cc:492] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:33.814976 11720 raft_consensus.cc:493] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.815606 11720 raft_consensus.cc:515] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.821663 11579 raft_consensus.cc:493] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.825377 11713 consensus_queue.cc:1048] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:33.830171 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0081c481615e480ea8dfafe6a806ee7f" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:33.830935 11447 raft_consensus.cc:2468] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:33.832386 11484 leader_election.cc:304] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:33.833395 11724 raft_consensus.cc:2804] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:33.833882 11724 raft_consensus.cc:493] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:33.834280 11724 raft_consensus.cc:3060] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.841235 11568 tablet_bootstrap.cc:654] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.842777 11579 raft_consensus.cc:515] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.846093 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee281c78e97848f59386b1d97f96bcfe" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
W20260709 12:26:33.847822 11336 leader_election.cc:343] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.848390 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0081c481615e480ea8dfafe6a806ee7f" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:33.846218 11724 raft_consensus.cc:515] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.849197 11295 raft_consensus.cc:2468] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:33.848308 11720 leader_election.cc:290] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:33.850025 11579 leader_election.cc:290] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:33.851073 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee281c78e97848f59386b1d97f96bcfe" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:33.851295 11660 raft_consensus.cc:493] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.851929 11660 raft_consensus.cc:515] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.853377 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0081c481615e480ea8dfafe6a806ee7f" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:33.853688 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0081c481615e480ea8dfafe6a806ee7f" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:33.854190 11295 raft_consensus.cc:3060] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:33.853329 11336 leader_election.cc:343] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.855293 11336 leader_election.cc:304] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433
I20260709 12:26:33.855465 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9836ae3a7d04d53ab2d78656c19c8f2" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:33.854346 11447 raft_consensus.cc:3060] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.854208 11660 leader_election.cc:290] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:33.860617 11590 raft_consensus.cc:493] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.861227 11590 raft_consensus.cc:515] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.863198 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e9836ae3a7d04d53ab2d78656c19c8f2" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:33.864336 11446 raft_consensus.cc:2393] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 663d88d9634e4720a695c7cd69eaa25e in current term 1: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:33.872936 11261 leader_election.cc:304] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:33.853307 11724 leader_election.cc:290] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:33.874851 11660 raft_consensus.cc:3060] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.882856 11624 consensus_queue.cc:1048] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:33.885383 11590 leader_election.cc:290] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:33.887661 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a5d23b0089f4d37a0c63d127537ec93" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
W20260709 12:26:33.889950 11411 leader_election.cc:343] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.892762 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a5d23b0089f4d37a0c63d127537ec93" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
W20260709 12:26:33.896454 11410 leader_election.cc:343] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.896997 11410 leader_election.cc:304] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433
I20260709 12:26:33.897845 11590 raft_consensus.cc:2749] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:33.898044 11566 tablet_bootstrap.cc:654] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.901185 11295 raft_consensus.cc:2468] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:33.903163 11486 leader_election.cc:304] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:33.905409 11724 raft_consensus.cc:2804] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:33.906123 11724 raft_consensus.cc:697] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:33.907289 11724 consensus_queue.cc:237] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.912452 11571 ts_tablet_manager.cc:1434] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.135s	user 0.007s	sys 0.002s
I20260709 12:26:33.903844 11660 raft_consensus.cc:2749] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:33.924649 11571 tablet_bootstrap.cc:492] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:33.927485 11518 raft_consensus.cc:3060] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:33.917491 11447 raft_consensus.cc:2468] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
W20260709 12:26:33.938439 11336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 85150.7us
I20260709 12:26:33.937893 11579 raft_consensus.cc:2749] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:33.943153 11567 tablet_bootstrap.cc:492] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:33.945111 11567 ts_tablet_manager.cc:1403] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.179s	user 0.057s	sys 0.052s
I20260709 12:26:33.947453 11566 tablet_bootstrap.cc:492] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:33.947999 11566 ts_tablet_manager.cc:1403] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.145s	user 0.014s	sys 0.009s
I20260709 12:26:33.939975 11635 raft_consensus.cc:493] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.949175 11635 raft_consensus.cc:515] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
W20260709 12:26:33.949857 11244 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:33.948696 11567 raft_consensus.cc:359] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.950865 11567 raft_consensus.cc:385] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.951159 11567 raft_consensus.cc:740] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.951900 11567 consensus_queue.cc:260] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:33.952680 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3701a37fd75248f6b8b0ca0930791a9d" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:33.953645 11635 leader_election.cc:290] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:33.954342 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3701a37fd75248f6b8b0ca0930791a9d" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:33.954825 11590 raft_consensus.cc:493] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.955561 11590 raft_consensus.cc:515] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.957633 11720 raft_consensus.cc:493] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:33.958171 11720 raft_consensus.cc:515] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:33.964640 11571 tablet_bootstrap.cc:654] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.973573 11720 leader_election.cc:290] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:33.974671 11568 tablet_bootstrap.cc:492] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:33.974545 11567 ts_tablet_manager.cc:1434] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.029s	user 0.012s	sys 0.011s
I20260709 12:26:33.975309 11568 ts_tablet_manager.cc:1403] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.165s	user 0.028s	sys 0.027s
I20260709 12:26:33.975718 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0c4260f7779463dab087f51bf433ab8" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:33.976109 11567 tablet_bootstrap.cc:492] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:33.976500 11371 raft_consensus.cc:2468] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:33.978014 11485 leader_election.cc:304] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:33.979954 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2925eddaf8944bf5bf72cd650ed1e287" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:33.980372 11521 raft_consensus.cc:1275] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:26:33.982404 11566 raft_consensus.cc:359] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.983728 11566 raft_consensus.cc:385] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.984006 11566 raft_consensus.cc:740] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.961269 11590 leader_election.cc:290] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
W20260709 12:26:33.963860 11336 leader_election.cc:343] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.978279 11568 raft_consensus.cc:359] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.986357 11568 raft_consensus.cc:385] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:33.986675 11568 raft_consensus.cc:740] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:33.986897 11566 consensus_queue.cc:260] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:33.988484 11568 consensus_queue.cc:260] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:33.990996 11568 ts_tablet_manager.cc:1434] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:26:33.992075 11568 tablet_bootstrap.cc:492] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
W20260709 12:26:33.994848 11338 leader_election.cc:343] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:33.995623 11410 leader_election.cc:343] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:33.997267 11338 leader_election.cc:304] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:33.997752 11590 consensus_queue.cc:1048] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:33.998399 11568 tablet_bootstrap.cc:654] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:33.998831 11635 raft_consensus.cc:2749] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:34.001438 11566 ts_tablet_manager.cc:1434] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.053s	user 0.008s	sys 0.001s
I20260709 12:26:34.002692 11566 tablet_bootstrap.cc:492] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:34.003328 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2925eddaf8944bf5bf72cd650ed1e287" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
W20260709 12:26:34.005370 11412 leader_election.cc:343] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.005851 11412 leader_election.cc:304] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:34.011525 11632 raft_consensus.cc:2749] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:34.017511 11196 catalog_manager.cc:5697] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:34.020339 11371 raft_consensus.cc:1275] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 12:26:34.020812 11371 pending_rounds.cc:85] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e: Aborting all ops after (but not including) 0
I20260709 12:26:34.021050 11371 pending_rounds.cc:107] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:26:34.021332 11371 raft_consensus.cc:2889] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:34.012648 11724 raft_consensus.cc:2804] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:34.023046 11724 raft_consensus.cc:493] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.011116 11660 consensus_queue.cc:1048] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260709 12:26:34.023581 11724 raft_consensus.cc:3060] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:34.029507 11567 tablet_bootstrap.cc:654] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.030735 11703 consensus_queue.cc:1048] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:34.043268 11724 raft_consensus.cc:515] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.044674 11518 raft_consensus.cc:1240] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Rejecting Update request from peer 5a1ce6e048ab47d69c453b5218cb64c1 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260709 12:26:34.048542 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0c4260f7779463dab087f51bf433ab8" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:34.049356 11724 leader_election.cc:290] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:34.052345 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0c4260f7779463dab087f51bf433ab8" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:34.053258 11371 raft_consensus.cc:3060] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:34.055173 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b0c4260f7779463dab087f51bf433ab8" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:34.057375 11590 consensus_queue.cc:1059] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:34.058259 11590 raft_consensus.cc:3055] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:34.058563 11590 raft_consensus.cc:740] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:34.059412 11590 consensus_queue.cc:260] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.060647 11590 raft_consensus.cc:3060] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:34.060231 11571 tablet_bootstrap.cc:492] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:34.071853 11571 ts_tablet_manager.cc:1403] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.147s	user 0.017s	sys 0.014s
I20260709 12:26:34.076526 11371 raft_consensus.cc:2468] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 2.
I20260709 12:26:34.090505 11194 catalog_manager.cc:5697] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.092757 11485 leader_election.cc:304] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:34.094545 11724 raft_consensus.cc:2804] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:34.095288 11724 raft_consensus.cc:697] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:34.096480 11724 consensus_queue.cc:237] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.099417 11579 raft_consensus.cc:493] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.100230 11579 raft_consensus.cc:515] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.102152 11567 tablet_bootstrap.cc:492] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:34.102751 11567 ts_tablet_manager.cc:1403] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.127s	user 0.018s	sys 0.004s
I20260709 12:26:34.080669 11609 consensus_queue.cc:1048] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:34.104809 11579 leader_election.cc:290] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:34.105890 11567 raft_consensus.cc:359] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.106714 11567 raft_consensus.cc:385] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.106595 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea9a20aab6840c1afb32c29bac2f4a4" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:34.107210 11567 raft_consensus.cc:740] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.107983 11567 consensus_queue.cc:260] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.108757 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea9a20aab6840c1afb32c29bac2f4a4" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
W20260709 12:26:34.109851 11338 leader_election.cc:343] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.110231 11567 ts_tablet_manager.cc:1434] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:26:34.111423 11567 tablet_bootstrap.cc:492] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
W20260709 12:26:34.112917 11336 leader_election.cc:343] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.113482 11336 leader_election.cc:304] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:34.114455 11635 raft_consensus.cc:2749] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:34.117909 11567 tablet_bootstrap.cc:654] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.088656 11571 raft_consensus.cc:359] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.118942 11571 raft_consensus.cc:385] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.119237 11571 raft_consensus.cc:740] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.119990 11571 consensus_queue.cc:260] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.123386 11567 tablet_bootstrap.cc:492] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:34.123898 11567 ts_tablet_manager.cc:1403] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.013s	user 0.011s	sys 0.000s
I20260709 12:26:34.126453 11756 raft_consensus.cc:493] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.127279 11571 ts_tablet_manager.cc:1434] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.055s	user 0.007s	sys 0.000s
I20260709 12:26:34.126971 11756 raft_consensus.cc:515] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.127031 11567 raft_consensus.cc:359] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.127837 11567 raft_consensus.cc:385] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.128149 11567 raft_consensus.cc:740] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.128364 11571 tablet_bootstrap.cc:492] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:34.129436 11756 leader_election.cc:290] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:34.130221 11296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:34.130733 11635 consensus_queue.cc:1048] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:34.131510 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
W20260709 12:26:34.136248 11486 leader_election.cc:343] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:34.137166 11485 leader_election.cc:343] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.137631 11485 leader_election.cc:304] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:34.138336 11756 raft_consensus.cc:2749] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:34.140816 11568 tablet_bootstrap.cc:492] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:34.141331 11568 ts_tablet_manager.cc:1403] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.150s	user 0.035s	sys 0.048s
I20260709 12:26:34.140290 11196 catalog_manager.cc:5697] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 1 to 2, leader changed from 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195) to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:34.143260 11567 consensus_queue.cc:260] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.150329 11568 raft_consensus.cc:359] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.151212 11568 raft_consensus.cc:385] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.151508 11568 raft_consensus.cc:740] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.152419 11566 tablet_bootstrap.cc:654] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.152262 11568 consensus_queue.cc:260] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.152762 11567 ts_tablet_manager.cc:1434] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.028s	user 0.002s	sys 0.004s
I20260709 12:26:34.153877 11567 tablet_bootstrap.cc:492] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:34.154428 11568 ts_tablet_manager.cc:1434] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:26:34.155730 11568 tablet_bootstrap.cc:492] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:34.163925 11571 tablet_bootstrap.cc:654] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.165814 11567 tablet_bootstrap.cc:654] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.174436 11566 tablet_bootstrap.cc:492] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:34.175160 11566 ts_tablet_manager.cc:1403] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.173s	user 0.009s	sys 0.006s
I20260709 12:26:34.180810 11566 raft_consensus.cc:359] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.181555 11566 raft_consensus.cc:385] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.181813 11566 raft_consensus.cc:740] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.182456 11566 consensus_queue.cc:260] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.184566 11566 ts_tablet_manager.cc:1434] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:26:34.190263 11756 raft_consensus.cc:493] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.190968 11756 raft_consensus.cc:515] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.194723 11566 tablet_bootstrap.cc:492] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:34.197435 11579 raft_consensus.cc:493] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.198041 11579 raft_consensus.cc:515] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.199535 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "827a79573ac94525a9ff30c744c45164" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:34.200193 11579 leader_election.cc:290] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:34.200369 11446 raft_consensus.cc:2468] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:34.201987 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578fd826b687468ca8be54244c3b4bc0" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:34.202699 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "827a79573ac94525a9ff30c744c45164" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
W20260709 12:26:34.203725 11338 leader_election.cc:343] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.204021 11484 leader_election.cc:304] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:34.204923 11724 raft_consensus.cc:2804] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:34.199676 11756 leader_election.cc:290] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:34.206298 11568 tablet_bootstrap.cc:654] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:34.208029 11486 leader_election.cc:343] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.208277 11720 consensus_queue.cc:1048] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:34.205315 11724 raft_consensus.cc:493] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.209270 11724 raft_consensus.cc:3060] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.209638 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578fd826b687468ca8be54244c3b4bc0" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:34.209895 11566 tablet_bootstrap.cc:654] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:34.217989 11336 leader_election.cc:343] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.218621 11336 leader_election.cc:304] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:34.219601 11579 raft_consensus.cc:2749] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:34.212806 11720 consensus_queue.cc:1048] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:26:34.223327 11724 raft_consensus.cc:515] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.232267 11720 raft_consensus.cc:493] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.237190 11720 raft_consensus.cc:515] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.237661 11567 tablet_bootstrap.cc:492] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:34.238205 11567 ts_tablet_manager.cc:1403] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.085s	user 0.011s	sys 0.007s
I20260709 12:26:34.238413 11724 leader_election.cc:290] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:34.239369 11720 leader_election.cc:290] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:34.241087 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "827a79573ac94525a9ff30c744c45164" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:34.241602 11567 raft_consensus.cc:359] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.242048 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2925eddaf8944bf5bf72cd650ed1e287" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:34.242952 11447 raft_consensus.cc:2468] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:34.243947 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2925eddaf8944bf5bf72cd650ed1e287" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:34.239246 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "827a79573ac94525a9ff30c744c45164" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:34.244626 11484 leader_election.cc:304] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:34.244928 11446 raft_consensus.cc:3060] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.245517 11720 raft_consensus.cc:2804] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:34.245904 11720 raft_consensus.cc:493] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.246277 11720 raft_consensus.cc:3060] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.250870 11567 raft_consensus.cc:385] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.251256 11567 raft_consensus.cc:740] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.252106 11567 consensus_queue.cc:260] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.255432 11720 raft_consensus.cc:515] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.256026 11756 consensus_queue.cc:1048] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:26:34.259032 11486 leader_election.cc:343] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:34.260968 11486 leader_election.cc:343] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.265645 11566 tablet_bootstrap.cc:492] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:34.266160 11566 ts_tablet_manager.cc:1403] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.080s	user 0.017s	sys 0.016s
I20260709 12:26:34.266867 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2925eddaf8944bf5bf72cd650ed1e287" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:34.268219 11567 ts_tablet_manager.cc:1434] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.030s	user 0.008s	sys 0.002s
I20260709 12:26:34.267534 11447 raft_consensus.cc:3060] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.281096 11567 tablet_bootstrap.cc:492] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:34.264412 11446 raft_consensus.cc:2468] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:34.292362 11571 tablet_bootstrap.cc:492] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:34.292997 11571 ts_tablet_manager.cc:1403] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.165s	user 0.012s	sys 0.006s
I20260709 12:26:34.294904 11484 leader_election.cc:304] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:34.298821 11724 raft_consensus.cc:2804] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:34.299391 11724 raft_consensus.cc:697] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:34.300230 11724 consensus_queue.cc:237] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.301807 11447 raft_consensus.cc:2468] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:34.303184 11571 raft_consensus.cc:359] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.303993 11571 raft_consensus.cc:385] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.304322 11571 raft_consensus.cc:740] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.310014 11484 leader_election.cc:304] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:34.311156 11756 raft_consensus.cc:2804] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:34.312080 11756 raft_consensus.cc:697] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:34.311160 11571 consensus_queue.cc:260] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.312958 11756 consensus_queue.cc:237] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.316089 11571 ts_tablet_manager.cc:1434] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.002s
I20260709 12:26:34.315243 11566 raft_consensus.cc:359] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.317309 11566 raft_consensus.cc:385] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.318543 11713 raft_consensus.cc:493] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.319119 11713 raft_consensus.cc:515] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.321307 11713 leader_election.cc:290] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:34.322237 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea9a20aab6840c1afb32c29bac2f4a4" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
W20260709 12:26:34.324106 11412 leader_election.cc:343] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.324569 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2925eddaf8944bf5bf72cd650ed1e287" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
W20260709 12:26:34.326047 11486 leader_election.cc:343] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.331368 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea9a20aab6840c1afb32c29bac2f4a4" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:34.332150 11372 raft_consensus.cc:2468] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:34.339349 11568 tablet_bootstrap.cc:492] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:34.340466 11568 ts_tablet_manager.cc:1403] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.185s	user 0.018s	sys 0.016s
I20260709 12:26:34.317601 11566 raft_consensus.cc:740] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.342991 11411 leader_election.cc:304] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:34.322254 11720 leader_election.cc:290] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:34.371752 11566 consensus_queue.cc:260] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.344136 11632 raft_consensus.cc:2804] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:34.375494 11571 tablet_bootstrap.cc:492] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:34.384204 11567 tablet_bootstrap.cc:654] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.378695 11632 raft_consensus.cc:493] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.390215 11566 ts_tablet_manager.cc:1434] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.124s	user 0.007s	sys 0.002s
I20260709 12:26:34.388243 11760 raft_consensus.cc:493] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.392006 11760 raft_consensus.cc:515] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.394263 11760 leader_election.cc:290] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:34.389717 11632 raft_consensus.cc:3060] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.405097 11566 tablet_bootstrap.cc:492] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:34.405239 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3701a37fd75248f6b8b0ca0930791a9d" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:34.405774 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3701a37fd75248f6b8b0ca0930791a9d" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:34.406021 11372 raft_consensus.cc:2468] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:34.407579 11262 leader_election.cc:304] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:34.408665 11660 raft_consensus.cc:2804] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:34.409126 11660 raft_consensus.cc:493] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.409510 11660 raft_consensus.cc:3060] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.403401 11724 raft_consensus.cc:493] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.413918 11760 raft_consensus.cc:493] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.414649 11760 raft_consensus.cc:515] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.417009 11760 leader_election.cc:290] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:34.417069 11192 catalog_manager.cc:5697] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.424291 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:34.424844 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:34.425153 11518 raft_consensus.cc:2468] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:34.427196 11261 leader_election.cc:304] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:34.428519 11760 raft_consensus.cc:2804] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:34.428918 11760 raft_consensus.cc:493] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:34.430300 11262 leader_election.cc:343] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.430522 11632 raft_consensus.cc:515] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.431617 11777 consensus_queue.cc:1048] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:34.432746 11632 leader_election.cc:290] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:34.435117 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea9a20aab6840c1afb32c29bac2f4a4" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:34.434970 11724 raft_consensus.cc:515] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.436632 11372 raft_consensus.cc:3060] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.438316 11724 leader_election.cc:290] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:34.441553 11568 raft_consensus.cc:359] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.446808 11568 raft_consensus.cc:385] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.447274 11568 raft_consensus.cc:740] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.429276 11760 raft_consensus.cc:3060] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.456123 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578fd826b687468ca8be54244c3b4bc0" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:34.456933 11297 raft_consensus.cc:2468] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:34.421985 11192 catalog_manager.cc:5697] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.458431 11486 leader_election.cc:304] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:34.459388 11724 raft_consensus.cc:2804] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:34.459789 11724 raft_consensus.cc:493] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.460163 11724 raft_consensus.cc:3060] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.461658 11566 tablet_bootstrap.cc:654] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.481046 11372 raft_consensus.cc:2468] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:34.482767 11411 leader_election.cc:304] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:34.448258 11568 consensus_queue.cc:260] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.483965 11632 raft_consensus.cc:2804] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:34.477193 11660 raft_consensus.cc:515] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.485831 11568 ts_tablet_manager.cc:1434] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.145s	user 0.006s	sys 0.001s
I20260709 12:26:34.486620 11632 raft_consensus.cc:697] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:34.487001 11568 tablet_bootstrap.cc:492] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:34.486794 11445 raft_consensus.cc:2468] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:34.487771 11632 consensus_queue.cc:237] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.491277 11660 leader_election.cc:290] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:34.492291 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3701a37fd75248f6b8b0ca0930791a9d" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:34.492964 11372 raft_consensus.cc:3060] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.493305 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3701a37fd75248f6b8b0ca0930791a9d" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:34.494201 11568 tablet_bootstrap.cc:654] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.494156 11446 raft_consensus.cc:3060] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.497233 11760 raft_consensus.cc:515] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.505518 11571 tablet_bootstrap.cc:654] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.495944 11447 raft_consensus.cc:1275] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 9968c0ed1974454d828996824f859433: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260709 12:26:34.507616 11447 pending_rounds.cc:85] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1: Aborting all ops after (but not including) 0
I20260709 12:26:34.507892 11447 pending_rounds.cc:107] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260709 12:26:34.508129 11447 raft_consensus.cc:2889] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260709 12:26:34.515281 11778 consensus_queue.cc:1048] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:34.526603 11760 leader_election.cc:290] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:34.529548 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:34.530305 11518 raft_consensus.cc:3060] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.531174 11566 tablet_bootstrap.cc:492] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:34.531701 11566 ts_tablet_manager.cc:1403] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.127s	user 0.010s	sys 0.003s
I20260709 12:26:34.532934 11298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea9a20aab6840c1afb32c29bac2f4a4" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:34.534601 11566 raft_consensus.cc:359] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.535338 11566 raft_consensus.cc:385] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.535602 11566 raft_consensus.cc:740] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.536294 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:34.536374 11566 consensus_queue.cc:260] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.538823 11566 ts_tablet_manager.cc:1434] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.004s
I20260709 12:26:34.516845 11724 raft_consensus.cc:515] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
W20260709 12:26:34.544296 11262 leader_election.cc:343] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:34.551201 11412 leader_election.cc:343] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.553413 11724 leader_election.cc:290] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:34.554494 11724 consensus_queue.cc:1048] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:34.559051 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578fd826b687468ca8be54244c3b4bc0" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:34.559823 11373 raft_consensus.cc:2468] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:34.563854 11566 tablet_bootstrap.cc:492] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:34.568864 11518 raft_consensus.cc:2468] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:34.571389 11370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578fd826b687468ca8be54244c3b4bc0" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:34.572108 11370 raft_consensus.cc:3060] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.570531 11446 raft_consensus.cc:2468] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:34.572873 11566 tablet_bootstrap.cc:654] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.579526 11261 leader_election.cc:304] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:34.579965 11373 raft_consensus.cc:1275] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 9968c0ed1974454d828996824f859433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260709 12:26:34.580751 11660 raft_consensus.cc:2804] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:34.583185 11660 raft_consensus.cc:697] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:34.584110 11660 consensus_queue.cc:237] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.589089 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "578fd826b687468ca8be54244c3b4bc0" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:34.589533 11370 raft_consensus.cc:2468] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:34.591336 11778 consensus_queue.cc:1048] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:34.592546 11261 leader_election.cc:304] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e
I20260709 12:26:34.602093 11703 raft_consensus.cc:2804] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:34.602674 11703 raft_consensus.cc:697] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:34.593602 11724 consensus_queue.cc:1048] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:34.602077 11568 tablet_bootstrap.cc:492] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:34.578368 11760 raft_consensus.cc:493] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.604547 11568 ts_tablet_manager.cc:1403] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.118s	user 0.017s	sys 0.013s
I20260709 12:26:34.603434 11295 raft_consensus.cc:3060] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.609897 11609 raft_consensus.cc:493] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.610443 11609 raft_consensus.cc:515] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.612568 11609 leader_election.cc:290] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:34.614877 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a5d23b0089f4d37a0c63d127537ec93" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:34.621304 11447 raft_consensus.cc:2468] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:34.606671 11760 raft_consensus.cc:515] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.622849 11336 leader_election.cc:304] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:34.626263 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a5d23b0089f4d37a0c63d127537ec93" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:34.626567 11370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea9a20aab6840c1afb32c29bac2f4a4" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:34.627341 11370 raft_consensus.cc:2393] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 663d88d9634e4720a695c7cd69eaa25e in current term 1: Already voted for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in this term.
I20260709 12:26:34.627844 11571 tablet_bootstrap.cc:492] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:34.627622 11703 consensus_queue.cc:237] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.628556 11571 ts_tablet_manager.cc:1403] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.262s	user 0.014s	sys 0.004s
I20260709 12:26:34.623838 11609 raft_consensus.cc:2804] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:34.629168 11609 raft_consensus.cc:493] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.629738 11609 raft_consensus.cc:3060] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.630682 11760 leader_election.cc:290] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:34.626263 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea9a20aab6840c1afb32c29bac2f4a4" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
W20260709 12:26:34.632241 11336 leader_election.cc:343] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.618474 11372 raft_consensus.cc:2468] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:34.633951 11567 tablet_bootstrap.cc:492] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:34.634481 11567 ts_tablet_manager.cc:1403] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.354s	user 0.014s	sys 0.026s
I20260709 12:26:34.605316 11485 leader_election.cc:304] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:34.652104 11778 raft_consensus.cc:2804] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:34.652201 11567 raft_consensus.cc:359] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.653163 11778 raft_consensus.cc:697] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:34.653368 11567 raft_consensus.cc:385] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.653854 11567 raft_consensus.cc:740] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.654812 11567 consensus_queue.cc:260] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.656862 11567 ts_tablet_manager.cc:1434] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.022s	user 0.007s	sys 0.000s
I20260709 12:26:34.658047 11567 tablet_bootstrap.cc:492] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:34.658591 11609 raft_consensus.cc:515] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.651391 11194 catalog_manager.cc:5697] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:34.664702 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a5d23b0089f4d37a0c63d127537ec93" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:34.665552 11447 raft_consensus.cc:3060] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.660928 11609 leader_election.cc:290] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:34.654443 11778 consensus_queue.cc:237] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.665242 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a5d23b0089f4d37a0c63d127537ec93" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:34.668103 11520 raft_consensus.cc:3060] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:34.670746 11485 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:44567, user_credentials={real_user=slave}} blocked reactor thread for 69325.4us
I20260709 12:26:34.678617 11520 raft_consensus.cc:2468] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:34.680239 11336 leader_election.cc:304] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:34.681488 11609 raft_consensus.cc:2804] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:34.681051 11571 raft_consensus.cc:359] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.682090 11609 raft_consensus.cc:697] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:34.682153 11571 raft_consensus.cc:385] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.695036 11571 raft_consensus.cc:740] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.683120 11609 consensus_queue.cc:237] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.664777 11567 tablet_bootstrap.cc:654] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.693683 11447 raft_consensus.cc:2468] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:34.646502 11261 leader_election.cc:304] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:34.704860 11295 raft_consensus.cc:2468] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:34.705202 11760 raft_consensus.cc:3060] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:34.704702 11261 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:36253, user_credentials={real_user=slave}} blocked reactor thread for 67443us
I20260709 12:26:34.711262 11566 tablet_bootstrap.cc:492] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:34.711890 11566 ts_tablet_manager.cc:1403] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.148s	user 0.035s	sys 0.040s
I20260709 12:26:34.711756 11571 consensus_queue.cc:260] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.715118 11566 raft_consensus.cc:359] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.715888 11566 raft_consensus.cc:385] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.716195 11566 raft_consensus.cc:740] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.716907 11566 consensus_queue.cc:260] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.717623 11571 ts_tablet_manager.cc:1434] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.089s	user 0.010s	sys 0.000s
I20260709 12:26:34.718988 11571 tablet_bootstrap.cc:492] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:34.715896 11568 raft_consensus.cc:359] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.723532 11568 raft_consensus.cc:385] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.723927 11568 raft_consensus.cc:740] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.724764 11568 consensus_queue.cc:260] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.725494 11571 tablet_bootstrap.cc:654] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.727185 11566 ts_tablet_manager.cc:1434] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:26:34.731621 11568 ts_tablet_manager.cc:1434] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.127s	user 0.005s	sys 0.004s
I20260709 12:26:34.731871 11778 consensus_queue.cc:1048] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:34.733191 11568 tablet_bootstrap.cc:492] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:34.734073 11192 catalog_manager.cc:5697] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 1 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.728466 11566 tablet_bootstrap.cc:492] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:34.748786 11571 tablet_bootstrap.cc:492] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:34.749366 11571 ts_tablet_manager.cc:1403] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.031s	user 0.022s	sys 0.004s
I20260709 12:26:34.752235 11571 raft_consensus.cc:359] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.752919 11571 raft_consensus.cc:385] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.753204 11571 raft_consensus.cc:740] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.753324 11568 tablet_bootstrap.cc:654] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.753912 11571 consensus_queue.cc:260] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.757323 11571 ts_tablet_manager.cc:1434] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:26:34.759294 11571 tablet_bootstrap.cc:492] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:34.757763 11193 catalog_manager.cc:5697] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 1 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.762046 11297 raft_consensus.cc:3060] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.767326 11566 tablet_bootstrap.cc:654] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.768025 11571 tablet_bootstrap.cc:654] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.785264 11571 tablet_bootstrap.cc:492] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:34.786285 11571 ts_tablet_manager.cc:1403] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.027s	user 0.014s	sys 0.005s
I20260709 12:26:34.789359 11571 raft_consensus.cc:359] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.790105 11571 raft_consensus.cc:385] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.790463 11571 raft_consensus.cc:740] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.791610 11571 consensus_queue.cc:260] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.793651 11568 tablet_bootstrap.cc:492] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:34.794148 11571 ts_tablet_manager.cc:1434] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:26:34.796797 11566 tablet_bootstrap.cc:492] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:34.809615 11566 ts_tablet_manager.cc:1403] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.081s	user 0.011s	sys 0.005s
I20260709 12:26:34.810101 11571 tablet_bootstrap.cc:492] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:34.812633 11566 raft_consensus.cc:359] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.813411 11566 raft_consensus.cc:385] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.813716 11566 raft_consensus.cc:740] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.794214 11568 ts_tablet_manager.cc:1403] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.061s	user 0.011s	sys 0.000s
I20260709 12:26:34.794706 11624 raft_consensus.cc:493] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.797286 11635 raft_consensus.cc:493] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.816109 11635 raft_consensus.cc:515] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.819774 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:34.819084 11195 catalog_manager.cc:5697] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.820660 11520 raft_consensus.cc:2393] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 05b93ba4435f40fb97b54343c4fdd71e in current term 1: Already voted for candidate 663d88d9634e4720a695c7cd69eaa25e in this term.
I20260709 12:26:34.820814 11566 consensus_queue.cc:260] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.819984 11568 raft_consensus.cc:359] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.821848 11568 raft_consensus.cc:385] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.822160 11568 raft_consensus.cc:740] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.823338 11635 leader_election.cc:290] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:34.815171 11624 raft_consensus.cc:515] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.823064 11568 consensus_queue.cc:260] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.826540 11624 leader_election.cc:290] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:34.827271 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee281c78e97848f59386b1d97f96bcfe" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:34.827628 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee281c78e97848f59386b1d97f96bcfe" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:34.828033 11369 raft_consensus.cc:2468] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
W20260709 12:26:34.829309 11410 leader_election.cc:343] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.829471 11571 tablet_bootstrap.cc:654] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.830790 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:34.832088 11760 raft_consensus.cc:2749] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:34.833184 11566 ts_tablet_manager.cc:1434] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.023s	user 0.006s	sys 0.001s
I20260709 12:26:34.832310 11568 ts_tablet_manager.cc:1434] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.018s	user 0.005s	sys 0.002s
I20260709 12:26:34.834829 11411 leader_election.cc:304] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:34.835639 11568 tablet_bootstrap.cc:492] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:34.836529 11566 tablet_bootstrap.cc:492] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:34.837915 11632 raft_consensus.cc:2804] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:34.842903 11632 raft_consensus.cc:493] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:34.843302 11632 raft_consensus.cc:3060] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.847290 11724 consensus_queue.cc:1048] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:34.847987 11338 leader_election.cc:304] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:34.850176 11571 tablet_bootstrap.cc:492] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:34.850751 11571 ts_tablet_manager.cc:1403] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.041s	user 0.020s	sys 0.004s
I20260709 12:26:34.850939 11609 raft_consensus.cc:3060] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.851850 11632 raft_consensus.cc:515] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.853778 11571 raft_consensus.cc:359] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.854489 11571 raft_consensus.cc:385] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.854799 11571 raft_consensus.cc:740] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.855571 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee281c78e97848f59386b1d97f96bcfe" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:34.855484 11571 consensus_queue.cc:260] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.856245 11369 raft_consensus.cc:3060] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.857695 11571 ts_tablet_manager.cc:1434] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.000s
I20260709 12:26:34.857617 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ee281c78e97848f59386b1d97f96bcfe" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:34.858937 11571 tablet_bootstrap.cc:492] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
W20260709 12:26:34.859704 11410 leader_election.cc:343] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:34.860678 11632 leader_election.cc:290] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:34.865927 11571 tablet_bootstrap.cc:654] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.866110 11369 raft_consensus.cc:2468] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:34.867528 11411 leader_election.cc:304] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:34.869030 11724 consensus_queue.cc:1048] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:34.871861 11632 raft_consensus.cc:2804] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:34.872547 11632 raft_consensus.cc:697] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:34.873839 11567 tablet_bootstrap.cc:492] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:34.874370 11567 ts_tablet_manager.cc:1403] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.217s	user 0.039s	sys 0.032s
I20260709 12:26:34.873602 11632 consensus_queue.cc:237] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.877311 11567 raft_consensus.cc:359] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.878063 11567 raft_consensus.cc:385] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.878343 11567 raft_consensus.cc:740] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.880725 11566 tablet_bootstrap.cc:654] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.882895 11295 raft_consensus.cc:3060] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:34.883208 11567 consensus_queue.cc:260] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.886278 11567 ts_tablet_manager.cc:1434] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:26:34.886461 11568 tablet_bootstrap.cc:654] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.887420 11567 tablet_bootstrap.cc:492] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:34.893321 11196 catalog_manager.cc:5697] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.894276 11776 consensus_queue.cc:1048] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:34.894220 11567 tablet_bootstrap.cc:654] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.923221 11568 tablet_bootstrap.cc:492] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:34.923952 11568 ts_tablet_manager.cc:1403] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.089s	user 0.017s	sys 0.003s
I20260709 12:26:34.926918 11566 tablet_bootstrap.cc:492] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:34.927826 11566 ts_tablet_manager.cc:1403] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.094s	user 0.021s	sys 0.000s
I20260709 12:26:34.920034 11571 tablet_bootstrap.cc:492] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:34.923128 11609 raft_consensus.cc:2749] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:34.928989 11571 ts_tablet_manager.cc:1403] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.070s	user 0.014s	sys 0.011s
I20260709 12:26:34.913714 11194 catalog_manager.cc:5697] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:34.935770 11568 raft_consensus.cc:359] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.936735 11568 raft_consensus.cc:385] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.936619 11571 raft_consensus.cc:359] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.937256 11568 raft_consensus.cc:740] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.937573 11571 raft_consensus.cc:385] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.937979 11571 raft_consensus.cc:740] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.938364 11568 consensus_queue.cc:260] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.938724 11571 consensus_queue.cc:260] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.941170 11568 ts_tablet_manager.cc:1434] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.017s	user 0.006s	sys 0.000s
I20260709 12:26:34.941502 11571 ts_tablet_manager.cc:1434] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 12:26:34.942534 11568 tablet_bootstrap.cc:492] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:34.942718 11571 tablet_bootstrap.cc:492] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:34.944660 11566 raft_consensus.cc:359] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.945662 11566 raft_consensus.cc:385] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.946012 11566 raft_consensus.cc:740] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.947300 11566 consensus_queue.cc:260] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:34.957350 11566 ts_tablet_manager.cc:1434] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.001s
I20260709 12:26:34.958562 11566 tablet_bootstrap.cc:492] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:34.961938 11571 tablet_bootstrap.cc:654] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.968297 11568 tablet_bootstrap.cc:654] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.978673 11566 tablet_bootstrap.cc:654] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:34.988854 11566 tablet_bootstrap.cc:492] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:34.989351 11566 ts_tablet_manager.cc:1403] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.031s	user 0.010s	sys 0.003s
I20260709 12:26:34.989161 11567 tablet_bootstrap.cc:492] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:34.989974 11567 ts_tablet_manager.cc:1403] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.103s	user 0.013s	sys 0.007s
I20260709 12:26:34.992885 11567 raft_consensus.cc:359] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:34.993588 11567 raft_consensus.cc:385] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.993866 11567 raft_consensus.cc:740] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.995451 11609 raft_consensus.cc:493] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:34.996035 11568 tablet_bootstrap.cc:492] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:34.995983 11609 raft_consensus.cc:515] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:34.996588 11568 ts_tablet_manager.cc:1403] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.054s	user 0.014s	sys 0.000s
I20260709 12:26:34.996418 11566 raft_consensus.cc:359] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:34.997195 11566 raft_consensus.cc:385] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:34.997486 11566 raft_consensus.cc:740] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:34.998167 11609 leader_election.cc:290] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:34.998346 11566 consensus_queue.cc:260] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.000636 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:35.000923 11566 ts_tablet_manager.cc:1434] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.011s	user 0.003s	sys 0.004s
I20260709 12:26:35.001333 11571 tablet_bootstrap.cc:492] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:35.001859 11571 ts_tablet_manager.cc:1403] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.059s	user 0.010s	sys 0.001s
I20260709 12:26:35.001478 11568 raft_consensus.cc:359] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:35.002274 11568 raft_consensus.cc:385] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.002475 11660 raft_consensus.cc:493] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.002565 11568 raft_consensus.cc:740] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.003106 11660 raft_consensus.cc:515] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.003823 11568 consensus_queue.cc:260] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:35.004590 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:35.005240 11446 raft_consensus.cc:2468] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:35.006814 11568 ts_tablet_manager.cc:1434] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.010s	user 0.010s	sys 0.000s
W20260709 12:26:35.007438 11336 leader_election.cc:343] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:35.007691 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf22a8b3fa4491c92c735f8151982d3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:35.007997 11568 tablet_bootstrap.cc:492] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:35.008448 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf22a8b3fa4491c92c735f8151982d3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:35.008740 11660 leader_election.cc:290] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:34.994618 11567 consensus_queue.cc:260] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:35.009179 11520 raft_consensus.cc:2468] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:35.011417 11567 ts_tablet_manager.cc:1434] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:26:35.002082 11566 tablet_bootstrap.cc:492] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:35.014434 11261 leader_election.cc:304] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:35.015112 11568 tablet_bootstrap.cc:654] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.015906 11660 raft_consensus.cc:2804] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:35.016317 11660 raft_consensus.cc:493] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:35.016661 11660 raft_consensus.cc:3060] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.016914 11567 tablet_bootstrap.cc:492] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:35.018167 11632 consensus_queue.cc:1048] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:35.019421 11760 consensus_queue.cc:1048] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:35.024161 11567 tablet_bootstrap.cc:654] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.024375 11579 raft_consensus.cc:493] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.024919 11579 raft_consensus.cc:515] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.026656 11336 leader_election.cc:304] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:35.027908 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0175551ed6284d01b80e6a34e24a07ed" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:35.028616 11297 raft_consensus.cc:2468] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:35.029145 11579 leader_election.cc:290] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:35.029765 11579 raft_consensus.cc:2804] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:35.030112 11338 leader_election.cc:304] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:35.030879 11566 tablet_bootstrap.cc:654] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.031126 11635 raft_consensus.cc:2804] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:35.031540 11635 raft_consensus.cc:493] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:35.031890 11635 raft_consensus.cc:3060] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.030130 11579 raft_consensus.cc:493] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:35.035434 11579 raft_consensus.cc:3060] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.045300 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0175551ed6284d01b80e6a34e24a07ed" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:35.046195 11447 raft_consensus.cc:2468] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:35.005034 11571 raft_consensus.cc:359] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.061138 11571 raft_consensus.cc:385] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.061652 11571 raft_consensus.cc:740] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.070143 11571 consensus_queue.cc:260] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.074126 11369 raft_consensus.cc:2468] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:35.075785 11632 consensus_queue.cc:1048] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:35.099969 11571 ts_tablet_manager.cc:1434] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.098s	user 0.006s	sys 0.004s
I20260709 12:26:35.102512 11571 tablet_bootstrap.cc:492] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:35.098901 11579 raft_consensus.cc:515] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:35.113817 11567 tablet_bootstrap.cc:492] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:35.115841 11567 ts_tablet_manager.cc:1403] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.099s	user 0.030s	sys 0.012s
I20260709 12:26:35.118278 11660 raft_consensus.cc:515] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.120923 11568 tablet_bootstrap.cc:492] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:35.121474 11568 ts_tablet_manager.cc:1403] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.114s	user 0.048s	sys 0.042s
W20260709 12:26:35.130461 11244 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:35.137600 11566 tablet_bootstrap.cc:492] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:35.138152 11566 ts_tablet_manager.cc:1403] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.136s	user 0.014s	sys 0.027s
I20260709 12:26:35.127159 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf22a8b3fa4491c92c735f8151982d3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:35.107419 11635 raft_consensus.cc:515] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.131492 11660 leader_election.cc:290] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
W20260709 12:26:35.125579 11336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:36253, user_credentials={real_user=slave}} blocked reactor thread for 81413.9us
I20260709 12:26:35.142279 11520 raft_consensus.cc:3060] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.142840 11568 raft_consensus.cc:359] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.143601 11568 raft_consensus.cc:385] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.143874 11568 raft_consensus.cc:740] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.144527 11568 consensus_queue.cc:260] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.146003 11609 raft_consensus.cc:493] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.146733 11568 ts_tablet_manager.cc:1434] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.025s	user 0.009s	sys 0.000s
I20260709 12:26:35.146562 11609 raft_consensus.cc:515] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.147818 11568 tablet_bootstrap.cc:492] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:35.148759 11609 leader_election.cc:290] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:35.159834 11819 raft_consensus.cc:493] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.158034 11776 consensus_queue.cc:1048] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.004s
I20260709 12:26:35.161006 11819 raft_consensus.cc:515] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.162181 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4438988b178419385f7ec82ef7c99d2" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:35.164201 11298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4438988b178419385f7ec82ef7c99d2" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
W20260709 12:26:35.166325 11338 leader_election.cc:343] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:35.167701 11635 leader_election.cc:290] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:35.169974 11571 tablet_bootstrap.cc:654] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:35.164482 11818 log.cc:927] Time spent T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e: Append to log took a long time: real 0.055s	user 0.001s	sys 0.000s
I20260709 12:26:35.170564 11298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0175551ed6284d01b80e6a34e24a07ed" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:35.172335 11298 raft_consensus.cc:3060] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.172677 11566 raft_consensus.cc:359] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:35.177067 11566 raft_consensus.cc:385] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.177353 11566 raft_consensus.cc:740] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.180004 11566 consensus_queue.cc:260] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
W20260709 12:26:35.180871 11336 leader_election.cc:343] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:35.182711 11566 ts_tablet_manager.cc:1434] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.044s	user 0.008s	sys 0.001s
I20260709 12:26:35.184007 11566 tablet_bootstrap.cc:492] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:35.181464 11336 leader_election.cc:304] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:35.186247 11609 raft_consensus.cc:2749] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:35.189359 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0175551ed6284d01b80e6a34e24a07ed" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:35.190213 11447 raft_consensus.cc:3060] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.191766 11819 leader_election.cc:290] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:35.192556 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0f568d1ffe4160afdf9d1bb4ba7c42" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
W20260709 12:26:35.194205 11338 leader_election.cc:343] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:35.196384 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0f568d1ffe4160afdf9d1bb4ba7c42" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:35.197198 11445 raft_consensus.cc:2468] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:35.199683 11447 raft_consensus.cc:2468] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:35.199949 11624 raft_consensus.cc:493] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.200542 11624 raft_consensus.cc:515] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:35.201586 11579 leader_election.cc:290] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:35.202472 11566 tablet_bootstrap.cc:654] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.205114 11567 raft_consensus.cc:359] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:35.205871 11567 raft_consensus.cc:385] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.206168 11567 raft_consensus.cc:740] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.206863 11567 consensus_queue.cc:260] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:35.208268 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47908558560f437a92445e6ab3a30a57" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:35.208959 11518 raft_consensus.cc:2468] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:35.209076 11567 ts_tablet_manager.cc:1434] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.093s	user 0.006s	sys 0.000s
I20260709 12:26:35.211318 11624 leader_election.cc:290] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:35.213742 11760 consensus_queue.cc:1048] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:35.215672 11370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf22a8b3fa4491c92c735f8151982d3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:35.216346 11370 raft_consensus.cc:3060] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.216511 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47908558560f437a92445e6ab3a30a57" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:35.217275 11297 raft_consensus.cc:2468] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:35.214838 11410 leader_election.cc:304] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:35.219398 11624 raft_consensus.cc:2804] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:35.219796 11624 raft_consensus.cc:493] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:35.220165 11624 raft_consensus.cc:3060] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.222404 11568 tablet_bootstrap.cc:654] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.244475 11824 consensus_queue.cc:1048] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:35.245784 11567 tablet_bootstrap.cc:492] T ee281c78e97848f59386b1d97f96bcfe P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:35.246747 11824 consensus_queue.cc:1048] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:35.250063 11336 leader_election.cc:304] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e
W20260709 12:26:35.255744 11336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:36253, user_credentials={real_user=slave}} blocked reactor thread for 55202.2us
I20260709 12:26:35.258914 11635 raft_consensus.cc:2804] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:35.259550 11635 raft_consensus.cc:493] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:35.259987 11635 raft_consensus.cc:3060] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.250968 11776 raft_consensus.cc:493] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.258281 11336 leader_election.cc:304] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:35.272722 11635 raft_consensus.cc:515] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.279804 11571 tablet_bootstrap.cc:492] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:35.250224 11520 raft_consensus.cc:2468] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:35.285127 11571 ts_tablet_manager.cc:1403] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.183s	user 0.019s	sys 0.004s
I20260709 12:26:35.284152 11624 raft_consensus.cc:515] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:35.309787 11566 tablet_bootstrap.cc:492] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:35.326231 11566 ts_tablet_manager.cc:1403] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.142s	user 0.007s	sys 0.013s
I20260709 12:26:35.326674 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:35.327888 11624 leader_election.cc:290] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:35.329159 11566 raft_consensus.cc:359] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.329882 11566 raft_consensus.cc:385] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.332824 11517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47908558560f437a92445e6ab3a30a57" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:35.333485 11517 raft_consensus.cc:3060] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.333921 11571 raft_consensus.cc:359] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.285333 11776 raft_consensus.cc:515] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:35.293521 11819 raft_consensus.cc:2804] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:35.336243 11819 raft_consensus.cc:697] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:35.336556 11776 leader_election.cc:290] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:35.317600 11724 consensus_queue.cc:1048] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:35.337255 11521 raft_consensus.cc:3060] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.337495 11819 consensus_queue.cc:237] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.321811 11298 raft_consensus.cc:2468] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:35.325381 11567 tablet_bootstrap.cc:654] T ee281c78e97848f59386b1d97f96bcfe P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.340803 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0f568d1ffe4160afdf9d1bb4ba7c42" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:35.347954 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:35.340958 11566 raft_consensus.cc:740] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.334231 11635 leader_election.cc:290] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:35.319087 11579 consensus_queue.cc:1048] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:35.321153 11370 raft_consensus.cc:2468] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:35.341223 11571 raft_consensus.cc:385] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.350229 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:35.358987 11372 raft_consensus.cc:2393] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 1: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:35.351316 11566 consensus_queue.cc:260] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.362331 11566 ts_tablet_manager.cc:1434] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.036s	user 0.005s	sys 0.001s
I20260709 12:26:35.363261 11297 raft_consensus.cc:3060] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.358119 11571 raft_consensus.cc:740] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.369534 11571 consensus_queue.cc:260] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.356439 11447 raft_consensus.cc:3060] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.391026 11566 tablet_bootstrap.cc:492] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:35.392266 11817 consensus_queue.cc:1048] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260709 12:26:35.395889 11568 tablet_bootstrap.cc:492] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:35.396517 11568 ts_tablet_manager.cc:1403] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.249s	user 0.011s	sys 0.005s
I20260709 12:26:35.399024 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8b0f568d1ffe4160afdf9d1bb4ba7c42" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:35.399746 11446 raft_consensus.cc:3060] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.400995 11566 tablet_bootstrap.cc:654] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:35.390021 11412 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 76919.9us
I20260709 12:26:35.419231 11571 ts_tablet_manager.cc:1434] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.132s	user 0.009s	sys 0.000s
I20260709 12:26:35.423059 11568 raft_consensus.cc:359] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:35.423731 11568 raft_consensus.cc:385] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.423961 11568 raft_consensus.cc:740] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.427821 11571 tablet_bootstrap.cc:492] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:35.429039 11568 consensus_queue.cc:260] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:35.418156 11521 raft_consensus.cc:2468] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:35.438256 11568 ts_tablet_manager.cc:1434] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.041s	user 0.006s	sys 0.002s
I20260709 12:26:35.450593 11296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "47908558560f437a92445e6ab3a30a57" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:35.451462 11296 raft_consensus.cc:3060] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.452242 11447 raft_consensus.cc:2468] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:35.463738 11568 tablet_bootstrap.cc:492] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:35.435247 11262 leader_election.cc:304] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:35.466969 11760 raft_consensus.cc:2804] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:35.467550 11760 raft_consensus.cc:697] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:35.468477 11760 consensus_queue.cc:237] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.472450 11296 raft_consensus.cc:2468] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:35.485704 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:35.487159 11444 raft_consensus.cc:2393] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 1: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:35.489969 11446 raft_consensus.cc:2468] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:35.492300 11567 tablet_bootstrap.cc:492] T ee281c78e97848f59386b1d97f96bcfe P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:35.492211 11484 leader_election.cc:304] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:35.493079 11567 ts_tablet_manager.cc:1403] T ee281c78e97848f59386b1d97f96bcfe P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.247s	user 0.018s	sys 0.015s
I20260709 12:26:35.494086 11715 raft_consensus.cc:2749] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:35.509030 11566 tablet_bootstrap.cc:492] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:35.516116 11566 ts_tablet_manager.cc:1403] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.127s	user 0.019s	sys 0.005s
I20260709 12:26:35.541585 11567 raft_consensus.cc:359] T ee281c78e97848f59386b1d97f96bcfe P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:35.515249 11336 leader_election.cc:304] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:35.542724 11567 raft_consensus.cc:385] T ee281c78e97848f59386b1d97f96bcfe P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.544206 11567 raft_consensus.cc:740] T ee281c78e97848f59386b1d97f96bcfe P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.545421 11567 consensus_queue.cc:260] T ee281c78e97848f59386b1d97f96bcfe P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:35.548278 11567 ts_tablet_manager.cc:1434] T ee281c78e97848f59386b1d97f96bcfe P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.055s	user 0.011s	sys 0.000s
I20260709 12:26:35.549574 11567 tablet_bootstrap.cc:492] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
W20260709 12:26:35.553670 11336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 50019.4us
I20260709 12:26:35.563220 11336 leader_election.cc:304] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:35.558719 11412 leader_election.cc:304] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:35.564524 11632 raft_consensus.cc:2804] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:35.565042 11632 raft_consensus.cc:697] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:35.555225 11579 raft_consensus.cc:2804] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:35.566656 11517 raft_consensus.cc:2468] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:35.569985 11609 raft_consensus.cc:2804] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:35.570595 11609 raft_consensus.cc:697] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:35.571486 11609 consensus_queue.cc:237] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.565989 11632 consensus_queue.cc:237] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
W20260709 12:26:35.580262 11410 consensus_peers.cc:597] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Couldn't send request to peer 9968c0ed1974454d828996824f859433. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:35.583736 11571 tablet_bootstrap.cc:654] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:35.594331 11829 log.cc:927] Time spent T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433: Append to log took a long time: real 0.129s	user 0.000s	sys 0.001s
I20260709 12:26:35.617771 11566 raft_consensus.cc:359] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.630365 11566 raft_consensus.cc:385] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.631769 11566 raft_consensus.cc:740] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.613020 11579 raft_consensus.cc:697] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:35.634855 11819 raft_consensus.cc:493] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.651870 11624 consensus_queue.cc:1048] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:35.656361 11566 consensus_queue.cc:260] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.659891 11297 raft_consensus.cc:2468] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:35.660264 11819 raft_consensus.cc:515] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.693639 11566 ts_tablet_manager.cc:1434] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.177s	user 0.010s	sys 0.001s
I20260709 12:26:35.685173 11579 consensus_queue.cc:237] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:35.718824 11568 tablet_bootstrap.cc:654] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.732826 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ce080ae4c98415a868195fb1df17f18" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
W20260709 12:26:35.734593 11336 leader_election.cc:343] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:35.738070 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ce080ae4c98415a868195fb1df17f18" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:35.738829 11297 raft_consensus.cc:2468] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:35.740031 11338 leader_election.cc:304] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:35.741005 11819 leader_election.cc:290] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:35.741411 11609 raft_consensus.cc:2804] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:35.741861 11609 raft_consensus.cc:493] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:35.742231 11609 raft_consensus.cc:3060] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.743090 11566 tablet_bootstrap.cc:492] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:35.752936 11567 tablet_bootstrap.cc:654] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.760798 11568 tablet_bootstrap.cc:492] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:35.761632 11568 ts_tablet_manager.cc:1403] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.301s	user 0.012s	sys 0.017s
I20260709 12:26:35.767033 11571 tablet_bootstrap.cc:492] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:35.767736 11571 ts_tablet_manager.cc:1403] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.343s	user 0.016s	sys 0.009s
I20260709 12:26:35.768997 11195 catalog_manager.cc:5697] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 1 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:35.766660 11191 catalog_manager.cc:5697] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:35.771098 11571 raft_consensus.cc:359] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:35.771837 11571 raft_consensus.cc:385] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.772141 11571 raft_consensus.cc:740] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.772195 11568 raft_consensus.cc:359] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:35.774623 11566 tablet_bootstrap.cc:654] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.773197 11568 raft_consensus.cc:385] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.775916 11568 raft_consensus.cc:740] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.776803 11568 consensus_queue.cc:260] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:35.777799 11571 consensus_queue.cc:260] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:35.780454 11571 ts_tablet_manager.cc:1434] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.000s
I20260709 12:26:35.782164 11571 tablet_bootstrap.cc:492] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:35.783769 11568 ts_tablet_manager.cc:1434] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.022s	user 0.006s	sys 0.004s
I20260709 12:26:35.790283 11568 tablet_bootstrap.cc:492] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:35.793040 11579 consensus_queue.cc:1048] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:35.800413 11568 tablet_bootstrap.cc:654] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.809614 11571 tablet_bootstrap.cc:654] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.788744 11609 raft_consensus.cc:515] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.841745 11566 tablet_bootstrap.cc:492] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:35.842281 11566 ts_tablet_manager.cc:1403] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.099s	user 0.009s	sys 0.006s
I20260709 12:26:35.844699 11571 tablet_bootstrap.cc:492] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:35.845284 11571 ts_tablet_manager.cc:1403] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.064s	user 0.022s	sys 0.006s
I20260709 12:26:35.845122 11566 raft_consensus.cc:359] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:35.845870 11566 raft_consensus.cc:385] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.846185 11566 raft_consensus.cc:740] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.846903 11566 consensus_queue.cc:260] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:35.846982 11579 consensus_queue.cc:1048] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:35.850375 11566 ts_tablet_manager.cc:1434] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.008s	user 0.009s	sys 0.000s
I20260709 12:26:35.851603 11566 tablet_bootstrap.cc:492] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:35.855142 11571 raft_consensus.cc:359] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:35.855933 11571 raft_consensus.cc:385] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.856208 11571 raft_consensus.cc:740] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.856916 11571 consensus_queue.cc:260] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:35.859208 11571 ts_tablet_manager.cc:1434] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:26:35.860287 11571 tablet_bootstrap.cc:492] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:35.863258 11566 tablet_bootstrap.cc:654] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.867285 11571 tablet_bootstrap.cc:654] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.869904 11519 raft_consensus.cc:3060] T ee281c78e97848f59386b1d97f96bcfe P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.875422 11566 tablet_bootstrap.cc:492] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:35.875555 11624 raft_consensus.cc:493] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.876053 11566 ts_tablet_manager.cc:1403] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.025s	user 0.007s	sys 0.010s
I20260709 12:26:35.876622 11624 raft_consensus.cc:515] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:35.882025 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8471ace194cf46238a8cd2862aa8cd5d" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
W20260709 12:26:35.884606 11410 leader_election.cc:343] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:35.886930 11567 tablet_bootstrap.cc:492] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:35.887511 11567 ts_tablet_manager.cc:1403] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.338s	user 0.035s	sys 0.013s
I20260709 12:26:35.891009 11517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ce080ae4c98415a868195fb1df17f18" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:35.884004 11566 raft_consensus.cc:359] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.892542 11566 raft_consensus.cc:385] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.892877 11566 raft_consensus.cc:740] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
W20260709 12:26:35.893460 11336 leader_election.cc:343] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:35.896605 11609 leader_election.cc:290] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:35.893672 11566 consensus_queue.cc:260] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.906009 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8471ace194cf46238a8cd2862aa8cd5d" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:35.907979 11566 ts_tablet_manager.cc:1434] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.031s	user 0.006s	sys 0.000s
I20260709 12:26:35.908182 11193 catalog_manager.cc:5697] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:35.903141 11567 raft_consensus.cc:359] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.914566 11567 raft_consensus.cc:385] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.915050 11567 raft_consensus.cc:740] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.903877 11624 leader_election.cc:290] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:35.903644 11824 consensus_queue.cc:1048] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:35.904532 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ce080ae4c98415a868195fb1df17f18" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:35.916594 11567 consensus_queue.cc:260] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.917821 11294 raft_consensus.cc:3060] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:35.919842 11567 ts_tablet_manager.cc:1434] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.032s	user 0.006s	sys 0.002s
I20260709 12:26:35.921038 11567 tablet_bootstrap.cc:492] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:35.923439 11193 catalog_manager.cc:5697] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:35.932679 11193 catalog_manager.cc:5697] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:35.898953 11571 tablet_bootstrap.cc:492] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
W20260709 12:26:35.909946 11412 leader_election.cc:343] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:35.941985 11412 leader_election.cc:304] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:35.942945 11624 raft_consensus.cc:2749] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:35.942008 11571 ts_tablet_manager.cc:1403] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.082s	user 0.016s	sys 0.019s
I20260709 12:26:35.909057 11566 tablet_bootstrap.cc:492] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:35.954646 11568 tablet_bootstrap.cc:492] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:35.946384 11571 raft_consensus.cc:359] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.955189 11571 raft_consensus.cc:385] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.955561 11571 raft_consensus.cc:740] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.956246 11571 consensus_queue.cc:260] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.957938 11571 ts_tablet_manager.cc:1434] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.014s	user 0.003s	sys 0.000s
I20260709 12:26:35.955248 11568 ts_tablet_manager.cc:1403] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.165s	user 0.015s	sys 0.017s
I20260709 12:26:35.941220 11567 tablet_bootstrap.cc:654] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:35.963243 11264 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.29.201.254:38597, user_credentials={real_user=slave}} blocked reactor thread for 52375.1us
I20260709 12:26:35.969249 11568 raft_consensus.cc:359] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.970028 11568 raft_consensus.cc:385] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:35.970305 11568 raft_consensus.cc:740] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:35.970997 11660 raft_consensus.cc:493] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.971534 11660 raft_consensus.cc:515] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.971702 11568 consensus_queue.cc:260] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:35.973835 11660 leader_election.cc:290] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:35.975766 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16773253264a4f8282ea1ea5b59a1ec6" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:35.976468 11445 raft_consensus.cc:2468] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:35.977084 11294 raft_consensus.cc:2468] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:35.977587 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16773253264a4f8282ea1ea5b59a1ec6" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:35.977603 11660 consensus_queue.cc:1048] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:35.978359 11373 raft_consensus.cc:2468] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:35.980930 11566 tablet_bootstrap.cc:654] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:35.982383 11338 leader_election.cc:304] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:35.984448 11609 consensus_queue.cc:1048] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:35.987607 11817 raft_consensus.cc:2804] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:35.988183 11817 raft_consensus.cc:697] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:35.989101 11817 consensus_queue.cc:237] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.991775 11567 tablet_bootstrap.cc:492] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:35.992483 11567 ts_tablet_manager.cc:1403] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.072s	user 0.012s	sys 0.004s
I20260709 12:26:35.992722 11609 raft_consensus.cc:493] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:35.993391 11609 raft_consensus.cc:515] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:35.997665 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91b161cce9ce49c58571f631c6810266" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:35.998777 11609 leader_election.cc:290] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:35.998425 11567 raft_consensus.cc:359] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:35.999162 11517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91b161cce9ce49c58571f631c6810266" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:35.999409 11261 leader_election.cc:304] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:36.000278 11660 raft_consensus.cc:2804] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:36.000727 11660 raft_consensus.cc:493] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.001099 11660 raft_consensus.cc:3060] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:36.001348 11336 leader_election.cc:343] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:36.002135 11336 leader_election.cc:343] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:36.002578 11336 leader_election.cc:304] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433
I20260709 12:26:35.999428 11567 raft_consensus.cc:385] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.004285 11579 raft_consensus.cc:2749] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:36.004410 11567 raft_consensus.cc:740] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.006718 11567 consensus_queue.cc:260] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:36.008168 11571 tablet_bootstrap.cc:492] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:36.011713 11567 ts_tablet_manager.cc:1434] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.019s	user 0.009s	sys 0.000s
I20260709 12:26:36.013187 11567 tablet_bootstrap.cc:492] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:35.974253 11568 ts_tablet_manager.cc:1434] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.016s	user 0.000s	sys 0.006s
I20260709 12:26:36.023636 11568 tablet_bootstrap.cc:492] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:36.026609 11624 raft_consensus.cc:493] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.031919 11624 raft_consensus.cc:515] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.035333 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:36.036265 11567 tablet_bootstrap.cc:654] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.044495 11703 raft_consensus.cc:493] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.045135 11703 raft_consensus.cc:515] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.020752 11660 raft_consensus.cc:515] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:36.067416 11571 tablet_bootstrap.cc:654] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:36.082077 11410 leader_election.cc:343] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:36.081284 11624 leader_election.cc:290] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:36.086843 11566 tablet_bootstrap.cc:492] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:36.091004 11566 ts_tablet_manager.cc:1403] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.182s	user 0.018s	sys 0.053s
I20260709 12:26:36.094043 11566 raft_consensus.cc:359] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:36.095031 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:36.095100 11566 raft_consensus.cc:385] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.095739 11566 raft_consensus.cc:740] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.095824 11297 raft_consensus.cc:2468] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:36.096613 11566 consensus_queue.cc:260] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:36.099073 11566 ts_tablet_manager.cc:1434] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:26:36.096640 11632 consensus_queue.cc:1048] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260709 12:26:36.101862 11778 raft_consensus.cc:493] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.102599 11778 raft_consensus.cc:515] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:36.101826 11566 tablet_bootstrap.cc:492] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:36.097859 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16773253264a4f8282ea1ea5b59a1ec6" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:36.105373 11372 raft_consensus.cc:3060] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.106244 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ce080ae4c98415a868195fb1df17f18" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:36.107849 11778 leader_election.cc:290] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:36.121701 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ce080ae4c98415a868195fb1df17f18" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:36.122728 11297 raft_consensus.cc:2393] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 1: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:36.123515 11412 leader_election.cc:304] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:36.131603 11486 leader_election.cc:304] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:36.142375 11828 raft_consensus.cc:2804] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:36.143181 11828 raft_consensus.cc:493] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.143680 11828 raft_consensus.cc:3060] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.152474 11778 raft_consensus.cc:3060] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.154732 11660 leader_election.cc:290] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:36.167285 11566 tablet_bootstrap.cc:654] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.171463 11624 consensus_queue.cc:1048] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:36.175403 11568 tablet_bootstrap.cc:654] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.179781 11844 consensus_queue.cc:1048] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:36.180658 11571 tablet_bootstrap.cc:492] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:36.181198 11571 ts_tablet_manager.cc:1403] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.173s	user 0.013s	sys 0.015s
I20260709 12:26:36.195834 11571 raft_consensus.cc:359] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:36.196965 11571 raft_consensus.cc:385] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.197295 11571 raft_consensus.cc:740] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.197582 11844 raft_consensus.cc:493] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.198210 11844 raft_consensus.cc:515] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.198237 11567 tablet_bootstrap.cc:492] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:36.203169 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eda5b4099d041c9aef56884ca4e594a" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:36.203225 11567 ts_tablet_manager.cc:1403] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.190s	user 0.031s	sys 0.012s
I20260709 12:26:36.205984 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eda5b4099d041c9aef56884ca4e594a" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:36.207841 11844 leader_election.cc:290] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
W20260709 12:26:36.208659 11336 leader_election.cc:343] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:36.209734 11336 leader_election.cc:343] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:36.209371 11567 raft_consensus.cc:359] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.210381 11567 raft_consensus.cc:385] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.210350 11336 leader_election.cc:304] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433
I20260709 12:26:36.210924 11567 raft_consensus.cc:740] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.211604 11579 consensus_queue.cc:1048] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:36.211879 11567 consensus_queue.cc:260] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.213141 11579 raft_consensus.cc:2749] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:36.213348 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:36.198117 11571 consensus_queue.cc:260] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:36.225492 11703 leader_election.cc:290] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:36.227247 11567 ts_tablet_manager.cc:1434] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.022s	user 0.008s	sys 0.000s
I20260709 12:26:36.232949 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16773253264a4f8282ea1ea5b59a1ec6" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:36.233732 11445 raft_consensus.cc:3060] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.235472 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:36.236385 11519 raft_consensus.cc:2468] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:36.237803 11261 leader_election.cc:304] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:36.234776 11571 ts_tablet_manager.cc:1434] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.053s	user 0.010s	sys 0.001s
I20260709 12:26:36.242146 11571 tablet_bootstrap.cc:492] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:36.243453 11567 tablet_bootstrap.cc:492] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:36.244942 11568 tablet_bootstrap.cc:492] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:36.246963 11660 raft_consensus.cc:2804] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:36.288525 11660 raft_consensus.cc:493] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.289047 11660 raft_consensus.cc:3060] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.292445 11567 tablet_bootstrap.cc:654] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:36.294389 11244 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:36.296788 11817 consensus_queue.cc:1048] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:36.295404 11778 raft_consensus.cc:2749] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:36.302989 11568 ts_tablet_manager.cc:1403] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.280s	user 0.016s	sys 0.002s
I20260709 12:26:36.307464 11568 raft_consensus.cc:359] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.308228 11568 raft_consensus.cc:385] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.308525 11568 raft_consensus.cc:740] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.309782 11567 tablet_bootstrap.cc:492] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:36.310333 11567 ts_tablet_manager.cc:1403] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.067s	user 0.011s	sys 0.000s
I20260709 12:26:36.263731 11445 raft_consensus.cc:2468] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:36.266568 11566 tablet_bootstrap.cc:492] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:36.313407 11566 ts_tablet_manager.cc:1403] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.212s	user 0.013s	sys 0.009s
I20260709 12:26:36.316511 11261 leader_election.cc:304] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:36.258023 11828 raft_consensus.cc:515] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.317150 11660 raft_consensus.cc:515] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.285280 11372 raft_consensus.cc:2468] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:36.321978 11660 leader_election.cc:290] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:36.309252 11568 consensus_queue.cc:260] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.317754 11824 raft_consensus.cc:2804] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:36.323392 11824 raft_consensus.cc:697] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:36.324602 11568 ts_tablet_manager.cc:1434] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.021s	user 0.007s	sys 0.000s
I20260709 12:26:36.324214 11824 consensus_queue.cc:237] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:36.325708 11568 tablet_bootstrap.cc:492] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:36.328326 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:36.318531 11566 raft_consensus.cc:359] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.330382 11566 raft_consensus.cc:385] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.330680 11566 raft_consensus.cc:740] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.330869 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:36.331324 11566 consensus_queue.cc:260] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.331858 11521 raft_consensus.cc:3060] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.332572 11568 tablet_bootstrap.cc:654] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.333606 11566 ts_tablet_manager.cc:1434] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.000s
I20260709 12:26:36.319110 11444 raft_consensus.cc:2393] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 663d88d9634e4720a695c7cd69eaa25e in current term 1: Already voted for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in this term.
W20260709 12:26:36.335253 11876 log.cc:927] Time spent T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e: Append to log took a long time: real 0.059s	user 0.000s	sys 0.000s
I20260709 12:26:36.339989 11566 tablet_bootstrap.cc:492] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:36.340483 11517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:36.335259 11194 catalog_manager.cc:5697] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:36.355556 11566 tablet_bootstrap.cc:654] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.356666 11828 leader_election.cc:290] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:36.357143 11703 raft_consensus.cc:493] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.357730 11703 raft_consensus.cc:515] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:36.359143 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:36.360097 11294 raft_consensus.cc:2393] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5a1ce6e048ab47d69c453b5218cb64c1 in current term 1: Already voted for candidate 663d88d9634e4720a695c7cd69eaa25e in this term.
I20260709 12:26:36.360488 11703 leader_election.cc:290] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:36.361392 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4438988b178419385f7ec82ef7c99d2" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:36.362154 11519 raft_consensus.cc:2468] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:36.363320 11412 leader_election.cc:304] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:36.363543 11261 leader_election.cc:304] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:36.366307 11632 raft_consensus.cc:2749] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:26:36.369469 11571 tablet_bootstrap.cc:654] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.374010 11567 raft_consensus.cc:359] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.375142 11567 raft_consensus.cc:385] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.375483 11567 raft_consensus.cc:740] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.378619 11567 consensus_queue.cc:260] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.371306 11703 raft_consensus.cc:2804] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:36.387176 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4438988b178419385f7ec82ef7c99d2" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:36.397418 11372 raft_consensus.cc:2468] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:36.399031 11703 raft_consensus.cc:493] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.399799 11703 raft_consensus.cc:3060] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.425133 11874 raft_consensus.cc:493] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.425827 11874 raft_consensus.cc:515] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.428732 11874 leader_election.cc:290] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:36.430178 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eda5b4099d041c9aef56884ca4e594a" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:36.432184 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eda5b4099d041c9aef56884ca4e594a" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:36.432965 11369 raft_consensus.cc:2468] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:36.434405 11485 leader_election.cc:304] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
W20260709 12:26:36.437127 11484 leader_election.cc:343] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:36.441807 11778 raft_consensus.cc:2804] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:36.442349 11778 raft_consensus.cc:493] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.443851 11778 raft_consensus.cc:3060] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.447849 11567 ts_tablet_manager.cc:1434] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.137s	user 0.012s	sys 0.000s
I20260709 12:26:36.457327 11703 raft_consensus.cc:515] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:36.459942 11703 leader_election.cc:290] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:36.460691 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4438988b178419385f7ec82ef7c99d2" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:36.461308 11521 raft_consensus.cc:2468] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:36.462941 11261 leader_election.cc:304] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:36.461587 11518 raft_consensus.cc:3060] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.464174 11703 raft_consensus.cc:2804] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:36.465476 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4438988b178419385f7ec82ef7c99d2" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:36.465772 11703 raft_consensus.cc:697] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:36.466192 11372 raft_consensus.cc:3060] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.466975 11703 consensus_queue.cc:237] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.475615 11567 tablet_bootstrap.cc:492] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:36.470288 11778 raft_consensus.cc:515] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.480698 11195 catalog_manager.cc:5697] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 1 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:36.483078 11566 tablet_bootstrap.cc:492] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:36.483937 11566 ts_tablet_manager.cc:1403] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.150s	user 0.033s	sys 0.023s
I20260709 12:26:36.486658 11372 raft_consensus.cc:2468] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:36.493635 11566 raft_consensus.cc:359] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:36.494333 11566 raft_consensus.cc:385] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.494596 11566 raft_consensus.cc:740] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.502941 11844 consensus_queue.cc:1048] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:36.505283 11567 tablet_bootstrap.cc:654] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.508276 11262 leader_election.cc:304] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:36.509299 11703 raft_consensus.cc:2804] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:36.509040 11566 consensus_queue.cc:260] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:36.510138 11703 raft_consensus.cc:697] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:36.510820 11518 raft_consensus.cc:2468] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:36.511762 11703 consensus_queue.cc:237] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:36.517465 11566 ts_tablet_manager.cc:1434] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.033s	user 0.008s	sys 0.000s
I20260709 12:26:36.523186 11566 tablet_bootstrap.cc:492] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:36.517122 11568 tablet_bootstrap.cc:492] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:36.518734 11660 raft_consensus.cc:493] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.526335 11660 raft_consensus.cc:515] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:36.528707 11660 leader_election.cc:290] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:36.530372 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:36.530959 11566 tablet_bootstrap.cc:654] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.531131 11369 raft_consensus.cc:2468] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:36.531155 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:36.533531 11262 leader_election.cc:304] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:36.534152 11817 consensus_queue.cc:1048] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:36.534688 11660 raft_consensus.cc:2804] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:36.535319 11660 raft_consensus.cc:493] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.535761 11660 raft_consensus.cc:3060] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:36.537345 11261 leader_election.cc:343] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:36.519639 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eda5b4099d041c9aef56884ca4e594a" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:36.543066 11372 raft_consensus.cc:3060] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.545493 11778 leader_election.cc:290] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:36.547458 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eda5b4099d041c9aef56884ca4e594a" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
W20260709 12:26:36.549209 11484 leader_election.cc:343] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:36.550549 11571 tablet_bootstrap.cc:492] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:36.551167 11571 ts_tablet_manager.cc:1403] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.309s	user 0.008s	sys 0.008s
I20260709 12:26:36.554153 11571 raft_consensus.cc:359] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.551167 11568 ts_tablet_manager.cc:1403] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.226s	user 0.022s	sys 0.028s
I20260709 12:26:36.554894 11571 raft_consensus.cc:385] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.555204 11571 raft_consensus.cc:740] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.558440 11568 raft_consensus.cc:359] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.559319 11568 raft_consensus.cc:385] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.559469 11567 tablet_bootstrap.cc:492] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:36.559631 11568 raft_consensus.cc:740] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.560130 11567 ts_tablet_manager.cc:1403] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.085s	user 0.019s	sys 0.006s
I20260709 12:26:36.563614 11567 raft_consensus.cc:359] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.564357 11567 raft_consensus.cc:385] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.564637 11567 raft_consensus.cc:740] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.565503 11571 consensus_queue.cc:260] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.554533 11660 raft_consensus.cc:515] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:36.568575 11566 tablet_bootstrap.cc:492] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:36.569095 11566 ts_tablet_manager.cc:1403] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.046s	user 0.014s	sys 0.000s
I20260709 12:26:36.569667 11568 consensus_queue.cc:260] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.572154 11566 raft_consensus.cc:359] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.572896 11566 raft_consensus.cc:385] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.573159 11566 raft_consensus.cc:740] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.573899 11566 consensus_queue.cc:260] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.576139 11571 ts_tablet_manager.cc:1434] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.025s	user 0.005s	sys 0.001s
I20260709 12:26:36.576224 11660 leader_election.cc:290] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:36.577387 11571 tablet_bootstrap.cc:492] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:36.578750 11568 ts_tablet_manager.cc:1434] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.001s
I20260709 12:26:36.578949 11567 consensus_queue.cc:260] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.579793 11568 tablet_bootstrap.cc:492] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:36.579818 11778 raft_consensus.cc:493] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.580435 11778 raft_consensus.cc:515] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.581007 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:36.578395 11372 raft_consensus.cc:2468] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:36.585635 11485 leader_election.cc:304] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:36.586978 11874 raft_consensus.cc:2804] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:36.587674 11874 raft_consensus.cc:697] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:36.588531 11566 ts_tablet_manager.cc:1434] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.019s	user 0.006s	sys 0.000s
W20260709 12:26:36.589025 11261 leader_election.cc:343] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:36.588392 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:36.588531 11874 consensus_queue.cc:237] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.592187 11571 tablet_bootstrap.cc:654] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.590276 11369 raft_consensus.cc:3060] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.586875 11567 ts_tablet_manager.cc:1434] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.001s
I20260709 12:26:36.589609 11566 tablet_bootstrap.cc:492] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:36.597165 11778 leader_election.cc:290] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:36.597414 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8471ace194cf46238a8cd2862aa8cd5d" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:36.587018 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8471ace194cf46238a8cd2862aa8cd5d" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:36.598165 11294 raft_consensus.cc:2468] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:36.598511 11445 raft_consensus.cc:2468] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:36.599464 11635 raft_consensus.cc:493] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.599821 11486 leader_election.cc:304] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:36.600092 11635 raft_consensus.cc:515] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.601586 11778 raft_consensus.cc:2804] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:36.602020 11778 raft_consensus.cc:493] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.602413 11778 raft_consensus.cc:3060] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.604492 11635 leader_election.cc:290] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:36.605837 11567 tablet_bootstrap.cc:492] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:36.607074 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "019a6ed4536442de96c44bde2d713c8b" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
W20260709 12:26:36.609242 11336 leader_election.cc:343] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:36.611444 11778 raft_consensus.cc:515] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.612056 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "019a6ed4536442de96c44bde2d713c8b" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:36.613139 11567 tablet_bootstrap.cc:654] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.616976 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8471ace194cf46238a8cd2862aa8cd5d" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:36.617642 11294 raft_consensus.cc:3060] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.616350 11778 leader_election.cc:290] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:36.622205 11369 raft_consensus.cc:2468] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:36.621791 11193 catalog_manager.cc:5697] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:36.622736 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8471ace194cf46238a8cd2862aa8cd5d" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:36.623387 11445 raft_consensus.cc:3060] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.623929 11262 leader_election.cc:304] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1
W20260709 12:26:36.624585 11336 leader_election.cc:343] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:36.625102 11336 leader_election.cc:304] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433
I20260709 12:26:36.625288 11660 raft_consensus.cc:2804] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:36.625924 11635 raft_consensus.cc:2749] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:36.626063 11660 raft_consensus.cc:697] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:36.629097 11566 tablet_bootstrap.cc:654] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.627103 11660 consensus_queue.cc:237] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:36.659456 11294 raft_consensus.cc:2468] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:36.663650 11486 leader_election.cc:304] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:36.669525 11568 tablet_bootstrap.cc:654] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.675669 11445 raft_consensus.cc:2468] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:36.676043 11566 tablet_bootstrap.cc:492] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:36.681924 11566 ts_tablet_manager.cc:1403] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.093s	user 0.013s	sys 0.001s
I20260709 12:26:36.673592 11778 raft_consensus.cc:2804] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:36.683125 11778 raft_consensus.cc:697] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:36.684110 11778 consensus_queue.cc:237] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.684937 11566 raft_consensus.cc:359] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.685920 11566 raft_consensus.cc:385] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.686236 11566 raft_consensus.cc:740] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.686991 11566 consensus_queue.cc:260] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.689240 11566 ts_tablet_manager.cc:1434] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.007s	user 0.004s	sys 0.001s
I20260709 12:26:36.690894 11571 tablet_bootstrap.cc:492] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:36.691972 11566 tablet_bootstrap.cc:492] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:36.692131 11571 ts_tablet_manager.cc:1403] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.115s	user 0.026s	sys 0.012s
I20260709 12:26:36.695842 11567 tablet_bootstrap.cc:492] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:36.696993 11571 raft_consensus.cc:359] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:36.697808 11571 raft_consensus.cc:385] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.698156 11571 raft_consensus.cc:740] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.698807 11827 raft_consensus.cc:493] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.702281 11827 raft_consensus.cc:515] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.704602 11827 leader_election.cc:290] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:36.705662 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91b161cce9ce49c58571f631c6810266" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:36.706426 11369 raft_consensus.cc:2468] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:36.706717 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91b161cce9ce49c58571f631c6810266" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:36.707852 11411 leader_election.cc:304] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
W20260709 12:26:36.708715 11410 leader_election.cc:343] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:36.709609 11827 raft_consensus.cc:2804] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:36.710119 11827 raft_consensus.cc:493] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.710567 11827 raft_consensus.cc:3060] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.701752 11568 tablet_bootstrap.cc:492] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:36.710744 11567 ts_tablet_manager.cc:1403] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.105s	user 0.008s	sys 0.003s
I20260709 12:26:36.700026 11195 catalog_manager.cc:5697] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:36.711297 11568 ts_tablet_manager.cc:1403] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.132s	user 0.007s	sys 0.006s
I20260709 12:26:36.713675 11567 raft_consensus.cc:359] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.713925 11568 raft_consensus.cc:359] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.714417 11567 raft_consensus.cc:385] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.714546 11568 raft_consensus.cc:385] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.714779 11567 raft_consensus.cc:740] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.708907 11571 consensus_queue.cc:260] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:36.717718 11571 ts_tablet_manager.cc:1434] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.024s	user 0.006s	sys 0.000s
I20260709 12:26:36.714869 11568 raft_consensus.cc:740] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.719480 11571 tablet_bootstrap.cc:492] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:36.721747 11828 raft_consensus.cc:493] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.722297 11828 raft_consensus.cc:515] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.723773 11568 consensus_queue.cc:260] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.725709 11827 raft_consensus.cc:515] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.728483 11827 leader_election.cc:290] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:36.728847 11567 consensus_queue.cc:260] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.729794 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91b161cce9ce49c58571f631c6810266" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
W20260709 12:26:36.731406 11410 leader_election.cc:343] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:36.733235 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91b161cce9ce49c58571f631c6810266" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:36.734437 11369 raft_consensus.cc:3060] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.735014 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eda5b4099d041c9aef56884ca4e594a" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:36.737381 11568 ts_tablet_manager.cc:1434] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.026s	user 0.006s	sys 0.000s
I20260709 12:26:36.738117 11567 ts_tablet_manager.cc:1434] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.027s	user 0.006s	sys 0.000s
I20260709 12:26:36.738466 11568 tablet_bootstrap.cc:492] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:36.739121 11567 tablet_bootstrap.cc:492] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:36.733754 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eda5b4099d041c9aef56884ca4e594a" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:36.739885 11373 raft_consensus.cc:2393] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a1ce6e048ab47d69c453b5218cb64c1 in current term 1: Already voted for candidate 9968c0ed1974454d828996824f859433 in this term.
I20260709 12:26:36.748621 11194 catalog_manager.cc:5697] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 1 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:36.744411 11828 leader_election.cc:290] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:36.754257 11411 leader_election.cc:304] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433
I20260709 12:26:36.750751 11194 catalog_manager.cc:5697] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 1 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:36.758425 11194 catalog_manager.cc:5697] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 1 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:36.762897 11571 tablet_bootstrap.cc:654] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.772276 11568 tablet_bootstrap.cc:654] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.757464 11827 raft_consensus.cc:3060] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.775705 11660 raft_consensus.cc:493] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.776270 11660 raft_consensus.cc:515] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.782302 11566 tablet_bootstrap.cc:654] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.791767 11567 tablet_bootstrap.cc:654] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.801973 11703 consensus_queue.cc:1048] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:36.825062 11369 raft_consensus.cc:2468] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:36.826210 11568 tablet_bootstrap.cc:492] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:36.826810 11568 ts_tablet_manager.cc:1403] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.089s	user 0.009s	sys 0.006s
I20260709 12:26:36.830976 11827 raft_consensus.cc:2749] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:36.851464 11411 leader_election.cc:304] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:36.851557 11660 leader_election.cc:290] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:36.853562 11632 raft_consensus.cc:2804] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:36.854146 11632 raft_consensus.cc:697] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:36.854566 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8f2372474d94a11bb1e044fc69c60e5" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:36.855084 11632 consensus_queue.cc:237] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.855630 11519 raft_consensus.cc:2468] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:36.857501 11261 leader_election.cc:304] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:36.858445 11660 raft_consensus.cc:2804] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:36.858852 11660 raft_consensus.cc:493] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:36.859215 11660 raft_consensus.cc:3060] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.865440 11571 tablet_bootstrap.cc:492] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:36.866289 11571 ts_tablet_manager.cc:1403] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.147s	user 0.017s	sys 0.006s
I20260709 12:26:36.870339 11571 raft_consensus.cc:359] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.871316 11571 raft_consensus.cc:385] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.871695 11571 raft_consensus.cc:740] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.872659 11571 consensus_queue.cc:260] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.873117 11703 consensus_queue.cc:1048] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:36.876256 11571 ts_tablet_manager.cc:1434] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:26:36.880765 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8f2372474d94a11bb1e044fc69c60e5" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:36.881680 11568 raft_consensus.cc:359] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.882774 11568 raft_consensus.cc:385] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.883152 11568 raft_consensus.cc:740] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.881677 11445 raft_consensus.cc:2468] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:36.884159 11568 consensus_queue.cc:260] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.892975 11568 ts_tablet_manager.cc:1434] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.066s	user 0.004s	sys 0.003s
I20260709 12:26:36.898605 11568 tablet_bootstrap.cc:492] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:36.894526 11571 tablet_bootstrap.cc:492] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:36.925554 11567 tablet_bootstrap.cc:492] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:36.926194 11567 ts_tablet_manager.cc:1403] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.187s	user 0.010s	sys 0.006s
I20260709 12:26:36.929301 11567 raft_consensus.cc:359] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.930033 11567 raft_consensus.cc:385] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.930307 11567 raft_consensus.cc:740] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.931696 11567 consensus_queue.cc:260] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.936439 11567 ts_tablet_manager.cc:1434] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.004s
I20260709 12:26:36.939338 11567 tablet_bootstrap.cc:492] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:36.946136 11874 raft_consensus.cc:493] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:36.946879 11874 raft_consensus.cc:515] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:36.949225 11874 leader_election.cc:290] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:36.960489 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91b161cce9ce49c58571f631c6810266" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:36.961777 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91b161cce9ce49c58571f631c6810266" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:36.962709 11369 raft_consensus.cc:2393] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 1: Already voted for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in this term.
I20260709 12:26:36.963929 11485 leader_election.cc:304] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:36.964195 11660 raft_consensus.cc:515] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:36.965267 11874 raft_consensus.cc:3060] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.966562 11660 leader_election.cc:290] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:36.968032 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8f2372474d94a11bb1e044fc69c60e5" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:36.968686 11519 raft_consensus.cc:3060] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:36.968861 11566 tablet_bootstrap.cc:492] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:36.969432 11566 ts_tablet_manager.cc:1403] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.278s	user 0.029s	sys 0.056s
I20260709 12:26:36.977097 11568 tablet_bootstrap.cc:654] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:36.984655 11566 raft_consensus.cc:359] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:36.987726 11566 raft_consensus.cc:385] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:36.988032 11566 raft_consensus.cc:740] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:36.988915 11566 consensus_queue.cc:260] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:36.983613 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8f2372474d94a11bb1e044fc69c60e5" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:36.993999 11446 raft_consensus.cc:3060] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:37.001523 11519 raft_consensus.cc:2468] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:37.001937 11446 raft_consensus.cc:2468] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:37.004316 11194 catalog_manager.cc:5697] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:37.008906 11566 ts_tablet_manager.cc:1434] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.039s	user 0.009s	sys 0.000s
I20260709 12:26:37.012490 11874 raft_consensus.cc:2749] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:37.010982 11566 tablet_bootstrap.cc:492] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:37.018281 11567 tablet_bootstrap.cc:654] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:37.024358 11261 leader_election.cc:304] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:37.025673 11571 tablet_bootstrap.cc:654] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:37.027204 11703 raft_consensus.cc:2804] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:37.027815 11703 raft_consensus.cc:697] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:37.028728 11703 consensus_queue.cc:237] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:37.041060 11566 tablet_bootstrap.cc:654] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:37.058343 11703 consensus_queue.cc:1048] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:26:37.069725 11703 raft_consensus.cc:493] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.071308 11703 raft_consensus.cc:515] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:37.079778 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc185f43f658441ba207bdfdc0e79a7f" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:37.080641 11446 raft_consensus.cc:2468] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:37.082774 11261 leader_election.cc:304] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:37.084013 11824 consensus_queue.cc:1048] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:37.085682 11824 raft_consensus.cc:2804] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:37.086055 11824 raft_consensus.cc:493] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.086434 11824 raft_consensus.cc:3060] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:37.096441 11660 raft_consensus.cc:493] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.097105 11660 raft_consensus.cc:515] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.101651 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d019dd2bd864166abac89cc3fb1be76" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
W20260709 12:26:37.103294 11261 leader_election.cc:343] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:37.103982 11660 leader_election.cc:290] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:37.108101 11660 consensus_queue.cc:1048] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:37.108721 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d019dd2bd864166abac89cc3fb1be76" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:37.110281 11660 consensus_queue.cc:1048] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:26:37.110744 11262 leader_election.cc:343] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:37.111964 11262 leader_election.cc:304] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433
I20260709 12:26:37.079840 11862 consensus_queue.cc:1048] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.043s
I20260709 12:26:37.124898 11862 raft_consensus.cc:2749] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:37.073633 11571 tablet_bootstrap.cc:492] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:37.126868 11571 ts_tablet_manager.cc:1403] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.233s	user 0.013s	sys 0.003s
I20260709 12:26:37.128244 11874 consensus_queue.cc:1048] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:37.132534 11703 leader_election.cc:290] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:37.137879 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc185f43f658441ba207bdfdc0e79a7f" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:37.139207 11635 raft_consensus.cc:493] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.139782 11635 raft_consensus.cc:515] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:37.141882 11635 leader_election.cc:290] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:37.144645 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75195e7d58a548658a538f4628a056d8" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:37.144905 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75195e7d58a548658a538f4628a056d8" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
W20260709 12:26:37.146443 11336 leader_election.cc:343] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:37.134968 11571 raft_consensus.cc:359] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.151062 11571 raft_consensus.cc:385] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.151474 11571 raft_consensus.cc:740] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.153786 11568 tablet_bootstrap.cc:492] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:37.154716 11568 ts_tablet_manager.cc:1403] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.256s	user 0.012s	sys 0.009s
I20260709 12:26:37.152472 11571 consensus_queue.cc:260] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
W20260709 12:26:37.156666 11336 leader_election.cc:343] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:37.157660 11567 tablet_bootstrap.cc:492] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:37.158247 11567 ts_tablet_manager.cc:1403] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.219s	user 0.028s	sys 0.046s
I20260709 12:26:37.157383 11336 leader_election.cc:304] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433
W20260709 12:26:37.158612 11261 leader_election.cc:343] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:37.161872 11567 raft_consensus.cc:359] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.169088 11567 raft_consensus.cc:385] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.169409 11567 raft_consensus.cc:740] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.171535 11567 consensus_queue.cc:260] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.163661 11571 ts_tablet_manager.cc:1434] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.036s	user 0.008s	sys 0.001s
I20260709 12:26:37.164302 11635 raft_consensus.cc:2749] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:37.180850 11571 tablet_bootstrap.cc:492] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:37.195832 11567 ts_tablet_manager.cc:1434] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.037s	user 0.006s	sys 0.001s
I20260709 12:26:37.199256 11567 tablet_bootstrap.cc:492] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:37.225591 11874 raft_consensus.cc:493] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.226272 11874 raft_consensus.cc:515] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.228430 11874 leader_election.cc:290] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:37.240794 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d019dd2bd864166abac89cc3fb1be76" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:37.241559 11294 raft_consensus.cc:2468] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:37.246436 11817 raft_consensus.cc:493] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.247074 11817 raft_consensus.cc:515] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.249202 11817 leader_election.cc:290] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:37.251655 11486 leader_election.cc:304] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:37.254796 11778 raft_consensus.cc:2804] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:37.255242 11778 raft_consensus.cc:493] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.253969 11191 catalog_manager.cc:5697] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 1 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:37.255594 11778 raft_consensus.cc:3060] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:37.260964 11715 consensus_queue.cc:1048] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:37.204326 11824 raft_consensus.cc:515] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:37.267939 11824 leader_election.cc:290] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:37.282382 11778 raft_consensus.cc:515] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.274012 11567 tablet_bootstrap.cc:654] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:37.275426 11261 consensus_peers.cc:597] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e -> Peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Couldn't send request to peer 5a1ce6e048ab47d69c453b5218cb64c1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:37.275242 11568 raft_consensus.cc:359] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:37.276158 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc185f43f658441ba207bdfdc0e79a7f" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:37.295321 11568 raft_consensus.cc:385] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.301992 11568 raft_consensus.cc:740] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.310401 11568 consensus_queue.cc:260] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:37.303193 11446 raft_consensus.cc:3060] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:37.307029 11778 leader_election.cc:290] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:37.321398 11568 ts_tablet_manager.cc:1434] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.166s	user 0.011s	sys 0.000s
I20260709 12:26:37.322329 11296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d019dd2bd864166abac89cc3fb1be76" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:37.323195 11296 raft_consensus.cc:2468] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:37.330407 11568 tablet_bootstrap.cc:492] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:37.336208 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d019dd2bd864166abac89cc3fb1be76" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:37.347536 11338 leader_election.cc:304] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:37.354880 11294 raft_consensus.cc:3060] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:37.363580 11566 tablet_bootstrap.cc:492] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:37.364307 11566 ts_tablet_manager.cc:1403] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.354s	user 0.020s	sys 0.002s
I20260709 12:26:37.368532 11817 raft_consensus.cc:2804] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:37.369129 11817 raft_consensus.cc:493] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.369500 11817 raft_consensus.cc:3060] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:37.370193 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc185f43f658441ba207bdfdc0e79a7f" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
W20260709 12:26:37.375954 11261 leader_election.cc:343] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:37.379516 11294 raft_consensus.cc:2468] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:37.380510 11579 raft_consensus.cc:493] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.381086 11579 raft_consensus.cc:515] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:37.384361 11579 leader_election.cc:290] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:37.382598 11486 leader_election.cc:304] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:37.388130 11917 raft_consensus.cc:2804] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:37.388351 11296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7179cfd71f5c4bcf8beac36a18e4c94e" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:37.388875 11917 raft_consensus.cc:697] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:37.390003 11917 consensus_queue.cc:237] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.392356 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7179cfd71f5c4bcf8beac36a18e4c94e" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:37.393849 11446 raft_consensus.cc:2468] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:37.399377 11571 tablet_bootstrap.cc:654] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:37.395848 11566 raft_consensus.cc:359] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:37.400753 11261 leader_election.cc:304] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:37.401703 11566 raft_consensus.cc:385] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.401904 11703 raft_consensus.cc:2804] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:37.403115 11566 raft_consensus.cc:740] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.406744 11827 raft_consensus.cc:493] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.407307 11827 raft_consensus.cc:515] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:37.406252 11566 consensus_queue.cc:260] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:37.409408 11827 leader_election.cc:290] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:37.413223 11293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:37.412645 11566 ts_tablet_manager.cc:1434] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.048s	user 0.007s	sys 0.000s
I20260709 12:26:37.416260 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d019dd2bd864166abac89cc3fb1be76" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:37.420171 11517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d019dd2bd864166abac89cc3fb1be76" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:37.422209 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:37.428828 11566 tablet_bootstrap.cc:492] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:37.435794 11568 tablet_bootstrap.cc:654] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:37.417294 11703 raft_consensus.cc:697] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:37.416718 11370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d019dd2bd864166abac89cc3fb1be76" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:37.437925 11703 consensus_queue.cc:237] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:37.438723 11296 raft_consensus.cc:2468] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
W20260709 12:26:37.417850 11336 leader_election.cc:343] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:37.467861 11715 consensus_queue.cc:1048] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:26:37.462456 11336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:36253, user_credentials={real_user=slave}} blocked reactor thread for 44654.3us
I20260709 12:26:37.447341 11817 raft_consensus.cc:515] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.486462 11874 consensus_queue.cc:1048] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:37.489331 11567 tablet_bootstrap.cc:492] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:37.490144 11567 ts_tablet_manager.cc:1403] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.291s	user 0.019s	sys 0.010s
I20260709 12:26:37.491881 11632 consensus_queue.cc:1048] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:37.493283 11338 leader_election.cc:304] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:37.503083 11919 raft_consensus.cc:2804] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:37.503545 11919 raft_consensus.cc:493] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.503902 11919 raft_consensus.cc:3060] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:37.509234 11817 leader_election.cc:290] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:37.510154 11373 raft_consensus.cc:2393] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 1: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:37.461692 11411 leader_election.cc:304] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
W20260709 12:26:37.512971 11411 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:44567, user_credentials={real_user=slave}} blocked reactor thread for 56677.3us
I20260709 12:26:37.514377 11632 raft_consensus.cc:2749] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:37.515467 11566 tablet_bootstrap.cc:654] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:37.525442 11571 tablet_bootstrap.cc:492] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:37.525332 11567 raft_consensus.cc:359] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:37.526068 11571 ts_tablet_manager.cc:1403] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.346s	user 0.013s	sys 0.003s
I20260709 12:26:37.526198 11567 raft_consensus.cc:385] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.526577 11567 raft_consensus.cc:740] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.527216 11567 consensus_queue.cc:260] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:37.531278 11567 ts_tablet_manager.cc:1434] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.041s	user 0.003s	sys 0.004s
I20260709 12:26:37.532704 11862 consensus_queue.cc:1048] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:37.532814 11571 raft_consensus.cc:359] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.533720 11571 raft_consensus.cc:385] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.533994 11571 raft_consensus.cc:740] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.538661 11571 consensus_queue.cc:260] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.551787 11571 ts_tablet_manager.cc:1434] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.025s	user 0.007s	sys 0.002s
I20260709 12:26:37.553380 11571 tablet_bootstrap.cc:492] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:37.554412 11567 tablet_bootstrap.cc:492] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
W20260709 12:26:37.581197 11484 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:36253, user_credentials={real_user=slave}} blocked reactor thread for 104195us
I20260709 12:26:37.600903 11514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d019dd2bd864166abac89cc3fb1be76" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:37.594585 11919 raft_consensus.cc:515] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:37.640410 11296 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1d019dd2bd864166abac89cc3fb1be76" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:37.641405 11296 raft_consensus.cc:2393] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 05b93ba4435f40fb97b54343c4fdd71e in current term 1: Already voted for candidate 9968c0ed1974454d828996824f859433 in this term.
I20260709 12:26:37.660341 11930 consensus_queue.cc:1048] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:26:37.661350 11244 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:37.653653 11919 leader_election.cc:290] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:37.679977 11568 tablet_bootstrap.cc:492] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:37.680600 11568 ts_tablet_manager.cc:1403] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.356s	user 0.023s	sys 0.007s
I20260709 12:26:37.689378 11566 tablet_bootstrap.cc:492] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:37.689973 11566 ts_tablet_manager.cc:1403] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.261s	user 0.020s	sys 0.029s
I20260709 12:26:37.692400 11566 raft_consensus.cc:359] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:37.693068 11566 raft_consensus.cc:385] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.693312 11566 raft_consensus.cc:740] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.694057 11571 tablet_bootstrap.cc:654] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:37.693984 11566 consensus_queue.cc:260] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:37.695413 11298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7179cfd71f5c4bcf8beac36a18e4c94e" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:37.695669 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7179cfd71f5c4bcf8beac36a18e4c94e" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:37.696152 11298 raft_consensus.cc:3060] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:37.696092 11338 leader_election.cc:304] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:37.703662 11579 raft_consensus.cc:2749] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:26:37.708333 11567 tablet_bootstrap.cc:654] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:37.725248 11568 raft_consensus.cc:359] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:37.726114 11568 raft_consensus.cc:385] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.726409 11568 raft_consensus.cc:740] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.727285 11568 consensus_queue.cc:260] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:37.733565 11298 raft_consensus.cc:2468] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:37.701799 11193 catalog_manager.cc:5697] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 1 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:37.740499 11338 leader_election.cc:304] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:37.741458 11579 raft_consensus.cc:2804] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:37.742202 11579 raft_consensus.cc:697] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:37.743034 11579 consensus_queue.cc:237] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:37.745941 11446 raft_consensus.cc:3060] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:37.748385 11567 tablet_bootstrap.cc:492] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:37.748934 11567 ts_tablet_manager.cc:1403] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.217s	user 0.011s	sys 0.002s
W20260709 12:26:37.750941 11336 leader_election.cc:343] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:37.752063 11567 raft_consensus.cc:359] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.752923 11567 raft_consensus.cc:385] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.753250 11567 raft_consensus.cc:740] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.758173 11567 consensus_queue.cc:260] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.760254 11568 ts_tablet_manager.cc:1434] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.079s	user 0.009s	sys 0.000s
I20260709 12:26:37.762072 11567 ts_tablet_manager.cc:1434] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 12:26:37.763123 11567 tablet_bootstrap.cc:492] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:37.765044 11568 tablet_bootstrap.cc:492] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:37.765260 11195 catalog_manager.cc:5697] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:37.775691 11568 tablet_bootstrap.cc:654] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:37.778338 11567 tablet_bootstrap.cc:654] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:37.785213 11566 ts_tablet_manager.cc:1434] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.095s	user 0.003s	sys 0.003s
I20260709 12:26:37.786357 11566 tablet_bootstrap.cc:492] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:37.790395 11567 tablet_bootstrap.cc:492] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:37.791049 11568 tablet_bootstrap.cc:492] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:37.791100 11567 ts_tablet_manager.cc:1403] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.028s	user 0.012s	sys 0.000s
I20260709 12:26:37.791960 11568 ts_tablet_manager.cc:1403] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.029s	user 0.015s	sys 0.004s
I20260709 12:26:37.795083 11567 raft_consensus.cc:359] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:37.795989 11567 raft_consensus.cc:385] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.795306 11568 raft_consensus.cc:359] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:37.796581 11567 raft_consensus.cc:740] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.796878 11568 raft_consensus.cc:385] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.797515 11567 consensus_queue.cc:260] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:37.799844 11567 ts_tablet_manager.cc:1434] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:26:37.800849 11567 tablet_bootstrap.cc:492] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:37.797358 11568 raft_consensus.cc:740] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.803303 11568 consensus_queue.cc:260] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:37.807093 11568 ts_tablet_manager.cc:1434] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.015s	user 0.009s	sys 0.000s
I20260709 12:26:37.808082 11568 tablet_bootstrap.cc:492] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:37.808806 11567 tablet_bootstrap.cc:654] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:37.817512 11568 tablet_bootstrap.cc:654] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:37.821940 11824 raft_consensus.cc:493] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.822752 11824 raft_consensus.cc:515] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:37.824832 11824 leader_election.cc:290] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:37.827047 11567 tablet_bootstrap.cc:492] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:37.827732 11567 ts_tablet_manager.cc:1403] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.027s	user 0.013s	sys 0.009s
I20260709 12:26:37.827600 11514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c068ba8213a415b9df4e5380d0ad37e" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:37.831322 11567 raft_consensus.cc:359] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:37.832240 11567 raft_consensus.cc:385] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.833288 11567 raft_consensus.cc:740] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.833369 11568 tablet_bootstrap.cc:492] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
W20260709 12:26:37.833602 11261 leader_election.cc:343] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:37.834112 11568 ts_tablet_manager.cc:1403] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.026s	user 0.007s	sys 0.010s
I20260709 12:26:37.834494 11567 consensus_queue.cc:260] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:37.837631 11567 ts_tablet_manager.cc:1434] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:26:37.838786 11567 tablet_bootstrap.cc:492] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:37.848222 11567 tablet_bootstrap.cc:654] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:37.852859 11916 raft_consensus.cc:493] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.853644 11916 raft_consensus.cc:515] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:37.857400 11715 raft_consensus.cc:493] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.859984 11919 raft_consensus.cc:493] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:37.860627 11919 raft_consensus.cc:515] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.863855 11567 tablet_bootstrap.cc:492] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:37.864030 11514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "610a732cba674c24bb228d619f52fce7" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:37.864598 11567 ts_tablet_manager.cc:1403] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.026s	user 0.016s	sys 0.002s
I20260709 12:26:37.864807 11514 raft_consensus.cc:2468] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:37.858283 11715 raft_consensus.cc:515] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:37.869673 11567 raft_consensus.cc:359] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:37.866460 11916 leader_election.cc:290] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:37.870543 11715 leader_election.cc:290] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:37.873523 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "019a6ed4536442de96c44bde2d713c8b" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:37.886577 11373 raft_consensus.cc:2468] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:37.866562 11336 leader_election.cc:304] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:37.877502 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc185f43f658441ba207bdfdc0e79a7f" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:37.879351 11927 consensus_queue.cc:1048] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.010s
I20260709 12:26:37.890298 11485 leader_election.cc:304] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:37.879519 11443 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c068ba8213a415b9df4e5380d0ad37e" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:37.892042 11579 raft_consensus.cc:2804] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:37.892490 11579 raft_consensus.cc:493] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.892832 11579 raft_consensus.cc:3060] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:37.893321 11927 raft_consensus.cc:2804] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:37.893925 11927 raft_consensus.cc:493] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:37.894429 11927 raft_consensus.cc:3060] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:37.870589 11567 raft_consensus.cc:385] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.897748 11567 raft_consensus.cc:740] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.905838 11568 raft_consensus.cc:359] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:37.906857 11568 raft_consensus.cc:385] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:37.907176 11568 raft_consensus.cc:740] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:37.908257 11514 raft_consensus.cc:3060] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:37.914618 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "610a732cba674c24bb228d619f52fce7" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:37.919090 11919 leader_election.cc:290] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:37.919766 11916 consensus_queue.cc:1048] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:37.924708 11579 raft_consensus.cc:515] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:37.925707 11929 consensus_queue.cc:1048] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:37.928359 11298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "610a732cba674c24bb228d619f52fce7" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:37.931361 11943 consensus_queue.cc:1048] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:37.903245 11567 consensus_queue.cc:260] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:37.942116 11567 ts_tablet_manager.cc:1434] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.077s	user 0.007s	sys 0.002s
I20260709 12:26:37.944761 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "610a732cba674c24bb228d619f52fce7" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:37.955530 11519 raft_consensus.cc:3060] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:37.925961 11261 leader_election.cc:343] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:37.961112 11261 leader_election.cc:304] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433
I20260709 12:26:37.962232 11824 raft_consensus.cc:2749] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:37.953995 11194 catalog_manager.cc:5697] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:37.969906 11338 leader_election.cc:343] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:37.970708 11568 consensus_queue.cc:260] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:37.927496 11927 raft_consensus.cc:515] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
W20260709 12:26:37.979157 11338 leader_election.cc:343] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:37.983014 11927 leader_election.cc:290] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:37.984376 11567 tablet_bootstrap.cc:492] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:37.991675 11568 ts_tablet_manager.cc:1434] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.157s	user 0.007s	sys 0.000s
I20260709 12:26:37.992821 11568 tablet_bootstrap.cc:492] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:37.992878 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "019a6ed4536442de96c44bde2d713c8b" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:37.993712 11369 raft_consensus.cc:3060] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:37.994828 11824 consensus_queue.cc:1048] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:37.982372 11519 raft_consensus.cc:2468] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:38.007866 11579 leader_election.cc:290] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:38.010248 11566 tablet_bootstrap.cc:700] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e: Time spent opening tablet: real 0.124s	user 0.004s	sys 0.000s
I20260709 12:26:38.018931 11566 tablet_bootstrap.cc:654] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.027415 11568 tablet_bootstrap.cc:654] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.029959 11336 leader_election.cc:304] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:38.036844 11919 raft_consensus.cc:2804] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:38.052234 11567 tablet_bootstrap.cc:654] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.060909 11443 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc185f43f658441ba207bdfdc0e79a7f" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:38.062937 11442 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "019a6ed4536442de96c44bde2d713c8b" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:38.072532 11369 raft_consensus.cc:2468] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:38.073884 11919 raft_consensus.cc:697] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:38.074878 11919 consensus_queue.cc:237] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:38.075186 11485 leader_election.cc:304] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:38.077219 11715 raft_consensus.cc:2804] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:38.077833 11715 raft_consensus.cc:697] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:38.080046 11715 consensus_queue.cc:237] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.083058 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "019a6ed4536442de96c44bde2d713c8b" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:38.075704 11484 leader_election.cc:304] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e
W20260709 12:26:38.092453 11484 leader_election.cc:343] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:38.093883 11874 raft_consensus.cc:2749] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:38.094925 11571 tablet_bootstrap.cc:492] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:38.095484 11571 ts_tablet_manager.cc:1403] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.542s	user 0.078s	sys 0.077s
I20260709 12:26:38.098340 11571 raft_consensus.cc:359] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.099087 11571 raft_consensus.cc:385] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.099361 11571 raft_consensus.cc:740] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.100023 11571 consensus_queue.cc:260] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.102190 11571 ts_tablet_manager.cc:1434] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:26:38.105104 11567 tablet_bootstrap.cc:492] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:38.105789 11567 ts_tablet_manager.cc:1403] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.122s	user 0.012s	sys 0.008s
I20260709 12:26:38.111405 11943 raft_consensus.cc:493] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.112769 11943 raft_consensus.cc:515] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.113515 11942 raft_consensus.cc:493] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.114195 11942 raft_consensus.cc:515] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.115902 11567 raft_consensus.cc:359] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:38.116707 11567 raft_consensus.cc:385] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.116991 11567 raft_consensus.cc:740] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.117650 11514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75195e7d58a548658a538f4628a056d8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:38.117833 11567 consensus_queue.cc:260] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:38.118402 11514 raft_consensus.cc:2468] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:38.119835 11410 leader_election.cc:304] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
W20260709 12:26:38.111272 11484 leader_election.cc:343] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:38.120220 11567 ts_tablet_manager.cc:1434] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.000s
I20260709 12:26:38.120823 11935 raft_consensus.cc:2804] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:38.121331 11935 raft_consensus.cc:493] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.120915 11571 tablet_bootstrap.cc:492] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:38.121789 11935 raft_consensus.cc:3060] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:38.122660 11942 leader_election.cc:290] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:38.123457 11943 leader_election.cc:290] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:38.127029 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7179cfd71f5c4bcf8beac36a18e4c94e" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:38.127882 11297 raft_consensus.cc:2393] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a1ce6e048ab47d69c453b5218cb64c1 in current term 1: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:38.135175 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7179cfd71f5c4bcf8beac36a18e4c94e" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:38.134439 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75195e7d58a548658a538f4628a056d8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:38.136893 11566 tablet_bootstrap.cc:492] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:38.137055 11369 raft_consensus.cc:2468] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:38.137434 11566 ts_tablet_manager.cc:1403] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.351s	user 0.019s	sys 0.007s
I20260709 12:26:38.138482 11411 leader_election.cc:304] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:38.139359 11827 raft_consensus.cc:3060] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:38.121743 11567 tablet_bootstrap.cc:492] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:38.144624 11194 catalog_manager.cc:5697] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:38.148274 11568 tablet_bootstrap.cc:492] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:38.148833 11568 ts_tablet_manager.cc:1403] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.156s	user 0.017s	sys 0.004s
I20260709 12:26:38.151635 11568 raft_consensus.cc:359] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:38.152316 11568 raft_consensus.cc:385] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.152595 11568 raft_consensus.cc:740] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.153345 11935 raft_consensus.cc:515] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.140727 11566 raft_consensus.cc:359] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.155433 11566 raft_consensus.cc:385] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.155711 11566 raft_consensus.cc:740] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.156332 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75195e7d58a548658a538f4628a056d8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:38.156994 11369 raft_consensus.cc:3060] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:38.159607 11571 tablet_bootstrap.cc:654] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.159857 11827 raft_consensus.cc:2749] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:38.156397 11566 consensus_queue.cc:260] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.164683 11566 ts_tablet_manager.cc:1434] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.026s	user 0.002s	sys 0.003s
I20260709 12:26:38.165750 11566 tablet_bootstrap.cc:492] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:38.166106 11369 raft_consensus.cc:2468] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:38.168578 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "75195e7d58a548658a538f4628a056d8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:38.169442 11519 raft_consensus.cc:3060] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:38.171732 11566 tablet_bootstrap.cc:654] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.178849 11916 raft_consensus.cc:493] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.179114 11935 leader_election.cc:290] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:38.179512 11916 raft_consensus.cc:515] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:38.178388 11519 raft_consensus.cc:2468] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:38.181442 11410 leader_election.cc:304] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:38.181866 11916 leader_election.cc:290] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:38.182508 11827 raft_consensus.cc:2804] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:38.183118 11827 raft_consensus.cc:697] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:38.183981 11827 consensus_queue.cc:237] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.184500 11571 tablet_bootstrap.cc:492] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:38.185336 11571 ts_tablet_manager.cc:1403] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.065s	user 0.008s	sys 0.004s
I20260709 12:26:38.187395 11567 tablet_bootstrap.cc:654] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.188088 11571 raft_consensus.cc:359] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.191514 11571 raft_consensus.cc:385] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.191852 11571 raft_consensus.cc:740] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.192886 11571 consensus_queue.cc:260] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.194770 11571 ts_tablet_manager.cc:1434] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:26:38.190515 11568 consensus_queue.cc:260] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:38.198745 11568 ts_tablet_manager.cc:1434] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.050s	user 0.009s	sys 0.000s
I20260709 12:26:38.199319 11566 tablet_bootstrap.cc:492] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:38.199754 11568 tablet_bootstrap.cc:492] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:38.199864 11566 ts_tablet_manager.cc:1403] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.034s	user 0.007s	sys 0.004s
I20260709 12:26:38.201076 11919 consensus_queue.cc:1048] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:38.187505 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32c7abdd125b4f38b6398a9b0e92fedd" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:38.203842 11566 raft_consensus.cc:359] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:38.204734 11566 raft_consensus.cc:385] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.205009 11566 raft_consensus.cc:740] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
W20260709 12:26:38.206213 11485 leader_election.cc:343] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:38.206409 11566 consensus_queue.cc:260] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:38.207557 11568 tablet_bootstrap.cc:654] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.208638 11566 ts_tablet_manager.cc:1434] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.002s
I20260709 12:26:38.209264 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32c7abdd125b4f38b6398a9b0e92fedd" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:38.209678 11566 tablet_bootstrap.cc:492] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:38.211220 11571 tablet_bootstrap.cc:492] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
W20260709 12:26:38.215585 11486 leader_election.cc:343] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:38.216116 11486 leader_election.cc:304] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:38.217373 11916 raft_consensus.cc:2749] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:38.217998 11567 tablet_bootstrap.cc:492] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:38.218593 11567 ts_tablet_manager.cc:1403] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.097s	user 0.006s	sys 0.005s
I20260709 12:26:38.236423 11817 consensus_queue.cc:1048] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:38.249799 11568 tablet_bootstrap.cc:492] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:38.250481 11568 ts_tablet_manager.cc:1403] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.051s	user 0.010s	sys 0.000s
I20260709 12:26:38.253681 11568 raft_consensus.cc:359] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.254580 11568 raft_consensus.cc:385] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.258517 11568 raft_consensus.cc:740] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.259382 11568 consensus_queue.cc:260] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.261097 11568 ts_tablet_manager.cc:1434] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.010s	user 0.004s	sys 0.000s
I20260709 12:26:38.261945 11579 raft_consensus.cc:493] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.262151 11568 tablet_bootstrap.cc:492] T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:38.262491 11579 raft_consensus.cc:515] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.264463 11579 leader_election.cc:290] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:38.271044 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431c66d13d0f4904a42b33665a9c09e1" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:38.271157 11194 catalog_manager.cc:5697] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:38.272732 11336 leader_election.cc:343] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:38.274206 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431c66d13d0f4904a42b33665a9c09e1" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
W20260709 12:26:38.276412 11336 leader_election.cc:343] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:38.276979 11336 leader_election.cc:304] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433
I20260709 12:26:38.281211 11579 raft_consensus.cc:2749] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:38.288419 11191 catalog_manager.cc:5697] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:38.291260 11568 tablet_bootstrap.cc:654] T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.293289 11827 raft_consensus.cc:493] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.293895 11827 raft_consensus.cc:515] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.298794 11567 raft_consensus.cc:359] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.299615 11567 raft_consensus.cc:385] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.299906 11567 raft_consensus.cc:740] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.300609 11827 leader_election.cc:290] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:38.299919 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f74457716f0d46568488137f1cb73fa8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:38.301833 11297 raft_consensus.cc:2468] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:38.301774 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f74457716f0d46568488137f1cb73fa8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:38.302753 11373 raft_consensus.cc:2468] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:38.303503 11566 tablet_bootstrap.cc:654] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.303508 11412 leader_election.cc:304] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:38.305330 11827 raft_consensus.cc:2804] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:38.305748 11827 raft_consensus.cc:493] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.306236 11827 raft_consensus.cc:3060] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:38.306559 11571 tablet_bootstrap.cc:654] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.309558 11567 consensus_queue.cc:260] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.318934 11567 ts_tablet_manager.cc:1434] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.100s	user 0.003s	sys 0.003s
I20260709 12:26:38.325330 11935 raft_consensus.cc:493] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.326538 11916 raft_consensus.cc:493] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.326278 11935 raft_consensus.cc:515] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:38.327106 11916 raft_consensus.cc:515] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:38.329303 11916 leader_election.cc:290] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:38.331254 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c068ba8213a415b9df4e5380d0ad37e" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:38.331848 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c068ba8213a415b9df4e5380d0ad37e" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:38.332170 11297 raft_consensus.cc:2468] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:38.334568 11412 leader_election.cc:304] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:38.335264 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c068ba8213a415b9df4e5380d0ad37e" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:38.335939 11828 raft_consensus.cc:2804] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:38.336473 11828 raft_consensus.cc:493] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.336899 11828 raft_consensus.cc:3060] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:38.337102 11514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c068ba8213a415b9df4e5380d0ad37e" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:38.337759 11514 raft_consensus.cc:2468] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:38.339033 11567 tablet_bootstrap.cc:492] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:38.344154 11935 leader_election.cc:290] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:38.355245 11571 tablet_bootstrap.cc:492] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:38.355768 11571 ts_tablet_manager.cc:1403] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.145s	user 0.011s	sys 0.002s
I20260709 12:26:38.360965 11571 raft_consensus.cc:359] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.361624 11571 raft_consensus.cc:385] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.353761 11566 tablet_bootstrap.cc:492] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:38.361927 11571 raft_consensus.cc:740] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.362419 11566 ts_tablet_manager.cc:1403] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.153s	user 0.008s	sys 0.006s
I20260709 12:26:38.365513 11566 raft_consensus.cc:359] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.366079 11567 tablet_bootstrap.cc:654] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.366302 11566 raft_consensus.cc:385] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.368899 11571 consensus_queue.cc:260] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.370014 11566 raft_consensus.cc:740] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.370882 11566 consensus_queue.cc:260] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.371376 11571 ts_tablet_manager.cc:1434] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.002s
I20260709 12:26:38.369407 11827 raft_consensus.cc:515] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.373145 11571 tablet_bootstrap.cc:492] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:38.361279 11828 raft_consensus.cc:515] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:38.375183 11446 raft_consensus.cc:2393] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 1: Already voted for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in this term.
I20260709 12:26:38.375726 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f74457716f0d46568488137f1cb73fa8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:38.376448 11373 raft_consensus.cc:3060] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:38.378271 11568 tablet_bootstrap.cc:492] T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:38.378891 11568 ts_tablet_manager.cc:1403] T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.117s	user 0.029s	sys 0.016s
I20260709 12:26:38.378978 11514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c068ba8213a415b9df4e5380d0ad37e" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:38.379707 11514 raft_consensus.cc:3060] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:38.380121 11571 tablet_bootstrap.cc:654] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.383345 11828 leader_election.cc:290] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:38.383529 11484 leader_election.cc:304] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:38.384377 11566 ts_tablet_manager.cc:1434] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.022s	user 0.005s	sys 0.002s
I20260709 12:26:38.381980 11568 raft_consensus.cc:359] T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.384917 11568 raft_consensus.cc:385] T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.385229 11568 raft_consensus.cc:740] T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.386113 11294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0c068ba8213a415b9df4e5380d0ad37e" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:38.386826 11294 raft_consensus.cc:3060] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:38.388224 11373 raft_consensus.cc:2468] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:38.391054 11411 leader_election.cc:304] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:38.391866 11567 tablet_bootstrap.cc:492] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:38.392421 11567 ts_tablet_manager.cc:1403] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.071s	user 0.011s	sys 0.000s
I20260709 12:26:38.393209 11935 raft_consensus.cc:2804] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:38.393811 11935 raft_consensus.cc:697] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:38.389456 11827 leader_election.cc:290] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:38.396168 11567 raft_consensus.cc:359] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.397186 11567 raft_consensus.cc:385] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.398757 11567 raft_consensus.cc:740] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.399192 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f74457716f0d46568488137f1cb73fa8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:38.399992 11297 raft_consensus.cc:3060] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:38.400749 11567 consensus_queue.cc:260] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.402256 11935 consensus_queue.cc:237] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.407687 11571 tablet_bootstrap.cc:492] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:38.408305 11571 ts_tablet_manager.cc:1403] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.035s	user 0.010s	sys 0.007s
I20260709 12:26:38.412175 11566 tablet_bootstrap.cc:492] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:38.413115 11514 raft_consensus.cc:2468] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:38.414822 11294 raft_consensus.cc:2468] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:38.426182 11410 leader_election.cc:304] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:38.415416 11568 consensus_queue.cc:260] T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.417102 11567 ts_tablet_manager.cc:1434] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.000s
I20260709 12:26:38.427160 11827 raft_consensus.cc:2804] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:38.427664 11827 raft_consensus.cc:697] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:38.427796 11567 tablet_bootstrap.cc:492] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:38.428499 11827 consensus_queue.cc:237] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:38.411077 11571 raft_consensus.cc:359] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:38.430716 11571 raft_consensus.cc:385] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.431025 11571 raft_consensus.cc:740] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.431731 11571 consensus_queue.cc:260] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:38.434213 11571 ts_tablet_manager.cc:1434] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.026s	user 0.007s	sys 0.000s
I20260709 12:26:38.435391 11571 tablet_bootstrap.cc:492] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:38.437827 11568 ts_tablet_manager.cc:1434] T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.059s	user 0.006s	sys 0.000s
I20260709 12:26:38.425011 11916 raft_consensus.cc:2749] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:38.442456 11568 tablet_bootstrap.cc:492] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:38.441205 11567 tablet_bootstrap.cc:654] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.441234 11571 tablet_bootstrap.cc:654] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.447269 11191 catalog_manager.cc:5697] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:38.423736 11566 tablet_bootstrap.cc:654] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.450531 11568 tablet_bootstrap.cc:654] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.462248 11566 tablet_bootstrap.cc:492] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:38.462816 11566 ts_tablet_manager.cc:1403] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.051s	user 0.011s	sys 0.003s
I20260709 12:26:38.465512 11566 raft_consensus.cc:359] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.469254 11567 tablet_bootstrap.cc:492] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:38.469813 11567 ts_tablet_manager.cc:1403] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.042s	user 0.010s	sys 0.002s
I20260709 12:26:38.472656 11567 raft_consensus.cc:359] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:38.473526 11567 raft_consensus.cc:385] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.473814 11567 raft_consensus.cc:740] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.474460 11952 consensus_queue.cc:1048] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:38.474570 11567 consensus_queue.cc:260] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:38.466234 11566 raft_consensus.cc:385] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.475602 11566 raft_consensus.cc:740] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.476332 11566 consensus_queue.cc:260] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.480482 11567 ts_tablet_manager.cc:1434] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.010s	user 0.008s	sys 0.000s
I20260709 12:26:38.481930 11567 tablet_bootstrap.cc:492] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:38.477020 11297 raft_consensus.cc:2468] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:38.487162 11566 ts_tablet_manager.cc:1434] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.024s	user 0.005s	sys 0.002s
I20260709 12:26:38.488332 11566 tablet_bootstrap.cc:492] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:38.491659 11567 tablet_bootstrap.cc:654] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.493911 11571 tablet_bootstrap.cc:492] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:38.494678 11571 ts_tablet_manager.cc:1403] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.059s	user 0.008s	sys 0.004s
I20260709 12:26:38.496150 11191 catalog_manager.cc:5697] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:38.499514 11566 tablet_bootstrap.cc:654] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.518319 11568 tablet_bootstrap.cc:492] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:38.519116 11568 ts_tablet_manager.cc:1403] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.077s	user 0.017s	sys 0.016s
I20260709 12:26:38.520291 11952 raft_consensus.cc:493] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.524940 11952 raft_consensus.cc:515] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:38.521688 11703 raft_consensus.cc:493] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.526196 11703 raft_consensus.cc:515] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.530552 11703 leader_election.cc:290] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:38.530572 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32c7abdd125b4f38b6398a9b0e92fedd" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:38.532217 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:38.533594 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:38.536790 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32c7abdd125b4f38b6398a9b0e92fedd" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:38.538360 11446 raft_consensus.cc:3060] T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:38.539652 11338 leader_election.cc:343] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:38.539754 11566 tablet_bootstrap.cc:492] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:38.540460 11566 ts_tablet_manager.cc:1403] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.052s	user 0.009s	sys 0.016s
I20260709 12:26:38.527212 11952 leader_election.cc:290] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
W20260709 12:26:38.541072 11262 leader_election.cc:343] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:38.542089 11916 raft_consensus.cc:493] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.542908 11571 raft_consensus.cc:359] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.543910 11571 raft_consensus.cc:385] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.544256 11571 raft_consensus.cc:740] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.542577 11916 raft_consensus.cc:515] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.549845 11916 leader_election.cc:290] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:38.550568 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e637809da62b408a91639b907355cad2" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:38.551402 11373 raft_consensus.cc:2468] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:38.538364 11519 raft_consensus.cc:2468] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:38.554747 11485 leader_election.cc:304] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:38.555745 11874 consensus_queue.cc:1048] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:38.556385 11715 raft_consensus.cc:2804] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:38.556425 11336 leader_election.cc:304] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:38.556901 11715 raft_consensus.cc:493] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.557334 11715 raft_consensus.cc:3060] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:38.561101 11952 raft_consensus.cc:2804] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:38.561721 11952 raft_consensus.cc:493] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:38.562170 11952 raft_consensus.cc:3060] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:38.564657 11828 raft_consensus.cc:493] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.565448 11828 raft_consensus.cc:515] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:38.567090 11566 raft_consensus.cc:359] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:38.569177 11566 raft_consensus.cc:385] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.568929 11571 consensus_queue.cc:260] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.569630 11566 raft_consensus.cc:740] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.571338 11566 consensus_queue.cc:260] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:38.573009 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e637809da62b408a91639b907355cad2" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:38.573804 11446 raft_consensus.cc:2468] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:38.578897 11566 ts_tablet_manager.cc:1434] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.038s	user 0.007s	sys 0.000s
I20260709 12:26:38.579948 11566 tablet_bootstrap.cc:492] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:38.696889 11571 ts_tablet_manager.cc:1434] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.202s	user 0.008s	sys 0.000s
I20260709 12:26:38.701268 11567 tablet_bootstrap.cc:492] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:38.702081 11567 ts_tablet_manager.cc:1403] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.220s	user 0.022s	sys 0.034s
I20260709 12:26:38.705731 11571 tablet_bootstrap.cc:492] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:38.787091 11986 raft_consensus.cc:493] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.796446 11828 leader_election.cc:290] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
W20260709 12:26:38.833850 11261 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:36253, user_credentials={real_user=slave}} blocked reactor thread for 68962.4us
I20260709 12:26:38.789080 11986 raft_consensus.cc:515] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.837422 11568 raft_consensus.cc:359] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.839833 11568 raft_consensus.cc:385] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.840168 11568 raft_consensus.cc:740] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.852305 11568 consensus_queue.cc:260] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:38.860142 11931 raft_consensus.cc:493] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.860903 11931 raft_consensus.cc:515] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:38.865576 11567 raft_consensus.cc:359] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:38.866417 11567 raft_consensus.cc:385] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:38.867638 11567 raft_consensus.cc:740] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:38.873428 11567 consensus_queue.cc:260] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:38.877050 11571 tablet_bootstrap.cc:654] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.882367 11579 raft_consensus.cc:493] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.882946 11579 raft_consensus.cc:515] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
W20260709 12:26:38.875939 11982 log.cc:927] Time spent T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433: Append to log took a long time: real 0.168s	user 0.000s	sys 0.001s
I20260709 12:26:38.885672 11567 ts_tablet_manager.cc:1434] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.183s	user 0.011s	sys 0.000s
I20260709 12:26:38.887763 11568 ts_tablet_manager.cc:1434] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.368s	user 0.006s	sys 0.003s
I20260709 12:26:38.899276 11579 leader_election.cc:290] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
W20260709 12:26:38.895169 11486 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 45727.4us
I20260709 12:26:38.901542 11568 tablet_bootstrap.cc:492] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:38.911091 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:38.911855 11446 raft_consensus.cc:2468] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
I20260709 12:26:38.917004 11566 tablet_bootstrap.cc:654] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:38.921624 11447 raft_consensus.cc:2468] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:38.923696 11952 raft_consensus.cc:515] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:38.916778 11336 leader_election.cc:304] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
W20260709 12:26:38.930576 11985 log.cc:927] Time spent T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1: Append to log took a long time: real 0.096s	user 0.001s	sys 0.000s
I20260709 12:26:38.933480 12001 raft_consensus.cc:493] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.934127 11370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e637809da62b408a91639b907355cad2" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:38.941754 11568 tablet_bootstrap.cc:654] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:39.004381 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bccc5d3239f3484eb991213e177b66cc" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:38.966514 11370 raft_consensus.cc:2468] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 0.
I20260709 12:26:38.994071 11988 raft_consensus.cc:493] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:38.982654 11931 leader_election.cc:290] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:39.011385 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bccc5d3239f3484eb991213e177b66cc" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:39.026510 11986 leader_election.cc:290] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:39.027907 11993 consensus_queue.cc:1048] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.062s
I20260709 12:26:39.028731 11515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e637809da62b408a91639b907355cad2" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:39.029273 11989 raft_consensus.cc:2804] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:39.029716 11989 raft_consensus.cc:493] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:39.030040 11989 raft_consensus.cc:3060] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:38.995544 12001 raft_consensus.cc:515] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:39.030853 11297 raft_consensus.cc:2468] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:39.031034 11715 raft_consensus.cc:515] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:39.031985 12001 leader_election.cc:290] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:39.032977 12002 raft_consensus.cc:493] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.033507 11715 leader_election.cc:290] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:39.033461 12002 raft_consensus.cc:515] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.035645 12002 leader_election.cc:290] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:39.048461 11261 leader_election.cc:304] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 05b93ba4435f40fb97b54343c4fdd71e
I20260709 12:26:39.052222 11952 leader_election.cc:290] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:39.054399 11996 raft_consensus.cc:2804] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:39.058809 11996 raft_consensus.cc:493] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:39.059208 11996 raft_consensus.cc:3060] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:39.059584 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:39.039654 11988 raft_consensus.cc:515] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:39.062942 11567 tablet_bootstrap.cc:492] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
I20260709 12:26:39.064623 11988 leader_election.cc:290] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:39.065869 11367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:39.067176 11566 tablet_bootstrap.cc:492] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:39.067739 11566 ts_tablet_manager.cc:1403] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.488s	user 0.024s	sys 0.008s
I20260709 12:26:39.069809 11567 tablet_bootstrap.cc:654] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:39.059912 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e637809da62b408a91639b907355cad2" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:39.073092 11373 raft_consensus.cc:3060] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:39.073958 11996 raft_consensus.cc:515] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:39.075145 11515 raft_consensus.cc:2393] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a1ce6e048ab47d69c453b5218cb64c1 in current term 1: Already voted for candidate 9968c0ed1974454d828996824f859433 in this term.
W20260709 12:26:39.077283 11261 leader_election.cc:343] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:39.078011 11931 consensus_queue.cc:1048] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:39.082049 11996 leader_election.cc:290] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:39.086728 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "610a732cba674c24bb228d619f52fce7" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:39.087252 11516 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bccc5d3239f3484eb991213e177b66cc" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:39.087904 11516 raft_consensus.cc:2468] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:39.072332 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bccc5d3239f3484eb991213e177b66cc" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:39.097091 11261 leader_election.cc:304] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
W20260709 12:26:39.097654 11484 leader_election.cc:343] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:39.107295 11566 raft_consensus.cc:359] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.108635 11566 raft_consensus.cc:385] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:39.109071 11566 raft_consensus.cc:740] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
W20260709 12:26:39.116365 11244 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:39.112756 11566 consensus_queue.cc:260] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
W20260709 12:26:39.147655 11261 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 50641.9us
I20260709 12:26:39.158079 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431c66d13d0f4904a42b33665a9c09e1" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
W20260709 12:26:39.160681 11484 leader_election.cc:343] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:39.172417 11566 ts_tablet_manager.cc:1434] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.104s	user 0.013s	sys 0.006s
W20260709 12:26:39.172154 11262 leader_election.cc:343] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:39.174742 11262 leader_election.cc:304] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:39.180209 11566 tablet_bootstrap.cc:492] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:39.185293 11294 raft_consensus.cc:3060] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:39.171388 11929 raft_consensus.cc:2804] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:39.173522 12002 raft_consensus.cc:493] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.178721 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "610a732cba674c24bb228d619f52fce7" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:39.180056 11442 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:39.188943 11929 raft_consensus.cc:493] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:39.207118 11929 raft_consensus.cc:3060] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:39.209245 11992 raft_consensus.cc:2749] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:39.232685 11571 tablet_bootstrap.cc:492] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:39.233384 11571 ts_tablet_manager.cc:1403] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.535s	user 0.026s	sys 0.025s
I20260709 12:26:39.210523 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e637809da62b408a91639b907355cad2" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:39.223083 11442 raft_consensus.cc:3060] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:39.230413 11485 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:44567, user_credentials={real_user=slave}} blocked reactor thread for 46012.7us
I20260709 12:26:39.199774 12002 raft_consensus.cc:515] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.272094 11486 leader_election.cc:304] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:39.273083 11715 raft_consensus.cc:2804] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:39.273530 11715 raft_consensus.cc:493] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:39.273877 11715 raft_consensus.cc:3060] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:26:39.281072 11412 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 83779.2us
I20260709 12:26:39.282125 11715 raft_consensus.cc:515] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:39.294394 11568 tablet_bootstrap.cc:492] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:39.295001 11568 ts_tablet_manager.cc:1403] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.394s	user 0.016s	sys 0.004s
I20260709 12:26:39.238421 11943 consensus_queue.cc:1048] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
I20260709 12:26:39.302112 11715 leader_election.cc:290] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:39.303488 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bccc5d3239f3484eb991213e177b66cc" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:39.308815 11566 tablet_bootstrap.cc:654] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:39.299489 11989 raft_consensus.cc:515] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:39.317588 12002 leader_election.cc:290] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:39.312076 11567 tablet_bootstrap.cc:492] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:39.398036 11567 ts_tablet_manager.cc:1403] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.335s	user 0.008s	sys 0.037s
I20260709 12:26:39.319847 12013 raft_consensus.cc:493] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.257714 11445 raft_consensus.cc:3060] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:39.337389 11367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:39.344358 11568 raft_consensus.cc:359] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:39.482164 11568 raft_consensus.cc:385] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:39.482611 11568 raft_consensus.cc:740] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:39.483450 11568 consensus_queue.cc:260] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:39.355196 11373 raft_consensus.cc:2468] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:39.295908 11293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:39.492249 11293 raft_consensus.cc:2393] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 05b93ba4435f40fb97b54343c4fdd71e in current term 1: Already voted for candidate 663d88d9634e4720a695c7cd69eaa25e in this term.
I20260709 12:26:39.494345 11568 ts_tablet_manager.cc:1434] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.199s	user 0.007s	sys 0.001s
I20260709 12:26:39.497737 11568 tablet_bootstrap.cc:492] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:39.498749 11929 raft_consensus.cc:515] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
W20260709 12:26:39.526849 11336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 286336us
I20260709 12:26:39.530766 11295 raft_consensus.cc:2393] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 1: Already voted for candidate 663d88d9634e4720a695c7cd69eaa25e in this term.
I20260709 12:26:39.500098 11442 raft_consensus.cc:2468] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:39.533330 11367 raft_consensus.cc:2393] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 663d88d9634e4720a695c7cd69eaa25e in current term 1: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:39.509182 12013 raft_consensus.cc:515] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.554862 11994 consensus_queue.cc:1048] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.215s
I20260709 12:26:39.563985 11516 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32c7abdd125b4f38b6398a9b0e92fedd" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:39.581921 11571 raft_consensus.cc:359] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.583336 11571 raft_consensus.cc:385] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:39.583684 11571 raft_consensus.cc:740] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:39.585851 11571 consensus_queue.cc:260] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.584654 11929 leader_election.cc:290] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:39.588371 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431c66d13d0f4904a42b33665a9c09e1" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:39.588596 11567 raft_consensus.cc:359] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:39.589164 11371 raft_consensus.cc:2468] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:39.584292 11989 leader_election.cc:290] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
W20260709 12:26:39.595217 11338 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 297117us
I20260709 12:26:39.592816 11987 consensus_queue.cc:1048] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260709 12:26:39.599745 11485 leader_election.cc:304] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:39.589334 11567 raft_consensus.cc:385] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:39.602519 11567 raft_consensus.cc:740] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:39.603009 11990 raft_consensus.cc:2804] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:39.603570 11990 raft_consensus.cc:697] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:39.603372 11567 consensus_queue.cc:260] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:39.604485 11990 consensus_queue.cc:237] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:39.605775 11567 ts_tablet_manager.cc:1434] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.207s	user 0.005s	sys 0.000s
I20260709 12:26:39.607779 11520 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bccc5d3239f3484eb991213e177b66cc" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:39.577979 11516 raft_consensus.cc:2468] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:39.608596 11520 raft_consensus.cc:2393] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 663d88d9634e4720a695c7cd69eaa25e in current term 1: Already voted for candidate 9968c0ed1974454d828996824f859433 in this term.
I20260709 12:26:39.609709 12013 leader_election.cc:290] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:39.610460 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bccc5d3239f3484eb991213e177b66cc" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:39.613056 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "359eb877601e4d519d6e7d6f3835b9a2" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:39.616217 11447 raft_consensus.cc:3060] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:39.617448 11261 leader_election.cc:304] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:39.619012 12002 raft_consensus.cc:2804] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:39.619407 12002 raft_consensus.cc:493] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:39.619760 12002 raft_consensus.cc:3060] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:39.623333 11445 raft_consensus.cc:2468] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:39.634239 11571 ts_tablet_manager.cc:1434] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.401s	user 0.007s	sys 0.001s
I20260709 12:26:39.636902 11571 tablet_bootstrap.cc:492] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
W20260709 12:26:39.635430 11261 leader_election.cc:343] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:39.639528 11261 leader_election.cc:304] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:39.648643 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bccc5d3239f3484eb991213e177b66cc" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
W20260709 12:26:39.654882 11486 leader_election.cc:343] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:39.676395 11262 leader_election.cc:304] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433
I20260709 12:26:39.677577 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:39.678478 11444 raft_consensus.cc:2393] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 05b93ba4435f40fb97b54343c4fdd71e in current term 1: Already voted for candidate 663d88d9634e4720a695c7cd69eaa25e in this term.
I20260709 12:26:39.684594 11261 leader_election.cc:304] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433
I20260709 12:26:39.663396 11571 tablet_bootstrap.cc:654] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:39.688148 11567 tablet_bootstrap.cc:492] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433: Bootstrap starting.
W20260709 12:26:39.638621 12022 log.cc:927] Time spent T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e: Append to log took a long time: real 0.064s	user 0.001s	sys 0.000s
I20260709 12:26:39.647922 11996 raft_consensus.cc:2804] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:39.703778 11368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32c7abdd125b4f38b6398a9b0e92fedd" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:39.707237 11368 raft_consensus.cc:2393] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 663d88d9634e4720a695c7cd69eaa25e in current term 1: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:39.695081 11994 consensus_queue.cc:1048] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260709 12:26:39.711436 11485 leader_election.cc:304] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:39.698288 11929 raft_consensus.cc:2749] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:39.714036 12013 raft_consensus.cc:2804] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:39.695226 11992 raft_consensus.cc:2749] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:26:39.733317 11568 tablet_bootstrap.cc:654] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
W20260709 12:26:39.735337 11338 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 139115us
I20260709 12:26:39.666509 12002 raft_consensus.cc:515] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.741890 11411 leader_election.cc:304] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:39.714478 12013 raft_consensus.cc:493] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260709 12:26:39.751927 11262 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:44567, user_credentials={real_user=slave}} blocked reactor thread for 57746.4us
I20260709 12:26:39.742826 11827 raft_consensus.cc:2764] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260709 12:26:39.748859 12013 raft_consensus.cc:3060] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:39.745424 12020 consensus_queue.cc:1048] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.015s
I20260709 12:26:39.773841 11566 tablet_bootstrap.cc:492] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:39.774430 11566 ts_tablet_manager.cc:1403] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.600s	user 0.017s	sys 0.018s
I20260709 12:26:39.776224 12002 leader_election.cc:290] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:39.766381 11367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "359eb877601e4d519d6e7d6f3835b9a2" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:39.749928 11987 consensus_queue.cc:1048] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260709 12:26:39.799319 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32c7abdd125b4f38b6398a9b0e92fedd" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:39.815980 11521 raft_consensus.cc:3060] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:39.817200 11567 tablet_bootstrap.cc:654] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:39.759909 11996 raft_consensus.cc:697] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:39.834359 11931 raft_consensus.cc:493] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.835355 11931 raft_consensus.cc:515] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:39.840418 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f75f882db94790be4eaf2994c44f61" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:39.822455 11367 raft_consensus.cc:2468] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
W20260709 12:26:39.843547 11484 leader_election.cc:343] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:39.845005 11566 raft_consensus.cc:359] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.845862 11566 raft_consensus.cc:385] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:39.846207 11566 raft_consensus.cc:740] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:39.857771 11566 consensus_queue.cc:260] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.844018 11447 raft_consensus.cc:2468] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:39.869859 11484 leader_election.cc:304] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:39.873308 11566 ts_tablet_manager.cc:1434] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.094s	user 0.008s	sys 0.000s
I20260709 12:26:39.878319 11703 raft_consensus.cc:493] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.912014 11568 tablet_bootstrap.cc:492] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:39.912865 11568 ts_tablet_manager.cc:1403] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.415s	user 0.016s	sys 0.018s
I20260709 12:26:39.892642 11931 leader_election.cc:290] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:39.919515 12027 raft_consensus.cc:493] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:39.920138 12027 raft_consensus.cc:515] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.915508 11703 raft_consensus.cc:515] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.922791 11991 raft_consensus.cc:2804] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:39.923617 11991 raft_consensus.cc:697] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:39.924676 11703 leader_election.cc:290] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:39.924800 11991 consensus_queue.cc:237] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:39.925922 11370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff00b27947404e52acea283e9381d837" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:39.909922 11992 raft_consensus.cc:493] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5a1ce6e048ab47d69c453b5218cb64c1)
I20260709 12:26:39.926826 11992 raft_consensus.cc:515] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:39.928359 11571 tablet_bootstrap.cc:492] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:39.928936 11571 ts_tablet_manager.cc:1403] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.292s	user 0.018s	sys 0.003s
I20260709 12:26:39.884867 11485 leader_election.cc:304] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:39.930440 11992 leader_election.cc:290] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:39.931416 11999 raft_consensus.cc:2804] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:39.884650 11996 consensus_queue.cc:237] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:39.931921 11999 raft_consensus.cc:493] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:39.932376 11367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8236893668af4eaa93bc53b6715f6cf0" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:39.930569 11445 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8236893668af4eaa93bc53b6715f6cf0" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
W20260709 12:26:39.931598 11485 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:44567, user_credentials={real_user=slave}} blocked reactor thread for 50404.9us
I20260709 12:26:39.932405 11999 raft_consensus.cc:3060] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:39.936280 11516 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff00b27947404e52acea283e9381d837" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:39.892083 11515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32c7abdd125b4f38b6398a9b0e92fedd" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
W20260709 12:26:39.938313 11261 leader_election.cc:343] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9968c0ed1974454d828996824f859433 (127.29.201.196:34569): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:39.945300 11261 leader_election.cc:304] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:39.943384 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f75f882db94790be4eaf2994c44f61" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:39.938651 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32c7abdd125b4f38b6398a9b0e92fedd" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:39.953955 11567 tablet_bootstrap.cc:492] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433: No bootstrap required, opened a new log
I20260709 12:26:39.955165 11567 ts_tablet_manager.cc:1403] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433: Time spent bootstrapping tablet: real 0.271s	user 0.013s	sys 0.008s
I20260709 12:26:39.956391 11931 raft_consensus.cc:493] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 05b93ba4435f40fb97b54343c4fdd71e)
W20260709 12:26:39.957710 11262 leader_election.cc:343] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:39.958338 11486 leader_election.cc:343] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:39.958666 11262 leader_election.cc:304] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433
I20260709 12:26:39.958741 11486 leader_election.cc:304] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:39.959717 11703 raft_consensus.cc:2749] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:39.960564 11916 raft_consensus.cc:2749] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:39.961064 11372 raft_consensus.cc:2393] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 663d88d9634e4720a695c7cd69eaa25e in current term 1: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:39.957034 11931 raft_consensus.cc:515] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:39.950433 11521 raft_consensus.cc:2468] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
W20260709 12:26:39.954262 11410 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 74826.6us
I20260709 12:26:39.963129 11931 leader_election.cc:290] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:39.965349 11567 raft_consensus.cc:359] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.966185 11567 raft_consensus.cc:385] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:39.966504 11567 raft_consensus.cc:740] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Initialized, Role: FOLLOWER
I20260709 12:26:39.967269 11567 consensus_queue.cc:260] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.968309 11566 tablet_bootstrap.cc:492] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
W20260709 12:26:39.969374 12025 log.cc:927] Time spent T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433: Append to log took a long time: real 0.139s	user 0.000s	sys 0.001s
I20260709 12:26:39.970748 11567 ts_tablet_manager.cc:1434] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433: Time spent starting tablet: real 0.015s	user 0.007s	sys 0.000s
I20260709 12:26:39.971515 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "393b5b835714456ab06eeecb9b4a116d" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:39.973529 11336 leader_election.cc:304] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:39.952760 11992 raft_consensus.cc:2749] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:39.983151 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "393b5b835714456ab06eeecb9b4a116d" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:39.986340 11262 leader_election.cc:304] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433
I20260709 12:26:39.988271 11571 raft_consensus.cc:359] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.989140 11571 raft_consensus.cc:385] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:39.989418 11571 raft_consensus.cc:740] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:39.998974 11989 raft_consensus.cc:2804] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:40.001157 11989 raft_consensus.cc:697] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:40.001624 11999 raft_consensus.cc:515] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:40.002080 11989 consensus_queue.cc:237] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:39.999339 12029 raft_consensus.cc:2749] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:26:40.005153 11999 leader_election.cc:290] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:40.007819 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "359eb877601e4d519d6e7d6f3835b9a2" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
W20260709 12:26:40.012362 11486 leader_election.cc:343] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:40.000396 11485 leader_election.cc:304] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:40.021456 11571 consensus_queue.cc:260] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:40.022560 11931 raft_consensus.cc:2749] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:39.957683 12013 raft_consensus.cc:515] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:40.066680 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32c7abdd125b4f38b6398a9b0e92fedd" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:40.067610 11297 raft_consensus.cc:2393] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 05b93ba4435f40fb97b54343c4fdd71e in current term 1: Already voted for candidate 663d88d9634e4720a695c7cd69eaa25e in this term.
I20260709 12:26:40.069818 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:40.083237 12027 leader_election.cc:290] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:40.090060 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "359eb877601e4d519d6e7d6f3835b9a2" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:40.092231 12013 leader_election.cc:290] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:40.093451 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431c66d13d0f4904a42b33665a9c09e1" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:40.107993 11568 raft_consensus.cc:359] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:40.108937 11568 raft_consensus.cc:385] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:40.109248 11568 raft_consensus.cc:740] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:40.110524 11521 raft_consensus.cc:2393] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 05b93ba4435f40fb97b54343c4fdd71e in current term 1: Already voted for candidate 9968c0ed1974454d828996824f859433 in this term.
I20260709 12:26:40.110064 11568 consensus_queue.cc:260] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:40.112703 11568 ts_tablet_manager.cc:1434] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.199s	user 0.007s	sys 0.000s
I20260709 12:26:40.115659 11568 tablet_bootstrap.cc:492] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:40.116808 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "359eb877601e4d519d6e7d6f3835b9a2" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:40.117734 11373 raft_consensus.cc:3060] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:40.126428 11447 raft_consensus.cc:3060] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:40.135389 11571 ts_tablet_manager.cc:1434] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.206s	user 0.008s	sys 0.002s
I20260709 12:26:40.138844 11566 tablet_bootstrap.cc:654] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:40.139976 11571 tablet_bootstrap.cc:492] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e: Bootstrap starting.
I20260709 12:26:40.151144 11293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "359eb877601e4d519d6e7d6f3835b9a2" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:40.158303 11447 raft_consensus.cc:2468] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:40.198529 11373 raft_consensus.cc:2468] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:40.204020 11338 leader_election.cc:304] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:40.205180 11635 raft_consensus.cc:2749] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:26:40.213470 11484 leader_election.cc:304] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:40.216914 11931 raft_consensus.cc:2804] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:40.215914 11990 consensus_queue.cc:1048] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20260709 12:26:40.244165 11931 raft_consensus.cc:697] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:40.255589 11568 tablet_bootstrap.cc:654] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:40.246492 11931 consensus_queue.cc:237] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:40.263024 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431c66d13d0f4904a42b33665a9c09e1" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:40.264427 11373 raft_consensus.cc:3060] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:40.261426 11988 raft_consensus.cc:493] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.301222 11988 raft_consensus.cc:515] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:40.323345 11373 raft_consensus.cc:2468] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:40.330592 11293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff00b27947404e52acea283e9381d837" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:40.331511 11293 raft_consensus.cc:2468] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 0.
I20260709 12:26:40.335796 11485 leader_election.cc:304] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:40.337308 11997 raft_consensus.cc:2804] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:40.339442 11997 raft_consensus.cc:697] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
W20260709 12:26:40.347550 11412 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 67820.3us
I20260709 12:26:40.351636 11486 leader_election.cc:304] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:40.356066 11571 tablet_bootstrap.cc:654] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:40.367009 12013 raft_consensus.cc:2804] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260709 12:26:40.368950 11410 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 165324us
I20260709 12:26:40.388690 12013 raft_consensus.cc:493] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:40.405814 12013 raft_consensus.cc:3060] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:40.420645 11566 tablet_bootstrap.cc:492] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:40.440056 11566 ts_tablet_manager.cc:1403] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.472s	user 0.017s	sys 0.010s
I20260709 12:26:40.424820 11988 leader_election.cc:290] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:40.398725 11997 consensus_queue.cc:237] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
W20260709 12:26:40.430514 11338 leader_election.cc:343] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:40.455471 11338 leader_election.cc:304] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:40.432291 11635 raft_consensus.cc:493] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.457324 11635 raft_consensus.cc:515] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:40.463534 12020 raft_consensus.cc:2749] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:26:40.483433 11568 tablet_bootstrap.cc:492] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:40.484095 11568 ts_tablet_manager.cc:1403] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.369s	user 0.019s	sys 0.006s
I20260709 12:26:40.492169 11635 leader_election.cc:290] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:40.493255 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff00b27947404e52acea283e9381d837" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
W20260709 12:26:40.492208 11485 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:44567, user_credentials={real_user=slave}} blocked reactor thread for 59404.3us
I20260709 12:26:40.506500 12009 consensus_queue.cc:1048] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:40.506875 11566 raft_consensus.cc:359] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:40.507699 11566 raft_consensus.cc:385] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:40.507982 11566 raft_consensus.cc:740] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:40.508651 11566 consensus_queue.cc:260] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:40.510221 11566 ts_tablet_manager.cc:1434] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.070s	user 0.005s	sys 0.001s
I20260709 12:26:40.515491 11566 tablet_bootstrap.cc:492] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:40.517186 11568 raft_consensus.cc:359] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:40.520334 11568 raft_consensus.cc:385] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:40.520731 11568 raft_consensus.cc:740] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
W20260709 12:26:40.523424 11261 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 113078us
I20260709 12:26:40.526145 11298 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff00b27947404e52acea283e9381d837" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:40.526906 11298 raft_consensus.cc:2468] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 0.
W20260709 12:26:40.542994 11244 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:26:40.549450 11338 leader_election.cc:304] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:40.556741 11568 consensus_queue.cc:260] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:40.555254 11817 raft_consensus.cc:2804] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:40.566092 11817 raft_consensus.cc:493] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:40.566588 11817 raft_consensus.cc:3060] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:40.562067 12058 consensus_queue.cc:1048] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260709 12:26:40.560618 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff00b27947404e52acea283e9381d837" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:40.583124 12060 consensus_queue.cc:1048] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260709 12:26:40.590920 12013 raft_consensus.cc:515] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:40.604686 11571 tablet_bootstrap.cc:492] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e: No bootstrap required, opened a new log
I20260709 12:26:40.605453 11571 ts_tablet_manager.cc:1403] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e: Time spent bootstrapping tablet: real 0.469s	user 0.028s	sys 0.013s
I20260709 12:26:40.605532 11521 raft_consensus.cc:2393] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 05b93ba4435f40fb97b54343c4fdd71e in current term 1: Already voted for candidate 9968c0ed1974454d828996824f859433 in this term.
I20260709 12:26:40.609781 11568 ts_tablet_manager.cc:1434] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.125s	user 0.007s	sys 0.001s
I20260709 12:26:40.611300 11568 tablet_bootstrap.cc:492] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1: Bootstrap starting.
I20260709 12:26:40.616551 11571 raft_consensus.cc:359] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:40.617780 11571 raft_consensus.cc:385] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:40.618191 11571 raft_consensus.cc:740] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Initialized, Role: FOLLOWER
I20260709 12:26:40.624307 12031 raft_consensus.cc:493] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.625864 12029 consensus_queue.cc:1048] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:40.630702 11997 consensus_queue.cc:1048] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:40.631399 12031 raft_consensus.cc:515] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:40.632417 12028 consensus_queue.cc:1048] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:40.633275 12013 leader_election.cc:290] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:40.633936 12031 leader_election.cc:290] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:40.635612 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff00b27947404e52acea283e9381d837" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:40.642567 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f75f882db94790be4eaf2994c44f61" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:40.643303 11446 raft_consensus.cc:2468] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:40.644098 11297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff00b27947404e52acea283e9381d837" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:40.645138 11297 raft_consensus.cc:3060] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:40.647619 12028 consensus_queue.cc:1048] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:40.649595 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f75f882db94790be4eaf2994c44f61" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:40.650399 11521 raft_consensus.cc:2468] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:40.628791 11566 tablet_bootstrap.cc:654] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:40.644377 11261 leader_election.cc:304] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:40.668363 11571 consensus_queue.cc:260] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:40.673210 11297 raft_consensus.cc:2468] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:40.675357 11486 leader_election.cc:304] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e
W20260709 12:26:40.687220 12062 log.cc:927] Time spent T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433: Append to log took a long time: real 0.072s	user 0.000s	sys 0.001s
I20260709 12:26:40.680580 12013 raft_consensus.cc:2804] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:40.691128 12013 raft_consensus.cc:697] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:40.692183 12013 consensus_queue.cc:237] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:40.695340 11817 raft_consensus.cc:515] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:40.673460 11703 raft_consensus.cc:2804] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:40.704964 11571 ts_tablet_manager.cc:1434] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e: Time spent starting tablet: real 0.099s	user 0.006s	sys 0.002s
I20260709 12:26:40.710350 11817 leader_election.cc:290] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:40.700713 11703 raft_consensus.cc:493] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:40.723948 11703 raft_consensus.cc:3060] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:40.699144 11517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff00b27947404e52acea283e9381d837" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:40.730274 11568 tablet_bootstrap.cc:654] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1: Neither blocks nor log segments found. Creating new log.
I20260709 12:26:40.741879 11295 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff00b27947404e52acea283e9381d837" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:40.737018 11372 raft_consensus.cc:2393] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 1: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:40.745828 11295 raft_consensus.cc:2393] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 05b93ba4435f40fb97b54343c4fdd71e in current term 1: Already voted for candidate 9968c0ed1974454d828996824f859433 in this term.
I20260709 12:26:40.755873 11338 leader_election.cc:304] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:40.782684 12028 raft_consensus.cc:2749] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:26:40.791157 11715 consensus_queue.cc:1048] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:40.828625 11929 raft_consensus.cc:493] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:26:40.829229 11929 raft_consensus.cc:515] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:40.831681 11568 tablet_bootstrap.cc:492] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1: No bootstrap required, opened a new log
I20260709 12:26:40.832372 11568 ts_tablet_manager.cc:1403] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent bootstrapping tablet: real 0.221s	user 0.019s	sys 0.000s
I20260709 12:26:40.831631 11929 leader_election.cc:290] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:40.834326 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:40.853363 11568 raft_consensus.cc:359] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:40.854204 11568 raft_consensus.cc:385] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:40.854503 11568 raft_consensus.cc:740] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Initialized, Role: FOLLOWER
I20260709 12:26:40.863049 11194 catalog_manager.cc:5697] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 1 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:40.865087 11194 catalog_manager.cc:5697] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } }
I20260709 12:26:40.877493 11568 consensus_queue.cc:260] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:40.879251 11446 raft_consensus.cc:2468] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:40.881009 11568 ts_tablet_manager.cc:1434] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1: Time spent starting tablet: real 0.048s	user 0.009s	sys 0.000s
I20260709 12:26:40.881078 11261 leader_election.cc:304] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:40.883697 11566 tablet_bootstrap.cc:492] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e: No bootstrap required, opened a new log
I20260709 12:26:40.912258 11566 ts_tablet_manager.cc:1403] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.399s	user 0.018s	sys 0.003s
I20260709 12:26:40.913582 12058 raft_consensus.cc:2804] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:26:40.914072 12058 raft_consensus.cc:493] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:26:40.914419 12058 raft_consensus.cc:3060] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:40.915414 11566 raft_consensus.cc:359] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:40.916393 11566 raft_consensus.cc:385] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:26:40.916862 11566 raft_consensus.cc:740] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:40.917762 11566 consensus_queue.cc:260] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:40.886065 11703 raft_consensus.cc:515] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:40.906864 11367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:40.885825 11715 consensus_queue.cc:1048] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:40.919063 11367 raft_consensus.cc:2468] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 0.
I20260709 12:26:40.921090 11566 ts_tablet_manager.cc:1434] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.003s
I20260709 12:26:40.925209 12060 consensus_queue.cc:1048] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:40.934267 11443 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f75f882db94790be4eaf2994c44f61" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:40.935068 11443 raft_consensus.cc:3060] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:40.936256 11703 leader_election.cc:290] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:40.946514 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f75f882db94790be4eaf2994c44f61" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:40.949250 11518 raft_consensus.cc:3060] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:40.975612 12058 raft_consensus.cc:515] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:40.977665 11443 raft_consensus.cc:2468] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:40.996256 11295 raft_consensus.cc:3060] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:41.000741 11261 leader_election.cc:304] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:41.001668 11996 raft_consensus.cc:2804] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:41.002274 11996 raft_consensus.cc:697] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:41.002790 11518 raft_consensus.cc:2468] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:41.003252 11996 consensus_queue.cc:237] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:41.024148 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:41.025712 11372 raft_consensus.cc:3060] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:41.038425 12058 leader_election.cc:290] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:41.039669 11443 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:41.040793 11443 raft_consensus.cc:3060] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:26:41.040922 11372 raft_consensus.cc:2468] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:41.042459 11262 leader_election.cc:304] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:41.046130 12058 raft_consensus.cc:2804] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:26:41.047006 12058 raft_consensus.cc:697] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:41.048023 12009 consensus_queue.cc:1048] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:41.048123 12058 consensus_queue.cc:237] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:41.050375 11443 raft_consensus.cc:2468] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 1.
I20260709 12:26:41.069802 11191 catalog_manager.cc:5697] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } }
I20260709 12:26:41.075238 11191 catalog_manager.cc:5697] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 1 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } }
I20260709 12:26:41.119691 11194 catalog_manager.cc:5697] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } }
I20260709 12:26:41.122255 11194 catalog_manager.cc:5697] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } }
I20260709 12:26:41.165211 11191 catalog_manager.cc:5697] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 1 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:41.167096 11191 catalog_manager.cc:5697] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 0 to 1, leader changed from <none> to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 1 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:41.209619 11195 catalog_manager.cc:5697] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 1 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:41.308554 12060 consensus_queue.cc:1048] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:41.315882 12060 consensus_queue.cc:1048] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:41.495056 11703 consensus_queue.cc:1048] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:41.509202 11996 consensus_queue.cc:1048] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:26:41.679888 12058 consensus_queue.cc:1048] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20260709 12:26:41.773478 11410 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 72703us
I20260709 12:26:41.793880 12061 consensus_queue.cc:1048] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:26:41.805831 30503 tablet_server.cc:179] TabletServer@127.29.201.193:0 shutting down...
W20260709 12:26:41.806352 11336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 46968us
W20260709 12:26:41.894373 11338 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 1 similar messages]
W20260709 12:26:42.014999 11338 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 182046us
W20260709 12:26:42.029206 11412 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 131000us
W20260709 12:26:42.092115 11486 consensus_peers.cc:597] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.108603 11486 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 48594.1us
W20260709 12:26:42.114804 11338 consensus_peers.cc:597] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.121812 11486 consensus_peers.cc:597] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.121347 11338 consensus_peers.cc:597] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.133335 11338 consensus_peers.cc:597] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.139629 11338 consensus_peers.cc:597] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.145536 11412 consensus_peers.cc:597] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.153141 11338 consensus_peers.cc:597] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.157351 11338 consensus_peers.cc:597] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.167129 11338 consensus_peers.cc:597] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.191783 11486 consensus_peers.cc:597] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.195421 11338 consensus_peers.cc:597] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.235195 11338 consensus_peers.cc:597] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.241772 11412 consensus_peers.cc:597] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.245332 11412 consensus_peers.cc:597] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.246436 11412 consensus_peers.cc:597] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.247219 11412 consensus_peers.cc:597] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.247852 11412 consensus_peers.cc:597] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.248466 11412 consensus_peers.cc:597] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.249044 11412 consensus_peers.cc:597] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.249214 11486 consensus_peers.cc:597] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.249697 11412 consensus_peers.cc:597] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.249982 11486 consensus_peers.cc:597] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.250348 11412 consensus_peers.cc:597] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.254181 11412 consensus_peers.cc:597] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.255115 11412 consensus_peers.cc:597] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.260252 11486 consensus_peers.cc:597] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.265466 11486 consensus_peers.cc:597] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.266093 11486 consensus_peers.cc:597] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.266677 11486 consensus_peers.cc:597] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.267226 11486 consensus_peers.cc:597] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:42.273335 11486 consensus_peers.cc:597] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:42.280788 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:42.282215 30503 tablet_replica.cc:333] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.284003 30503 raft_consensus.cc:2243] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.284884 30503 raft_consensus.cc:2272] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.289064 30503 tablet_replica.cc:333] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.290109 30503 raft_consensus.cc:2243] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.290925 30503 raft_consensus.cc:2272] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.295728 30503 tablet_replica.cc:333] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.296854 30503 raft_consensus.cc:2243] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.297621 30503 raft_consensus.cc:2272] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.301355 30503 tablet_replica.cc:333] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.302101 30503 raft_consensus.cc:2243] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:42.303144 30503 raft_consensus.cc:2272] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.305958 30503 tablet_replica.cc:333] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.306694 30503 raft_consensus.cc:2243] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:42.307607 30503 raft_consensus.cc:2272] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.309933 30503 tablet_replica.cc:333] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.310592 30503 raft_consensus.cc:2243] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:42.311602 30503 raft_consensus.cc:2272] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.314728 30503 tablet_replica.cc:333] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.315488 30503 raft_consensus.cc:2243] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:42.316497 30503 raft_consensus.cc:2272] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.318919 30503 tablet_replica.cc:333] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.319594 30503 raft_consensus.cc:2243] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:42.320518 30503 raft_consensus.cc:2272] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.323122 30503 tablet_replica.cc:333] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.323876 30503 raft_consensus.cc:2243] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.324568 30503 raft_consensus.cc:2272] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.330442 30503 tablet_replica.cc:333] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.331480 30503 raft_consensus.cc:2243] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:42.333207 30503 raft_consensus.cc:2272] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.336894 30503 tablet_replica.cc:333] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.338063 30503 raft_consensus.cc:2243] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.338857 30503 raft_consensus.cc:2272] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.342774 30503 tablet_replica.cc:333] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.343995 30503 raft_consensus.cc:2243] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:42.345175 30503 raft_consensus.cc:2272] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.349561 30503 tablet_replica.cc:333] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.350428 30503 raft_consensus.cc:2243] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.351240 30503 raft_consensus.cc:2272] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.353919 30503 tablet_replica.cc:333] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.354751 30503 raft_consensus.cc:2243] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:42.355836 30503 raft_consensus.cc:2272] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.359236 30503 tablet_replica.cc:333] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.360147 30503 raft_consensus.cc:2243] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.360957 30503 raft_consensus.cc:2272] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.364213 30503 tablet_replica.cc:333] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.365044 30503 raft_consensus.cc:2243] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.365792 30503 raft_consensus.cc:2272] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.369778 30503 tablet_replica.cc:333] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.370677 30503 raft_consensus.cc:2243] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.371312 30503 raft_consensus.cc:2272] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.374724 30503 tablet_replica.cc:333] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.375397 30503 raft_consensus.cc:2243] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.376158 30503 raft_consensus.cc:2272] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.378978 30503 tablet_replica.cc:333] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.379757 30503 raft_consensus.cc:2243] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:42.380745 30503 raft_consensus.cc:2272] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.383158 30503 tablet_replica.cc:333] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.383874 30503 raft_consensus.cc:2243] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.384425 30503 raft_consensus.cc:2272] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.386961 30503 tablet_replica.cc:333] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.387583 30503 raft_consensus.cc:2243] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.388064 30503 raft_consensus.cc:2272] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.390332 30503 tablet_replica.cc:333] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.391068 30503 raft_consensus.cc:2243] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.391577 30503 raft_consensus.cc:2272] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.394063 30503 tablet_replica.cc:333] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.394927 30503 raft_consensus.cc:2243] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.395568 30503 raft_consensus.cc:2272] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.398936 30503 tablet_replica.cc:333] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.399897 30503 raft_consensus.cc:2243] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:42.401643 30503 raft_consensus.cc:2272] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.406071 30503 tablet_replica.cc:333] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.407150 30503 raft_consensus.cc:2243] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.407827 30503 raft_consensus.cc:2272] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.411326 30503 tablet_replica.cc:333] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.412335 30503 raft_consensus.cc:2243] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.413025 30503 raft_consensus.cc:2272] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.417268 30503 tablet_replica.cc:333] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.418128 30503 raft_consensus.cc:2243] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.418825 30503 raft_consensus.cc:2272] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.421878 30503 tablet_replica.cc:333] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.422698 30503 raft_consensus.cc:2243] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.423396 30503 raft_consensus.cc:2272] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.426960 30503 tablet_replica.cc:333] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.427810 30503 raft_consensus.cc:2243] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.428496 30503 raft_consensus.cc:2272] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.431941 30503 tablet_replica.cc:333] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.432731 30503 raft_consensus.cc:2243] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.433511 30503 raft_consensus.cc:2272] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.436636 30503 tablet_replica.cc:333] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.437659 30503 raft_consensus.cc:2243] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.438401 30503 raft_consensus.cc:2272] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.441886 30503 tablet_replica.cc:333] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.442487 30503 raft_consensus.cc:2243] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.443176 30503 raft_consensus.cc:2272] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.445747 30503 tablet_replica.cc:333] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.446412 30503 raft_consensus.cc:2243] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.446962 30503 raft_consensus.cc:2272] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.449136 30503 tablet_replica.cc:333] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.449733 30503 raft_consensus.cc:2243] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.450218 30503 raft_consensus.cc:2272] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.452464 30503 tablet_replica.cc:333] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.453048 30503 raft_consensus.cc:2243] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.453521 30503 raft_consensus.cc:2272] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.455821 30503 tablet_replica.cc:333] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.456386 30503 raft_consensus.cc:2243] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.456887 30503 raft_consensus.cc:2272] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.459221 30503 tablet_replica.cc:333] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.459822 30503 raft_consensus.cc:2243] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:42.460686 30503 raft_consensus.cc:2272] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.462916 30503 tablet_replica.cc:333] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.463471 30503 raft_consensus.cc:2243] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:42.464700 30503 raft_consensus.cc:2272] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.467695 30503 tablet_replica.cc:333] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.468515 30503 raft_consensus.cc:2243] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:42.469695 30503 raft_consensus.cc:2272] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.472316 30503 tablet_replica.cc:333] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.473155 30503 raft_consensus.cc:2243] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.473811 30503 raft_consensus.cc:2272] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.477361 30503 tablet_replica.cc:333] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.478132 30503 raft_consensus.cc:2243] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.478832 30503 raft_consensus.cc:2272] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.482091 30503 tablet_replica.cc:333] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.482892 30503 raft_consensus.cc:2243] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.483616 30503 raft_consensus.cc:2272] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.486938 30503 tablet_replica.cc:333] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.487666 30503 raft_consensus.cc:2243] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.488286 30503 raft_consensus.cc:2272] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.492033 30503 tablet_replica.cc:333] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.492928 30503 raft_consensus.cc:2243] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.493630 30503 raft_consensus.cc:2272] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.496908 30503 tablet_replica.cc:333] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e: stopping tablet replica
I20260709 12:26:42.497601 30503 raft_consensus.cc:2243] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:42.498214 30503 raft_consensus.cc:2272] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:42.591061 30503 tablet_server.cc:196] TabletServer@127.29.201.193:0 shutdown complete.
I20260709 12:26:42.716190 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:26:42.730139 12112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:42.732595 12113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:26:42.738163 12115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:26:42.739708 30503 server_base.cc:1061] running on GCE node
I20260709 12:26:42.744940 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:26:42.745287 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:26:42.745524 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600002745501 us; error 0 us; skew 500 ppm
I20260709 12:26:42.746037 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:26:42.749948 30503 webserver.cc:533] Webserver started at http://127.29.201.193:35155/ using document root <none> and password file <none>
I20260709 12:26:42.750728 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:26:42.751353 30503 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:26:42.756786 30503 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.004s	sys 0.001s
I20260709 12:26:42.765745 12120 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:26:42.766863 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:26:42.767266 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-0-root
uuid: "663d88d9634e4720a695c7cd69eaa25e"
format_stamp: "Formatted at 2026-07-09 12:26:28 on dist-test-slave-qgf5"
I20260709 12:26:42.767834 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.RestartTserver.1783599553313367-30503-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:26:42.788331 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:26:42.789745 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:26:42.791384 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:26:42.797371 12127 ts_tablet_manager.cc:548] Loading tablet metadata (0/45 complete)
I20260709 12:26:43.092703 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (45 total tablets, 45 live tablets)
I20260709 12:26:43.093006 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.297s	user 0.010s	sys 0.001s
I20260709 12:26:43.093261 30503 ts_tablet_manager.cc:600] Registering tablets (0/45 complete)
I20260709 12:26:43.099434 12127 tablet_bootstrap.cc:492] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.114413 12127 tablet_bootstrap.cc:492] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.115571 12127 tablet_bootstrap.cc:492] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.116276 12127 ts_tablet_manager.cc:1403] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.017s	user 0.015s	sys 0.000s
I20260709 12:26:43.120220 12127 raft_consensus.cc:359] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.120936 12127 raft_consensus.cc:740] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.121739 12127 consensus_queue.cc:260] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.128305 12127 ts_tablet_manager.cc:1434] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.004s
I20260709 12:26:43.129068 12127 tablet_bootstrap.cc:492] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.146097 12127 tablet_bootstrap.cc:492] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.147148 12127 tablet_bootstrap.cc:492] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.147833 12127 ts_tablet_manager.cc:1403] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.019s	user 0.015s	sys 0.000s
I20260709 12:26:43.150394 12127 raft_consensus.cc:359] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.151120 12127 raft_consensus.cc:740] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.151954 12127 consensus_queue.cc:260] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.153770 12127 ts_tablet_manager.cc:1434] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:26:43.154594 12127 tablet_bootstrap.cc:492] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.170167 12127 tablet_bootstrap.cc:492] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.171321 12127 tablet_bootstrap.cc:492] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.172008 12127 ts_tablet_manager.cc:1403] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.018s	user 0.011s	sys 0.004s
I20260709 12:26:43.174170 12127 raft_consensus.cc:359] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.174758 12127 raft_consensus.cc:740] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.175423 12127 consensus_queue.cc:260] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.176939 12127 ts_tablet_manager.cc:1434] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:26:43.177626 12127 tablet_bootstrap.cc:492] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.190943 12127 tablet_bootstrap.cc:492] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.191960 12127 tablet_bootstrap.cc:492] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.192623 12127 ts_tablet_manager.cc:1403] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.015s	user 0.012s	sys 0.000s
I20260709 12:26:43.195196 12127 raft_consensus.cc:359] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.195890 12127 raft_consensus.cc:740] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.196657 12127 consensus_queue.cc:260] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.198586 12127 ts_tablet_manager.cc:1434] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:26:43.199445 12127 tablet_bootstrap.cc:492] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.225813 12127 tablet_bootstrap.cc:492] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.226975 12127 tablet_bootstrap.cc:492] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.230945 12127 ts_tablet_manager.cc:1403] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.032s	user 0.016s	sys 0.011s
I20260709 12:26:43.233803 12127 raft_consensus.cc:359] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.234570 12127 raft_consensus.cc:740] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.235551 12127 consensus_queue.cc:260] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.237509 12127 ts_tablet_manager.cc:1434] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:26:43.238384 12127 tablet_bootstrap.cc:492] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.255898 12127 tablet_bootstrap.cc:492] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.256743 12127 tablet_bootstrap.cc:492] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.257311 12127 ts_tablet_manager.cc:1403] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.019s	user 0.014s	sys 0.000s
I20260709 12:26:43.259464 12127 raft_consensus.cc:359] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.260152 12127 raft_consensus.cc:740] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.260854 12127 consensus_queue.cc:260] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.262724 12127 ts_tablet_manager.cc:1434] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 12:26:43.263423 12127 tablet_bootstrap.cc:492] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.276338 12127 tablet_bootstrap.cc:492] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.277125 12127 tablet_bootstrap.cc:492] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.277673 12127 ts_tablet_manager.cc:1403] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.014s	user 0.011s	sys 0.000s
I20260709 12:26:43.279623 12127 raft_consensus.cc:359] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.280158 12127 raft_consensus.cc:740] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.280999 12127 consensus_queue.cc:260] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.282753 12127 ts_tablet_manager.cc:1434] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 12:26:43.283442 12127 tablet_bootstrap.cc:492] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.295802 12127 tablet_bootstrap.cc:492] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.296629 12127 tablet_bootstrap.cc:492] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.297207 12127 ts_tablet_manager.cc:1403] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.014s	user 0.007s	sys 0.003s
I20260709 12:26:43.299270 12127 raft_consensus.cc:359] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.300012 12127 raft_consensus.cc:740] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.300629 12127 consensus_queue.cc:260] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.302181 12127 ts_tablet_manager.cc:1434] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:26:43.302922 12127 tablet_bootstrap.cc:492] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.315136 12127 tablet_bootstrap.cc:492] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.316094 12127 tablet_bootstrap.cc:492] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.316716 12127 ts_tablet_manager.cc:1403] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.014s	user 0.011s	sys 0.000s
I20260709 12:26:43.318920 12127 raft_consensus.cc:359] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.319612 12127 raft_consensus.cc:740] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.320258 12127 consensus_queue.cc:260] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.322284 12127 ts_tablet_manager.cc:1434] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:26:43.323092 12127 tablet_bootstrap.cc:492] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.337265 11827 raft_consensus.cc:493] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.337942 11827 raft_consensus.cc:515] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.340168 11827 leader_election.cc:290] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:43.344760 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:43.346776 30503 ts_tablet_manager.cc:616] Registered 45 tablets
W20260709 12:26:43.347179 11412 leader_election.cc:336] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111)
I20260709 12:26:43.348250 11411 leader_election.cc:304] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:43.347208 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.254s	user 0.235s	sys 0.009s
I20260709 12:26:43.349432 11827 raft_consensus.cc:2749] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:43.354831 12127 tablet_bootstrap.cc:492] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.356110 12127 tablet_bootstrap.cc:492] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.357992 12127 ts_tablet_manager.cc:1403] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.034s	user 0.018s	sys 0.013s
I20260709 12:26:43.360930 12127 raft_consensus.cc:359] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.361714 12127 raft_consensus.cc:740] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.362615 12127 consensus_queue.cc:260] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.365718 12127 ts_tablet_manager.cc:1434] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:26:43.366647 12127 tablet_bootstrap.cc:492] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.383626 12127 tablet_bootstrap.cc:492] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.384603 12127 tablet_bootstrap.cc:492] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.385246 12127 ts_tablet_manager.cc:1403] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.019s	user 0.012s	sys 0.002s
I20260709 12:26:43.387784 12127 raft_consensus.cc:359] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.388414 12127 raft_consensus.cc:740] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.389122 12127 consensus_queue.cc:260] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.391352 12127 ts_tablet_manager.cc:1434] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.002s
I20260709 12:26:43.392336 12127 tablet_bootstrap.cc:492] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.409521 12127 tablet_bootstrap.cc:492] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.410890 12127 tablet_bootstrap.cc:492] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.411671 12127 ts_tablet_manager.cc:1403] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.020s	user 0.014s	sys 0.004s
I20260709 12:26:43.413168 11827 raft_consensus.cc:493] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.413950 11827 raft_consensus.cc:515] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.415073 12127 raft_consensus.cc:359] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.415892 12127 raft_consensus.cc:740] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.416730 12127 consensus_queue.cc:260] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.417680 11517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8f2372474d94a11bb1e044fc69c60e5" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:43.418818 12127 ts_tablet_manager.cc:1434] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:26:43.419852 12127 tablet_bootstrap.cc:492] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.422802 11827 leader_election.cc:290] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
W20260709 12:26:43.428133 11412 leader_election.cc:336] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111)
I20260709 12:26:43.428794 11412 leader_election.cc:304] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:43.429769 11827 raft_consensus.cc:2749] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:43.436439 12127 tablet_bootstrap.cc:492] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.437598 12127 tablet_bootstrap.cc:492] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.438408 12127 ts_tablet_manager.cc:1403] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.019s	user 0.010s	sys 0.004s
I20260709 12:26:43.441016 12127 raft_consensus.cc:359] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.441699 12127 raft_consensus.cc:740] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.442340 12135 raft_consensus.cc:493] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.442710 12127 consensus_queue.cc:260] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.443194 12135 raft_consensus.cc:515] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.445222 12127 ts_tablet_manager.cc:1434] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:26:43.446200 12127 tablet_bootstrap.cc:492] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.447275 12135 leader_election.cc:290] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:43.448774 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:43.449572 11447 raft_consensus.cc:2468] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:43.452450 11336 leader_election.cc:304] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:43.453384 12135 raft_consensus.cc:2804] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:43.454059 12135 raft_consensus.cc:493] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.454653 12135 raft_consensus.cc:3060] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.466239 12135 raft_consensus.cc:515] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.469514 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
W20260709 12:26:43.469767 11338 leader_election.cc:336] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111)
I20260709 12:26:43.470139 11447 raft_consensus.cc:3060] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.478755 12135 leader_election.cc:290] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:43.481350 11447 raft_consensus.cc:2468] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 2.
I20260709 12:26:43.483000 11336 leader_election.cc:304] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:43.484097 11827 raft_consensus.cc:493] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.484987 11827 raft_consensus.cc:515] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.488758 11517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:43.492309 12131 raft_consensus.cc:493] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.492820 12099 raft_consensus.cc:493] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.493003 12131 raft_consensus.cc:515] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.493453 12099 raft_consensus.cc:515] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.496589 12099 leader_election.cc:290] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
W20260709 12:26:43.496956 11412 leader_election.cc:336] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111)
I20260709 12:26:43.497687 11412 leader_election.cc:304] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:43.497920 12095 raft_consensus.cc:493] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.498613 12095 raft_consensus.cc:515] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.499657 12127 tablet_bootstrap.cc:492] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.501155 12095 leader_election.cc:290] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:43.501856 12095 raft_consensus.cc:2749] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:43.501957 12127 tablet_bootstrap.cc:492] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.502924 12127 ts_tablet_manager.cc:1403] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.057s	user 0.014s	sys 0.034s
I20260709 12:26:43.502900 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4438988b178419385f7ec82ef7c99d2" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:43.506292 12127 raft_consensus.cc:359] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.507145 12127 raft_consensus.cc:740] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.507597 12131 leader_election.cc:290] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:43.508473 12127 consensus_queue.cc:260] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.509452 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:43.510165 11447 raft_consensus.cc:2468] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:43.511158 12127 ts_tablet_manager.cc:1434] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.001s
I20260709 12:26:43.512176 12127 tablet_bootstrap.cc:492] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:43.511885 11484 leader_election.cc:304] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:43.513243 12131 raft_consensus.cc:2804] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:43.513661 12131 raft_consensus.cc:493] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.514052 12131 raft_consensus.cc:3060] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.484170 12135 raft_consensus.cc:2804] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:43.516636 12135 raft_consensus.cc:697] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:43.517757 12135 consensus_queue.cc:237] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.519712 11827 leader_election.cc:290] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
W20260709 12:26:43.528689 11338 leader_election.cc:336] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111)
W20260709 12:26:43.530594 11412 leader_election.cc:336] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111)
I20260709 12:26:43.532739 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:43.540551 11411 leader_election.cc:304] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:43.541591 11827 raft_consensus.cc:2749] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:43.564845 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.193:37401
I20260709 12:26:43.565102 12203 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.193:37401 every 8 connection(s)
I20260709 12:26:43.565268 11827 raft_consensus.cc:493] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.566200 11827 raft_consensus.cc:515] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.570411 11517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f75f882db94790be4eaf2994c44f61" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:43.572144 12028 raft_consensus.cc:493] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.572997 12028 raft_consensus.cc:515] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.576368 12028 leader_election.cc:290] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:43.579496 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3701a37fd75248f6b8b0ca0930791a9d" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:43.589059 11827 leader_election.cc:290] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:43.584164 11195 catalog_manager.cc:5697] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 1 to 2, leader changed from 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193) to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 2 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:43.594497 12131 raft_consensus.cc:515] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.597424 12135 raft_consensus.cc:493] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.598397 12135 raft_consensus.cc:515] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.602847 12028 raft_consensus.cc:493] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.603606 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:43.604385 11446 raft_consensus.cc:2468] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:43.604470 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:43.603623 12028 raft_consensus.cc:515] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.606647 12131 leader_election.cc:290] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:43.611697 12028 leader_election.cc:290] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:43.605724 11447 raft_consensus.cc:3060] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.611728 11517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:43.615381 11336 leader_election.cc:304] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:43.616552 12028 raft_consensus.cc:2804] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:43.616964 12028 raft_consensus.cc:493] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.617340 12028 raft_consensus.cc:3060] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.629169 12127 tablet_bootstrap.cc:492] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.629943 12135 leader_election.cc:290] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:43.630429 12127 tablet_bootstrap.cc:492] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.631313 12127 ts_tablet_manager.cc:1403] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.119s	user 0.037s	sys 0.031s
I20260709 12:26:43.633208 11447 raft_consensus.cc:2468] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 2.
I20260709 12:26:43.634663 11484 leader_election.cc:304] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:43.635917 12099 raft_consensus.cc:2804] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:43.636963 12028 raft_consensus.cc:515] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.642200 12174 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3701a37fd75248f6b8b0ca0930791a9d" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:43.642807 12172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:43.643775 12172 raft_consensus.cc:2468] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:43.643035 12170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:43.645113 12173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:43.650269 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:43.651377 11446 raft_consensus.cc:3060] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:26:43.667833 11338 leader_election.cc:343] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:43.668572 11338 leader_election.cc:304] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:43.670078 12091 raft_consensus.cc:2749] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:43.673363 12060 raft_consensus.cc:493] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.674090 12060 raft_consensus.cc:515] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.674808 12191 raft_consensus.cc:493] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.675417 12191 raft_consensus.cc:515] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.678272 12095 raft_consensus.cc:493] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.679134 12095 raft_consensus.cc:515] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.682307 12060 leader_election.cc:290] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:43.682828 12135 raft_consensus.cc:493] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.683823 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:43.684571 11373 raft_consensus.cc:2468] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:43.685503 12211 raft_consensus.cc:493] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.685814 11485 leader_election.cc:304] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:43.685850 11338 leader_election.cc:304] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 9968c0ed1974454d828996824f859433
I20260709 12:26:43.681764 12095 leader_election.cc:290] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:43.634482 12127 raft_consensus.cc:359] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.690908 12127 raft_consensus.cc:740] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:43.694237 12170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f75f882db94790be4eaf2994c44f61" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:43.695086 12091 raft_consensus.cc:2804] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:43.695182 12127 consensus_queue.cc:260] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.695911 12091 raft_consensus.cc:493] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.696486 12091 raft_consensus.cc:3060] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.683698 12135 raft_consensus.cc:515] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.697685 12168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16773253264a4f8282ea1ea5b59a1ec6" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:43.699247 12211 raft_consensus.cc:515] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.702706 12127 ts_tablet_manager.cc:1434] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.071s	user 0.009s	sys 0.000s
I20260709 12:26:43.703744 12127 tablet_bootstrap.cc:492] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
W20260709 12:26:43.703744 11412 leader_election.cc:343] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:43.705349 11412 leader_election.cc:304] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:43.706866 12095 raft_consensus.cc:2749] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:43.699749 12135 leader_election.cc:290] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:43.710144 12211 leader_election.cc:290] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
W20260709 12:26:43.713805 11412 leader_election.cc:343] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:43.701481 12028 leader_election.cc:290] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:43.720424 12191 leader_election.cc:290] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:43.724377 11443 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16773253264a4f8282ea1ea5b59a1ec6" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:43.725385 11443 raft_consensus.cc:2468] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:43.728153 11443 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:43.729111 11443 raft_consensus.cc:2468] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
W20260709 12:26:43.732581 11485 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.194:44567, user_credentials={real_user=slave}} blocked reactor thread for 46978.1us
I20260709 12:26:43.733902 11446 raft_consensus.cc:2468] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 2.
I20260709 12:26:43.735872 11336 leader_election.cc:304] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:43.738112 12170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:43.738656 12216 raft_consensus.cc:493] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.738866 12028 raft_consensus.cc:2804] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:43.739377 12028 raft_consensus.cc:493] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.739300 12216 raft_consensus.cc:515] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.739943 12028 raft_consensus.cc:3060] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.744145 12212 raft_consensus.cc:493] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.744772 12212 raft_consensus.cc:515] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
W20260709 12:26:43.747550 11412 leader_election.cc:343] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:43.747916 11338 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 62156.8us
I20260709 12:26:43.748313 12060 raft_consensus.cc:493] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
W20260709 12:26:43.748752 11338 leader_election.cc:343] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:43.748930 12060 raft_consensus.cc:515] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
W20260709 12:26:43.749452 11338 leader_election.cc:343] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:43.760573 11336 leader_election.cc:304] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:43.764089 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16773253264a4f8282ea1ea5b59a1ec6" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:43.722489 12215 raft_consensus.cc:2804] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:43.783883 11827 raft_consensus.cc:493] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.786800 11827 raft_consensus.cc:515] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.792606 11827 leader_election.cc:290] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:43.793442 12213 raft_consensus.cc:493] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.793614 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f75f882db94790be4eaf2994c44f61" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:43.794354 11446 raft_consensus.cc:2468] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:43.794165 12213 raft_consensus.cc:515] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.785018 12215 raft_consensus.cc:493] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.796365 11484 leader_election.cc:304] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:43.803700 12216 leader_election.cc:290] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:43.804606 12172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16773253264a4f8282ea1ea5b59a1ec6" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:43.806679 12213 leader_election.cc:290] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
W20260709 12:26:43.809669 11485 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:44567, user_credentials={real_user=slave}} blocked reactor thread for 68569us
I20260709 12:26:43.808193 12135 raft_consensus.cc:2804] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:43.807979 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc185f43f658441ba207bdfdc0e79a7f" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:43.816879 11446 raft_consensus.cc:2468] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:43.797348 12215 raft_consensus.cc:3060] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.820580 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8f2372474d94a11bb1e044fc69c60e5" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" is_pre_election: true
I20260709 12:26:43.821789 11446 raft_consensus.cc:2468] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:43.820780 11484 leader_election.cc:304] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:43.807089 12060 leader_election.cc:290] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:43.828049 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:43.804637 12216 raft_consensus.cc:2804] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:43.829599 12216 raft_consensus.cc:493] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.830200 12216 raft_consensus.cc:3060] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.832535 12091 raft_consensus.cc:515] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.833333 12212 leader_election.cc:290] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:43.838688 12214 raft_consensus.cc:493] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.818048 12099 raft_consensus.cc:697] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:43.839418 12214 raft_consensus.cc:515] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.845389 11484 leader_election.cc:304] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:43.818362 12135 raft_consensus.cc:493] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.855090 12219 raft_consensus.cc:2804] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:43.852438 12212 raft_consensus.cc:493] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
W20260709 12:26:43.863755 11338 leader_election.cc:343] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:43.856921 12135 raft_consensus.cc:3060] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:26:43.870003 11336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:36253, user_credentials={real_user=slave}} blocked reactor thread for 109415us
I20260709 12:26:43.858748 12219 raft_consensus.cc:493] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.873785 11336 leader_election.cc:304] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:43.865746 12131 raft_consensus.cc:2804] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:43.858345 12191 raft_consensus.cc:493] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.862285 12173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc185f43f658441ba207bdfdc0e79a7f" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:43.840791 12099 consensus_queue.cc:237] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.887414 12131 raft_consensus.cc:493] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.890770 12131 raft_consensus.cc:3060] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.890967 12191 raft_consensus.cc:515] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.891631 12060 raft_consensus.cc:493] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.892181 12060 raft_consensus.cc:515] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.860888 12212 raft_consensus.cc:515] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.876112 12219 raft_consensus.cc:3060] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.883807 12091 leader_election.cc:290] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:43.903213 12213 raft_consensus.cc:2804] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:43.910058 12220 raft_consensus.cc:493] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:43.918320 12220 raft_consensus.cc:515] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:43.919337 12060 leader_election.cc:290] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:43.911577 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc185f43f658441ba207bdfdc0e79a7f" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:43.912114 12214 leader_election.cc:290] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:43.889073 12172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:43.931239 12220 leader_election.cc:290] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:43.932438 11515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:43.936648 12127 tablet_bootstrap.cc:492] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:43.940694 12174 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:43.941713 12174 raft_consensus.cc:2468] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 2.
I20260709 12:26:43.944339 12191 leader_election.cc:290] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:43.944981 12028 raft_consensus.cc:515] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.946336 11373 raft_consensus.cc:2393] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a1ce6e048ab47d69c453b5218cb64c1 in current term 2: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:43.948510 11514 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:43.949348 11514 raft_consensus.cc:2468] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 2.
I20260709 12:26:43.950449 12028 leader_election.cc:290] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:43.925272 12213 raft_consensus.cc:697] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:43.951716 12212 leader_election.cc:290] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:43.951984 12213 consensus_queue.cc:237] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.958114 12131 raft_consensus.cc:515] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.968374 11411 leader_election.cc:304] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:43.971633 11827 raft_consensus.cc:3060] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.973968 12127 tablet_bootstrap.cc:492] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:43.975358 12127 ts_tablet_manager.cc:1403] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.272s	user 0.048s	sys 0.022s
I20260709 12:26:43.977568 11338 leader_election.cc:304] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:43.981003 11367 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3701a37fd75248f6b8b0ca0930791a9d" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:43.981909 11367 raft_consensus.cc:2468] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 1.
I20260709 12:26:43.982345 12167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
W20260709 12:26:43.985445 11338 leader_election.cc:343] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:43.987785 11411 leader_election.cc:304] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:43.930050 12216 raft_consensus.cc:515] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:43.983101 12167 raft_consensus.cc:3060] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:26:43.996480 11412 leader_election.cc:343] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:43.997267 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8f2372474d94a11bb1e044fc69c60e5" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:43.998272 11447 raft_consensus.cc:3060] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.999374 12172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf22a8b3fa4491c92c735f8151982d3" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:44.000838 12172 raft_consensus.cc:2468] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:44.002275 12174 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3701a37fd75248f6b8b0ca0930791a9d" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:44.002763 12217 raft_consensus.cc:2804] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:44.004328 12127 raft_consensus.cc:359] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:44.005669 12127 raft_consensus.cc:740] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:44.006417 12127 consensus_queue.cc:260] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:43.993472 12214 raft_consensus.cc:2804] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:26:44.003166 12216 leader_election.cc:290] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
W20260709 12:26:44.012012 11412 leader_election.cc:343] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:44.012797 11447 raft_consensus.cc:2468] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 2.
I20260709 12:26:44.013100 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f75f882db94790be4eaf2994c44f61" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:44.013935 11446 raft_consensus.cc:3060] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:43.976166 12215 raft_consensus.cc:515] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:44.020164 11484 leader_election.cc:304] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:44.023006 11447 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16773253264a4f8282ea1ea5b59a1ec6" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:44.027204 12127 ts_tablet_manager.cc:1434] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.051s	user 0.007s	sys 0.000s
I20260709 12:26:44.028524 12127 tablet_bootstrap.cc:492] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:44.051889 11446 raft_consensus.cc:2468] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 2.
W20260709 12:26:44.064625 11410 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 59477.5us
I20260709 12:26:44.006863 12217 raft_consensus.cc:493] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:44.011451 12214 raft_consensus.cc:493] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:44.072968 12217 raft_consensus.cc:3060] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:26:44.112528 11336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 96054.8us
I20260709 12:26:44.079793 12214 raft_consensus.cc:3060] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:44.136371 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:44.139990 11518 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf22a8b3fa4491c92c735f8151982d3" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:44.141060 11518 raft_consensus.cc:2468] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:44.142366 12226 raft_consensus.cc:2804] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:44.131310 11515 raft_consensus.cc:2393] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 05b93ba4435f40fb97b54343c4fdd71e in current term 2: Already voted for candidate 9968c0ed1974454d828996824f859433 in this term.
I20260709 12:26:44.145874 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf22a8b3fa4491c92c735f8151982d3" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e" is_pre_election: true
I20260709 12:26:44.146756 11372 raft_consensus.cc:2468] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:44.147066 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4438988b178419385f7ec82ef7c99d2" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433" is_pre_election: true
I20260709 12:26:44.147904 11519 raft_consensus.cc:2468] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 1.
I20260709 12:26:44.150897 12170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4438988b178419385f7ec82ef7c99d2" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:44.158402 11485 leader_election.cc:304] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:44.164865 11827 raft_consensus.cc:2749] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:44.177351 11484 leader_election.cc:304] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:44.138473 12131 leader_election.cc:290] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:44.167587 12226 raft_consensus.cc:697] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:44.214916 11338 leader_election.cc:304] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:44.204659 12231 raft_consensus.cc:2804] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:26:44.221344 11336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:36253, user_credentials={real_user=slave}} blocked reactor thread for 99066.4us
W20260709 12:26:44.231885 11338 leader_election.cc:343] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:44.231581 12131 raft_consensus.cc:2804] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election won for term 2
W20260709 12:26:44.245952 11484 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:36253, user_credentials={real_user=slave}} blocked reactor thread for 68681us
I20260709 12:26:44.226512 12213 raft_consensus.cc:2804] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:44.250974 12131 raft_consensus.cc:697] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:44.212900 12135 raft_consensus.cc:515] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:44.253911 12215 leader_election.cc:290] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:44.229991 12231 raft_consensus.cc:493] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:44.260455 11195 catalog_manager.cc:5697] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 1 to 2, leader changed from 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193) to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 2 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:44.261919 11447 raft_consensus.cc:2468] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 2.
I20260709 12:26:44.291306 12131 consensus_queue.cc:237] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:44.308688 11369 raft_consensus.cc:2393] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a1ce6e048ab47d69c453b5218cb64c1 in current term 2: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:44.268085 12213 raft_consensus.cc:493] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:44.357298 11370 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:44.358116 12135 leader_election.cc:290] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:44.358381 11370 raft_consensus.cc:2393] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 2: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:44.280371 12231 raft_consensus.cc:3060] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:44.262606 12226 consensus_queue.cc:237] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:44.362718 12213 raft_consensus.cc:3060] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:44.392650 12219 raft_consensus.cc:515] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:44.394237 11521 raft_consensus.cc:2393] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a1ce6e048ab47d69c453b5218cb64c1 in current term 2: Already voted for candidate 9968c0ed1974454d828996824f859433 in this term.
I20260709 12:26:44.373950 12167 raft_consensus.cc:2468] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 2.
I20260709 12:26:44.398933 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc185f43f658441ba207bdfdc0e79a7f" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:44.399704 11446 raft_consensus.cc:3060] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:44.404381 11338 leader_election.cc:304] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:44.404186 11336 leader_election.cc:304] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 663d88d9634e4720a695c7cd69eaa25e
W20260709 12:26:44.473667 11336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 60324.2us
I20260709 12:26:44.397833 12219 leader_election.cc:290] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:44.472445 12135 raft_consensus.cc:2804] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:44.484961 12135 raft_consensus.cc:697] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:44.494797 12167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:44.496166 12217 raft_consensus.cc:515] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
W20260709 12:26:44.502045 11338 consensus_peers.cc:597] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:44.504210 11338 consensus_peers.cc:597] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.514199 12164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc185f43f658441ba207bdfdc0e79a7f" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:44.514884 12204 heartbeater.cc:344] Connected to a master server at 127.29.201.254:38597
I20260709 12:26:44.515403 12204 heartbeater.cc:461] Registering TS with master...
I20260709 12:26:44.516019 12164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8f2372474d94a11bb1e044fc69c60e5" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:44.516686 12204 heartbeater.cc:507] Master 127.29.201.254:38597 requested a full tablet report, sending...
I20260709 12:26:44.478874 12211 raft_consensus.cc:2804] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:26:44.518147 12211 raft_consensus.cc:493] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:44.518579 12211 raft_consensus.cc:3060] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:44.517395 12164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f75f882db94790be4eaf2994c44f61" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:44.523690 11336 leader_election.cc:304] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:44.524259 12164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:44.525200 12164 raft_consensus.cc:2393] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 2: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:44.525755 12167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:44.485878 12135 consensus_queue.cc:237] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:44.526851 12168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4438988b178419385f7ec82ef7c99d2" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:44.527451 12167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc185f43f658441ba207bdfdc0e79a7f" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:44.513578 12165 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf22a8b3fa4491c92c735f8151982d3" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:44.498502 12166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fee758b4d91541148427a240c8f1db4e" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:44.485812 12214 raft_consensus.cc:515] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
W20260709 12:26:44.506323 11338 leader_election.cc:343] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:44.528085 12168 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:44.528920 12167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70f75f882db94790be4eaf2994c44f61" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:44.543632 12168 raft_consensus.cc:2468] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 2.
I20260709 12:26:44.526151 12164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8f2372474d94a11bb1e044fc69c60e5" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:44.529008 12165 raft_consensus.cc:2468] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 1.
I20260709 12:26:44.529886 12231 raft_consensus.cc:515] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:44.530040 12217 leader_election.cc:290] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:44.530192 12166 raft_consensus.cc:2393] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 2: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:44.551359 12211 raft_consensus.cc:515] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:44.532853 12213 raft_consensus.cc:515] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
W20260709 12:26:44.538492 11486 leader_election.cc:343] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:44.586370 11486 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 47912.2us
W20260709 12:26:44.587105 11486 leader_election.cc:343] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:44.588053 11486 leader_election.cc:343] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:44.588999 11486 leader_election.cc:304] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:44.554095 11446 raft_consensus.cc:2468] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 2.
I20260709 12:26:44.554589 12231 leader_election.cc:290] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:44.590236 12216 raft_consensus.cc:2749] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260709 12:26:44.592226 11486 leader_election.cc:343] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:44.592772 12213 leader_election.cc:290] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
W20260709 12:26:44.593369 11486 leader_election.cc:343] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:44.594149 11486 leader_election.cc:304] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:44.539116 12173 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e" is_pre_election: true
I20260709 12:26:44.595234 12231 raft_consensus.cc:2749] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:44.563177 11193 catalog_manager.cc:5697] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 1 to 2, leader changed from 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193) to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:44.600409 11486 leader_election.cc:343] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:44.564381 12166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
W20260709 12:26:44.567348 11338 consensus_peers.cc:597] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.613790 11338 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 46626.2us
I20260709 12:26:44.541615 12238 raft_consensus.cc:2804] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:44.534966 11373 raft_consensus.cc:2393] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 3: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:44.540825 12214 leader_election.cc:290] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 3 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
W20260709 12:26:44.581489 11412 consensus_peers.cc:597] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.625622 11412 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:37401, user_credentials={real_user=slave}} blocked reactor thread for 44255.4us
W20260709 12:26:44.627195 11412 consensus_peers.cc:597] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.557643 11372 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf22a8b3fa4491c92c735f8151982d3" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
W20260709 12:26:44.617372 11338 consensus_peers.cc:597] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:44.613663 11336 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 47011.4us
I20260709 12:26:44.624054 11486 leader_election.cc:304] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
W20260709 12:26:44.634122 11486 leader_election.cc:343] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:26:44.635802 11486 consensus_peers.cc:597] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:44.637485 11486 leader_election.cc:343] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:44.638880 11372 raft_consensus.cc:2393] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 2: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
W20260709 12:26:44.640532 11410 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.196:34569, user_credentials={real_user=slave}} blocked reactor thread for 234460us
W20260709 12:26:44.641132 11486 leader_election.cc:343] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:44.640017 12211 leader_election.cc:290] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:44.638772 12166 raft_consensus.cc:3060] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:44.639119 11443 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:44.663331 11521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4438988b178419385f7ec82ef7c99d2" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:44.665699 11521 raft_consensus.cc:3060] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:44.641180 12226 raft_consensus.cc:2804] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:26:44.661234 11410 leader_election.cc:304] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:44.646090 12238 raft_consensus.cc:697] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:44.684587 11444 raft_consensus.cc:1275] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:44.684708 12262 raft_consensus.cc:2749] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:44.677644 11519 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf22a8b3fa4491c92c735f8151982d3" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:44.686501 11519 raft_consensus.cc:2393] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 05b93ba4435f40fb97b54343c4fdd71e in current term 2: Already voted for candidate 9968c0ed1974454d828996824f859433 in this term.
I20260709 12:26:44.686028 12238 consensus_queue.cc:237] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:44.686803 11371 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3701a37fd75248f6b8b0ca0930791a9d" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:44.687559 11371 raft_consensus.cc:3060] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:44.693627 11521 raft_consensus.cc:2468] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 2.
I20260709 12:26:44.644179 12174 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf22a8b3fa4491c92c735f8151982d3" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:44.698940 11371 raft_consensus.cc:2468] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 2.
I20260709 12:26:44.699167 12174 raft_consensus.cc:3060] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:44.667027 12226 raft_consensus.cc:493] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:44.703943 12226 raft_consensus.cc:3060] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:44.704244 11411 leader_election.cc:304] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:44.676406 11443 raft_consensus.cc:3060] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:44.657676 11515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:44.711141 12169 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16773253264a4f8282ea1ea5b59a1ec6" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:44.711328 12174 raft_consensus.cc:2468] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9968c0ed1974454d828996824f859433 in term 2.
I20260709 12:26:44.713335 11447 raft_consensus.cc:1275] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 9968c0ed1974454d828996824f859433: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:26:44.715839 11338 leader_election.cc:343] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:44.719393 11443 raft_consensus.cc:2468] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 2.
I20260709 12:26:44.674786 11484 leader_election.cc:304] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 9968c0ed1974454d828996824f859433; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:44.720214 12258 raft_consensus.cc:2749] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:26:44.722808 12167 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3701a37fd75248f6b8b0ca0930791a9d" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:44.723141 11411 leader_election.cc:304] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:44.724287 12256 raft_consensus.cc:2804] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:44.724822 12256 raft_consensus.cc:697] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:44.725762 12256 consensus_queue.cc:237] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
W20260709 12:26:44.730516 11484 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:36253, user_credentials={real_user=slave}} blocked reactor thread for 55897.5us
I20260709 12:26:44.732332 11336 leader_election.cc:304] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:44.738812 12245 raft_consensus.cc:2804] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:44.739632 12245 raft_consensus.cc:697] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:44.733920 12060 raft_consensus.cc:2804] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:44.741079 12060 raft_consensus.cc:697] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:44.740746 12245 consensus_queue.cc:237] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:44.742363 12060 consensus_queue.cc:237] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:44.763969 11486 leader_election.cc:304] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:44.768184 12231 raft_consensus.cc:2804] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:44.774469 12231 raft_consensus.cc:697] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Becoming Leader. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:44.783216 12231 consensus_queue.cc:237] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:44.787691 12219 consensus_queue.cc:1048] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:44.798215 12166 raft_consensus.cc:2468] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 3.
I20260709 12:26:44.808044 12226 raft_consensus.cc:515] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:44.813220 11338 leader_election.cc:304] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
W20260709 12:26:44.815668 12127 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.517s	user 0.022s	sys 0.014s
W20260709 12:26:44.816164 12127 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.518s	user 0.022s	sys 0.014s
I20260709 12:26:44.814180 12214 raft_consensus.cc:2804] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:26:44.819290 12214 raft_consensus.cc:697] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:44.820255 12214 consensus_queue.cc:237] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
W20260709 12:26:44.824599 11412 consensus_peers.cc:597] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.833336 11446 raft_consensus.cc:1275] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:26:44.833814 11412 consensus_peers.cc:597] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.834964 11412 leader_election.cc:343] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:44.822342 12127 tablet_bootstrap.cc:492] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:44.845718 12127 tablet_bootstrap.cc:492] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:44.846843 12127 ts_tablet_manager.cc:1403] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.819s	user 0.041s	sys 0.015s
I20260709 12:26:44.826546 11515 raft_consensus.cc:2393] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 05b93ba4435f40fb97b54343c4fdd71e in current term 3: Already voted for candidate 9968c0ed1974454d828996824f859433 in this term.
I20260709 12:26:44.855350 12164 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1cf22a8b3fa4491c92c735f8151982d3" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:44.859345 12164 raft_consensus.cc:2393] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 05b93ba4435f40fb97b54343c4fdd71e in current term 2: Already voted for candidate 9968c0ed1974454d828996824f859433 in this term.
I20260709 12:26:44.861436 11338 leader_election.cc:304] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e; no voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433
I20260709 12:26:44.863745 12127 raft_consensus.cc:359] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:44.864600 12127 raft_consensus.cc:740] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:44.867377 12127 consensus_queue.cc:260] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:44.875356 12226 leader_election.cc:290] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 3 election: Requested vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:44.883028 11447 raft_consensus.cc:1275] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 9968c0ed1974454d828996824f859433: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:26:44.884223 11412 consensus_peers.cc:597] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.884635 12127 ts_tablet_manager.cc:1434] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.037s	user 0.005s	sys 0.003s
W20260709 12:26:44.885229 11412 consensus_peers.cc:597] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.870273 12245 raft_consensus.cc:2749] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260709 12:26:44.883406 12166 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
I20260709 12:26:44.890194 12226 consensus_queue.cc:1048] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.010s
I20260709 12:26:44.905957 12166 raft_consensus.cc:2393] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 9968c0ed1974454d828996824f859433 in current term 3: Already voted for candidate 05b93ba4435f40fb97b54343c4fdd71e in this term.
I20260709 12:26:44.907402 11447 raft_consensus.cc:1275] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 9968c0ed1974454d828996824f859433: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:44.910084 12218 consensus_queue.cc:1048] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:44.922111 12170 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b4438988b178419385f7ec82ef7c99d2" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
W20260709 12:26:44.923954 11338 leader_election.cc:343] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:26:44.939072 11336 leader_election.cc:304] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:44.946806 12224 consensus_queue.cc:1048] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:44.950867 12239 raft_consensus.cc:2804] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:44.951509 12239 raft_consensus.cc:697] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:44.952373 12239 consensus_queue.cc:237] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
W20260709 12:26:44.953362 11412 consensus_peers.cc:597] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.957599 11364 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "9968c0ed1974454d828996824f859433" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:44.958916 12028 consensus_queue.cc:1048] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:26:44.969897 11338 consensus_peers.cc:597] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.970721 11338 consensus_peers.cc:597] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:44.971637 11338 consensus_peers.cc:597] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:44.975355 12170 raft_consensus.cc:3060] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:44.957628 11193 ts_manager.cc:194] Re-registered known tserver with Master: 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401)
I20260709 12:26:45.041046 11515 raft_consensus.cc:1275] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:45.051386 12232 consensus_queue.cc:1048] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.011s
W20260709 12:26:45.068545 11338 consensus_peers.cc:597] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.082325 11485 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:44567, user_credentials={real_user=slave}} blocked reactor thread for 53052.1us
W20260709 12:26:45.105715 11338 consensus_peers.cc:597] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:45.119266 12127 tablet_bootstrap.cc:492] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:45.126243 11485 leader_election.cc:304] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
W20260709 12:26:45.127537 11486 consensus_peers.cc:597] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:45.130398 12231 raft_consensus.cc:2749] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260709 12:26:45.148048 11486 consensus_peers.cc:597] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:45.160341 11195 catalog_manager.cc:5697] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 1 to 2, leader changed from 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193) to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:45.169615 11195 catalog_manager.cc:5697] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 1 to 2, leader changed from 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193) to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:45.172492 11195 catalog_manager.cc:5697] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 1 to 2, leader changed from 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193) to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:45.175439 12173 raft_consensus.cc:1275] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:45.185405 11195 catalog_manager.cc:5697] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 reported cstate change: term changed from 1 to 2, leader changed from 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193) to 9968c0ed1974454d828996824f859433 (127.29.201.196). New cstate: current_term: 2 leader_uuid: "9968c0ed1974454d828996824f859433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:45.210858 12243 consensus_queue.cc:1048] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:45.224707 12173 raft_consensus.cc:1275] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Refusing update from remote peer 9968c0ed1974454d828996824f859433: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:26:45.226920 11338 consensus_peers.cc:597] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:45.246251 11373 raft_consensus.cc:1275] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 9968c0ed1974454d828996824f859433: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:26:45.237493 11484 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:36253, user_credentials={real_user=slave}} blocked reactor thread for 51929.2us
I20260709 12:26:45.262348 11443 raft_consensus.cc:1275] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:45.267045 11368 raft_consensus.cc:1275] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 5a1ce6e048ab47d69c453b5218cb64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:45.237069 12218 consensus_queue.cc:1048] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260709 12:26:45.268646 12227 consensus_queue.cc:1048] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:45.280431 12217 consensus_queue.cc:1048] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.013s
I20260709 12:26:45.279163 12243 consensus_queue.cc:1048] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.020s
W20260709 12:26:45.318997 11338 consensus_peers.cc:597] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:45.325518 11515 raft_consensus.cc:1275] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 3 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:26:45.336118 12243 consensus_queue.cc:1048] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:26:45.337347 11412 consensus_peers.cc:597] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:45.417196 12173 raft_consensus.cc:1275] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:26:45.417513 11517 raft_consensus.cc:1275] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:45.419309 12243 consensus_queue.cc:1048] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:45.419546 12239 consensus_queue.cc:1048] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:45.424219 11444 raft_consensus.cc:1275] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:45.426651 12127 tablet_bootstrap.cc:492] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:45.427773 12127 tablet_bootstrap.cc:492] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:45.427861 12243 consensus_queue.cc:1048] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:45.428609 12127 ts_tablet_manager.cc:1403] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.310s	user 0.019s	sys 0.000s
I20260709 12:26:45.436335 12127 raft_consensus.cc:359] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:45.439491 12127 raft_consensus.cc:740] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:45.440375 12127 consensus_queue.cc:260] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:45.445763 11194 catalog_manager.cc:5697] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 1 to 2, leader changed from 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193) to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 2 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:45.448135 12127 ts_tablet_manager.cc:1434] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.000s
I20260709 12:26:45.447880 11194 catalog_manager.cc:5697] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 1 to 2, leader changed from 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193) to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 2 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:45.449075 12127 tablet_bootstrap.cc:492] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:45.449766 11194 catalog_manager.cc:5697] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 1 to 2, leader changed from 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193) to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 2 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:45.463765 11194 catalog_manager.cc:5697] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 2 to 3, leader changed from 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193) to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 3 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:45.465564 11194 catalog_manager.cc:5697] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 1 to 2, leader changed from 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193) to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 2 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
W20260709 12:26:45.470754 11486 consensus_peers.cc:597] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20260709 12:26:45.471909 11486 consensus_peers.cc:597] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:45.476856 11446 raft_consensus.cc:1275] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 9968c0ed1974454d828996824f859433: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260709 12:26:45.501583 11338 consensus_peers.cc:597] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:45.508826 12282 consensus_queue.cc:1048] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:26:45.527953 12298 log.cc:927] Time spent T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1: Append to log took a long time: real 0.053s	user 0.000s	sys 0.003s
I20260709 12:26:45.541007 11191 catalog_manager.cc:5697] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 1 to 2, leader changed from 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193) to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 2 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:45.546475 11193 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:41984
I20260709 12:26:45.567844 12204 heartbeater.cc:499] Master 127.29.201.254:38597 was elected leader, sending a full tablet report...
I20260709 12:26:45.584995 12127 tablet_bootstrap.cc:492] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:45.586095 12127 tablet_bootstrap.cc:492] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:45.586887 12127 ts_tablet_manager.cc:1403] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.138s	user 0.016s	sys 0.019s
I20260709 12:26:45.589615 12127 raft_consensus.cc:359] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:45.590355 12127 raft_consensus.cc:740] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:45.591221 12127 consensus_queue.cc:260] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:45.593434 12127 ts_tablet_manager.cc:1434] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:26:45.594269 12127 tablet_bootstrap.cc:492] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
W20260709 12:26:45.651706 11486 consensus_peers.cc:597] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:45.654803 12127 tablet_bootstrap.cc:492] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:45.655905 12127 tablet_bootstrap.cc:492] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:45.656607 12127 ts_tablet_manager.cc:1403] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.063s	user 0.023s	sys 0.035s
I20260709 12:26:45.659261 12127 raft_consensus.cc:359] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:45.659996 12127 raft_consensus.cc:740] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:45.660728 12127 consensus_queue.cc:260] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:45.662565 12127 ts_tablet_manager.cc:1434] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.007s	sys 0.000s
I20260709 12:26:45.663430 12127 tablet_bootstrap.cc:492] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:45.701130 12127 tablet_bootstrap.cc:492] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:45.702490 12127 tablet_bootstrap.cc:492] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
W20260709 12:26:45.702984 11486 consensus_peers.cc:597] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:45.703222 12127 ts_tablet_manager.cc:1403] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.040s	user 0.018s	sys 0.019s
I20260709 12:26:45.706307 12127 raft_consensus.cc:359] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:45.707056 12127 raft_consensus.cc:740] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:45.707789 12127 consensus_queue.cc:260] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:45.709723 12127 ts_tablet_manager.cc:1434] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:26:45.710572 12127 tablet_bootstrap.cc:492] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:45.749461 12127 tablet_bootstrap.cc:492] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:45.750530 12127 tablet_bootstrap.cc:492] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:45.751277 12127 ts_tablet_manager.cc:1403] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.041s	user 0.020s	sys 0.014s
I20260709 12:26:45.753902 12127 raft_consensus.cc:359] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:45.754578 12127 raft_consensus.cc:740] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:45.755390 12127 consensus_queue.cc:260] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:45.759338 12127 ts_tablet_manager.cc:1434] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:26:45.762724 12127 tablet_bootstrap.cc:492] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:45.776957 12170 raft_consensus.cc:3060] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:45.800661 12127 tablet_bootstrap.cc:492] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:45.801833 12127 tablet_bootstrap.cc:492] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:45.802917 12127 ts_tablet_manager.cc:1403] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.041s	user 0.019s	sys 0.020s
I20260709 12:26:45.808153 12127 raft_consensus.cc:359] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:45.809051 12127 raft_consensus.cc:740] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:45.809760 12127 consensus_queue.cc:260] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:45.811689 12127 ts_tablet_manager.cc:1434] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:26:45.812426 12127 tablet_bootstrap.cc:492] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:45.833760 12127 tablet_bootstrap.cc:492] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:45.834595 12127 tablet_bootstrap.cc:492] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:45.835470 12127 ts_tablet_manager.cc:1403] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.023s	user 0.009s	sys 0.008s
I20260709 12:26:45.838243 12127 raft_consensus.cc:359] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:45.839000 12127 raft_consensus.cc:740] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:45.839668 12127 consensus_queue.cc:260] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:45.841830 12127 ts_tablet_manager.cc:1434] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:26:45.843130 12127 tablet_bootstrap.cc:492] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:45.858043 12127 tablet_bootstrap.cc:492] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:45.859086 12127 tablet_bootstrap.cc:492] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:45.859835 12127 ts_tablet_manager.cc:1403] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.017s	user 0.014s	sys 0.001s
I20260709 12:26:45.862655 12127 raft_consensus.cc:359] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:45.863418 12127 raft_consensus.cc:740] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:45.864151 12127 consensus_queue.cc:260] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:45.866503 12127 ts_tablet_manager.cc:1434] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:26:45.867624 12127 tablet_bootstrap.cc:492] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:45.885389 12127 tablet_bootstrap.cc:492] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:45.886427 12127 tablet_bootstrap.cc:492] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:45.887239 12127 ts_tablet_manager.cc:1403] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.020s	user 0.014s	sys 0.004s
I20260709 12:26:45.889865 12127 raft_consensus.cc:359] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:45.890689 12127 raft_consensus.cc:740] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:45.891448 12127 consensus_queue.cc:260] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:45.893932 12127 ts_tablet_manager.cc:1434] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:26:45.895198 12127 tablet_bootstrap.cc:492] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:45.994841 12127 tablet_bootstrap.cc:492] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:45.995882 12127 tablet_bootstrap.cc:492] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:45.996603 12127 ts_tablet_manager.cc:1403] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.102s	user 0.040s	sys 0.023s
I20260709 12:26:45.999626 12127 raft_consensus.cc:359] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.000361 12127 raft_consensus.cc:740] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.001061 12127 consensus_queue.cc:260] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.002774 12127 ts_tablet_manager.cc:1434] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:26:46.004314 12127 tablet_bootstrap.cc:492] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.019713 12127 tablet_bootstrap.cc:492] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.020746 12127 tablet_bootstrap.cc:492] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.021453 12127 ts_tablet_manager.cc:1403] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.017s	user 0.012s	sys 0.002s
I20260709 12:26:46.023965 12127 raft_consensus.cc:359] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.024685 12127 raft_consensus.cc:740] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.025477 12127 consensus_queue.cc:260] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.027434 12127 ts_tablet_manager.cc:1434] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.006s	sys 0.002s
I20260709 12:26:46.028398 12127 tablet_bootstrap.cc:492] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.057938 12127 tablet_bootstrap.cc:492] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.058990 12127 tablet_bootstrap.cc:492] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.059641 12127 ts_tablet_manager.cc:1403] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.031s	user 0.019s	sys 0.011s
I20260709 12:26:46.062559 12127 raft_consensus.cc:359] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:46.063444 12127 raft_consensus.cc:740] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.064306 12127 consensus_queue.cc:260] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:46.067437 12127 ts_tablet_manager.cc:1434] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.001s
I20260709 12:26:46.068342 12127 tablet_bootstrap.cc:492] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.084735 12127 tablet_bootstrap.cc:492] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.085762 12127 tablet_bootstrap.cc:492] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.086514 12127 ts_tablet_manager.cc:1403] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.018s	user 0.009s	sys 0.004s
I20260709 12:26:46.089629 12127 raft_consensus.cc:359] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.090346 12127 raft_consensus.cc:740] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.091018 12127 consensus_queue.cc:260] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
W20260709 12:26:46.091984 11486 consensus_peers.cc:597] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20260709 12:26:46.093152 12127 ts_tablet_manager.cc:1434] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:26:46.094285 12127 tablet_bootstrap.cc:492] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.112324 12127 tablet_bootstrap.cc:492] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.113188 12127 tablet_bootstrap.cc:492] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.113797 12127 ts_tablet_manager.cc:1403] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.020s	user 0.013s	sys 0.000s
I20260709 12:26:46.115947 12127 raft_consensus.cc:359] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.116500 12127 raft_consensus.cc:740] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.117141 12127 consensus_queue.cc:260] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.119306 12127 ts_tablet_manager.cc:1434] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.005s	user 0.008s	sys 0.000s
I20260709 12:26:46.120436 12127 tablet_bootstrap.cc:492] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.125381 12173 raft_consensus.cc:3060] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:46.146860 12127 tablet_bootstrap.cc:492] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.147867 12127 tablet_bootstrap.cc:492] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.148576 12127 ts_tablet_manager.cc:1403] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.028s	user 0.016s	sys 0.007s
I20260709 12:26:46.151326 12127 raft_consensus.cc:359] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:46.152107 12127 raft_consensus.cc:740] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.152742 12127 consensus_queue.cc:260] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:46.155223 12127 ts_tablet_manager.cc:1434] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:26:46.156030 12127 tablet_bootstrap.cc:492] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.172891 12127 tablet_bootstrap.cc:492] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.173908 12127 tablet_bootstrap.cc:492] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.174721 12127 ts_tablet_manager.cc:1403] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.019s	user 0.015s	sys 0.003s
I20260709 12:26:46.177232 12127 raft_consensus.cc:359] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.178028 12127 raft_consensus.cc:740] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.178864 12127 consensus_queue.cc:260] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.181145 12127 ts_tablet_manager.cc:1434] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.000s	sys 0.003s
I20260709 12:26:46.182487 12127 tablet_bootstrap.cc:492] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.201299 12127 tablet_bootstrap.cc:492] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.202152 12127 tablet_bootstrap.cc:492] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.202926 12127 ts_tablet_manager.cc:1403] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.021s	user 0.014s	sys 0.003s
I20260709 12:26:46.206979 12127 raft_consensus.cc:359] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.207713 12127 raft_consensus.cc:740] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.208446 12127 consensus_queue.cc:260] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.209975 12127 ts_tablet_manager.cc:1434] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.007s	user 0.003s	sys 0.000s
I20260709 12:26:46.211113 12127 tablet_bootstrap.cc:492] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.227624 12173 raft_consensus.cc:3060] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:46.231208 12127 tablet_bootstrap.cc:492] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.232326 12127 tablet_bootstrap.cc:492] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.233028 12127 ts_tablet_manager.cc:1403] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.022s	user 0.014s	sys 0.005s
I20260709 12:26:46.235901 12127 raft_consensus.cc:359] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.236632 12127 raft_consensus.cc:740] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.237452 12127 consensus_queue.cc:260] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.239979 12127 ts_tablet_manager.cc:1434] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:26:46.241143 12127 tablet_bootstrap.cc:492] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.258018 12127 tablet_bootstrap.cc:492] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.258996 12127 tablet_bootstrap.cc:492] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.259684 12127 ts_tablet_manager.cc:1403] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.019s	user 0.016s	sys 0.001s
I20260709 12:26:46.262293 12127 raft_consensus.cc:359] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.262992 12127 raft_consensus.cc:740] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.263695 12127 consensus_queue.cc:260] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.265769 12127 ts_tablet_manager.cc:1434] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:26:46.266544 12127 tablet_bootstrap.cc:492] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.287171 12127 tablet_bootstrap.cc:492] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.288110 12127 tablet_bootstrap.cc:492] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.288815 12127 ts_tablet_manager.cc:1403] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.022s	user 0.017s	sys 0.003s
I20260709 12:26:46.291085 12127 raft_consensus.cc:359] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.291702 12127 raft_consensus.cc:740] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.292362 12127 consensus_queue.cc:260] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.294405 12127 ts_tablet_manager.cc:1434] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:26:46.295959 12127 tablet_bootstrap.cc:492] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.311476 12127 tablet_bootstrap.cc:492] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.312374 12127 tablet_bootstrap.cc:492] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.312980 12127 ts_tablet_manager.cc:1403] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.017s	user 0.005s	sys 0.009s
I20260709 12:26:46.315208 12127 raft_consensus.cc:359] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.315910 12127 raft_consensus.cc:740] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.316622 12127 consensus_queue.cc:260] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.318967 12127 ts_tablet_manager.cc:1434] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:26:46.320976 12127 tablet_bootstrap.cc:492] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.339668 12170 raft_consensus.cc:3060] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:46.345777 12173 raft_consensus.cc:3060] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:46.353386 12164 raft_consensus.cc:3060] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:46.393862 12174 raft_consensus.cc:3060] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:46.422537 12127 tablet_bootstrap.cc:492] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.423782 12127 tablet_bootstrap.cc:492] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.424732 12127 ts_tablet_manager.cc:1403] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.104s	user 0.044s	sys 0.042s
I20260709 12:26:46.427739 12127 raft_consensus.cc:359] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.428673 12127 raft_consensus.cc:740] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.429571 12127 consensus_queue.cc:260] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.434372 12127 ts_tablet_manager.cc:1434] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:26:46.435379 12127 tablet_bootstrap.cc:492] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.455823 12127 tablet_bootstrap.cc:492] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.457979 12127 tablet_bootstrap.cc:492] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.461736 12127 ts_tablet_manager.cc:1403] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.027s	user 0.008s	sys 0.013s
I20260709 12:26:46.464581 12127 raft_consensus.cc:359] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.465209 12127 raft_consensus.cc:740] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.465905 12127 consensus_queue.cc:260] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.468171 12127 ts_tablet_manager.cc:1434] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:26:46.468974 12127 tablet_bootstrap.cc:492] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.485446 12127 tablet_bootstrap.cc:492] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.486374 12127 tablet_bootstrap.cc:492] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.487154 12127 ts_tablet_manager.cc:1403] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.018s	user 0.013s	sys 0.004s
I20260709 12:26:46.489447 12127 raft_consensus.cc:359] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.490070 12127 raft_consensus.cc:740] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.490805 12127 consensus_queue.cc:260] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.492668 12127 ts_tablet_manager.cc:1434] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:26:46.493861 12127 tablet_bootstrap.cc:492] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.531654 12127 tablet_bootstrap.cc:492] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.532832 12127 tablet_bootstrap.cc:492] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.533635 12127 ts_tablet_manager.cc:1403] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.040s	user 0.012s	sys 0.012s
I20260709 12:26:46.536451 12127 raft_consensus.cc:359] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:46.537189 12127 raft_consensus.cc:740] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.537891 12127 consensus_queue.cc:260] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:46.540828 12127 ts_tablet_manager.cc:1434] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:26:46.541674 12127 tablet_bootstrap.cc:492] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.559518 12127 tablet_bootstrap.cc:492] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.560670 12127 tablet_bootstrap.cc:492] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.561398 12127 ts_tablet_manager.cc:1403] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.020s	user 0.014s	sys 0.003s
I20260709 12:26:46.563527 12127 raft_consensus.cc:359] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.564091 12127 raft_consensus.cc:740] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.564749 12127 consensus_queue.cc:260] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:46.566792 12127 ts_tablet_manager.cc:1434] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.005s	user 0.007s	sys 0.000s
I20260709 12:26:46.568005 12127 tablet_bootstrap.cc:492] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.598047 12127 tablet_bootstrap.cc:492] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.599074 12127 tablet_bootstrap.cc:492] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.599742 12127 ts_tablet_manager.cc:1403] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.032s	user 0.012s	sys 0.004s
I20260709 12:26:46.602080 12127 raft_consensus.cc:359] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.602669 12127 raft_consensus.cc:740] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.603493 12127 consensus_queue.cc:260] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:46.605733 12127 ts_tablet_manager.cc:1434] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.001s
I20260709 12:26:46.608372 12127 tablet_bootstrap.cc:492] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap starting.
I20260709 12:26:46.622710 12127 tablet_bootstrap.cc:492] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:26:46.623705 12127 tablet_bootstrap.cc:492] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e: Bootstrap complete.
I20260709 12:26:46.624423 12127 ts_tablet_manager.cc:1403] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e: Time spent bootstrapping tablet: real 0.016s	user 0.014s	sys 0.000s
I20260709 12:26:46.627156 12127 raft_consensus.cc:359] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:46.627877 12127 raft_consensus.cc:740] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Initialized, Role: FOLLOWER
I20260709 12:26:46.628582 12127 consensus_queue.cc:260] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:46.630848 12127 ts_tablet_manager.cc:1434] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:26:46.789402 12174 raft_consensus.cc:3060] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:46.920779 12164 raft_consensus.cc:3060] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:53.911456 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:53.970737 11373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab"
dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
mode: GRACEFUL
new_leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
 from {username='slave'} at 127.0.0.1:53470
I20260709 12:26:53.971843 11373 raft_consensus.cc:606] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Received request to transfer leadership to TS 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:53.987504 11444 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6788efa5ae124a478e06ef927cb631ff"
dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:35062
I20260709 12:26:53.988372 11444 raft_consensus.cc:606] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:53.991451 11373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "7179cfd71f5c4bcf8beac36a18e4c94e"
dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:53470
I20260709 12:26:53.992053 11373 raft_consensus.cc:606] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:53.994493 11373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602"
dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:53470
I20260709 12:26:53.995136 11373 raft_consensus.cc:606] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:53.997623 11373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd"
dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:53470
I20260709 12:26:53.998086 11373 raft_consensus.cc:606] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:54.000298 11373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086"
dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:53470
I20260709 12:26:54.000727 11373 raft_consensus.cc:606] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:54.013343 11515 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ff00b27947404e52acea283e9381d837"
dest_uuid: "9968c0ed1974454d828996824f859433"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:34516
I20260709 12:26:54.014039 11515 raft_consensus.cc:606] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:54.014432 12357 raft_consensus.cc:993] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e: : Instructing follower 5a1ce6e048ab47d69c453b5218cb64c1 to start an election
I20260709 12:26:54.015070 12357 raft_consensus.cc:1081] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Signalling peer 5a1ce6e048ab47d69c453b5218cb64c1 to start an election
I20260709 12:26:54.015812 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 7
I20260709 12:26:54.016906 11444 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab"
dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
 from {username='slave'} at 127.0.0.1:42394
I20260709 12:26:54.017376 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:54.017686 11444 raft_consensus.cc:493] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:54.018225 11444 raft_consensus.cc:3060] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:54.019136 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:54.034205 11444 raft_consensus.cc:515] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:54.036984 11444 leader_election.cc:290] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:54.037829 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:54.038707 11373 raft_consensus.cc:3055] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:54.039011 11373 raft_consensus.cc:740] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:54.039810 11515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:54.040490 11515 raft_consensus.cc:3060] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:54.040639 12354 raft_consensus.cc:993] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:54.041162 12354 raft_consensus.cc:1081] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:54.039840 11373 consensus_queue.cc:260] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:54.043711 11373 raft_consensus.cc:3060] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:54.047052 12164 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ff00b27947404e52acea283e9381d837"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50170
I20260709 12:26:54.047665 12164 raft_consensus.cc:493] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:54.047981 12164 raft_consensus.cc:3060] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:54.051793 11373 raft_consensus.cc:2468] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 2.
I20260709 12:26:54.053499 11411 leader_election.cc:304] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:54.054656 12348 raft_consensus.cc:2804] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:54.056078 12348 raft_consensus.cc:697] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:54.057076 12348 consensus_queue.cc:237] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:54.060812 12164 raft_consensus.cc:515] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:54.087787 11515 raft_consensus.cc:2468] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 2.
I20260709 12:26:54.112895 12164 leader_election.cc:290] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:54.152715 11193 catalog_manager.cc:5697] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 1 to 2, leader changed from 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194) to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 2 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:54.169862 12357 raft_consensus.cc:993] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:54.170498 12357 raft_consensus.cc:1081] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:54.172441 11515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff00b27947404e52acea283e9381d837" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:54.173309 12164 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50600
I20260709 12:26:54.174006 12164 raft_consensus.cc:493] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:54.174382 12164 raft_consensus.cc:3060] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:54.181831 11515 raft_consensus.cc:3055] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:54.187036 12164 raft_consensus.cc:515] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:54.187737 11515 raft_consensus.cc:740] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:54.182710 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ff00b27947404e52acea283e9381d837" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:54.192312 11515 consensus_queue.cc:260] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:54.194403 11515 raft_consensus.cc:3060] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:54.195791 11368 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:54.190359 11373 raft_consensus.cc:3060] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:54.196796 11368 raft_consensus.cc:3055] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:26:54.197307 11368 raft_consensus.cc:740] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:54.198470 11368 consensus_queue.cc:260] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:54.211174 11368 raft_consensus.cc:3060] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:54.213982 12164 leader_election.cc:290] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 3 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:54.223877 11373 raft_consensus.cc:2468] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:54.229089 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86bb3bcc0a6449a9adc6c21b722f6602" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:54.229151 11515 raft_consensus.cc:2468] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:54.230005 11444 raft_consensus.cc:3060] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:54.235524 12121 leader_election.cc:304] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:54.223970 11368 raft_consensus.cc:2468] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 3.
I20260709 12:26:54.247430 12379 raft_consensus.cc:2804] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:54.247424 12122 leader_election.cc:304] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:54.251096 12375 raft_consensus.cc:993] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:54.251874 12375 raft_consensus.cc:1081] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:54.259804 11444 raft_consensus.cc:2468] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 3.
I20260709 12:26:54.260919 12381 raft_consensus.cc:2804] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:26:54.261544 12381 raft_consensus.cc:697] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 3 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:54.262715 12381 consensus_queue.cc:237] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:54.264780 12170 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "7179cfd71f5c4bcf8beac36a18e4c94e"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50600
I20260709 12:26:54.265496 12170 raft_consensus.cc:493] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:54.265919 12170 raft_consensus.cc:3060] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:54.277750 12379 raft_consensus.cc:697] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:54.278961 12379 consensus_queue.cc:237] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:54.283901 12170 raft_consensus.cc:515] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:54.290076 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7179cfd71f5c4bcf8beac36a18e4c94e" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:54.290747 11444 raft_consensus.cc:3060] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:54.291219 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7179cfd71f5c4bcf8beac36a18e4c94e" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:54.291877 11369 raft_consensus.cc:3055] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:54.292222 11369 raft_consensus.cc:740] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:54.293155 11369 consensus_queue.cc:260] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:54.294293 11369 raft_consensus.cc:3060] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:54.303014 11193 catalog_manager.cc:5697] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 1 to 2, leader changed from 9968c0ed1974454d828996824f859433 (127.29.201.196) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 2 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:54.309805 11369 raft_consensus.cc:2468] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:54.312103 12170 leader_election.cc:290] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:54.313884 12122 leader_election.cc:304] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:54.314931 12379 raft_consensus.cc:2804] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:54.315812 12379 raft_consensus.cc:697] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:54.316912 12379 consensus_queue.cc:237] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:54.322197 11444 raft_consensus.cc:2468] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:54.340106 11193 catalog_manager.cc:5697] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 2 to 3, leader changed from 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 3 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:54.341717 11193 catalog_manager.cc:5697] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 1 to 2, leader changed from 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 2 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:54.407754 12364 raft_consensus.cc:993] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:54.408236 12357 raft_consensus.cc:1081] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:54.410050 12170 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50600
I20260709 12:26:54.410650 12170 raft_consensus.cc:493] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:54.411036 12170 raft_consensus.cc:3060] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:54.417272 12170 raft_consensus.cc:515] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:54.419332 12170 leader_election.cc:290] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 3 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:54.420006 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:54.420378 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b3c0ce8bbe4f4392bdac15d4d35f84dd" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:54.420720 11369 raft_consensus.cc:3055] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:26:54.421026 12370 raft_consensus.cc:993] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:54.421152 11369 raft_consensus.cc:740] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:54.421778 12385 raft_consensus.cc:1081] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:54.421042 11444 raft_consensus.cc:3060] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:54.423694 12173 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6788efa5ae124a478e06ef927cb631ff"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50610
I20260709 12:26:54.424286 12173 raft_consensus.cc:493] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:54.424675 12173 raft_consensus.cc:3060] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:54.422127 11369 consensus_queue.cc:260] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:54.427224 11369 raft_consensus.cc:3060] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:54.430017 11444 raft_consensus.cc:2468] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 3.
I20260709 12:26:54.431603 12121 leader_election.cc:304] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:54.432453 12379 raft_consensus.cc:2804] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:26:54.432986 12379 raft_consensus.cc:697] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 3 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:54.434031 12379 consensus_queue.cc:237] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:54.434880 11369 raft_consensus.cc:2468] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 3.
I20260709 12:26:54.434981 12173 raft_consensus.cc:515] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:54.438494 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6788efa5ae124a478e06ef927cb631ff" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:54.439595 11444 raft_consensus.cc:3055] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:54.440063 11444 raft_consensus.cc:740] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:54.441337 11444 consensus_queue.cc:260] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:54.442608 11444 raft_consensus.cc:3060] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:54.437320 12173 leader_election.cc:290] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:54.438503 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6788efa5ae124a478e06ef927cb631ff" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:54.449046 11369 raft_consensus.cc:3060] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:54.453052 11193 catalog_manager.cc:5697] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 2 to 3, leader changed from 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 3 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:54.456043 11369 raft_consensus.cc:2468] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:54.456342 11444 raft_consensus.cc:2468] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:54.457108 12122 leader_election.cc:304] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:54.457914 12379 raft_consensus.cc:2804] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:54.458393 12379 raft_consensus.cc:697] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:54.459241 12379 consensus_queue.cc:237] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:54.473698 11193 catalog_manager.cc:5697] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 1 to 2, leader changed from 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 2 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:54.520540 11373 raft_consensus.cc:1275] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 5a1ce6e048ab47d69c453b5218cb64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:54.521771 12385 consensus_queue.cc:1048] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:54.538434 11512 raft_consensus.cc:1275] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Refusing update from remote peer 5a1ce6e048ab47d69c453b5218cb64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:54.540284 12385 consensus_queue.cc:1048] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:54.579896 12364 raft_consensus.cc:993] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:54.580471 12357 raft_consensus.cc:1081] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:54.582443 12170 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50600
I20260709 12:26:54.583076 12170 raft_consensus.cc:493] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:54.583500 12170 raft_consensus.cc:3060] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:26:54.591614 12170 raft_consensus.cc:515] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:54.593839 12170 leader_election.cc:290] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 4 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:54.594961 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:54.594961 11515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fa781c13298b41e2ba513a74a2455086" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:54.595834 11369 raft_consensus.cc:3055] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:26:54.596186 11369 raft_consensus.cc:740] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:54.597049 11369 consensus_queue.cc:260] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:54.598433 11369 raft_consensus.cc:3060] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:26:54.595860 11515 raft_consensus.cc:3060] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:26:54.606472 11369 raft_consensus.cc:2468] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 4.
I20260709 12:26:54.610682 12122 leader_election.cc:304] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:54.611428 11515 raft_consensus.cc:2468] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 4.
I20260709 12:26:54.611506 12379 raft_consensus.cc:2804] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:26:54.612279 12379 raft_consensus.cc:697] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 4 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:54.613266 12379 consensus_queue.cc:237] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:54.623948 11196 catalog_manager.cc:5697] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 3 to 4, leader changed from 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 4 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:54.661351 11444 raft_consensus.cc:1275] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:54.662477 12379 consensus_queue.cc:1048] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:54.677312 11369 raft_consensus.cc:1275] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:54.683081 12379 consensus_queue.cc:1048] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:54.689590 11515 raft_consensus.cc:1275] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:54.691915 12381 consensus_queue.cc:1048] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:54.705673 11372 raft_consensus.cc:1275] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:54.708570 12379 consensus_queue.cc:1048] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:54.808466 11372 raft_consensus.cc:1275] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:54.809842 12379 consensus_queue.cc:1048] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:54.819543 11443 raft_consensus.cc:1275] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:54.823884 12380 consensus_queue.cc:1048] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:54.909827 11372 raft_consensus.cc:1275] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:54.911217 12380 consensus_queue.cc:1048] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:54.917898 11444 raft_consensus.cc:1275] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:54.919535 12380 consensus_queue.cc:1048] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:54.985597 11369 raft_consensus.cc:1275] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:54.986946 12381 consensus_queue.cc:1048] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:54.997290 11444 raft_consensus.cc:1275] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:54.998955 12381 consensus_queue.cc:1048] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:55.097481 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:55.101784 11512 raft_consensus.cc:1275] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 4 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:26:55.103263 12381 consensus_queue.cc:1048] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:55.117843 11369 raft_consensus.cc:1275] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 4 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260709 12:26:55.120347 12379 consensus_queue.cc:1048] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:55.181195 11515 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "431c66d13d0f4904a42b33665a9c09e1"
dest_uuid: "9968c0ed1974454d828996824f859433"
mode: GRACEFUL
new_leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
 from {username='slave'} at 127.0.0.1:34516
I20260709 12:26:55.181846 11515 raft_consensus.cc:606] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Received request to transfer leadership to TS 05b93ba4435f40fb97b54343c4fdd71e
I20260709 12:26:55.189051 11515 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "5eda5b4099d041c9aef56884ca4e594a"
dest_uuid: "9968c0ed1974454d828996824f859433"
mode: GRACEFUL
new_leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
 from {username='slave'} at 127.0.0.1:34516
I20260709 12:26:55.189777 11515 raft_consensus.cc:606] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Received request to transfer leadership to TS 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:55.192478 11444 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "91b161cce9ce49c58571f631c6810266"
dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
mode: GRACEFUL
new_leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
 from {username='slave'} at 127.0.0.1:35062
I20260709 12:26:55.192936 11444 raft_consensus.cc:606] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Received request to transfer leadership to TS 05b93ba4435f40fb97b54343c4fdd71e
I20260709 12:26:55.195410 11515 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "e313f6102e9941248ba9069990945017"
dest_uuid: "9968c0ed1974454d828996824f859433"
mode: GRACEFUL
new_leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
 from {username='slave'} at 127.0.0.1:34516
I20260709 12:26:55.195889 11515 raft_consensus.cc:606] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Received request to transfer leadership to TS 5a1ce6e048ab47d69c453b5218cb64c1
I20260709 12:26:55.198235 11444 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fea9a20aab6840c1afb32c29bac2f4a4"
dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:35062
I20260709 12:26:55.198753 11444 raft_consensus.cc:606] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:55.199966 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:26:55.201712 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:55.202433 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:55.211917 12348 raft_consensus.cc:993] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:55.212567 12385 raft_consensus.cc:1081] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:55.214087 12173 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fea9a20aab6840c1afb32c29bac2f4a4"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50610
I20260709 12:26:55.214756 12173 raft_consensus.cc:493] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:55.215101 12173 raft_consensus.cc:3060] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.221077 12173 raft_consensus.cc:515] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:55.222980 12173 leader_election.cc:290] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:55.224228 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea9a20aab6840c1afb32c29bac2f4a4" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:55.224964 11369 raft_consensus.cc:3060] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.227711 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fea9a20aab6840c1afb32c29bac2f4a4" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:55.228636 11444 raft_consensus.cc:3055] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:55.229199 11444 raft_consensus.cc:740] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:55.231206 11444 consensus_queue.cc:260] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:55.232694 11444 raft_consensus.cc:3060] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.239888 11369 raft_consensus.cc:2468] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:55.241216 12122 leader_election.cc:304] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:55.242059 12379 raft_consensus.cc:2804] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:55.242597 12379 raft_consensus.cc:697] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:55.243760 12379 consensus_queue.cc:237] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:55.246730 11444 raft_consensus.cc:2468] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:55.265271 11193 catalog_manager.cc:5697] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 1 to 2, leader changed from 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 2 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:55.307240 12348 raft_consensus.cc:993] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1: : Instructing follower 05b93ba4435f40fb97b54343c4fdd71e to start an election
I20260709 12:26:55.307726 12385 raft_consensus.cc:1081] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Signalling peer 05b93ba4435f40fb97b54343c4fdd71e to start an election
I20260709 12:26:55.309208 11373 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "91b161cce9ce49c58571f631c6810266"
dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
 from {username='slave'} at 127.0.0.1:47074
I20260709 12:26:55.309770 11373 raft_consensus.cc:493] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:55.310109 11373 raft_consensus.cc:3060] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.317665 11373 raft_consensus.cc:515] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:55.320317 11373 leader_election.cc:290] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:55.321094 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91b161cce9ce49c58571f631c6810266" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:55.321487 11515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "91b161cce9ce49c58571f631c6810266" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:55.321767 11444 raft_consensus.cc:3055] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:55.322099 11444 raft_consensus.cc:740] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:55.322104 11515 raft_consensus.cc:3060] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.323077 11444 consensus_queue.cc:260] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:55.324420 11444 raft_consensus.cc:3060] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.330108 11515 raft_consensus.cc:2468] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 2.
I20260709 12:26:55.331845 11336 leader_election.cc:304] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:55.332751 12364 raft_consensus.cc:2804] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:55.333238 12364 raft_consensus.cc:697] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:55.334144 12364 consensus_queue.cc:237] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:55.337311 11444 raft_consensus.cc:2468] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 2.
I20260709 12:26:55.346876 11193 catalog_manager.cc:5697] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 1 to 2, leader changed from 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195) to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 2 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:55.370096 12363 raft_consensus.cc:993] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433: : Instructing follower 5a1ce6e048ab47d69c453b5218cb64c1 to start an election
I20260709 12:26:55.370777 12369 raft_consensus.cc:1081] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Signalling peer 5a1ce6e048ab47d69c453b5218cb64c1 to start an election
I20260709 12:26:55.372321 11443 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "5eda5b4099d041c9aef56884ca4e594a"
dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
 from {username='slave'} at 127.0.0.1:42398
I20260709 12:26:55.372834 11443 raft_consensus.cc:493] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:55.373164 11443 raft_consensus.cc:3060] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.378325 11443 raft_consensus.cc:515] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:55.380232 11443 leader_election.cc:290] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:55.381124 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eda5b4099d041c9aef56884ca4e594a" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:55.381191 11515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5eda5b4099d041c9aef56884ca4e594a" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:55.381907 11373 raft_consensus.cc:3060] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.381976 11515 raft_consensus.cc:3055] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:55.382491 11515 raft_consensus.cc:740] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:55.383306 11515 consensus_queue.cc:260] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:55.384399 11515 raft_consensus.cc:3060] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.389267 11373 raft_consensus.cc:2468] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 2.
I20260709 12:26:55.390836 11515 raft_consensus.cc:2468] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 2.
I20260709 12:26:55.390990 11411 leader_election.cc:304] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:55.391961 12385 raft_consensus.cc:2804] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:55.392462 12385 raft_consensus.cc:697] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:55.393218 12385 consensus_queue.cc:237] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:55.408561 11193 catalog_manager.cc:5697] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 1 to 2, leader changed from 9968c0ed1974454d828996824f859433 (127.29.201.196) to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 2 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:55.487017 12363 raft_consensus.cc:993] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433: : Instructing follower 5a1ce6e048ab47d69c453b5218cb64c1 to start an election
I20260709 12:26:55.487638 12363 raft_consensus.cc:1081] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Signalling peer 5a1ce6e048ab47d69c453b5218cb64c1 to start an election
I20260709 12:26:55.489076 11443 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "e313f6102e9941248ba9069990945017"
dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
 from {username='slave'} at 127.0.0.1:42398
I20260709 12:26:55.489581 11443 raft_consensus.cc:493] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:55.489931 11443 raft_consensus.cc:3060] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.492427 12369 raft_consensus.cc:993] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433: : Instructing follower 05b93ba4435f40fb97b54343c4fdd71e to start an election
I20260709 12:26:55.493088 12363 raft_consensus.cc:1081] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Signalling peer 05b93ba4435f40fb97b54343c4fdd71e to start an election
I20260709 12:26:55.494696 11373 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "431c66d13d0f4904a42b33665a9c09e1"
dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
 from {username='slave'} at 127.0.0.1:47096
I20260709 12:26:55.495280 11373 raft_consensus.cc:493] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:55.495630 11373 raft_consensus.cc:3060] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.496802 11443 raft_consensus.cc:515] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:55.499629 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e313f6102e9941248ba9069990945017" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:55.500226 11369 raft_consensus.cc:3060] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.502843 11373 raft_consensus.cc:515] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:55.505770 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431c66d13d0f4904a42b33665a9c09e1" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:55.505844 11373 leader_election.cc:290] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:55.506978 11515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "431c66d13d0f4904a42b33665a9c09e1" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:55.507582 11515 raft_consensus.cc:3055] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:55.507869 11515 raft_consensus.cc:740] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:55.508466 11369 raft_consensus.cc:2468] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 2.
I20260709 12:26:55.508665 11515 consensus_queue.cc:260] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:55.509511 11411 leader_election.cc:304] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:55.509690 11515 raft_consensus.cc:3060] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.510382 12385 raft_consensus.cc:2804] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:55.506448 11444 raft_consensus.cc:3060] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.510705 11512 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e313f6102e9941248ba9069990945017" candidate_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:55.510876 12385 raft_consensus.cc:697] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 LEADER]: Becoming Leader. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:55.511401 11512 raft_consensus.cc:3055] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:55.511801 11512 raft_consensus.cc:740] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:55.511957 12385 consensus_queue.cc:237] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:55.512635 11512 consensus_queue.cc:260] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:55.513811 11512 raft_consensus.cc:3060] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:55.509652 11443 leader_election.cc:290] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [CANDIDATE]: Term 2 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:55.519901 11515 raft_consensus.cc:2468] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 2.
I20260709 12:26:55.522004 11512 raft_consensus.cc:2468] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a1ce6e048ab47d69c453b5218cb64c1 in term 2.
I20260709 12:26:55.522799 11444 raft_consensus.cc:2468] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 2.
I20260709 12:26:55.524549 11336 leader_election.cc:304] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 5a1ce6e048ab47d69c453b5218cb64c1; no voters: 
I20260709 12:26:55.526237 12357 raft_consensus.cc:2804] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:55.526773 12357 raft_consensus.cc:697] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:55.527693 12357 consensus_queue.cc:237] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:55.537570 11193 catalog_manager.cc:5697] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 reported cstate change: term changed from 1 to 2, leader changed from 9968c0ed1974454d828996824f859433 (127.29.201.196) to 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195). New cstate: current_term: 2 leader_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:55.560469 11193 catalog_manager.cc:5697] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 1 to 2, leader changed from 9968c0ed1974454d828996824f859433 (127.29.201.196) to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 2 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:55.748055 11369 raft_consensus.cc:1275] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:55.749248 12408 consensus_queue.cc:1048] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:55.752851 11515 raft_consensus.cc:1275] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:55.754415 12357 consensus_queue.cc:1048] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:55.796325 11443 raft_consensus.cc:1275] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:55.798017 12357 consensus_queue.cc:1048] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:55.799731 11444 raft_consensus.cc:1275] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:55.805799 12379 consensus_queue.cc:1048] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:55.867010 11515 raft_consensus.cc:1275] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Refusing update from remote peer 5a1ce6e048ab47d69c453b5218cb64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:55.868521 12348 consensus_queue.cc:1048] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:55.881431 11373 raft_consensus.cc:1275] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 5a1ce6e048ab47d69c453b5218cb64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:55.882692 12348 consensus_queue.cc:1048] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:55.921200 11373 raft_consensus.cc:1275] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 5a1ce6e048ab47d69c453b5218cb64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:55.922441 12348 consensus_queue.cc:1048] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:55.949765 11512 raft_consensus.cc:1275] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Refusing update from remote peer 5a1ce6e048ab47d69c453b5218cb64c1: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:55.951107 12385 consensus_queue.cc:1048] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:55.970819 11444 raft_consensus.cc:1275] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:55.972476 12357 consensus_queue.cc:1048] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:55.981731 11512 raft_consensus.cc:1275] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:55.983592 12415 consensus_queue.cc:1048] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:56.257453 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:56.307534 11373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "1ce080ae4c98415a868195fb1df17f18"
dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:53470
I20260709 12:26:56.308095 11373 raft_consensus.cc:606] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:56.310781 11446 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab"
dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
mode: GRACEFUL
new_leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
 from {username='slave'} at 127.0.0.1:35062
I20260709 12:26:56.311290 11446 raft_consensus.cc:606] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 LEADER]: Received request to transfer leadership to TS 05b93ba4435f40fb97b54343c4fdd71e
I20260709 12:26:56.313789 11512 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "827a79573ac94525a9ff30c744c45164"
dest_uuid: "9968c0ed1974454d828996824f859433"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:34516
I20260709 12:26:56.314321 11512 raft_consensus.cc:606] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:56.316918 11446 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "bd76933be5f240a1a5c069775527f562"
dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:35062
I20260709 12:26:56.317509 11446 raft_consensus.cc:606] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:56.319906 11373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "db58e810d4a04064bbf12daef58026f3"
dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:53470
I20260709 12:26:56.320405 11373 raft_consensus.cc:606] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:56.321409 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260709 12:26:56.323040 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:56.323712 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:56.405287 12385 raft_consensus.cc:993] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1: : Instructing follower 05b93ba4435f40fb97b54343c4fdd71e to start an election
I20260709 12:26:56.405954 12425 raft_consensus.cc:1081] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 LEADER]: Signalling peer 05b93ba4435f40fb97b54343c4fdd71e to start an election
I20260709 12:26:56.407658 11373 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab"
dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
 from {username='slave'} at 127.0.0.1:47074
I20260709 12:26:56.408228 11373 raft_consensus.cc:493] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:56.408568 11373 raft_consensus.cc:3060] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:56.417398 11373 raft_consensus.cc:515] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:56.423326 11373 leader_election.cc:290] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 3 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:56.425987 11517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:56.428262 11517 raft_consensus.cc:3060] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:56.430251 12364 raft_consensus.cc:993] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:56.432207 12364 raft_consensus.cc:1081] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:56.433913 12173 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "db58e810d4a04064bbf12daef58026f3"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50600
I20260709 12:26:56.434469 12173 raft_consensus.cc:493] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:56.420722 11446 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "267085844c6943b08c0f42b220e7e8ab" candidate_uuid: "05b93ba4435f40fb97b54343c4fdd71e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:56.434860 12173 raft_consensus.cc:3060] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:56.435052 11446 raft_consensus.cc:3055] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:26:56.435544 11446 raft_consensus.cc:740] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:56.436547 11446 consensus_queue.cc:260] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:56.441321 11446 raft_consensus.cc:3060] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:56.440163 11517 raft_consensus.cc:2468] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 3.
I20260709 12:26:56.449568 11336 leader_election.cc:304] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:56.450510 12415 raft_consensus.cc:2804] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:26:56.451043 12415 raft_consensus.cc:697] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 3 LEADER]: Becoming Leader. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:56.451889 12415 consensus_queue.cc:237] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } }
I20260709 12:26:56.453867 12173 raft_consensus.cc:515] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:56.454345 11446 raft_consensus.cc:2468] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05b93ba4435f40fb97b54343c4fdd71e in term 3.
I20260709 12:26:56.456554 12173 leader_election.cc:290] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:56.457772 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db58e810d4a04064bbf12daef58026f3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:56.459149 11373 raft_consensus.cc:3055] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:56.459508 11373 raft_consensus.cc:740] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:56.459987 11517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "db58e810d4a04064bbf12daef58026f3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:56.460628 11517 raft_consensus.cc:3060] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:56.460439 11373 consensus_queue.cc:260] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:56.461925 11373 raft_consensus.cc:3060] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:56.475521 11373 raft_consensus.cc:2468] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:56.477231 12122 leader_election.cc:304] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:56.478195 12437 raft_consensus.cc:2804] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:56.478739 12437 raft_consensus.cc:697] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:56.479009 11517 raft_consensus.cc:2468] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:56.479631 12437 consensus_queue.cc:237] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:56.490392 11193 catalog_manager.cc:5697] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e reported cstate change: term changed from 2 to 3, leader changed from 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195) to 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194). New cstate: current_term: 3 leader_uuid: "05b93ba4435f40fb97b54343c4fdd71e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:56.501202 11194 catalog_manager.cc:5697] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 1 to 2, leader changed from 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 2 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:56.638015 12415 raft_consensus.cc:993] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:56.638546 12357 raft_consensus.cc:1081] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:56.640196 12173 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "1ce080ae4c98415a868195fb1df17f18"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50600
I20260709 12:26:56.640760 12173 raft_consensus.cc:493] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:56.641065 12173 raft_consensus.cc:3060] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:56.647931 12173 raft_consensus.cc:515] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:56.649880 12173 leader_election.cc:290] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:56.650981 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ce080ae4c98415a868195fb1df17f18" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:56.651718 11369 raft_consensus.cc:3055] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:56.652007 11369 raft_consensus.cc:740] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:56.652684 11517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1ce080ae4c98415a868195fb1df17f18" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:56.652860 11369 consensus_queue.cc:260] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:56.653326 11517 raft_consensus.cc:3060] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:56.654173 11369 raft_consensus.cc:3060] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:56.661942 11517 raft_consensus.cc:2468] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:56.662004 11369 raft_consensus.cc:2468] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:56.662621 11512 raft_consensus.cc:1240] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Rejecting Update request from peer 05b93ba4435f40fb97b54343c4fdd71e for earlier term 1. Current term is 2. Ops: []
I20260709 12:26:56.663069 12121 leader_election.cc:304] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:56.663975 12437 raft_consensus.cc:2804] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:56.665335 12437 raft_consensus.cc:697] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:56.666440 12437 consensus_queue.cc:237] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:56.669409 12363 raft_consensus.cc:993] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:56.670032 12369 raft_consensus.cc:1081] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:56.672237 12173 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "827a79573ac94525a9ff30c744c45164"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50170
I20260709 12:26:56.672860 12173 raft_consensus.cc:493] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:56.673298 12173 raft_consensus.cc:3060] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:56.680099 11194 catalog_manager.cc:5697] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 1 to 2, leader changed from 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 2 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:56.683075 12173 raft_consensus.cc:515] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:56.685231 12173 leader_election.cc:290] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:56.686352 11512 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "827a79573ac94525a9ff30c744c45164" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:56.687053 11512 raft_consensus.cc:3055] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:56.687346 11512 raft_consensus.cc:740] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:56.688107 11512 consensus_queue.cc:260] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:56.689354 11512 raft_consensus.cc:3060] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:56.686424 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "827a79573ac94525a9ff30c744c45164" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:56.693827 11444 raft_consensus.cc:3060] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:56.700480 11444 raft_consensus.cc:2468] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:56.701653 12121 leader_election.cc:304] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:56.702729 12437 raft_consensus.cc:2804] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:56.703375 12437 raft_consensus.cc:697] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:56.704417 12437 consensus_queue.cc:237] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:56.705574 11512 raft_consensus.cc:2468] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:56.714617 11194 catalog_manager.cc:5697] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 1 to 2, leader changed from 9968c0ed1974454d828996824f859433 (127.29.201.196) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 2 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:56.744410 12385 raft_consensus.cc:993] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:56.744971 12432 raft_consensus.cc:1081] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:56.746531 12164 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "bd76933be5f240a1a5c069775527f562"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50610
I20260709 12:26:56.747125 12164 raft_consensus.cc:493] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:56.747484 12164 raft_consensus.cc:3060] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:56.754750 12164 raft_consensus.cc:515] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:56.757334 11369 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd76933be5f240a1a5c069775527f562" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:56.757987 11369 raft_consensus.cc:3060] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:56.758191 12164 leader_election.cc:290] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:56.759639 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bd76933be5f240a1a5c069775527f562" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:56.760309 11444 raft_consensus.cc:3055] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:56.760605 11444 raft_consensus.cc:740] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5a1ce6e048ab47d69c453b5218cb64c1, State: Running, Role: LEADER
I20260709 12:26:56.761361 11444 consensus_queue.cc:260] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:56.762617 11444 raft_consensus.cc:3060] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:56.765756 11369 raft_consensus.cc:2468] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:56.767056 12122 leader_election.cc:304] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:56.767818 12437 raft_consensus.cc:2804] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:56.768268 12437 raft_consensus.cc:697] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:56.769223 12437 consensus_queue.cc:237] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:56.769745 11444 raft_consensus.cc:2468] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:56.780594 11194 catalog_manager.cc:5697] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 1 to 2, leader changed from 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 2 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:56.855690 11512 raft_consensus.cc:1275] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 3 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:56.857023 12415 consensus_queue.cc:1048] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:56.876559 11446 raft_consensus.cc:1275] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Refusing update from remote peer 05b93ba4435f40fb97b54343c4fdd71e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:56.878173 12357 consensus_queue.cc:1048] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:56.893293 11373 raft_consensus.cc:1275] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:56.894920 12436 consensus_queue.cc:1048] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:56.920213 11512 raft_consensus.cc:1275] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:56.922012 12437 consensus_queue.cc:1048] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:57.062366 11512 raft_consensus.cc:1275] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:57.063607 12437 consensus_queue.cc:1048] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:57.077775 11369 raft_consensus.cc:1275] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:57.079270 12379 consensus_queue.cc:1048] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:57.257287 11512 raft_consensus.cc:1275] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:57.258545 12379 consensus_queue.cc:1048] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:57.268441 11444 raft_consensus.cc:1275] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:57.270071 12379 consensus_queue.cc:1048] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:57.344575 11373 raft_consensus.cc:1275] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:57.345800 12379 consensus_queue.cc:1048] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:57.362527 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:57.359269 11444 raft_consensus.cc:1275] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:57.373517 12379 consensus_queue.cc:1048] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:57.412009 11373 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8"
dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:53470
I20260709 12:26:57.412631 11373 raft_consensus.cc:606] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:57.415278 11512 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3"
dest_uuid: "9968c0ed1974454d828996824f859433"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:34516
I20260709 12:26:57.415846 11512 raft_consensus.cc:606] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:57.418416 11512 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d8586c90fc9848248b387776dc1f7db6"
dest_uuid: "9968c0ed1974454d828996824f859433"
mode: GRACEFUL
new_leader_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:34516
I20260709 12:26:57.419034 11512 raft_consensus.cc:606] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Received request to transfer leadership to TS 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:57.420059 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:26:57.421669 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:57.422501 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:57.798053 12363 raft_consensus.cc:993] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:57.798519 12462 raft_consensus.cc:1081] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:57.799983 12170 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50170
I20260709 12:26:57.800508 12170 raft_consensus.cc:493] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:57.800885 12170 raft_consensus.cc:3060] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:57.807750 12170 raft_consensus.cc:515] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:57.810336 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:57.810395 12170 leader_election.cc:290] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 3 election: Requested vote from peers 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253), 9968c0ed1974454d828996824f859433 (127.29.201.196:34569)
I20260709 12:26:57.811056 11444 raft_consensus.cc:3060] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:57.811472 11515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8da681b4f1a47ff8327c26e796ad3c3" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:57.812127 11515 raft_consensus.cc:3055] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:26:57.812431 11515 raft_consensus.cc:740] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:57.813267 11515 consensus_queue.cc:260] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:57.814397 11515 raft_consensus.cc:3060] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:57.818619 11444 raft_consensus.cc:2468] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 3.
I20260709 12:26:57.820132 12121 leader_election.cc:304] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5a1ce6e048ab47d69c453b5218cb64c1, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:57.821040 12379 raft_consensus.cc:2804] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:26:57.821578 12379 raft_consensus.cc:697] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 3 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:57.821338 11515 raft_consensus.cc:2468] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 3.
I20260709 12:26:57.822522 12379 consensus_queue.cc:237] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } }
I20260709 12:26:57.833684 12461 raft_consensus.cc:993] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:57.834278 12461 raft_consensus.cc:1081] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:57.834542 12363 raft_consensus.cc:993] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433: : Instructing follower 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:57.835264 12363 raft_consensus.cc:1081] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Signalling peer 663d88d9634e4720a695c7cd69eaa25e to start an election
I20260709 12:26:57.837174 12164 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50600
I20260709 12:26:57.837723 12164 raft_consensus.cc:493] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:57.838070 12164 raft_consensus.cc:3060] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:57.843793 11196 catalog_manager.cc:5697] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 2 to 3, leader changed from 9968c0ed1974454d828996824f859433 (127.29.201.196) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 3 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } }
I20260709 12:26:57.846917 12164 raft_consensus.cc:515] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:57.849553 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:57.850157 11373 raft_consensus.cc:3055] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:26:57.850481 11373 raft_consensus.cc:740] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 05b93ba4435f40fb97b54343c4fdd71e, State: Running, Role: LEADER
I20260709 12:26:57.851346 11373 consensus_queue.cc:260] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:57.852043 12170 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d8586c90fc9848248b387776dc1f7db6"
dest_uuid: "663d88d9634e4720a695c7cd69eaa25e"
 from {username='slave'} at 127.0.0.1:50170
I20260709 12:26:57.852484 11373 raft_consensus.cc:3060] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:57.852854 12170 raft_consensus.cc:493] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:26:57.853242 12170 raft_consensus.cc:3060] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:57.861486 11373 raft_consensus.cc:2468] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 3.
I20260709 12:26:57.863276 12122 leader_election.cc:304] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e; no voters: 
I20260709 12:26:57.864929 12379 raft_consensus.cc:2804] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:26:57.865451 12379 raft_consensus.cc:697] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 3 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:57.866359 12379 consensus_queue.cc:237] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } }
I20260709 12:26:57.870968 12164 leader_election.cc:290] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 3 election: Requested vote from peers 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567), 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253)
I20260709 12:26:57.871165 11444 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "50c3c1a7849646fc95a663b6cd0427e8" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5a1ce6e048ab47d69c453b5218cb64c1"
I20260709 12:26:57.871981 11444 raft_consensus.cc:3060] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:26:57.907431 12170 raft_consensus.cc:515] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:57.910555 11515 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8586c90fc9848248b387776dc1f7db6" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9968c0ed1974454d828996824f859433"
I20260709 12:26:57.911206 11515 raft_consensus.cc:3055] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:26:57.911420 11515 raft_consensus.cc:740] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9968c0ed1974454d828996824f859433, State: Running, Role: LEADER
I20260709 12:26:57.912348 11515 consensus_queue.cc:260] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:57.913411 11515 raft_consensus.cc:3060] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:57.919521 12170 leader_election.cc:290] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Requested vote from peers 9968c0ed1974454d828996824f859433 (127.29.201.196:34569), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:57.920941 11444 raft_consensus.cc:2468] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 3.
I20260709 12:26:57.924340 11373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d8586c90fc9848248b387776dc1f7db6" candidate_uuid: "663d88d9634e4720a695c7cd69eaa25e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "05b93ba4435f40fb97b54343c4fdd71e"
I20260709 12:26:57.925019 11373 raft_consensus.cc:3060] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:26:57.925382 11515 raft_consensus.cc:2468] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:57.926478 12121 leader_election.cc:304] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 663d88d9634e4720a695c7cd69eaa25e, 9968c0ed1974454d828996824f859433; no voters: 
I20260709 12:26:57.931031 12437 raft_consensus.cc:2804] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:26:57.931566 12437 raft_consensus.cc:697] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Becoming Leader. State: Replica: 663d88d9634e4720a695c7cd69eaa25e, State: Running, Role: LEADER
I20260709 12:26:57.932410 12437 consensus_queue.cc:237] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:57.934855 11373 raft_consensus.cc:2468] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 663d88d9634e4720a695c7cd69eaa25e in term 2.
I20260709 12:26:57.939819 11196 catalog_manager.cc:5697] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 2 to 3, leader changed from 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 3 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:57.956529 11196 catalog_manager.cc:5697] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e reported cstate change: term changed from 1 to 2, leader changed from 9968c0ed1974454d828996824f859433 (127.29.201.196) to 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193). New cstate: current_term: 2 leader_uuid: "663d88d9634e4720a695c7cd69eaa25e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } health_report { overall_health: UNKNOWN } } }
I20260709 12:26:58.277460 11512 raft_consensus.cc:1275] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 3 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:58.278826 12437 consensus_queue.cc:1048] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:58.290210 11446 raft_consensus.cc:1275] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:58.291798 12437 consensus_queue.cc:1048] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:58.389731 11373 raft_consensus.cc:1275] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:58.391047 12437 consensus_queue.cc:1048] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:26:58.404013 11512 raft_consensus.cc:1275] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:26:58.406150 12437 consensus_queue.cc:1048] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:26:58.427448 11373 raft_consensus.cc:1275] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:58.433475 12437 consensus_queue.cc:1048] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:26:58.444931 11446 raft_consensus.cc:1275] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Refusing update from remote peer 663d88d9634e4720a695c7cd69eaa25e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:26:58.446446 12437 consensus_queue.cc:1048] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a1ce6e048ab47d69c453b5218cb64c1" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:26:58.465708 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:58.518834 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:26:58.520314 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:26:58.520987 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:58.578460 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:26:58.623209 30503 tablet_server.cc:179] TabletServer@127.29.201.193:37401 shutting down...
W20260709 12:26:58.631507 11338 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 105 similar messages]
W20260709 12:26:58.636087 11338 consensus_peers.cc:597] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:58.640576 11338 consensus_peers.cc:597] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:58.644066 11338 consensus_peers.cc:597] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:58.650285 11486 consensus_peers.cc:597] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:58.657186 11486 consensus_peers.cc:597] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.668597 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:58.670324 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.671337 30503 raft_consensus.cc:2243] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.671962 30503 raft_consensus.cc:2272] T 359eb877601e4d519d6e7d6f3835b9a2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:58.672725 11486 consensus_peers.cc:597] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.674870 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.675772 30503 raft_consensus.cc:2243] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.676363 30503 raft_consensus.cc:2272] T b4438988b178419385f7ec82ef7c99d2 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.679296 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.680017 30503 raft_consensus.cc:2243] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:26:58.680932 30503 raft_consensus.cc:2272] T 50c3c1a7849646fc95a663b6cd0427e8 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.684545 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.685272 30503 raft_consensus.cc:2243] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.685840 30503 raft_consensus.cc:2272] T 3701a37fd75248f6b8b0ca0930791a9d P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:58.688161 11486 consensus_peers.cc:597] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.689328 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.690284 30503 raft_consensus.cc:2243] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.690800 30503 raft_consensus.cc:2272] T 0081c481615e480ea8dfafe6a806ee7f P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.693912 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.694943 30503 raft_consensus.cc:2243] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.698797 30503 raft_consensus.cc:2272] T f74457716f0d46568488137f1cb73fa8 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.701983 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.703471 30503 raft_consensus.cc:2243] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.704106 30503 raft_consensus.cc:2272] T 8471ace194cf46238a8cd2862aa8cd5d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:58.704454 11338 consensus_peers.cc:597] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.707337 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.708458 30503 raft_consensus.cc:2243] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.709069 30503 raft_consensus.cc:2272] T e57eb43303724eae812867b69fa94d9d P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.712661 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.714093 30503 raft_consensus.cc:2243] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.714877 30503 raft_consensus.cc:2272] T c8f2372474d94a11bb1e044fc69c60e5 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.718895 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.720062 30503 raft_consensus.cc:2243] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.720656 30503 raft_consensus.cc:2272] T 610a732cba674c24bb228d619f52fce7 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.723894 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.724853 30503 raft_consensus.cc:2243] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:26:58.726073 30503 raft_consensus.cc:2272] T b8da681b4f1a47ff8327c26e796ad3c3 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.730893 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.731593 30503 raft_consensus.cc:2243] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.732154 30503 raft_consensus.cc:2272] T 0175551ed6284d01b80e6a34e24a07ed P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.734849 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.735710 30503 raft_consensus.cc:2243] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.736217 30503 raft_consensus.cc:2272] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:58.737112 11486 consensus_peers.cc:597] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.739223 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.740010 30503 raft_consensus.cc:2243] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:26:58.741130 30503 raft_consensus.cc:2272] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:58.742355 11486 consensus_peers.cc:597] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.744022 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.744951 30503 raft_consensus.cc:2243] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.745585 30503 raft_consensus.cc:2272] T 70f75f882db94790be4eaf2994c44f61 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.748087 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.748701 30503 raft_consensus.cc:2243] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.749109 30503 raft_consensus.cc:2272] T 57d6a333ba744c8389730bd74362cb43 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.751497 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.752336 30503 raft_consensus.cc:2243] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:58.753262 30503 raft_consensus.cc:2272] T 827a79573ac94525a9ff30c744c45164 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.755752 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.756317 30503 raft_consensus.cc:2243] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:58.757071 30503 raft_consensus.cc:2272] T 7179cfd71f5c4bcf8beac36a18e4c94e P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.759809 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.760524 30503 raft_consensus.cc:2243] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:58.761605 30503 raft_consensus.cc:2272] T 6788efa5ae124a478e06ef927cb631ff P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:58.763736 11412 consensus_peers.cc:597] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.764881 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.765815 30503 raft_consensus.cc:2243] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 12:26:58.766052 11338 consensus_peers.cc:597] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.766444 30503 raft_consensus.cc:2272] T 16773253264a4f8282ea1ea5b59a1ec6 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.769037 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.769784 30503 raft_consensus.cc:2243] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.770285 30503 raft_consensus.cc:2272] T 6c4cc7481ed24aa6991531d818969460 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.772989 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.773610 30503 raft_consensus.cc:2243] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.774016 30503 raft_consensus.cc:2272] T bfa4ec6799b5454a9d750ba5a79fafa3 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.776302 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.776885 30503 raft_consensus.cc:2243] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.777293 30503 raft_consensus.cc:2272] T bccc5d3239f3484eb991213e177b66cc P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.779559 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.780448 30503 raft_consensus.cc:2243] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.781036 30503 raft_consensus.cc:2272] T cc185f43f658441ba207bdfdc0e79a7f P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.784108 30503 tablet_replica.cc:333] stopping tablet replica
W20260709 12:26:58.784492 11486 consensus_peers.cc:597] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.784936 30503 raft_consensus.cc:2243] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:26:58.785830 30503 raft_consensus.cc:2272] T 86bb3bcc0a6449a9adc6c21b722f6602 P 663d88d9634e4720a695c7cd69eaa25e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.788686 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.789424 30503 raft_consensus.cc:2243] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.789943 30503 raft_consensus.cc:2272] T e9836ae3a7d04d53ab2d78656c19c8f2 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.792459 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.793048 30503 raft_consensus.cc:2243] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.793454 30503 raft_consensus.cc:2272] T 0c068ba8213a415b9df4e5380d0ad37e P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.795583 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.796172 30503 raft_consensus.cc:2243] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:58.796949 30503 raft_consensus.cc:2272] T db58e810d4a04064bbf12daef58026f3 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.799261 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.799850 30503 raft_consensus.cc:2243] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.800283 30503 raft_consensus.cc:2272] T 47908558560f437a92445e6ab3a30a57 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.802395 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.803040 30503 raft_consensus.cc:2243] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.803493 30503 raft_consensus.cc:2272] T 1cf22a8b3fa4491c92c735f8151982d3 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.806559 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.807469 30503 raft_consensus.cc:2243] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:58.808554 30503 raft_consensus.cc:2272] T ff00b27947404e52acea283e9381d837 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.811584 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.812362 30503 raft_consensus.cc:2243] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:26:58.813187 11486 consensus_peers.cc:597] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.813341 30503 raft_consensus.cc:2272] T fea9a20aab6840c1afb32c29bac2f4a4 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.815914 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.816557 30503 raft_consensus.cc:2243] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:58.817329 30503 raft_consensus.cc:2272] T bd76933be5f240a1a5c069775527f562 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.819687 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.820478 30503 raft_consensus.cc:2243] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.821022 30503 raft_consensus.cc:2272] T 1d019dd2bd864166abac89cc3fb1be76 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.824174 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.824972 30503 raft_consensus.cc:2243] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.825363 30503 raft_consensus.cc:2272] T 6d4384a4da6846e39db4bcd0330cb822 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.827479 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.828056 30503 raft_consensus.cc:2243] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.828433 30503 raft_consensus.cc:2272] T 2925eddaf8944bf5bf72cd650ed1e287 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.830677 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.831374 30503 raft_consensus.cc:2243] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:26:58.832260 30503 raft_consensus.cc:2272] T fa781c13298b41e2ba513a74a2455086 P 663d88d9634e4720a695c7cd69eaa25e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.835027 30503 tablet_replica.cc:333] stopping tablet replica
W20260709 12:26:58.835165 11412 consensus_peers.cc:597] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.835788 30503 raft_consensus.cc:2243] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.836344 30503 raft_consensus.cc:2272] T 8236893668af4eaa93bc53b6715f6cf0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.838550 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.839160 30503 raft_consensus.cc:2243] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:58.840202 30503 raft_consensus.cc:2272] T d8586c90fc9848248b387776dc1f7db6 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.843152 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.843770 30503 raft_consensus.cc:2243] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.844177 30503 raft_consensus.cc:2272] T 32c7abdd125b4f38b6398a9b0e92fedd P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.846292 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.846966 30503 raft_consensus.cc:2243] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.847384 30503 raft_consensus.cc:2272] T 5734e27937894275a18ba504ffedd114 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.849800 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.850435 30503 raft_consensus.cc:2243] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.850975 30503 raft_consensus.cc:2272] T 578fd826b687468ca8be54244c3b4bc0 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.853737 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.854250 30503 raft_consensus.cc:2243] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:58.855087 30503 raft_consensus.cc:2272] T 1ce080ae4c98415a868195fb1df17f18 P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:58.856355 11412 consensus_peers.cc:597] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.857307 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.857925 30503 raft_consensus.cc:2243] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.858355 30503 raft_consensus.cc:2272] T c91074436b5b44779742b3478249f283 P 663d88d9634e4720a695c7cd69eaa25e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:58.860685 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:26:58.861178 30503 raft_consensus.cc:2243] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:58.861698 30503 raft_consensus.cc:2272] T fee758b4d91541148427a240c8f1db4e P 663d88d9634e4720a695c7cd69eaa25e [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:58.907536 11338 consensus_peers.cc:597] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:58.933190 30503 tablet_server.cc:196] TabletServer@127.29.201.193:37401 shutdown complete.
W20260709 12:26:58.939602 11412 consensus_peers.cc:597] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:58.944977 11338 consensus_peers.cc:597] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:58.951998 11486 consensus_peers.cc:597] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:58.955500 11486 consensus_peers.cc:597] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:58.958801 11412 consensus_peers.cc:597] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:58.982020 11412 consensus_peers.cc:597] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.011400 11412 consensus_peers.cc:597] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.019450 11412 consensus_peers.cc:597] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.021781 11486 consensus_peers.cc:597] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.028800 11412 consensus_peers.cc:597] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.042515 11338 consensus_peers.cc:597] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.046411 11486 consensus_peers.cc:597] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.050434 30503 tablet_server.cc:179] TabletServer@127.29.201.194:0 shutting down...
W20260709 12:26:59.075623 11411 consensus_peers.cc:597] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.092211 11485 consensus_peers.cc:597] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.095050 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:59.096035 30503 tablet_replica.cc:333] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.096943 30503 raft_consensus.cc:2243] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.097577 30503 raft_consensus.cc:2272] T 359eb877601e4d519d6e7d6f3835b9a2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.101470 30503 tablet_replica.cc:333] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.102387 30503 raft_consensus.cc:2243] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.102981 30503 raft_consensus.cc:2272] T 86bb3bcc0a6449a9adc6c21b722f6602 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.106331 30503 tablet_replica.cc:333] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.107380 30503 raft_consensus.cc:2243] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.108837 30503 raft_consensus.cc:2272] T 32c7abdd125b4f38b6398a9b0e92fedd P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:59.109118 11485 consensus_peers.cc:597] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.111840 30503 tablet_replica.cc:333] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
W20260709 12:26:59.111922 11485 consensus_peers.cc:597] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.112895 30503 raft_consensus.cc:2243] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.113957 30503 raft_consensus.cc:2272] T 431c66d13d0f4904a42b33665a9c09e1 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.117556 30503 tablet_replica.cc:333] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.118611 30503 raft_consensus.cc:2243] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.119781 30503 raft_consensus.cc:2272] T 610a732cba674c24bb228d619f52fce7 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.122725 30503 tablet_replica.cc:333] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.123533 30503 raft_consensus.cc:2243] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.124125 30503 raft_consensus.cc:2272] T f74457716f0d46568488137f1cb73fa8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.126495 30503 tablet_replica.cc:333] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.127156 30503 raft_consensus.cc:2243] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.127637 30503 raft_consensus.cc:2272] T 019a6ed4536442de96c44bde2d713c8b P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.129781 30503 tablet_replica.cc:333] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.130416 30503 raft_consensus.cc:2243] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.130908 30503 raft_consensus.cc:2272] T 50c3c1a7849646fc95a663b6cd0427e8 P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.133958 30503 tablet_replica.cc:333] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.134598 30503 raft_consensus.cc:2243] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.135113 30503 raft_consensus.cc:2272] T e637809da62b408a91639b907355cad2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.137107 30503 tablet_replica.cc:333] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.137717 30503 raft_consensus.cc:2243] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.138553 30503 raft_consensus.cc:2272] T 91b161cce9ce49c58571f631c6810266 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.140895 30503 tablet_replica.cc:333] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.141417 30503 raft_consensus.cc:2243] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.141906 30503 raft_consensus.cc:2272] T 5eda5b4099d041c9aef56884ca4e594a P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.144912 30503 tablet_replica.cc:333] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.145931 30503 raft_consensus.cc:2243] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.146476 30503 raft_consensus.cc:2272] T 1ce080ae4c98415a868195fb1df17f18 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:59.147100 11411 consensus_peers.cc:597] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.148085 11338 consensus_peers.cc:597] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.149420 30503 tablet_replica.cc:333] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
W20260709 12:26:59.149791 11411 consensus_peers.cc:597] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.150252 30503 raft_consensus.cc:2243] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.150810 30503 raft_consensus.cc:2272] T ff00b27947404e52acea283e9381d837 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.152832 30503 tablet_replica.cc:333] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.153617 30503 raft_consensus.cc:2243] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.154170 30503 raft_consensus.cc:2272] T 1cf22a8b3fa4491c92c735f8151982d3 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.156690 30503 tablet_replica.cc:333] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.157235 30503 raft_consensus.cc:2243] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.157986 30503 raft_consensus.cc:2272] T b4438988b178419385f7ec82ef7c99d2 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.160049 30503 tablet_replica.cc:333] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.160584 30503 raft_consensus.cc:2243] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.160953 30503 raft_consensus.cc:2272] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.162905 30503 tablet_replica.cc:333] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.163492 30503 raft_consensus.cc:2243] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.164250 30503 raft_consensus.cc:2272] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.166101 30503 tablet_replica.cc:333] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.166706 30503 raft_consensus.cc:2243] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.167140 30503 raft_consensus.cc:2272] T 578fd826b687468ca8be54244c3b4bc0 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.170055 30503 tablet_replica.cc:333] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.170868 30503 raft_consensus.cc:2243] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.171386 30503 raft_consensus.cc:2272] T 75195e7d58a548658a538f4628a056d8 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.174018 30503 tablet_replica.cc:333] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.174788 30503 raft_consensus.cc:2243] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.175318 30503 raft_consensus.cc:2272] T ee281c78e97848f59386b1d97f96bcfe P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.178011 30503 tablet_replica.cc:333] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.178596 30503 raft_consensus.cc:2243] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.179473 30503 raft_consensus.cc:2272] T fee758b4d91541148427a240c8f1db4e P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.181649 30503 tablet_replica.cc:333] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.182235 30503 raft_consensus.cc:2243] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.182750 30503 raft_consensus.cc:2272] T bd76933be5f240a1a5c069775527f562 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.185182 30503 tablet_replica.cc:333] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.185906 30503 raft_consensus.cc:2243] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.186437 30503 raft_consensus.cc:2272] T fea9a20aab6840c1afb32c29bac2f4a4 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:59.188979 11412 consensus_peers.cc:597] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Couldn't send request to peer 663d88d9634e4720a695c7cd69eaa25e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.189159 30503 tablet_replica.cc:333] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.190192 30503 raft_consensus.cc:2243] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.190797 30503 raft_consensus.cc:2272] T b0c4260f7779463dab087f51bf433ab8 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.193890 30503 tablet_replica.cc:333] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
W20260709 12:26:59.194607 11485 consensus_peers.cc:597] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.194720 30503 raft_consensus.cc:2243] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.196049 30503 raft_consensus.cc:2272] T e9836ae3a7d04d53ab2d78656c19c8f2 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.198892 30503 tablet_replica.cc:333] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.199730 30503 raft_consensus.cc:2243] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.200711 30503 raft_consensus.cc:2272] T 267085844c6943b08c0f42b220e7e8ab P 05b93ba4435f40fb97b54343c4fdd71e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.203689 30503 tablet_replica.cc:333] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.204720 30503 raft_consensus.cc:2243] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.205513 30503 raft_consensus.cc:2272] T e57eb43303724eae812867b69fa94d9d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.208295 30503 tablet_replica.cc:333] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.209123 30503 raft_consensus.cc:2243] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.209646 30503 raft_consensus.cc:2272] T fa781c13298b41e2ba513a74a2455086 P 05b93ba4435f40fb97b54343c4fdd71e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.212407 30503 tablet_replica.cc:333] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.213014 30503 raft_consensus.cc:2243] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.213460 30503 raft_consensus.cc:2272] T 8236893668af4eaa93bc53b6715f6cf0 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.215500 30503 tablet_replica.cc:333] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.216122 30503 raft_consensus.cc:2243] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.216548 30503 raft_consensus.cc:2272] T d8586c90fc9848248b387776dc1f7db6 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.219008 30503 tablet_replica.cc:333] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.219534 30503 raft_consensus.cc:2243] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.220317 30503 raft_consensus.cc:2272] T 0175551ed6284d01b80e6a34e24a07ed P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.222370 30503 tablet_replica.cc:333] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.222970 30503 raft_consensus.cc:2243] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.223431 30503 raft_consensus.cc:2272] T 1d019dd2bd864166abac89cc3fb1be76 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.225881 30503 tablet_replica.cc:333] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.226699 30503 raft_consensus.cc:2243] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.227190 30503 raft_consensus.cc:2272] T 6d4384a4da6846e39db4bcd0330cb822 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:59.228051 11485 consensus_peers.cc:597] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.229781 30503 tablet_replica.cc:333] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.230592 30503 raft_consensus.cc:2243] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.231735 30503 raft_consensus.cc:2272] T 393b5b835714456ab06eeecb9b4a116d P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.234004 30503 tablet_replica.cc:333] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.234542 30503 raft_consensus.cc:2243] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.235148 30503 raft_consensus.cc:2272] T 3701a37fd75248f6b8b0ca0930791a9d P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.237126 30503 tablet_replica.cc:333] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.237730 30503 raft_consensus.cc:2243] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.238588 30503 raft_consensus.cc:2272] T 4a5d23b0089f4d37a0c63d127537ec93 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.241128 30503 tablet_replica.cc:333] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.241663 30503 raft_consensus.cc:2243] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.242110 30503 raft_consensus.cc:2272] T 7179cfd71f5c4bcf8beac36a18e4c94e P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:59.242930 11411 consensus_peers.cc:597] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.244284 30503 tablet_replica.cc:333] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.244932 30503 raft_consensus.cc:2243] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.245374 30503 raft_consensus.cc:2272] T 57d6a333ba744c8389730bd74362cb43 P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.247453 30503 tablet_replica.cc:333] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.248023 30503 raft_consensus.cc:2243] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.248432 30503 raft_consensus.cc:2272] T e313f6102e9941248ba9069990945017 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.250502 30503 tablet_replica.cc:333] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.251070 30503 raft_consensus.cc:2243] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.251468 30503 raft_consensus.cc:2272] T db58e810d4a04064bbf12daef58026f3 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.253491 30503 tablet_replica.cc:333] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.254166 30503 raft_consensus.cc:2243] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.255232 30503 raft_consensus.cc:2272] T 43ed653326ff4ae6b3c7daa594b3228b P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.258054 30503 tablet_replica.cc:333] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.258666 30503 raft_consensus.cc:2243] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.259475 30503 raft_consensus.cc:2272] T 16773253264a4f8282ea1ea5b59a1ec6 P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.261507 30503 tablet_replica.cc:333] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.261997 30503 raft_consensus.cc:2243] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.262409 30503 raft_consensus.cc:2272] T 6788efa5ae124a478e06ef927cb631ff P 05b93ba4435f40fb97b54343c4fdd71e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.264683 30503 tablet_replica.cc:333] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e: stopping tablet replica
I20260709 12:26:59.265290 30503 raft_consensus.cc:2243] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.265816 30503 raft_consensus.cc:2272] T f32b983bac2949418ec7a6bf8a0ecdea P 05b93ba4435f40fb97b54343c4fdd71e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:59.320113 11485 consensus_peers.cc:597] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.323118 11411 consensus_peers.cc:597] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.348881 30503 tablet_server.cc:196] TabletServer@127.29.201.194:0 shutdown complete.
W20260709 12:26:59.349442 11411 consensus_peers.cc:597] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:44567: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.358151 11485 consensus_peers.cc:597] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:44567: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.444692 30503 tablet_server.cc:179] TabletServer@127.29.201.195:0 shutting down...
W20260709 12:26:59.456710 11485 consensus_peers.cc:597] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:44567: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.462458 11411 consensus_peers.cc:597] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:44567: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.472070 11484 consensus_peers.cc:597] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 -> Peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Couldn't send request to peer 5a1ce6e048ab47d69c453b5218cb64c1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.476287 11411 consensus_peers.cc:597] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 -> Peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Couldn't send request to peer 05b93ba4435f40fb97b54343c4fdd71e. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:44567: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.504577 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:59.505599 30503 tablet_replica.cc:333] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.506804 30503 raft_consensus.cc:2243] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.507319 30503 raft_consensus.cc:2272] T 431c66d13d0f4904a42b33665a9c09e1 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.509366 30503 tablet_replica.cc:333] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.509963 30503 raft_consensus.cc:2243] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.510356 30503 raft_consensus.cc:2272] T bccc5d3239f3484eb991213e177b66cc P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.512454 30503 tablet_replica.cc:333] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.513399 30503 raft_consensus.cc:2243] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.513815 30503 raft_consensus.cc:2272] T 50c3c1a7849646fc95a663b6cd0427e8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.515892 30503 tablet_replica.cc:333] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.516505 30503 raft_consensus.cc:2243] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.516907 30503 raft_consensus.cc:2272] T 86bb3bcc0a6449a9adc6c21b722f6602 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.518790 30503 tablet_replica.cc:333] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.519706 30503 raft_consensus.cc:2243] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.520371 30503 raft_consensus.cc:2272] T cc185f43f658441ba207bdfdc0e79a7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.522732 30503 tablet_replica.cc:333] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.523340 30503 raft_consensus.cc:2243] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.525915 30503 raft_consensus.cc:2272] T 5eda5b4099d041c9aef56884ca4e594a P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.528260 30503 tablet_replica.cc:333] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.528851 30503 raft_consensus.cc:2243] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.529295 30503 raft_consensus.cc:2272] T e637809da62b408a91639b907355cad2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.531571 30503 tablet_replica.cc:333] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.532338 30503 raft_consensus.cc:2243] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.532842 30503 raft_consensus.cc:2272] T 91b161cce9ce49c58571f631c6810266 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.534778 30503 tablet_replica.cc:333] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.535346 30503 raft_consensus.cc:2243] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.535778 30503 raft_consensus.cc:2272] T b8da681b4f1a47ff8327c26e796ad3c3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.537557 30503 tablet_replica.cc:333] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.538115 30503 raft_consensus.cc:2243] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.538537 30503 raft_consensus.cc:2272] T 8471ace194cf46238a8cd2862aa8cd5d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.541023 30503 tablet_replica.cc:333] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.541905 30503 raft_consensus.cc:2243] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.542502 30503 raft_consensus.cc:2272] T b3c0ce8bbe4f4392bdac15d4d35f84dd P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.545259 30503 tablet_replica.cc:333] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.546216 30503 raft_consensus.cc:2243] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.546883 30503 raft_consensus.cc:2272] T 8b0f568d1ffe4160afdf9d1bb4ba7c42 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.549269 30503 tablet_replica.cc:333] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.549872 30503 raft_consensus.cc:2243] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.550981 30503 raft_consensus.cc:2272] T 47908558560f437a92445e6ab3a30a57 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.553936 30503 tablet_replica.cc:333] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.554859 30503 raft_consensus.cc:2243] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.555933 30503 raft_consensus.cc:2272] T 0c068ba8213a415b9df4e5380d0ad37e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.558020 30503 tablet_replica.cc:333] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.558620 30503 raft_consensus.cc:2243] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.559144 30503 raft_consensus.cc:2272] T e9836ae3a7d04d53ab2d78656c19c8f2 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.561889 30503 tablet_replica.cc:333] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.562458 30503 raft_consensus.cc:2243] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.562937 30503 raft_consensus.cc:2272] T 267085844c6943b08c0f42b220e7e8ab P 5a1ce6e048ab47d69c453b5218cb64c1 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.564870 30503 tablet_replica.cc:333] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.565506 30503 raft_consensus.cc:2243] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.566367 30503 raft_consensus.cc:2272] T 75195e7d58a548658a538f4628a056d8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.568437 30503 tablet_replica.cc:333] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.569016 30503 raft_consensus.cc:2243] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.569903 30503 raft_consensus.cc:2272] T ee281c78e97848f59386b1d97f96bcfe P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.572110 30503 tablet_replica.cc:333] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.572808 30503 raft_consensus.cc:2243] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.573262 30503 raft_consensus.cc:2272] T bd76933be5f240a1a5c069775527f562 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.575380 30503 tablet_replica.cc:333] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.576036 30503 raft_consensus.cc:2243] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.576472 30503 raft_consensus.cc:2272] T fea9a20aab6840c1afb32c29bac2f4a4 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.578856 30503 tablet_replica.cc:333] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.579501 30503 raft_consensus.cc:2243] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.580295 30503 raft_consensus.cc:2272] T 3701a37fd75248f6b8b0ca0930791a9d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.584301 30503 tablet_replica.cc:333] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.584923 30503 raft_consensus.cc:2243] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.585386 30503 raft_consensus.cc:2272] T 4a5d23b0089f4d37a0c63d127537ec93 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.587571 30503 tablet_replica.cc:333] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.588219 30503 raft_consensus.cc:2243] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.588832 30503 raft_consensus.cc:2272] T 2925eddaf8944bf5bf72cd650ed1e287 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.591581 30503 tablet_replica.cc:333] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.592355 30503 raft_consensus.cc:2243] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.592929 30503 raft_consensus.cc:2272] T 70f75f882db94790be4eaf2994c44f61 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:59.595546 11484 consensus_peers.cc:597] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 -> Peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Couldn't send request to peer 5a1ce6e048ab47d69c453b5218cb64c1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.595679 30503 tablet_replica.cc:333] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.596496 30503 raft_consensus.cc:2243] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.596952 30503 raft_consensus.cc:2272] T 7179cfd71f5c4bcf8beac36a18e4c94e P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.599233 30503 tablet_replica.cc:333] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.599828 30503 raft_consensus.cc:2243] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.600288 30503 raft_consensus.cc:2272] T 827a79573ac94525a9ff30c744c45164 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.602314 30503 tablet_replica.cc:333] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.602952 30503 raft_consensus.cc:2243] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.603804 30503 raft_consensus.cc:2272] T 57d6a333ba744c8389730bd74362cb43 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.606233 30503 tablet_replica.cc:333] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.606889 30503 raft_consensus.cc:2243] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.607741 30503 raft_consensus.cc:2272] T e313f6102e9941248ba9069990945017 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.610316 30503 tablet_replica.cc:333] T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.611128 30503 raft_consensus.cc:2243] T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.611723 30503 raft_consensus.cc:2272] T 019a6ed4536442de96c44bde2d713c8b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.614511 30503 tablet_replica.cc:333] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.615311 30503 raft_consensus.cc:2243] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.616204 30503 raft_consensus.cc:2272] T f74457716f0d46568488137f1cb73fa8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.618383 30503 tablet_replica.cc:333] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.618993 30503 raft_consensus.cc:2243] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.619427 30503 raft_consensus.cc:2272] T 0081c481615e480ea8dfafe6a806ee7f P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.621821 30503 tablet_replica.cc:333] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.622406 30503 raft_consensus.cc:2243] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.622886 30503 raft_consensus.cc:2272] T b0c4260f7779463dab087f51bf433ab8 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.625020 30503 tablet_replica.cc:333] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.625627 30503 raft_consensus.cc:2243] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.626523 30503 raft_consensus.cc:2272] T bfa4ec6799b5454a9d750ba5a79fafa3 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.629380 30503 tablet_replica.cc:333] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.630205 30503 raft_consensus.cc:2243] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.631259 30503 raft_consensus.cc:2272] T f32b983bac2949418ec7a6bf8a0ecdea P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.633913 30503 tablet_replica.cc:333] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.634928 30503 raft_consensus.cc:2243] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.636061 30503 raft_consensus.cc:2272] T 6c4cc7481ed24aa6991531d818969460 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.638849 30503 tablet_replica.cc:333] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.639703 30503 raft_consensus.cc:2243] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.640312 30503 raft_consensus.cc:2272] T 0175551ed6284d01b80e6a34e24a07ed P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.643177 30503 tablet_replica.cc:333] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.643986 30503 raft_consensus.cc:2243] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.644579 30503 raft_consensus.cc:2272] T 16773253264a4f8282ea1ea5b59a1ec6 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.647078 30503 tablet_replica.cc:333] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.647868 30503 raft_consensus.cc:2243] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.648432 30503 raft_consensus.cc:2272] T 6788efa5ae124a478e06ef927cb631ff P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.651197 30503 tablet_replica.cc:333] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.653117 30503 raft_consensus.cc:2243] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.653609 30503 raft_consensus.cc:2272] T c8f2372474d94a11bb1e044fc69c60e5 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:59.653750 11484 consensus_peers.cc:597] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 -> Peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Couldn't send request to peer 5a1ce6e048ab47d69c453b5218cb64c1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.655184 11484 consensus_peers.cc:597] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 -> Peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Couldn't send request to peer 5a1ce6e048ab47d69c453b5218cb64c1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.656102 30503 tablet_replica.cc:333] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.656725 30503 raft_consensus.cc:2243] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.657155 30503 raft_consensus.cc:2272] T 43ed653326ff4ae6b3c7daa594b3228b P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.659361 30503 tablet_replica.cc:333] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.659909 30503 raft_consensus.cc:2243] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.660722 30503 raft_consensus.cc:2272] T c91074436b5b44779742b3478249f283 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.662827 30503 tablet_replica.cc:333] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.663336 30503 raft_consensus.cc:2243] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.664104 30503 raft_consensus.cc:2272] T 8236893668af4eaa93bc53b6715f6cf0 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.666245 30503 tablet_replica.cc:333] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.666795 30503 raft_consensus.cc:2243] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.667212 30503 raft_consensus.cc:2272] T 393b5b835714456ab06eeecb9b4a116d P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.669128 30503 tablet_replica.cc:333] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1: stopping tablet replica
I20260709 12:26:59.669606 30503 raft_consensus.cc:2243] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.670378 30503 raft_consensus.cc:2272] T 5734e27937894275a18ba504ffedd114 P 5a1ce6e048ab47d69c453b5218cb64c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.757928 30503 tablet_server.cc:196] TabletServer@127.29.201.195:0 shutdown complete.
W20260709 12:26:59.763345 11484 consensus_peers.cc:597] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 -> Peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Couldn't send request to peer 5a1ce6e048ab47d69c453b5218cb64c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.195:36253: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:26:59.795325 11484 consensus_peers.cc:597] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 -> Peer 5a1ce6e048ab47d69c453b5218cb64c1 (127.29.201.195:36253): Couldn't send request to peer 5a1ce6e048ab47d69c453b5218cb64c1. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.195:36253: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:26:59.846151 30503 tablet_server.cc:179] TabletServer@127.29.201.196:0 shutting down...
I20260709 12:26:59.891112 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:26:59.891901 30503 tablet_replica.cc:333] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.892583 30503 raft_consensus.cc:2243] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.893512 30503 raft_consensus.cc:2272] T 359eb877601e4d519d6e7d6f3835b9a2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.895826 30503 tablet_replica.cc:333] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.896472 30503 raft_consensus.cc:2243] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.897308 30503 raft_consensus.cc:2272] T bccc5d3239f3484eb991213e177b66cc P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.899549 30503 tablet_replica.cc:333] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.900202 30503 raft_consensus.cc:2243] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.900671 30503 raft_consensus.cc:2272] T 431c66d13d0f4904a42b33665a9c09e1 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.903327 12462 raft_consensus.cc:493] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:59.903427 30503 tablet_replica.cc:333] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.903821 12462 raft_consensus.cc:515] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:59.904330 30503 raft_consensus.cc:2243] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.904987 30503 raft_consensus.cc:2272] T 32c7abdd125b4f38b6398a9b0e92fedd P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.907145 12462 leader_election.cc:290] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:59.908730 30503 tablet_replica.cc:333] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.910183 30503 raft_consensus.cc:2243] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.911422 12488 raft_consensus.cc:493] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:59.911986 30503 raft_consensus.cc:2272] T cc185f43f658441ba207bdfdc0e79a7f P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.912031 12488 raft_consensus.cc:515] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
W20260709 12:26:59.913097 11486 leader_election.cc:336] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111)
W20260709 12:26:59.913897 11485 leader_election.cc:336] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Network error: Client connection negotiation failed: client connection to 127.29.201.194:44567: connect: Connection refused (error 111)
I20260709 12:26:59.914354 11485 leader_election.cc:304] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:59.914871 12488 leader_election.cc:290] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:59.915462 12488 raft_consensus.cc:2749] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:26:59.918035 30503 tablet_replica.cc:333] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.919107 30503 raft_consensus.cc:2243] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.920486 30503 raft_consensus.cc:2272] T e637809da62b408a91639b907355cad2 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:59.921696 11485 leader_election.cc:336] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Network error: Client connection negotiation failed: client connection to 127.29.201.194:44567: connect: Connection refused (error 111)
W20260709 12:26:59.922453 11486 leader_election.cc:336] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111)
I20260709 12:26:59.923020 11486 leader_election.cc:304] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:59.923753 30503 tablet_replica.cc:333] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.923806 12488 raft_consensus.cc:2749] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260709 12:26:59.924731 30503 raft_consensus.cc:2243] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.925338 30503 raft_consensus.cc:2272] T 91b161cce9ce49c58571f631c6810266 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.928416 30503 tablet_replica.cc:333] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.929155 30503 raft_consensus.cc:2243] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.929747 30503 raft_consensus.cc:2272] T 5eda5b4099d041c9aef56884ca4e594a P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.932060 30503 tablet_replica.cc:333] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.932651 30503 raft_consensus.cc:2243] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.933075 30503 raft_consensus.cc:2272] T 610a732cba674c24bb228d619f52fce7 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.935364 30503 tablet_replica.cc:333] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.936113 30503 raft_consensus.cc:2243] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.936739 30503 raft_consensus.cc:2272] T b8da681b4f1a47ff8327c26e796ad3c3 P 9968c0ed1974454d828996824f859433 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.939009 30503 tablet_replica.cc:333] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.939700 30503 raft_consensus.cc:2243] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.940152 12488 raft_consensus.cc:493] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 663d88d9634e4720a695c7cd69eaa25e)
I20260709 12:26:59.940366 30503 raft_consensus.cc:2272] T 1ce080ae4c98415a868195fb1df17f18 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.940869 12488 raft_consensus.cc:515] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9968c0ed1974454d828996824f859433" member_type: VOTER last_known_addr { host: "127.29.201.196" port: 34569 } } peers { permanent_uuid: "663d88d9634e4720a695c7cd69eaa25e" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 37401 } } peers { permanent_uuid: "05b93ba4435f40fb97b54343c4fdd71e" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 44567 } }
I20260709 12:26:59.943315 12488 leader_election.cc:290] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401), 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567)
I20260709 12:26:59.943625 30503 tablet_replica.cc:333] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.944664 30503 raft_consensus.cc:2243] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.945379 30503 raft_consensus.cc:2272] T b4438988b178419385f7ec82ef7c99d2 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:26:59.947409 11486 leader_election.cc:336] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 663d88d9634e4720a695c7cd69eaa25e (127.29.201.193:37401): Network error: Client connection negotiation failed: client connection to 127.29.201.193:37401: connect: Connection refused (error 111)
W20260709 12:26:59.948032 11485 leader_election.cc:336] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer 05b93ba4435f40fb97b54343c4fdd71e (127.29.201.194:44567): Network error: Client connection negotiation failed: client connection to 127.29.201.194:44567: connect: Connection refused (error 111)
I20260709 12:26:59.948477 11485 leader_election.cc:304] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9968c0ed1974454d828996824f859433; no voters: 05b93ba4435f40fb97b54343c4fdd71e, 663d88d9634e4720a695c7cd69eaa25e
I20260709 12:26:59.948700 30503 tablet_replica.cc:333] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.949348 12488 raft_consensus.cc:2749] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260709 12:26:59.949656 30503 raft_consensus.cc:2243] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.950348 30503 raft_consensus.cc:2272] T 75195e7d58a548658a538f4628a056d8 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.952751 30503 tablet_replica.cc:333] T ee281c78e97848f59386b1d97f96bcfe P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.953398 30503 raft_consensus.cc:2243] T ee281c78e97848f59386b1d97f96bcfe P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.953892 30503 raft_consensus.cc:2272] T ee281c78e97848f59386b1d97f96bcfe P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.956071 30503 tablet_replica.cc:333] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.956698 30503 raft_consensus.cc:2243] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.957176 30503 raft_consensus.cc:2272] T c91074436b5b44779742b3478249f283 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.960001 30503 tablet_replica.cc:333] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.960856 30503 raft_consensus.cc:2243] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.961509 30503 raft_consensus.cc:2272] T fee758b4d91541148427a240c8f1db4e P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.964272 30503 tablet_replica.cc:333] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.964865 30503 raft_consensus.cc:2243] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.965663 30503 raft_consensus.cc:2272] T b0c4260f7779463dab087f51bf433ab8 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.967758 30503 tablet_replica.cc:333] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.968379 30503 raft_consensus.cc:2243] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.969213 30503 raft_consensus.cc:2272] T 019a6ed4536442de96c44bde2d713c8b P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.971498 30503 tablet_replica.cc:333] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.972108 30503 raft_consensus.cc:2243] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.972966 30503 raft_consensus.cc:2272] T 0081c481615e480ea8dfafe6a806ee7f P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.975337 30503 tablet_replica.cc:333] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.975893 30503 raft_consensus.cc:2243] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.976343 30503 raft_consensus.cc:2272] T 4a5d23b0089f4d37a0c63d127537ec93 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.978399 30503 tablet_replica.cc:333] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.979117 30503 raft_consensus.cc:2243] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.979559 30503 raft_consensus.cc:2272] T fa781c13298b41e2ba513a74a2455086 P 9968c0ed1974454d828996824f859433 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.981611 30503 tablet_replica.cc:333] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.982190 30503 raft_consensus.cc:2243] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.983129 30503 raft_consensus.cc:2272] T 2925eddaf8944bf5bf72cd650ed1e287 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.985312 30503 tablet_replica.cc:333] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.985864 30503 raft_consensus.cc:2243] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.986805 30503 raft_consensus.cc:2272] T 70f75f882db94790be4eaf2994c44f61 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.988864 30503 tablet_replica.cc:333] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.989408 30503 raft_consensus.cc:2243] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.989835 30503 raft_consensus.cc:2272] T 827a79573ac94525a9ff30c744c45164 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.991990 30503 tablet_replica.cc:333] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.992573 30503 raft_consensus.cc:2243] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:26:59.993001 30503 raft_consensus.cc:2272] T e313f6102e9941248ba9069990945017 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.995041 30503 tablet_replica.cc:333] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.995594 30503 raft_consensus.cc:2243] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.996351 30503 raft_consensus.cc:2272] T 1d019dd2bd864166abac89cc3fb1be76 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:26:59.998349 30503 tablet_replica.cc:333] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:26:59.998909 30503 raft_consensus.cc:2243] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:26:59.999682 30503 raft_consensus.cc:2272] T 6d4384a4da6846e39db4bcd0330cb822 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.001652 30503 tablet_replica.cc:333] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.002290 30503 raft_consensus.cc:2243] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:00.002871 30503 raft_consensus.cc:2272] T bfa4ec6799b5454a9d750ba5a79fafa3 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.005023 30503 tablet_replica.cc:333] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.005651 30503 raft_consensus.cc:2243] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:00.006126 30503 raft_consensus.cc:2272] T f32b983bac2949418ec7a6bf8a0ecdea P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.008168 30503 tablet_replica.cc:333] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.008677 30503 raft_consensus.cc:2243] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:00.009074 30503 raft_consensus.cc:2272] T 6c4cc7481ed24aa6991531d818969460 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.011162 30503 tablet_replica.cc:333] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.011667 30503 raft_consensus.cc:2243] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:00.012035 30503 raft_consensus.cc:2272] T 393b5b835714456ab06eeecb9b4a116d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.014309 30503 tablet_replica.cc:333] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.014968 30503 raft_consensus.cc:2243] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:00.015887 30503 raft_consensus.cc:2272] T c8f2372474d94a11bb1e044fc69c60e5 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.018536 30503 tablet_replica.cc:333] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.019222 30503 raft_consensus.cc:2243] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:00.019681 30503 raft_consensus.cc:2272] T 43ed653326ff4ae6b3c7daa594b3228b P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.022028 30503 tablet_replica.cc:333] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.022619 30503 raft_consensus.cc:2243] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:00.023471 30503 raft_consensus.cc:2272] T 578fd826b687468ca8be54244c3b4bc0 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.025627 30503 tablet_replica.cc:333] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.026165 30503 raft_consensus.cc:2243] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:00.026619 30503 raft_consensus.cc:2272] T 5734e27937894275a18ba504ffedd114 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.028681 30503 tablet_replica.cc:333] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.029191 30503 raft_consensus.cc:2243] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:00.029608 30503 raft_consensus.cc:2272] T 0c068ba8213a415b9df4e5380d0ad37e P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.031627 30503 tablet_replica.cc:333] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.032150 30503 raft_consensus.cc:2243] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:00.032595 30503 raft_consensus.cc:2272] T 267085844c6943b08c0f42b220e7e8ab P 9968c0ed1974454d828996824f859433 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.034394 30503 tablet_replica.cc:333] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.034945 30503 raft_consensus.cc:2243] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:00.035373 30503 raft_consensus.cc:2272] T db58e810d4a04064bbf12daef58026f3 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.037267 30503 tablet_replica.cc:333] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.037766 30503 raft_consensus.cc:2243] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:00.038197 30503 raft_consensus.cc:2272] T 47908558560f437a92445e6ab3a30a57 P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.040191 30503 tablet_replica.cc:333] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.040714 30503 raft_consensus.cc:2243] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:00.041559 30503 raft_consensus.cc:2272] T 8471ace194cf46238a8cd2862aa8cd5d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.043673 30503 tablet_replica.cc:333] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.044210 30503 raft_consensus.cc:2243] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:00.044656 30503 raft_consensus.cc:2272] T e57eb43303724eae812867b69fa94d9d P 9968c0ed1974454d828996824f859433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.046684 30503 tablet_replica.cc:333] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.047184 30503 raft_consensus.cc:2243] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:00.047597 30503 raft_consensus.cc:2272] T ff00b27947404e52acea283e9381d837 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.049578 30503 tablet_replica.cc:333] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.050074 30503 raft_consensus.cc:2243] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:00.050971 30503 raft_consensus.cc:2272] T 1cf22a8b3fa4491c92c735f8151982d3 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.053016 30503 tablet_replica.cc:333] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433: stopping tablet replica
I20260709 12:27:00.053486 30503 raft_consensus.cc:2243] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:00.053880 30503 raft_consensus.cc:2272] T d8586c90fc9848248b387776dc1f7db6 P 9968c0ed1974454d828996824f859433 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.126044 30503 tablet_server.cc:196] TabletServer@127.29.201.196:0 shutdown complete.
I20260709 12:27:00.191583 30503 master.cc:562] Master@127.29.201.254:38597 shutting down...
I20260709 12:27:00.220132 30503 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:00.220741 30503 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:00.221071 30503 tablet_replica.cc:333] T 00000000000000000000000000000000 P 160751aa6dc745d783d91c4f9039e337: stopping tablet replica
I20260709 12:27:00.244637 30503 master.cc:584] Master@127.29.201.254:38597 shutdown complete.
[       OK ] LeaderRebalancerTest.RestartTserver (31668 ms)
[ RUN      ] LeaderRebalancerTest.TestMaintenanceMode
I20260709 12:27:00.340788 30503 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.201.254:44111
I20260709 12:27:00.342160 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:00.348390 12496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:00.348433 12497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:00.349677 12499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:00.350174 30503 server_base.cc:1061] running on GCE node
I20260709 12:27:00.351421 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:00.351608 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:00.351725 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600020351715 us; error 0 us; skew 500 ppm
I20260709 12:27:00.352242 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:00.355201 30503 webserver.cc:533] Webserver started at http://127.29.201.254:39047/ using document root <none> and password file <none>
I20260709 12:27:00.355675 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:00.355829 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:00.356070 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:00.357501 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/master-0-root/instance:
uuid: "b425b396d0724794b79814fc3056dbd3"
format_stamp: "Formatted at 2026-07-09 12:27:00 on dist-test-slave-qgf5"
I20260709 12:27:00.362190 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.003s
I20260709 12:27:00.366063 12504 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:00.367056 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.000s	sys 0.003s
I20260709 12:27:00.367381 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/master-0-root
uuid: "b425b396d0724794b79814fc3056dbd3"
format_stamp: "Formatted at 2026-07-09 12:27:00 on dist-test-slave-qgf5"
I20260709 12:27:00.367715 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:00.391824 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:00.393270 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:00.433830 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.254:44111
I20260709 12:27:00.433954 12555 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.254:44111 every 8 connection(s)
I20260709 12:27:00.438010 12556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:00.450554 12556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3: Bootstrap starting.
I20260709 12:27:00.456053 12556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:00.460698 12556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3: No bootstrap required, opened a new log
I20260709 12:27:00.462990 12556 raft_consensus.cc:359] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b425b396d0724794b79814fc3056dbd3" member_type: VOTER }
I20260709 12:27:00.463389 12556 raft_consensus.cc:385] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:00.463580 12556 raft_consensus.cc:740] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b425b396d0724794b79814fc3056dbd3, State: Initialized, Role: FOLLOWER
I20260709 12:27:00.464089 12556 consensus_queue.cc:260] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b425b396d0724794b79814fc3056dbd3" member_type: VOTER }
I20260709 12:27:00.464504 12556 raft_consensus.cc:399] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:27:00.464833 12556 raft_consensus.cc:493] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:27:00.465076 12556 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:00.470337 12556 raft_consensus.cc:515] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b425b396d0724794b79814fc3056dbd3" member_type: VOTER }
I20260709 12:27:00.470970 12556 leader_election.cc:304] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b425b396d0724794b79814fc3056dbd3; no voters: 
I20260709 12:27:00.472198 12556 leader_election.cc:290] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:27:00.472546 12559 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:00.473969 12559 raft_consensus.cc:697] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [term 1 LEADER]: Becoming Leader. State: Replica: b425b396d0724794b79814fc3056dbd3, State: Running, Role: LEADER
I20260709 12:27:00.474783 12559 consensus_queue.cc:237] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b425b396d0724794b79814fc3056dbd3" member_type: VOTER }
I20260709 12:27:00.475642 12556 sys_catalog.cc:565] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:27:00.479642 12560 sys_catalog.cc:455] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b425b396d0724794b79814fc3056dbd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b425b396d0724794b79814fc3056dbd3" member_type: VOTER } }
I20260709 12:27:00.480131 12561 sys_catalog.cc:455] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b425b396d0724794b79814fc3056dbd3. Latest consensus state: current_term: 1 leader_uuid: "b425b396d0724794b79814fc3056dbd3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b425b396d0724794b79814fc3056dbd3" member_type: VOTER } }
I20260709 12:27:00.480579 12560 sys_catalog.cc:458] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:00.480829 12561 sys_catalog.cc:458] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:00.483871 12566 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:27:00.492266 12566 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:27:00.495421 30503 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:27:00.502879 12566 catalog_manager.cc:1383] Generated new cluster ID: b604531af23d4c2b84cb4a00c5ee11bb
I20260709 12:27:00.503187 12566 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:27:00.526461 12566 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:27:00.528024 12566 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:27:00.543131 12566 catalog_manager.cc:6070] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3: Generated new TSK 0
I20260709 12:27:00.543752 12566 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:27:00.562247 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:00.568641 12577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:00.569511 12578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:00.571529 12580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:00.572167 30503 server_base.cc:1061] running on GCE node
I20260709 12:27:00.573225 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:00.573439 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:00.573637 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600020573599 us; error 0 us; skew 500 ppm
I20260709 12:27:00.574249 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:00.576704 30503 webserver.cc:533] Webserver started at http://127.29.201.193:45725/ using document root <none> and password file <none>
I20260709 12:27:00.577278 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:00.577492 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:00.577777 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:00.578980 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-0-root/instance:
uuid: "76884a9b95e64c5e95e950afda820ece"
format_stamp: "Formatted at 2026-07-09 12:27:00 on dist-test-slave-qgf5"
I20260709 12:27:00.583745 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:27:00.587383 12585 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:00.588235 30503 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.001s
I20260709 12:27:00.588577 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-0-root
uuid: "76884a9b95e64c5e95e950afda820ece"
format_stamp: "Formatted at 2026-07-09 12:27:00 on dist-test-slave-qgf5"
I20260709 12:27:00.588960 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:00.611359 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:00.612663 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:00.614220 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:00.616851 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:00.617094 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:00.617357 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:00.617551 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:00.658653 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.193:39205
I20260709 12:27:00.658838 12647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.193:39205 every 8 connection(s)
I20260709 12:27:00.663844 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:00.671257 12652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:00.671510 12653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:00.675277 12655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:00.675428 30503 server_base.cc:1061] running on GCE node
I20260709 12:27:00.677114 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:00.677456 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:00.677709 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600020677689 us; error 0 us; skew 500 ppm
I20260709 12:27:00.678283 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:00.680485 12648 heartbeater.cc:344] Connected to a master server at 127.29.201.254:44111
I20260709 12:27:00.680965 12648 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:00.681314 30503 webserver.cc:533] Webserver started at http://127.29.201.194:44381/ using document root <none> and password file <none>
I20260709 12:27:00.682029 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:00.681926 12648 heartbeater.cc:507] Master 127.29.201.254:44111 requested a full tablet report, sending...
I20260709 12:27:00.682447 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:00.682840 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:00.684121 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-1-root/instance:
uuid: "dd8de60841be432785aa6bde9d170a08"
format_stamp: "Formatted at 2026-07-09 12:27:00 on dist-test-slave-qgf5"
I20260709 12:27:00.684624 12521 ts_manager.cc:194] Registered new tserver with Master: 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205)
I20260709 12:27:00.686487 12521 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51258
I20260709 12:27:00.689340 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:27:00.693279 12660 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:00.694283 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:27:00.694702 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-1-root
uuid: "dd8de60841be432785aa6bde9d170a08"
format_stamp: "Formatted at 2026-07-09 12:27:00 on dist-test-slave-qgf5"
I20260709 12:27:00.695003 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:00.718901 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:00.720255 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:00.721923 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:00.724467 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:00.724830 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:00.725101 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:00.725294 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:00.767719 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.194:33355
I20260709 12:27:00.767843 12722 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.194:33355 every 8 connection(s)
I20260709 12:27:00.772670 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:00.780782 12726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:00.781201 12727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:00.785756 30503 server_base.cc:1061] running on GCE node
W20260709 12:27:00.785812 12729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:00.786657 12723 heartbeater.cc:344] Connected to a master server at 127.29.201.254:44111
I20260709 12:27:00.787175 12723 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:00.787456 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:00.787763 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:00.787968 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600020787952 us; error 0 us; skew 500 ppm
I20260709 12:27:00.788223 12723 heartbeater.cc:507] Master 127.29.201.254:44111 requested a full tablet report, sending...
I20260709 12:27:00.788836 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:00.790889 12521 ts_manager.cc:194] Registered new tserver with Master: dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:00.791819 30503 webserver.cc:533] Webserver started at http://127.29.201.195:33649/ using document root <none> and password file <none>
I20260709 12:27:00.792400 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:00.792622 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:00.792500 12521 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51270
I20260709 12:27:00.793097 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:00.794458 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-2-root/instance:
uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
format_stamp: "Formatted at 2026-07-09 12:27:00 on dist-test-slave-qgf5"
I20260709 12:27:00.799068 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:27:00.802470 12734 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:00.803359 30503 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.003s	sys 0.000s
I20260709 12:27:00.803694 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-2-root
uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
format_stamp: "Formatted at 2026-07-09 12:27:00 on dist-test-slave-qgf5"
I20260709 12:27:00.804008 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:00.825708 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:00.827127 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:00.828715 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:00.831136 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:00.831388 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:00.831650 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:00.831835 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:00.872138 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.195:44647
I20260709 12:27:00.872258 12796 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.195:44647 every 8 connection(s)
I20260709 12:27:00.886772 12797 heartbeater.cc:344] Connected to a master server at 127.29.201.254:44111
I20260709 12:27:00.887220 12797 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:00.888003 12797 heartbeater.cc:507] Master 127.29.201.254:44111 requested a full tablet report, sending...
I20260709 12:27:00.890190 12521 ts_manager.cc:194] Registered new tserver with Master: 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:00.890300 30503 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014915593s
I20260709 12:27:00.891878 12521 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51278
I20260709 12:27:00.918599 12521 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51288:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:27:00.922161 12521 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:27:01.003925 12613 tablet_service.cc:1511] Processing CreateTablet for tablet ade88c76c8f34edfac61ded6c3d472f6 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:27:01.005676 12613 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade88c76c8f34edfac61ded6c3d472f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.008361 12762 tablet_service.cc:1511] Processing CreateTablet for tablet ade88c76c8f34edfac61ded6c3d472f6 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:27:01.010079 12762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade88c76c8f34edfac61ded6c3d472f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.010510 12612 tablet_service.cc:1511] Processing CreateTablet for tablet 66d601ec378b47b9834fefc2ccf51ce3 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:27:01.012210 12612 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66d601ec378b47b9834fefc2ccf51ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.012840 12761 tablet_service.cc:1511] Processing CreateTablet for tablet 66d601ec378b47b9834fefc2ccf51ce3 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:27:01.014533 12761 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66d601ec378b47b9834fefc2ccf51ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.015298 12611 tablet_service.cc:1511] Processing CreateTablet for tablet b909763b4c1045178a6fc9fd1a58250f (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:27:01.017197 12611 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b909763b4c1045178a6fc9fd1a58250f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.021245 12760 tablet_service.cc:1511] Processing CreateTablet for tablet b909763b4c1045178a6fc9fd1a58250f (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:27:01.023387 12760 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b909763b4c1045178a6fc9fd1a58250f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.027725 12610 tablet_service.cc:1511] Processing CreateTablet for tablet 15118b927e72411bb330ef7b8d8d3a92 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:27:01.029498 12610 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15118b927e72411bb330ef7b8d8d3a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.040056 12609 tablet_service.cc:1511] Processing CreateTablet for tablet 6f0ff2156af842de82d34a04aab9cdf4 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:27:01.041839 12609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f0ff2156af842de82d34a04aab9cdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.040056 12759 tablet_service.cc:1511] Processing CreateTablet for tablet 15118b927e72411bb330ef7b8d8d3a92 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:27:01.043138 12757 tablet_service.cc:1511] Processing CreateTablet for tablet fbd279d00e7142afab793d8849e54bf0 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:27:01.043345 12755 tablet_service.cc:1511] Processing CreateTablet for tablet 128ccf33a3764acc9fa913611d1a32f0 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:27:01.044608 12759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15118b927e72411bb330ef7b8d8d3a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.045184 12756 tablet_service.cc:1511] Processing CreateTablet for tablet a47a8c7a8101401297cf692f5358a7d2 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:27:01.046759 12755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 128ccf33a3764acc9fa913611d1a32f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.041872 12758 tablet_service.cc:1511] Processing CreateTablet for tablet 6f0ff2156af842de82d34a04aab9cdf4 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:27:01.048185 12758 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f0ff2156af842de82d34a04aab9cdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.051777 12756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a47a8c7a8101401297cf692f5358a7d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.055328 12757 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbd279d00e7142afab793d8849e54bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.062395 12608 tablet_service.cc:1511] Processing CreateTablet for tablet fbd279d00e7142afab793d8849e54bf0 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:27:01.064181 12608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbd279d00e7142afab793d8849e54bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.068486 12607 tablet_service.cc:1511] Processing CreateTablet for tablet a47a8c7a8101401297cf692f5358a7d2 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:27:01.070276 12607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a47a8c7a8101401297cf692f5358a7d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.071996 12687 tablet_service.cc:1511] Processing CreateTablet for tablet 66d601ec378b47b9834fefc2ccf51ce3 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260709 12:27:01.073861 12687 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66d601ec378b47b9834fefc2ccf51ce3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.076188 12685 tablet_service.cc:1511] Processing CreateTablet for tablet 15118b927e72411bb330ef7b8d8d3a92 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260709 12:27:01.078298 12685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15118b927e72411bb330ef7b8d8d3a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.087756 12680 tablet_service.cc:1511] Processing CreateTablet for tablet cc7e1418d7c145b88a45a159d25e21cc (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:27:01.087925 12688 tablet_service.cc:1511] Processing CreateTablet for tablet ade88c76c8f34edfac61ded6c3d472f6 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260709 12:27:01.089887 12680 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc7e1418d7c145b88a45a159d25e21cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.080029 12684 tablet_service.cc:1511] Processing CreateTablet for tablet 6f0ff2156af842de82d34a04aab9cdf4 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260709 12:27:01.091405 12684 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f0ff2156af842de82d34a04aab9cdf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.091171 12681 tablet_service.cc:1511] Processing CreateTablet for tablet 128ccf33a3764acc9fa913611d1a32f0 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:27:01.085817 12754 tablet_service.cc:1511] Processing CreateTablet for tablet cc7e1418d7c145b88a45a159d25e21cc (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:27:01.093037 12681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 128ccf33a3764acc9fa913611d1a32f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.093323 12754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc7e1418d7c145b88a45a159d25e21cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.086882 12686 tablet_service.cc:1511] Processing CreateTablet for tablet b909763b4c1045178a6fc9fd1a58250f (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260709 12:27:01.087792 12683 tablet_service.cc:1511] Processing CreateTablet for tablet fbd279d00e7142afab793d8849e54bf0 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260709 12:27:01.095480 12686 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b909763b4c1045178a6fc9fd1a58250f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.096377 12683 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbd279d00e7142afab793d8849e54bf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.088189 12753 tablet_service.cc:1511] Processing CreateTablet for tablet 6041407ddd7647db97a3e044abd5574e (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:27:01.097961 12753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6041407ddd7647db97a3e044abd5574e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.084261 12682 tablet_service.cc:1511] Processing CreateTablet for tablet a47a8c7a8101401297cf692f5358a7d2 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260709 12:27:01.099747 12682 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a47a8c7a8101401297cf692f5358a7d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.101415 12688 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ade88c76c8f34edfac61ded6c3d472f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.120939 12606 tablet_service.cc:1511] Processing CreateTablet for tablet 128ccf33a3764acc9fa913611d1a32f0 (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260709 12:27:01.120329 12679 tablet_service.cc:1511] Processing CreateTablet for tablet 6041407ddd7647db97a3e044abd5574e (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:27:01.124930 12606 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 128ccf33a3764acc9fa913611d1a32f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.125531 12679 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6041407ddd7647db97a3e044abd5574e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.140524 12605 tablet_service.cc:1511] Processing CreateTablet for tablet cc7e1418d7c145b88a45a159d25e21cc (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260709 12:27:01.142238 12605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc7e1418d7c145b88a45a159d25e21cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.148504 12604 tablet_service.cc:1511] Processing CreateTablet for tablet 6041407ddd7647db97a3e044abd5574e (DEFAULT_TABLE table=test-workload [id=adced6e2e2b845159dfda4eea254c43e]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260709 12:27:01.150094 12604 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6041407ddd7647db97a3e044abd5574e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:01.167428 12813 tablet_bootstrap.cc:492] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08: Bootstrap starting.
I20260709 12:27:01.176703 12813 tablet_bootstrap.cc:654] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.237083 12812 tablet_bootstrap.cc:492] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Bootstrap starting.
I20260709 12:27:01.244000 12812 tablet_bootstrap.cc:654] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.256652 12816 tablet_bootstrap.cc:492] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:01.269062 12813 tablet_bootstrap.cc:492] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08: No bootstrap required, opened a new log
I20260709 12:27:01.304695 12813 ts_tablet_manager.cc:1403] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08: Time spent bootstrapping tablet: real 0.138s	user 0.025s	sys 0.000s
I20260709 12:27:01.309531 12813 raft_consensus.cc:359] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.310324 12813 raft_consensus.cc:385] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.310663 12813 raft_consensus.cc:740] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.311609 12813 consensus_queue.cc:260] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.344408 12723 heartbeater.cc:499] Master 127.29.201.254:44111 was elected leader, sending a full tablet report...
I20260709 12:27:01.344790 12813 ts_tablet_manager.cc:1434] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08: Time spent starting tablet: real 0.040s	user 0.017s	sys 0.019s
I20260709 12:27:01.346042 12813 tablet_bootstrap.cc:492] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08: Bootstrap starting.
I20260709 12:27:01.358031 12812 tablet_bootstrap.cc:492] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: No bootstrap required, opened a new log
I20260709 12:27:01.358721 12812 ts_tablet_manager.cc:1403] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent bootstrapping tablet: real 0.122s	user 0.021s	sys 0.020s
I20260709 12:27:01.359164 12816 tablet_bootstrap.cc:654] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.360865 12813 tablet_bootstrap.cc:654] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.361889 12812 raft_consensus.cc:359] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.366922 12812 raft_consensus.cc:385] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.367338 12812 raft_consensus.cc:740] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.368242 12812 consensus_queue.cc:260] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.370450 12816 tablet_bootstrap.cc:492] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece: No bootstrap required, opened a new log
I20260709 12:27:01.371404 12816 ts_tablet_manager.cc:1403] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.115s	user 0.014s	sys 0.004s
I20260709 12:27:01.374686 12797 heartbeater.cc:499] Master 127.29.201.254:44111 was elected leader, sending a full tablet report...
I20260709 12:27:01.375173 12812 ts_tablet_manager.cc:1434] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.001s
I20260709 12:27:01.376385 12812 tablet_bootstrap.cc:492] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Bootstrap starting.
I20260709 12:27:01.376151 12816 raft_consensus.cc:359] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.376964 12816 raft_consensus.cc:385] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.377262 12813 tablet_bootstrap.cc:492] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08: No bootstrap required, opened a new log
W20260709 12:27:01.377596 12798 tablet.cc:2406] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:27:01.377934 12813 ts_tablet_manager.cc:1403] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08: Time spent bootstrapping tablet: real 0.032s	user 0.020s	sys 0.003s
I20260709 12:27:01.377301 12816 raft_consensus.cc:740] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.379871 12816 consensus_queue.cc:260] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.381229 12813 raft_consensus.cc:359] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.382030 12813 raft_consensus.cc:385] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.382368 12813 raft_consensus.cc:740] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.383279 12812 tablet_bootstrap.cc:654] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.383203 12813 consensus_queue.cc:260] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.390259 12813 ts_tablet_manager.cc:1434] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.008s
I20260709 12:27:01.391624 12813 tablet_bootstrap.cc:492] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08: Bootstrap starting.
I20260709 12:27:01.401467 12813 tablet_bootstrap.cc:654] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.414477 12648 heartbeater.cc:499] Master 127.29.201.254:44111 was elected leader, sending a full tablet report...
I20260709 12:27:01.415239 12816 ts_tablet_manager.cc:1434] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.043s	user 0.015s	sys 0.024s
I20260709 12:27:01.416435 12816 tablet_bootstrap.cc:492] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:01.421001 12812 tablet_bootstrap.cc:492] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: No bootstrap required, opened a new log
I20260709 12:27:01.421782 12812 ts_tablet_manager.cc:1403] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent bootstrapping tablet: real 0.046s	user 0.015s	sys 0.007s
I20260709 12:27:01.423337 12816 tablet_bootstrap.cc:654] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.425122 12812 raft_consensus.cc:359] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.426066 12812 raft_consensus.cc:385] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.426446 12812 raft_consensus.cc:740] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.427006 12813 tablet_bootstrap.cc:492] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08: No bootstrap required, opened a new log
I20260709 12:27:01.427821 12813 ts_tablet_manager.cc:1403] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08: Time spent bootstrapping tablet: real 0.036s	user 0.013s	sys 0.008s
I20260709 12:27:01.427366 12812 consensus_queue.cc:260] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.429849 12812 ts_tablet_manager.cc:1434] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:27:01.430995 12812 tablet_bootstrap.cc:492] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e: Bootstrap starting.
I20260709 12:27:01.431288 12813 raft_consensus.cc:359] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.432320 12813 raft_consensus.cc:385] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.432687 12813 raft_consensus.cc:740] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.433710 12813 consensus_queue.cc:260] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.439065 12812 tablet_bootstrap.cc:654] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.439186 12813 ts_tablet_manager.cc:1434] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:27:01.441653 12813 tablet_bootstrap.cc:492] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08: Bootstrap starting.
I20260709 12:27:01.443599 12816 tablet_bootstrap.cc:492] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece: No bootstrap required, opened a new log
I20260709 12:27:01.444535 12816 ts_tablet_manager.cc:1403] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.028s	user 0.018s	sys 0.005s
I20260709 12:27:01.449294 12816 raft_consensus.cc:359] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.450594 12816 raft_consensus.cc:385] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.451117 12816 raft_consensus.cc:740] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.451666 12813 tablet_bootstrap.cc:654] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.452437 12816 consensus_queue.cc:260] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.455942 12816 ts_tablet_manager.cc:1434] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.011s	user 0.006s	sys 0.002s
I20260709 12:27:01.457168 12816 tablet_bootstrap.cc:492] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:01.457639 12812 tablet_bootstrap.cc:492] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e: No bootstrap required, opened a new log
I20260709 12:27:01.458353 12812 ts_tablet_manager.cc:1403] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent bootstrapping tablet: real 0.028s	user 0.019s	sys 0.007s
I20260709 12:27:01.462149 12812 raft_consensus.cc:359] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.463146 12812 raft_consensus.cc:385] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.463563 12812 raft_consensus.cc:740] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.464709 12812 consensus_queue.cc:260] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.464709 12816 tablet_bootstrap.cc:654] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.467412 12812 ts_tablet_manager.cc:1434] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:27:01.468670 12812 tablet_bootstrap.cc:492] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Bootstrap starting.
I20260709 12:27:01.470116 12813 tablet_bootstrap.cc:492] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08: No bootstrap required, opened a new log
I20260709 12:27:01.470775 12813 ts_tablet_manager.cc:1403] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08: Time spent bootstrapping tablet: real 0.029s	user 0.016s	sys 0.006s
I20260709 12:27:01.474879 12813 raft_consensus.cc:359] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:01.475808 12813 raft_consensus.cc:385] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.476622 12812 tablet_bootstrap.cc:654] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.476209 12813 raft_consensus.cc:740] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.479782 12813 consensus_queue.cc:260] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:01.482848 12813 ts_tablet_manager.cc:1434] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.002s
I20260709 12:27:01.484249 12813 tablet_bootstrap.cc:492] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08: Bootstrap starting.
I20260709 12:27:01.485105 12816 tablet_bootstrap.cc:492] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece: No bootstrap required, opened a new log
I20260709 12:27:01.485775 12816 ts_tablet_manager.cc:1403] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.029s	user 0.011s	sys 0.008s
I20260709 12:27:01.485746 12812 tablet_bootstrap.cc:492] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e: No bootstrap required, opened a new log
I20260709 12:27:01.487309 12812 ts_tablet_manager.cc:1403] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent bootstrapping tablet: real 0.019s	user 0.013s	sys 0.000s
W20260709 12:27:01.490182 12569 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:01.491067 12812 raft_consensus.cc:359] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.491297 12816 raft_consensus.cc:359] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.492051 12813 tablet_bootstrap.cc:654] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.492158 12812 raft_consensus.cc:385] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.492566 12816 raft_consensus.cc:385] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.492833 12812 raft_consensus.cc:740] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.493062 12816 raft_consensus.cc:740] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.493996 12816 consensus_queue.cc:260] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.494072 12812 consensus_queue.cc:260] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.498661 12816 ts_tablet_manager.cc:1434] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 12:27:01.499941 12816 tablet_bootstrap.cc:492] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:01.500480 12812 ts_tablet_manager.cc:1434] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.000s
I20260709 12:27:01.502992 12812 tablet_bootstrap.cc:492] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Bootstrap starting.
I20260709 12:27:01.506652 12816 tablet_bootstrap.cc:654] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.512986 12813 tablet_bootstrap.cc:492] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08: No bootstrap required, opened a new log
I20260709 12:27:01.513720 12813 ts_tablet_manager.cc:1403] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08: Time spent bootstrapping tablet: real 0.030s	user 0.017s	sys 0.009s
I20260709 12:27:01.517197 12812 tablet_bootstrap.cc:654] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.518162 12820 raft_consensus.cc:493] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.517839 12813 raft_consensus.cc:359] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.519062 12813 raft_consensus.cc:385] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.518857 12820 raft_consensus.cc:515] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.519511 12813 raft_consensus.cc:740] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.520747 12813 consensus_queue.cc:260] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.523299 12813 ts_tablet_manager.cc:1434] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:27:01.524123 12816 tablet_bootstrap.cc:492] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece: No bootstrap required, opened a new log
I20260709 12:27:01.524554 12813 tablet_bootstrap.cc:492] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08: Bootstrap starting.
I20260709 12:27:01.524722 12816 ts_tablet_manager.cc:1403] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.025s	user 0.012s	sys 0.006s
I20260709 12:27:01.531458 12816 raft_consensus.cc:359] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.532514 12816 raft_consensus.cc:385] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.532935 12816 raft_consensus.cc:740] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.534035 12816 consensus_queue.cc:260] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.537003 12816 ts_tablet_manager.cc:1434] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.012s	user 0.003s	sys 0.003s
I20260709 12:27:01.537696 12812 tablet_bootstrap.cc:492] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e: No bootstrap required, opened a new log
I20260709 12:27:01.538312 12816 tablet_bootstrap.cc:492] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:01.538415 12812 ts_tablet_manager.cc:1403] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent bootstrapping tablet: real 0.036s	user 0.018s	sys 0.005s
I20260709 12:27:01.542311 12812 raft_consensus.cc:359] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.543396 12812 raft_consensus.cc:385] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.543823 12812 raft_consensus.cc:740] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.544745 12812 consensus_queue.cc:260] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.534555 12813 tablet_bootstrap.cc:654] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.551380 12816 tablet_bootstrap.cc:654] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.552628 12817 raft_consensus.cc:493] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.553361 12817 raft_consensus.cc:515] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.553699 12820 leader_election.cc:290] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:01.556260 12821 raft_consensus.cc:493] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.556862 12821 raft_consensus.cc:515] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.564821 12821 leader_election.cc:290] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:01.566128 12813 tablet_bootstrap.cc:492] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08: No bootstrap required, opened a new log
I20260709 12:27:01.566211 12817 leader_election.cc:290] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:01.566990 12813 ts_tablet_manager.cc:1403] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08: Time spent bootstrapping tablet: real 0.043s	user 0.015s	sys 0.005s
I20260709 12:27:01.570714 12816 tablet_bootstrap.cc:492] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece: No bootstrap required, opened a new log
I20260709 12:27:01.571609 12816 ts_tablet_manager.cc:1403] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.034s	user 0.010s	sys 0.008s
I20260709 12:27:01.572161 12813 raft_consensus.cc:359] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.573377 12813 raft_consensus.cc:385] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.573791 12813 raft_consensus.cc:740] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.574939 12813 consensus_queue.cc:260] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.578373 12812 ts_tablet_manager.cc:1434] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent starting tablet: real 0.039s	user 0.009s	sys 0.003s
I20260709 12:27:01.578781 12816 raft_consensus.cc:359] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.579732 12816 raft_consensus.cc:385] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.580049 12816 raft_consensus.cc:740] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.580065 12812 tablet_bootstrap.cc:492] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e: Bootstrap starting.
I20260709 12:27:01.581056 12816 consensus_queue.cc:260] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.583770 12816 ts_tablet_manager.cc:1434] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.012s	user 0.005s	sys 0.004s
I20260709 12:27:01.584852 12816 tablet_bootstrap.cc:492] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:01.591876 12813 ts_tablet_manager.cc:1434] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08: Time spent starting tablet: real 0.024s	user 0.009s	sys 0.001s
I20260709 12:27:01.593693 12813 tablet_bootstrap.cc:492] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08: Bootstrap starting.
I20260709 12:27:01.598294 12812 tablet_bootstrap.cc:654] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.609530 12813 tablet_bootstrap.cc:654] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.616043 12816 tablet_bootstrap.cc:654] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.616104 12817 raft_consensus.cc:493] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.618237 12817 raft_consensus.cc:515] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.632608 12817 leader_election.cc:290] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:01.646901 12812 tablet_bootstrap.cc:492] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e: No bootstrap required, opened a new log
I20260709 12:27:01.647609 12812 ts_tablet_manager.cc:1403] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent bootstrapping tablet: real 0.068s	user 0.023s	sys 0.000s
I20260709 12:27:01.646896 12813 tablet_bootstrap.cc:492] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08: No bootstrap required, opened a new log
I20260709 12:27:01.650556 12813 ts_tablet_manager.cc:1403] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08: Time spent bootstrapping tablet: real 0.057s	user 0.019s	sys 0.000s
I20260709 12:27:01.652829 12812 raft_consensus.cc:359] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.653963 12812 raft_consensus.cc:385] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.654417 12834 raft_consensus.cc:493] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.654335 12812 raft_consensus.cc:740] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.655010 12834 raft_consensus.cc:515] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.656898 12821 raft_consensus.cc:493] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.657434 12821 raft_consensus.cc:515] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.658016 12834 leader_election.cc:290] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:01.657878 12812 consensus_queue.cc:260] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.659483 12821 leader_election.cc:290] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:01.656311 12813 raft_consensus.cc:359] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.660984 12813 raft_consensus.cc:385] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.661288 12813 raft_consensus.cc:740] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.664374 12698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "128ccf33a3764acc9fa913611d1a32f0" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08" is_pre_election: true
I20260709 12:27:01.666517 12697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15118b927e72411bb330ef7b8d8d3a92" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08" is_pre_election: true
I20260709 12:27:01.667804 12697 raft_consensus.cc:2468] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 0.
I20260709 12:27:01.669348 12736 leader_election.cc:304] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:01.670151 12821 raft_consensus.cc:493] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.670514 12623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "128ccf33a3764acc9fa913611d1a32f0" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece" is_pre_election: true
I20260709 12:27:01.670723 12821 raft_consensus.cc:515] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.671343 12623 raft_consensus.cc:2468] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 0.
I20260709 12:27:01.672886 12622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15118b927e72411bb330ef7b8d8d3a92" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece" is_pre_election: true
I20260709 12:27:01.673099 12821 leader_election.cc:290] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:01.671162 12812 ts_tablet_manager.cc:1434] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent starting tablet: real 0.023s	user 0.007s	sys 0.003s
I20260709 12:27:01.674677 12812 tablet_bootstrap.cc:492] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Bootstrap starting.
I20260709 12:27:01.674750 12834 raft_consensus.cc:2804] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.675256 12834 raft_consensus.cc:493] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.675637 12834 raft_consensus.cc:3060] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.662770 12813 consensus_queue.cc:260] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.665761 12698 raft_consensus.cc:2468] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 0.
I20260709 12:27:01.678579 12736 leader_election.cc:304] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:01.679427 12820 raft_consensus.cc:2804] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.679780 12820 raft_consensus.cc:493] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.680104 12820 raft_consensus.cc:3060] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.684383 12812 tablet_bootstrap.cc:654] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.685474 12813 ts_tablet_manager.cc:1434] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08: Time spent starting tablet: real 0.035s	user 0.006s	sys 0.000s
I20260709 12:27:01.686751 12813 tablet_bootstrap.cc:492] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08: Bootstrap starting.
I20260709 12:27:01.688496 12834 raft_consensus.cc:515] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
W20260709 12:27:01.691206 12735 leader_election.cc:343] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.692713 12622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece" is_pre_election: true
I20260709 12:27:01.693423 12622 raft_consensus.cc:2468] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 0.
I20260709 12:27:01.694475 12623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd279d00e7142afab793d8849e54bf0" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece" is_pre_election: true
I20260709 12:27:01.695987 12661 leader_election.cc:304] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:01.696784 12816 tablet_bootstrap.cc:492] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece: No bootstrap required, opened a new log
I20260709 12:27:01.699997 12816 ts_tablet_manager.cc:1403] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.115s	user 0.021s	sys 0.015s
I20260709 12:27:01.700439 12817 raft_consensus.cc:2804] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.700901 12817 raft_consensus.cc:493] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.701241 12817 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.701855 12623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15118b927e72411bb330ef7b8d8d3a92" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:01.700470 12698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "15118b927e72411bb330ef7b8d8d3a92" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08"
I20260709 12:27:01.698868 12834 leader_election.cc:290] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:01.703413 12698 raft_consensus.cc:3060] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:01.703739 12735 leader_election.cc:343] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.703764 12834 raft_consensus.cc:493] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.704562 12834 raft_consensus.cc:515] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.705392 12812 tablet_bootstrap.cc:492] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e: No bootstrap required, opened a new log
I20260709 12:27:01.708190 12697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd279d00e7142afab793d8849e54bf0" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08" is_pre_election: true
I20260709 12:27:01.708953 12697 raft_consensus.cc:2468] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 0.
I20260709 12:27:01.709934 12813 tablet_bootstrap.cc:654] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.710304 12736 leader_election.cc:304] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:01.712257 12623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd279d00e7142afab793d8849e54bf0" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece" is_pre_election: true
W20260709 12:27:01.713876 12735 leader_election.cc:343] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:01.713982 12661 leader_election.cc:343] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.715417 12772 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a47a8c7a8101401297cf692f5358a7d2" candidate_uuid: "76884a9b95e64c5e95e950afda820ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" is_pre_election: true
I20260709 12:27:01.715662 12849 raft_consensus.cc:2804] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.716204 12772 raft_consensus.cc:2468] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76884a9b95e64c5e95e950afda820ece in term 0.
I20260709 12:27:01.705999 12812 ts_tablet_manager.cc:1403] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent bootstrapping tablet: real 0.032s	user 0.010s	sys 0.004s
I20260709 12:27:01.717150 12771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f0ff2156af842de82d34a04aab9cdf4" candidate_uuid: "76884a9b95e64c5e95e950afda820ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" is_pre_election: true
I20260709 12:27:01.718473 12697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a47a8c7a8101401297cf692f5358a7d2" candidate_uuid: "76884a9b95e64c5e95e950afda820ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08" is_pre_election: true
I20260709 12:27:01.719192 12697 raft_consensus.cc:2468] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76884a9b95e64c5e95e950afda820ece in term 0.
I20260709 12:27:01.720261 12696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f0ff2156af842de82d34a04aab9cdf4" candidate_uuid: "76884a9b95e64c5e95e950afda820ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08" is_pre_election: true
I20260709 12:27:01.721724 12695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3" candidate_uuid: "76884a9b95e64c5e95e950afda820ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08" is_pre_election: true
I20260709 12:27:01.722131 12587 leader_election.cc:304] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08; no voters: 
W20260709 12:27:01.723649 12587 leader_election.cc:343] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.723861 12771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3" candidate_uuid: "76884a9b95e64c5e95e950afda820ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" is_pre_election: true
W20260709 12:27:01.724573 12587 leader_election.cc:343] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.725493 12834 leader_election.cc:290] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:01.726861 12821 raft_consensus.cc:2804] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.727424 12821 raft_consensus.cc:493] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.727862 12821 raft_consensus.cc:3060] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.729166 12698 raft_consensus.cc:2468] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 1.
I20260709 12:27:01.729517 12817 raft_consensus.cc:515] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.730683 12736 leader_election.cc:304] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08; no voters: 76884a9b95e64c5e95e950afda820ece
I20260709 12:27:01.732134 12834 raft_consensus.cc:2804] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.732735 12623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:01.733529 12623 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:01.726148 12587 leader_election.cc:343] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.734913 12587 leader_election.cc:304] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece; no voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08
I20260709 12:27:01.737329 12853 raft_consensus.cc:2749] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:01.716187 12849 raft_consensus.cc:493] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.739251 12849 raft_consensus.cc:3060] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.741307 12834 raft_consensus.cc:697] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Becoming Leader. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:01.742420 12834 consensus_queue.cc:237] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.745127 12821 raft_consensus.cc:515] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.728058 12820 raft_consensus.cc:515] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.748064 12770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
I20260709 12:27:01.748890 12770 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.748116 12812 raft_consensus.cc:359] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:01.749625 12812 raft_consensus.cc:385] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.749926 12812 raft_consensus.cc:740] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.744593 12772 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd279d00e7142afab793d8849e54bf0" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" is_pre_election: true
I20260709 12:27:01.750541 12622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "128ccf33a3764acc9fa913611d1a32f0" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:01.745425 12771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" is_pre_election: true
I20260709 12:27:01.745530 12623 raft_consensus.cc:2468] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 1.
I20260709 12:27:01.750442 12820 leader_election.cc:290] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:01.746716 12817 leader_election.cc:290] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:01.751331 12622 raft_consensus.cc:3060] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.753314 12817 raft_consensus.cc:493] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.752625 12698 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "128ccf33a3764acc9fa913611d1a32f0" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08"
I20260709 12:27:01.703416 12816 raft_consensus.cc:359] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:01.754485 12816 raft_consensus.cc:385] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.754563 12698 raft_consensus.cc:3060] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.755918 12821 leader_election.cc:290] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 election: Requested vote from peers dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:01.756732 12812 consensus_queue.cc:260] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:01.759771 12812 ts_tablet_manager.cc:1434] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent starting tablet: real 0.042s	user 0.007s	sys 0.001s
I20260709 12:27:01.759881 12661 leader_election.cc:304] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08; no voters: 
W20260709 12:27:01.747548 12587 leader_election.cc:343] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.761102 12812 tablet_bootstrap.cc:492] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e: Bootstrap starting.
I20260709 12:27:01.761484 12587 leader_election.cc:304] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece; no voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08
I20260709 12:27:01.763942 12820 raft_consensus.cc:493] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.766872 12820 raft_consensus.cc:515] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:01.769678 12820 leader_election.cc:290] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:01.765286 12813 tablet_bootstrap.cc:492] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08: No bootstrap required, opened a new log
I20260709 12:27:01.770495 12771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a47a8c7a8101401297cf692f5358a7d2" candidate_uuid: "76884a9b95e64c5e95e950afda820ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
I20260709 12:27:01.771168 12696 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a47a8c7a8101401297cf692f5358a7d2" candidate_uuid: "76884a9b95e64c5e95e950afda820ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08"
I20260709 12:27:01.771294 12771 raft_consensus.cc:3060] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.771835 12821 raft_consensus.cc:2749] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:01.771994 12695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08" is_pre_election: true
I20260709 12:27:01.770498 12623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece" is_pre_election: true
I20260709 12:27:01.773028 12849 raft_consensus.cc:515] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.771864 12696 raft_consensus.cc:3060] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.781980 12771 raft_consensus.cc:2468] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76884a9b95e64c5e95e950afda820ece in term 1.
I20260709 12:27:01.769268 12622 raft_consensus.cc:2468] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 1.
I20260709 12:27:01.754878 12816 raft_consensus.cc:740] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.765885 12854 raft_consensus.cc:2804] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.783910 12735 leader_election.cc:304] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece; no voters: 
I20260709 12:27:01.753808 12817 raft_consensus.cc:515] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:01.786761 12812 tablet_bootstrap.cc:654] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.787209 12820 raft_consensus.cc:2804] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.787894 12695 raft_consensus.cc:2468] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 0.
I20260709 12:27:01.787283 12816 consensus_queue.cc:260] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:01.788659 12852 raft_consensus.cc:493] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.793669 12852 raft_consensus.cc:515] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.794893 12623 raft_consensus.cc:2468] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 0.
I20260709 12:27:01.796657 12852 leader_election.cc:290] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:01.797290 12696 raft_consensus.cc:2468] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76884a9b95e64c5e95e950afda820ece in term 1.
I20260709 12:27:01.798010 12849 leader_election.cc:290] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:01.770578 12813 ts_tablet_manager.cc:1403] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08: Time spent bootstrapping tablet: real 0.084s	user 0.020s	sys 0.022s
I20260709 12:27:01.791882 12854 raft_consensus.cc:697] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Becoming Leader. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:01.792603 12772 raft_consensus.cc:2393] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd8de60841be432785aa6bde9d170a08 in current term 1: Already voted for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in this term.
I20260709 12:27:01.792732 12736 leader_election.cc:304] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:01.800729 12771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" is_pre_election: true
I20260709 12:27:01.799906 12817 leader_election.cc:290] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647), 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205)
I20260709 12:27:01.794288 12770 raft_consensus.cc:2468] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 1.
I20260709 12:27:01.801086 12854 consensus_queue.cc:237] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.805222 12813 raft_consensus.cc:359] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.805883 12813 raft_consensus.cc:385] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.806176 12813 raft_consensus.cc:740] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.802675 12812 tablet_bootstrap.cc:492] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e: No bootstrap required, opened a new log
I20260709 12:27:01.802563 12849 raft_consensus.cc:2804] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.806938 12812 ts_tablet_manager.cc:1403] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent bootstrapping tablet: real 0.046s	user 0.014s	sys 0.006s
I20260709 12:27:01.785738 12857 raft_consensus.cc:493] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:01.807161 12849 raft_consensus.cc:493] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.807646 12849 raft_consensus.cc:3060] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.807535 12857 raft_consensus.cc:515] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.810134 12857 leader_election.cc:290] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:01.809971 12812 raft_consensus.cc:359] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.810964 12812 raft_consensus.cc:385] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.811255 12812 raft_consensus.cc:740] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.812003 12812 consensus_queue.cc:260] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.813881 12813 consensus_queue.cc:260] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.814281 12812 ts_tablet_manager.cc:1434] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:27:01.815488 12812 tablet_bootstrap.cc:492] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Bootstrap starting.
I20260709 12:27:01.816470 12849 raft_consensus.cc:515] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:01.819155 12771 raft_consensus.cc:2393] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd8de60841be432785aa6bde9d170a08 in current term 1: Already voted for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in this term.
I20260709 12:27:01.819844 12849 leader_election.cc:290] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:01.802675 12816 ts_tablet_manager.cc:1434] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.102s	user 0.007s	sys 0.002s
I20260709 12:27:01.821478 12816 tablet_bootstrap.cc:492] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:01.821735 12812 tablet_bootstrap.cc:654] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.823174 12813 ts_tablet_manager.cc:1434] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08: Time spent starting tablet: real 0.021s	user 0.006s	sys 0.000s
I20260709 12:27:01.860503 12587 leader_election.cc:304] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece; no voters: 
I20260709 12:27:01.864326 12621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:01.866967 12622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece" is_pre_election: true
I20260709 12:27:01.789733 12698 raft_consensus.cc:2468] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 1.
I20260709 12:27:01.873555 12623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd279d00e7142afab793d8849e54bf0" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:01.875357 12697 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08" is_pre_election: true
I20260709 12:27:01.875831 12695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd279d00e7142afab793d8849e54bf0" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08"
W20260709 12:27:01.876506 12735 leader_election.cc:343] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.876853 12694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08"
I20260709 12:27:01.877089 12695 raft_consensus.cc:3060] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.866186 12621 raft_consensus.cc:3060] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.824393 12820 raft_consensus.cc:697] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Becoming Leader. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:01.878034 12662 leader_election.cc:304] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd8de60841be432785aa6bde9d170a08; no voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece
I20260709 12:27:01.871706 12821 raft_consensus.cc:2804] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.881037 12821 raft_consensus.cc:697] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 1 LEADER]: Becoming Leader. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Running, Role: LEADER
I20260709 12:27:01.882082 12820 consensus_queue.cc:237] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.882879 12821 consensus_queue.cc:237] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
W20260709 12:27:01.884948 12736 leader_election.cc:343] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.885694 12736 leader_election.cc:304] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e; no voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08
I20260709 12:27:01.888034 12771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" is_pre_election: true
I20260709 12:27:01.889552 12771 raft_consensus.cc:2468] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 0.
I20260709 12:27:01.895707 12622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece" is_pre_election: true
I20260709 12:27:01.891824 12694 raft_consensus.cc:3060] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.900871 12662 leader_election.cc:304] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:01.902380 12858 raft_consensus.cc:2749] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:01.918032 12661 leader_election.cc:304] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd8de60841be432785aa6bde9d170a08; no voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece
W20260709 12:27:01.923359 12735 leader_election.cc:343] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.880124 12813 tablet_bootstrap.cc:492] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08: Bootstrap starting.
I20260709 12:27:01.937426 12695 raft_consensus.cc:2468] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 1.
I20260709 12:27:01.940927 12816 tablet_bootstrap.cc:654] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.944218 12519 catalog_manager.cc:5697] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195). New cstate: current_term: 1 leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:01.947181 12854 raft_consensus.cc:2804] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:01.947340 12622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece" is_pre_election: true
I20260709 12:27:01.947873 12854 raft_consensus.cc:493] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:01.948318 12854 raft_consensus.cc:3060] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:01.952021 12736 leader_election.cc:304] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08; no voters: 76884a9b95e64c5e95e950afda820ece
I20260709 12:27:01.953990 12858 raft_consensus.cc:2804] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.954558 12858 raft_consensus.cc:697] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Becoming Leader. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
W20260709 12:27:01.955858 12661 leader_election.cc:343] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:01.955955 12858 consensus_queue.cc:237] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.960333 12694 raft_consensus.cc:2468] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 1.
I20260709 12:27:01.961221 12817 raft_consensus.cc:2749] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:01.967387 12857 raft_consensus.cc:2749] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:01.968261 12812 tablet_bootstrap.cc:492] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e: No bootstrap required, opened a new log
I20260709 12:27:01.969170 12812 ts_tablet_manager.cc:1403] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent bootstrapping tablet: real 0.154s	user 0.010s	sys 0.008s
I20260709 12:27:01.968034 12520 catalog_manager.cc:5697] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd8de60841be432785aa6bde9d170a08 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "dd8de60841be432785aa6bde9d170a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:01.973867 12736 leader_election.cc:304] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:01.975507 12812 raft_consensus.cc:359] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.975976 12820 raft_consensus.cc:2804] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:01.976485 12812 raft_consensus.cc:385] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:01.976862 12812 raft_consensus.cc:740] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Initialized, Role: FOLLOWER
I20260709 12:27:01.977059 12820 raft_consensus.cc:697] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Becoming Leader. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:01.977969 12812 consensus_queue.cc:260] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:01.979319 12820 consensus_queue.cc:237] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:01.980644 12813 tablet_bootstrap.cc:654] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:01.981285 12812 ts_tablet_manager.cc:1434] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent starting tablet: real 0.012s	user 0.011s	sys 0.000s
I20260709 12:27:01.984592 12621 raft_consensus.cc:2468] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 1.
I20260709 12:27:01.993121 12812 tablet_bootstrap.cc:492] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Bootstrap starting.
I20260709 12:27:01.993486 12816 tablet_bootstrap.cc:492] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece: No bootstrap required, opened a new log
I20260709 12:27:01.996014 12816 ts_tablet_manager.cc:1403] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.175s	user 0.019s	sys 0.018s
I20260709 12:27:02.001943 12816 raft_consensus.cc:359] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.005807 12816 raft_consensus.cc:385] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.008999 12816 raft_consensus.cc:740] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.009908 12816 consensus_queue.cc:260] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.002938 12854 raft_consensus.cc:515] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.010775 12813 tablet_bootstrap.cc:492] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08: No bootstrap required, opened a new log
I20260709 12:27:02.011626 12813 ts_tablet_manager.cc:1403] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08: Time spent bootstrapping tablet: real 0.188s	user 0.017s	sys 0.005s
I20260709 12:27:02.013162 12816 ts_tablet_manager.cc:1434] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.016s	user 0.006s	sys 0.003s
I20260709 12:27:02.014858 12816 tablet_bootstrap.cc:492] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:02.015189 12771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
I20260709 12:27:02.015709 12621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:02.015961 12771 raft_consensus.cc:3060] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.013909 12521 catalog_manager.cc:5697] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece reported cstate change: term changed from 0 to 1, leader changed from <none> to 76884a9b95e64c5e95e950afda820ece (127.29.201.193). New cstate: current_term: 1 leader_uuid: "76884a9b95e64c5e95e950afda820ece" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:02.016808 12813 raft_consensus.cc:359] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
W20260709 12:27:02.017308 12661 leader_election.cc:343] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.017689 12813 raft_consensus.cc:385] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.017963 12813 raft_consensus.cc:740] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.018834 12813 consensus_queue.cc:260] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.022436 12813 ts_tablet_manager.cc:1434] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:27:02.023499 12816 tablet_bootstrap.cc:654] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.023916 12813 tablet_bootstrap.cc:492] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08: Bootstrap starting.
I20260709 12:27:02.024284 12854 leader_election.cc:290] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:02.026273 12812 tablet_bootstrap.cc:654] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.029600 12854 raft_consensus.cc:493] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.030424 12854 raft_consensus.cc:515] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.031227 12813 tablet_bootstrap.cc:654] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.029182 12771 raft_consensus.cc:2468] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 1.
I20260709 12:27:02.033670 12621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f0ff2156af842de82d34a04aab9cdf4" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece" is_pre_election: true
I20260709 12:27:02.034519 12621 raft_consensus.cc:2468] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 0.
I20260709 12:27:02.035092 12662 leader_election.cc:304] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08; no voters: 76884a9b95e64c5e95e950afda820ece
I20260709 12:27:02.036221 12854 leader_election.cc:290] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:02.036935 12854 raft_consensus.cc:2804] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:02.037441 12854 raft_consensus.cc:697] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Becoming Leader. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:02.038597 12854 consensus_queue.cc:237] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.038970 12770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f0ff2156af842de82d34a04aab9cdf4" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" is_pre_election: true
I20260709 12:27:02.040076 12661 leader_election.cc:304] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:02.041090 12817 raft_consensus.cc:2804] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:02.041483 12817 raft_consensus.cc:493] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:02.041926 12817 raft_consensus.cc:3060] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.043332 12812 tablet_bootstrap.cc:492] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e: No bootstrap required, opened a new log
I20260709 12:27:02.043846 12816 tablet_bootstrap.cc:492] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece: No bootstrap required, opened a new log
I20260709 12:27:02.043919 12812 ts_tablet_manager.cc:1403] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent bootstrapping tablet: real 0.060s	user 0.011s	sys 0.004s
I20260709 12:27:02.044674 12816 ts_tablet_manager.cc:1403] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.030s	user 0.018s	sys 0.012s
I20260709 12:27:02.047465 12812 raft_consensus.cc:359] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.047703 12816 raft_consensus.cc:359] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.048363 12812 raft_consensus.cc:385] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.048520 12816 raft_consensus.cc:385] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.048719 12812 raft_consensus.cc:740] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.048970 12816 raft_consensus.cc:740] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.049621 12812 consensus_queue.cc:260] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.049893 12816 consensus_queue.cc:260] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.040103 12770 raft_consensus.cc:2468] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 0.
I20260709 12:27:02.052340 12816 ts_tablet_manager.cc:1434] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.007s	user 0.002s	sys 0.004s
I20260709 12:27:02.052549 12812 ts_tablet_manager.cc:1434] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:27:02.053943 12816 tablet_bootstrap.cc:492] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:02.061774 12816 tablet_bootstrap.cc:654] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.068032 12820 raft_consensus.cc:493] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.070320 12820 raft_consensus.cc:515] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.069442 12817 raft_consensus.cc:515] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.073195 12820 leader_election.cc:290] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:02.073232 12816 tablet_bootstrap.cc:492] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece: No bootstrap required, opened a new log
I20260709 12:27:02.074012 12816 ts_tablet_manager.cc:1403] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.021s	user 0.010s	sys 0.009s
I20260709 12:27:02.074013 12694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08" is_pre_election: true
I20260709 12:27:02.074471 12622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece" is_pre_election: true
I20260709 12:27:02.075417 12622 raft_consensus.cc:2468] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 0.
W20260709 12:27:02.075953 12736 leader_election.cc:343] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.077272 12735 leader_election.cc:304] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece; no voters: dd8de60841be432785aa6bde9d170a08
I20260709 12:27:02.077610 12621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f0ff2156af842de82d34a04aab9cdf4" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:02.077962 12816 raft_consensus.cc:359] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.078365 12621 raft_consensus.cc:3060] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.078589 12820 raft_consensus.cc:2804] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:02.079077 12816 raft_consensus.cc:385] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.079187 12820 raft_consensus.cc:493] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:02.079488 12816 raft_consensus.cc:740] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.079674 12820 raft_consensus.cc:3060] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.080518 12816 consensus_queue.cc:260] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.083632 12816 ts_tablet_manager.cc:1434] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.001s
I20260709 12:27:02.085178 12816 tablet_bootstrap.cc:492] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:02.090427 12820 raft_consensus.cc:515] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.092235 12816 tablet_bootstrap.cc:654] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:02.094875 12694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08"
I20260709 12:27:02.095082 12622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:02.094349 12520 catalog_manager.cc:5697] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195). New cstate: current_term: 1 leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:02.095734 12622 raft_consensus.cc:3060] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:02.097100 12736 leader_election.cc:343] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:02.096621 12520 catalog_manager.cc:5697] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195). New cstate: current_term: 1 leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:02.097929 12820 leader_election.cc:290] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:02.098987 12520 catalog_manager.cc:5697] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195). New cstate: current_term: 1 leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:02.099725 12621 raft_consensus.cc:2468] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 1.
I20260709 12:27:02.101784 12661 leader_election.cc:304] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:02.102691 12857 raft_consensus.cc:2804] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:02.103147 12857 raft_consensus.cc:697] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Becoming Leader. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:02.104131 12857 consensus_queue.cc:237] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.105412 12770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f0ff2156af842de82d34a04aab9cdf4" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
I20260709 12:27:02.106109 12770 raft_consensus.cc:3060] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.107476 12816 tablet_bootstrap.cc:492] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece: No bootstrap required, opened a new log
I20260709 12:27:02.108034 12816 ts_tablet_manager.cc:1403] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.023s	user 0.007s	sys 0.008s
I20260709 12:27:02.111515 12816 raft_consensus.cc:359] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.112504 12816 raft_consensus.cc:385] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.112897 12816 raft_consensus.cc:740] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.113829 12816 consensus_queue.cc:260] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.118105 12770 raft_consensus.cc:2468] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 1.
I20260709 12:27:02.118590 12622 raft_consensus.cc:2468] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 1.
I20260709 12:27:02.119915 12817 leader_election.cc:290] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:02.120121 12735 leader_election.cc:304] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece; no voters: dd8de60841be432785aa6bde9d170a08
I20260709 12:27:02.121532 12820 raft_consensus.cc:2804] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:02.121990 12816 ts_tablet_manager.cc:1434] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:27:02.122095 12820 raft_consensus.cc:697] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Becoming Leader. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:02.122575 12813 tablet_bootstrap.cc:492] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08: No bootstrap required, opened a new log
I20260709 12:27:02.123190 12813 ts_tablet_manager.cc:1403] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08: Time spent bootstrapping tablet: real 0.100s	user 0.012s	sys 0.003s
I20260709 12:27:02.123099 12820 consensus_queue.cc:237] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.133976 12813 raft_consensus.cc:359] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.135191 12813 raft_consensus.cc:385] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:02.135514 12813 raft_consensus.cc:740] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Initialized, Role: FOLLOWER
I20260709 12:27:02.136341 12813 consensus_queue.cc:260] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.136170 12519 catalog_manager.cc:5697] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195). New cstate: current_term: 1 leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:02.137591 12520 catalog_manager.cc:5697] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd8de60841be432785aa6bde9d170a08 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "dd8de60841be432785aa6bde9d170a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:02.138738 12813 ts_tablet_manager.cc:1434] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08: Time spent starting tablet: real 0.015s	user 0.008s	sys 0.000s
I20260709 12:27:02.139395 12520 catalog_manager.cc:5697] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd8de60841be432785aa6bde9d170a08 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "dd8de60841be432785aa6bde9d170a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:02.141646 12622 raft_consensus.cc:3060] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.149588 12820 consensus_queue.cc:1048] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:02.155172 12817 raft_consensus.cc:493] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.156041 12817 raft_consensus.cc:515] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.158321 12817 leader_election.cc:290] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:02.159137 12621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece" is_pre_election: true
I20260709 12:27:02.159832 12621 raft_consensus.cc:2468] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 0.
I20260709 12:27:02.160665 12770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" is_pre_election: true
I20260709 12:27:02.161242 12661 leader_election.cc:304] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:02.161546 12770 raft_consensus.cc:2468] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 0.
I20260709 12:27:02.162408 12817 raft_consensus.cc:2804] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:02.162985 12817 raft_consensus.cc:493] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:02.163410 12817 raft_consensus.cc:3060] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.166373 12857 raft_consensus.cc:493] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.166992 12857 raft_consensus.cc:515] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.171132 12621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece" is_pre_election: true
I20260709 12:27:02.172369 12621 raft_consensus.cc:2393] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd8de60841be432785aa6bde9d170a08 in current term 1: Already voted for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in this term.
I20260709 12:27:02.173872 12817 raft_consensus.cc:515] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.176708 12770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" is_pre_election: true
I20260709 12:27:02.177213 12817 leader_election.cc:290] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:02.180437 12662 leader_election.cc:304] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd8de60841be432785aa6bde9d170a08; no voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece
I20260709 12:27:02.182179 12770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
I20260709 12:27:02.183194 12770 raft_consensus.cc:3060] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.185556 12621 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:02.186558 12621 raft_consensus.cc:3060] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.187503 12849 consensus_queue.cc:1048] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:02.191129 12821 raft_consensus.cc:493] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.191519 12817 raft_consensus.cc:3060] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.192070 12821 raft_consensus.cc:515] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.194758 12857 leader_election.cc:290] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:02.196480 12771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc" candidate_uuid: "76884a9b95e64c5e95e950afda820ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" is_pre_election: true
I20260709 12:27:02.197451 12771 raft_consensus.cc:2393] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76884a9b95e64c5e95e950afda820ece in current term 1: Already voted for candidate dd8de60841be432785aa6bde9d170a08 in this term.
I20260709 12:27:02.194561 12821 leader_election.cc:290] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:02.201992 12695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc" candidate_uuid: "76884a9b95e64c5e95e950afda820ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08" is_pre_election: true
I20260709 12:27:02.204428 12587 leader_election.cc:304] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece; no voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08
I20260709 12:27:02.206066 12821 raft_consensus.cc:3060] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.206889 12621 raft_consensus.cc:2468] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 1.
I20260709 12:27:02.208859 12661 leader_election.cc:304] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:02.209003 12817 raft_consensus.cc:2749] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:02.208971 12770 raft_consensus.cc:2468] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 1.
I20260709 12:27:02.210986 12857 raft_consensus.cc:2804] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:02.211663 12857 raft_consensus.cc:697] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Becoming Leader. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:02.212812 12857 consensus_queue.cc:237] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.228169 12821 raft_consensus.cc:2749] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:02.242257 12520 catalog_manager.cc:5697] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd8de60841be432785aa6bde9d170a08 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "dd8de60841be432785aa6bde9d170a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:02.296458 12817 consensus_queue.cc:1048] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:02.313280 12817 consensus_queue.cc:1048] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:02.328963 12849 consensus_queue.cc:1048] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:02.336719 12821 raft_consensus.cc:493] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:02.337291 12821 raft_consensus.cc:515] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:02.339540 12821 leader_election.cc:290] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:02.341542 30503 ts_manager.cc:295] Set tserver state for 76884a9b95e64c5e95e950afda820ece to MAINTENANCE_MODE
I20260709 12:27:02.342936 12770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd279d00e7142afab793d8849e54bf0" candidate_uuid: "76884a9b95e64c5e95e950afda820ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" is_pre_election: true
I20260709 12:27:02.343941 12695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd279d00e7142afab793d8849e54bf0" candidate_uuid: "76884a9b95e64c5e95e950afda820ece" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08" is_pre_election: true
I20260709 12:27:02.344923 12695 raft_consensus.cc:2393] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 76884a9b95e64c5e95e950afda820ece in current term 1: Already voted for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in this term.
I20260709 12:27:02.343029 30503 tablet_server.cc:179] TabletServer@127.29.201.193:0 shutting down...
I20260709 12:27:02.346369 12587 leader_election.cc:304] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece; no voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08
I20260709 12:27:02.346762 12858 consensus_queue.cc:1048] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:02.347538 12853 raft_consensus.cc:3060] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:02.363119 12853 raft_consensus.cc:2749] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:02.363504 12849 consensus_queue.cc:1048] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:02.397194 12853 consensus_queue.cc:1048] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260709 12:27:02.401459 12735 consensus_peers.cc:597] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:02.407343 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:02.410660 30503 tablet_replica.cc:333] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece: stopping tablet replica
I20260709 12:27:02.411602 30503 raft_consensus.cc:2243] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.412341 30503 raft_consensus.cc:2272] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.416975 12895 consensus_queue.cc:1048] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:02.417685 30503 tablet_replica.cc:333] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece: stopping tablet replica
I20260709 12:27:02.418699 30503 raft_consensus.cc:2243] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.419705 30503 raft_consensus.cc:2272] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.424023 30503 tablet_replica.cc:333] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece: stopping tablet replica
I20260709 12:27:02.424875 30503 raft_consensus.cc:2243] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.425365 30503 raft_consensus.cc:2272] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.428285 30503 tablet_replica.cc:333] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece: stopping tablet replica
I20260709 12:27:02.429232 30503 raft_consensus.cc:2243] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:02.430104 30503 raft_consensus.cc:2272] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.433418 30503 tablet_replica.cc:333] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece: stopping tablet replica
I20260709 12:27:02.434044 30503 raft_consensus.cc:2243] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.434448 30503 raft_consensus.cc:2272] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.436766 30503 tablet_replica.cc:333] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece: stopping tablet replica
I20260709 12:27:02.437397 30503 raft_consensus.cc:2243] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.437832 30503 raft_consensus.cc:2272] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.440172 30503 tablet_replica.cc:333] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece: stopping tablet replica
I20260709 12:27:02.440748 30503 raft_consensus.cc:2243] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.441188 30503 raft_consensus.cc:2272] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.443570 30503 tablet_replica.cc:333] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece: stopping tablet replica
I20260709 12:27:02.444123 30503 raft_consensus.cc:2243] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.444555 30503 raft_consensus.cc:2272] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.446738 30503 tablet_replica.cc:333] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece: stopping tablet replica
I20260709 12:27:02.447244 30503 raft_consensus.cc:2243] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.447687 30503 raft_consensus.cc:2272] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.450284 30503 tablet_replica.cc:333] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece: stopping tablet replica
I20260709 12:27:02.450860 30503 raft_consensus.cc:2243] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:02.451298 30503 raft_consensus.cc:2272] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:02.477427 12817 consensus_queue.cc:1048] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:02.477626 12858 consensus_queue.cc:1048] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:02.489229 30503 tablet_server.cc:196] TabletServer@127.29.201.193:0 shutdown complete.
W20260709 12:27:02.493072 12569 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:27:02.500721 12735 consensus_peers.cc:597] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:02.514220 12661 consensus_peers.cc:597] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:02.515015 12661 consensus_peers.cc:597] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:02.538261 12735 consensus_peers.cc:597] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:02.608453 12854 consensus_queue.cc:1048] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:02.711182 12849 consensus_queue.cc:1048] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:27:02.746907 12735 consensus_peers.cc:597] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:02.751863 12854 consensus_queue.cc:1048] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:27:02.767046 12661 consensus_peers.cc:597] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:02.838356 12661 consensus_peers.cc:597] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:02.858059 12735 consensus_peers.cc:597] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:03.494239 12569 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:27:03.698670 12661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111) [suppressed 124 similar messages]
I20260709 12:27:03.916889 12817 raft_consensus.cc:493] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76884a9b95e64c5e95e950afda820ece)
I20260709 12:27:03.917687 12817 raft_consensus.cc:515] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:03.919669 12817 leader_election.cc:290] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:03.920392 12858 raft_consensus.cc:493] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 76884a9b95e64c5e95e950afda820ece)
I20260709 12:27:03.921137 12770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a47a8c7a8101401297cf692f5358a7d2" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" is_pre_election: true
I20260709 12:27:03.921029 12858 raft_consensus.cc:515] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:03.922072 12770 raft_consensus.cc:2468] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 1.
I20260709 12:27:03.923369 12858 leader_election.cc:290] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:03.923440 12662 leader_election.cc:304] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08; no voters: 
W20260709 12:27:03.924220 12661 leader_election.cc:336] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111)
I20260709 12:27:03.924446 12817 raft_consensus.cc:2804] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260709 12:27:03.924922 12817 raft_consensus.cc:493] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 76884a9b95e64c5e95e950afda820ece)
I20260709 12:27:03.925351 12817 raft_consensus.cc:3060] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:03.925130 12695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a47a8c7a8101401297cf692f5358a7d2" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "dd8de60841be432785aa6bde9d170a08" is_pre_election: true
W20260709 12:27:03.927933 12735 leader_election.cc:336] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111)
I20260709 12:27:03.933341 12817 raft_consensus.cc:515] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:03.934504 12695 raft_consensus.cc:2393] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in current term 2: Already voted for candidate dd8de60841be432785aa6bde9d170a08 in this term.
I20260709 12:27:03.935427 12817 leader_election.cc:290] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 2 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:03.936758 12736 leader_election.cc:304] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e; no voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08
I20260709 12:27:03.936811 12770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a47a8c7a8101401297cf692f5358a7d2" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
I20260709 12:27:03.937579 12770 raft_consensus.cc:3060] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:03.940187 12661 leader_election.cc:336] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111)
I20260709 12:27:03.946519 12770 raft_consensus.cc:2468] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 2.
I20260709 12:27:03.947402 12858 raft_consensus.cc:2749] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:03.949401 12662 leader_election.cc:304] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08; no voters: 76884a9b95e64c5e95e950afda820ece
I20260709 12:27:03.950464 12817 raft_consensus.cc:2804] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:03.951707 12817 raft_consensus.cc:697] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Becoming Leader. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:03.952756 12817 consensus_queue.cc:237] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:03.961176 12519 catalog_manager.cc:5697] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 reported cstate change: term changed from 1 to 2, leader changed from 76884a9b95e64c5e95e950afda820ece (127.29.201.193) to dd8de60841be432785aa6bde9d170a08 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "dd8de60841be432785aa6bde9d170a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:04.480995 12661 consensus_peers.cc:597] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:04.495523 12569 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:04.505194 12770 raft_consensus.cc:1275] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:04.507695 12817 consensus_queue.cc:1048] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260709 12:27:04.689845 12735 consensus_peers.cc:597] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:04.733174 12735 consensus_peers.cc:597] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:05.014389 12661 consensus_peers.cc:597] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:05.054735 12735 consensus_peers.cc:597] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:05.076931 12661 consensus_peers.cc:597] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:05.123134 12661 consensus_peers.cc:597] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:05.151777 12735 consensus_peers.cc:597] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:05.229414 12661 consensus_peers.cc:597] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:05.353072 12735 consensus_peers.cc:597] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:05.496811 12569 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:27:06.498216 12569 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
W20260709 12:27:06.934870 12661 consensus_peers.cc:597] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:07.303225 12735 consensus_peers.cc:597] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:07.361246 12735 consensus_peers.cc:597] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:07.481901 12661 consensus_peers.cc:597] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:07.492199 12735 consensus_peers.cc:597] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20260709 12:27:07.499338 12569 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:07.534394 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:07.542989 12919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:07.543018 12918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:07.544318 12921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:07.545848 30503 server_base.cc:1061] running on GCE node
I20260709 12:27:07.549376 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:07.549690 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:07.549914 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600027549897 us; error 0 us; skew 500 ppm
I20260709 12:27:07.550813 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:07.555277 30503 webserver.cc:533] Webserver started at http://127.29.201.193:45725/ using document root <none> and password file <none>
I20260709 12:27:07.556133 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:07.556349 30503 fs_manager.cc:365] Using existing metadata directory in first data directory
I20260709 12:27:07.561077 30503 fs_manager.cc:714] Time spent opening directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:27:07.565042 12926 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:07.566004 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.002s	sys 0.000s
I20260709 12:27:07.566355 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-0-root
uuid: "76884a9b95e64c5e95e950afda820ece"
format_stamp: "Formatted at 2026-07-09 12:27:00 on dist-test-slave-qgf5"
I20260709 12:27:07.566805 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.TestMaintenanceMode.1783599553313367-30503-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260709 12:27:07.572397 12661 consensus_peers.cc:597] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:07.584334 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:07.585808 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:07.587708 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20260709 12:27:07.591830 12735 consensus_peers.cc:597] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:07.592483 12933 ts_tablet_manager.cc:548] Loading tablet metadata (0/10 complete)
I20260709 12:27:07.652433 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (10 total tablets, 10 live tablets)
I20260709 12:27:07.652858 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.061s	user 0.003s	sys 0.000s
I20260709 12:27:07.653203 30503 ts_tablet_manager.cc:600] Registering tablets (0/10 complete)
I20260709 12:27:07.658253 12933 tablet_bootstrap.cc:492] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:07.674022 12933 tablet_bootstrap.cc:492] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:07.677839 12933 tablet_bootstrap.cc:492] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece: Bootstrap complete.
W20260709 12:27:07.678555 12735 consensus_peers.cc:597] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:07.678615 12933 ts_tablet_manager.cc:1403] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.021s	user 0.017s	sys 0.000s
I20260709 12:27:07.681922 12933 raft_consensus.cc:359] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.682686 12933 raft_consensus.cc:740] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:07.683455 12933 consensus_queue.cc:260] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.686224 12933 ts_tablet_manager.cc:1434] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:27:07.687124 12933 tablet_bootstrap.cc:492] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
W20260709 12:27:07.704473 12661 consensus_peers.cc:597] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260709 12:27:07.704798 12933 tablet_bootstrap.cc:492] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:07.705750 12933 tablet_bootstrap.cc:492] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece: Bootstrap complete.
I20260709 12:27:07.706444 30503 ts_tablet_manager.cc:616] Registered 10 tablets
I20260709 12:27:07.706498 12933 ts_tablet_manager.cc:1403] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.020s	user 0.015s	sys 0.000s
I20260709 12:27:07.706877 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.054s	user 0.051s	sys 0.004s
I20260709 12:27:07.709230 12933 raft_consensus.cc:359] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.709816 12933 raft_consensus.cc:740] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:07.710593 12933 consensus_queue.cc:260] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.712385 12933 ts_tablet_manager.cc:1434] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.005s	user 0.004s	sys 0.000s
I20260709 12:27:07.713143 12933 tablet_bootstrap.cc:492] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:07.725373 12933 tablet_bootstrap.cc:492] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:07.726156 12933 tablet_bootstrap.cc:492] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece: Bootstrap complete.
I20260709 12:27:07.726899 12933 ts_tablet_manager.cc:1403] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.014s	user 0.003s	sys 0.010s
I20260709 12:27:07.729130 12933 raft_consensus.cc:359] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.729985 12933 raft_consensus.cc:740] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:07.731019 12933 consensus_queue.cc:260] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.733214 12933 ts_tablet_manager.cc:1434] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.006s	user 0.003s	sys 0.002s
I20260709 12:27:07.734176 12933 tablet_bootstrap.cc:492] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:07.745309 12933 tablet_bootstrap.cc:492] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:07.746196 12933 tablet_bootstrap.cc:492] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece: Bootstrap complete.
I20260709 12:27:07.747052 12933 ts_tablet_manager.cc:1403] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.013s	user 0.009s	sys 0.004s
I20260709 12:27:07.749792 12933 raft_consensus.cc:359] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.750509 12933 raft_consensus.cc:740] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:07.751283 12933 consensus_queue.cc:260] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.752852 12933 ts_tablet_manager.cc:1434] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.005s	user 0.003s	sys 0.000s
I20260709 12:27:07.753559 12933 tablet_bootstrap.cc:492] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:07.767728 12933 tablet_bootstrap.cc:492] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:07.768707 12933 tablet_bootstrap.cc:492] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece: Bootstrap complete.
I20260709 12:27:07.769781 12933 ts_tablet_manager.cc:1403] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.016s	user 0.013s	sys 0.000s
I20260709 12:27:07.772243 12933 raft_consensus.cc:359] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.772809 12933 raft_consensus.cc:740] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:07.773455 12933 consensus_queue.cc:260] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.775452 12933 ts_tablet_manager.cc:1434] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.005s	user 0.005s	sys 0.000s
I20260709 12:27:07.776297 12933 tablet_bootstrap.cc:492] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:07.776852 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.193:39205
I20260709 12:27:07.776969 12995 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.193:39205 every 8 connection(s)
I20260709 12:27:07.792641 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:07.797767 12933 tablet_bootstrap.cc:492] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:07.798833 12933 tablet_bootstrap.cc:492] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece: Bootstrap complete.
I20260709 12:27:07.799505 12933 ts_tablet_manager.cc:1403] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.023s	user 0.015s	sys 0.003s
I20260709 12:27:07.801987 12933 raft_consensus.cc:359] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.802697 12933 raft_consensus.cc:740] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:07.803653 12933 consensus_queue.cc:260] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.806064 12933 ts_tablet_manager.cc:1434] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.006s	user 0.005s	sys 0.000s
I20260709 12:27:07.814318 12933 tablet_bootstrap.cc:492] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:07.814553 12997 heartbeater.cc:344] Connected to a master server at 127.29.201.254:44111
I20260709 12:27:07.816032 12997 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:07.817036 12997 heartbeater.cc:507] Master 127.29.201.254:44111 requested a full tablet report, sending...
I20260709 12:27:07.829797 12695 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
mode: GRACEFUL
new_leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:55960
I20260709 12:27:07.830498 12695 raft_consensus.cc:606] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Received request to transfer leadership to TS 1b03c1a2d74a436ea0be8b84f9b3f75e
I20260709 12:27:07.830925 12520 ts_manager.cc:194] Re-registered known tserver with Master: 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205)
I20260709 12:27:07.838145 12933 tablet_bootstrap.cc:492] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:07.839478 12933 tablet_bootstrap.cc:492] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece: Bootstrap complete.
I20260709 12:27:07.840278 12933 ts_tablet_manager.cc:1403] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.026s	user 0.015s	sys 0.006s
I20260709 12:27:07.843205 12520 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:51938
I20260709 12:27:07.843389 12933 raft_consensus.cc:359] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.844302 12933 raft_consensus.cc:740] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:07.845108 12933 consensus_queue.cc:260] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.847375 12771 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
mode: GRACEFUL
new_leader_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:40580
I20260709 12:27:07.847910 12997 heartbeater.cc:499] Master 127.29.201.254:44111 was elected leader, sending a full tablet report...
I20260709 12:27:07.848131 12771 raft_consensus.cc:606] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Received request to transfer leadership to TS dd8de60841be432785aa6bde9d170a08
I20260709 12:27:07.848356 12933 ts_tablet_manager.cc:1434] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.001s
I20260709 12:27:07.849982 12933 tablet_bootstrap.cc:492] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:07.851234 13008 raft_consensus.cc:993] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08: : Instructing follower 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:07.851699 12695 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
mode: GRACEFUL
new_leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:55960
I20260709 12:27:07.852413 12695 raft_consensus.cc:606] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Received request to transfer leadership to TS 1b03c1a2d74a436ea0be8b84f9b3f75e
I20260709 12:27:07.854285 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:27:07.854333 12912 raft_consensus.cc:1081] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Signalling peer 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:07.855072 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:07.855809 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:07.856566 12771 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:37760
I20260709 12:27:07.857292 12771 raft_consensus.cc:493] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:07.857697 12771 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:07.868047 12771 raft_consensus.cc:515] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.877904 12695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dd8de60841be432785aa6bde9d170a08"
I20260709 12:27:07.879151 12695 raft_consensus.cc:3055] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:07.879671 12695 raft_consensus.cc:740] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:07.887952 12695 consensus_queue.cc:260] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.889415 12933 tablet_bootstrap.cc:492] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:07.891099 12933 tablet_bootstrap.cc:492] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece: Bootstrap complete.
I20260709 12:27:07.892175 12933 ts_tablet_manager.cc:1403] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.043s	user 0.026s	sys 0.012s
I20260709 12:27:07.894829 12695 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:07.896399 12969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:07.897606 12969 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:07.896152 12933 raft_consensus.cc:359] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.900739 12933 raft_consensus.cc:740] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:07.901608 12933 consensus_queue.cc:260] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.902813 12771 leader_election.cc:290] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 2 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:07.904384 12933 ts_tablet_manager.cc:1434] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.012s	user 0.006s	sys 0.000s
I20260709 12:27:07.905385 12933 tablet_bootstrap.cc:492] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:07.906401 12969 raft_consensus.cc:2468] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 2.
I20260709 12:27:07.908151 12735 leader_election.cc:304] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece; no voters: 
I20260709 12:27:07.909080 12916 raft_consensus.cc:2804] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:07.911075 12916 raft_consensus.cc:697] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Becoming Leader. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:07.912160 12916 consensus_queue.cc:237] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.927670 12520 catalog_manager.cc:5697] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e reported cstate change: term changed from 1 to 2, leader changed from dd8de60841be432785aa6bde9d170a08 (127.29.201.194) to 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195). New cstate: current_term: 2 leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:07.932142 12933 tablet_bootstrap.cc:492] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:07.933171 12933 tablet_bootstrap.cc:492] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece: Bootstrap complete.
I20260709 12:27:07.933966 12933 ts_tablet_manager.cc:1403] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.029s	user 0.019s	sys 0.004s
I20260709 12:27:07.934304 12695 raft_consensus.cc:2468] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 2.
I20260709 12:27:07.937260 12933 raft_consensus.cc:359] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:07.937897 12933 raft_consensus.cc:740] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:07.938534 12933 consensus_queue.cc:260] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:07.940653 12933 ts_tablet_manager.cc:1434] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:27:07.941552 12933 tablet_bootstrap.cc:492] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece: Bootstrap starting.
I20260709 12:27:07.950220 13012 raft_consensus.cc:993] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e: : Instructing follower dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:07.950838 12916 raft_consensus.cc:1081] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Signalling peer dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:07.952466 12695 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:53578
I20260709 12:27:07.953040 12695 raft_consensus.cc:493] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:07.953425 12695 raft_consensus.cc:3060] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:07.956892 12933 tablet_bootstrap.cc:492] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260709 12:27:07.957988 12933 tablet_bootstrap.cc:492] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece: Bootstrap complete.
I20260709 12:27:07.958797 12933 ts_tablet_manager.cc:1403] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece: Time spent bootstrapping tablet: real 0.017s	user 0.016s	sys 0.000s
I20260709 12:27:07.960295 12695 raft_consensus.cc:515] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.962621 12695 leader_election.cc:290] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 2 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:07.961866 12933 raft_consensus.cc:359] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.963658 12970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:07.963835 12771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
I20260709 12:27:07.964025 12933 raft_consensus.cc:740] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76884a9b95e64c5e95e950afda820ece, State: Initialized, Role: FOLLOWER
I20260709 12:27:07.964653 12771 raft_consensus.cc:3055] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:07.965058 12771 raft_consensus.cc:740] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:07.965245 12933 consensus_queue.cc:260] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.966331 12970 raft_consensus.cc:3060] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:07.966121 12771 consensus_queue.cc:260] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.967705 12771 raft_consensus.cc:3060] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:07.967842 12933 ts_tablet_manager.cc:1434] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.000s
I20260709 12:27:07.974068 12970 raft_consensus.cc:2468] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 2.
I20260709 12:27:07.974475 12771 raft_consensus.cc:2468] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 2.
I20260709 12:27:07.976158 12662 leader_election.cc:304] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:07.977262 13009 raft_consensus.cc:2804] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:07.977851 13009 raft_consensus.cc:697] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Becoming Leader. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:07.978863 13009 consensus_queue.cc:237] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:07.990784 12521 catalog_manager.cc:5697] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 reported cstate change: term changed from 1 to 2, leader changed from 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195) to dd8de60841be432785aa6bde9d170a08 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "dd8de60841be432785aa6bde9d170a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:08.024647 12970 raft_consensus.cc:3060] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:08.281111 12912 raft_consensus.cc:993] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08: : Instructing follower 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:08.281562 13009 raft_consensus.cc:1081] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Signalling peer 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:08.283355 12771 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:37760
I20260709 12:27:08.283955 12771 raft_consensus.cc:493] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:08.284287 12771 raft_consensus.cc:3060] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:08.290308 12771 raft_consensus.cc:515] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:08.292088 12771 leader_election.cc:290] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 2 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:08.293016 12970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:08.293242 12695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dd8de60841be432785aa6bde9d170a08"
I20260709 12:27:08.293617 12970 raft_consensus.cc:3060] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:08.293900 12695 raft_consensus.cc:3055] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:08.294261 12695 raft_consensus.cc:740] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:08.295019 12695 consensus_queue.cc:260] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:08.296137 12695 raft_consensus.cc:3060] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:08.299705 12970 raft_consensus.cc:2468] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 2.
I20260709 12:27:08.301031 12735 leader_election.cc:304] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece; no voters: 
I20260709 12:27:08.301857 12916 raft_consensus.cc:2804] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:08.302328 12916 raft_consensus.cc:697] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Becoming Leader. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:08.303184 12695 raft_consensus.cc:2468] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 2.
I20260709 12:27:08.303148 12916 consensus_queue.cc:237] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:08.309105 12695 raft_consensus.cc:1275] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:08.313831 12916 consensus_queue.cc:1048] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:08.315924 12520 catalog_manager.cc:5697] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e reported cstate change: term changed from 1 to 2, leader changed from dd8de60841be432785aa6bde9d170a08 (127.29.201.194) to 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195). New cstate: current_term: 2 leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:08.334291 12970 raft_consensus.cc:1275] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:08.335784 13029 consensus_queue.cc:1048] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:08.465847 12970 raft_consensus.cc:1275] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:08.465848 12771 raft_consensus.cc:1275] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:08.467777 13009 consensus_queue.cc:1048] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:08.468504 12912 consensus_queue.cc:1048] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
W20260709 12:27:08.500412 12569 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:08.709681 12695 raft_consensus.cc:1275] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:08.711329 12916 consensus_queue.cc:1048] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:08.720708 12970 raft_consensus.cc:1275] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:08.722818 13012 consensus_queue.cc:1048] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:08.868095 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:08.876123 12771 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
mode: GRACEFUL
new_leader_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:40580
I20260709 12:27:08.876881 12771 raft_consensus.cc:606] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Received request to transfer leadership to TS dd8de60841be432785aa6bde9d170a08
I20260709 12:27:08.879998 12695 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
mode: GRACEFUL
new_leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:55960
I20260709 12:27:08.880636 12695 raft_consensus.cc:606] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Received request to transfer leadership to TS 1b03c1a2d74a436ea0be8b84f9b3f75e
I20260709 12:27:08.883616 12771 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
mode: GRACEFUL
new_leader_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:40580
I20260709 12:27:08.884182 12771 raft_consensus.cc:606] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Received request to transfer leadership to TS dd8de60841be432785aa6bde9d170a08
I20260709 12:27:08.885427 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:27:08.885998 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:08.886648 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:09.088945 12912 raft_consensus.cc:993] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08: : Instructing follower 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:09.089568 13009 raft_consensus.cc:1081] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Signalling peer 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:09.091223 12771 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:37760
I20260709 12:27:09.091706 12771 raft_consensus.cc:493] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:09.091969 12771 raft_consensus.cc:3060] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:09.097510 12771 raft_consensus.cc:515] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:09.099253 12771 leader_election.cc:290] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 3 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:09.100157 12970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:09.100271 12695 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66d601ec378b47b9834fefc2ccf51ce3" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dd8de60841be432785aa6bde9d170a08"
I20260709 12:27:09.100776 12970 raft_consensus.cc:3060] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:09.101006 12695 raft_consensus.cc:3055] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:09.101394 12695 raft_consensus.cc:740] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:09.102180 12695 consensus_queue.cc:260] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:09.103252 12695 raft_consensus.cc:3060] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:09.106933 12970 raft_consensus.cc:2468] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 3.
I20260709 12:27:09.108121 12735 leader_election.cc:304] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece; no voters: 
I20260709 12:27:09.108886 13012 raft_consensus.cc:2804] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:09.109398 13012 raft_consensus.cc:697] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 LEADER]: Becoming Leader. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:09.109803 12695 raft_consensus.cc:2468] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 3.
I20260709 12:27:09.110195 13012 consensus_queue.cc:237] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:09.117444 12521 catalog_manager.cc:5697] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e reported cstate change: term changed from 2 to 3, leader changed from dd8de60841be432785aa6bde9d170a08 (127.29.201.194) to 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195). New cstate: current_term: 3 leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:09.125277 13029 raft_consensus.cc:993] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e: : Instructing follower dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:09.126174 13029 raft_consensus.cc:1081] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Signalling peer dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:09.128041 12694 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:53578
I20260709 12:27:09.128656 12694 raft_consensus.cc:493] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:09.129056 12694 raft_consensus.cc:3060] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:09.135078 12694 raft_consensus.cc:515] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:09.137027 12694 leader_election.cc:290] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 3 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:09.138011 12969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:09.138201 12771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
I20260709 12:27:09.138789 12969 raft_consensus.cc:3060] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:09.138898 12771 raft_consensus.cc:3055] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:09.139400 12771 raft_consensus.cc:740] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:09.140089 12771 consensus_queue.cc:260] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:09.141155 12771 raft_consensus.cc:3060] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:09.144536 12969 raft_consensus.cc:2468] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 3.
I20260709 12:27:09.145767 12661 leader_election.cc:304] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:09.146469 13009 raft_consensus.cc:2804] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:09.146543 12771 raft_consensus.cc:2468] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 3.
I20260709 12:27:09.146978 13009 raft_consensus.cc:697] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 3 LEADER]: Becoming Leader. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:09.148221 13009 consensus_queue.cc:237] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:09.155119 12521 catalog_manager.cc:5697] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 reported cstate change: term changed from 2 to 3, leader changed from 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195) to dd8de60841be432785aa6bde9d170a08 (127.29.201.194). New cstate: current_term: 3 leader_uuid: "dd8de60841be432785aa6bde9d170a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:09.199998 13029 raft_consensus.cc:993] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e: : Instructing follower dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:09.200484 12916 raft_consensus.cc:1081] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Signalling peer dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:09.202252 12694 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:53578
I20260709 12:27:09.202836 12694 raft_consensus.cc:493] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:09.203171 12694 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:09.208777 12694 raft_consensus.cc:515] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:09.210526 12694 leader_election.cc:290] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 3 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:09.211571 12969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:09.211571 12771 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
I20260709 12:27:09.212538 12969 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:09.212698 12771 raft_consensus.cc:3055] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:09.213146 12771 raft_consensus.cc:740] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:09.213917 12771 consensus_queue.cc:260] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:09.214984 12771 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:09.217981 12969 raft_consensus.cc:2468] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 3.
I20260709 12:27:09.219236 12661 leader_election.cc:304] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:09.220150 13009 raft_consensus.cc:2804] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:09.220619 12771 raft_consensus.cc:2468] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 3.
I20260709 12:27:09.220688 13009 raft_consensus.cc:697] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 3 LEADER]: Becoming Leader. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:09.222205 13009 consensus_queue.cc:237] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:09.228446 12521 catalog_manager.cc:5697] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 reported cstate change: term changed from 2 to 3, leader changed from 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195) to dd8de60841be432785aa6bde9d170a08 (127.29.201.194). New cstate: current_term: 3 leader_uuid: "dd8de60841be432785aa6bde9d170a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:09.501447 12569 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:09.555712 12969 raft_consensus.cc:1275] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:09.557137 13029 consensus_queue.cc:1048] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:09.564153 12970 raft_consensus.cc:1275] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:09.566475 12912 consensus_queue.cc:1048] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:09.573530 12694 raft_consensus.cc:1275] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:09.576233 12916 consensus_queue.cc:1048] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:09.585080 12771 raft_consensus.cc:1275] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:09.587723 12912 consensus_queue.cc:1048] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:09.601295 12969 raft_consensus.cc:1275] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:09.602582 12912 consensus_queue.cc:1048] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:09.611681 12771 raft_consensus.cc:1275] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:09.614693 13009 consensus_queue.cc:1048] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:09.893723 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:09.901556 12694 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a47a8c7a8101401297cf692f5358a7d2"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
mode: GRACEFUL
new_leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:55960
I20260709 12:27:09.902107 12694 raft_consensus.cc:606] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Received request to transfer leadership to TS 1b03c1a2d74a436ea0be8b84f9b3f75e
I20260709 12:27:09.905088 12694 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
mode: GRACEFUL
new_leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:55960
I20260709 12:27:09.905839 12694 raft_consensus.cc:606] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Received request to transfer leadership to TS 1b03c1a2d74a436ea0be8b84f9b3f75e
I20260709 12:27:09.909014 12770 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "fbd279d00e7142afab793d8849e54bf0"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
mode: GRACEFUL
new_leader_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:40580
I20260709 12:27:09.909646 12770 raft_consensus.cc:606] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Received request to transfer leadership to TS dd8de60841be432785aa6bde9d170a08
I20260709 12:27:09.911091 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:27:09.911757 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:09.912384 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:10.085510 13012 raft_consensus.cc:993] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: : Instructing follower dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:10.086129 13029 raft_consensus.cc:1081] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Signalling peer dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:10.087873 12694 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "fbd279d00e7142afab793d8849e54bf0"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:53578
I20260709 12:27:10.088383 12694 raft_consensus.cc:493] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:10.088693 12694 raft_consensus.cc:3060] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.094332 12694 raft_consensus.cc:515] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:10.096987 12694 leader_election.cc:290] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 2 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:10.097566 12969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd279d00e7142afab793d8849e54bf0" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:10.097584 12770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fbd279d00e7142afab793d8849e54bf0" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
I20260709 12:27:10.098706 12969 raft_consensus.cc:3060] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.098770 12770 raft_consensus.cc:3055] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:10.099346 12770 raft_consensus.cc:740] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:10.100345 12770 consensus_queue.cc:260] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:10.101854 12770 raft_consensus.cc:3060] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.107672 12969 raft_consensus.cc:2468] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 2.
I20260709 12:27:10.109278 12661 leader_election.cc:304] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:10.110049 13042 raft_consensus.cc:2804] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:10.110306 12770 raft_consensus.cc:2468] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 2.
I20260709 12:27:10.110678 13042 raft_consensus.cc:697] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Becoming Leader. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:10.112409 13042 consensus_queue.cc:237] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:10.121481 12520 catalog_manager.cc:5697] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 reported cstate change: term changed from 1 to 2, leader changed from 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195) to dd8de60841be432785aa6bde9d170a08 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "dd8de60841be432785aa6bde9d170a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:10.203624 12912 raft_consensus.cc:993] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08: : Instructing follower 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:10.204206 13042 raft_consensus.cc:1081] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Signalling peer 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:10.205943 12770 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:37760
I20260709 12:27:10.206498 12770 raft_consensus.cc:493] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:10.206902 12770 raft_consensus.cc:3060] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.212344 12770 raft_consensus.cc:515] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:10.214179 12770 leader_election.cc:290] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 2 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:10.215260 12969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:10.215270 12694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dd8de60841be432785aa6bde9d170a08"
I20260709 12:27:10.215996 12969 raft_consensus.cc:3060] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.216099 12694 raft_consensus.cc:3055] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:10.216588 12694 raft_consensus.cc:740] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:10.217417 12694 consensus_queue.cc:260] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:10.218719 12694 raft_consensus.cc:3060] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:10.222988 12969 raft_consensus.cc:2468] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 2.
I20260709 12:27:10.224264 12735 leader_election.cc:304] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece; no voters: 
I20260709 12:27:10.224932 12694 raft_consensus.cc:2468] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 2.
I20260709 12:27:10.225116 13029 raft_consensus.cc:2804] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:10.225723 13029 raft_consensus.cc:697] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Becoming Leader. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:10.226744 13029 consensus_queue.cc:237] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:10.234326 12520 catalog_manager.cc:5697] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e reported cstate change: term changed from 1 to 2, leader changed from dd8de60841be432785aa6bde9d170a08 (127.29.201.194) to 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195). New cstate: current_term: 2 leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:10.310496 12912 raft_consensus.cc:993] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08: : Instructing follower 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:10.311209 12912 raft_consensus.cc:1081] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Signalling peer 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:10.312835 12770 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a47a8c7a8101401297cf692f5358a7d2"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:37760
I20260709 12:27:10.313513 12770 raft_consensus.cc:493] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:10.313841 12770 raft_consensus.cc:3060] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:10.319262 12770 raft_consensus.cc:515] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:10.320916 12770 leader_election.cc:290] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 3 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:10.322016 12969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a47a8c7a8101401297cf692f5358a7d2" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:10.322113 12694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a47a8c7a8101401297cf692f5358a7d2" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dd8de60841be432785aa6bde9d170a08"
I20260709 12:27:10.322769 12969 raft_consensus.cc:3060] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:10.322851 12694 raft_consensus.cc:3055] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:10.323324 12694 raft_consensus.cc:740] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:10.324085 12694 consensus_queue.cc:260] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:10.325104 12694 raft_consensus.cc:3060] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:10.328526 12969 raft_consensus.cc:2468] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 3.
I20260709 12:27:10.330005 12735 leader_election.cc:304] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece; no voters: 
I20260709 12:27:10.330773 12694 raft_consensus.cc:2468] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 3.
I20260709 12:27:10.330895 13029 raft_consensus.cc:2804] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:10.331591 13029 raft_consensus.cc:697] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 LEADER]: Becoming Leader. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:10.332542 13029 consensus_queue.cc:237] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:10.340855 12520 catalog_manager.cc:5697] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e reported cstate change: term changed from 2 to 3, leader changed from dd8de60841be432785aa6bde9d170a08 (127.29.201.194) to 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195). New cstate: current_term: 3 leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:10.502609 12569 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:10.638226 12969 raft_consensus.cc:1275] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:10.639801 12912 consensus_queue.cc:1048] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:10.651353 12770 raft_consensus.cc:1275] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:10.652896 12912 consensus_queue.cc:1048] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:10.722522 12694 raft_consensus.cc:1275] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:10.723961 13029 consensus_queue.cc:1048] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:10.734648 12969 raft_consensus.cc:1275] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:10.736546 13029 consensus_queue.cc:1048] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:10.737833 12969 raft_consensus.cc:1275] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:10.740278 13012 consensus_queue.cc:1048] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:10.754673 12694 raft_consensus.cc:1275] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:10.756224 13012 consensus_queue.cc:1048] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:10.919482 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:10.926914 12694 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
mode: GRACEFUL
new_leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:55960
I20260709 12:27:10.927565 12694 raft_consensus.cc:606] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 3 LEADER]: Received request to transfer leadership to TS 1b03c1a2d74a436ea0be8b84f9b3f75e
I20260709 12:27:10.930493 12770 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
mode: GRACEFUL
new_leader_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:40580
I20260709 12:27:10.931094 12770 raft_consensus.cc:606] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Received request to transfer leadership to TS dd8de60841be432785aa6bde9d170a08
I20260709 12:27:10.933903 12770 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
mode: GRACEFUL
new_leader_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:40580
I20260709 12:27:10.934474 12770 raft_consensus.cc:606] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Received request to transfer leadership to TS dd8de60841be432785aa6bde9d170a08
I20260709 12:27:10.935633 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:27:10.936175 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:10.936898 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:11.057457 13029 raft_consensus.cc:993] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e: : Instructing follower dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:11.058161 13012 raft_consensus.cc:1081] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Signalling peer dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:11.059808 12694 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:53578
I20260709 12:27:11.060366 12694 raft_consensus.cc:493] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:11.060772 12694 raft_consensus.cc:3060] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:11.066339 12694 raft_consensus.cc:515] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:11.068311 12694 leader_election.cc:290] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 2 election: Requested vote from peers 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647), 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205)
I20260709 12:27:11.069258 12770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
I20260709 12:27:11.069684 12970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:11.070041 12770 raft_consensus.cc:3055] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:11.070411 12970 raft_consensus.cc:3060] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:11.070408 12770 raft_consensus.cc:740] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:11.071386 12770 consensus_queue.cc:260] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:11.072350 12770 raft_consensus.cc:3060] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:11.076285 12970 raft_consensus.cc:2468] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 2.
I20260709 12:27:11.077271 12770 raft_consensus.cc:2468] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 2.
I20260709 12:27:11.077607 12661 leader_election.cc:304] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:11.078485 13057 raft_consensus.cc:2804] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:11.079011 13057 raft_consensus.cc:697] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Becoming Leader. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:11.079800 13057 consensus_queue.cc:237] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:11.087350 12521 catalog_manager.cc:5697] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 reported cstate change: term changed from 1 to 2, leader changed from 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195) to dd8de60841be432785aa6bde9d170a08 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "dd8de60841be432785aa6bde9d170a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:11.262094 12912 raft_consensus.cc:993] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08: : Instructing follower 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:11.262676 13069 raft_consensus.cc:1081] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 3 LEADER]: Signalling peer 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:11.264868 12770 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:37760
I20260709 12:27:11.265421 12770 raft_consensus.cc:493] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:11.265829 12770 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:11.272913 12770 raft_consensus.cc:515] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:11.275206 12770 leader_election.cc:290] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 4 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:11.276472 12694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dd8de60841be432785aa6bde9d170a08"
I20260709 12:27:11.276815 12970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:11.277304 12694 raft_consensus.cc:3055] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:27:11.277442 12970 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:11.277733 12694 raft_consensus.cc:740] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:11.278772 12694 consensus_queue.cc:260] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:11.280220 12694 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:11.286993 12970 raft_consensus.cc:2468] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 4.
I20260709 12:27:11.288659 12735 leader_election.cc:304] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece; no voters: 
I20260709 12:27:11.288965 12694 raft_consensus.cc:2468] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 4.
I20260709 12:27:11.289695 13012 raft_consensus.cc:2804] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:27:11.290307 13012 raft_consensus.cc:697] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 4 LEADER]: Becoming Leader. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:11.291239 13012 consensus_queue.cc:237] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:11.301324 12521 catalog_manager.cc:5697] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e reported cstate change: term changed from 3 to 4, leader changed from dd8de60841be432785aa6bde9d170a08 (127.29.201.194) to 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195). New cstate: current_term: 4 leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:11.364662 13029 raft_consensus.cc:993] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e: : Instructing follower dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:11.365265 13012 raft_consensus.cc:1081] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Signalling peer dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:11.367024 12694 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:53578
I20260709 12:27:11.367507 12694 raft_consensus.cc:493] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:11.367780 12694 raft_consensus.cc:3060] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:11.374464 12694 raft_consensus.cc:515] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:11.376187 12694 leader_election.cc:290] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 3 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:11.377182 12770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
I20260709 12:27:11.377254 12970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cc7e1418d7c145b88a45a159d25e21cc" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:11.377933 12770 raft_consensus.cc:3055] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:11.378041 12970 raft_consensus.cc:3060] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:11.378227 12770 raft_consensus.cc:740] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:11.379117 12770 consensus_queue.cc:260] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:11.380018 12770 raft_consensus.cc:3060] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:11.383875 12970 raft_consensus.cc:2468] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 3.
I20260709 12:27:11.385043 12770 raft_consensus.cc:2468] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 3.
I20260709 12:27:11.385201 12661 leader_election.cc:304] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:11.386045 13069 raft_consensus.cc:2804] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:11.386688 13069 raft_consensus.cc:697] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 3 LEADER]: Becoming Leader. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:11.387619 13069 consensus_queue.cc:237] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:11.394421 12521 catalog_manager.cc:5697] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 reported cstate change: term changed from 2 to 3, leader changed from 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195) to dd8de60841be432785aa6bde9d170a08 (127.29.201.194). New cstate: current_term: 3 leader_uuid: "dd8de60841be432785aa6bde9d170a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:11.464744 12770 raft_consensus.cc:1275] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.466207 13069 consensus_queue.cc:1048] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:11.477099 12970 raft_consensus.cc:1275] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:11.478953 12912 consensus_queue.cc:1048] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:27:11.503649 12569 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:11.844099 12969 raft_consensus.cc:1275] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 4 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:27:11.845706 13012 consensus_queue.cc:1048] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:27:11.855108 12694 raft_consensus.cc:1275] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 4 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:27:11.857681 13061 consensus_queue.cc:1048] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:27:11.927789 12970 raft_consensus.cc:1275] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:11.932621 13057 consensus_queue.cc:1048] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:27:11.946918 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:11.951032 12770 raft_consensus.cc:1275] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:11.954984 13080 consensus_queue.cc:1048] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:27:11.970752 12770 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
mode: GRACEFUL
new_leader_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:40580
I20260709 12:27:11.971863 12770 raft_consensus.cc:606] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 4 LEADER]: Received request to transfer leadership to TS dd8de60841be432785aa6bde9d170a08
I20260709 12:27:11.978256 12694 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
mode: GRACEFUL
new_leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:55960
I20260709 12:27:11.979009 12694 raft_consensus.cc:606] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Received request to transfer leadership to TS 1b03c1a2d74a436ea0be8b84f9b3f75e
I20260709 12:27:11.982518 12694 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
mode: GRACEFUL
new_leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:55960
I20260709 12:27:11.983228 12694 raft_consensus.cc:606] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 3 LEADER]: Received request to transfer leadership to TS 1b03c1a2d74a436ea0be8b84f9b3f75e
I20260709 12:27:11.984387 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 3
I20260709 12:27:11.984923 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:11.985658 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:12.023073 13080 raft_consensus.cc:993] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08: : Instructing follower 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:12.023699 13080 raft_consensus.cc:1081] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Signalling peer 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:12.025455 12770 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:37760
I20260709 12:27:12.026000 12770 raft_consensus.cc:493] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:12.026355 12770 raft_consensus.cc:3060] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:12.034524 12770 raft_consensus.cc:515] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:12.036365 12770 leader_election.cc:290] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 3 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:12.037582 12970 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:12.037665 12694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ade88c76c8f34edfac61ded6c3d472f6" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "dd8de60841be432785aa6bde9d170a08"
I20260709 12:27:12.038584 12970 raft_consensus.cc:3060] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:12.038798 12694 raft_consensus.cc:3055] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:12.039438 12694 raft_consensus.cc:740] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:12.040504 12694 consensus_queue.cc:260] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:12.041975 12694 raft_consensus.cc:3060] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:12.049201 12970 raft_consensus.cc:2468] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 3.
I20260709 12:27:12.051170 12735 leader_election.cc:304] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, 76884a9b95e64c5e95e950afda820ece; no voters: 
I20260709 12:27:12.052840 13061 raft_consensus.cc:2804] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:12.053409 13061 raft_consensus.cc:697] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 LEADER]: Becoming Leader. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:12.054916 12694 raft_consensus.cc:2468] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 3.
I20260709 12:27:12.054785 13061 consensus_queue.cc:237] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } }
I20260709 12:27:12.062558 12521 catalog_manager.cc:5697] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e reported cstate change: term changed from 2 to 3, leader changed from dd8de60841be432785aa6bde9d170a08 (127.29.201.194) to 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195). New cstate: current_term: 3 leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:12.174990 13057 raft_consensus.cc:993] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08: : Instructing follower 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:12.175570 13080 raft_consensus.cc:1081] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 3 LEADER]: Signalling peer 1b03c1a2d74a436ea0be8b84f9b3f75e to start an election
I20260709 12:27:12.177628 12770 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f"
dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
 from {username='slave'} at 127.0.0.1:37760
I20260709 12:27:12.178190 12770 raft_consensus.cc:493] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:12.178493 12770 raft_consensus.cc:3060] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:12.184609 12770 raft_consensus.cc:515] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:12.186403 12770 leader_election.cc:290] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 4 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355)
I20260709 12:27:12.187493 12694 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "dd8de60841be432785aa6bde9d170a08"
I20260709 12:27:12.187671 12969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b909763b4c1045178a6fc9fd1a58250f" candidate_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:12.188237 12694 raft_consensus.cc:3055] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 3 LEADER]: Stepping down as leader of term 3
I20260709 12:27:12.188557 12694 raft_consensus.cc:740] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:12.188354 12969 raft_consensus.cc:3060] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:12.189913 12694 consensus_queue.cc:260] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:12.191182 12694 raft_consensus.cc:3060] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Advancing to term 4
I20260709 12:27:12.196789 12694 raft_consensus.cc:2468] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 4.
I20260709 12:27:12.196789 12969 raft_consensus.cc:2468] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b03c1a2d74a436ea0be8b84f9b3f75e in term 4.
I20260709 12:27:12.198468 12736 leader_election.cc:304] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b03c1a2d74a436ea0be8b84f9b3f75e, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:12.199440 13012 raft_consensus.cc:2804] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 4 FOLLOWER]: Leader election won for term 4
I20260709 12:27:12.199906 13012 raft_consensus.cc:697] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 4 LEADER]: Becoming Leader. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:12.200866 13012 consensus_queue.cc:237] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:12.210264 12521 catalog_manager.cc:5697] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e reported cstate change: term changed from 3 to 4, leader changed from dd8de60841be432785aa6bde9d170a08 (127.29.201.194) to 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195). New cstate: current_term: 4 leader_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:12.286087 13012 raft_consensus.cc:993] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e: : Instructing follower dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:12.286989 13029 raft_consensus.cc:1081] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 4 LEADER]: Signalling peer dd8de60841be432785aa6bde9d170a08 to start an election
I20260709 12:27:12.289139 12694 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e"
dest_uuid: "dd8de60841be432785aa6bde9d170a08"
 from {username='slave'} at 127.0.0.1:53578
I20260709 12:27:12.289780 12694 raft_consensus.cc:493] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:12.290110 12694 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:27:12.296020 12694 raft_consensus.cc:515] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:12.297706 12694 leader_election.cc:290] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 5 election: Requested vote from peers 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205), 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195:44647)
I20260709 12:27:12.298916 12770 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e"
I20260709 12:27:12.298910 12969 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6041407ddd7647db97a3e044abd5574e" candidate_uuid: "dd8de60841be432785aa6bde9d170a08" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: true dest_uuid: "76884a9b95e64c5e95e950afda820ece"
I20260709 12:27:12.299587 12770 raft_consensus.cc:3055] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 4 LEADER]: Stepping down as leader of term 4
I20260709 12:27:12.299842 12969 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:27:12.299930 12770 raft_consensus.cc:740] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 1b03c1a2d74a436ea0be8b84f9b3f75e, State: Running, Role: LEADER
I20260709 12:27:12.300855 12770 consensus_queue.cc:260] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:12.302162 12770 raft_consensus.cc:3060] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 4 FOLLOWER]: Advancing to term 5
I20260709 12:27:12.307588 12969 raft_consensus.cc:2468] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 5.
I20260709 12:27:12.307803 12770 raft_consensus.cc:2468] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd8de60841be432785aa6bde9d170a08 in term 5.
I20260709 12:27:12.309031 12661 leader_election.cc:304] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76884a9b95e64c5e95e950afda820ece, dd8de60841be432785aa6bde9d170a08; no voters: 
I20260709 12:27:12.309906 13080 raft_consensus.cc:2804] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 5 FOLLOWER]: Leader election won for term 5
I20260709 12:27:12.310370 13080 raft_consensus.cc:697] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 5 LEADER]: Becoming Leader. State: Replica: dd8de60841be432785aa6bde9d170a08, State: Running, Role: LEADER
I20260709 12:27:12.311298 13080 consensus_queue.cc:237] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } }
I20260709 12:27:12.318603 12521 catalog_manager.cc:5697] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 reported cstate change: term changed from 4 to 5, leader changed from 1b03c1a2d74a436ea0be8b84f9b3f75e (127.29.201.195) to dd8de60841be432785aa6bde9d170a08 (127.29.201.194). New cstate: current_term: 5 leader_uuid: "dd8de60841be432785aa6bde9d170a08" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:12.505015 12569 auto_rebalancer.cc:288] Could not retrieve cluster info: Illegal state: not all tservers available for tablet placement
I20260709 12:27:12.578753 12969 raft_consensus.cc:1275] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:12.580360 13029 consensus_queue.cc:1048] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:12.593199 12694 raft_consensus.cc:1275] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:12.595275 13061 consensus_queue.cc:1048] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:12.654807 12694 raft_consensus.cc:1275] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 4 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:27:12.656242 13061 consensus_queue.cc:1048] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd8de60841be432785aa6bde9d170a08" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:27:12.666814 12969 raft_consensus.cc:1275] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 4 FOLLOWER]: Refusing update from remote peer 1b03c1a2d74a436ea0be8b84f9b3f75e: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260709 12:27:12.668999 13029 consensus_queue.cc:1048] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:27:12.787966 12969 raft_consensus.cc:1275] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 5 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:27:12.789784 13087 consensus_queue.cc:1048] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76884a9b95e64c5e95e950afda820ece" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260709 12:27:12.799291 12770 raft_consensus.cc:1275] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 5 FOLLOWER]: Refusing update from remote peer dd8de60841be432785aa6bde9d170a08: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260709 12:27:12.800855 13087 consensus_queue.cc:1048] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b03c1a2d74a436ea0be8b84f9b3f75e" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 44647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260709 12:27:12.992399 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:13.026791 30503 tablet_server.cc:179] TabletServer@127.29.201.193:39205 shutting down...
W20260709 12:27:13.042268 12735 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 87 similar messages]
W20260709 12:27:13.046372 12735 consensus_peers.cc:597] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:13.059794 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:13.060838 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:13.061621 30503 raft_consensus.cc:2243] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.062201 30503 raft_consensus.cc:2272] T 66d601ec378b47b9834fefc2ccf51ce3 P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.066720 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:13.067683 30503 raft_consensus.cc:2243] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.068536 30503 raft_consensus.cc:2272] T a47a8c7a8101401297cf692f5358a7d2 P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.072546 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:13.073484 30503 raft_consensus.cc:2243] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.074147 30503 raft_consensus.cc:2272] T 6f0ff2156af842de82d34a04aab9cdf4 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:13.074236 12735 consensus_peers.cc:597] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:13.077768 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:13.078418 30503 raft_consensus.cc:2243] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.078972 30503 raft_consensus.cc:2272] T cc7e1418d7c145b88a45a159d25e21cc P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.081648 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:13.082172 30503 raft_consensus.cc:2243] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.082717 30503 raft_consensus.cc:2272] T b909763b4c1045178a6fc9fd1a58250f P 76884a9b95e64c5e95e950afda820ece [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.087268 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:13.088131 30503 raft_consensus.cc:2243] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 12:27:13.088240 12735 consensus_peers.cc:597] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:13.089011 30503 raft_consensus.cc:2272] T fbd279d00e7142afab793d8849e54bf0 P 76884a9b95e64c5e95e950afda820ece [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:13.092244 12735 consensus_peers.cc:597] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:13.092666 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:13.093441 30503 raft_consensus.cc:2243] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.093909 30503 raft_consensus.cc:2272] T 15118b927e72411bb330ef7b8d8d3a92 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.096500 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:13.097165 30503 raft_consensus.cc:2243] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.097664 30503 raft_consensus.cc:2272] T ade88c76c8f34edfac61ded6c3d472f6 P 76884a9b95e64c5e95e950afda820ece [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.101030 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:13.101636 30503 raft_consensus.cc:2243] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.102077 30503 raft_consensus.cc:2272] T 128ccf33a3764acc9fa913611d1a32f0 P 76884a9b95e64c5e95e950afda820ece [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.104408 30503 tablet_replica.cc:333] stopping tablet replica
I20260709 12:27:13.104943 30503 raft_consensus.cc:2243] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.105423 30503 raft_consensus.cc:2272] T 6041407ddd7647db97a3e044abd5574e P 76884a9b95e64c5e95e950afda820ece [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.140031 30503 tablet_server.cc:196] TabletServer@127.29.201.193:39205 shutdown complete.
I20260709 12:27:13.172719 30503 tablet_server.cc:179] TabletServer@127.29.201.194:0 shutting down...
W20260709 12:27:13.187189 12661 consensus_peers.cc:597] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:13.212713 12661 consensus_peers.cc:597] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:13.215873 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:13.217027 30503 tablet_replica.cc:333] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08: stopping tablet replica
W20260709 12:27:13.217073 12735 consensus_peers.cc:597] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:13.218154 30503 raft_consensus.cc:2243] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.218885 30503 raft_consensus.cc:2272] T 66d601ec378b47b9834fefc2ccf51ce3 P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.221501 30503 tablet_replica.cc:333] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08: stopping tablet replica
I20260709 12:27:13.222203 30503 raft_consensus.cc:2243] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.222795 30503 raft_consensus.cc:2272] T b909763b4c1045178a6fc9fd1a58250f P dd8de60841be432785aa6bde9d170a08 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.225641 30503 tablet_replica.cc:333] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08: stopping tablet replica
I20260709 12:27:13.226379 30503 raft_consensus.cc:2243] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.227008 30503 raft_consensus.cc:2272] T a47a8c7a8101401297cf692f5358a7d2 P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.229694 30503 tablet_replica.cc:333] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08: stopping tablet replica
I20260709 12:27:13.230367 30503 raft_consensus.cc:2243] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.230937 30503 raft_consensus.cc:2272] T 15118b927e72411bb330ef7b8d8d3a92 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.233732 30503 tablet_replica.cc:333] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08: stopping tablet replica
I20260709 12:27:13.234402 30503 raft_consensus.cc:2243] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:13.235627 30503 raft_consensus.cc:2272] T fbd279d00e7142afab793d8849e54bf0 P dd8de60841be432785aa6bde9d170a08 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.238565 30503 tablet_replica.cc:333] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08: stopping tablet replica
I20260709 12:27:13.239199 30503 raft_consensus.cc:2243] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.239652 30503 raft_consensus.cc:2272] T ade88c76c8f34edfac61ded6c3d472f6 P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.242892 30503 tablet_replica.cc:333] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08: stopping tablet replica
I20260709 12:27:13.243494 30503 raft_consensus.cc:2243] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.243999 30503 raft_consensus.cc:2272] T 128ccf33a3764acc9fa913611d1a32f0 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.246605 30503 tablet_replica.cc:333] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08: stopping tablet replica
I20260709 12:27:13.247366 30503 raft_consensus.cc:2243] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 5 LEADER]: Raft consensus shutting down.
I20260709 12:27:13.248353 30503 raft_consensus.cc:2272] T 6041407ddd7647db97a3e044abd5574e P dd8de60841be432785aa6bde9d170a08 [term 5 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:13.251029 12661 consensus_peers.cc:597] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 -> Peer 76884a9b95e64c5e95e950afda820ece (127.29.201.193:39205): Couldn't send request to peer 76884a9b95e64c5e95e950afda820ece. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39205: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:13.252077 30503 tablet_replica.cc:333] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08: stopping tablet replica
I20260709 12:27:13.252653 30503 raft_consensus.cc:2243] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:13.253480 30503 raft_consensus.cc:2272] T 6f0ff2156af842de82d34a04aab9cdf4 P dd8de60841be432785aa6bde9d170a08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.256028 30503 tablet_replica.cc:333] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08: stopping tablet replica
I20260709 12:27:13.256520 30503 raft_consensus.cc:2243] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:13.257254 30503 raft_consensus.cc:2272] T cc7e1418d7c145b88a45a159d25e21cc P dd8de60841be432785aa6bde9d170a08 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:13.273301 12736 consensus_peers.cc:597] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355): Couldn't send request to peer dd8de60841be432785aa6bde9d170a08. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:13.292377 30503 tablet_server.cc:196] TabletServer@127.29.201.194:0 shutdown complete.
I20260709 12:27:13.325588 30503 tablet_server.cc:179] TabletServer@127.29.201.195:0 shutting down...
I20260709 12:27:13.353799 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:13.354599 30503 tablet_replica.cc:333] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e: stopping tablet replica
I20260709 12:27:13.355392 30503 raft_consensus.cc:2243] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:13.356498 30503 raft_consensus.cc:2272] T 66d601ec378b47b9834fefc2ccf51ce3 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.359536 30503 tablet_replica.cc:333] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e: stopping tablet replica
I20260709 12:27:13.360405 30503 raft_consensus.cc:2243] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.361074 30503 raft_consensus.cc:2272] T 6f0ff2156af842de82d34a04aab9cdf4 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:13.364563 12736 consensus_peers.cc:597] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e -> Peer dd8de60841be432785aa6bde9d170a08 (127.29.201.194:33355): Couldn't send request to peer dd8de60841be432785aa6bde9d170a08. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:33355: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:13.364892 30503 tablet_replica.cc:333] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e: stopping tablet replica
I20260709 12:27:13.365790 30503 raft_consensus.cc:2243] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.366318 30503 raft_consensus.cc:2272] T cc7e1418d7c145b88a45a159d25e21cc P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.368708 30503 tablet_replica.cc:333] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e: stopping tablet replica
I20260709 12:27:13.369383 30503 raft_consensus.cc:2243] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:13.370412 30503 raft_consensus.cc:2272] T a47a8c7a8101401297cf692f5358a7d2 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.372872 30503 tablet_replica.cc:333] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: stopping tablet replica
I20260709 12:27:13.373534 30503 raft_consensus.cc:2243] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.374059 30503 raft_consensus.cc:2272] T fbd279d00e7142afab793d8849e54bf0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.376570 30503 tablet_replica.cc:333] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e: stopping tablet replica
I20260709 12:27:13.377166 30503 raft_consensus.cc:2243] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:13.378159 30503 raft_consensus.cc:2272] T 15118b927e72411bb330ef7b8d8d3a92 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.380752 30503 tablet_replica.cc:333] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e: stopping tablet replica
I20260709 12:27:13.381345 30503 raft_consensus.cc:2243] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:13.382195 30503 raft_consensus.cc:2272] T ade88c76c8f34edfac61ded6c3d472f6 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.385162 30503 tablet_replica.cc:333] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e: stopping tablet replica
I20260709 12:27:13.385865 30503 raft_consensus.cc:2243] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 5 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:13.386364 30503 raft_consensus.cc:2272] T 6041407ddd7647db97a3e044abd5574e P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 5 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.389304 30503 tablet_replica.cc:333] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e: stopping tablet replica
I20260709 12:27:13.389919 30503 raft_consensus.cc:2243] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:13.390990 30503 raft_consensus.cc:2272] T 128ccf33a3764acc9fa913611d1a32f0 P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.393600 30503 tablet_replica.cc:333] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e: stopping tablet replica
I20260709 12:27:13.394098 30503 raft_consensus.cc:2243] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 4 LEADER]: Raft consensus shutting down.
I20260709 12:27:13.395049 30503 raft_consensus.cc:2272] T b909763b4c1045178a6fc9fd1a58250f P 1b03c1a2d74a436ea0be8b84f9b3f75e [term 4 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.430395 30503 tablet_server.cc:196] TabletServer@127.29.201.195:0 shutdown complete.
I20260709 12:27:13.457114 30503 master.cc:562] Master@127.29.201.254:44111 shutting down...
I20260709 12:27:13.482398 30503 raft_consensus.cc:2243] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:13.483076 30503 raft_consensus.cc:2272] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:13.483397 30503 tablet_replica.cc:333] T 00000000000000000000000000000000 P b425b396d0724794b79814fc3056dbd3: stopping tablet replica
I20260709 12:27:13.504504 30503 master.cc:584] Master@127.29.201.254:44111 shutdown complete.
[       OK ] LeaderRebalancerTest.TestMaintenanceMode (13213 ms)
[ RUN      ] LeaderRebalancerTest.MultiTableLeaderBalance
I20260709 12:27:13.554209 30503 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.201.254:35873
I20260709 12:27:13.555547 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:13.562179 13104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:13.562290 13103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:13.562779 13106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:13.563686 30503 server_base.cc:1061] running on GCE node
I20260709 12:27:13.564690 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:13.564888 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:13.565025 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600033565012 us; error 0 us; skew 500 ppm
I20260709 12:27:13.565546 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:13.568482 30503 webserver.cc:533] Webserver started at http://127.29.201.254:43107/ using document root <none> and password file <none>
I20260709 12:27:13.569028 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:13.569198 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:13.569463 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:13.570780 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/master-0-root/instance:
uuid: "d3aaac24a32e4658bbaf5f24d12d03aa"
format_stamp: "Formatted at 2026-07-09 12:27:13 on dist-test-slave-qgf5"
I20260709 12:27:13.575894 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.001s
I20260709 12:27:13.579969 13111 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:13.581015 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.001s
I20260709 12:27:13.581293 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/master-0-root
uuid: "d3aaac24a32e4658bbaf5f24d12d03aa"
format_stamp: "Formatted at 2026-07-09 12:27:13 on dist-test-slave-qgf5"
I20260709 12:27:13.581635 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:13.600382 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:13.601722 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:13.643159 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.254:35873
I20260709 12:27:13.643355 13162 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.254:35873 every 8 connection(s)
I20260709 12:27:13.647428 13163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:13.659276 13163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa: Bootstrap starting.
I20260709 12:27:13.664263 13163 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:13.668920 13163 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa: No bootstrap required, opened a new log
I20260709 12:27:13.671219 13163 raft_consensus.cc:359] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3aaac24a32e4658bbaf5f24d12d03aa" member_type: VOTER }
I20260709 12:27:13.671656 13163 raft_consensus.cc:385] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:13.671840 13163 raft_consensus.cc:740] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3aaac24a32e4658bbaf5f24d12d03aa, State: Initialized, Role: FOLLOWER
I20260709 12:27:13.672363 13163 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3aaac24a32e4658bbaf5f24d12d03aa" member_type: VOTER }
I20260709 12:27:13.672798 13163 raft_consensus.cc:399] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:27:13.672994 13163 raft_consensus.cc:493] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:27:13.673205 13163 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:13.678476 13163 raft_consensus.cc:515] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3aaac24a32e4658bbaf5f24d12d03aa" member_type: VOTER }
I20260709 12:27:13.679054 13163 leader_election.cc:304] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3aaac24a32e4658bbaf5f24d12d03aa; no voters: 
I20260709 12:27:13.680331 13163 leader_election.cc:290] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:27:13.680732 13166 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:13.682191 13166 raft_consensus.cc:697] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [term 1 LEADER]: Becoming Leader. State: Replica: d3aaac24a32e4658bbaf5f24d12d03aa, State: Running, Role: LEADER
I20260709 12:27:13.682996 13166 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3aaac24a32e4658bbaf5f24d12d03aa" member_type: VOTER }
I20260709 12:27:13.683432 13163 sys_catalog.cc:565] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:27:13.688477 13168 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3aaac24a32e4658bbaf5f24d12d03aa. Latest consensus state: current_term: 1 leader_uuid: "d3aaac24a32e4658bbaf5f24d12d03aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3aaac24a32e4658bbaf5f24d12d03aa" member_type: VOTER } }
I20260709 12:27:13.689435 13168 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:13.689270 13167 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3aaac24a32e4658bbaf5f24d12d03aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3aaac24a32e4658bbaf5f24d12d03aa" member_type: VOTER } }
I20260709 12:27:13.690505 13167 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:13.694583 13174 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:27:13.702797 13174 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:27:13.703949 30503 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:27:13.712488 13174 catalog_manager.cc:1383] Generated new cluster ID: 15aaee6f2ebe4f73b4013e4332002735
I20260709 12:27:13.712862 13174 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:27:13.734215 13174 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:27:13.735785 13174 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:27:13.752832 13174 catalog_manager.cc:6070] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa: Generated new TSK 0
I20260709 12:27:13.753684 13174 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:27:13.771663 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:13.778450 13184 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:13.778769 13185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:13.780689 13187 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:13.781282 30503 server_base.cc:1061] running on GCE node
I20260709 12:27:13.782335 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:13.782559 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:13.782820 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600033782798 us; error 0 us; skew 500 ppm
I20260709 12:27:13.783458 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:13.786299 30503 webserver.cc:533] Webserver started at http://127.29.201.193:44715/ using document root <none> and password file <none>
I20260709 12:27:13.786871 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:13.787039 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:13.787371 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:13.788549 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-0-root/instance:
uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22"
format_stamp: "Formatted at 2026-07-09 12:27:13 on dist-test-slave-qgf5"
I20260709 12:27:13.793485 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 12:27:13.797322 13192 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:13.798172 30503 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.000s	sys 0.002s
I20260709 12:27:13.798529 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-0-root
uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22"
format_stamp: "Formatted at 2026-07-09 12:27:13 on dist-test-slave-qgf5"
I20260709 12:27:13.798882 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:13.829764 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:13.831257 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:13.832880 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:13.835793 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:13.836088 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:13.836366 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:13.836576 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:13.880493 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.193:42945
I20260709 12:27:13.880609 13254 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.193:42945 every 8 connection(s)
I20260709 12:27:13.886281 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:13.894076 13259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:13.894459 13260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:13.897989 30503 server_base.cc:1061] running on GCE node
W20260709 12:27:13.898694 13263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:13.899986 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:13.900298 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:13.900503 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600033900483 us; error 0 us; skew 500 ppm
I20260709 12:27:13.901273 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:13.901594 13255 heartbeater.cc:344] Connected to a master server at 127.29.201.254:35873
I20260709 12:27:13.902138 13255 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:13.903234 13255 heartbeater.cc:507] Master 127.29.201.254:35873 requested a full tablet report, sending...
I20260709 12:27:13.904461 30503 webserver.cc:533] Webserver started at http://127.29.201.194:39293/ using document root <none> and password file <none>
I20260709 12:27:13.905082 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:13.905347 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:13.905719 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:13.906049 13128 ts_manager.cc:194] Registered new tserver with Master: 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945)
I20260709 12:27:13.907531 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-1-root/instance:
uuid: "8efac806c91b40f38e07639bd5ba445d"
format_stamp: "Formatted at 2026-07-09 12:27:13 on dist-test-slave-qgf5"
I20260709 12:27:13.908828 13128 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58570
I20260709 12:27:13.913894 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.006s	user 0.006s	sys 0.000s
I20260709 12:27:13.917958 13267 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:13.918843 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:27:13.919209 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-1-root
uuid: "8efac806c91b40f38e07639bd5ba445d"
format_stamp: "Formatted at 2026-07-09 12:27:13 on dist-test-slave-qgf5"
I20260709 12:27:13.919492 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:13.933223 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:13.934502 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:13.936146 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:13.938519 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:13.938827 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20260709 12:27:13.939082 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:13.939262 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:13.980856 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.194:32805
I20260709 12:27:13.980993 13329 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.194:32805 every 8 connection(s)
I20260709 12:27:13.986770 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:13.994725 13333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:13.997862 13334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:13.998502 13330 heartbeater.cc:344] Connected to a master server at 127.29.201.254:35873
I20260709 12:27:13.998970 13330 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:13.999675 13330 heartbeater.cc:507] Master 127.29.201.254:35873 requested a full tablet report, sending...
I20260709 12:27:14.002211 13128 ts_manager.cc:194] Registered new tserver with Master: 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805)
I20260709 12:27:14.002921 30503 server_base.cc:1061] running on GCE node
W20260709 12:27:14.002948 13336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:14.004151 13128 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58582
I20260709 12:27:14.004233 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:14.004601 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:14.004824 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600034004803 us; error 0 us; skew 500 ppm
I20260709 12:27:14.005771 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:14.008638 30503 webserver.cc:533] Webserver started at http://127.29.201.195:39799/ using document root <none> and password file <none>
I20260709 12:27:14.009160 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:14.009356 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:14.009634 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:14.010871 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-2-root/instance:
uuid: "3960c41fdd944168b675855e44737cac"
format_stamp: "Formatted at 2026-07-09 12:27:14 on dist-test-slave-qgf5"
I20260709 12:27:14.015455 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.004s	sys 0.000s
I20260709 12:27:14.019116 13341 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:14.019937 30503 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.002s
I20260709 12:27:14.020255 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-2-root
uuid: "3960c41fdd944168b675855e44737cac"
format_stamp: "Formatted at 2026-07-09 12:27:14 on dist-test-slave-qgf5"
I20260709 12:27:14.020522 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.MultiTableLeaderBalance.1783599553313367-30503-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:14.037503 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:14.038913 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:14.040441 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:14.042856 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:14.043098 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:14.043362 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:14.043561 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:14.092351 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.195:36755
I20260709 12:27:14.092458 13403 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.195:36755 every 8 connection(s)
I20260709 12:27:14.108244 13404 heartbeater.cc:344] Connected to a master server at 127.29.201.254:35873
I20260709 12:27:14.108748 13404 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:14.109555 13404 heartbeater.cc:507] Master 127.29.201.254:35873 requested a full tablet report, sending...
I20260709 12:27:14.111756 13128 ts_manager.cc:194] Registered new tserver with Master: 3960c41fdd944168b675855e44737cac (127.29.201.195:36755)
I20260709 12:27:14.112643 30503 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015986123s
I20260709 12:27:14.113483 13128 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58588
I20260709 12:27:14.141341 13128 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58590:
name: "multi_table_leader_balance_table1"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:27:14.143656 13128 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:27:14.193316 13369 tablet_service.cc:1511] Processing CreateTablet for tablet 9dfe53b95c194ea18f1cc7126f1f7719 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=86411fe1d1b746d7a10ff219f32ad62e]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:27:14.194170 13368 tablet_service.cc:1511] Processing CreateTablet for tablet 19da758fe12148fcbce4c890bdf31287 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=86411fe1d1b746d7a10ff219f32ad62e]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:27:14.194989 13369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dfe53b95c194ea18f1cc7126f1f7719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:14.196534 13368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19da758fe12148fcbce4c890bdf31287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:14.199760 13220 tablet_service.cc:1511] Processing CreateTablet for tablet 9dfe53b95c194ea18f1cc7126f1f7719 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=86411fe1d1b746d7a10ff219f32ad62e]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:27:14.199999 13219 tablet_service.cc:1511] Processing CreateTablet for tablet 19da758fe12148fcbce4c890bdf31287 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=86411fe1d1b746d7a10ff219f32ad62e]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:27:14.201447 13220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dfe53b95c194ea18f1cc7126f1f7719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:14.203047 13295 tablet_service.cc:1511] Processing CreateTablet for tablet 9dfe53b95c194ea18f1cc7126f1f7719 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=86411fe1d1b746d7a10ff219f32ad62e]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:27:14.203033 13219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19da758fe12148fcbce4c890bdf31287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:14.204828 13295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dfe53b95c194ea18f1cc7126f1f7719. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:14.211917 13294 tablet_service.cc:1511] Processing CreateTablet for tablet 19da758fe12148fcbce4c890bdf31287 (DEFAULT_TABLE table=multi_table_leader_balance_table1 [id=86411fe1d1b746d7a10ff219f32ad62e]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:27:14.213524 13294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19da758fe12148fcbce4c890bdf31287. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:14.224344 13419 tablet_bootstrap.cc:492] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac: Bootstrap starting.
I20260709 12:27:14.232558 13419 tablet_bootstrap.cc:654] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:14.244433 13419 tablet_bootstrap.cc:492] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac: No bootstrap required, opened a new log
I20260709 12:27:14.245010 13419 ts_tablet_manager.cc:1403] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac: Time spent bootstrapping tablet: real 0.021s	user 0.017s	sys 0.000s
I20260709 12:27:14.248154 13419 raft_consensus.cc:359] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.248876 13419 raft_consensus.cc:385] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:14.249176 13419 raft_consensus.cc:740] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3960c41fdd944168b675855e44737cac, State: Initialized, Role: FOLLOWER
I20260709 12:27:14.250176 13419 consensus_queue.cc:260] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.252046 13420 tablet_bootstrap.cc:492] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Bootstrap starting.
I20260709 12:27:14.259776 13419 ts_tablet_manager.cc:1434] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac: Time spent starting tablet: real 0.014s	user 0.011s	sys 0.005s
I20260709 12:27:14.260725 13404 heartbeater.cc:499] Master 127.29.201.254:35873 was elected leader, sending a full tablet report...
I20260709 12:27:14.262261 13419 tablet_bootstrap.cc:492] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac: Bootstrap starting.
I20260709 12:27:14.259944 13420 tablet_bootstrap.cc:654] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:14.266281 13423 tablet_bootstrap.cc:492] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d: Bootstrap starting.
I20260709 12:27:14.273001 13419 tablet_bootstrap.cc:654] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:14.274822 13423 tablet_bootstrap.cc:654] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:14.284252 13423 tablet_bootstrap.cc:492] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d: No bootstrap required, opened a new log
I20260709 12:27:14.284420 13420 tablet_bootstrap.cc:492] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: No bootstrap required, opened a new log
I20260709 12:27:14.284829 13423 ts_tablet_manager.cc:1403] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d: Time spent bootstrapping tablet: real 0.020s	user 0.000s	sys 0.012s
I20260709 12:27:14.285144 13420 ts_tablet_manager.cc:1403] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Time spent bootstrapping tablet: real 0.034s	user 0.019s	sys 0.005s
I20260709 12:27:14.285187 13419 tablet_bootstrap.cc:492] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac: No bootstrap required, opened a new log
I20260709 12:27:14.285959 13419 ts_tablet_manager.cc:1403] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac: Time spent bootstrapping tablet: real 0.024s	user 0.008s	sys 0.012s
I20260709 12:27:14.288465 13420 raft_consensus.cc:359] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:14.289206 13420 raft_consensus.cc:385] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:14.289052 13423 raft_consensus.cc:359] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:14.289608 13420 raft_consensus.cc:740] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7dd3fe520c4a949bd7b8b68fc9fe22, State: Initialized, Role: FOLLOWER
I20260709 12:27:14.289347 13419 raft_consensus.cc:359] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:14.289950 13423 raft_consensus.cc:385] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:14.290198 13419 raft_consensus.cc:385] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:14.290364 13423 raft_consensus.cc:740] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efac806c91b40f38e07639bd5ba445d, State: Initialized, Role: FOLLOWER
I20260709 12:27:14.290576 13419 raft_consensus.cc:740] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3960c41fdd944168b675855e44737cac, State: Initialized, Role: FOLLOWER
I20260709 12:27:14.290728 13420 consensus_queue.cc:260] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:14.291440 13419 consensus_queue.cc:260] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:14.291343 13423 consensus_queue.cc:260] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:14.294230 13420 ts_tablet_manager.cc:1434] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:27:14.294427 13419 ts_tablet_manager.cc:1434] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.004s
I20260709 12:27:14.295434 13255 heartbeater.cc:499] Master 127.29.201.254:35873 was elected leader, sending a full tablet report...
I20260709 12:27:14.295614 13420 tablet_bootstrap.cc:492] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Bootstrap starting.
I20260709 12:27:14.297508 13330 heartbeater.cc:499] Master 127.29.201.254:35873 was elected leader, sending a full tablet report...
I20260709 12:27:14.303086 13423 ts_tablet_manager.cc:1434] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d: Time spent starting tablet: real 0.018s	user 0.008s	sys 0.003s
I20260709 12:27:14.304356 13423 tablet_bootstrap.cc:492] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d: Bootstrap starting.
I20260709 12:27:14.304785 13428 raft_consensus.cc:493] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:14.305450 13428 raft_consensus.cc:515] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:14.314865 13428 leader_election.cc:290] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3960c41fdd944168b675855e44737cac (127.29.201.195:36755), 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945)
I20260709 12:27:14.315117 13423 tablet_bootstrap.cc:654] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:14.304808 13420 tablet_bootstrap.cc:654] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:14.323725 13379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dfe53b95c194ea18f1cc7126f1f7719" candidate_uuid: "8efac806c91b40f38e07639bd5ba445d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960c41fdd944168b675855e44737cac" is_pre_election: true
I20260709 12:27:14.325011 13379 raft_consensus.cc:2468] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8efac806c91b40f38e07639bd5ba445d in term 0.
I20260709 12:27:14.328569 13268 leader_election.cc:304] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3960c41fdd944168b675855e44737cac, 8efac806c91b40f38e07639bd5ba445d; no voters: 
I20260709 12:27:14.329586 13428 raft_consensus.cc:2804] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:14.330036 13428 raft_consensus.cc:493] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:14.330399 13428 raft_consensus.cc:3060] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:14.332532 13423 tablet_bootstrap.cc:492] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d: No bootstrap required, opened a new log
I20260709 12:27:14.333282 13423 ts_tablet_manager.cc:1403] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d: Time spent bootstrapping tablet: real 0.029s	user 0.001s	sys 0.018s
I20260709 12:27:14.335882 13420 tablet_bootstrap.cc:492] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: No bootstrap required, opened a new log
I20260709 12:27:14.336555 13420 ts_tablet_manager.cc:1403] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Time spent bootstrapping tablet: real 0.041s	user 0.011s	sys 0.009s
I20260709 12:27:14.337430 13230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dfe53b95c194ea18f1cc7126f1f7719" candidate_uuid: "8efac806c91b40f38e07639bd5ba445d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" is_pre_election: true
I20260709 12:27:14.338325 13230 raft_consensus.cc:2468] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8efac806c91b40f38e07639bd5ba445d in term 0.
I20260709 12:27:14.339035 13428 raft_consensus.cc:515] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:14.339695 13423 raft_consensus.cc:359] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.340721 13423 raft_consensus.cc:385] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:14.340250 13420 raft_consensus.cc:359] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.341197 13423 raft_consensus.cc:740] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efac806c91b40f38e07639bd5ba445d, State: Initialized, Role: FOLLOWER
I20260709 12:27:14.341293 13420 raft_consensus.cc:385] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:14.341475 13428 leader_election.cc:290] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [CANDIDATE]: Term 1 election: Requested vote from peers 3960c41fdd944168b675855e44737cac (127.29.201.195:36755), 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945)
I20260709 12:27:14.341810 13420 raft_consensus.cc:740] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7dd3fe520c4a949bd7b8b68fc9fe22, State: Initialized, Role: FOLLOWER
I20260709 12:27:14.342255 13379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dfe53b95c194ea18f1cc7126f1f7719" candidate_uuid: "8efac806c91b40f38e07639bd5ba445d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960c41fdd944168b675855e44737cac"
I20260709 12:27:14.342999 13379 raft_consensus.cc:3060] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:14.343181 13230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dfe53b95c194ea18f1cc7126f1f7719" candidate_uuid: "8efac806c91b40f38e07639bd5ba445d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22"
I20260709 12:27:14.342950 13423 consensus_queue.cc:260] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.343161 13420 consensus_queue.cc:260] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.344316 13230 raft_consensus.cc:3060] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:14.346009 13423 ts_tablet_manager.cc:1434] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d: Time spent starting tablet: real 0.012s	user 0.007s	sys 0.000s
I20260709 12:27:14.351181 13420 ts_tablet_manager.cc:1434] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Time spent starting tablet: real 0.014s	user 0.010s	sys 0.000s
I20260709 12:27:14.353814 13230 raft_consensus.cc:2468] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8efac806c91b40f38e07639bd5ba445d in term 1.
I20260709 12:27:14.355403 13268 leader_election.cc:304] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7dd3fe520c4a949bd7b8b68fc9fe22, 8efac806c91b40f38e07639bd5ba445d; no voters: 
I20260709 12:27:14.356433 13428 raft_consensus.cc:2804] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:14.356788 13379 raft_consensus.cc:2468] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8efac806c91b40f38e07639bd5ba445d in term 1.
I20260709 12:27:14.359599 13428 raft_consensus.cc:697] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 1 LEADER]: Becoming Leader. State: Replica: 8efac806c91b40f38e07639bd5ba445d, State: Running, Role: LEADER
I20260709 12:27:14.360755 13428 consensus_queue.cc:237] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:14.369545 13127 catalog_manager.cc:5697] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d reported cstate change: term changed from 0 to 1, leader changed from <none> to 8efac806c91b40f38e07639bd5ba445d (127.29.201.194). New cstate: current_term: 1 leader_uuid: "8efac806c91b40f38e07639bd5ba445d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:14.434772 13427 raft_consensus.cc:493] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:14.435388 13427 raft_consensus.cc:515] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.437657 13427 leader_election.cc:290] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805), 3960c41fdd944168b675855e44737cac (127.29.201.195:36755)
I20260709 12:27:14.449410 13305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19da758fe12148fcbce4c890bdf31287" candidate_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efac806c91b40f38e07639bd5ba445d" is_pre_election: true
I20260709 12:27:14.450183 13305 raft_consensus.cc:2468] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a7dd3fe520c4a949bd7b8b68fc9fe22 in term 0.
I20260709 12:27:14.450608 13379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19da758fe12148fcbce4c890bdf31287" candidate_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960c41fdd944168b675855e44737cac" is_pre_election: true
I20260709 12:27:14.451352 13379 raft_consensus.cc:2468] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a7dd3fe520c4a949bd7b8b68fc9fe22 in term 0.
I20260709 12:27:14.451520 13193 leader_election.cc:304] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7dd3fe520c4a949bd7b8b68fc9fe22, 8efac806c91b40f38e07639bd5ba445d; no voters: 
I20260709 12:27:14.452440 13427 raft_consensus.cc:2804] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:14.452761 13427 raft_consensus.cc:493] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:14.453047 13427 raft_consensus.cc:3060] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:14.458778 13427 raft_consensus.cc:515] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.460511 13427 leader_election.cc:290] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [CANDIDATE]: Term 1 election: Requested vote from peers 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805), 3960c41fdd944168b675855e44737cac (127.29.201.195:36755)
I20260709 12:27:14.461624 13305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19da758fe12148fcbce4c890bdf31287" candidate_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efac806c91b40f38e07639bd5ba445d"
I20260709 12:27:14.461892 13379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19da758fe12148fcbce4c890bdf31287" candidate_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960c41fdd944168b675855e44737cac"
I20260709 12:27:14.462476 13379 raft_consensus.cc:3060] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:14.462476 13305 raft_consensus.cc:3060] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:14.471237 13379 raft_consensus.cc:2468] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7dd3fe520c4a949bd7b8b68fc9fe22 in term 1.
I20260709 12:27:14.471237 13305 raft_consensus.cc:2468] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7dd3fe520c4a949bd7b8b68fc9fe22 in term 1.
I20260709 12:27:14.472689 13193 leader_election.cc:304] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7dd3fe520c4a949bd7b8b68fc9fe22, 3960c41fdd944168b675855e44737cac; no voters: 
I20260709 12:27:14.473517 13427 raft_consensus.cc:2804] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:14.474874 13427 raft_consensus.cc:697] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 LEADER]: Becoming Leader. State: Replica: 0a7dd3fe520c4a949bd7b8b68fc9fe22, State: Running, Role: LEADER
I20260709 12:27:14.475672 13427 consensus_queue.cc:237] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.484088 13127 catalog_manager.cc:5697] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:14.556954 13127 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58600:
name: "multi_table_leader_balance_table2"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\377\377\377?""\004\001\000\377\377\377?"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:27:14.559325 13127 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table multi_table_leader_balance_table2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:27:14.585855 13219 tablet_service.cc:1511] Processing CreateTablet for tablet a89ad6d2be0e4850945958ab820c7545 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=f4235fd37b564f5381cdb682b2f753bf]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:27:14.587311 13219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89ad6d2be0e4850945958ab820c7545. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:14.587409 13295 tablet_service.cc:1511] Processing CreateTablet for tablet a89ad6d2be0e4850945958ab820c7545 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=f4235fd37b564f5381cdb682b2f753bf]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:27:14.588843 13295 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89ad6d2be0e4850945958ab820c7545. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:14.588969 13369 tablet_service.cc:1511] Processing CreateTablet for tablet a89ad6d2be0e4850945958ab820c7545 (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=f4235fd37b564f5381cdb682b2f753bf]), partition=RANGE (key) PARTITION VALUES < 1073741823
I20260709 12:27:14.590318 13369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a89ad6d2be0e4850945958ab820c7545. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:14.596087 13220 tablet_service.cc:1511] Processing CreateTablet for tablet a9d78091f8e4409382a42ce2392046ed (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=f4235fd37b564f5381cdb682b2f753bf]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:27:14.597743 13220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9d78091f8e4409382a42ce2392046ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:14.605167 13420 tablet_bootstrap.cc:492] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Bootstrap starting.
I20260709 12:27:14.607949 13423 tablet_bootstrap.cc:492] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d: Bootstrap starting.
I20260709 12:27:14.609304 13419 tablet_bootstrap.cc:492] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac: Bootstrap starting.
I20260709 12:27:14.612969 13420 tablet_bootstrap.cc:654] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:14.613266 13294 tablet_service.cc:1511] Processing CreateTablet for tablet a9d78091f8e4409382a42ce2392046ed (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=f4235fd37b564f5381cdb682b2f753bf]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:27:14.614749 13294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9d78091f8e4409382a42ce2392046ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:14.616966 13419 tablet_bootstrap.cc:654] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:14.620635 13423 tablet_bootstrap.cc:654] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:14.621624 13369 tablet_service.cc:1511] Processing CreateTablet for tablet a9d78091f8e4409382a42ce2392046ed (DEFAULT_TABLE table=multi_table_leader_balance_table2 [id=f4235fd37b564f5381cdb682b2f753bf]), partition=RANGE (key) PARTITION 1073741823 <= VALUES
I20260709 12:27:14.623389 13369 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9d78091f8e4409382a42ce2392046ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:14.627921 13419 tablet_bootstrap.cc:492] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac: No bootstrap required, opened a new log
I20260709 12:27:14.628523 13419 ts_tablet_manager.cc:1403] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac: Time spent bootstrapping tablet: real 0.019s	user 0.009s	sys 0.007s
I20260709 12:27:14.631868 13419 raft_consensus.cc:359] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.632612 13419 raft_consensus.cc:385] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:14.632884 13419 raft_consensus.cc:740] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3960c41fdd944168b675855e44737cac, State: Initialized, Role: FOLLOWER
I20260709 12:27:14.634464 13419 consensus_queue.cc:260] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.635663 13420 tablet_bootstrap.cc:492] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: No bootstrap required, opened a new log
I20260709 12:27:14.636264 13420 ts_tablet_manager.cc:1403] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Time spent bootstrapping tablet: real 0.031s	user 0.010s	sys 0.003s
I20260709 12:27:14.639608 13420 raft_consensus.cc:359] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.640416 13420 raft_consensus.cc:385] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:14.640904 13420 raft_consensus.cc:740] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7dd3fe520c4a949bd7b8b68fc9fe22, State: Initialized, Role: FOLLOWER
I20260709 12:27:14.663165 13419 ts_tablet_manager.cc:1434] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac: Time spent starting tablet: real 0.034s	user 0.010s	sys 0.000s
I20260709 12:27:14.664508 13420 consensus_queue.cc:260] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.667999 13420 ts_tablet_manager.cc:1434] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Time spent starting tablet: real 0.031s	user 0.010s	sys 0.000s
I20260709 12:27:14.669515 13420 tablet_bootstrap.cc:492] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Bootstrap starting.
I20260709 12:27:14.672554 13423 tablet_bootstrap.cc:492] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d: No bootstrap required, opened a new log
I20260709 12:27:14.673239 13423 ts_tablet_manager.cc:1403] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d: Time spent bootstrapping tablet: real 0.066s	user 0.009s	sys 0.015s
I20260709 12:27:14.677312 13423 raft_consensus.cc:359] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.678335 13423 raft_consensus.cc:385] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:14.678727 13423 raft_consensus.cc:740] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efac806c91b40f38e07639bd5ba445d, State: Initialized, Role: FOLLOWER
I20260709 12:27:14.679636 13423 consensus_queue.cc:260] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.681028 13419 tablet_bootstrap.cc:492] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac: Bootstrap starting.
I20260709 12:27:14.685300 13420 tablet_bootstrap.cc:654] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:14.686784 13423 ts_tablet_manager.cc:1434] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d: Time spent starting tablet: real 0.013s	user 0.008s	sys 0.003s
I20260709 12:27:14.688890 13423 tablet_bootstrap.cc:492] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d: Bootstrap starting.
I20260709 12:27:14.692301 13419 tablet_bootstrap.cc:654] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:14.692914 13420 tablet_bootstrap.cc:492] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22: No bootstrap required, opened a new log
I20260709 12:27:14.693584 13420 ts_tablet_manager.cc:1403] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Time spent bootstrapping tablet: real 0.024s	user 0.015s	sys 0.000s
I20260709 12:27:14.696360 13423 tablet_bootstrap.cc:654] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:14.697324 13420 raft_consensus.cc:359] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.698292 13420 raft_consensus.cc:385] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:14.698664 13420 raft_consensus.cc:740] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a7dd3fe520c4a949bd7b8b68fc9fe22, State: Initialized, Role: FOLLOWER
I20260709 12:27:14.699501 13420 consensus_queue.cc:260] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.703239 13420 ts_tablet_manager.cc:1434] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 12:27:14.709646 13423 tablet_bootstrap.cc:492] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d: No bootstrap required, opened a new log
I20260709 12:27:14.710398 13423 ts_tablet_manager.cc:1403] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d: Time spent bootstrapping tablet: real 0.022s	user 0.009s	sys 0.008s
I20260709 12:27:14.713804 13423 raft_consensus.cc:359] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.714792 13423 raft_consensus.cc:385] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:14.715137 13423 raft_consensus.cc:740] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efac806c91b40f38e07639bd5ba445d, State: Initialized, Role: FOLLOWER
I20260709 12:27:14.716212 13423 consensus_queue.cc:260] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.718329 13423 ts_tablet_manager.cc:1434] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:27:14.724381 13419 tablet_bootstrap.cc:492] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac: No bootstrap required, opened a new log
I20260709 12:27:14.725060 13419 ts_tablet_manager.cc:1403] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac: Time spent bootstrapping tablet: real 0.044s	user 0.025s	sys 0.011s
I20260709 12:27:14.728073 13419 raft_consensus.cc:359] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.728727 13419 raft_consensus.cc:385] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:14.729032 13419 raft_consensus.cc:740] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3960c41fdd944168b675855e44737cac, State: Initialized, Role: FOLLOWER
I20260709 12:27:14.729830 13419 consensus_queue.cc:260] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.732472 13419 ts_tablet_manager.cc:1434] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac: Time spent starting tablet: real 0.007s	user 0.009s	sys 0.000s
I20260709 12:27:14.735962 13427 raft_consensus.cc:493] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:14.736562 13427 raft_consensus.cc:515] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.738700 13427 leader_election.cc:290] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805), 3960c41fdd944168b675855e44737cac (127.29.201.195:36755)
I20260709 12:27:14.763172 13379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9d78091f8e4409382a42ce2392046ed" candidate_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960c41fdd944168b675855e44737cac" is_pre_election: true
I20260709 12:27:14.763999 13379 raft_consensus.cc:2468] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a7dd3fe520c4a949bd7b8b68fc9fe22 in term 0.
I20260709 12:27:14.763028 13305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9d78091f8e4409382a42ce2392046ed" candidate_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efac806c91b40f38e07639bd5ba445d" is_pre_election: true
I20260709 12:27:14.765442 13193 leader_election.cc:304] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7dd3fe520c4a949bd7b8b68fc9fe22, 3960c41fdd944168b675855e44737cac; no voters: 
I20260709 12:27:14.765857 13305 raft_consensus.cc:2468] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a7dd3fe520c4a949bd7b8b68fc9fe22 in term 0.
I20260709 12:27:14.766388 13427 raft_consensus.cc:2804] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:14.766817 13427 raft_consensus.cc:493] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:14.767252 13427 raft_consensus.cc:3060] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:14.774981 13427 raft_consensus.cc:515] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.777406 13427 leader_election.cc:290] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [CANDIDATE]: Term 1 election: Requested vote from peers 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805), 3960c41fdd944168b675855e44737cac (127.29.201.195:36755)
I20260709 12:27:14.778546 13305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9d78091f8e4409382a42ce2392046ed" candidate_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efac806c91b40f38e07639bd5ba445d"
I20260709 12:27:14.779364 13305 raft_consensus.cc:3060] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:14.780210 13379 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9d78091f8e4409382a42ce2392046ed" candidate_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3960c41fdd944168b675855e44737cac"
I20260709 12:27:14.781085 13379 raft_consensus.cc:3060] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:14.813542 13379 raft_consensus.cc:2468] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7dd3fe520c4a949bd7b8b68fc9fe22 in term 1.
I20260709 12:27:14.892936 13193 leader_election.cc:304] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7dd3fe520c4a949bd7b8b68fc9fe22, 3960c41fdd944168b675855e44737cac; no voters: 
I20260709 12:27:14.904450 13427 raft_consensus.cc:2804] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:14.905058 13427 raft_consensus.cc:697] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 LEADER]: Becoming Leader. State: Replica: 0a7dd3fe520c4a949bd7b8b68fc9fe22, State: Running, Role: LEADER
I20260709 12:27:14.906010 13427 consensus_queue.cc:237] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.907493 13305 raft_consensus.cc:2468] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7dd3fe520c4a949bd7b8b68fc9fe22 in term 1.
W20260709 12:27:14.922770 13176 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:14.928185 13125 catalog_manager.cc:5697] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:14.938887 13422 raft_consensus.cc:493] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:14.939574 13422 raft_consensus.cc:515] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.939992 13427 consensus_queue.cc:1048] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:14.942348 13422 leader_election.cc:290] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945), 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805)
I20260709 12:27:14.964311 13305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a89ad6d2be0e4850945958ab820c7545" candidate_uuid: "3960c41fdd944168b675855e44737cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efac806c91b40f38e07639bd5ba445d" is_pre_election: true
I20260709 12:27:14.965369 13305 raft_consensus.cc:2468] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3960c41fdd944168b675855e44737cac in term 0.
I20260709 12:27:14.967371 13342 leader_election.cc:304] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3960c41fdd944168b675855e44737cac, 8efac806c91b40f38e07639bd5ba445d; no voters: 
I20260709 12:27:14.968888 13422 raft_consensus.cc:2804] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:14.969439 13422 raft_consensus.cc:493] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:14.969882 13422 raft_consensus.cc:3060] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:14.975821 13230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a89ad6d2be0e4850945958ab820c7545" candidate_uuid: "3960c41fdd944168b675855e44737cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" is_pre_election: true
I20260709 12:27:14.976799 13230 raft_consensus.cc:2468] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3960c41fdd944168b675855e44737cac in term 0.
I20260709 12:27:14.982362 13427 consensus_queue.cc:1048] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:14.990767 13428 consensus_queue.cc:1048] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:14.992749 13422 raft_consensus.cc:515] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:14.996970 13229 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a89ad6d2be0e4850945958ab820c7545" candidate_uuid: "3960c41fdd944168b675855e44737cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22"
I20260709 12:27:14.997622 13229 raft_consensus.cc:3060] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:14.998888 13304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a89ad6d2be0e4850945958ab820c7545" candidate_uuid: "3960c41fdd944168b675855e44737cac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efac806c91b40f38e07639bd5ba445d"
I20260709 12:27:14.999845 13304 raft_consensus.cc:3060] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:15.000849 13428 consensus_queue.cc:1048] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:15.010437 13229 raft_consensus.cc:2468] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3960c41fdd944168b675855e44737cac in term 1.
I20260709 12:27:15.011142 13422 leader_election.cc:290] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 1 election: Requested vote from peers 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945), 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805)
I20260709 12:27:15.011924 13342 leader_election.cc:304] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7dd3fe520c4a949bd7b8b68fc9fe22, 3960c41fdd944168b675855e44737cac; no voters: 
I20260709 12:27:15.012007 13304 raft_consensus.cc:2468] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3960c41fdd944168b675855e44737cac in term 1.
I20260709 12:27:15.012948 13422 raft_consensus.cc:2804] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:15.015543 13422 raft_consensus.cc:697] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 1 LEADER]: Becoming Leader. State: Replica: 3960c41fdd944168b675855e44737cac, State: Running, Role: LEADER
I20260709 12:27:15.017097 13422 consensus_queue.cc:237] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:15.040007 13125 catalog_manager.cc:5697] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac reported cstate change: term changed from 0 to 1, leader changed from <none> to 3960c41fdd944168b675855e44737cac (127.29.201.195). New cstate: current_term: 1 leader_uuid: "3960c41fdd944168b675855e44737cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:15.096030 13305 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "9dfe53b95c194ea18f1cc7126f1f7719"
dest_uuid: "8efac806c91b40f38e07639bd5ba445d"
mode: GRACEFUL
new_leader_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22"
 from {username='slave'} at 127.0.0.1:59632
I20260709 12:27:15.096576 13305 raft_consensus.cc:606] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 1 LEADER]: Received request to transfer leadership to TS 0a7dd3fe520c4a949bd7b8b68fc9fe22
I20260709 12:27:15.111706 13230 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "19da758fe12148fcbce4c890bdf31287"
dest_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22"
mode: GRACEFUL
new_leader_uuid: "3960c41fdd944168b675855e44737cac"
 from {username='slave'} at 127.0.0.1:39678
I20260709 12:27:15.112272 13230 raft_consensus.cc:606] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 LEADER]: Received request to transfer leadership to TS 3960c41fdd944168b675855e44737cac
I20260709 12:27:15.129043 13378 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a89ad6d2be0e4850945958ab820c7545"
dest_uuid: "3960c41fdd944168b675855e44737cac"
mode: GRACEFUL
new_leader_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22"
 from {username='slave'} at 127.0.0.1:39166
I20260709 12:27:15.129770 13378 raft_consensus.cc:606] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 1 LEADER]: Received request to transfer leadership to TS 0a7dd3fe520c4a949bd7b8b68fc9fe22
I20260709 12:27:15.323576 13436 consensus_queue.cc:1048] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:15.334779 13457 consensus_queue.cc:1048] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:15.479166 13422 consensus_queue.cc:1048] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:15.493930 13470 consensus_queue.cc:1048] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:15.503695 13470 raft_consensus.cc:993] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac: : Instructing follower 0a7dd3fe520c4a949bd7b8b68fc9fe22 to start an election
I20260709 12:27:15.504205 13422 raft_consensus.cc:1081] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 1 LEADER]: Signalling peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 to start an election
I20260709 12:27:15.506135 13230 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a89ad6d2be0e4850945958ab820c7545"
dest_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22"
 from {username='slave'} at 127.0.0.1:39674
I20260709 12:27:15.506875 13230 raft_consensus.cc:493] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:15.507236 13230 raft_consensus.cc:3060] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:15.513053 13230 raft_consensus.cc:515] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:15.514804 13230 leader_election.cc:290] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [CANDIDATE]: Term 2 election: Requested vote from peers 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805), 3960c41fdd944168b675855e44737cac (127.29.201.195:36755)
I20260709 12:27:15.515730 13304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a89ad6d2be0e4850945958ab820c7545" candidate_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8efac806c91b40f38e07639bd5ba445d"
I20260709 12:27:15.516223 13378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a89ad6d2be0e4850945958ab820c7545" candidate_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3960c41fdd944168b675855e44737cac"
I20260709 12:27:15.516501 13304 raft_consensus.cc:3060] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:15.516989 13378 raft_consensus.cc:3055] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:15.517320 13378 raft_consensus.cc:740] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 3960c41fdd944168b675855e44737cac, State: Running, Role: LEADER
I20260709 12:27:15.518069 13378 consensus_queue.cc:260] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:15.519145 13378 raft_consensus.cc:3060] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:15.523293 13304 raft_consensus.cc:2468] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7dd3fe520c4a949bd7b8b68fc9fe22 in term 2.
I20260709 12:27:15.524544 13193 leader_election.cc:304] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7dd3fe520c4a949bd7b8b68fc9fe22, 8efac806c91b40f38e07639bd5ba445d; no voters: 
I20260709 12:27:15.525301 13436 raft_consensus.cc:2804] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:15.525797 13436 raft_consensus.cc:697] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 LEADER]: Becoming Leader. State: Replica: 0a7dd3fe520c4a949bd7b8b68fc9fe22, State: Running, Role: LEADER
I20260709 12:27:15.526271 13378 raft_consensus.cc:2468] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7dd3fe520c4a949bd7b8b68fc9fe22 in term 2.
I20260709 12:27:15.526731 13436 consensus_queue.cc:237] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:15.534039 13126 catalog_manager.cc:5697] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 reported cstate change: term changed from 1 to 2, leader changed from 3960c41fdd944168b675855e44737cac (127.29.201.195) to 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193). New cstate: current_term: 2 leader_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:15.550444 13464 raft_consensus.cc:993] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d: : Instructing follower 0a7dd3fe520c4a949bd7b8b68fc9fe22 to start an election
I20260709 12:27:15.551011 13458 raft_consensus.cc:1081] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 1 LEADER]: Signalling peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 to start an election
I20260709 12:27:15.552562 13230 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "9dfe53b95c194ea18f1cc7126f1f7719"
dest_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22"
 from {username='slave'} at 127.0.0.1:39666
I20260709 12:27:15.553151 13230 raft_consensus.cc:493] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:15.553508 13230 raft_consensus.cc:3060] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:15.558965 13230 raft_consensus.cc:515] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:15.561190 13230 leader_election.cc:290] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [CANDIDATE]: Term 2 election: Requested vote from peers 3960c41fdd944168b675855e44737cac (127.29.201.195:36755), 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805)
I20260709 12:27:15.562153 13378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dfe53b95c194ea18f1cc7126f1f7719" candidate_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3960c41fdd944168b675855e44737cac"
I20260709 12:27:15.562404 13304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dfe53b95c194ea18f1cc7126f1f7719" candidate_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8efac806c91b40f38e07639bd5ba445d"
I20260709 12:27:15.563031 13378 raft_consensus.cc:3060] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:15.563118 13304 raft_consensus.cc:3055] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:15.563536 13304 raft_consensus.cc:740] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8efac806c91b40f38e07639bd5ba445d, State: Running, Role: LEADER
I20260709 12:27:15.564213 13304 consensus_queue.cc:260] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:15.565152 13304 raft_consensus.cc:3060] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:15.568848 13378 raft_consensus.cc:2468] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7dd3fe520c4a949bd7b8b68fc9fe22 in term 2.
I20260709 12:27:15.570182 13193 leader_election.cc:304] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a7dd3fe520c4a949bd7b8b68fc9fe22, 3960c41fdd944168b675855e44737cac; no voters: 
I20260709 12:27:15.570216 13304 raft_consensus.cc:2468] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a7dd3fe520c4a949bd7b8b68fc9fe22 in term 2.
I20260709 12:27:15.571193 13436 raft_consensus.cc:2804] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:15.572078 13436 raft_consensus.cc:697] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 LEADER]: Becoming Leader. State: Replica: 0a7dd3fe520c4a949bd7b8b68fc9fe22, State: Running, Role: LEADER
I20260709 12:27:15.572826 13436 consensus_queue.cc:237] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:15.579716 13427 raft_consensus.cc:993] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: : Instructing follower 3960c41fdd944168b675855e44737cac to start an election
I20260709 12:27:15.580211 13436 raft_consensus.cc:1081] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 LEADER]: Signalling peer 3960c41fdd944168b675855e44737cac to start an election
I20260709 12:27:15.580570 13125 catalog_manager.cc:5697] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 reported cstate change: term changed from 1 to 2, leader changed from 8efac806c91b40f38e07639bd5ba445d (127.29.201.194) to 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193). New cstate: current_term: 2 leader_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:15.581780 13378 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "19da758fe12148fcbce4c890bdf31287"
dest_uuid: "3960c41fdd944168b675855e44737cac"
 from {username='slave'} at 127.0.0.1:39152
I20260709 12:27:15.582429 13378 raft_consensus.cc:493] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:15.582839 13378 raft_consensus.cc:3060] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:15.591181 13378 raft_consensus.cc:515] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:15.593456 13378 leader_election.cc:290] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 2 election: Requested vote from peers 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945), 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805)
I20260709 12:27:15.595558 13304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19da758fe12148fcbce4c890bdf31287" candidate_uuid: "3960c41fdd944168b675855e44737cac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "8efac806c91b40f38e07639bd5ba445d"
I20260709 12:27:15.595608 13230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "19da758fe12148fcbce4c890bdf31287" candidate_uuid: "3960c41fdd944168b675855e44737cac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22"
I20260709 12:27:15.596810 13304 raft_consensus.cc:3060] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:15.596987 13230 raft_consensus.cc:3055] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:15.597718 13230 raft_consensus.cc:740] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 0a7dd3fe520c4a949bd7b8b68fc9fe22, State: Running, Role: LEADER
I20260709 12:27:15.599042 13230 consensus_queue.cc:260] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:15.601434 13230 raft_consensus.cc:3060] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:15.605511 13304 raft_consensus.cc:2468] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3960c41fdd944168b675855e44737cac in term 2.
I20260709 12:27:15.607290 13342 leader_election.cc:304] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3960c41fdd944168b675855e44737cac, 8efac806c91b40f38e07639bd5ba445d; no voters: 
I20260709 12:27:15.608295 13422 raft_consensus.cc:2804] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:15.608903 13422 raft_consensus.cc:697] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [term 2 LEADER]: Becoming Leader. State: Replica: 3960c41fdd944168b675855e44737cac, State: Running, Role: LEADER
I20260709 12:27:15.609918 13422 consensus_queue.cc:237] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:15.611327 13230 raft_consensus.cc:2468] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3960c41fdd944168b675855e44737cac in term 2.
I20260709 12:27:15.617998 13126 catalog_manager.cc:5697] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac reported cstate change: term changed from 1 to 2, leader changed from 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193) to 3960c41fdd944168b675855e44737cac (127.29.201.195). New cstate: current_term: 2 leader_uuid: "3960c41fdd944168b675855e44737cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:15.796355 13251 debug-util.cc:398] Leaking SignalData structure 0x7b0800151620 after lost signal to thread 30506
W20260709 12:27:15.797609 13251 debug-util.cc:398] Leaking SignalData structure 0x7b08001c1fe0 after lost signal to thread 13162
I20260709 12:27:15.888478 30503 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table1
I20260709 12:27:15.890166 30503 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table1, leader rebalance finish, leader transfer count: 0
I20260709 12:27:15.890617 30503 auto_leader_rebalancer.cc:141] leader rebalance for table multi_table_leader_balance_table2
I20260709 12:27:15.908416 13230 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a89ad6d2be0e4850945958ab820c7545"
dest_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22"
mode: GRACEFUL
new_leader_uuid: "8efac806c91b40f38e07639bd5ba445d"
 from {username='slave'} at 127.0.0.1:39690
I20260709 12:27:15.909235 13230 raft_consensus.cc:606] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 LEADER]: Received request to transfer leadership to TS 8efac806c91b40f38e07639bd5ba445d
I20260709 12:27:15.910653 30503 auto_leader_rebalancer.cc:440] table: multi_table_leader_balance_table2, leader rebalance finish, leader transfer count: 1
I20260709 12:27:15.922114 13304 raft_consensus.cc:1275] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Refusing update from remote peer 0a7dd3fe520c4a949bd7b8b68fc9fe22: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:15.923617 13436 consensus_queue.cc:1048] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:15.935799 13427 raft_consensus.cc:993] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: : Instructing follower 8efac806c91b40f38e07639bd5ba445d to start an election
I20260709 12:27:15.936483 13457 raft_consensus.cc:1081] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 LEADER]: Signalling peer 8efac806c91b40f38e07639bd5ba445d to start an election
I20260709 12:27:15.939078 13304 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a89ad6d2be0e4850945958ab820c7545"
dest_uuid: "8efac806c91b40f38e07639bd5ba445d"
 from {username='slave'} at 127.0.0.1:59610
I20260709 12:27:15.939704 13304 raft_consensus.cc:493] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:15.940079 13304 raft_consensus.cc:3060] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:15.942341 13378 raft_consensus.cc:1275] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Refusing update from remote peer 0a7dd3fe520c4a949bd7b8b68fc9fe22: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:15.946985 13427 consensus_queue.cc:1048] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:15.951195 13304 raft_consensus.cc:515] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:15.953097 13305 raft_consensus.cc:1240] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 3 FOLLOWER]: Rejecting Update request from peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 for earlier term 2. Current term is 3. Ops: []
I20260709 12:27:15.957043 13474 consensus_queue.cc:1059] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:15.958348 13474 raft_consensus.cc:3055] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:15.958889 13474 raft_consensus.cc:740] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 0a7dd3fe520c4a949bd7b8b68fc9fe22, State: Running, Role: LEADER
I20260709 12:27:15.959964 13474 consensus_queue.cc:260] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:15.960409 13230 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a89ad6d2be0e4850945958ab820c7545" candidate_uuid: "8efac806c91b40f38e07639bd5ba445d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22"
I20260709 12:27:15.961575 13474 raft_consensus.cc:3060] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:15.962611 13304 leader_election.cc:290] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [CANDIDATE]: Term 3 election: Requested vote from peers 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945), 3960c41fdd944168b675855e44737cac (127.29.201.195:36755)
I20260709 12:27:15.966143 13378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a89ad6d2be0e4850945958ab820c7545" candidate_uuid: "8efac806c91b40f38e07639bd5ba445d" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "3960c41fdd944168b675855e44737cac"
I20260709 12:27:15.966847 13378 raft_consensus.cc:3060] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:15.974455 13378 raft_consensus.cc:2468] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8efac806c91b40f38e07639bd5ba445d in term 3.
I20260709 12:27:15.975795 13268 leader_election.cc:304] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3960c41fdd944168b675855e44737cac, 8efac806c91b40f38e07639bd5ba445d; no voters: 
I20260709 12:27:15.976799 13458 raft_consensus.cc:2804] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:15.977366 13458 raft_consensus.cc:697] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 3 LEADER]: Becoming Leader. State: Replica: 8efac806c91b40f38e07639bd5ba445d, State: Running, Role: LEADER
I20260709 12:27:15.978294 13458 consensus_queue.cc:237] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:15.980145 13230 raft_consensus.cc:2468] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8efac806c91b40f38e07639bd5ba445d in term 3.
I20260709 12:27:15.987684 13125 catalog_manager.cc:5697] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d reported cstate change: term changed from 2 to 3, leader changed from 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193) to 8efac806c91b40f38e07639bd5ba445d (127.29.201.194). New cstate: current_term: 3 leader_uuid: "8efac806c91b40f38e07639bd5ba445d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:16.007449 30503 tablet_server.cc:179] TabletServer@127.29.201.193:0 shutting down...
I20260709 12:27:16.041190 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:16.041900 30503 tablet_replica.cc:333] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22: stopping tablet replica
I20260709 12:27:16.042541 30503 raft_consensus.cc:2243] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.043510 30503 raft_consensus.cc:2272] T a9d78091f8e4409382a42ce2392046ed P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.045950 30503 tablet_replica.cc:333] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: stopping tablet replica
I20260709 12:27:16.046833 30503 raft_consensus.cc:2243] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.047466 30503 raft_consensus.cc:2272] T a89ad6d2be0e4850945958ab820c7545 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.050213 30503 tablet_replica.cc:333] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: stopping tablet replica
I20260709 12:27:16.050935 30503 raft_consensus.cc:2243] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:16.051403 30503 raft_consensus.cc:2272] T 19da758fe12148fcbce4c890bdf31287 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:16.053603 30503 tablet_replica.cc:333] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: stopping tablet replica
I20260709 12:27:16.054153 30503 raft_consensus.cc:2243] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:16.055114 30503 pending_rounds.cc:70] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Trying to abort 1 pending ops.
I20260709 12:27:16.055354 30503 pending_rounds.cc:77] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7305625745715515392 op_type: NO_OP noop_request { }
I20260709 12:27:16.055754 30503 raft_consensus.cc:2889] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260709 12:27:16.056069 30503 raft_consensus.cc:2272] T 9dfe53b95c194ea18f1cc7126f1f7719 P 0a7dd3fe520c4a949bd7b8b68fc9fe22 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:16.089671 13342 consensus_peers.cc:597] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac -> Peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945): Couldn't send request to peer 0a7dd3fe520c4a949bd7b8b68fc9fe22. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:42945: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:16.128244 13304 raft_consensus.cc:1275] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Refusing update from remote peer 3960c41fdd944168b675855e44737cac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:16.130378 13422 consensus_queue.cc:1048] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:16.419842 13378 raft_consensus.cc:1275] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 3 FOLLOWER]: Refusing update from remote peer 8efac806c91b40f38e07639bd5ba445d: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:16.421579 13458 consensus_queue.cc:1048] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260709 12:27:16.448639 13268 consensus_peers.cc:597] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d -> Peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945): Couldn't send request to peer 0a7dd3fe520c4a949bd7b8b68fc9fe22. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:42945: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:17.084357 30503 thread.cc:527] Waited for 1000ms trying to join with diag-logger (tid 13251)
I20260709 12:27:17.268234 13458 raft_consensus.cc:493] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a7dd3fe520c4a949bd7b8b68fc9fe22)
I20260709 12:27:17.268986 13458 raft_consensus.cc:515] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:17.271124 13458 leader_election.cc:290] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945), 3960c41fdd944168b675855e44737cac (127.29.201.195:36755)
I20260709 12:27:17.273146 13378 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9d78091f8e4409382a42ce2392046ed" candidate_uuid: "8efac806c91b40f38e07639bd5ba445d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "3960c41fdd944168b675855e44737cac" is_pre_election: true
W20260709 12:27:17.275676 13268 leader_election.cc:336] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945): Network error: Client connection negotiation failed: client connection to 127.29.201.193:42945: connect: Connection refused (error 111)
I20260709 12:27:17.276157 13268 leader_election.cc:304] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8efac806c91b40f38e07639bd5ba445d; no voters: 0a7dd3fe520c4a949bd7b8b68fc9fe22, 3960c41fdd944168b675855e44737cac
I20260709 12:27:17.277119 13458 raft_consensus.cc:2749] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260709 12:27:17.283161 13478 raft_consensus.cc:493] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0a7dd3fe520c4a949bd7b8b68fc9fe22)
I20260709 12:27:17.283823 13478 raft_consensus.cc:515] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:17.285815 13478 leader_election.cc:290] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945), 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805)
I20260709 12:27:17.288033 13304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9d78091f8e4409382a42ce2392046ed" candidate_uuid: "3960c41fdd944168b675855e44737cac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8efac806c91b40f38e07639bd5ba445d" is_pre_election: true
I20260709 12:27:17.288825 13304 raft_consensus.cc:2468] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3960c41fdd944168b675855e44737cac in term 1.
I20260709 12:27:17.290220 13342 leader_election.cc:304] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3960c41fdd944168b675855e44737cac, 8efac806c91b40f38e07639bd5ba445d; no voters: 
I20260709 12:27:17.291085 13478 raft_consensus.cc:2804] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260709 12:27:17.291159 13342 leader_election.cc:336] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945): Network error: Client connection negotiation failed: client connection to 127.29.201.193:42945: connect: Connection refused (error 111)
I20260709 12:27:17.291536 13478 raft_consensus.cc:493] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0a7dd3fe520c4a949bd7b8b68fc9fe22)
I20260709 12:27:17.291976 13478 raft_consensus.cc:3060] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:17.297873 13478 raft_consensus.cc:515] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:17.299976 13478 leader_election.cc:290] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 2 election: Requested vote from peers 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945), 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805)
I20260709 12:27:17.301581 13304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a9d78091f8e4409382a42ce2392046ed" candidate_uuid: "3960c41fdd944168b675855e44737cac" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8efac806c91b40f38e07639bd5ba445d"
I20260709 12:27:17.302354 13304 raft_consensus.cc:3060] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:17.304916 13342 leader_election.cc:336] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945): Network error: Client connection negotiation failed: client connection to 127.29.201.193:42945: connect: Connection refused (error 111)
I20260709 12:27:17.309789 13304 raft_consensus.cc:2468] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3960c41fdd944168b675855e44737cac in term 2.
I20260709 12:27:17.311101 13342 leader_election.cc:304] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3960c41fdd944168b675855e44737cac, 8efac806c91b40f38e07639bd5ba445d; no voters: 0a7dd3fe520c4a949bd7b8b68fc9fe22
I20260709 12:27:17.312104 13478 raft_consensus.cc:2804] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:17.313300 13478 raft_consensus.cc:697] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 2 LEADER]: Becoming Leader. State: Replica: 3960c41fdd944168b675855e44737cac, State: Running, Role: LEADER
I20260709 12:27:17.314170 13478 consensus_queue.cc:237] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } }
I20260709 12:27:17.322695 13125 catalog_manager.cc:5697] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac reported cstate change: term changed from 1 to 2, leader changed from 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193) to 3960c41fdd944168b675855e44737cac (127.29.201.195). New cstate: current_term: 2 leader_uuid: "3960c41fdd944168b675855e44737cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:17.789144 13478 raft_consensus.cc:493] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:17.789747 13478 raft_consensus.cc:515] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:17.791847 13478 leader_election.cc:290] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945), 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805)
I20260709 12:27:17.793875 13304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dfe53b95c194ea18f1cc7126f1f7719" candidate_uuid: "3960c41fdd944168b675855e44737cac" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8efac806c91b40f38e07639bd5ba445d" is_pre_election: true
I20260709 12:27:17.794764 13304 raft_consensus.cc:2468] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3960c41fdd944168b675855e44737cac in term 2.
I20260709 12:27:17.796658 13342 leader_election.cc:304] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3960c41fdd944168b675855e44737cac, 8efac806c91b40f38e07639bd5ba445d; no voters: 
I20260709 12:27:17.797505 13478 raft_consensus.cc:2804] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260709 12:27:17.797855 13478 raft_consensus.cc:493] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:17.798152 13478 raft_consensus.cc:3060] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:27:17.803371 13342 leader_election.cc:336] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945): Network error: Client connection negotiation failed: client connection to 127.29.201.193:42945: connect: Connection refused (error 111)
I20260709 12:27:17.804339 13478 raft_consensus.cc:515] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:17.806728 13478 leader_election.cc:290] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 3 election: Requested vote from peers 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945), 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805)
I20260709 12:27:17.808482 13304 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9dfe53b95c194ea18f1cc7126f1f7719" candidate_uuid: "3960c41fdd944168b675855e44737cac" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8efac806c91b40f38e07639bd5ba445d"
I20260709 12:27:17.809259 13304 raft_consensus.cc:3060] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:27:17.810595 13342 leader_election.cc:336] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945): Network error: Client connection negotiation failed: client connection to 127.29.201.193:42945: connect: Connection refused (error 111)
I20260709 12:27:17.818465 13304 raft_consensus.cc:2468] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3960c41fdd944168b675855e44737cac in term 3.
I20260709 12:27:17.820288 13342 leader_election.cc:304] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3960c41fdd944168b675855e44737cac, 8efac806c91b40f38e07639bd5ba445d; no voters: 0a7dd3fe520c4a949bd7b8b68fc9fe22
I20260709 12:27:17.821383 13478 raft_consensus.cc:2804] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:17.823503 13478 raft_consensus.cc:697] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 3 LEADER]: Becoming Leader. State: Replica: 3960c41fdd944168b675855e44737cac, State: Running, Role: LEADER
I20260709 12:27:17.824435 13478 consensus_queue.cc:237] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } }
I20260709 12:27:17.834985 13125 catalog_manager.cc:5697] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac reported cstate change: term changed from 2 to 3, leader changed from 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193) to 3960c41fdd944168b675855e44737cac (127.29.201.195). New cstate: current_term: 3 leader_uuid: "3960c41fdd944168b675855e44737cac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3960c41fdd944168b675855e44737cac" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 36755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a7dd3fe520c4a949bd7b8b68fc9fe22" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 42945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:17.877688 13304 raft_consensus.cc:1275] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Refusing update from remote peer 3960c41fdd944168b675855e44737cac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:17.879242 13478 consensus_queue.cc:1048] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:27:17.901068 13342 consensus_peers.cc:597] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac -> Peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945): Couldn't send request to peer 0a7dd3fe520c4a949bd7b8b68fc9fe22. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:42945: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:18.084969 30503 thread.cc:527] Waited for 2000ms trying to join with diag-logger (tid 13251)
W20260709 12:27:18.196537 13342 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.201.193:42945: connect: Connection refused (error 111) [suppressed 23 similar messages]
W20260709 12:27:18.230988 13342 consensus_peers.cc:597] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac -> Peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945): Couldn't send request to peer 0a7dd3fe520c4a949bd7b8b68fc9fe22. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:42945: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:18.360477 13304 raft_consensus.cc:1275] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 3 FOLLOWER]: Refusing update from remote peer 3960c41fdd944168b675855e44737cac: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260709 12:27:18.361881 13494 consensus_queue.cc:1048] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [LEADER]: Connected to new peer: Peer: permanent_uuid: "8efac806c91b40f38e07639bd5ba445d" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 32805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260709 12:27:18.683262 13342 consensus_peers.cc:597] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac -> Peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945): Couldn't send request to peer 0a7dd3fe520c4a949bd7b8b68fc9fe22. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:42945: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20260709 12:27:19.085676 30503 thread.cc:527] Waited for 3000ms trying to join with diag-logger (tid 13251)
W20260709 12:27:19.101850 13268 consensus_peers.cc:597] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d -> Peer 0a7dd3fe520c4a949bd7b8b68fc9fe22 (127.29.201.193:42945): Couldn't send request to peer 0a7dd3fe520c4a949bd7b8b68fc9fe22. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:42945: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260709 12:27:19.508025 30503 tablet_server.cc:196] TabletServer@127.29.201.193:0 shutdown complete.
I20260709 12:27:19.523509 30503 tablet_server.cc:179] TabletServer@127.29.201.194:0 shutting down...
I20260709 12:27:19.544879 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:19.545539 30503 tablet_replica.cc:333] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d: stopping tablet replica
I20260709 12:27:19.546099 30503 raft_consensus.cc:2243] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:19.546576 30503 raft_consensus.cc:2272] T a9d78091f8e4409382a42ce2392046ed P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:19.548776 30503 tablet_replica.cc:333] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d: stopping tablet replica
I20260709 12:27:19.549327 30503 raft_consensus.cc:2243] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:19.550170 30503 raft_consensus.cc:2272] T a89ad6d2be0e4850945958ab820c7545 P 8efac806c91b40f38e07639bd5ba445d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:19.552242 30503 tablet_replica.cc:333] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d: stopping tablet replica
I20260709 12:27:19.552717 30503 raft_consensus.cc:2243] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:19.553103 30503 raft_consensus.cc:2272] T 19da758fe12148fcbce4c890bdf31287 P 8efac806c91b40f38e07639bd5ba445d [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:19.555044 30503 tablet_replica.cc:333] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d: stopping tablet replica
I20260709 12:27:19.555477 30503 raft_consensus.cc:2243] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:19.555855 30503 raft_consensus.cc:2272] T 9dfe53b95c194ea18f1cc7126f1f7719 P 8efac806c91b40f38e07639bd5ba445d [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:19.570931 30503 tablet_server.cc:196] TabletServer@127.29.201.194:0 shutdown complete.
W20260709 12:27:19.571328 13342 consensus_peers.cc:597] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac -> Peer 8efac806c91b40f38e07639bd5ba445d (127.29.201.194:32805): Couldn't send request to peer 8efac806c91b40f38e07639bd5ba445d. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:32805: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:19.585443 30503 tablet_server.cc:179] TabletServer@127.29.201.195:0 shutting down...
I20260709 12:27:19.607703 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:19.608412 30503 tablet_replica.cc:333] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac: stopping tablet replica
I20260709 12:27:19.609010 30503 raft_consensus.cc:2243] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:19.609937 30503 raft_consensus.cc:2272] T a9d78091f8e4409382a42ce2392046ed P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:19.612054 30503 tablet_replica.cc:333] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac: stopping tablet replica
I20260709 12:27:19.612613 30503 raft_consensus.cc:2243] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:19.613092 30503 raft_consensus.cc:2272] T a89ad6d2be0e4850945958ab820c7545 P 3960c41fdd944168b675855e44737cac [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:19.615171 30503 tablet_replica.cc:333] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac: stopping tablet replica
I20260709 12:27:19.615696 30503 raft_consensus.cc:2243] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:19.616549 30503 raft_consensus.cc:2272] T 9dfe53b95c194ea18f1cc7126f1f7719 P 3960c41fdd944168b675855e44737cac [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:19.618602 30503 tablet_replica.cc:333] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac: stopping tablet replica
I20260709 12:27:19.619175 30503 raft_consensus.cc:2243] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:19.620041 30503 raft_consensus.cc:2272] T 19da758fe12148fcbce4c890bdf31287 P 3960c41fdd944168b675855e44737cac [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:19.644945 30503 tablet_server.cc:196] TabletServer@127.29.201.195:0 shutdown complete.
I20260709 12:27:19.658947 30503 master.cc:562] Master@127.29.201.254:35873 shutting down...
I20260709 12:27:19.677892 30503 raft_consensus.cc:2243] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:19.678432 30503 raft_consensus.cc:2272] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:19.678766 30503 tablet_replica.cc:333] T 00000000000000000000000000000000 P d3aaac24a32e4658bbaf5f24d12d03aa: stopping tablet replica
I20260709 12:27:19.699733 30503 master.cc:584] Master@127.29.201.254:35873 shutdown complete.
[       OK ] LeaderRebalancerTest.MultiTableLeaderBalance (6175 ms)
[----------] 6 tests from LeaderRebalancerTest (486239 ms total)

[----------] 2 tests from FilterSoftDeletedTableTest
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0
I20260709 12:27:19.730222 30503 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.201.254:44713
I20260709 12:27:19.731551 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:19.737490 13508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:19.738124 13509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:19.738868 13511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:19.739295 30503 server_base.cc:1061] running on GCE node
I20260709 12:27:19.740253 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:19.740471 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:19.740649 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600039740626 us; error 0 us; skew 500 ppm
I20260709 12:27:19.741196 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:19.743790 30503 webserver.cc:533] Webserver started at http://127.29.201.254:35491/ using document root <none> and password file <none>
I20260709 12:27:19.744323 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:19.744545 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:19.744827 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:19.745944 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/master-0-root/instance:
uuid: "dd0a13d159d04fe094fb12e9ed83929a"
format_stamp: "Formatted at 2026-07-09 12:27:19 on dist-test-slave-qgf5"
I20260709 12:27:19.750828 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.007s	sys 0.000s
I20260709 12:27:19.754468 13516 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:19.755599 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:27:19.755887 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/master-0-root
uuid: "dd0a13d159d04fe094fb12e9ed83929a"
format_stamp: "Formatted at 2026-07-09 12:27:19 on dist-test-slave-qgf5"
I20260709 12:27:19.756156 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:19.771773 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:19.772877 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:19.811125 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.254:44713
I20260709 12:27:19.811237 13567 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.254:44713 every 8 connection(s)
I20260709 12:27:19.815266 13568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:19.827077 13568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a: Bootstrap starting.
I20260709 12:27:19.831812 13568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:19.836305 13568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a: No bootstrap required, opened a new log
I20260709 12:27:19.838614 13568 raft_consensus.cc:359] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0a13d159d04fe094fb12e9ed83929a" member_type: VOTER }
I20260709 12:27:19.839037 13568 raft_consensus.cc:385] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:19.839339 13568 raft_consensus.cc:740] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd0a13d159d04fe094fb12e9ed83929a, State: Initialized, Role: FOLLOWER
I20260709 12:27:19.839967 13568 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0a13d159d04fe094fb12e9ed83929a" member_type: VOTER }
I20260709 12:27:19.840471 13568 raft_consensus.cc:399] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:27:19.840677 13568 raft_consensus.cc:493] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:27:19.840991 13568 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:19.846325 13568 raft_consensus.cc:515] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0a13d159d04fe094fb12e9ed83929a" member_type: VOTER }
I20260709 12:27:19.846956 13568 leader_election.cc:304] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd0a13d159d04fe094fb12e9ed83929a; no voters: 
I20260709 12:27:19.848186 13568 leader_election.cc:290] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:27:19.848563 13571 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:19.850046 13571 raft_consensus.cc:697] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [term 1 LEADER]: Becoming Leader. State: Replica: dd0a13d159d04fe094fb12e9ed83929a, State: Running, Role: LEADER
I20260709 12:27:19.850831 13571 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0a13d159d04fe094fb12e9ed83929a" member_type: VOTER }
I20260709 12:27:19.851163 13568 sys_catalog.cc:565] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:27:19.855096 13573 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd0a13d159d04fe094fb12e9ed83929a. Latest consensus state: current_term: 1 leader_uuid: "dd0a13d159d04fe094fb12e9ed83929a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0a13d159d04fe094fb12e9ed83929a" member_type: VOTER } }
I20260709 12:27:19.855865 13573 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:19.855096 13572 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd0a13d159d04fe094fb12e9ed83929a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd0a13d159d04fe094fb12e9ed83929a" member_type: VOTER } }
I20260709 12:27:19.856848 13572 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:19.860563 13577 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:27:19.868678 13577 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:27:19.869745 30503 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:27:19.878304 13577 catalog_manager.cc:1383] Generated new cluster ID: a12b261008ae45f1aa8507fc01b202a6
I20260709 12:27:19.878695 13577 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:27:19.905454 13577 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:27:19.906975 13577 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:27:19.920230 13577 catalog_manager.cc:6070] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a: Generated new TSK 0
I20260709 12:27:19.920924 13577 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:27:19.937781 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:19.944643 13589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:19.945549 13590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:19.948217 13592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:19.948285 30503 server_base.cc:1061] running on GCE node
I20260709 12:27:19.949514 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:19.949748 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:19.949993 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600039949969 us; error 0 us; skew 500 ppm
I20260709 12:27:19.950770 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:19.953686 30503 webserver.cc:533] Webserver started at http://127.29.201.193:39081/ using document root <none> and password file <none>
I20260709 12:27:19.954281 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:19.954525 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:19.954902 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:19.956174 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-0-root/instance:
uuid: "99369f74684341af8f9ba9ff6c225216"
format_stamp: "Formatted at 2026-07-09 12:27:19 on dist-test-slave-qgf5"
I20260709 12:27:19.961668 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.001s	sys 0.005s
I20260709 12:27:19.966109 13597 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:19.967161 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:27:19.967526 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-0-root
uuid: "99369f74684341af8f9ba9ff6c225216"
format_stamp: "Formatted at 2026-07-09 12:27:19 on dist-test-slave-qgf5"
I20260709 12:27:19.967871 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:19.993427 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:19.994952 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:19.996670 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:19.999331 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:19.999598 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:19.999884 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:20.000092 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:20.045456 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.193:39059
I20260709 12:27:20.045557 13659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.193:39059 every 8 connection(s)
I20260709 12:27:20.051066 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:20.061502 13664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:20.066208 13665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:20.067651 30503 server_base.cc:1061] running on GCE node
W20260709 12:27:20.068651 13667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:20.070600 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:20.070958 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:20.071215 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600040071195 us; error 0 us; skew 500 ppm
I20260709 12:27:20.072105 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:20.073004 13660 heartbeater.cc:344] Connected to a master server at 127.29.201.254:44713
I20260709 12:27:20.073635 13660 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:20.074695 13660 heartbeater.cc:507] Master 127.29.201.254:44713 requested a full tablet report, sending...
I20260709 12:27:20.075567 30503 webserver.cc:533] Webserver started at http://127.29.201.194:45807/ using document root <none> and password file <none>
I20260709 12:27:20.076225 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:20.076473 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:20.076789 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:20.077164 13533 ts_manager.cc:194] Registered new tserver with Master: 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059)
I20260709 12:27:20.078120 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-1-root/instance:
uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
format_stamp: "Formatted at 2026-07-09 12:27:20 on dist-test-slave-qgf5"
I20260709 12:27:20.079214 13533 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46774
I20260709 12:27:20.083411 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.005s	user 0.003s	sys 0.001s
I20260709 12:27:20.087262 13672 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:20.088145 30503 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.002s	sys 0.002s
I20260709 12:27:20.088503 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-1-root
uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
format_stamp: "Formatted at 2026-07-09 12:27:20 on dist-test-slave-qgf5"
I20260709 12:27:20.088869 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:20.104641 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:20.105901 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:20.107781 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:20.110491 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:20.110867 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.001s	sys 0.000s
I20260709 12:27:20.111172 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:20.111375 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:20.154809 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.194:45095
I20260709 12:27:20.154933 13734 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.194:45095 every 8 connection(s)
I20260709 12:27:20.160593 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:20.168763 13738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:20.170894 13739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:20.175449 13741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:20.175117 30503 server_base.cc:1061] running on GCE node
I20260709 12:27:20.176529 13735 heartbeater.cc:344] Connected to a master server at 127.29.201.254:44713
I20260709 12:27:20.176898 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260709 12:27:20.177026 13735 heartbeater.cc:461] Registering TS with master...
W20260709 12:27:20.177261 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:20.177484 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600040177466 us; error 0 us; skew 500 ppm
I20260709 12:27:20.177999 13735 heartbeater.cc:507] Master 127.29.201.254:44713 requested a full tablet report, sending...
I20260709 12:27:20.178321 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:20.180634 13533 ts_manager.cc:194] Registered new tserver with Master: f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:20.181280 30503 webserver.cc:533] Webserver started at http://127.29.201.195:44223/ using document root <none> and password file <none>
I20260709 12:27:20.181869 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:20.182102 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:20.182262 13533 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46782
I20260709 12:27:20.182379 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:20.183912 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-2-root/instance:
uuid: "a5e7dc958a3f45de8436922490e08ef8"
format_stamp: "Formatted at 2026-07-09 12:27:20 on dist-test-slave-qgf5"
I20260709 12:27:20.188506 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:27:20.192155 13746 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:20.193096 30503 fs_manager.cc:730] Time spent opening block manager: real 0.002s	user 0.001s	sys 0.000s
I20260709 12:27:20.193451 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-2-root
uuid: "a5e7dc958a3f45de8436922490e08ef8"
format_stamp: "Formatted at 2026-07-09 12:27:20 on dist-test-slave-qgf5"
I20260709 12:27:20.193768 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_0.1783599553313367-30503-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:20.230901 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:20.232088 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:20.233760 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:20.236269 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:20.236510 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:20.236757 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:20.236929 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:20.277264 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.195:42677
I20260709 12:27:20.277446 13808 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.195:42677 every 8 connection(s)
I20260709 12:27:20.292299 13809 heartbeater.cc:344] Connected to a master server at 127.29.201.254:44713
I20260709 12:27:20.292857 13809 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:20.293632 13809 heartbeater.cc:507] Master 127.29.201.254:44713 requested a full tablet report, sending...
I20260709 12:27:20.296149 13533 ts_manager.cc:194] Registered new tserver with Master: a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:20.296706 30503 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.016292679s
I20260709 12:27:20.298254 13533 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46784
I20260709 12:27:20.325377 13533 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46796:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:27:20.328068 13533 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:27:20.402419 13700 tablet_service.cc:1511] Processing CreateTablet for tablet 93fd5bdcec364b0fa44abe6a94e822ef (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:20.404124 13700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93fd5bdcec364b0fa44abe6a94e822ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.407986 13699 tablet_service.cc:1511] Processing CreateTablet for tablet 81f900f101f543fa8b3cfea0fb81f776 (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:20.409691 13699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f900f101f543fa8b3cfea0fb81f776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.412441 13773 tablet_service.cc:1511] Processing CreateTablet for tablet 81f900f101f543fa8b3cfea0fb81f776 (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:20.414132 13773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f900f101f543fa8b3cfea0fb81f776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.414620 13774 tablet_service.cc:1511] Processing CreateTablet for tablet 93fd5bdcec364b0fa44abe6a94e822ef (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:20.416096 13774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93fd5bdcec364b0fa44abe6a94e822ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.419880 13697 tablet_service.cc:1511] Processing CreateTablet for tablet 6145a89b623b4ec5b638ec5fe3c030cc (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:20.421581 13697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6145a89b623b4ec5b638ec5fe3c030cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.423230 13625 tablet_service.cc:1511] Processing CreateTablet for tablet 93fd5bdcec364b0fa44abe6a94e822ef (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:20.424801 13625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93fd5bdcec364b0fa44abe6a94e822ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.424535 13770 tablet_service.cc:1511] Processing CreateTablet for tablet 0a1f458064a14018ac4e716f1cda3b73 (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:20.426154 13770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a1f458064a14018ac4e716f1cda3b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.426738 13623 tablet_service.cc:1511] Processing CreateTablet for tablet 5633dbff576b44b8914b2cc23f91de93 (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:20.412564 13772 tablet_service.cc:1511] Processing CreateTablet for tablet 5633dbff576b44b8914b2cc23f91de93 (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:20.427721 13624 tablet_service.cc:1511] Processing CreateTablet for tablet 81f900f101f543fa8b3cfea0fb81f776 (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:20.428248 13623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5633dbff576b44b8914b2cc23f91de93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.429142 13624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f900f101f543fa8b3cfea0fb81f776. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.429384 13772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5633dbff576b44b8914b2cc23f91de93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.418300 13698 tablet_service.cc:1511] Processing CreateTablet for tablet 5633dbff576b44b8914b2cc23f91de93 (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:20.423281 13771 tablet_service.cc:1511] Processing CreateTablet for tablet 6145a89b623b4ec5b638ec5fe3c030cc (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:20.431638 13698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5633dbff576b44b8914b2cc23f91de93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.432113 13771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6145a89b623b4ec5b638ec5fe3c030cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.434546 13696 tablet_service.cc:1511] Processing CreateTablet for tablet 0a1f458064a14018ac4e716f1cda3b73 (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:20.436241 13696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a1f458064a14018ac4e716f1cda3b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.441020 13769 tablet_service.cc:1511] Processing CreateTablet for tablet 2f47887de35d404f83ab0edf3cabb706 (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:20.443883 13768 tablet_service.cc:1511] Processing CreateTablet for tablet f4f25524761a4060871c3cc27c79f9df (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:20.445461 13769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f47887de35d404f83ab0edf3cabb706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.447046 13768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f25524761a4060871c3cc27c79f9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.452715 13767 tablet_service.cc:1511] Processing CreateTablet for tablet 66feb633e0dd4d549c04578e7eac038a (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:20.452975 13622 tablet_service.cc:1511] Processing CreateTablet for tablet 6145a89b623b4ec5b638ec5fe3c030cc (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:20.455102 13622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6145a89b623b4ec5b638ec5fe3c030cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.454448 13767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66feb633e0dd4d549c04578e7eac038a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.459471 13766 tablet_service.cc:1511] Processing CreateTablet for tablet 33fa2959a148483984e00bb7fdcd0dcb (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:20.461163 13766 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33fa2959a148483984e00bb7fdcd0dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.461681 13621 tablet_service.cc:1511] Processing CreateTablet for tablet 0a1f458064a14018ac4e716f1cda3b73 (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:20.463209 13621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a1f458064a14018ac4e716f1cda3b73. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.476524 13620 tablet_service.cc:1511] Processing CreateTablet for tablet 2f47887de35d404f83ab0edf3cabb706 (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:20.477345 13695 tablet_service.cc:1511] Processing CreateTablet for tablet 2f47887de35d404f83ab0edf3cabb706 (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:20.479700 13694 tablet_service.cc:1511] Processing CreateTablet for tablet f4f25524761a4060871c3cc27c79f9df (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:20.480604 13692 tablet_service.cc:1511] Processing CreateTablet for tablet 33fa2959a148483984e00bb7fdcd0dcb (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:20.481005 13618 tablet_service.cc:1511] Processing CreateTablet for tablet 66feb633e0dd4d549c04578e7eac038a (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:20.482288 13693 tablet_service.cc:1511] Processing CreateTablet for tablet 66feb633e0dd4d549c04578e7eac038a (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:20.483369 13617 tablet_service.cc:1511] Processing CreateTablet for tablet 33fa2959a148483984e00bb7fdcd0dcb (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:20.479700 13619 tablet_service.cc:1511] Processing CreateTablet for tablet f4f25524761a4060871c3cc27c79f9df (DEFAULT_TABLE table=test-workload [id=e7124cde96c042018259e07a8a5f853e]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:20.478191 13620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f47887de35d404f83ab0edf3cabb706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.485488 13619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f25524761a4060871c3cc27c79f9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.486132 13617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33fa2959a148483984e00bb7fdcd0dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.484606 13695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f47887de35d404f83ab0edf3cabb706. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.487396 13824 tablet_bootstrap.cc:492] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:20.491012 13693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66feb633e0dd4d549c04578e7eac038a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.492131 13618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66feb633e0dd4d549c04578e7eac038a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.493674 13694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4f25524761a4060871c3cc27c79f9df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.494010 13824 tablet_bootstrap.cc:654] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:20.495085 13692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33fa2959a148483984e00bb7fdcd0dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:20.573004 13824 tablet_bootstrap.cc:492] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:20.573637 13824 ts_tablet_manager.cc:1403] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.087s	user 0.019s	sys 0.025s
I20260709 12:27:20.576844 13824 raft_consensus.cc:359] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:20.577605 13824 raft_consensus.cc:385] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:20.577909 13824 raft_consensus.cc:740] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:20.578974 13824 consensus_queue.cc:260] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:20.619491 13825 tablet_bootstrap.cc:492] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:20.649152 13825 tablet_bootstrap.cc:654] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:20.671413 13827 tablet_bootstrap.cc:492] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:20.681102 13660 heartbeater.cc:499] Master 127.29.201.254:44713 was elected leader, sending a full tablet report...
I20260709 12:27:20.687029 13824 ts_tablet_manager.cc:1434] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.113s	user 0.034s	sys 0.048s
I20260709 12:27:20.688573 13824 tablet_bootstrap.cc:492] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:20.690451 13825 tablet_bootstrap.cc:492] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:20.691253 13825 ts_tablet_manager.cc:1403] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.072s	user 0.025s	sys 0.004s
I20260709 12:27:20.692772 13827 tablet_bootstrap.cc:654] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:20.694778 13824 tablet_bootstrap.cc:654] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:20.694875 13825 raft_consensus.cc:359] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.696048 13825 raft_consensus.cc:385] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:20.696386 13825 raft_consensus.cc:740] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:20.697372 13825 consensus_queue.cc:260] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.705747 13809 heartbeater.cc:499] Master 127.29.201.254:44713 was elected leader, sending a full tablet report...
I20260709 12:27:20.706943 13825 ts_tablet_manager.cc:1434] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.015s	user 0.012s	sys 0.000s
I20260709 12:27:20.708369 13825 tablet_bootstrap.cc:492] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:20.716066 13828 raft_consensus.cc:493] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.716977 13828 raft_consensus.cc:515] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:20.726966 13828 leader_election.cc:290] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:20.741029 13824 tablet_bootstrap.cc:492] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:20.742020 13824 ts_tablet_manager.cc:1403] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.054s	user 0.031s	sys 0.017s
I20260709 12:27:20.750715 13827 tablet_bootstrap.cc:492] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:20.751544 13825 tablet_bootstrap.cc:654] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:20.752771 13824 raft_consensus.cc:359] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.753672 13824 raft_consensus.cc:385] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:20.754006 13824 raft_consensus.cc:740] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:20.754683 13827 ts_tablet_manager.cc:1403] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.084s	user 0.034s	sys 0.016s
I20260709 12:27:20.755055 13824 consensus_queue.cc:260] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.757931 13824 ts_tablet_manager.cc:1434] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.015s	user 0.012s	sys 0.000s
I20260709 12:27:20.759212 13824 tablet_bootstrap.cc:492] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:20.762072 13827 raft_consensus.cc:359] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.762897 13827 raft_consensus.cc:385] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:20.763204 13827 raft_consensus.cc:740] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:20.766443 13824 tablet_bootstrap.cc:654] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:20.768214 13827 consensus_queue.cc:260] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.768687 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
W20260709 12:27:20.770845 13600 leader_election.cc:343] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:20.773682 13784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:20.774552 13827 ts_tablet_manager.cc:1434] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.019s	user 0.007s	sys 0.003s
I20260709 12:27:20.773782 13735 heartbeater.cc:499] Master 127.29.201.254:44713 was elected leader, sending a full tablet report...
I20260709 12:27:20.775852 13827 tablet_bootstrap.cc:492] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:20.777976 13824 tablet_bootstrap.cc:492] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:20.778586 13824 ts_tablet_manager.cc:1403] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.020s	user 0.011s	sys 0.001s
W20260709 12:27:20.780149 13601 leader_election.cc:343] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:20.782151 13827 tablet_bootstrap.cc:654] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:20.781864 13824 raft_consensus.cc:359] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.783573 13824 raft_consensus.cc:385] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:20.783912 13824 raft_consensus.cc:740] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:20.784755 13824 consensus_queue.cc:260] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.785848 13825 tablet_bootstrap.cc:492] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:20.786535 13825 ts_tablet_manager.cc:1403] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.078s	user 0.004s	sys 0.008s
I20260709 12:27:20.787607 13824 ts_tablet_manager.cc:1434] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.009s	user 0.004s	sys 0.003s
I20260709 12:27:20.789222 13824 tablet_bootstrap.cc:492] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:20.782889 13601 leader_election.cc:304] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216; no voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:20.790984 13828 raft_consensus.cc:2749] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:20.795959 13825 raft_consensus.cc:359] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.796746 13825 raft_consensus.cc:385] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:20.797012 13825 raft_consensus.cc:740] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:20.797798 13825 consensus_queue.cc:260] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.800057 13825 ts_tablet_manager.cc:1434] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.000s
I20260709 12:27:20.801195 13825 tablet_bootstrap.cc:492] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:20.804833 13824 tablet_bootstrap.cc:654] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:20.806932 13825 tablet_bootstrap.cc:654] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:20.837462 13827 tablet_bootstrap.cc:492] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:20.838061 13827 ts_tablet_manager.cc:1403] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.062s	user 0.021s	sys 0.035s
I20260709 12:27:20.838164 13825 tablet_bootstrap.cc:492] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:20.838868 13824 tablet_bootstrap.cc:492] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:20.838884 13825 ts_tablet_manager.cc:1403] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.038s	user 0.012s	sys 0.016s
I20260709 12:27:20.839650 13824 ts_tablet_manager.cc:1403] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.051s	user 0.028s	sys 0.015s
I20260709 12:27:20.840759 13827 raft_consensus.cc:359] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.841462 13827 raft_consensus.cc:385] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:20.841744 13827 raft_consensus.cc:740] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:20.842446 13827 consensus_queue.cc:260] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.842552 13825 raft_consensus.cc:359] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.843482 13825 raft_consensus.cc:385] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:20.843842 13825 raft_consensus.cc:740] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:20.844731 13827 ts_tablet_manager.cc:1434] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.006s	user 0.008s	sys 0.000s
I20260709 12:27:20.844612 13824 raft_consensus.cc:359] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.844740 13825 consensus_queue.cc:260] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.846020 13827 tablet_bootstrap.cc:492] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:20.846258 13836 raft_consensus.cc:493] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.846866 13836 raft_consensus.cc:515] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.847692 13825 ts_tablet_manager.cc:1434] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:27:20.848881 13825 tablet_bootstrap.cc:492] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:20.851992 13836 leader_election.cc:290] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:20.853928 13827 tablet_bootstrap.cc:654] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:20.857178 13825 tablet_bootstrap.cc:654] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:20.845538 13824 raft_consensus.cc:385] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:20.857822 13824 raft_consensus.cc:740] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:20.859989 13824 consensus_queue.cc:260] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.864243 13832 raft_consensus.cc:493] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.865151 13832 raft_consensus.cc:515] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.868816 13824 ts_tablet_manager.cc:1434] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.029s	user 0.007s	sys 0.001s
I20260709 12:27:20.875546 13832 leader_election.cc:290] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:20.885315 13828 raft_consensus.cc:493] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.886044 13828 raft_consensus.cc:515] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
W20260709 12:27:20.889212 13581 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:20.891351 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33fa2959a148483984e00bb7fdcd0dcb" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
W20260709 12:27:20.893230 13600 leader_election.cc:343] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:20.878787 13824 tablet_bootstrap.cc:492] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:20.897353 13828 leader_election.cc:290] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:20.899325 13784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33fa2959a148483984e00bb7fdcd0dcb" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
W20260709 12:27:20.905117 13601 leader_election.cc:343] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:20.905982 13601 leader_election.cc:304] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216; no voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:20.913898 13828 raft_consensus.cc:2749] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:20.915263 13832 raft_consensus.cc:493] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.916085 13828 raft_consensus.cc:493] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.916298 13832 raft_consensus.cc:515] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.917008 13828 raft_consensus.cc:515] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.920215 13832 leader_election.cc:290] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:20.921831 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6145a89b623b4ec5b638ec5fe3c030cc" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
W20260709 12:27:20.924433 13600 leader_election.cc:343] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:20.925827 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6145a89b623b4ec5b638ec5fe3c030cc" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:20.929685 13828 leader_election.cc:290] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:20.932418 13784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6145a89b623b4ec5b638ec5fe3c030cc" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:20.933637 13784 raft_consensus.cc:2468] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 0.
I20260709 12:27:20.935443 13601 leader_election.cc:304] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:20.936847 13828 raft_consensus.cc:2804] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:20.937273 13828 raft_consensus.cc:493] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:20.937134 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a1f458064a14018ac4e716f1cda3b73" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:20.937897 13828 raft_consensus.cc:3060] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:20.938342 13710 raft_consensus.cc:2468] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 0.
W20260709 12:27:20.940385 13749 leader_election.cc:343] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:20.941991 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6145a89b623b4ec5b638ec5fe3c030cc" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:20.943387 13634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a1f458064a14018ac4e716f1cda3b73" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:20.949836 13828 raft_consensus.cc:515] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.943822 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a1f458064a14018ac4e716f1cda3b73" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:20.951179 13635 raft_consensus.cc:2393] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5e7dc958a3f45de8436922490e08ef8 in current term 1: Already voted for candidate 99369f74684341af8f9ba9ff6c225216 in this term.
I20260709 12:27:20.952417 13749 leader_election.cc:304] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
W20260709 12:27:20.954227 13675 leader_election.cc:343] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:20.957065 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6145a89b623b4ec5b638ec5fe3c030cc" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
W20260709 12:27:20.958536 13749 leader_election.cc:343] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:20.959102 13600 leader_election.cc:343] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:20.957634 13832 raft_consensus.cc:2804] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:20.959842 13832 raft_consensus.cc:493] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:20.960289 13832 raft_consensus.cc:3060] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:20.948926 13824 tablet_bootstrap.cc:654] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:20.964222 13828 leader_election.cc:290] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:20.960013 13749 leader_election.cc:304] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8; no voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:20.965684 13784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6145a89b623b4ec5b638ec5fe3c030cc" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:20.966883 13825 tablet_bootstrap.cc:492] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:20.966905 13784 raft_consensus.cc:3060] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:20.967847 13825 ts_tablet_manager.cc:1403] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.119s	user 0.038s	sys 0.056s
I20260709 12:27:20.968727 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a1f458064a14018ac4e716f1cda3b73" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:20.972687 13827 tablet_bootstrap.cc:492] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:20.972244 13825 raft_consensus.cc:359] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.973402 13825 raft_consensus.cc:385] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:20.973541 13827 ts_tablet_manager.cc:1403] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.128s	user 0.038s	sys 0.057s
I20260709 12:27:20.973846 13825 raft_consensus.cc:740] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:20.975456 13825 consensus_queue.cc:260] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.977034 13784 raft_consensus.cc:2468] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 1.
I20260709 12:27:20.977623 13827 raft_consensus.cc:359] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.978528 13827 raft_consensus.cc:385] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:20.978953 13827 raft_consensus.cc:740] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:20.978758 13601 leader_election.cc:304] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:20.980158 13828 raft_consensus.cc:2804] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:20.979970 13827 consensus_queue.cc:260] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.983174 13827 ts_tablet_manager.cc:1434] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.002s
I20260709 12:27:20.985445 13827 tablet_bootstrap.cc:492] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:20.992786 13828 raft_consensus.cc:697] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Becoming Leader. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:20.994043 13827 tablet_bootstrap.cc:654] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:20.994560 13836 raft_consensus.cc:493] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:20.995177 13836 raft_consensus.cc:515] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:20.997947 13836 leader_election.cc:290] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.000638 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81f900f101f543fa8b3cfea0fb81f776" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
W20260709 12:27:21.003328 13675 leader_election.cc:343] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:21.003558 13782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81f900f101f543fa8b3cfea0fb81f776" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:21.004482 13782 raft_consensus.cc:2468] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 0.
I20260709 12:27:20.994036 13828 consensus_queue.cc:237] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.007275 13676 leader_election.cc:304] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 99369f74684341af8f9ba9ff6c225216
I20260709 12:27:21.009042 13836 raft_consensus.cc:2804] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:21.009591 13836 raft_consensus.cc:493] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.010023 13836 raft_consensus.cc:3060] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:21.013132 13749 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.193:39059, user_credentials={real_user=slave}} blocked reactor thread for 53221.7us
I20260709 12:27:21.013453 13849 raft_consensus.cc:2749] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:21.017143 13832 raft_consensus.cc:515] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.018464 13825 ts_tablet_manager.cc:1434] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.050s	user 0.016s	sys 0.010s
I20260709 12:27:21.019095 13783 raft_consensus.cc:2393] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f3189d868dea45edb84dd2d2aa69c3e9 in current term 1: Already voted for candidate a5e7dc958a3f45de8436922490e08ef8 in this term.
I20260709 12:27:21.021989 13676 leader_election.cc:304] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3189d868dea45edb84dd2d2aa69c3e9; no voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:21.025470 13825 tablet_bootstrap.cc:492] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:21.027010 13836 raft_consensus.cc:515] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.030946 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81f900f101f543fa8b3cfea0fb81f776" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216"
W20260709 12:27:21.035570 13675 leader_election.cc:343] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:21.037674 13827 tablet_bootstrap.cc:492] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:21.040262 13827 ts_tablet_manager.cc:1403] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.055s	user 0.029s	sys 0.011s
I20260709 12:27:21.041244 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a1f458064a14018ac4e716f1cda3b73" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:21.042022 13710 raft_consensus.cc:3060] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.044376 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a1f458064a14018ac4e716f1cda3b73" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:21.044095 13827 raft_consensus.cc:359] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.045094 13827 raft_consensus.cc:385] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.045467 13827 raft_consensus.cc:740] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
W20260709 12:27:21.046329 13749 leader_election.cc:343] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:21.046696 13836 leader_election.cc:290] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.046409 13827 consensus_queue.cc:260] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.048429 13533 catalog_manager.cc:5697] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99369f74684341af8f9ba9ff6c225216 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "99369f74684341af8f9ba9ff6c225216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:21.049396 13827 ts_tablet_manager.cc:1434] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:27:21.039440 13832 leader_election.cc:290] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:21.050398 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81f900f101f543fa8b3cfea0fb81f776" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:21.039479 13824 tablet_bootstrap.cc:492] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:21.051165 13783 raft_consensus.cc:3060] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.034418 13825 tablet_bootstrap.cc:654] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.052093 13824 ts_tablet_manager.cc:1403] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.174s	user 0.029s	sys 0.018s
I20260709 12:27:21.054989 13710 raft_consensus.cc:2468] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 1.
I20260709 12:27:21.056087 13824 raft_consensus.cc:359] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.056756 13749 leader_election.cc:304] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 99369f74684341af8f9ba9ff6c225216
I20260709 12:27:21.057111 13824 raft_consensus.cc:385] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.057593 13824 raft_consensus.cc:740] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.058367 13832 raft_consensus.cc:2804] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:21.059047 13832 raft_consensus.cc:697] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Becoming Leader. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:21.058559 13824 consensus_queue.cc:260] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.060051 13832 consensus_queue.cc:237] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.061388 13783 raft_consensus.cc:2468] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 1.
I20260709 12:27:21.062036 13824 ts_tablet_manager.cc:1434] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.000s
I20260709 12:27:21.063433 13824 tablet_bootstrap.cc:492] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:21.063345 13676 leader_election.cc:304] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 99369f74684341af8f9ba9ff6c225216
I20260709 12:27:21.068933 13827 tablet_bootstrap.cc:492] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:21.075274 13854 raft_consensus.cc:2804] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:21.075552 13824 tablet_bootstrap.cc:654] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.076165 13854 raft_consensus.cc:697] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 LEADER]: Becoming Leader. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:21.077401 13854 consensus_queue.cc:237] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.082792 13836 raft_consensus.cc:2749] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:21.082103 13531 catalog_manager.cc:5697] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "a5e7dc958a3f45de8436922490e08ef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:21.087651 13827 tablet_bootstrap.cc:654] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.096028 13825 tablet_bootstrap.cc:492] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:21.096688 13825 ts_tablet_manager.cc:1403] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.071s	user 0.028s	sys 0.002s
I20260709 12:27:21.098281 13824 tablet_bootstrap.cc:492] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:21.086400 13836 raft_consensus.cc:493] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.099109 13824 ts_tablet_manager.cc:1403] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.036s	user 0.019s	sys 0.004s
I20260709 12:27:21.099385 13836 raft_consensus.cc:515] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.100428 13825 raft_consensus.cc:359] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:21.101223 13825 raft_consensus.cc:385] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.102555 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f25524761a4060871c3cc27c79f9df" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:21.102754 13824 raft_consensus.cc:359] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.103143 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f25524761a4060871c3cc27c79f9df" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:21.103653 13824 raft_consensus.cc:385] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.104022 13824 raft_consensus.cc:740] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.101744 13836 leader_election.cc:290] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.101589 13825 raft_consensus.cc:740] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
W20260709 12:27:21.105527 13675 leader_election.cc:343] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:21.105566 13824 consensus_queue.cc:260] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.106684 13825 consensus_queue.cc:260] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
W20260709 12:27:21.109041 13676 leader_election.cc:343] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:21.110922 13824 ts_tablet_manager.cc:1434] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.009s	user 0.003s	sys 0.003s
I20260709 12:27:21.113034 13825 ts_tablet_manager.cc:1434] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 12:27:21.113524 13824 tablet_bootstrap.cc:492] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:21.114221 13825 tablet_bootstrap.cc:492] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:21.119813 13676 leader_election.cc:304] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3189d868dea45edb84dd2d2aa69c3e9; no voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:21.120908 13854 raft_consensus.cc:2749] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:21.123010 13824 tablet_bootstrap.cc:654] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.123057 13825 tablet_bootstrap.cc:654] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.124809 13827 tablet_bootstrap.cc:492] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:21.125515 13827 ts_tablet_manager.cc:1403] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.057s	user 0.016s	sys 0.000s
I20260709 12:27:21.130043 13827 raft_consensus.cc:359] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:21.131006 13827 raft_consensus.cc:385] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.131403 13827 raft_consensus.cc:740] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.132423 13827 consensus_queue.cc:260] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:21.133267 13825 tablet_bootstrap.cc:492] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:21.133904 13825 ts_tablet_manager.cc:1403] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.020s	user 0.007s	sys 0.008s
I20260709 12:27:21.134609 13828 raft_consensus.cc:493] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.135214 13827 ts_tablet_manager.cc:1434] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.000s
I20260709 12:27:21.135545 13828 raft_consensus.cc:515] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.136605 13827 tablet_bootstrap.cc:492] T 6145a89b623b4ec5b638ec5fe3c030cc P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:21.137527 13825 raft_consensus.cc:359] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.138370 13825 raft_consensus.cc:385] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.138737 13825 raft_consensus.cc:740] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.139847 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a1f458064a14018ac4e716f1cda3b73" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:21.140626 13710 raft_consensus.cc:2393] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99369f74684341af8f9ba9ff6c225216 in current term 1: Already voted for candidate a5e7dc958a3f45de8436922490e08ef8 in this term.
I20260709 12:27:21.141861 13851 raft_consensus.cc:493] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.142421 13851 raft_consensus.cc:515] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.144065 13825 consensus_queue.cc:260] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.144577 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a1f458064a14018ac4e716f1cda3b73" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:21.144906 13851 leader_election.cc:290] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.145169 13827 tablet_bootstrap.cc:654] T 6145a89b623b4ec5b638ec5fe3c030cc P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.146605 13824 tablet_bootstrap.cc:492] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:21.147202 13824 ts_tablet_manager.cc:1403] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.034s	user 0.008s	sys 0.019s
I20260709 12:27:21.148051 13825 ts_tablet_manager.cc:1434] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.014s	user 0.009s	sys 0.001s
I20260709 12:27:21.148444 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5633dbff576b44b8914b2cc23f91de93" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:21.149156 13783 raft_consensus.cc:2468] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 0.
I20260709 12:27:21.149454 13825 tablet_bootstrap.cc:492] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:21.150609 13601 leader_election.cc:304] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216; no voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:21.150466 13824 raft_consensus.cc:359] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.151328 13824 raft_consensus.cc:385] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.151640 13824 raft_consensus.cc:740] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.151976 13601 leader_election.cc:304] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: 
I20260709 12:27:21.152388 13824 consensus_queue.cc:260] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.153326 13851 raft_consensus.cc:3060] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.159261 13825 tablet_bootstrap.cc:654] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.164994 13827 tablet_bootstrap.cc:492] T 6145a89b623b4ec5b638ec5fe3c030cc P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:21.159247 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5633dbff576b44b8914b2cc23f91de93" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:21.165964 13827 ts_tablet_manager.cc:1403] T 6145a89b623b4ec5b638ec5fe3c030cc P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.030s	user 0.017s	sys 0.003s
I20260709 12:27:21.166929 13828 leader_election.cc:290] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.168385 13828 raft_consensus.cc:2804] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:21.169224 13828 raft_consensus.cc:493] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.169896 13828 raft_consensus.cc:3060] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.166366 13710 raft_consensus.cc:2468] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 0.
I20260709 12:27:21.171257 13827 raft_consensus.cc:359] T 6145a89b623b4ec5b638ec5fe3c030cc P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.172261 13827 raft_consensus.cc:385] T 6145a89b623b4ec5b638ec5fe3c030cc P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.172605 13827 raft_consensus.cc:740] T 6145a89b623b4ec5b638ec5fe3c030cc P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.173076 13825 tablet_bootstrap.cc:492] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:21.173745 13825 ts_tablet_manager.cc:1403] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.025s	user 0.017s	sys 0.004s
I20260709 12:27:21.173863 13827 consensus_queue.cc:260] T 6145a89b623b4ec5b638ec5fe3c030cc P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.177093 13827 ts_tablet_manager.cc:1434] T 6145a89b623b4ec5b638ec5fe3c030cc P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.011s	user 0.005s	sys 0.006s
I20260709 12:27:21.177728 13825 raft_consensus.cc:359] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.178431 13827 tablet_bootstrap.cc:492] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:21.178695 13825 raft_consensus.cc:385] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.179086 13825 raft_consensus.cc:740] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.180307 13825 consensus_queue.cc:260] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.183450 13825 ts_tablet_manager.cc:1434] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.002s
I20260709 12:27:21.184747 13825 tablet_bootstrap.cc:492] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:21.187100 13827 tablet_bootstrap.cc:654] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.197891 13851 raft_consensus.cc:2749] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:21.200817 13825 tablet_bootstrap.cc:654] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.204643 13532 catalog_manager.cc:5697] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
W20260709 12:27:21.198037 13601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:42677, user_credentials={real_user=slave}} blocked reactor thread for 46134.3us
I20260709 12:27:21.208374 13824 ts_tablet_manager.cc:1434] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.061s	user 0.027s	sys 0.032s
I20260709 12:27:21.209738 13824 tablet_bootstrap.cc:492] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:21.209561 13828 raft_consensus.cc:515] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.213243 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5633dbff576b44b8914b2cc23f91de93" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:21.214001 13710 raft_consensus.cc:3060] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.216466 13825 tablet_bootstrap.cc:492] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:21.217077 13825 ts_tablet_manager.cc:1403] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.033s	user 0.008s	sys 0.012s
I20260709 12:27:21.218227 13824 tablet_bootstrap.cc:654] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.219995 13825 raft_consensus.cc:359] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.220754 13825 raft_consensus.cc:385] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.221029 13825 raft_consensus.cc:740] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.221751 13825 consensus_queue.cc:260] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.224255 13825 ts_tablet_manager.cc:1434] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:27:21.224961 13828 leader_election.cc:290] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.225283 13825 tablet_bootstrap.cc:492] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:21.227604 13827 tablet_bootstrap.cc:492] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:21.228206 13827 ts_tablet_manager.cc:1403] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.050s	user 0.010s	sys 0.013s
I20260709 12:27:21.228255 13824 tablet_bootstrap.cc:492] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:21.228878 13824 ts_tablet_manager.cc:1403] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.019s	user 0.004s	sys 0.012s
I20260709 12:27:21.231807 13827 raft_consensus.cc:359] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.231962 13824 raft_consensus.cc:359] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.232718 13827 raft_consensus.cc:385] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.232753 13824 raft_consensus.cc:385] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.233397 13824 raft_consensus.cc:740] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.233724 13825 tablet_bootstrap.cc:654] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.233345 13827 raft_consensus.cc:740] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.234350 13824 consensus_queue.cc:260] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.235466 13827 consensus_queue.cc:260] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.236716 13824 ts_tablet_manager.cc:1434] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:27:21.237711 13824 tablet_bootstrap.cc:492] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:21.237998 13827 ts_tablet_manager.cc:1434] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.009s	user 0.008s	sys 0.001s
I20260709 12:27:21.239113 13827 tablet_bootstrap.cc:492] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:21.245231 13824 tablet_bootstrap.cc:654] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.245764 13827 tablet_bootstrap.cc:654] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.246160 13825 tablet_bootstrap.cc:492] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:21.246789 13825 ts_tablet_manager.cc:1403] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.022s	user 0.013s	sys 0.006s
I20260709 12:27:21.250135 13825 raft_consensus.cc:359] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.250896 13825 raft_consensus.cc:385] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.251178 13825 raft_consensus.cc:740] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.251960 13825 consensus_queue.cc:260] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.253911 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5633dbff576b44b8914b2cc23f91de93" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:21.254484 13783 raft_consensus.cc:3060] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.256652 13825 ts_tablet_manager.cc:1434] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:27:21.260566 13827 tablet_bootstrap.cc:492] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:21.261427 13827 ts_tablet_manager.cc:1403] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.023s	user 0.006s	sys 0.012s
I20260709 12:27:21.264680 13710 raft_consensus.cc:2468] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 1.
I20260709 12:27:21.268291 13824 tablet_bootstrap.cc:492] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:21.268872 13824 ts_tablet_manager.cc:1403] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.031s	user 0.020s	sys 0.008s
I20260709 12:27:21.269743 13783 raft_consensus.cc:2468] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 1.
I20260709 12:27:21.271736 13600 leader_election.cc:304] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:21.271754 13824 raft_consensus.cc:359] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.272656 13824 raft_consensus.cc:385] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.272939 13824 raft_consensus.cc:740] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.273770 13824 consensus_queue.cc:260] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.274343 13827 raft_consensus.cc:359] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.274829 13828 raft_consensus.cc:2804] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:21.275074 13827 raft_consensus.cc:385] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.275465 13828 raft_consensus.cc:697] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Becoming Leader. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:21.275928 13824 ts_tablet_manager.cc:1434] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:27:21.276448 13828 consensus_queue.cc:237] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.275477 13827 raft_consensus.cc:740] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.278270 13827 consensus_queue.cc:260] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.279412 13832 raft_consensus.cc:493] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.279940 13832 raft_consensus.cc:515] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.280418 13827 ts_tablet_manager.cc:1434] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.019s	user 0.010s	sys 0.000s
I20260709 12:27:21.281635 13827 tablet_bootstrap.cc:492] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:21.282667 13832 leader_election.cc:290] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:21.286726 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
W20260709 12:27:21.288383 13749 leader_election.cc:343] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:21.288507 13827 tablet_bootstrap.cc:654] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:21.286741 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:21.290012 13635 raft_consensus.cc:2468] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 0.
I20260709 12:27:21.291375 13749 leader_election.cc:304] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:21.292704 13832 raft_consensus.cc:2804] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:21.293094 13832 raft_consensus.cc:493] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.293433 13832 raft_consensus.cc:3060] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.300642 13832 raft_consensus.cc:515] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.302202 13827 tablet_bootstrap.cc:492] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:21.302891 13827 ts_tablet_manager.cc:1403] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.021s	user 0.019s	sys 0.000s
I20260709 12:27:21.303289 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:21.303973 13635 raft_consensus.cc:3060] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.304195 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:21.303809 13533 catalog_manager.cc:5697] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99369f74684341af8f9ba9ff6c225216 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "99369f74684341af8f9ba9ff6c225216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:21.305923 13827 raft_consensus.cc:359] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.306666 13827 raft_consensus.cc:385] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:21.307087 13827 raft_consensus.cc:740] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:21.307950 13827 consensus_queue.cc:260] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.310417 13827 ts_tablet_manager.cc:1434] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
W20260709 12:27:21.312907 13749 leader_election.cc:343] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:21.313620 13832 leader_election.cc:290] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:21.320281 13635 raft_consensus.cc:2468] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 1.
I20260709 12:27:21.322018 13749 leader_election.cc:304] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:21.322954 13832 raft_consensus.cc:2804] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:21.323726 13832 raft_consensus.cc:697] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Becoming Leader. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:21.324577 13832 consensus_queue.cc:237] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.334825 13532 catalog_manager.cc:5697] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "a5e7dc958a3f45de8436922490e08ef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:21.340883 13828 raft_consensus.cc:493] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.341536 13828 raft_consensus.cc:515] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.344135 13828 leader_election.cc:290] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.344652 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f25524761a4060871c3cc27c79f9df" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:21.344887 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f25524761a4060871c3cc27c79f9df" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:21.345387 13710 raft_consensus.cc:2468] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 0.
I20260709 12:27:21.345520 13783 raft_consensus.cc:2468] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 0.
I20260709 12:27:21.346418 13600 leader_election.cc:304] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:21.347277 13828 raft_consensus.cc:2804] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:21.347671 13828 raft_consensus.cc:493] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.347950 13828 raft_consensus.cc:3060] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.356163 13828 raft_consensus.cc:515] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.358323 13828 leader_election.cc:290] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.359328 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f25524761a4060871c3cc27c79f9df" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:21.360167 13710 raft_consensus.cc:3060] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.364111 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f25524761a4060871c3cc27c79f9df" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:21.364929 13783 raft_consensus.cc:3060] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.372838 13783 raft_consensus.cc:2468] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 1.
I20260709 12:27:21.374053 13601 leader_election.cc:304] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: 
I20260709 12:27:21.374161 13710 raft_consensus.cc:2468] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 1.
I20260709 12:27:21.375056 13828 raft_consensus.cc:2804] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:21.375574 13828 raft_consensus.cc:697] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Becoming Leader. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:21.376443 13828 consensus_queue.cc:237] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.384712 13532 catalog_manager.cc:5697] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99369f74684341af8f9ba9ff6c225216 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "99369f74684341af8f9ba9ff6c225216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:21.386572 13854 raft_consensus.cc:493] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.387306 13710 raft_consensus.cc:3060] T 6145a89b623b4ec5b638ec5fe3c030cc P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.387171 13854 raft_consensus.cc:515] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.388003 13828 raft_consensus.cc:493] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.388506 13828 raft_consensus.cc:515] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.389422 13854 leader_election.cc:290] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.391819 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f47887de35d404f83ab0edf3cabb706" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:21.392174 13784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f47887de35d404f83ab0edf3cabb706" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:21.392541 13635 raft_consensus.cc:2468] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 0.
I20260709 12:27:21.392817 13784 raft_consensus.cc:2468] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 0.
I20260709 12:27:21.393924 13675 leader_election.cc:304] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:21.394840 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f47887de35d404f83ab0edf3cabb706" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:21.395259 13828 leader_election.cc:290] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.395587 13783 raft_consensus.cc:2468] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 0.
I20260709 12:27:21.397102 13601 leader_election.cc:304] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: 
I20260709 12:27:21.397114 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f47887de35d404f83ab0edf3cabb706" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:21.397661 13854 raft_consensus.cc:2804] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:21.397576 13828 consensus_queue.cc:1048] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:21.398164 13868 raft_consensus.cc:2804] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:21.398236 13854 raft_consensus.cc:493] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.398736 13868 raft_consensus.cc:493] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.398792 13854 raft_consensus.cc:3060] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.399205 13868 raft_consensus.cc:3060] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.407403 13868 raft_consensus.cc:515] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.407207 13854 raft_consensus.cc:515] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.409749 13710 raft_consensus.cc:2393] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99369f74684341af8f9ba9ff6c225216 in current term 1: Already voted for candidate f3189d868dea45edb84dd2d2aa69c3e9 in this term.
I20260709 12:27:21.411335 13868 leader_election.cc:290] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.412565 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f47887de35d404f83ab0edf3cabb706" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:21.413275 13783 raft_consensus.cc:3060] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.415608 13784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f47887de35d404f83ab0edf3cabb706" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:21.417212 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f47887de35d404f83ab0edf3cabb706" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:21.418150 13708 raft_consensus.cc:2393] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 99369f74684341af8f9ba9ff6c225216 in current term 1: Already voted for candidate f3189d868dea45edb84dd2d2aa69c3e9 in this term.
I20260709 12:27:21.414971 13854 leader_election.cc:290] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.419664 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f47887de35d404f83ab0edf3cabb706" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:21.420560 13635 raft_consensus.cc:2393] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f3189d868dea45edb84dd2d2aa69c3e9 in current term 1: Already voted for candidate 99369f74684341af8f9ba9ff6c225216 in this term.
I20260709 12:27:21.422993 13676 leader_election.cc:304] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3189d868dea45edb84dd2d2aa69c3e9; no voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:21.425215 13868 consensus_queue.cc:1048] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:21.427016 13854 raft_consensus.cc:2749] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260709 12:27:21.427834 13783 raft_consensus.cc:2468] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 1.
I20260709 12:27:21.429183 13601 leader_election.cc:304] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:21.430173 13868 raft_consensus.cc:2804] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:21.430951 13868 raft_consensus.cc:697] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Becoming Leader. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:21.432099 13868 consensus_queue.cc:237] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.441231 13533 catalog_manager.cc:5697] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99369f74684341af8f9ba9ff6c225216 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "99369f74684341af8f9ba9ff6c225216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:21.447403 13832 raft_consensus.cc:493] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.448052 13832 raft_consensus.cc:515] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.450492 13832 leader_election.cc:290] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:21.451392 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33fa2959a148483984e00bb7fdcd0dcb" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:21.452162 13635 raft_consensus.cc:2468] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 0.
I20260709 12:27:21.453253 13749 leader_election.cc:304] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: 
I20260709 12:27:21.453274 13709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33fa2959a148483984e00bb7fdcd0dcb" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:21.453702 13832 raft_consensus.cc:493] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.454201 13709 raft_consensus.cc:2468] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 0.
I20260709 12:27:21.454242 13849 raft_consensus.cc:2804] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:21.454342 13832 raft_consensus.cc:515] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:21.454954 13849 raft_consensus.cc:493] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.455848 13849 raft_consensus.cc:3060] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.456528 13832 leader_election.cc:290] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059)
I20260709 12:27:21.457413 13709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:21.457759 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:21.458171 13709 raft_consensus.cc:2468] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 0.
I20260709 12:27:21.458443 13635 raft_consensus.cc:2468] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 0.
I20260709 12:27:21.459482 13749 leader_election.cc:304] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:21.460212 13832 raft_consensus.cc:2804] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:21.460536 13832 raft_consensus.cc:493] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:21.460830 13832 raft_consensus.cc:3060] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.463828 13849 raft_consensus.cc:515] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.465864 13849 leader_election.cc:290] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:21.466997 13709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33fa2959a148483984e00bb7fdcd0dcb" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:21.467614 13709 raft_consensus.cc:3060] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.467338 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33fa2959a148483984e00bb7fdcd0dcb" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:21.468148 13635 raft_consensus.cc:3060] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.468019 13832 raft_consensus.cc:515] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:21.470276 13832 leader_election.cc:290] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059)
I20260709 12:27:21.472009 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:21.472754 13708 raft_consensus.cc:3060] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.472872 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:21.473639 13633 raft_consensus.cc:3060] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.476027 13709 raft_consensus.cc:2468] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 1.
I20260709 12:27:21.476292 13635 raft_consensus.cc:2468] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 1.
I20260709 12:27:21.479478 13749 leader_election.cc:304] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: 
I20260709 12:27:21.480291 13832 raft_consensus.cc:2804] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:21.481177 13832 raft_consensus.cc:697] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Becoming Leader. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:21.481483 13708 raft_consensus.cc:2468] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 1.
I20260709 12:27:21.482026 13633 raft_consensus.cc:2468] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 1.
I20260709 12:27:21.482115 13832 consensus_queue.cc:237] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.482939 13749 leader_election.cc:304] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:21.483846 13849 raft_consensus.cc:2804] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:21.484432 13849 raft_consensus.cc:697] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Becoming Leader. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:21.485246 13849 consensus_queue.cc:237] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:21.491071 13832 consensus_queue.cc:1048] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:21.492163 13533 catalog_manager.cc:5697] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "a5e7dc958a3f45de8436922490e08ef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:21.504451 13850 consensus_queue.cc:1048] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:21.515142 13531 catalog_manager.cc:5697] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "a5e7dc958a3f45de8436922490e08ef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:21.526991 13854 consensus_queue.cc:1048] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:21.538479 13633 raft_consensus.cc:3060] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.547570 13854 consensus_queue.cc:1048] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:21.564291 13854 raft_consensus.cc:493] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:21.564909 13854 raft_consensus.cc:515] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.566872 13854 leader_election.cc:290] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.567584 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:21.567955 13784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:21.568555 13633 raft_consensus.cc:2393] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f3189d868dea45edb84dd2d2aa69c3e9 in current term 1: Already voted for candidate a5e7dc958a3f45de8436922490e08ef8 in this term.
I20260709 12:27:21.569581 13675 leader_election.cc:304] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3189d868dea45edb84dd2d2aa69c3e9; no voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:21.570263 13854 raft_consensus.cc:3060] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:21.575897 13854 raft_consensus.cc:2749] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:21.598783 13784 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef"
dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
mode: GRACEFUL
new_leader_uuid: "99369f74684341af8f9ba9ff6c225216"
 from {username='slave'} at 127.0.0.1:47416
I20260709 12:27:21.599402 13784 raft_consensus.cc:606] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Received request to transfer leadership to TS 99369f74684341af8f9ba9ff6c225216
I20260709 12:27:21.614032 13708 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "81f900f101f543fa8b3cfea0fb81f776"
dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
mode: GRACEFUL
new_leader_uuid: "99369f74684341af8f9ba9ff6c225216"
 from {username='slave'} at 127.0.0.1:52112
I20260709 12:27:21.614822 13708 raft_consensus.cc:606] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 LEADER]: Received request to transfer leadership to TS 99369f74684341af8f9ba9ff6c225216
I20260709 12:27:21.626981 13784 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0a1f458064a14018ac4e716f1cda3b73"
dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
mode: GRACEFUL
new_leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:47416
I20260709 12:27:21.627526 13784 raft_consensus.cc:606] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Received request to transfer leadership to TS f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:21.643388 13633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2f47887de35d404f83ab0edf3cabb706"
dest_uuid: "99369f74684341af8f9ba9ff6c225216"
mode: GRACEFUL
new_leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:41114
I20260709 12:27:21.644053 13633 raft_consensus.cc:606] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Received request to transfer leadership to TS f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:21.650569 13633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f4f25524761a4060871c3cc27c79f9df"
dest_uuid: "99369f74684341af8f9ba9ff6c225216"
mode: GRACEFUL
new_leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:41114
I20260709 12:27:21.651274 13633 raft_consensus.cc:606] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Received request to transfer leadership to TS f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:21.657326 13784 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a"
dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
mode: GRACEFUL
new_leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:47416
I20260709 12:27:21.657958 13784 raft_consensus.cc:606] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Received request to transfer leadership to TS f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:21.768187 13868 consensus_queue.cc:1048] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:21.780687 13868 consensus_queue.cc:1048] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:21.784653 13868 consensus_queue.cc:1048] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:21.790591 13828 raft_consensus.cc:993] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216: : Instructing follower f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:21.791165 13828 raft_consensus.cc:1081] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Signalling peer f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:21.793102 13708 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f4f25524761a4060871c3cc27c79f9df"
dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:52100
I20260709 12:27:21.793743 13708 raft_consensus.cc:493] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:21.794133 13708 raft_consensus.cc:3060] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.802119 13868 consensus_queue.cc:1048] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:21.808197 13708 raft_consensus.cc:515] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.811997 13708 leader_election.cc:290] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.811909 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f25524761a4060871c3cc27c79f9df" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:21.812716 13633 raft_consensus.cc:3055] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:21.813083 13633 raft_consensus.cc:740] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:21.813907 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f4f25524761a4060871c3cc27c79f9df" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:21.813951 13633 consensus_queue.cc:260] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.814700 13783 raft_consensus.cc:3060] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.815652 13633 raft_consensus.cc:3060] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.822849 13633 raft_consensus.cc:2468] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 2.
I20260709 12:27:21.823778 13783 raft_consensus.cc:2468] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 2.
I20260709 12:27:21.824088 13675 leader_election.cc:304] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:21.825834 13854 raft_consensus.cc:2804] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:21.826442 13854 raft_consensus.cc:697] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Becoming Leader. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:21.827426 13854 consensus_queue.cc:237] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.828811 13849 consensus_queue.cc:1048] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:21.835763 13531 catalog_manager.cc:5697] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 reported cstate change: term changed from 1 to 2, leader changed from 99369f74684341af8f9ba9ff6c225216 (127.29.201.193) to f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:21.842479 13832 consensus_queue.cc:1048] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:21.851095 13850 raft_consensus.cc:993] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8: : Instructing follower f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:21.851667 13832 raft_consensus.cc:1081] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Signalling peer f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:21.853773 13709 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a"
dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:52108
I20260709 12:27:21.855150 13709 raft_consensus.cc:493] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:21.855525 13709 raft_consensus.cc:3060] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.862759 13709 raft_consensus.cc:515] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.865684 13709 leader_election.cc:290] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.866493 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:21.867300 13633 raft_consensus.cc:3060] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.867213 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:21.868064 13783 raft_consensus.cc:3055] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:21.868438 13783 raft_consensus.cc:740] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:21.869413 13783 consensus_queue.cc:260] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.870936 13783 raft_consensus.cc:3060] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.875895 13633 raft_consensus.cc:2468] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 2.
I20260709 12:27:21.877498 13675 leader_election.cc:304] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:21.878342 13854 raft_consensus.cc:2804] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:21.878794 13783 raft_consensus.cc:2468] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 2.
I20260709 12:27:21.878962 13854 raft_consensus.cc:697] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Becoming Leader. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:21.880110 13854 consensus_queue.cc:237] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.887619 13531 catalog_manager.cc:5697] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 reported cstate change: term changed from 1 to 2, leader changed from a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195) to f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:21.903306 13832 consensus_queue.cc:1048] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:21.916290 13904 consensus_queue.cc:1048] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:21.916523 13850 consensus_queue.cc:1048] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:21.926275 13832 raft_consensus.cc:993] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8: : Instructing follower f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:21.927562 13850 raft_consensus.cc:1081] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Signalling peer f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:21.931517 13868 consensus_queue.cc:1048] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:21.937513 13708 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0a1f458064a14018ac4e716f1cda3b73"
dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:52108
I20260709 12:27:21.938273 13708 raft_consensus.cc:493] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:21.938915 13708 raft_consensus.cc:3060] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.939327 13832 raft_consensus.cc:993] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8: : Instructing follower 99369f74684341af8f9ba9ff6c225216 to start an election
I20260709 12:27:21.939906 13832 raft_consensus.cc:1081] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Signalling peer 99369f74684341af8f9ba9ff6c225216 to start an election
I20260709 12:27:21.947470 13867 raft_consensus.cc:993] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216: : Instructing follower f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:21.948117 13867 raft_consensus.cc:1081] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Signalling peer f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:21.952119 13710 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2f47887de35d404f83ab0edf3cabb706"
dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:52100
I20260709 12:27:21.952903 13633 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef"
dest_uuid: "99369f74684341af8f9ba9ff6c225216"
 from {username='slave'} at 127.0.0.1:41108
I20260709 12:27:21.953650 13633 raft_consensus.cc:493] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:21.954094 13633 raft_consensus.cc:3060] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.955960 13867 consensus_queue.cc:1048] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:21.957288 13710 raft_consensus.cc:493] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:21.957917 13710 raft_consensus.cc:3060] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.965055 13708 raft_consensus.cc:515] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.967723 13708 leader_election.cc:290] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.968932 13784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a1f458064a14018ac4e716f1cda3b73" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:21.969659 13784 raft_consensus.cc:3055] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:21.970005 13784 raft_consensus.cc:740] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:21.970096 13849 consensus_queue.cc:1048] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:21.970881 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0a1f458064a14018ac4e716f1cda3b73" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:21.971576 13635 raft_consensus.cc:3060] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.971226 13784 consensus_queue.cc:260] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.971745 13633 raft_consensus.cc:515] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:21.972783 13784 raft_consensus.cc:3060] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.975008 13707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:21.975728 13707 raft_consensus.cc:3060] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:21.977545 13710 raft_consensus.cc:515] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.979907 13710 leader_election.cc:290] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:21.981642 13782 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f47887de35d404f83ab0edf3cabb706" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:21.982668 13635 raft_consensus.cc:2468] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 2.
I20260709 12:27:21.986183 13675 leader_election.cc:304] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:21.987931 13857 raft_consensus.cc:2804] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:21.988649 13857 raft_consensus.cc:697] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Becoming Leader. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:21.989887 13857 consensus_queue.cc:237] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:21.995990 13707 raft_consensus.cc:2468] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 2.
I20260709 12:27:21.998219 13633 leader_election.cc:290] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 2 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:22.001060 13600 leader_election.cc:304] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:22.002132 13868 raft_consensus.cc:2804] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:22.002691 13868 raft_consensus.cc:697] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Becoming Leader. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:22.003034 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f47887de35d404f83ab0edf3cabb706" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:22.003799 13635 raft_consensus.cc:3055] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:22.003568 13868 consensus_queue.cc:237] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:22.004217 13635 raft_consensus.cc:740] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:22.005165 13635 consensus_queue.cc:260] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:22.006083 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:22.006511 13635 raft_consensus.cc:3060] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Advancing to term 2
W20260709 12:27:22.006469 13867 consensus_queue.cc:1175] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "a5e7dc958a3f45de8436922490e08ef8" responder_term: 1 status { last_received { term: 1 index: 1 } last_committed_idx: 1 last_received_current_leader { term: 1 index: 1 } }
I20260709 12:27:22.009020 13836 raft_consensus.cc:993] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9: : Instructing follower 99369f74684341af8f9ba9ff6c225216 to start an election
I20260709 12:27:22.009544 13836 raft_consensus.cc:1081] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 LEADER]: Signalling peer 99369f74684341af8f9ba9ff6c225216 to start an election
I20260709 12:27:22.006870 13783 raft_consensus.cc:3055] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:22.011008 13784 raft_consensus.cc:2468] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 2.
I20260709 12:27:22.011265 13783 raft_consensus.cc:740] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:22.012315 13783 consensus_queue.cc:260] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:22.013294 13633 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "81f900f101f543fa8b3cfea0fb81f776"
dest_uuid: "99369f74684341af8f9ba9ff6c225216"
 from {username='slave'} at 127.0.0.1:41102
I20260709 12:27:22.014012 13633 raft_consensus.cc:493] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:22.014031 13783 raft_consensus.cc:3060] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.014447 13633 raft_consensus.cc:3060] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.015702 13635 raft_consensus.cc:2468] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 2.
I20260709 12:27:22.021445 13783 raft_consensus.cc:2468] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 2.
I20260709 12:27:22.069079 13675 leader_election.cc:304] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:22.083920 13854 raft_consensus.cc:2804] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:22.085215 13854 raft_consensus.cc:697] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Becoming Leader. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:22.087322 13854 consensus_queue.cc:237] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:22.104038 13633 raft_consensus.cc:515] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:22.105508 13531 catalog_manager.cc:5697] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 reported cstate change: term changed from 1 to 2, leader changed from 99369f74684341af8f9ba9ff6c225216 (127.29.201.193) to f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:22.108479 13633 leader_election.cc:290] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 2 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:22.108724 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81f900f101f543fa8b3cfea0fb81f776" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:22.109709 13708 raft_consensus.cc:3055] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:22.109555 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "81f900f101f543fa8b3cfea0fb81f776" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:22.110231 13708 raft_consensus.cc:740] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:22.111064 13783 raft_consensus.cc:3060] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.108757 13531 catalog_manager.cc:5697] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 reported cstate change: term changed from 1 to 2, leader changed from a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195) to f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:22.110536 13533 catalog_manager.cc:5697] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 reported cstate change: term changed from 1 to 2, leader changed from a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195) to 99369f74684341af8f9ba9ff6c225216 (127.29.201.193). New cstate: current_term: 2 leader_uuid: "99369f74684341af8f9ba9ff6c225216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:22.112355 13708 consensus_queue.cc:260] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:22.115476 13708 raft_consensus.cc:3060] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.126058 13783 raft_consensus.cc:2468] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 2.
I20260709 12:27:22.127972 13601 leader_election.cc:304] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: 
I20260709 12:27:22.128202 13708 raft_consensus.cc:2468] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 2.
I20260709 12:27:22.129406 13868 raft_consensus.cc:2804] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:22.130064 13868 raft_consensus.cc:697] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Becoming Leader. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:22.131480 13868 consensus_queue.cc:237] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:22.141160 13533 catalog_manager.cc:5697] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 reported cstate change: term changed from 1 to 2, leader changed from f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194) to 99369f74684341af8f9ba9ff6c225216 (127.29.201.193). New cstate: current_term: 2 leader_uuid: "99369f74684341af8f9ba9ff6c225216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:22.289145 13783 raft_consensus.cc:1275] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.290902 13854 consensus_queue.cc:1048] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:22.304081 13633 raft_consensus.cc:1275] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.305797 13854 consensus_queue.cc:1048] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:22.314215 13783 raft_consensus.cc:1275] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.315912 13854 consensus_queue.cc:1048] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:22.327579 13633 raft_consensus.cc:1275] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.329517 13857 consensus_queue.cc:1048] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:22.394905 13783 raft_consensus.cc:1275] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Refusing update from remote peer 99369f74684341af8f9ba9ff6c225216: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.396332 13868 consensus_queue.cc:1048] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:22.407284 13708 raft_consensus.cc:1275] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Refusing update from remote peer 99369f74684341af8f9ba9ff6c225216: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.409801 13828 consensus_queue.cc:1048] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:22.472522 13783 raft_consensus.cc:3060] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:22.482793 13783 raft_consensus.cc:1275] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.484686 13836 consensus_queue.cc:1048] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:22.494498 13633 raft_consensus.cc:1275] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.496554 13854 consensus_queue.cc:1048] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:22.549851 13633 raft_consensus.cc:1275] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.551436 13854 consensus_queue.cc:1048] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:22.562675 13783 raft_consensus.cc:1275] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.564684 13854 consensus_queue.cc:1048] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:22.659132 13783 raft_consensus.cc:1275] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Refusing update from remote peer 99369f74684341af8f9ba9ff6c225216: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.660827 13828 consensus_queue.cc:1048] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:22.672986 13708 raft_consensus.cc:1275] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Refusing update from remote peer 99369f74684341af8f9ba9ff6c225216: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:22.674717 13867 consensus_queue.cc:1048] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:24.694077 13531 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35484:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:27:24.696925 13531 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:27:24.754776 13624 tablet_service.cc:1511] Processing CreateTablet for tablet c0059d80d6d442499891fcc2bf42f899 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:24.756105 13695 tablet_service.cc:1511] Processing CreateTablet for tablet c0059d80d6d442499891fcc2bf42f899 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:24.756696 13624 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0059d80d6d442499891fcc2bf42f899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.757831 13695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0059d80d6d442499891fcc2bf42f899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.759285 13619 tablet_service.cc:1511] Processing CreateTablet for tablet 2a777f237f174c6fb0a469141cc8c426 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:24.760877 13619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a777f237f174c6fb0a469141cc8c426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.761302 13766 tablet_service.cc:1511] Processing CreateTablet for tablet c0059d80d6d442499891fcc2bf42f899 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:24.762861 13766 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0059d80d6d442499891fcc2bf42f899. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.765086 13769 tablet_service.cc:1511] Processing CreateTablet for tablet 2a777f237f174c6fb0a469141cc8c426 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:24.766880 13769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a777f237f174c6fb0a469141cc8c426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.767984 13692 tablet_service.cc:1511] Processing CreateTablet for tablet d6581db8dd724835827972f64e4bb115 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:24.768831 13618 tablet_service.cc:1511] Processing CreateTablet for tablet d6581db8dd724835827972f64e4bb115 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:24.769481 13692 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6581db8dd724835827972f64e4bb115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.770272 13618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6581db8dd724835827972f64e4bb115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.771441 13693 tablet_service.cc:1511] Processing CreateTablet for tablet 2a777f237f174c6fb0a469141cc8c426 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:24.772888 13693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a777f237f174c6fb0a469141cc8c426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.773979 13621 tablet_service.cc:1511] Processing CreateTablet for tablet b9551779678d40728ad7cf78f0551fa6 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:24.775892 13768 tablet_service.cc:1511] Processing CreateTablet for tablet d6581db8dd724835827972f64e4bb115 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:24.776969 13621 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9551779678d40728ad7cf78f0551fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.777370 13768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6581db8dd724835827972f64e4bb115. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.792392 13772 tablet_service.cc:1511] Processing CreateTablet for tablet b9551779678d40728ad7cf78f0551fa6 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:24.794021 13772 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9551779678d40728ad7cf78f0551fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.794819 13696 tablet_service.cc:1511] Processing CreateTablet for tablet b9551779678d40728ad7cf78f0551fa6 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:24.796382 13696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9551779678d40728ad7cf78f0551fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.799997 13697 tablet_service.cc:1511] Processing CreateTablet for tablet eec8f9faf1894326ba84678fd84d81f7 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:24.801587 13697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec8f9faf1894326ba84678fd84d81f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.804821 13620 tablet_service.cc:1511] Processing CreateTablet for tablet eec8f9faf1894326ba84678fd84d81f7 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:24.806366 13620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec8f9faf1894326ba84678fd84d81f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.810660 13774 tablet_service.cc:1511] Processing CreateTablet for tablet eec8f9faf1894326ba84678fd84d81f7 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:24.812214 13774 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eec8f9faf1894326ba84678fd84d81f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.819651 13773 tablet_service.cc:1511] Processing CreateTablet for tablet d656052161734f809877b26b2eaa3487 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:24.821285 13773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d656052161734f809877b26b2eaa3487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.832214 13617 tablet_service.cc:1511] Processing CreateTablet for tablet d656052161734f809877b26b2eaa3487 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:24.833909 13617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d656052161734f809877b26b2eaa3487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.837073 13623 tablet_service.cc:1511] Processing CreateTablet for tablet 04da14f36947466f88a8dbfaa58505f2 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:24.839972 13622 tablet_service.cc:1511] Processing CreateTablet for tablet 96138840b8c545ab90f1d0be4086de59 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:24.842644 13625 tablet_service.cc:1511] Processing CreateTablet for tablet a7582c4042164df48db37a656afff05d (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:24.845214 13700 tablet_service.cc:1511] Processing CreateTablet for tablet d656052161734f809877b26b2eaa3487 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:24.847671 13700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d656052161734f809877b26b2eaa3487. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.855835 13623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04da14f36947466f88a8dbfaa58505f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.858978 13625 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7582c4042164df48db37a656afff05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.864527 13622 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96138840b8c545ab90f1d0be4086de59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.868127 13771 tablet_service.cc:1511] Processing CreateTablet for tablet 96138840b8c545ab90f1d0be4086de59 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:24.880337 13771 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96138840b8c545ab90f1d0be4086de59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.866108 13770 tablet_service.cc:1511] Processing CreateTablet for tablet 04da14f36947466f88a8dbfaa58505f2 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:24.883128 13770 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04da14f36947466f88a8dbfaa58505f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.868036 13699 tablet_service.cc:1511] Processing CreateTablet for tablet 96138840b8c545ab90f1d0be4086de59 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:24.887590 13699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96138840b8c545ab90f1d0be4086de59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.869438 13767 tablet_service.cc:1511] Processing CreateTablet for tablet a7582c4042164df48db37a656afff05d (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:24.871323 13698 tablet_service.cc:1511] Processing CreateTablet for tablet 04da14f36947466f88a8dbfaa58505f2 (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:24.873509 13694 tablet_service.cc:1511] Processing CreateTablet for tablet a7582c4042164df48db37a656afff05d (DEFAULT_TABLE table=soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:24.891103 13694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7582c4042164df48db37a656afff05d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.899030 13698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04da14f36947466f88a8dbfaa58505f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:24.903514 13767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7582c4042164df48db37a656afff05d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260709 12:27:24.939551 13581 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:24.952229 13930 tablet_bootstrap.cc:492] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:24.998960 13931 tablet_bootstrap.cc:492] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:25.009236 13933 tablet_bootstrap.cc:492] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:25.016976 13930 tablet_bootstrap.cc:654] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.024943 13933 tablet_bootstrap.cc:654] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.038780 13931 tablet_bootstrap.cc:654] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.063166 13933 tablet_bootstrap.cc:492] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:25.063710 13930 tablet_bootstrap.cc:492] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:25.064484 13930 ts_tablet_manager.cc:1403] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.113s	user 0.007s	sys 0.025s
I20260709 12:27:25.064512 13931 tablet_bootstrap.cc:492] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:25.065106 13931 ts_tablet_manager.cc:1403] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.066s	user 0.017s	sys 0.004s
I20260709 12:27:25.063766 13933 ts_tablet_manager.cc:1403] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.055s	user 0.016s	sys 0.024s
I20260709 12:27:25.067852 13931 raft_consensus.cc:359] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.067732 13930 raft_consensus.cc:359] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.068655 13931 raft_consensus.cc:385] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.068745 13930 raft_consensus.cc:385] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.069015 13931 raft_consensus.cc:740] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.069218 13930 raft_consensus.cc:740] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.069952 13933 raft_consensus.cc:359] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.070041 13931 consensus_queue.cc:260] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.070114 13930 consensus_queue.cc:260] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.070938 13933 raft_consensus.cc:385] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.073290 13931 ts_tablet_manager.cc:1434] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:27:25.073295 13930 ts_tablet_manager.cc:1434] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:27:25.074702 13931 tablet_bootstrap.cc:492] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:25.074805 13930 tablet_bootstrap.cc:492] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:25.071323 13933 raft_consensus.cc:740] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.080379 13933 consensus_queue.cc:260] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.082733 13931 tablet_bootstrap.cc:654] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.083287 13933 ts_tablet_manager.cc:1434] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.016s	user 0.005s	sys 0.003s
I20260709 12:27:25.084506 13933 tablet_bootstrap.cc:492] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:25.086870 13930 tablet_bootstrap.cc:654] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.092273 13933 tablet_bootstrap.cc:654] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.121474 13904 raft_consensus.cc:493] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.122182 13904 raft_consensus.cc:515] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.126356 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a777f237f174c6fb0a469141cc8c426" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:25.130353 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a777f237f174c6fb0a469141cc8c426" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:25.131474 13708 raft_consensus.cc:2468] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 0.
I20260709 12:27:25.136515 13904 leader_election.cc:290] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
W20260709 12:27:25.134274 13749 leader_election.cc:343] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:25.141880 13749 leader_election.cc:304] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 99369f74684341af8f9ba9ff6c225216
I20260709 12:27:25.143967 13904 raft_consensus.cc:2804] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:25.145956 13904 raft_consensus.cc:493] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:25.162853 13904 raft_consensus.cc:3060] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.198712 13904 raft_consensus.cc:515] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.203625 13930 tablet_bootstrap.cc:492] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:25.204276 13930 ts_tablet_manager.cc:1403] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.130s	user 0.038s	sys 0.071s
I20260709 12:27:25.204836 13933 tablet_bootstrap.cc:492] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:25.205024 13931 tablet_bootstrap.cc:492] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:25.205296 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a777f237f174c6fb0a469141cc8c426" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:25.205485 13933 ts_tablet_manager.cc:1403] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.121s	user 0.052s	sys 0.065s
I20260709 12:27:25.205792 13931 ts_tablet_manager.cc:1403] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.131s	user 0.054s	sys 0.071s
W20260709 12:27:25.207006 13749 leader_election.cc:343] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:25.207631 13930 raft_consensus.cc:359] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.208417 13930 raft_consensus.cc:385] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.208730 13930 raft_consensus.cc:740] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.209333 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a777f237f174c6fb0a469141cc8c426" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:25.209213 13933 raft_consensus.cc:359] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.209488 13930 consensus_queue.cc:260] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.209635 13931 raft_consensus.cc:359] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.210120 13708 raft_consensus.cc:3060] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.210124 13933 raft_consensus.cc:385] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.210822 13931 raft_consensus.cc:385] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.210664 13904 leader_election.cc:290] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:25.215389 13931 raft_consensus.cc:740] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.210902 13933 raft_consensus.cc:740] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.216403 13931 consensus_queue.cc:260] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.216840 13933 consensus_queue.cc:260] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.212180 13930 ts_tablet_manager.cc:1434] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:27:25.219391 13931 ts_tablet_manager.cc:1434] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.013s	user 0.006s	sys 0.002s
I20260709 12:27:25.219511 13933 ts_tablet_manager.cc:1434] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
I20260709 12:27:25.220541 13931 tablet_bootstrap.cc:492] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:25.220553 13933 tablet_bootstrap.cc:492] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:25.225835 13708 raft_consensus.cc:2468] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 1.
I20260709 12:27:25.228834 13931 tablet_bootstrap.cc:654] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.229626 13933 tablet_bootstrap.cc:654] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.231140 13930 tablet_bootstrap.cc:492] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:25.238948 13930 tablet_bootstrap.cc:654] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.248526 13933 tablet_bootstrap.cc:492] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:25.249184 13933 ts_tablet_manager.cc:1403] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.029s	user 0.012s	sys 0.007s
I20260709 12:27:25.253823 13933 raft_consensus.cc:359] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.254907 13933 raft_consensus.cc:385] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.255316 13933 raft_consensus.cc:740] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.256376 13933 consensus_queue.cc:260] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.259809 13933 ts_tablet_manager.cc:1434] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.010s	user 0.012s	sys 0.000s
I20260709 12:27:25.261373 13933 tablet_bootstrap.cc:492] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:25.271462 13933 tablet_bootstrap.cc:654] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.273878 13749 leader_election.cc:304] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 99369f74684341af8f9ba9ff6c225216
I20260709 12:27:25.276511 13904 raft_consensus.cc:2804] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:25.277256 13904 raft_consensus.cc:697] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Becoming Leader. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:25.278172 13904 consensus_queue.cc:237] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.284325 13931 tablet_bootstrap.cc:492] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:25.285009 13931 ts_tablet_manager.cc:1403] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.065s	user 0.018s	sys 0.039s
I20260709 12:27:25.285244 13930 tablet_bootstrap.cc:492] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:25.285854 13930 ts_tablet_manager.cc:1403] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.055s	user 0.013s	sys 0.029s
I20260709 12:27:25.288255 13931 raft_consensus.cc:359] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.289273 13931 raft_consensus.cc:385] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.289755 13931 raft_consensus.cc:740] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.289268 13930 raft_consensus.cc:359] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.290323 13930 raft_consensus.cc:385] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.290671 13930 raft_consensus.cc:740] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.290729 13931 consensus_queue.cc:260] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.284930 13933 tablet_bootstrap.cc:492] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:25.291505 13930 consensus_queue.cc:260] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.292335 13933 ts_tablet_manager.cc:1403] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.031s	user 0.014s	sys 0.003s
I20260709 12:27:25.293360 13931 ts_tablet_manager.cc:1434] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:27:25.294360 13930 ts_tablet_manager.cc:1434] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:27:25.294534 13931 tablet_bootstrap.cc:492] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:25.294070 13531 catalog_manager.cc:5697] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "a5e7dc958a3f45de8436922490e08ef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:25.295506 13930 tablet_bootstrap.cc:492] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:25.295708 13933 raft_consensus.cc:359] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.296598 13933 raft_consensus.cc:385] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.296898 13933 raft_consensus.cc:740] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.297753 13933 consensus_queue.cc:260] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.300428 13929 raft_consensus.cc:493] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.300455 13933 ts_tablet_manager.cc:1434] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:27:25.300998 13929 raft_consensus.cc:515] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.301833 13933 tablet_bootstrap.cc:492] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:25.303516 13929 leader_election.cc:290] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.304186 13931 tablet_bootstrap.cc:654] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.304452 13930 tablet_bootstrap.cc:654] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.306020 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6581db8dd724835827972f64e4bb115" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:25.308712 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6581db8dd724835827972f64e4bb115" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:25.310865 13933 tablet_bootstrap.cc:654] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:25.310959 13675 leader_election.cc:343] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:25.312441 13676 leader_election.cc:343] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:25.313098 13676 leader_election.cc:304] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3189d868dea45edb84dd2d2aa69c3e9; no voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:25.317780 13904 raft_consensus.cc:493] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.318507 13904 raft_consensus.cc:515] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.321412 13904 leader_election.cc:290] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:25.324707 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0059d80d6d442499891fcc2bf42f899" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:25.325500 13708 raft_consensus.cc:2468] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 0.
I20260709 12:27:25.325711 13930 tablet_bootstrap.cc:492] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:25.326320 13930 ts_tablet_manager.cc:1403] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.031s	user 0.012s	sys 0.005s
I20260709 12:27:25.327196 13749 leader_election.cc:304] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:25.328465 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0059d80d6d442499891fcc2bf42f899" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:25.329190 13633 raft_consensus.cc:2468] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 0.
I20260709 12:27:25.329722 13930 raft_consensus.cc:359] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.330461 13930 raft_consensus.cc:385] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.330930 13930 raft_consensus.cc:740] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.331688 13930 consensus_queue.cc:260] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.335520 13930 ts_tablet_manager.cc:1434] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.009s	user 0.009s	sys 0.001s
I20260709 12:27:25.336158 13929 raft_consensus.cc:2749] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:25.338574 13932 raft_consensus.cc:493] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.336675 13930 tablet_bootstrap.cc:492] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:25.339196 13932 raft_consensus.cc:515] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.341580 13932 leader_election.cc:290] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.342363 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a777f237f174c6fb0a469141cc8c426" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:25.343324 13708 raft_consensus.cc:2393] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 99369f74684341af8f9ba9ff6c225216 in current term 1: Already voted for candidate a5e7dc958a3f45de8436922490e08ef8 in this term.
I20260709 12:27:25.343516 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a777f237f174c6fb0a469141cc8c426" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:25.339179 13904 raft_consensus.cc:2804] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:25.346143 13904 raft_consensus.cc:493] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:25.346611 13904 raft_consensus.cc:3060] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.345537 13601 leader_election.cc:304] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216; no voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:25.348127 13868 raft_consensus.cc:3060] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.353571 13930 tablet_bootstrap.cc:654] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.366945 13930 tablet_bootstrap.cc:492] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:25.367673 13930 ts_tablet_manager.cc:1403] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.031s	user 0.017s	sys 0.003s
I20260709 12:27:25.372326 13930 raft_consensus.cc:359] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.373410 13930 raft_consensus.cc:385] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.373816 13930 raft_consensus.cc:740] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.374868 13930 consensus_queue.cc:260] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.377316 13930 ts_tablet_manager.cc:1434] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:27:25.378520 13930 tablet_bootstrap.cc:492] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:25.383894 13868 raft_consensus.cc:2749] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:25.384878 13940 raft_consensus.cc:493] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.385720 13940 raft_consensus.cc:515] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.387748 13930 tablet_bootstrap.cc:654] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.388525 13940 leader_election.cc:290] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:25.389878 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d656052161734f809877b26b2eaa3487" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
W20260709 12:27:25.392477 13749 leader_election.cc:343] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:25.393973 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d656052161734f809877b26b2eaa3487" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
W20260709 12:27:25.399300 13749 leader_election.cc:343] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:25.399863 13749 leader_election.cc:304] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8; no voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:25.400985 13930 tablet_bootstrap.cc:492] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:25.401667 13930 ts_tablet_manager.cc:1403] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.023s	user 0.015s	sys 0.006s
I20260709 12:27:25.405943 13940 raft_consensus.cc:2749] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:25.406221 13930 raft_consensus.cc:359] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.408128 13930 raft_consensus.cc:385] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.408434 13930 raft_consensus.cc:740] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.409350 13930 consensus_queue.cc:260] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.412707 13930 ts_tablet_manager.cc:1434] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.011s	user 0.004s	sys 0.005s
I20260709 12:27:25.416726 13930 tablet_bootstrap.cc:492] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:25.424021 13930 tablet_bootstrap.cc:654] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:25.438407 13676 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.201.195:42677, user_credentials={real_user=slave}} blocked reactor thread for 126146us
I20260709 12:27:25.478521 13868 raft_consensus.cc:493] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.479408 13868 raft_consensus.cc:515] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.483407 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6581db8dd724835827972f64e4bb115" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:25.484143 13708 raft_consensus.cc:2468] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 0.
I20260709 12:27:25.485143 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6581db8dd724835827972f64e4bb115" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:25.485638 13600 leader_election.cc:304] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:25.486876 13932 raft_consensus.cc:2804] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:25.487390 13932 raft_consensus.cc:493] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:25.487877 13932 raft_consensus.cc:3060] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:25.491250 13601 leader_election.cc:343] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:25.423924 13904 raft_consensus.cc:515] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.503376 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0059d80d6d442499891fcc2bf42f899" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:25.504328 13633 raft_consensus.cc:3060] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.528815 13932 raft_consensus.cc:515] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.541419 13930 tablet_bootstrap.cc:492] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:25.541674 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6581db8dd724835827972f64e4bb115" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:25.542069 13930 ts_tablet_manager.cc:1403] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.126s	user 0.028s	sys 0.074s
I20260709 12:27:25.542496 13707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6581db8dd724835827972f64e4bb115" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:25.543196 13707 raft_consensus.cc:3060] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:25.544606 13601 leader_election.cc:343] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:25.545430 13930 raft_consensus.cc:359] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.546176 13930 raft_consensus.cc:385] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.546487 13930 raft_consensus.cc:740] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.547276 13930 consensus_queue.cc:260] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.531143 13868 leader_election.cc:290] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.531850 13958 raft_consensus.cc:493] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.550693 13958 raft_consensus.cc:515] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.551280 13932 leader_election.cc:290] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.554108 13930 ts_tablet_manager.cc:1434] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.012s	user 0.000s	sys 0.012s
I20260709 12:27:25.554796 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec8f9faf1894326ba84678fd84d81f7" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:25.534322 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0059d80d6d442499891fcc2bf42f899" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:25.535379 13931 tablet_bootstrap.cc:492] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:25.555761 13708 raft_consensus.cc:3060] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.535444 13633 raft_consensus.cc:2468] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 1.
I20260709 12:27:25.536393 13933 tablet_bootstrap.cc:492] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:25.556592 13933 ts_tablet_manager.cc:1403] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.255s	user 0.099s	sys 0.112s
I20260709 12:27:25.557296 13749 leader_election.cc:304] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: 
W20260709 12:27:25.557832 13601 leader_election.cc:343] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:25.558740 13954 raft_consensus.cc:2804] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:25.559273 13954 raft_consensus.cc:697] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Becoming Leader. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:25.560024 13933 raft_consensus.cc:359] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.560138 13954 consensus_queue.cc:237] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.560875 13933 raft_consensus.cc:385] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.561204 13933 raft_consensus.cc:740] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.562027 13933 consensus_queue.cc:260] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.564527 13933 ts_tablet_manager.cc:1434] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:27:25.555998 13931 ts_tablet_manager.cc:1403] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.262s	user 0.051s	sys 0.160s
I20260709 12:27:25.567939 13931 raft_consensus.cc:359] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.568692 13931 raft_consensus.cc:385] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.568838 13933 tablet_bootstrap.cc:492] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:25.554802 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec8f9faf1894326ba84678fd84d81f7" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:25.569512 13958 leader_election.cc:290] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.533032 13904 leader_election.cc:290] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:25.569806 13710 raft_consensus.cc:2468] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 0.
I20260709 12:27:25.572168 13707 raft_consensus.cc:2468] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 1.
I20260709 12:27:25.572357 13600 leader_election.cc:304] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:25.555292 13930 tablet_bootstrap.cc:492] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:25.574354 13600 leader_election.cc:304] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:25.576995 13933 tablet_bootstrap.cc:654] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.578843 13958 raft_consensus.cc:2804] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:25.578954 13932 raft_consensus.cc:2804] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:25.579413 13958 raft_consensus.cc:493] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:25.579620 13932 raft_consensus.cc:697] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Becoming Leader. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:25.579869 13958 raft_consensus.cc:3060] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.580624 13932 consensus_queue.cc:237] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.582535 13868 raft_consensus.cc:493] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.583129 13868 raft_consensus.cc:515] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.585352 13868 leader_election.cc:290] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.586614 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9551779678d40728ad7cf78f0551fa6" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:25.589165 13707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9551779678d40728ad7cf78f0551fa6" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:25.590104 13708 raft_consensus.cc:2468] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 1.
I20260709 12:27:25.569031 13931 raft_consensus.cc:740] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
W20260709 12:27:25.591456 13600 leader_election.cc:343] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:25.591483 13931 consensus_queue.cc:260] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.593870 13783 raft_consensus.cc:2468] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 0.
I20260709 12:27:25.597388 13601 leader_election.cc:304] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:25.595506 13931 ts_tablet_manager.cc:1434] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.030s	user 0.005s	sys 0.002s
I20260709 12:27:25.598728 13930 tablet_bootstrap.cc:654] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.598856 13932 raft_consensus.cc:2804] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:25.599448 13932 raft_consensus.cc:493] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:25.600872 13932 raft_consensus.cc:3060] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.603754 13531 catalog_manager.cc:5697] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99369f74684341af8f9ba9ff6c225216 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "99369f74684341af8f9ba9ff6c225216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:25.604398 13958 raft_consensus.cc:515] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.605379 13933 tablet_bootstrap.cc:492] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:25.605958 13933 ts_tablet_manager.cc:1403] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.037s	user 0.002s	sys 0.027s
I20260709 12:27:25.606905 13958 leader_election.cc:290] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.599720 13931 tablet_bootstrap.cc:492] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:25.607728 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec8f9faf1894326ba84678fd84d81f7" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:25.608018 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec8f9faf1894326ba84678fd84d81f7" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:25.608652 13708 raft_consensus.cc:3060] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
W20260709 12:27:25.609552 13601 leader_election.cc:343] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:25.624696 13931 tablet_bootstrap.cc:654] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.618723 13933 raft_consensus.cc:359] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.621937 13930 tablet_bootstrap.cc:492] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:25.624554 13708 raft_consensus.cc:2468] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 1.
I20260709 12:27:25.625167 13933 raft_consensus.cc:385] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.625998 13933 raft_consensus.cc:740] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.615761 13932 raft_consensus.cc:515] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.626929 13600 leader_election.cc:304] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:25.622311 13533 catalog_manager.cc:5697] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "a5e7dc958a3f45de8436922490e08ef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:25.628505 13932 leader_election.cc:290] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.628829 13958 raft_consensus.cc:2804] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:25.629411 13958 raft_consensus.cc:697] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Becoming Leader. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:25.629753 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9551779678d40728ad7cf78f0551fa6" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:25.630000 13707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9551779678d40728ad7cf78f0551fa6" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:25.630388 13783 raft_consensus.cc:3060] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.630432 13958 consensus_queue.cc:237] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.625655 13930 ts_tablet_manager.cc:1403] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.071s	user 0.010s	sys 0.017s
W20260709 12:27:25.632354 13600 leader_election.cc:343] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:25.631119 13959 raft_consensus.cc:493] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.634058 13959 raft_consensus.cc:515] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.634469 13930 raft_consensus.cc:359] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.635272 13930 raft_consensus.cc:385] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.635558 13930 raft_consensus.cc:740] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.636389 13930 consensus_queue.cc:260] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.637594 13959 leader_election.cc:290] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.638821 13930 ts_tablet_manager.cc:1434] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.001s
I20260709 12:27:25.627061 13933 consensus_queue.cc:260] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.641669 13784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d656052161734f809877b26b2eaa3487" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:25.642580 13784 raft_consensus.cc:2468] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 0.
I20260709 12:27:25.643069 13933 ts_tablet_manager.cc:1434] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.037s	user 0.005s	sys 0.007s
I20260709 12:27:25.643764 13959 raft_consensus.cc:493] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.644197 13601 leader_election.cc:304] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: 
I20260709 12:27:25.644496 13930 tablet_bootstrap.cc:492] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216: Bootstrap starting.
I20260709 12:27:25.644340 13959 raft_consensus.cc:515] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.645246 13932 raft_consensus.cc:2804] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:25.645656 13932 raft_consensus.cc:493] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:25.646020 13932 raft_consensus.cc:3060] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.648299 13959 leader_election.cc:290] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.649118 13707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7582c4042164df48db37a656afff05d" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:25.649438 13784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7582c4042164df48db37a656afff05d" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
W20260709 12:27:25.651034 13601 leader_election.cc:343] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:25.651705 13600 leader_election.cc:343] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:25.652189 13600 leader_election.cc:304] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216; no voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:25.655910 13783 raft_consensus.cc:2468] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 1.
I20260709 12:27:25.659376 13930 tablet_bootstrap.cc:654] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.661731 13932 raft_consensus.cc:515] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.664438 13959 raft_consensus.cc:2749] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:25.665019 13933 tablet_bootstrap.cc:492] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:25.667172 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d656052161734f809877b26b2eaa3487" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:25.667840 13708 raft_consensus.cc:2468] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 0.
I20260709 12:27:25.678341 13601 leader_election.cc:304] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:25.680332 13959 raft_consensus.cc:2804] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:25.681115 13959 raft_consensus.cc:697] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Becoming Leader. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:25.683925 13932 leader_election.cc:290] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.684314 13707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d656052161734f809877b26b2eaa3487" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:25.685096 13707 raft_consensus.cc:3060] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.689234 13930 tablet_bootstrap.cc:492] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216: No bootstrap required, opened a new log
I20260709 12:27:25.689827 13930 ts_tablet_manager.cc:1403] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216: Time spent bootstrapping tablet: real 0.046s	user 0.018s	sys 0.009s
I20260709 12:27:25.692616 13930 raft_consensus.cc:359] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.693318 13930 raft_consensus.cc:385] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.693579 13930 raft_consensus.cc:740] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.693624 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d656052161734f809877b26b2eaa3487" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:25.694267 13783 raft_consensus.cc:3060] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.694491 13930 consensus_queue.cc:260] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.682322 13959 consensus_queue.cc:237] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.697240 13930 ts_tablet_manager.cc:1434] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:27:25.697292 13533 catalog_manager.cc:5697] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99369f74684341af8f9ba9ff6c225216 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "99369f74684341af8f9ba9ff6c225216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:25.707809 13783 raft_consensus.cc:2468] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 1.
I20260709 12:27:25.713821 13601 leader_election.cc:304] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: 
I20260709 12:27:25.714675 13932 raft_consensus.cc:2804] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:25.715166 13932 raft_consensus.cc:697] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Becoming Leader. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:25.715857 13707 raft_consensus.cc:2468] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 1.
I20260709 12:27:25.715900 13932 consensus_queue.cc:237] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.718339 13933 tablet_bootstrap.cc:654] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.720894 13931 tablet_bootstrap.cc:492] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:25.721550 13931 ts_tablet_manager.cc:1403] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.122s	user 0.021s	sys 0.034s
I20260709 12:27:25.723327 13533 catalog_manager.cc:5697] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99369f74684341af8f9ba9ff6c225216 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "99369f74684341af8f9ba9ff6c225216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:25.725181 13933 tablet_bootstrap.cc:492] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:25.725252 13931 raft_consensus.cc:359] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.726009 13933 ts_tablet_manager.cc:1403] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.061s	user 0.009s	sys 0.006s
I20260709 12:27:25.726397 13931 raft_consensus.cc:385] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.726804 13931 raft_consensus.cc:740] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.727727 13931 consensus_queue.cc:260] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.729653 13933 raft_consensus.cc:359] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.730516 13931 ts_tablet_manager.cc:1434] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.000s
I20260709 12:27:25.730536 13933 raft_consensus.cc:385] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.731181 13933 raft_consensus.cc:740] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.732071 13931 tablet_bootstrap.cc:492] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:25.732035 13933 consensus_queue.cc:260] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.734858 13933 ts_tablet_manager.cc:1434] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:27:25.739224 13933 tablet_bootstrap.cc:492] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:25.740551 13931 tablet_bootstrap.cc:654] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.746058 13933 tablet_bootstrap.cc:654] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.746176 13531 catalog_manager.cc:5697] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 reported cstate change: term changed from 0 to 1, leader changed from <none> to 99369f74684341af8f9ba9ff6c225216 (127.29.201.193). New cstate: current_term: 1 leader_uuid: "99369f74684341af8f9ba9ff6c225216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:25.749069 13932 raft_consensus.cc:493] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.749610 13932 raft_consensus.cc:515] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.751832 13932 leader_election.cc:290] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.753391 13707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04da14f36947466f88a8dbfaa58505f2" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:25.753706 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04da14f36947466f88a8dbfaa58505f2" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
W20260709 12:27:25.755182 13601 leader_election.cc:343] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:25.755802 13600 leader_election.cc:343] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:25.756258 13600 leader_election.cc:304] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216; no voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:25.757313 13932 raft_consensus.cc:2749] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:25.764976 13931 tablet_bootstrap.cc:492] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:25.765630 13931 ts_tablet_manager.cc:1403] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.034s	user 0.008s	sys 0.013s
I20260709 12:27:25.768606 13933 tablet_bootstrap.cc:492] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:25.768695 13931 raft_consensus.cc:359] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.769325 13933 ts_tablet_manager.cc:1403] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.030s	user 0.009s	sys 0.001s
I20260709 12:27:25.769740 13931 raft_consensus.cc:385] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.770134 13931 raft_consensus.cc:740] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.771092 13931 consensus_queue.cc:260] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.772809 13933 raft_consensus.cc:359] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.773680 13933 raft_consensus.cc:385] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.773880 13931 ts_tablet_manager.cc:1434] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.008s	user 0.007s	sys 0.000s
I20260709 12:27:25.773990 13933 raft_consensus.cc:740] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.775313 13931 tablet_bootstrap.cc:492] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:25.775543 13933 consensus_queue.cc:260] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.777995 13933 ts_tablet_manager.cc:1434] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:27:25.779138 13933 tablet_bootstrap.cc:492] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9: Bootstrap starting.
I20260709 12:27:25.784373 13931 tablet_bootstrap.cc:654] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.786392 13933 tablet_bootstrap.cc:654] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.791723 13931 tablet_bootstrap.cc:492] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:25.792537 13931 ts_tablet_manager.cc:1403] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.017s	user 0.011s	sys 0.004s
I20260709 12:27:25.796787 13931 raft_consensus.cc:359] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.797874 13931 raft_consensus.cc:385] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.798250 13931 raft_consensus.cc:740] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.799000 13931 consensus_queue.cc:260] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.800129 13933 tablet_bootstrap.cc:492] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9: No bootstrap required, opened a new log
I20260709 12:27:25.800657 13933 ts_tablet_manager.cc:1403] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9: Time spent bootstrapping tablet: real 0.022s	user 0.010s	sys 0.009s
I20260709 12:27:25.801239 13931 ts_tablet_manager.cc:1434] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.008s	user 0.008s	sys 0.000s
I20260709 12:27:25.802404 13931 tablet_bootstrap.cc:492] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:25.803130 13933 raft_consensus.cc:359] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.803839 13933 raft_consensus.cc:385] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.804105 13933 raft_consensus.cc:740] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.804809 13933 consensus_queue.cc:260] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.807036 13933 ts_tablet_manager.cc:1434] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.004s
I20260709 12:27:25.809929 13931 tablet_bootstrap.cc:654] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.819870 13931 tablet_bootstrap.cc:492] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:25.820417 13931 ts_tablet_manager.cc:1403] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.018s	user 0.012s	sys 0.004s
I20260709 12:27:25.823062 13954 consensus_queue.cc:1048] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:25.823349 13931 raft_consensus.cc:359] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.824126 13931 raft_consensus.cc:385] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.824328 13931 raft_consensus.cc:740] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.825050 13931 consensus_queue.cc:260] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.827379 13931 ts_tablet_manager.cc:1434] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.007s	user 0.008s	sys 0.000s
I20260709 12:27:25.828764 13931 tablet_bootstrap.cc:492] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8: Bootstrap starting.
I20260709 12:27:25.835111 13931 tablet_bootstrap.cc:654] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:25.835665 13954 consensus_queue.cc:1048] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:25.841460 13950 raft_consensus.cc:493] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.842146 13950 raft_consensus.cc:515] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.845498 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9551779678d40728ad7cf78f0551fa6" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:25.845752 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9551779678d40728ad7cf78f0551fa6" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:25.845726 13950 leader_election.cc:290] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.846796 13783 raft_consensus.cc:2393] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f3189d868dea45edb84dd2d2aa69c3e9 in current term 1: Already voted for candidate 99369f74684341af8f9ba9ff6c225216 in this term.
I20260709 12:27:25.848198 13676 leader_election.cc:304] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f3189d868dea45edb84dd2d2aa69c3e9; no voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:25.848381 13931 tablet_bootstrap.cc:492] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8: No bootstrap required, opened a new log
I20260709 12:27:25.849017 13931 ts_tablet_manager.cc:1403] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8: Time spent bootstrapping tablet: real 0.021s	user 0.010s	sys 0.008s
I20260709 12:27:25.849959 13950 raft_consensus.cc:3060] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.853569 13931 raft_consensus.cc:359] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.854395 13931 raft_consensus.cc:385] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:25.854710 13931 raft_consensus.cc:740] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Initialized, Role: FOLLOWER
I20260709 12:27:25.855537 13931 consensus_queue.cc:260] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.857762 13931 ts_tablet_manager.cc:1434] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:27:25.864187 13950 raft_consensus.cc:2749] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:25.908004 13950 raft_consensus.cc:493] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.908546 13950 raft_consensus.cc:515] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.910357 13950 leader_election.cc:290] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.911444 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04da14f36947466f88a8dbfaa58505f2" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:25.911513 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04da14f36947466f88a8dbfaa58505f2" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:25.912487 13783 raft_consensus.cc:2468] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 0.
I20260709 12:27:25.912487 13633 raft_consensus.cc:2468] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 0.
I20260709 12:27:25.913698 13676 leader_election.cc:304] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:25.914426 13950 raft_consensus.cc:2804] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:25.914871 13950 raft_consensus.cc:493] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:25.915195 13950 raft_consensus.cc:3060] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.921198 13929 raft_consensus.cc:493] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.921782 13929 raft_consensus.cc:515] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.923115 13950 raft_consensus.cc:515] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.923826 13929 leader_election.cc:290] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.924655 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7582c4042164df48db37a656afff05d" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:25.924964 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7582c4042164df48db37a656afff05d" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8" is_pre_election: true
I20260709 12:27:25.925215 13950 leader_election.cc:290] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.925657 13783 raft_consensus.cc:2468] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 0.
I20260709 12:27:25.925985 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04da14f36947466f88a8dbfaa58505f2" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:25.925622 13633 raft_consensus.cc:2468] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 0.
I20260709 12:27:25.926748 13635 raft_consensus.cc:3060] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.927001 13676 leader_election.cc:304] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:25.927285 13940 raft_consensus.cc:493] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.927389 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "04da14f36947466f88a8dbfaa58505f2" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:25.927958 13950 raft_consensus.cc:2804] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:25.927793 13940 raft_consensus.cc:515] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.928411 13950 raft_consensus.cc:493] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:25.928817 13950 raft_consensus.cc:3060] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.930030 13940 leader_election.cc:290] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:25.928092 13783 raft_consensus.cc:3060] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.932471 13707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96138840b8c545ab90f1d0be4086de59" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:25.933238 13707 raft_consensus.cc:2468] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 0.
I20260709 12:27:25.934561 13749 leader_election.cc:304] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:25.935460 13940 raft_consensus.cc:2804] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:25.935824 13940 raft_consensus.cc:493] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:25.936141 13940 raft_consensus.cc:3060] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.936810 13950 raft_consensus.cc:515] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.937752 13635 raft_consensus.cc:2468] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 1.
I20260709 12:27:25.939348 13675 leader_election.cc:304] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:25.939509 13783 raft_consensus.cc:2468] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 1.
I20260709 12:27:25.940165 13929 raft_consensus.cc:2804] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:25.940706 13929 raft_consensus.cc:697] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 LEADER]: Becoming Leader. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:25.939821 13635 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7582c4042164df48db37a656afff05d" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:25.941339 13635 raft_consensus.cc:3060] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.941576 13929 consensus_queue.cc:237] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.943475 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96138840b8c545ab90f1d0be4086de59" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:25.944195 13633 raft_consensus.cc:2468] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 0.
I20260709 12:27:25.947135 13940 raft_consensus.cc:515] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
W20260709 12:27:25.950208 13581 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:25.950050 13635 raft_consensus.cc:2468] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 1.
I20260709 12:27:25.950812 13634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96138840b8c545ab90f1d0be4086de59" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:25.952382 13634 raft_consensus.cc:3060] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.952659 13950 leader_election.cc:290] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:25.955691 13675 leader_election.cc:304] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:25.956146 13707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96138840b8c545ab90f1d0be4086de59" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:25.956965 13707 raft_consensus.cc:3060] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.956696 13950 raft_consensus.cc:2804] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:25.957708 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7582c4042164df48db37a656afff05d" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:25.957916 13954 raft_consensus.cc:493] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:25.958328 13950 raft_consensus.cc:697] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 LEADER]: Becoming Leader. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:25.958611 13783 raft_consensus.cc:3060] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.958767 13954 raft_consensus.cc:515] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.959326 13950 consensus_queue.cc:237] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.961115 13634 raft_consensus.cc:2468] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 1.
I20260709 12:27:25.961251 13954 leader_election.cc:290] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:25.963028 13749 leader_election.cc:304] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: 
I20260709 12:27:25.965023 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6581db8dd724835827972f64e4bb115" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:25.966276 13708 raft_consensus.cc:2393] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5e7dc958a3f45de8436922490e08ef8 in current term 1: Already voted for candidate 99369f74684341af8f9ba9ff6c225216 in this term.
I20260709 12:27:25.968555 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d6581db8dd724835827972f64e4bb115" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:25.968803 13783 raft_consensus.cc:2468] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 1.
I20260709 12:27:25.972129 13749 leader_election.cc:304] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8; no voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:25.972945 13904 raft_consensus.cc:2804] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:25.974820 13954 raft_consensus.cc:3060] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:25.978890 13904 raft_consensus.cc:697] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Becoming Leader. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:25.974848 13940 leader_election.cc:290] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:25.979983 13904 consensus_queue.cc:237] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:25.982935 13707 raft_consensus.cc:2468] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 1.
I20260709 12:27:25.982378 13531 catalog_manager.cc:5697] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:25.983158 13954 raft_consensus.cc:2749] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:25.992488 13529 catalog_manager.cc:5697] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "a5e7dc958a3f45de8436922490e08ef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:25.998311 13531 catalog_manager.cc:5697] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:26.009735 13904 raft_consensus.cc:493] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:26.010291 13904 raft_consensus.cc:515] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.012517 13904 leader_election.cc:290] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:26.013396 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec8f9faf1894326ba84678fd84d81f7" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99369f74684341af8f9ba9ff6c225216" is_pre_election: true
I20260709 12:27:26.013767 13932 consensus_queue.cc:1048] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:26.014536 13707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec8f9faf1894326ba84678fd84d81f7" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" is_pre_election: true
I20260709 12:27:26.015447 13707 raft_consensus.cc:2393] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a5e7dc958a3f45de8436922490e08ef8 in current term 1: Already voted for candidate 99369f74684341af8f9ba9ff6c225216 in this term.
I20260709 12:27:26.016919 13749 leader_election.cc:304] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8; no voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:26.018196 13904 raft_consensus.cc:3060] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:26.028606 13904 raft_consensus.cc:2749] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:26.033068 13868 consensus_queue.cc:1048] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:26.071074 13904 consensus_queue.cc:1048] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:26.081918 13940 consensus_queue.cc:1048] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:26.083900 13868 consensus_queue.cc:1048] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:26.105036 13958 consensus_queue.cc:1048] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:26.112439 13958 consensus_queue.cc:1048] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:26.131875 13868 consensus_queue.cc:1048] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:26.203878 13868 consensus_queue.cc:1048] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:26.215773 13932 consensus_queue.cc:1048] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:26.275825 13784 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c0059d80d6d442499891fcc2bf42f899"
dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
mode: GRACEFUL
new_leader_uuid: "99369f74684341af8f9ba9ff6c225216"
 from {username='slave'} at 127.0.0.1:47416
I20260709 12:27:26.276562 13784 raft_consensus.cc:606] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Received request to transfer leadership to TS 99369f74684341af8f9ba9ff6c225216
I20260709 12:27:26.282178 13784 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "2a777f237f174c6fb0a469141cc8c426"
dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
mode: GRACEFUL
new_leader_uuid: "99369f74684341af8f9ba9ff6c225216"
 from {username='slave'} at 127.0.0.1:47416
I20260709 12:27:26.282794 13784 raft_consensus.cc:606] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Received request to transfer leadership to TS 99369f74684341af8f9ba9ff6c225216
I20260709 12:27:26.294870 13633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "eec8f9faf1894326ba84678fd84d81f7"
dest_uuid: "99369f74684341af8f9ba9ff6c225216"
mode: GRACEFUL
new_leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:41114
I20260709 12:27:26.295540 13633 raft_consensus.cc:606] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Received request to transfer leadership to TS f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:26.301164 13633 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d656052161734f809877b26b2eaa3487"
dest_uuid: "99369f74684341af8f9ba9ff6c225216"
mode: GRACEFUL
new_leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:41114
I20260709 12:27:26.301780 13633 raft_consensus.cc:606] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Received request to transfer leadership to TS f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:26.311122 13784 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "96138840b8c545ab90f1d0be4086de59"
dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
mode: GRACEFUL
new_leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:47416
I20260709 12:27:26.311708 13784 raft_consensus.cc:606] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Received request to transfer leadership to TS f3189d868dea45edb84dd2d2aa69c3e9
I20260709 12:27:26.317034 13707 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "a7582c4042164df48db37a656afff05d"
dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
mode: GRACEFUL
new_leader_uuid: "a5e7dc958a3f45de8436922490e08ef8"
 from {username='slave'} at 127.0.0.1:52112
I20260709 12:27:26.317610 13707 raft_consensus.cc:606] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 LEADER]: Received request to transfer leadership to TS a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:26.343626 13950 consensus_queue.cc:1048] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:26.351130 13929 raft_consensus.cc:993] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9: : Instructing follower a5e7dc958a3f45de8436922490e08ef8 to start an election
I20260709 12:27:26.351643 13929 raft_consensus.cc:1081] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 LEADER]: Signalling peer a5e7dc958a3f45de8436922490e08ef8 to start an election
I20260709 12:27:26.353861 13783 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "a7582c4042164df48db37a656afff05d"
dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
 from {username='slave'} at 127.0.0.1:47414
I20260709 12:27:26.355855 13783 raft_consensus.cc:493] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:26.356243 13783 raft_consensus.cc:3060] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.358445 13950 consensus_queue.cc:1048] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:26.363356 13783 raft_consensus.cc:515] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.368412 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7582c4042164df48db37a656afff05d" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:26.368844 13707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a7582c4042164df48db37a656afff05d" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:26.369040 13633 raft_consensus.cc:3060] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.369525 13707 raft_consensus.cc:3055] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:26.369992 13707 raft_consensus.cc:740] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:26.370886 13707 consensus_queue.cc:260] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.372255 13707 raft_consensus.cc:3060] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.374707 13783 leader_election.cc:290] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 2 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:26.377923 13633 raft_consensus.cc:2468] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 2.
I20260709 12:27:26.380316 13749 leader_election.cc:304] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: 
I20260709 12:27:26.380336 13707 raft_consensus.cc:2468] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 2.
I20260709 12:27:26.381255 13940 raft_consensus.cc:2804] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:26.381784 13940 raft_consensus.cc:697] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [term 2 LEADER]: Becoming Leader. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:26.382663 13940 consensus_queue.cc:237] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.387147 13954 consensus_queue.cc:1048] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:26.393064 13531 catalog_manager.cc:5697] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 reported cstate change: term changed from 1 to 2, leader changed from f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194) to a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195). New cstate: current_term: 2 leader_uuid: "a5e7dc958a3f45de8436922490e08ef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:26.393875 13950 consensus_queue.cc:1048] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:26.405825 13904 consensus_queue.cc:1048] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:26.415890 13954 raft_consensus.cc:993] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8: : Instructing follower f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:26.416534 13954 raft_consensus.cc:1081] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Signalling peer f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:26.418825 13707 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "96138840b8c545ab90f1d0be4086de59"
dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:52108
I20260709 12:27:26.422845 13707 raft_consensus.cc:493] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:26.423367 13707 raft_consensus.cc:3060] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.427505 13929 consensus_queue.cc:1048] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:26.436508 13707 raft_consensus.cc:515] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.439092 13707 leader_election.cc:290] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:26.443591 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96138840b8c545ab90f1d0be4086de59" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:26.444170 13633 raft_consensus.cc:3060] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.445775 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96138840b8c545ab90f1d0be4086de59" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:26.446475 13783 raft_consensus.cc:3055] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:26.446807 13783 raft_consensus.cc:740] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:26.447548 13783 consensus_queue.cc:260] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.448762 13783 raft_consensus.cc:3060] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.453418 13633 raft_consensus.cc:2468] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 2.
I20260709 12:27:26.455081 13675 leader_election.cc:304] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:26.456009 13948 raft_consensus.cc:2804] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:26.456490 13948 raft_consensus.cc:697] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Becoming Leader. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:26.457307 13948 consensus_queue.cc:237] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.459137 13783 raft_consensus.cc:2468] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 2.
I20260709 12:27:26.467533 13533 catalog_manager.cc:5697] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 reported cstate change: term changed from 1 to 2, leader changed from a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195) to f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:26.480597 13983 raft_consensus.cc:993] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8: : Instructing follower 99369f74684341af8f9ba9ff6c225216 to start an election
I20260709 12:27:26.481145 13904 raft_consensus.cc:1081] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Signalling peer 99369f74684341af8f9ba9ff6c225216 to start an election
I20260709 12:27:26.482761 13633 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "2a777f237f174c6fb0a469141cc8c426"
dest_uuid: "99369f74684341af8f9ba9ff6c225216"
 from {username='slave'} at 127.0.0.1:41108
I20260709 12:27:26.483191 13633 raft_consensus.cc:493] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:26.483429 13633 raft_consensus.cc:3060] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.488582 13633 raft_consensus.cc:515] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.489629 13958 raft_consensus.cc:993] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216: : Instructing follower f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:26.490235 13932 raft_consensus.cc:1081] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Signalling peer f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:26.490291 13633 leader_election.cc:290] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 2 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:26.491199 13707 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a777f237f174c6fb0a469141cc8c426" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:26.491410 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a777f237f174c6fb0a469141cc8c426" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:26.491878 13707 raft_consensus.cc:3060] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.491930 13783 raft_consensus.cc:3055] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:26.492221 13710 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "eec8f9faf1894326ba84678fd84d81f7"
dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:52100
I20260709 12:27:26.492341 13783 raft_consensus.cc:740] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:26.492892 13710 raft_consensus.cc:493] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:26.493297 13710 raft_consensus.cc:3060] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.493417 13783 consensus_queue.cc:260] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.494915 13783 raft_consensus.cc:3060] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.499704 13707 raft_consensus.cc:2468] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 2.
I20260709 12:27:26.501281 13710 raft_consensus.cc:515] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.501456 13600 leader_election.cc:304] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:26.502312 13783 raft_consensus.cc:2468] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 2.
I20260709 12:27:26.502612 13932 raft_consensus.cc:2804] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:26.503213 13932 raft_consensus.cc:697] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Becoming Leader. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:26.504063 13932 consensus_queue.cc:237] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.504840 13904 raft_consensus.cc:993] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8: : Instructing follower 99369f74684341af8f9ba9ff6c225216 to start an election
I20260709 12:27:26.504923 13710 leader_election.cc:290] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:26.505447 13904 raft_consensus.cc:1081] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Signalling peer 99369f74684341af8f9ba9ff6c225216 to start an election
I20260709 12:27:26.505980 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec8f9faf1894326ba84678fd84d81f7" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:26.506565 13783 raft_consensus.cc:3060] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.507496 13633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eec8f9faf1894326ba84678fd84d81f7" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:26.508045 13633 raft_consensus.cc:3055] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:26.508284 13634 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c0059d80d6d442499891fcc2bf42f899"
dest_uuid: "99369f74684341af8f9ba9ff6c225216"
 from {username='slave'} at 127.0.0.1:41108
I20260709 12:27:26.508373 13633 raft_consensus.cc:740] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:26.509008 13634 raft_consensus.cc:493] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:26.509414 13634 raft_consensus.cc:3060] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.513198 13533 catalog_manager.cc:5697] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 reported cstate change: term changed from 1 to 2, leader changed from a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195) to 99369f74684341af8f9ba9ff6c225216 (127.29.201.193). New cstate: current_term: 2 leader_uuid: "99369f74684341af8f9ba9ff6c225216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:26.509603 13633 consensus_queue.cc:260] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.515482 13633 raft_consensus.cc:3060] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.516111 13783 raft_consensus.cc:2468] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 2.
I20260709 12:27:26.518476 13676 leader_election.cc:304] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:26.519377 13948 raft_consensus.cc:2804] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:26.519861 13948 raft_consensus.cc:697] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Becoming Leader. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:26.520697 13948 consensus_queue.cc:237] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.524055 13633 raft_consensus.cc:2468] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 2.
I20260709 12:27:26.516152 13634 raft_consensus.cc:515] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.530310 13710 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0059d80d6d442499891fcc2bf42f899" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:26.531081 13710 raft_consensus.cc:3060] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.531519 13783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0059d80d6d442499891fcc2bf42f899" candidate_uuid: "99369f74684341af8f9ba9ff6c225216" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:26.532163 13783 raft_consensus.cc:3055] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:26.532631 13783 raft_consensus.cc:740] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:26.532106 13533 catalog_manager.cc:5697] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 reported cstate change: term changed from 1 to 2, leader changed from 99369f74684341af8f9ba9ff6c225216 (127.29.201.193) to f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:26.533528 13783 consensus_queue.cc:260] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.534469 13783 raft_consensus.cc:3060] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.539469 13710 raft_consensus.cc:2468] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 2.
I20260709 12:27:26.540400 13707 raft_consensus.cc:1240] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Rejecting Update request from peer a5e7dc958a3f45de8436922490e08ef8 for earlier term 1. Current term is 2. Ops: []
I20260709 12:27:26.543468 13600 leader_election.cc:304] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:26.543999 13783 raft_consensus.cc:2468] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99369f74684341af8f9ba9ff6c225216 in term 2.
I20260709 12:27:26.544549 13958 raft_consensus.cc:2804] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:26.545100 13958 raft_consensus.cc:697] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Becoming Leader. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:26.545938 13958 consensus_queue.cc:237] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.530714 13634 leader_election.cc:290] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [CANDIDATE]: Term 2 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:26.555658 13533 catalog_manager.cc:5697] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 reported cstate change: term changed from 1 to 2, leader changed from a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195) to 99369f74684341af8f9ba9ff6c225216 (127.29.201.193). New cstate: current_term: 2 leader_uuid: "99369f74684341af8f9ba9ff6c225216" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:26.606837 13958 raft_consensus.cc:993] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216: : Instructing follower f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:26.607393 13959 raft_consensus.cc:1081] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Signalling peer f3189d868dea45edb84dd2d2aa69c3e9 to start an election
I20260709 12:27:26.608965 13710 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d656052161734f809877b26b2eaa3487"
dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
 from {username='slave'} at 127.0.0.1:52100
I20260709 12:27:26.609500 13710 raft_consensus.cc:493] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:26.609869 13710 raft_consensus.cc:3060] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.616704 13710 raft_consensus.cc:515] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.618521 13710 leader_election.cc:290] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195:42677)
I20260709 12:27:26.619616 13784 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d656052161734f809877b26b2eaa3487" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
I20260709 12:27:26.619936 13634 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d656052161734f809877b26b2eaa3487" candidate_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:26.620406 13784 raft_consensus.cc:3060] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.620565 13634 raft_consensus.cc:3055] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:26.620996 13634 raft_consensus.cc:740] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:26.621857 13634 consensus_queue.cc:260] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.623222 13634 raft_consensus.cc:3060] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:26.628844 13784 raft_consensus.cc:2468] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 2.
I20260709 12:27:26.630431 13676 leader_election.cc:304] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:26.631084 13634 raft_consensus.cc:2468] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3189d868dea45edb84dd2d2aa69c3e9 in term 2.
I20260709 12:27:26.631425 13948 raft_consensus.cc:2804] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:26.632025 13948 raft_consensus.cc:697] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Becoming Leader. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:26.632880 13948 consensus_queue.cc:237] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:26.640022 13533 catalog_manager.cc:5697] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 reported cstate change: term changed from 1 to 2, leader changed from 99369f74684341af8f9ba9ff6c225216 (127.29.201.193) to f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:26.764649 13634 raft_consensus.cc:1275] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Refusing update from remote peer a5e7dc958a3f45de8436922490e08ef8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:26.765957 13904 consensus_queue.cc:1048] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:26.773851 13710 raft_consensus.cc:1275] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Refusing update from remote peer a5e7dc958a3f45de8436922490e08ef8: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:26.775502 13954 consensus_queue.cc:1048] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:26.963409 13784 raft_consensus.cc:1275] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:26.964917 13948 consensus_queue.cc:1048] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:26.973565 13633 raft_consensus.cc:1275] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:26.983366 13948 consensus_queue.cc:1048] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:26.984086 13634 raft_consensus.cc:1275] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:26.987203 14001 consensus_queue.cc:1048] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:27.000801 13783 raft_consensus.cc:1275] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:27.002391 13929 consensus_queue.cc:1048] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:27.051901 13783 raft_consensus.cc:1275] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Refusing update from remote peer 99369f74684341af8f9ba9ff6c225216: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:27.053133 13932 consensus_queue.cc:1048] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:27.062192 13710 raft_consensus.cc:1275] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Refusing update from remote peer 99369f74684341af8f9ba9ff6c225216: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:27.064704 13959 consensus_queue.cc:1048] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:27.104008 13783 raft_consensus.cc:1275] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Refusing update from remote peer 99369f74684341af8f9ba9ff6c225216: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:27.105369 13959 consensus_queue.cc:1048] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:27.114825 13710 raft_consensus.cc:1275] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Refusing update from remote peer 99369f74684341af8f9ba9ff6c225216: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:27.116473 13958 consensus_queue.cc:1048] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:27.181551 13634 raft_consensus.cc:1275] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:27.183086 13929 consensus_queue.cc:1048] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:27.195495 13783 raft_consensus.cc:1275] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Refusing update from remote peer f3189d868dea45edb84dd2d2aa69c3e9: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:27.197118 14003 consensus_queue.cc:1048] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:29.321614 13533 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35484:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:27:29.343130 13533 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35484:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:27:29.373905 30503 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
W20260709 12:27:29.552762 14024 log.cc:927] Time spent T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
W20260709 12:27:29.536062 14026 log.cc:927] Time spent T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216: Append to log took a long time: real 0.054s	user 0.001s	sys 0.000s
W20260709 12:27:29.589013 14031 log.cc:927] Time spent T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9: Append to log took a long time: real 0.117s	user 0.000s	sys 0.002s
I20260709 12:27:29.686745 14058 tablet.cc:1724] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:27:29.690409 13601 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.195:42677, user_credentials={real_user=slave}} blocked reactor thread for 98199.2us
W20260709 12:27:29.682780 14045 log.cc:927] Time spent T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216: Append to log took a long time: real 0.059s	user 0.000s	sys 0.001s
W20260709 12:27:29.701364 14052 log.cc:927] Time spent T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8: Append to log took a long time: real 0.060s	user 0.002s	sys 0.000s
W20260709 12:27:29.683210 13600 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:45095, user_credentials={real_user=slave}} blocked reactor thread for 91260.3us
I20260709 12:27:29.740744 14061 tablet.cc:1724] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:27:29.774684 13810 tablet_mm_ops.cc:139] Rowset compaction is disabled (check --enable_rowset_compaction and disable_compaction in extra_config for tablet:04da14f36947466f88a8dbfaa58505f2)
I20260709 12:27:29.777904 13706 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "96138840b8c545ab90f1d0be4086de59"
dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
mode: GRACEFUL
new_leader_uuid: "a5e7dc958a3f45de8436922490e08ef8"
 from {username='slave'} at 127.0.0.1:52938
I20260709 12:27:29.778712 13706 raft_consensus.cc:606] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Received request to transfer leadership to TS a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:29.780158 14062 tablet.cc:1724] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.748276 14063 tablet.cc:1724] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:27:29.782440 13810 tablet_mm_ops.cc:304] Major delta compaction is disabled (check --enable_major_delta_compaction and disable_compaction in extra_config for tablet:04da14f36947466f88a8dbfaa58505f2)
I20260709 12:27:29.787708 14069 tablet.cc:1724] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.787506 14068 tablet.cc:1724] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:27:29.862380 14059 log.cc:927] Time spent T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216: Append to log took a long time: real 0.083s	user 0.001s	sys 0.000s
I20260709 12:27:29.865657 14072 tablet.cc:1724] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.871680 14077 tablet.cc:1724] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:27:29.885046 13810 tablet_mm_ops.cc:220] Minor delta compaction is disabled (check --enable_minor_delta_compaction and disable_compaction in extra_config for tablet:04da14f36947466f88a8dbfaa58505f2)
I20260709 12:27:29.859112 14078 tablet.cc:1724] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.906791 14076 tablet.cc:1724] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.916407 14063 tablet.cc:1724] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.928151 14071 raft_consensus.cc:993] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9: : Instructing follower a5e7dc958a3f45de8436922490e08ef8 to start an election
I20260709 12:27:29.928779 14071 raft_consensus.cc:1081] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Signalling peer a5e7dc958a3f45de8436922490e08ef8 to start an election
I20260709 12:27:29.928364 14058 tablet.cc:1724] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.932991 14068 tablet.cc:1724] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:27:29.940565 14081 log.cc:927] Time spent T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8: Append to log took a long time: real 0.058s	user 0.000s	sys 0.000s
I20260709 12:27:29.943087 14075 tablet.cc:1724] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.938928 13777 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "96138840b8c545ab90f1d0be4086de59"
dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
 from {username='slave'} at 127.0.0.1:47414
I20260709 12:27:29.945461 13777 raft_consensus.cc:493] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:29.946030 13777 raft_consensus.cc:3060] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:29.946286 14069 tablet.cc:1724] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.954224 14080 tablet.cc:1724] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.951700 14061 tablet.cc:1724] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.963950 14068 tablet.cc:1724] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.982391 14076 tablet.cc:1724] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.983584 14077 tablet.cc:1724] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.985446 14069 tablet.cc:1724] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.988421 14062 tablet.cc:1724] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:29.986886 14072 tablet.cc:1724] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:30.022244 14080 tablet.cc:1724] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:30.022899 13777 raft_consensus.cc:515] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:30.040742 14063 tablet.cc:1724] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:30.043736 14078 tablet.cc:1724] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:30.047595 13709 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96138840b8c545ab90f1d0be4086de59" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:30.048324 13709 raft_consensus.cc:3055] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:30.048627 13709 raft_consensus.cc:740] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:30.049456 13709 consensus_queue.cc:260] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:30.050720 13709 raft_consensus.cc:3060] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:30.046401 13777 leader_election.cc:290] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 3 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:30.053766 13634 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c0059d80d6d442499891fcc2bf42f899"
dest_uuid: "99369f74684341af8f9ba9ff6c225216"
mode: GRACEFUL
new_leader_uuid: "a5e7dc958a3f45de8436922490e08ef8"
 from {username='slave'} at 127.0.0.1:53784
I20260709 12:27:30.054391 13634 raft_consensus.cc:606] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Received request to transfer leadership to TS a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:30.025652 14058 tablet.cc:1724] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:30.059629 30503 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 2
I20260709 12:27:30.067137 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:30.059468 13632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "96138840b8c545ab90f1d0be4086de59" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:30.071827 13632 raft_consensus.cc:3060] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:30.082454 13708 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a"
dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
mode: GRACEFUL
new_leader_uuid: "a5e7dc958a3f45de8436922490e08ef8"
 from {username='slave'} at 127.0.0.1:52938
I20260709 12:27:30.082896 13709 raft_consensus.cc:2468] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 3.
I20260709 12:27:30.083343 13708 raft_consensus.cc:606] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Received request to transfer leadership to TS a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:30.085239 13632 raft_consensus.cc:2468] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 3.
I20260709 12:27:30.086589 13749 leader_election.cc:304] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:30.088238 14057 raft_consensus.cc:2804] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:30.088801 14057 raft_consensus.cc:697] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 3 LEADER]: Becoming Leader. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:30.089638 14057 consensus_queue.cc:237] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:30.091485 13632 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef"
dest_uuid: "99369f74684341af8f9ba9ff6c225216"
mode: GRACEFUL
new_leader_uuid: "a5e7dc958a3f45de8436922490e08ef8"
 from {username='slave'} at 127.0.0.1:53784
I20260709 12:27:30.092233 13632 raft_consensus.cc:606] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Received request to transfer leadership to TS a5e7dc958a3f45de8436922490e08ef8
I20260709 12:27:30.093865 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:27:30.094579 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:30.107154 13533 catalog_manager.cc:5697] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 reported cstate change: term changed from 2 to 3, leader changed from f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194) to a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195). New cstate: current_term: 3 leader_uuid: "a5e7dc958a3f45de8436922490e08ef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:30.108722 13531 catalog_manager.cc:6242] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a: soft_deleted_table [id=281afdf33e544aca89a1d989bf3b66f6] alter complete (version 1)
I20260709 12:27:30.267163 14082 raft_consensus.cc:993] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216: : Instructing follower a5e7dc958a3f45de8436922490e08ef8 to start an election
I20260709 12:27:30.267908 14036 raft_consensus.cc:1081] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Signalling peer a5e7dc958a3f45de8436922490e08ef8 to start an election
I20260709 12:27:30.269809 13784 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c0059d80d6d442499891fcc2bf42f899"
dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
 from {username='slave'} at 127.0.0.1:47400
I20260709 12:27:30.270407 13784 raft_consensus.cc:493] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:30.270807 13784 raft_consensus.cc:3060] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:30.276726 13784 raft_consensus.cc:515] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:30.278359 13784 leader_election.cc:290] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 3 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:30.279330 13632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0059d80d6d442499891fcc2bf42f899" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:30.280239 13632 raft_consensus.cc:3055] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:30.280584 13632 raft_consensus.cc:740] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:30.281363 13632 consensus_queue.cc:260] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:30.282394 13632 raft_consensus.cc:3060] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:30.283169 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c0059d80d6d442499891fcc2bf42f899" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: true dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:30.283819 13708 raft_consensus.cc:3060] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:30.288564 13632 raft_consensus.cc:2468] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 3.
I20260709 12:27:30.289597 13708 raft_consensus.cc:2468] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 3.
I20260709 12:27:30.290035 13749 leader_election.cc:304] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: 
I20260709 12:27:30.291028 14057 raft_consensus.cc:2804] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:30.291533 14057 raft_consensus.cc:697] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 3 LEADER]: Becoming Leader. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:30.292224 14057 consensus_queue.cc:237] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:30.300575 13533 catalog_manager.cc:5697] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 reported cstate change: term changed from 2 to 3, leader changed from 99369f74684341af8f9ba9ff6c225216 (127.29.201.193) to a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195). New cstate: current_term: 3 leader_uuid: "a5e7dc958a3f45de8436922490e08ef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:30.309461 14071 raft_consensus.cc:993] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9: : Instructing follower a5e7dc958a3f45de8436922490e08ef8 to start an election
I20260709 12:27:30.310139 14010 raft_consensus.cc:1081] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Signalling peer a5e7dc958a3f45de8436922490e08ef8 to start an election
I20260709 12:27:30.311893 13784 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a"
dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
 from {username='slave'} at 127.0.0.1:47414
I20260709 12:27:30.312453 13784 raft_consensus.cc:493] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:30.312739 13784 raft_consensus.cc:3060] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:30.318327 13784 raft_consensus.cc:515] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:30.319984 13784 leader_election.cc:290] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 3 election: Requested vote from peers 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059), f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095)
I20260709 12:27:30.321025 13632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:30.321429 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "66feb633e0dd4d549c04578e7eac038a" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:30.321667 13632 raft_consensus.cc:3060] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:30.322151 13708 raft_consensus.cc:3055] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:30.322443 13708 raft_consensus.cc:740] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f3189d868dea45edb84dd2d2aa69c3e9, State: Running, Role: LEADER
I20260709 12:27:30.323133 13708 consensus_queue.cc:260] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:30.324229 13708 raft_consensus.cc:3060] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:30.327927 13632 raft_consensus.cc:2468] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 3.
I20260709 12:27:30.329363 13749 leader_election.cc:304] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99369f74684341af8f9ba9ff6c225216, a5e7dc958a3f45de8436922490e08ef8; no voters: 
I20260709 12:27:30.329859 13708 raft_consensus.cc:2468] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 3.
I20260709 12:27:30.330487 14057 raft_consensus.cc:2804] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:30.331132 14057 raft_consensus.cc:697] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 3 LEADER]: Becoming Leader. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:30.331971 14057 consensus_queue.cc:237] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } }
I20260709 12:27:30.339504 13533 catalog_manager.cc:5697] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 reported cstate change: term changed from 2 to 3, leader changed from f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194) to a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195). New cstate: current_term: 3 leader_uuid: "a5e7dc958a3f45de8436922490e08ef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:30.351145 14082 raft_consensus.cc:993] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216: : Instructing follower a5e7dc958a3f45de8436922490e08ef8 to start an election
I20260709 12:27:30.351718 14036 raft_consensus.cc:1081] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Signalling peer a5e7dc958a3f45de8436922490e08ef8 to start an election
I20260709 12:27:30.353432 13784 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef"
dest_uuid: "a5e7dc958a3f45de8436922490e08ef8"
 from {username='slave'} at 127.0.0.1:47400
I20260709 12:27:30.354009 13784 raft_consensus.cc:493] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:30.354314 13784 raft_consensus.cc:3060] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:30.359465 13784 raft_consensus.cc:515] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:30.360991 13784 leader_election.cc:290] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 3 election: Requested vote from peers f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095), 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059)
I20260709 12:27:30.362071 13708 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f3189d868dea45edb84dd2d2aa69c3e9"
I20260709 12:27:30.362390 13632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "93fd5bdcec364b0fa44abe6a94e822ef" candidate_uuid: "a5e7dc958a3f45de8436922490e08ef8" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "99369f74684341af8f9ba9ff6c225216"
I20260709 12:27:30.362793 13708 raft_consensus.cc:3060] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:30.363065 13632 raft_consensus.cc:3055] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:30.363420 13632 raft_consensus.cc:740] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 99369f74684341af8f9ba9ff6c225216, State: Running, Role: LEADER
I20260709 12:27:30.364254 13632 consensus_queue.cc:260] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:30.365604 13632 raft_consensus.cc:3060] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:30.370348 13708 raft_consensus.cc:2468] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 3.
I20260709 12:27:30.371618 13749 leader_election.cc:304] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e7dc958a3f45de8436922490e08ef8, f3189d868dea45edb84dd2d2aa69c3e9; no voters: 
I20260709 12:27:30.372586 14057 raft_consensus.cc:2804] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:30.373310 13632 raft_consensus.cc:2468] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e7dc958a3f45de8436922490e08ef8 in term 3.
I20260709 12:27:30.373342 14057 raft_consensus.cc:697] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 3 LEADER]: Becoming Leader. State: Replica: a5e7dc958a3f45de8436922490e08ef8, State: Running, Role: LEADER
I20260709 12:27:30.375314 14057 consensus_queue.cc:237] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } }
I20260709 12:27:30.383756 13533 catalog_manager.cc:5697] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 reported cstate change: term changed from 2 to 3, leader changed from 99369f74684341af8f9ba9ff6c225216 (127.29.201.193) to a5e7dc958a3f45de8436922490e08ef8 (127.29.201.195). New cstate: current_term: 3 leader_uuid: "a5e7dc958a3f45de8436922490e08ef8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5e7dc958a3f45de8436922490e08ef8" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 42677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:30.509670 13632 raft_consensus.cc:1275] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Refusing update from remote peer a5e7dc958a3f45de8436922490e08ef8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:27:30.511121 14057 consensus_queue.cc:1048] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:27:30.521865 13708 raft_consensus.cc:1275] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Refusing update from remote peer a5e7dc958a3f45de8436922490e08ef8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:27:30.523811 14057 consensus_queue.cc:1048] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:27:30.674422 13709 raft_consensus.cc:1275] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Refusing update from remote peer a5e7dc958a3f45de8436922490e08ef8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:27:30.675935 14057 consensus_queue.cc:1048] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:27:30.684964 13632 raft_consensus.cc:1275] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Refusing update from remote peer a5e7dc958a3f45de8436922490e08ef8: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
I20260709 12:27:30.686902 14057 consensus_queue.cc:1048] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260709 12:27:30.810228 13709 raft_consensus.cc:1275] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Refusing update from remote peer a5e7dc958a3f45de8436922490e08ef8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:30.811700 14057 consensus_queue.cc:1048] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:30.821609 13632 raft_consensus.cc:1275] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Refusing update from remote peer a5e7dc958a3f45de8436922490e08ef8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:30.823695 14057 consensus_queue.cc:1048] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:30.836422 13709 raft_consensus.cc:1275] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Refusing update from remote peer a5e7dc958a3f45de8436922490e08ef8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:30.838589 14046 consensus_queue.cc:1048] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3189d868dea45edb84dd2d2aa69c3e9" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 45095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:27:30.843420 13634 raft_consensus.cc:1275] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Refusing update from remote peer a5e7dc958a3f45de8436922490e08ef8: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:30.845367 14057 consensus_queue.cc:1048] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99369f74684341af8f9ba9ff6c225216" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:27:31.096088 30503 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:27:31.103183 30503 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:27:31.103736 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:31.110917 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:31.111444 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:32.132711 30503 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:27:32.154361 30503 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:27:32.155078 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:32.164884 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:32.165445 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:33.166919 30503 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:27:33.173022 30503 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:27:33.173563 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:33.180236 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:33.180814 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:34.181963 30503 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:27:34.187196 30503 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:27:34.187678 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:34.195004 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:34.195537 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:35.196652 30503 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:27:35.201699 30503 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:27:35.202204 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:35.207471 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:35.207931 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:36.209022 30503 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:27:36.214747 30503 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:27:36.215206 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:36.221465 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:36.221891 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:37.222932 30503 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:27:37.228868 30503 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:27:37.229270 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:37.234483 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:37.235271 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:38.236584 30503 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:27:38.242966 30503 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:27:38.243474 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:38.248557 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:38.249024 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:39.250211 30503 auto_leader_rebalancer.cc:141] leader rebalance for table soft_deleted_table
I20260709 12:27:39.256201 30503 auto_leader_rebalancer.cc:440] table: soft_deleted_table, leader rebalance finish, leader transfer count: 0
I20260709 12:27:39.256781 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:39.264256 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:39.264724 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:40.335224 30503 tablet_server.cc:179] TabletServer@127.29.201.193:0 shutting down...
W20260709 12:27:40.354521 13749 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 11 similar messages]
W20260709 12:27:40.361063 13749 consensus_peers.cc:597] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 -> Peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Couldn't send request to peer 99369f74684341af8f9ba9ff6c225216. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:40.391973 13675 consensus_peers.cc:597] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 -> Peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Couldn't send request to peer 99369f74684341af8f9ba9ff6c225216. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:40.395977 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:40.397002 30503 tablet_replica.cc:333] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.398088 30503 raft_consensus.cc:2243] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.399121 30503 raft_consensus.cc:2272] T 96138840b8c545ab90f1d0be4086de59 P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.402004 30503 tablet_replica.cc:333] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.402824 30503 raft_consensus.cc:2243] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.403478 30503 raft_consensus.cc:2272] T d656052161734f809877b26b2eaa3487 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.407680 30503 tablet_replica.cc:333] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.408854 30503 raft_consensus.cc:2243] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.409767 30503 raft_consensus.cc:2272] T eec8f9faf1894326ba84678fd84d81f7 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.414423 30503 tablet_replica.cc:333] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.415271 30503 raft_consensus.cc:2243] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.416393 30503 raft_consensus.cc:2272] T d6581db8dd724835827972f64e4bb115 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.419247 30503 tablet_replica.cc:333] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.419910 30503 raft_consensus.cc:2243] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.420910 30503 raft_consensus.cc:2272] T 2a777f237f174c6fb0a469141cc8c426 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.423862 30503 tablet_replica.cc:333] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.424477 30503 raft_consensus.cc:2243] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.425107 30503 raft_consensus.cc:2272] T 04da14f36947466f88a8dbfaa58505f2 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.427881 30503 tablet_replica.cc:333] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.428488 30503 raft_consensus.cc:2243] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.429136 30503 raft_consensus.cc:2272] T 2f47887de35d404f83ab0edf3cabb706 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.434069 30503 tablet_replica.cc:333] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.435024 30503 raft_consensus.cc:2243] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.436025 30503 raft_consensus.cc:2272] T b9551779678d40728ad7cf78f0551fa6 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.438707 30503 tablet_replica.cc:333] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.439344 30503 raft_consensus.cc:2243] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.439950 30503 raft_consensus.cc:2272] T a7582c4042164df48db37a656afff05d P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.442730 30503 tablet_replica.cc:333] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.443286 30503 raft_consensus.cc:2243] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.443833 30503 raft_consensus.cc:2272] T 0a1f458064a14018ac4e716f1cda3b73 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.446395 30503 tablet_replica.cc:333] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.447082 30503 raft_consensus.cc:2243] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.447606 30503 raft_consensus.cc:2272] T c0059d80d6d442499891fcc2bf42f899 P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.450502 30503 tablet_replica.cc:333] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.451150 30503 raft_consensus.cc:2243] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.452154 30503 raft_consensus.cc:2272] T 81f900f101f543fa8b3cfea0fb81f776 P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.455096 30503 tablet_replica.cc:333] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.455729 30503 raft_consensus.cc:2243] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.456324 30503 raft_consensus.cc:2272] T 33fa2959a148483984e00bb7fdcd0dcb P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.459406 30503 tablet_replica.cc:333] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.460165 30503 raft_consensus.cc:2243] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.461623 30503 raft_consensus.cc:2272] T 6145a89b623b4ec5b638ec5fe3c030cc P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:40.464605 13749 consensus_peers.cc:597] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 -> Peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Couldn't send request to peer 99369f74684341af8f9ba9ff6c225216. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:40.465611 30503 tablet_replica.cc:333] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.466527 30503 raft_consensus.cc:2243] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.467237 30503 raft_consensus.cc:2272] T f4f25524761a4060871c3cc27c79f9df P 99369f74684341af8f9ba9ff6c225216 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.470202 30503 tablet_replica.cc:333] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.470916 30503 raft_consensus.cc:2243] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.471491 30503 raft_consensus.cc:2272] T 93fd5bdcec364b0fa44abe6a94e822ef P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.474294 30503 tablet_replica.cc:333] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.474988 30503 raft_consensus.cc:2243] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.476007 30503 raft_consensus.cc:2272] T 5633dbff576b44b8914b2cc23f91de93 P 99369f74684341af8f9ba9ff6c225216 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.478895 30503 tablet_replica.cc:333] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216: stopping tablet replica
I20260709 12:27:40.479499 30503 raft_consensus.cc:2243] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.480134 30503 raft_consensus.cc:2272] T 66feb633e0dd4d549c04578e7eac038a P 99369f74684341af8f9ba9ff6c225216 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:40.499349 13675 consensus_peers.cc:597] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 -> Peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Couldn't send request to peer 99369f74684341af8f9ba9ff6c225216. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:40.500100 13749 consensus_peers.cc:597] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 -> Peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Couldn't send request to peer 99369f74684341af8f9ba9ff6c225216. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:40.506771 13675 consensus_peers.cc:597] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 -> Peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Couldn't send request to peer 99369f74684341af8f9ba9ff6c225216. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:40.527082 30503 tablet_server.cc:196] TabletServer@127.29.201.193:0 shutdown complete.
I20260709 12:27:40.578392 30503 tablet_server.cc:179] TabletServer@127.29.201.194:0 shutting down...
I20260709 12:27:40.612355 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:40.613251 30503 tablet_replica.cc:333] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.614172 30503 raft_consensus.cc:2243] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.615356 30503 raft_consensus.cc:2272] T d656052161734f809877b26b2eaa3487 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.617894 30503 tablet_replica.cc:333] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.618566 30503 raft_consensus.cc:2243] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.619279 30503 raft_consensus.cc:2272] T 96138840b8c545ab90f1d0be4086de59 P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.621579 30503 tablet_replica.cc:333] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.622166 30503 raft_consensus.cc:2243] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.623287 30503 raft_consensus.cc:2272] T eec8f9faf1894326ba84678fd84d81f7 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.625849 30503 tablet_replica.cc:333] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.626792 30503 raft_consensus.cc:2243] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.627452 30503 raft_consensus.cc:2272] T d6581db8dd724835827972f64e4bb115 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.630332 30503 tablet_replica.cc:333] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.631053 30503 raft_consensus.cc:2243] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.631582 30503 raft_consensus.cc:2272] T 33fa2959a148483984e00bb7fdcd0dcb P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.634058 30503 tablet_replica.cc:333] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.634829 30503 raft_consensus.cc:2243] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.635366 30503 raft_consensus.cc:2272] T 81f900f101f543fa8b3cfea0fb81f776 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.637842 30503 tablet_replica.cc:333] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.638492 30503 raft_consensus.cc:2243] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.639133 30503 raft_consensus.cc:2272] T c0059d80d6d442499891fcc2bf42f899 P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.641541 30503 tablet_replica.cc:333] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.642129 30503 raft_consensus.cc:2243] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.642623 30503 raft_consensus.cc:2272] T 66feb633e0dd4d549c04578e7eac038a P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.645102 30503 tablet_replica.cc:333] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.645680 30503 raft_consensus.cc:2243] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.646606 30503 raft_consensus.cc:2272] T f4f25524761a4060871c3cc27c79f9df P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.648975 30503 tablet_replica.cc:333] T 6145a89b623b4ec5b638ec5fe3c030cc P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.649598 30503 raft_consensus.cc:2243] T 6145a89b623b4ec5b638ec5fe3c030cc P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.650074 30503 raft_consensus.cc:2272] T 6145a89b623b4ec5b638ec5fe3c030cc P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.653550 30503 tablet_replica.cc:333] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.654273 30503 raft_consensus.cc:2243] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.654996 30503 raft_consensus.cc:2272] T 2a777f237f174c6fb0a469141cc8c426 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:40.657537 13675 consensus_peers.cc:597] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 -> Peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Couldn't send request to peer 99369f74684341af8f9ba9ff6c225216. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:40.658444 30503 tablet_replica.cc:333] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.659161 30503 raft_consensus.cc:2243] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.659684 30503 raft_consensus.cc:2272] T 93fd5bdcec364b0fa44abe6a94e822ef P f3189d868dea45edb84dd2d2aa69c3e9 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.661998 30503 tablet_replica.cc:333] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.662566 30503 raft_consensus.cc:2243] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.663120 30503 raft_consensus.cc:2272] T 5633dbff576b44b8914b2cc23f91de93 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.665764 30503 tablet_replica.cc:333] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.666597 30503 raft_consensus.cc:2243] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.668138 30503 raft_consensus.cc:2272] T 04da14f36947466f88a8dbfaa58505f2 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:40.671993 13749 consensus_peers.cc:597] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 -> Peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Couldn't send request to peer f3189d868dea45edb84dd2d2aa69c3e9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:40.672087 30503 tablet_replica.cc:333] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.672876 30503 raft_consensus.cc:2243] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.673471 30503 raft_consensus.cc:2272] T b9551779678d40728ad7cf78f0551fa6 P f3189d868dea45edb84dd2d2aa69c3e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.675992 30503 tablet_replica.cc:333] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.676609 30503 raft_consensus.cc:2243] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.677570 30503 raft_consensus.cc:2272] T 2f47887de35d404f83ab0edf3cabb706 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.680017 30503 tablet_replica.cc:333] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.680694 30503 raft_consensus.cc:2243] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.681248 30503 raft_consensus.cc:2272] T a7582c4042164df48db37a656afff05d P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.683611 30503 tablet_replica.cc:333] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9: stopping tablet replica
I20260709 12:27:40.684135 30503 raft_consensus.cc:2243] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.685105 30503 raft_consensus.cc:2272] T 0a1f458064a14018ac4e716f1cda3b73 P f3189d868dea45edb84dd2d2aa69c3e9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.730448 30503 tablet_server.cc:196] TabletServer@127.29.201.194:0 shutdown complete.
W20260709 12:27:40.768362 13749 consensus_peers.cc:597] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 -> Peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Couldn't send request to peer 99369f74684341af8f9ba9ff6c225216. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:40.775398 30503 tablet_server.cc:179] TabletServer@127.29.201.195:0 shutting down...
W20260709 12:27:40.791813 13749 consensus_peers.cc:597] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 -> Peer 99369f74684341af8f9ba9ff6c225216 (127.29.201.193:39059): Couldn't send request to peer 99369f74684341af8f9ba9ff6c225216. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:39059: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:27:40.804615 13749 consensus_peers.cc:597] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 -> Peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Couldn't send request to peer f3189d868dea45edb84dd2d2aa69c3e9. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:45095: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:40.812094 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:27:40.813040 30503 tablet_replica.cc:333] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.813755 30503 raft_consensus.cc:2243] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.814738 30503 raft_consensus.cc:2272] T 96138840b8c545ab90f1d0be4086de59 P a5e7dc958a3f45de8436922490e08ef8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.817231 30503 tablet_replica.cc:333] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.817920 30503 raft_consensus.cc:2243] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.818540 30503 raft_consensus.cc:2272] T eec8f9faf1894326ba84678fd84d81f7 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.821655 30503 tablet_replica.cc:333] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
W20260709 12:27:40.822434 13749 consensus_peers.cc:597] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 -> Peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Couldn't send request to peer f3189d868dea45edb84dd2d2aa69c3e9. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:45095: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:40.822559 30503 raft_consensus.cc:2243] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.823324 30503 raft_consensus.cc:2272] T d6581db8dd724835827972f64e4bb115 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.825678 30503 tablet_replica.cc:333] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.826385 30503 raft_consensus.cc:2243] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.826963 30503 raft_consensus.cc:2272] T d656052161734f809877b26b2eaa3487 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.829281 30503 tablet_replica.cc:333] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.829939 30503 raft_consensus.cc:2243] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.830915 30503 raft_consensus.cc:2272] T 33fa2959a148483984e00bb7fdcd0dcb P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.833338 30503 tablet_replica.cc:333] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.834012 30503 raft_consensus.cc:2243] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.834553 30503 raft_consensus.cc:2272] T 04da14f36947466f88a8dbfaa58505f2 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.837061 30503 tablet_replica.cc:333] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.837703 30503 raft_consensus.cc:2243] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.838245 30503 raft_consensus.cc:2272] T b9551779678d40728ad7cf78f0551fa6 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.841310 30503 tablet_replica.cc:333] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.842065 30503 raft_consensus.cc:2243] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.842849 30503 raft_consensus.cc:2272] T 2f47887de35d404f83ab0edf3cabb706 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:40.843765 13749 consensus_peers.cc:597] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 -> Peer f3189d868dea45edb84dd2d2aa69c3e9 (127.29.201.194:45095): Couldn't send request to peer f3189d868dea45edb84dd2d2aa69c3e9. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:45095: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:27:40.845503 30503 tablet_replica.cc:333] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.846232 30503 raft_consensus.cc:2243] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.846822 30503 raft_consensus.cc:2272] T 81f900f101f543fa8b3cfea0fb81f776 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.849431 30503 tablet_replica.cc:333] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.850144 30503 raft_consensus.cc:2243] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.851166 30503 raft_consensus.cc:2272] T c0059d80d6d442499891fcc2bf42f899 P a5e7dc958a3f45de8436922490e08ef8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.854566 30503 tablet_replica.cc:333] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.855607 30503 raft_consensus.cc:2243] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.856770 30503 raft_consensus.cc:2272] T 66feb633e0dd4d549c04578e7eac038a P a5e7dc958a3f45de8436922490e08ef8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.859508 30503 tablet_replica.cc:333] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.860129 30503 raft_consensus.cc:2243] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.860570 30503 raft_consensus.cc:2272] T f4f25524761a4060871c3cc27c79f9df P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.862849 30503 tablet_replica.cc:333] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.863425 30503 raft_consensus.cc:2243] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.863899 30503 raft_consensus.cc:2272] T 6145a89b623b4ec5b638ec5fe3c030cc P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.866577 30503 tablet_replica.cc:333] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.867331 30503 raft_consensus.cc:2243] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.868113 30503 raft_consensus.cc:2272] T 2a777f237f174c6fb0a469141cc8c426 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.871272 30503 tablet_replica.cc:333] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.871781 30503 raft_consensus.cc:2243] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.872175 30503 raft_consensus.cc:2272] T 5633dbff576b44b8914b2cc23f91de93 P a5e7dc958a3f45de8436922490e08ef8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.874231 30503 tablet_replica.cc:333] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.874779 30503 raft_consensus.cc:2243] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.875491 30503 raft_consensus.cc:2272] T 93fd5bdcec364b0fa44abe6a94e822ef P a5e7dc958a3f45de8436922490e08ef8 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.877897 30503 tablet_replica.cc:333] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.878366 30503 raft_consensus.cc:2243] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:27:40.879202 30503 raft_consensus.cc:2272] T a7582c4042164df48db37a656afff05d P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:40.881350 30503 tablet_replica.cc:333] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8: stopping tablet replica
I20260709 12:27:40.881806 30503 raft_consensus.cc:2243] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:27:40.882246 30503 raft_consensus.cc:2272] T 0a1f458064a14018ac4e716f1cda3b73 P a5e7dc958a3f45de8436922490e08ef8 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:27:41.251767 13748 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
W20260709 12:27:41.711568 13748 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260709 12:27:41.905339 30503 tablet_server.cc:196] TabletServer@127.29.201.195:0 shutdown complete.
I20260709 12:27:41.948468 30503 master.cc:562] Master@127.29.201.254:44713 shutting down...
I20260709 12:27:41.973569 30503 raft_consensus.cc:2243] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:27:41.974289 30503 raft_consensus.cc:2272] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:27:41.974606 30503 tablet_replica.cc:333] T 00000000000000000000000000000000 P dd0a13d159d04fe094fb12e9ed83929a: stopping tablet replica
I20260709 12:27:41.995477 30503 master.cc:584] Master@127.29.201.254:44713 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/0 (22327 ms)
[ RUN      ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1
I20260709 12:27:42.057928 30503 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.201.254:37363
I20260709 12:27:42.059275 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:42.067278 14125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:42.068037 14124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:42.071293 14127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:42.071636 30503 server_base.cc:1061] running on GCE node
I20260709 12:27:42.072845 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:42.073078 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:42.073243 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600062073226 us; error 0 us; skew 500 ppm
I20260709 12:27:42.073796 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:42.076542 30503 webserver.cc:533] Webserver started at http://127.29.201.254:44315/ using document root <none> and password file <none>
I20260709 12:27:42.077078 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:42.077289 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:42.077579 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:42.078965 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/master-0-root/instance:
uuid: "981e75aa16d74694a15008175e8137dc"
format_stamp: "Formatted at 2026-07-09 12:27:42 on dist-test-slave-qgf5"
I20260709 12:27:42.083748 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.003s
I20260709 12:27:42.087649 14132 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:42.088652 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:27:42.088922 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/master-0-root
uuid: "981e75aa16d74694a15008175e8137dc"
format_stamp: "Formatted at 2026-07-09 12:27:42 on dist-test-slave-qgf5"
I20260709 12:27:42.089180 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:42.100914 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:42.102267 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:42.144482 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.254:37363
I20260709 12:27:42.144606 14183 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.254:37363 every 8 connection(s)
I20260709 12:27:42.149013 14184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.184768 14184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc: Bootstrap starting.
I20260709 12:27:42.191042 14184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:42.196455 14184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc: No bootstrap required, opened a new log
I20260709 12:27:42.199484 14184 raft_consensus.cc:359] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981e75aa16d74694a15008175e8137dc" member_type: VOTER }
I20260709 12:27:42.200119 14184 raft_consensus.cc:385] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:42.200436 14184 raft_consensus.cc:740] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 981e75aa16d74694a15008175e8137dc, State: Initialized, Role: FOLLOWER
I20260709 12:27:42.201233 14184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981e75aa16d74694a15008175e8137dc" member_type: VOTER }
I20260709 12:27:42.201889 14184 raft_consensus.cc:399] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260709 12:27:42.202142 14184 raft_consensus.cc:493] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260709 12:27:42.202438 14184 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:42.209308 14184 raft_consensus.cc:515] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981e75aa16d74694a15008175e8137dc" member_type: VOTER }
I20260709 12:27:42.209952 14184 leader_election.cc:304] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 981e75aa16d74694a15008175e8137dc; no voters: 
I20260709 12:27:42.211324 14184 leader_election.cc:290] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [CANDIDATE]: Term 1 election: Requested vote from peers 
I20260709 12:27:42.211786 14188 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:42.213331 14188 raft_consensus.cc:697] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [term 1 LEADER]: Becoming Leader. State: Replica: 981e75aa16d74694a15008175e8137dc, State: Running, Role: LEADER
I20260709 12:27:42.214236 14188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981e75aa16d74694a15008175e8137dc" member_type: VOTER }
I20260709 12:27:42.215526 14184 sys_catalog.cc:565] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [sys.catalog]: configured and running, proceeding with master startup.
I20260709 12:27:42.221609 14189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "981e75aa16d74694a15008175e8137dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981e75aa16d74694a15008175e8137dc" member_type: VOTER } }
I20260709 12:27:42.221670 14190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 981e75aa16d74694a15008175e8137dc. Latest consensus state: current_term: 1 leader_uuid: "981e75aa16d74694a15008175e8137dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "981e75aa16d74694a15008175e8137dc" member_type: VOTER } }
I20260709 12:27:42.222462 14189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:42.223707 14190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [sys.catalog]: This master's current role is: LEADER
I20260709 12:27:42.225749 14194 catalog_manager.cc:1511] Loading table and tablet metadata into memory...
I20260709 12:27:42.232997 14194 catalog_manager.cc:1520] Initializing Kudu cluster ID...
I20260709 12:27:42.236249 30503 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260709 12:27:42.243512 14194 catalog_manager.cc:1383] Generated new cluster ID: 2dd927a60e694f5c970b1b32261fc681
I20260709 12:27:42.243858 14194 catalog_manager.cc:1531] Initializing Kudu internal certificate authority...
I20260709 12:27:42.280786 14194 catalog_manager.cc:1406] Generated new certificate authority record
I20260709 12:27:42.282325 14194 catalog_manager.cc:1540] Loading token signing keys...
I20260709 12:27:42.303165 14194 catalog_manager.cc:6070] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc: Generated new TSK 0
I20260709 12:27:42.303951 14194 catalog_manager.cc:1550] Initializing in-progress tserver states...
I20260709 12:27:42.367910 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:42.376036 14206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:42.376981 14207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:42.379426 14209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:42.379492 30503 server_base.cc:1061] running on GCE node
I20260709 12:27:42.380951 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:42.381206 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:42.381399 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600062381378 us; error 0 us; skew 500 ppm
I20260709 12:27:42.381979 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:42.384732 30503 webserver.cc:533] Webserver started at http://127.29.201.193:36181/ using document root <none> and password file <none>
I20260709 12:27:42.385295 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:42.385514 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:42.385847 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:42.387458 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-0-root/instance:
uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
format_stamp: "Formatted at 2026-07-09 12:27:42 on dist-test-slave-qgf5"
I20260709 12:27:42.392575 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.005s	sys 0.000s
I20260709 12:27:42.396564 14214 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:42.397586 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.001s	sys 0.000s
I20260709 12:27:42.397939 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-0-root
uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
format_stamp: "Formatted at 2026-07-09 12:27:42 on dist-test-slave-qgf5"
I20260709 12:27:42.398285 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:42.421068 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:42.422509 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:42.424151 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:42.426940 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:42.427217 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:42.427496 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:42.427671 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:42.471601 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.193:41273
I20260709 12:27:42.471711 14276 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.193:41273 every 8 connection(s)
I20260709 12:27:42.477718 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:42.486558 14281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:42.487938 14282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:42.490968 14284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:42.494859 14277 heartbeater.cc:344] Connected to a master server at 127.29.201.254:37363
I20260709 12:27:42.495404 14277 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:42.496371 14277 heartbeater.cc:507] Master 127.29.201.254:37363 requested a full tablet report, sending...
I20260709 12:27:42.496488 30503 server_base.cc:1061] running on GCE node
I20260709 12:27:42.498000 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:42.498250 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:42.498466 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600062498441 us; error 0 us; skew 500 ppm
I20260709 12:27:42.499104 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:42.499225 14149 ts_manager.cc:194] Registered new tserver with Master: bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273)
I20260709 12:27:42.501739 14149 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:37186
I20260709 12:27:42.501982 30503 webserver.cc:533] Webserver started at http://127.29.201.194:35449/ using document root <none> and password file <none>
I20260709 12:27:42.502805 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:42.503038 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:42.503382 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:42.504767 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-1-root/instance:
uuid: "c403bb4886574f1db5b159d81a264617"
format_stamp: "Formatted at 2026-07-09 12:27:42 on dist-test-slave-qgf5"
I20260709 12:27:42.509806 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.006s	sys 0.000s
I20260709 12:27:42.513831 14289 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:42.514889 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.003s	sys 0.000s
I20260709 12:27:42.515225 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-1-root
uuid: "c403bb4886574f1db5b159d81a264617"
format_stamp: "Formatted at 2026-07-09 12:27:42 on dist-test-slave-qgf5"
I20260709 12:27:42.515527 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:42.537225 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:42.538538 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:42.540237 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:42.542775 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:42.543017 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:42.543263 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:42.543437 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:42.587417 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.194:33923
I20260709 12:27:42.587512 14351 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.194:33923 every 8 connection(s)
I20260709 12:27:42.592681 30503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260709 12:27:42.603520 14356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:42.604049 14355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260709 12:27:42.605865 14358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260709 12:27:42.609206 14352 heartbeater.cc:344] Connected to a master server at 127.29.201.254:37363
I20260709 12:27:42.609700 14352 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:42.610512 14352 heartbeater.cc:507] Master 127.29.201.254:37363 requested a full tablet report, sending...
I20260709 12:27:42.610764 30503 server_base.cc:1061] running on GCE node
I20260709 12:27:42.611907 30503 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260709 12:27:42.612147 30503 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260709 12:27:42.612337 30503 hybrid_clock.cc:648] HybridClock initialized: now 1783600062612303 us; error 0 us; skew 500 ppm
I20260709 12:27:42.612941 30503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260709 12:27:42.613131 14149 ts_manager.cc:194] Registered new tserver with Master: c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:42.615191 14149 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46976
I20260709 12:27:42.615875 30503 webserver.cc:533] Webserver started at http://127.29.201.195:33593/ using document root <none> and password file <none>
I20260709 12:27:42.616636 30503 fs_manager.cc:362] Metadata directory not provided
I20260709 12:27:42.616889 30503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260709 12:27:42.617235 30503 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260709 12:27:42.619015 30503 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-2-root/instance:
uuid: "7310265ac7544fd281403a40bf5d6d30"
format_stamp: "Formatted at 2026-07-09 12:27:42 on dist-test-slave-qgf5"
I20260709 12:27:42.623839 30503 fs_manager.cc:696] Time spent creating directory manager: real 0.004s	user 0.001s	sys 0.003s
I20260709 12:27:42.627808 14363 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:42.628691 30503 fs_manager.cc:730] Time spent opening block manager: real 0.003s	user 0.004s	sys 0.000s
I20260709 12:27:42.629036 30503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-2-root
uuid: "7310265ac7544fd281403a40bf5d6d30"
format_stamp: "Formatted at 2026-07-09 12:27:42 on dist-test-slave-qgf5"
I20260709 12:27:42.629396 30503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskWoexUp/test-tmp/auto_leader_rebalancer-test.0.FilterSoftDeletedTableTest.TestFilterSofteDeletedTable_1.1783599553313367-30503-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260709 12:27:42.654197 30503 rpc_server.cc:225] running with OpenSSL 1.1.1  11 Sep 2018
I20260709 12:27:42.655607 30503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260709 12:27:42.657182 30503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260709 12:27:42.659626 30503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260709 12:27:42.659883 30503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:42.660127 30503 ts_tablet_manager.cc:616] Registered 0 tablets
I20260709 12:27:42.660303 30503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s	user 0.000s	sys 0.000s
I20260709 12:27:42.704995 30503 rpc_server.cc:307] RPC server started. Bound to: 127.29.201.195:38553
I20260709 12:27:42.705117 14425 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.29.201.195:38553 every 8 connection(s)
I20260709 12:27:42.720515 14426 heartbeater.cc:344] Connected to a master server at 127.29.201.254:37363
I20260709 12:27:42.720947 14426 heartbeater.cc:461] Registering TS with master...
I20260709 12:27:42.721740 14426 heartbeater.cc:507] Master 127.29.201.254:37363 requested a full tablet report, sending...
I20260709 12:27:42.723836 14149 ts_manager.cc:194] Registered new tserver with Master: 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:42.724561 30503 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015938896s
I20260709 12:27:42.725381 14149 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:46986
I20260709 12:27:42.752413 14149 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47000:
name: "test-workload"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:27:42.755417 14149 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:27:42.842324 14317 tablet_service.cc:1511] Processing CreateTablet for tablet f71a8ae43eb54833a729dffab676352c (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:42.844005 14317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71a8ae43eb54833a729dffab676352c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.852579 14316 tablet_service.cc:1511] Processing CreateTablet for tablet c1d4d250a41b4c41b7e0ee5973f1143c (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:42.854745 14316 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1d4d250a41b4c41b7e0ee5973f1143c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.869127 14241 tablet_service.cc:1511] Processing CreateTablet for tablet c1d4d250a41b4c41b7e0ee5973f1143c (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:42.870826 14241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1d4d250a41b4c41b7e0ee5973f1143c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.871202 14237 tablet_service.cc:1511] Processing CreateTablet for tablet 834228cc45dd4a64bf3883ce6244969f (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:42.872881 14237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 834228cc45dd4a64bf3883ce6244969f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.875145 14391 tablet_service.cc:1511] Processing CreateTablet for tablet f71a8ae43eb54833a729dffab676352c (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:42.875909 14240 tablet_service.cc:1511] Processing CreateTablet for tablet 6c7f424db5614fef9965b16eaae1864c (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:42.876037 14315 tablet_service.cc:1511] Processing CreateTablet for tablet 6c7f424db5614fef9965b16eaae1864c (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:42.876730 14391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71a8ae43eb54833a729dffab676352c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.877405 14240 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c7f424db5614fef9965b16eaae1864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.877179 14238 tablet_service.cc:1511] Processing CreateTablet for tablet 90fa812051104a8cba9311995042ec56 (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:42.877784 14315 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c7f424db5614fef9965b16eaae1864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.878836 14238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90fa812051104a8cba9311995042ec56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.879765 14239 tablet_service.cc:1511] Processing CreateTablet for tablet 4a68fb8a14f5495fb30dd7dd8b58d11f (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:42.881407 14239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a68fb8a14f5495fb30dd7dd8b58d11f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.884253 14242 tablet_service.cc:1511] Processing CreateTablet for tablet f71a8ae43eb54833a729dffab676352c (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:42.886014 14242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71a8ae43eb54833a729dffab676352c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.887698 14314 tablet_service.cc:1511] Processing CreateTablet for tablet 4a68fb8a14f5495fb30dd7dd8b58d11f (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:42.887990 14390 tablet_service.cc:1511] Processing CreateTablet for tablet c1d4d250a41b4c41b7e0ee5973f1143c (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:42.889585 14390 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1d4d250a41b4c41b7e0ee5973f1143c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.889566 14314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a68fb8a14f5495fb30dd7dd8b58d11f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.892705 14236 tablet_service.cc:1511] Processing CreateTablet for tablet 8279557ea37842edb1dc6a1bd65340eb (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:42.895358 14313 tablet_service.cc:1511] Processing CreateTablet for tablet 90fa812051104a8cba9311995042ec56 (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:42.895941 14235 tablet_service.cc:1511] Processing CreateTablet for tablet ed0b601cccbc41c089d18bd6924693f1 (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:42.897128 14313 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90fa812051104a8cba9311995042ec56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.894383 14236 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8279557ea37842edb1dc6a1bd65340eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.898470 14234 tablet_service.cc:1511] Processing CreateTablet for tablet 0b256c149e1d418cb673a5372afad5c4 (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:42.900096 14235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed0b601cccbc41c089d18bd6924693f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.901161 14234 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b256c149e1d418cb673a5372afad5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.928748 14312 tablet_service.cc:1511] Processing CreateTablet for tablet 834228cc45dd4a64bf3883ce6244969f (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:42.932778 14311 tablet_service.cc:1511] Processing CreateTablet for tablet 8279557ea37842edb1dc6a1bd65340eb (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:42.935074 14312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 834228cc45dd4a64bf3883ce6244969f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.939066 14389 tablet_service.cc:1511] Processing CreateTablet for tablet 6c7f424db5614fef9965b16eaae1864c (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:42.940791 14389 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c7f424db5614fef9965b16eaae1864c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.940519 14310 tablet_service.cc:1511] Processing CreateTablet for tablet ed0b601cccbc41c089d18bd6924693f1 (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:42.943038 14388 tablet_service.cc:1511] Processing CreateTablet for tablet 4a68fb8a14f5495fb30dd7dd8b58d11f (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:42.944705 14388 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a68fb8a14f5495fb30dd7dd8b58d11f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.945392 14309 tablet_service.cc:1511] Processing CreateTablet for tablet 0b256c149e1d418cb673a5372afad5c4 (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:42.947906 14387 tablet_service.cc:1511] Processing CreateTablet for tablet 90fa812051104a8cba9311995042ec56 (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:42.949364 14387 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90fa812051104a8cba9311995042ec56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.951866 14386 tablet_service.cc:1511] Processing CreateTablet for tablet 834228cc45dd4a64bf3883ce6244969f (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:42.953394 14386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 834228cc45dd4a64bf3883ce6244969f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.955861 14385 tablet_service.cc:1511] Processing CreateTablet for tablet 8279557ea37842edb1dc6a1bd65340eb (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:42.957510 14385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8279557ea37842edb1dc6a1bd65340eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.959925 14384 tablet_service.cc:1511] Processing CreateTablet for tablet ed0b601cccbc41c089d18bd6924693f1 (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:42.961457 14384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed0b601cccbc41c089d18bd6924693f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.963984 14383 tablet_service.cc:1511] Processing CreateTablet for tablet 0b256c149e1d418cb673a5372afad5c4 (DEFAULT_TABLE table=test-workload [id=c0e27c6095d8478b8a51d43f1671550a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:42.969029 14383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b256c149e1d418cb673a5372afad5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.939095 14311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8279557ea37842edb1dc6a1bd65340eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.976743 14309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b256c149e1d418cb673a5372afad5c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:42.979432 14310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed0b601cccbc41c089d18bd6924693f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:43.020867 14441 tablet_bootstrap.cc:492] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:43.045898 14441 tablet_bootstrap.cc:654] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.052691 14442 tablet_bootstrap.cc:492] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:43.059861 14442 tablet_bootstrap.cc:654] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.085426 14441 tablet_bootstrap.cc:492] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:43.085952 14441 ts_tablet_manager.cc:1403] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.072s	user 0.006s	sys 0.007s
I20260709 12:27:43.091264 14441 raft_consensus.cc:359] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.092047 14441 raft_consensus.cc:385] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.092368 14441 raft_consensus.cc:740] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.094177 14441 consensus_queue.cc:260] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.097069 14444 tablet_bootstrap.cc:492] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:43.101366 14277 heartbeater.cc:499] Master 127.29.201.254:37363 was elected leader, sending a full tablet report...
I20260709 12:27:43.101794 14441 ts_tablet_manager.cc:1434] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.016s	user 0.007s	sys 0.004s
I20260709 12:27:43.104457 14442 tablet_bootstrap.cc:492] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:43.105058 14442 ts_tablet_manager.cc:1403] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.053s	user 0.004s	sys 0.015s
I20260709 12:27:43.107059 14441 tablet_bootstrap.cc:492] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:43.112468 14442 raft_consensus.cc:359] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.113384 14442 raft_consensus.cc:385] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.113705 14442 raft_consensus.cc:740] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.114565 14444 tablet_bootstrap.cc:654] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.114781 14442 consensus_queue.cc:260] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.116384 14441 tablet_bootstrap.cc:654] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.124547 14442 ts_tablet_manager.cc:1434] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.019s	user 0.008s	sys 0.008s
I20260709 12:27:43.125433 14352 heartbeater.cc:499] Master 127.29.201.254:37363 was elected leader, sending a full tablet report...
I20260709 12:27:43.129053 14441 tablet_bootstrap.cc:492] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:43.129078 14444 tablet_bootstrap.cc:492] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:43.129943 14441 ts_tablet_manager.cc:1403] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.023s	user 0.012s	sys 0.009s
I20260709 12:27:43.130059 14444 ts_tablet_manager.cc:1403] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.033s	user 0.015s	sys 0.006s
I20260709 12:27:43.133694 14444 raft_consensus.cc:359] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.134483 14444 raft_consensus.cc:385] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.134860 14444 raft_consensus.cc:740] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.135875 14442 tablet_bootstrap.cc:492] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:43.135818 14444 consensus_queue.cc:260] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.138990 14444 ts_tablet_manager.cc:1434] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.008s	user 0.004s	sys 0.004s
I20260709 12:27:43.139840 14426 heartbeater.cc:499] Master 127.29.201.254:37363 was elected leader, sending a full tablet report...
I20260709 12:27:43.143043 14442 tablet_bootstrap.cc:654] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.144665 14441 raft_consensus.cc:359] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.145638 14441 raft_consensus.cc:385] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.145805 14444 tablet_bootstrap.cc:492] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:43.146039 14441 raft_consensus.cc:740] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.147392 14441 consensus_queue.cc:260] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.150686 14441 ts_tablet_manager.cc:1434] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.020s	user 0.005s	sys 0.002s
I20260709 12:27:43.152312 14441 tablet_bootstrap.cc:492] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:43.154531 14444 tablet_bootstrap.cc:654] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.162252 14441 tablet_bootstrap.cc:654] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.167207 14444 tablet_bootstrap.cc:492] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:43.167869 14444 ts_tablet_manager.cc:1403] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.022s	user 0.011s	sys 0.008s
I20260709 12:27:43.171212 14444 raft_consensus.cc:359] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.172060 14444 raft_consensus.cc:385] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.172389 14444 raft_consensus.cc:740] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.173596 14444 consensus_queue.cc:260] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.177008 14444 ts_tablet_manager.cc:1434] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.009s	user 0.006s	sys 0.002s
I20260709 12:27:43.178335 14444 tablet_bootstrap.cc:492] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:43.185765 14444 tablet_bootstrap.cc:654] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.195043 14441 tablet_bootstrap.cc:492] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:43.195339 14444 tablet_bootstrap.cc:492] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:43.195339 14442 tablet_bootstrap.cc:492] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:43.195573 14441 ts_tablet_manager.cc:1403] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.044s	user 0.017s	sys 0.022s
I20260709 12:27:43.196249 14444 ts_tablet_manager.cc:1403] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.018s	user 0.014s	sys 0.004s
I20260709 12:27:43.196305 14442 ts_tablet_manager.cc:1403] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.061s	user 0.030s	sys 0.018s
I20260709 12:27:43.198602 14441 raft_consensus.cc:359] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.199371 14441 raft_consensus.cc:385] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.199633 14441 raft_consensus.cc:740] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.199827 14444 raft_consensus.cc:359] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.199584 14442 raft_consensus.cc:359] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.200759 14444 raft_consensus.cc:385] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.200815 14442 raft_consensus.cc:385] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.201207 14444 raft_consensus.cc:740] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.201411 14442 raft_consensus.cc:740] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.200839 14441 consensus_queue.cc:260] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.202172 14444 consensus_queue.cc:260] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.202385 14442 consensus_queue.cc:260] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.203624 14441 ts_tablet_manager.cc:1434] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:27:43.204573 14444 ts_tablet_manager.cc:1434] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:27:43.204764 14441 tablet_bootstrap.cc:492] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:43.205694 14444 tablet_bootstrap.cc:492] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:43.209437 14442 ts_tablet_manager.cc:1434] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.000s
W20260709 12:27:43.210134 14427 tablet.cc:2406] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260709 12:27:43.210691 14442 tablet_bootstrap.cc:492] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:43.213349 14444 tablet_bootstrap.cc:654] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.218039 14441 tablet_bootstrap.cc:654] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.219647 14442 tablet_bootstrap.cc:654] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.229852 14444 tablet_bootstrap.cc:492] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:43.230414 14444 ts_tablet_manager.cc:1403] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.025s	user 0.003s	sys 0.016s
W20260709 12:27:43.232419 14198 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:43.234565 14444 raft_consensus.cc:359] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.235790 14444 raft_consensus.cc:385] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.236193 14444 raft_consensus.cc:740] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.238420 14444 consensus_queue.cc:260] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.241189 14442 tablet_bootstrap.cc:492] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:43.241189 14441 tablet_bootstrap.cc:492] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:43.242146 14441 ts_tablet_manager.cc:1403] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.038s	user 0.013s	sys 0.016s
I20260709 12:27:43.242146 14442 ts_tablet_manager.cc:1403] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.032s	user 0.025s	sys 0.000s
I20260709 12:27:43.244524 14444 ts_tablet_manager.cc:1434] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.014s	user 0.006s	sys 0.002s
I20260709 12:27:43.245708 14444 tablet_bootstrap.cc:492] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:43.245673 14441 raft_consensus.cc:359] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.246791 14441 raft_consensus.cc:385] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.246255 14442 raft_consensus.cc:359] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.247107 14441 raft_consensus.cc:740] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.247321 14442 raft_consensus.cc:385] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.248138 14442 raft_consensus.cc:740] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.248133 14441 consensus_queue.cc:260] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.249701 14442 consensus_queue.cc:260] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.253587 14450 raft_consensus.cc:493] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.253820 14441 ts_tablet_manager.cc:1434] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.011s	user 0.007s	sys 0.000s
I20260709 12:27:43.254189 14450 raft_consensus.cc:515] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.254913 14441 tablet_bootstrap.cc:492] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:43.256899 14442 ts_tablet_manager.cc:1434] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.000s
I20260709 12:27:43.257076 14450 leader_election.cc:290] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:43.258209 14442 tablet_bootstrap.cc:492] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:43.263193 14441 tablet_bootstrap.cc:654] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.267223 14444 tablet_bootstrap.cc:654] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.273411 14442 tablet_bootstrap.cc:654] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.289521 14450 raft_consensus.cc:493] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.290215 14450 raft_consensus.cc:515] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.291357 14442 tablet_bootstrap.cc:492] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:43.292060 14442 ts_tablet_manager.cc:1403] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.034s	user 0.006s	sys 0.013s
I20260709 12:27:43.292340 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8279557ea37842edb1dc6a1bd65340eb" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:43.292994 14450 leader_election.cc:290] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:43.293287 14252 raft_consensus.cc:2468] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:43.292424 14444 tablet_bootstrap.cc:492] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:43.294950 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "834228cc45dd4a64bf3883ce6244969f" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:43.294998 14444 ts_tablet_manager.cc:1403] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.050s	user 0.012s	sys 0.008s
I20260709 12:27:43.296922 14442 raft_consensus.cc:359] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.297866 14442 raft_consensus.cc:385] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.298218 14442 raft_consensus.cc:740] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.298962 14441 tablet_bootstrap.cc:492] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:43.299592 14441 ts_tablet_manager.cc:1403] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.045s	user 0.015s	sys 0.003s
I20260709 12:27:43.299361 14442 consensus_queue.cc:260] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.298799 14364 leader_election.cc:304] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: 
I20260709 12:27:43.303049 14441 raft_consensus.cc:359] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.303823 14441 raft_consensus.cc:385] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.304132 14441 raft_consensus.cc:740] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.304404 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8279557ea37842edb1dc6a1bd65340eb" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:43.305105 14441 consensus_queue.cc:260] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.305979 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "834228cc45dd4a64bf3883ce6244969f" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:43.306740 14326 raft_consensus.cc:2468] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
W20260709 12:27:43.308534 14366 leader_election.cc:343] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.308655 14463 raft_consensus.cc:493] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.309372 14366 leader_election.cc:304] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:43.309755 14441 ts_tablet_manager.cc:1434] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
W20260709 12:27:43.310532 14364 leader_election.cc:343] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.310925 14441 tablet_bootstrap.cc:492] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:43.311736 14450 raft_consensus.cc:2804] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:43.312176 14450 raft_consensus.cc:493] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.299751 14444 raft_consensus.cc:359] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.312681 14450 raft_consensus.cc:3060] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.312911 14444 raft_consensus.cc:385] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.313306 14444 raft_consensus.cc:740] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.309276 14463 raft_consensus.cc:515] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.315946 14442 ts_tablet_manager.cc:1434] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.023s	user 0.008s	sys 0.001s
I20260709 12:27:43.320657 14442 tablet_bootstrap.cc:492] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:43.320429 14450 raft_consensus.cc:515] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.322063 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71a8ae43eb54833a729dffab676352c" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:43.322778 14326 raft_consensus.cc:2468] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:43.323348 14450 leader_election.cc:290] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:43.324357 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8279557ea37842edb1dc6a1bd65340eb" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:43.325116 14252 raft_consensus.cc:3060] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.329286 14442 tablet_bootstrap.cc:654] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.318897 14441 tablet_bootstrap.cc:654] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.330724 14366 leader_election.cc:304] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:43.331131 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8279557ea37842edb1dc6a1bd65340eb" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617"
W20260709 12:27:43.332882 14366 leader_election.cc:343] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.315088 14444 consensus_queue.cc:260] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.331914 14450 raft_consensus.cc:2804] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:43.333942 14450 raft_consensus.cc:493] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.312182 14464 raft_consensus.cc:2804] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:43.334482 14450 raft_consensus.cc:3060] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.335325 14464 raft_consensus.cc:493] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.335747 14464 raft_consensus.cc:3060] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.340814 14252 raft_consensus.cc:2468] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:43.343070 14444 ts_tablet_manager.cc:1434] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.046s	user 0.009s	sys 0.000s
I20260709 12:27:43.346122 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71a8ae43eb54833a729dffab676352c" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:43.350862 14251 raft_consensus.cc:2468] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:43.352743 14464 raft_consensus.cc:515] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.353787 14444 tablet_bootstrap.cc:492] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:43.355970 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "834228cc45dd4a64bf3883ce6244969f" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:43.356654 14450 raft_consensus.cc:515] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
W20260709 12:27:43.357671 14364 leader_election.cc:343] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.347016 14463 leader_election.cc:290] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273)
I20260709 12:27:43.362305 14464 leader_election.cc:290] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:43.364957 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "834228cc45dd4a64bf3883ce6244969f" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:43.366021 14326 raft_consensus.cc:3060] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.368544 14442 tablet_bootstrap.cc:492] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:43.369391 14442 ts_tablet_manager.cc:1403] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.049s	user 0.019s	sys 0.024s
I20260709 12:27:43.369952 14447 raft_consensus.cc:493] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.370520 14447 raft_consensus.cc:515] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.378484 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71a8ae43eb54833a729dffab676352c" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:43.380270 14447 leader_election.cc:290] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:43.373445 14442 raft_consensus.cc:359] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.380854 14251 raft_consensus.cc:3060] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.380990 14442 raft_consensus.cc:385] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.381815 14463 raft_consensus.cc:493] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.382414 14463 raft_consensus.cc:515] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.383641 14364 leader_election.cc:304] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: c403bb4886574f1db5b159d81a264617
I20260709 12:27:43.384856 14463 leader_election.cc:290] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:43.385485 14463 raft_consensus.cc:2804] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.385962 14463 raft_consensus.cc:697] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Leader. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:43.387058 14250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1d4d250a41b4c41b7e0ee5973f1143c" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:43.381368 14442 raft_consensus.cc:740] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.388087 14250 raft_consensus.cc:2468] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:43.389062 14442 consensus_queue.cc:260] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.390117 14364 leader_election.cc:304] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: 
I20260709 12:27:43.391515 14467 raft_consensus.cc:2804] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:43.392032 14326 raft_consensus.cc:2468] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:43.392095 14467 raft_consensus.cc:493] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.397009 14441 tablet_bootstrap.cc:492] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:43.397245 14467 raft_consensus.cc:3060] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.397629 14441 ts_tablet_manager.cc:1403] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.087s	user 0.013s	sys 0.016s
I20260709 12:27:43.401201 14441 raft_consensus.cc:359] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.401957 14441 raft_consensus.cc:385] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.402254 14441 raft_consensus.cc:740] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.403780 14366 leader_election.cc:304] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:43.392773 14450 leader_election.cc:290] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Requested vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273)
I20260709 12:27:43.406208 14464 raft_consensus.cc:2804] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.407176 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71a8ae43eb54833a729dffab676352c" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:43.407778 14464 raft_consensus.cc:697] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Leader. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:43.394153 14442 ts_tablet_manager.cc:1434] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.024s	user 0.007s	sys 0.004s
I20260709 12:27:43.408058 14326 raft_consensus.cc:3060] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.409152 14442 tablet_bootstrap.cc:492] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:43.408782 14464 consensus_queue.cc:237] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.390898 14251 raft_consensus.cc:2468] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:43.396626 14444 tablet_bootstrap.cc:654] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.407341 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1d4d250a41b4c41b7e0ee5973f1143c" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:43.411154 14327 raft_consensus.cc:2468] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:43.412897 14441 consensus_queue.cc:260] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.395329 14463 consensus_queue.cc:237] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.416249 14364 leader_election.cc:304] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: 
I20260709 12:27:43.410192 14467 raft_consensus.cc:515] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.419111 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c7f424db5614fef9965b16eaae1864c" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:43.421294 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1d4d250a41b4c41b7e0ee5973f1143c" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:43.421763 14450 raft_consensus.cc:2804] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.422093 14252 raft_consensus.cc:3060] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.422135 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c7f424db5614fef9965b16eaae1864c" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:43.422996 14401 raft_consensus.cc:2468] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 0.
I20260709 12:27:43.423610 14442 tablet_bootstrap.cc:654] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.436108 14450 raft_consensus.cc:697] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Leader. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:43.438220 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1d4d250a41b4c41b7e0ee5973f1143c" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:43.439471 14327 raft_consensus.cc:3060] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.439833 14450 consensus_queue.cc:237] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.436120 14467 leader_election.cc:290] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:43.435452 14251 raft_consensus.cc:2468] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 0.
I20260709 12:27:43.435320 14292 leader_election.cc:304] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:43.444204 14447 raft_consensus.cc:2804] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:43.444630 14447 raft_consensus.cc:493] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.444990 14447 raft_consensus.cc:3060] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.449717 14149 catalog_manager.cc:5697] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "7310265ac7544fd281403a40bf5d6d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:43.447036 14441 ts_tablet_manager.cc:1434] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.049s	user 0.010s	sys 0.000s
I20260709 12:27:43.454254 14252 raft_consensus.cc:2468] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:43.460304 14447 raft_consensus.cc:515] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.460902 14364 leader_election.cc:304] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: 
I20260709 12:27:43.463840 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c7f424db5614fef9965b16eaae1864c" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:43.464459 14401 raft_consensus.cc:3060] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.464560 14326 raft_consensus.cc:2468] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:43.466464 14441 tablet_bootstrap.cc:492] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:43.466687 14447 leader_election.cc:290] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:43.470450 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c7f424db5614fef9965b16eaae1864c" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:43.471174 14251 raft_consensus.cc:3060] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.472286 14327 raft_consensus.cc:2468] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:43.480867 14251 raft_consensus.cc:2468] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 1.
I20260709 12:27:43.481431 14442 tablet_bootstrap.cc:492] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:43.483337 14442 ts_tablet_manager.cc:1403] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.074s	user 0.016s	sys 0.008s
I20260709 12:27:43.484894 14290 leader_election.cc:304] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:43.481439 14444 tablet_bootstrap.cc:492] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:43.481657 14474 raft_consensus.cc:2804] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.486033 14474 raft_consensus.cc:697] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Leader. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:43.486027 14444 ts_tablet_manager.cc:1403] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.133s	user 0.017s	sys 0.016s
I20260709 12:27:43.487102 14474 consensus_queue.cc:237] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.487447 14442 raft_consensus.cc:359] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.488262 14446 raft_consensus.cc:493] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.488412 14442 raft_consensus.cc:385] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.488896 14446 raft_consensus.cc:515] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.491806 14446 leader_election.cc:290] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:43.492158 14444 raft_consensus.cc:359] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.488315 14441 tablet_bootstrap.cc:654] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.493252 14444 raft_consensus.cc:385] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.495422 14447 raft_consensus.cc:2804] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.488875 14442 raft_consensus.cc:740] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.501026 14446 raft_consensus.cc:493] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.503396 14446 raft_consensus.cc:515] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.503865 14442 consensus_queue.cc:260] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.495970 14401 raft_consensus.cc:2468] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 1.
I20260709 12:27:43.508919 14446 leader_election.cc:290] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:43.501379 14444 raft_consensus.cc:740] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.517186 14444 consensus_queue.cc:260] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.523409 14444 ts_tablet_manager.cc:1434] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.037s	user 0.009s	sys 0.002s
I20260709 12:27:43.530514 14444 tablet_bootstrap.cc:492] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:43.530326 14447 raft_consensus.cc:697] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Becoming Leader. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:43.539317 14442 ts_tablet_manager.cc:1434] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.056s	user 0.011s	sys 0.005s
I20260709 12:27:43.540755 14442 tablet_bootstrap.cc:492] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:43.541097 14447 consensus_queue.cc:237] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.544795 14483 raft_consensus.cc:493] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.545379 14483 raft_consensus.cc:515] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.549696 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed0b601cccbc41c089d18bd6924693f1" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:43.550158 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed0b601cccbc41c089d18bd6924693f1" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:43.550369 14251 raft_consensus.cc:2468] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 0.
W20260709 12:27:43.551791 14292 leader_election.cc:343] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.551791 14441 tablet_bootstrap.cc:492] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:43.552788 14441 ts_tablet_manager.cc:1403] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.087s	user 0.013s	sys 0.014s
I20260709 12:27:43.552412 14290 leader_election.cc:304] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 7310265ac7544fd281403a40bf5d6d30
I20260709 12:27:43.553758 14480 raft_consensus.cc:2804] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:43.552424 14483 leader_election.cc:290] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:43.554216 14480 raft_consensus.cc:493] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.554766 14480 raft_consensus.cc:3060] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.559828 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b256c149e1d418cb673a5372afad5c4" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:43.559926 14441 raft_consensus.cc:359] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.560788 14441 raft_consensus.cc:385] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.560839 14400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed0b601cccbc41c089d18bd6924693f1" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:43.561167 14441 raft_consensus.cc:740] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
W20260709 12:27:43.561530 14217 leader_election.cc:343] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.562287 14444 tablet_bootstrap.cc:654] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.562106 14441 consensus_queue.cc:260] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.559640 14442 tablet_bootstrap.cc:654] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.567102 14441 ts_tablet_manager.cc:1434] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:27:43.567035 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b256c149e1d418cb673a5372afad5c4" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:43.568542 14441 tablet_bootstrap.cc:492] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:43.568616 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed0b601cccbc41c089d18bd6924693f1" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
W20260709 12:27:43.569918 14217 leader_election.cc:343] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.570566 14217 leader_election.cc:304] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f; no voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617
I20260709 12:27:43.571704 14446 raft_consensus.cc:2749] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:43.571053 14149 catalog_manager.cc:5697] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "7310265ac7544fd281403a40bf5d6d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:43.572342 14217 leader_election.cc:343] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.573532 14149 catalog_manager.cc:5697] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "7310265ac7544fd281403a40bf5d6d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:43.576009 14444 tablet_bootstrap.cc:492] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:43.576902 14444 ts_tablet_manager.cc:1403] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.052s	user 0.014s	sys 0.000s
I20260709 12:27:43.577312 14441 tablet_bootstrap.cc:654] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.576187 14149 catalog_manager.cc:5697] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "7310265ac7544fd281403a40bf5d6d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:43.581640 14442 tablet_bootstrap.cc:492] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:43.581017 14444 raft_consensus.cc:359] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.582234 14442 ts_tablet_manager.cc:1403] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.042s	user 0.011s	sys 0.004s
I20260709 12:27:43.585806 14442 raft_consensus.cc:359] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.582172 14444 raft_consensus.cc:385] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.586589 14442 raft_consensus.cc:385] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.586783 14444 raft_consensus.cc:740] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.589738 14442 raft_consensus.cc:740] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.590286 14447 raft_consensus.cc:493] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.591018 14447 raft_consensus.cc:515] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.591338 14442 consensus_queue.cc:260] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.594208 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a68fb8a14f5495fb30dd7dd8b58d11f" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:43.594601 14442 ts_tablet_manager.cc:1434] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.012s	user 0.004s	sys 0.002s
W20260709 12:27:43.595803 14290 leader_election.cc:343] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.596197 14442 tablet_bootstrap.cc:492] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:43.596782 14147 catalog_manager.cc:5697] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 reported cstate change: term changed from 0 to 1, leader changed from <none> to c403bb4886574f1db5b159d81a264617 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "c403bb4886574f1db5b159d81a264617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:43.603194 14480 raft_consensus.cc:515] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.604710 14326 raft_consensus.cc:2393] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bd3bfb90591a4b8c95c500d57be3d44f in current term 1: Already voted for candidate c403bb4886574f1db5b159d81a264617 in this term.
I20260709 12:27:43.607141 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed0b601cccbc41c089d18bd6924693f1" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:43.607642 14400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a68fb8a14f5495fb30dd7dd8b58d11f" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:43.607854 14251 raft_consensus.cc:3060] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.608752 14217 leader_election.cc:304] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f; no voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617
I20260709 12:27:43.609356 14447 leader_election.cc:290] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:43.613274 14442 tablet_bootstrap.cc:654] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
W20260709 12:27:43.618505 14292 leader_election.cc:343] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:43.619179 14292 leader_election.cc:304] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c403bb4886574f1db5b159d81a264617; no voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:43.620564 14447 raft_consensus.cc:2749] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:43.621426 14444 consensus_queue.cc:260] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.624022 14444 ts_tablet_manager.cc:1434] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.047s	user 0.011s	sys 0.000s
I20260709 12:27:43.625280 14444 tablet_bootstrap.cc:492] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:43.626375 14480 leader_election.cc:290] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:43.628055 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ed0b601cccbc41c089d18bd6924693f1" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:43.629138 14401 raft_consensus.cc:3060] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.633222 14251 raft_consensus.cc:2468] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 1.
I20260709 12:27:43.633881 14446 raft_consensus.cc:2749] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:43.638536 14442 tablet_bootstrap.cc:492] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:43.639271 14442 ts_tablet_manager.cc:1403] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.043s	user 0.016s	sys 0.017s
I20260709 12:27:43.639174 14290 leader_election.cc:304] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:43.640364 14480 raft_consensus.cc:2804] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.641086 14480 raft_consensus.cc:697] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Becoming Leader. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:43.641970 14480 consensus_queue.cc:237] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.642494 14441 tablet_bootstrap.cc:492] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:43.643187 14441 ts_tablet_manager.cc:1403] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.075s	user 0.014s	sys 0.004s
I20260709 12:27:43.642709 14442 raft_consensus.cc:359] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.643612 14442 raft_consensus.cc:385] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.643993 14442 raft_consensus.cc:740] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.644843 14442 consensus_queue.cc:260] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.645159 14401 raft_consensus.cc:2468] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 1.
I20260709 12:27:43.647398 14442 ts_tablet_manager.cc:1434] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.000s
I20260709 12:27:43.648419 14442 tablet_bootstrap.cc:492] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:43.652423 14444 tablet_bootstrap.cc:654] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.654542 14442 tablet_bootstrap.cc:654] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.655602 14441 raft_consensus.cc:359] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.657474 14441 raft_consensus.cc:385] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.657790 14441 raft_consensus.cc:740] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.658598 14441 consensus_queue.cc:260] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.661149 14441 ts_tablet_manager.cc:1434] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.014s	user 0.001s	sys 0.004s
I20260709 12:27:43.665129 14147 catalog_manager.cc:5697] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 reported cstate change: term changed from 0 to 1, leader changed from <none> to c403bb4886574f1db5b159d81a264617 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "c403bb4886574f1db5b159d81a264617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:43.668735 14442 tablet_bootstrap.cc:492] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:43.669394 14442 ts_tablet_manager.cc:1403] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.021s	user 0.017s	sys 0.000s
I20260709 12:27:43.672489 14442 raft_consensus.cc:359] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.673378 14442 raft_consensus.cc:385] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.673705 14442 raft_consensus.cc:740] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.674326 14441 tablet_bootstrap.cc:492] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:43.674675 14442 consensus_queue.cc:260] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.677098 14442 ts_tablet_manager.cc:1434] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.000s
I20260709 12:27:43.679422 14474 raft_consensus.cc:493] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.680028 14474 raft_consensus.cc:515] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.681910 14444 tablet_bootstrap.cc:492] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:43.682579 14444 ts_tablet_manager.cc:1403] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.058s	user 0.016s	sys 0.019s
I20260709 12:27:43.684396 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90fa812051104a8cba9311995042ec56" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:43.685161 14326 raft_consensus.cc:2468] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:43.685866 14444 raft_consensus.cc:359] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.686687 14444 raft_consensus.cc:385] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.686781 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90fa812051104a8cba9311995042ec56" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:43.687042 14444 raft_consensus.cc:740] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.687392 14441 tablet_bootstrap.cc:654] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.687492 14251 raft_consensus.cc:2468] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:43.688086 14444 consensus_queue.cc:260] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.689076 14366 leader_election.cc:304] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:43.690059 14450 raft_consensus.cc:2804] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:43.690450 14450 raft_consensus.cc:493] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.691619 14450 raft_consensus.cc:3060] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.693409 14444 ts_tablet_manager.cc:1434] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.010s	user 0.006s	sys 0.000s
I20260709 12:27:43.695209 14444 tablet_bootstrap.cc:492] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:43.682336 14474 leader_election.cc:290] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:43.701355 14444 tablet_bootstrap.cc:654] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:43.701664 14450 raft_consensus.cc:515] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.705250 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90fa812051104a8cba9311995042ec56" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:43.705919 14251 raft_consensus.cc:3060] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.712379 14450 leader_election.cc:290] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:43.713403 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90fa812051104a8cba9311995042ec56" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:43.714411 14326 raft_consensus.cc:3060] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.715963 14441 tablet_bootstrap.cc:492] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:43.716573 14441 ts_tablet_manager.cc:1403] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.042s	user 0.022s	sys 0.013s
I20260709 12:27:43.718451 14444 tablet_bootstrap.cc:492] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:43.719133 14444 ts_tablet_manager.cc:1403] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.024s	user 0.006s	sys 0.011s
I20260709 12:27:43.719698 14441 raft_consensus.cc:359] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.720500 14441 raft_consensus.cc:385] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.720788 14441 raft_consensus.cc:740] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.721513 14441 consensus_queue.cc:260] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.722249 14326 raft_consensus.cc:2468] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:43.723682 14441 ts_tablet_manager.cc:1434] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.000s
I20260709 12:27:43.723745 14366 leader_election.cc:304] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:43.724459 14251 raft_consensus.cc:2468] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:43.722271 14444 raft_consensus.cc:359] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.725096 14450 raft_consensus.cc:2804] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.725543 14444 raft_consensus.cc:385] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:43.725757 14450 raft_consensus.cc:697] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Leader. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:43.725863 14444 raft_consensus.cc:740] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:43.726837 14450 consensus_queue.cc:237] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.726857 14444 consensus_queue.cc:260] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.729274 14444 ts_tablet_manager.cc:1434] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.010s	user 0.007s	sys 0.000s
I20260709 12:27:43.738127 14146 catalog_manager.cc:5697] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "7310265ac7544fd281403a40bf5d6d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:43.768477 14450 raft_consensus.cc:493] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.769088 14450 raft_consensus.cc:515] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.771144 14450 leader_election.cc:290] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:43.772276 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a68fb8a14f5495fb30dd7dd8b58d11f" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:43.772099 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a68fb8a14f5495fb30dd7dd8b58d11f" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:43.773130 14326 raft_consensus.cc:2468] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:43.773211 14251 raft_consensus.cc:2468] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:43.774328 14366 leader_election.cc:304] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:43.775256 14450 raft_consensus.cc:2804] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:43.775638 14450 raft_consensus.cc:493] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.775892 14450 raft_consensus.cc:3060] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.781490 14450 raft_consensus.cc:515] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.783350 14450 leader_election.cc:290] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:43.784152 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a68fb8a14f5495fb30dd7dd8b58d11f" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:43.784356 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a68fb8a14f5495fb30dd7dd8b58d11f" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:43.784847 14251 raft_consensus.cc:3060] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.785002 14326 raft_consensus.cc:3060] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.792312 14326 raft_consensus.cc:2468] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:43.792765 14251 raft_consensus.cc:2468] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:43.793543 14366 leader_election.cc:304] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:43.794587 14450 raft_consensus.cc:2804] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:43.795223 14450 raft_consensus.cc:697] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Leader. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:43.796066 14450 consensus_queue.cc:237] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.805068 14146 catalog_manager.cc:5697] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "7310265ac7544fd281403a40bf5d6d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:43.832484 14480 raft_consensus.cc:493] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.833195 14480 raft_consensus.cc:515] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.835249 14480 leader_election.cc:290] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:43.836095 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8279557ea37842edb1dc6a1bd65340eb" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:43.836339 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8279557ea37842edb1dc6a1bd65340eb" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:43.836967 14251 raft_consensus.cc:2393] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c403bb4886574f1db5b159d81a264617 in current term 1: Already voted for candidate 7310265ac7544fd281403a40bf5d6d30 in this term.
I20260709 12:27:43.837940 14290 leader_election.cc:304] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c403bb4886574f1db5b159d81a264617; no voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:43.838706 14480 raft_consensus.cc:3060] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.844743 14480 raft_consensus.cc:2749] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:43.848289 14450 consensus_queue.cc:1048] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:43.865864 14474 consensus_queue.cc:1048] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:43.885784 14474 consensus_queue.cc:1048] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:43.898533 14474 consensus_queue.cc:1048] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:43.917819 14251 raft_consensus.cc:3060] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:43.925221 14474 consensus_queue.cc:1048] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:43.941032 14474 consensus_queue.cc:1048] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:43.991932 14480 raft_consensus.cc:493] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:43.992545 14480 raft_consensus.cc:515] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:43.994719 14480 leader_election.cc:290] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:43.995429 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b256c149e1d418cb673a5372afad5c4" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:43.995841 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b256c149e1d418cb673a5372afad5c4" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:43.996302 14251 raft_consensus.cc:2468] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 0.
I20260709 12:27:43.996533 14401 raft_consensus.cc:2468] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 0.
I20260709 12:27:43.997507 14290 leader_election.cc:304] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:43.998317 14480 raft_consensus.cc:2804] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:43.998790 14480 raft_consensus.cc:493] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:43.999063 14480 raft_consensus.cc:3060] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.004907 14480 raft_consensus.cc:515] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.007052 14480 leader_election.cc:290] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:44.007745 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b256c149e1d418cb673a5372afad5c4" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:44.008164 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b256c149e1d418cb673a5372afad5c4" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:44.008505 14251 raft_consensus.cc:3060] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.008800 14401 raft_consensus.cc:3060] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:44.013284 14474 consensus_queue.cc:1048] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:44.016345 14251 raft_consensus.cc:2468] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 1.
I20260709 12:27:44.016342 14401 raft_consensus.cc:2468] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 1.
I20260709 12:27:44.018466 14290 leader_election.cc:304] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:44.021150 14480 raft_consensus.cc:2804] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:44.021648 14480 raft_consensus.cc:697] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Becoming Leader. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:44.024724 14474 consensus_queue.cc:1048] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:44.026517 14480 consensus_queue.cc:237] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.037832 14146 catalog_manager.cc:5697] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 reported cstate change: term changed from 0 to 1, leader changed from <none> to c403bb4886574f1db5b159d81a264617 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "c403bb4886574f1db5b159d81a264617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:44.078987 14480 consensus_queue.cc:1048] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:44.091233 14447 consensus_queue.cc:1048] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:44.111280 14474 consensus_queue.cc:1048] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:44.123651 14474 consensus_queue.cc:1048] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:44.150508 14447 consensus_queue.cc:1048] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:44.166556 14447 consensus_queue.cc:1048] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:44.238796 14474 consensus_queue.cc:1048] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:44.252671 14450 consensus_queue.cc:1048] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:44.293227 14401 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "f71a8ae43eb54833a729dffab676352c"
dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
mode: GRACEFUL
new_leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:54634
I20260709 12:27:44.293984 14401 raft_consensus.cc:606] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Received request to transfer leadership to TS bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:44.300179 14401 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c1d4d250a41b4c41b7e0ee5973f1143c"
dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
mode: GRACEFUL
new_leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:54634
I20260709 12:27:44.301064 14401 raft_consensus.cc:606] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Received request to transfer leadership to TS bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:44.303707 14467 raft_consensus.cc:993] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30: : Instructing follower bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:44.304214 14450 raft_consensus.cc:1081] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Signalling peer bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:44.305920 14251 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "f71a8ae43eb54833a729dffab676352c"
dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:55618
I20260709 12:27:44.306715 14251 raft_consensus.cc:493] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:44.307147 14251 raft_consensus.cc:3060] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.313689 14251 raft_consensus.cc:515] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.315821 14251 leader_election.cc:290] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Requested vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:44.316931 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71a8ae43eb54833a729dffab676352c" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:44.317340 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f71a8ae43eb54833a729dffab676352c" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:44.317740 14326 raft_consensus.cc:3060] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.318159 14401 raft_consensus.cc:3055] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:44.318600 14401 raft_consensus.cc:740] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:44.319655 14401 consensus_queue.cc:260] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.319942 14327 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6c7f424db5614fef9965b16eaae1864c"
dest_uuid: "c403bb4886574f1db5b159d81a264617"
mode: GRACEFUL
new_leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:47642
I20260709 12:27:44.320699 14327 raft_consensus.cc:606] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Received request to transfer leadership to TS bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:44.320957 14401 raft_consensus.cc:3060] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.326231 14326 raft_consensus.cc:2468] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:44.327725 14217 leader_election.cc:304] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:44.328141 14401 raft_consensus.cc:2468] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:44.328272 14400 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "4a68fb8a14f5495fb30dd7dd8b58d11f"
dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
mode: GRACEFUL
new_leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:54634
I20260709 12:27:44.328754 14446 raft_consensus.cc:2804] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:44.329269 14400 raft_consensus.cc:606] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Received request to transfer leadership to TS bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:44.330468 14446 raft_consensus.cc:697] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Becoming Leader. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Running, Role: LEADER
I20260709 12:27:44.331575 14446 consensus_queue.cc:237] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.336380 14400 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90fa812051104a8cba9311995042ec56"
dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
mode: GRACEFUL
new_leader_uuid: "c403bb4886574f1db5b159d81a264617"
 from {username='slave'} at 127.0.0.1:54634
I20260709 12:27:44.337195 14400 raft_consensus.cc:606] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Received request to transfer leadership to TS c403bb4886574f1db5b159d81a264617
I20260709 12:27:44.339969 14149 catalog_manager.cc:5697] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f reported cstate change: term changed from 1 to 2, leader changed from 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195) to bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193). New cstate: current_term: 2 leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:44.343739 14400 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "834228cc45dd4a64bf3883ce6244969f"
dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
mode: GRACEFUL
new_leader_uuid: "c403bb4886574f1db5b159d81a264617"
 from {username='slave'} at 127.0.0.1:54634
I20260709 12:27:44.344480 14400 raft_consensus.cc:606] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Received request to transfer leadership to TS c403bb4886574f1db5b159d81a264617
I20260709 12:27:44.350914 14400 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "8279557ea37842edb1dc6a1bd65340eb"
dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
mode: GRACEFUL
new_leader_uuid: "c403bb4886574f1db5b159d81a264617"
 from {username='slave'} at 127.0.0.1:54634
I20260709 12:27:44.351519 14400 raft_consensus.cc:606] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Received request to transfer leadership to TS c403bb4886574f1db5b159d81a264617
I20260709 12:27:44.359827 14326 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "0b256c149e1d418cb673a5372afad5c4"
dest_uuid: "c403bb4886574f1db5b159d81a264617"
mode: GRACEFUL
new_leader_uuid: "7310265ac7544fd281403a40bf5d6d30"
 from {username='slave'} at 127.0.0.1:47642
I20260709 12:27:44.360460 14326 raft_consensus.cc:606] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Received request to transfer leadership to TS 7310265ac7544fd281403a40bf5d6d30
I20260709 12:27:44.542502 14447 consensus_queue.cc:1048] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:44.550678 14480 raft_consensus.cc:993] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617: : Instructing follower 7310265ac7544fd281403a40bf5d6d30 to start an election
I20260709 12:27:44.551272 14517 raft_consensus.cc:1081] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Signalling peer 7310265ac7544fd281403a40bf5d6d30 to start an election
I20260709 12:27:44.553192 14401 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "0b256c149e1d418cb673a5372afad5c4"
dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
 from {username='slave'} at 127.0.0.1:54622
I20260709 12:27:44.556077 14401 raft_consensus.cc:493] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:44.556525 14401 raft_consensus.cc:3060] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.556861 14517 consensus_queue.cc:1048] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:44.563848 14401 raft_consensus.cc:515] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.570811 14401 leader_election.cc:290] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 2 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:44.573299 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b256c149e1d418cb673a5372afad5c4" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:44.574090 14252 raft_consensus.cc:3060] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.575835 14464 raft_consensus.cc:993] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30: : Instructing follower bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:44.576498 14464 raft_consensus.cc:1081] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Signalling peer bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:44.577121 14474 raft_consensus.cc:993] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30: : Instructing follower c403bb4886574f1db5b159d81a264617 to start an election
I20260709 12:27:44.577646 14474 raft_consensus.cc:1081] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Signalling peer c403bb4886574f1db5b159d81a264617 to start an election
I20260709 12:27:44.576004 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0b256c149e1d418cb673a5372afad5c4" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:44.578574 14251 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c1d4d250a41b4c41b7e0ee5973f1143c"
dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:55618
I20260709 12:27:44.578936 14326 raft_consensus.cc:3055] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:44.579314 14326 raft_consensus.cc:740] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:44.579314 14251 raft_consensus.cc:493] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:44.579996 14251 raft_consensus.cc:3060] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.580276 14326 consensus_queue.cc:260] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.581847 14326 raft_consensus.cc:3060] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.583072 14252 raft_consensus.cc:2468] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 2.
I20260709 12:27:44.584542 14364 leader_election.cc:304] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: 
I20260709 12:27:44.585515 14474 raft_consensus.cc:2804] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:44.586112 14474 raft_consensus.cc:697] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [term 2 LEADER]: Becoming Leader. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:44.587015 14474 consensus_queue.cc:237] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.587518 14327 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "834228cc45dd4a64bf3883ce6244969f"
dest_uuid: "c403bb4886574f1db5b159d81a264617"
 from {username='slave'} at 127.0.0.1:47618
I20260709 12:27:44.588264 14327 raft_consensus.cc:493] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:44.588687 14327 raft_consensus.cc:3060] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.588543 14251 raft_consensus.cc:515] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.592123 14325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1d4d250a41b4c41b7e0ee5973f1143c" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:44.592854 14325 raft_consensus.cc:3060] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.593631 14400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1d4d250a41b4c41b7e0ee5973f1143c" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:44.594317 14400 raft_consensus.cc:3055] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:44.594674 14400 raft_consensus.cc:740] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:44.595314 14251 leader_election.cc:290] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Requested vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:44.595572 14400 consensus_queue.cc:260] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.597002 14400 raft_consensus.cc:3060] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.597363 14327 raft_consensus.cc:515] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.600719 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "834228cc45dd4a64bf3883ce6244969f" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:44.601354 14251 raft_consensus.cc:3060] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.602154 14326 raft_consensus.cc:2468] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 2.
I20260709 12:27:44.605393 14400 raft_consensus.cc:2468] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:44.606611 14327 leader_election.cc:290] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 2 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:44.607424 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "834228cc45dd4a64bf3883ce6244969f" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:44.613126 14251 raft_consensus.cc:2468] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 2.
I20260709 12:27:44.606781 14217 leader_election.cc:304] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: 
I20260709 12:27:44.614455 14401 raft_consensus.cc:3055] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:44.614872 14446 raft_consensus.cc:2804] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:44.605401 14149 catalog_manager.cc:5697] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 reported cstate change: term changed from 1 to 2, leader changed from c403bb4886574f1db5b159d81a264617 (127.29.201.194) to 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195). New cstate: current_term: 2 leader_uuid: "7310265ac7544fd281403a40bf5d6d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:44.613126 14325 raft_consensus.cc:2468] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:44.615612 14290 leader_election.cc:304] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:44.615505 14446 raft_consensus.cc:697] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Becoming Leader. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Running, Role: LEADER
I20260709 12:27:44.616659 14517 raft_consensus.cc:2804] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:44.617192 14517 raft_consensus.cc:697] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [term 2 LEADER]: Becoming Leader. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:44.617501 14446 consensus_queue.cc:237] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.618073 14517 consensus_queue.cc:237] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.614883 14401 raft_consensus.cc:740] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:44.626444 14401 consensus_queue.cc:260] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.627942 14401 raft_consensus.cc:3060] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.628153 14146 catalog_manager.cc:5697] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f reported cstate change: term changed from 1 to 2, leader changed from 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195) to bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193). New cstate: current_term: 2 leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:44.631702 14147 catalog_manager.cc:5697] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 reported cstate change: term changed from 1 to 2, leader changed from 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195) to c403bb4886574f1db5b159d81a264617 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "c403bb4886574f1db5b159d81a264617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:44.635954 14401 raft_consensus.cc:2468] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 2.
I20260709 12:27:44.702803 14447 raft_consensus.cc:993] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617: : Instructing follower bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:44.703351 14517 raft_consensus.cc:1081] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Signalling peer bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:44.705181 14251 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6c7f424db5614fef9965b16eaae1864c"
dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:55626
I20260709 12:27:44.705792 14251 raft_consensus.cc:493] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:44.706167 14251 raft_consensus.cc:3060] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.713079 14251 raft_consensus.cc:515] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.715420 14251 leader_election.cc:290] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Requested vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:44.716373 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c7f424db5614fef9965b16eaae1864c" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:44.716521 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c7f424db5614fef9965b16eaae1864c" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:44.716879 14467 raft_consensus.cc:993] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30: : Instructing follower bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:44.717962 14327 raft_consensus.cc:3055] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:44.718125 14474 raft_consensus.cc:993] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30: : Instructing follower c403bb4886574f1db5b159d81a264617 to start an election
I20260709 12:27:44.718490 14467 raft_consensus.cc:1081] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Signalling peer bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:44.718698 14327 raft_consensus.cc:740] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:44.719233 14474 raft_consensus.cc:1081] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Signalling peer c403bb4886574f1db5b159d81a264617 to start an election
I20260709 12:27:44.720158 14327 consensus_queue.cc:260] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.720829 14251 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "4a68fb8a14f5495fb30dd7dd8b58d11f"
dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:55618
I20260709 12:27:44.721156 14325 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "90fa812051104a8cba9311995042ec56"
dest_uuid: "c403bb4886574f1db5b159d81a264617"
 from {username='slave'} at 127.0.0.1:47618
I20260709 12:27:44.721787 14251 raft_consensus.cc:493] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:44.721843 14325 raft_consensus.cc:493] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:44.722311 14251 raft_consensus.cc:3060] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.722386 14325 raft_consensus.cc:3060] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.717965 14401 raft_consensus.cc:3060] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.722455 14327 raft_consensus.cc:3060] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.730306 14325 raft_consensus.cc:515] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.730306 14251 raft_consensus.cc:515] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.734987 14400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a68fb8a14f5495fb30dd7dd8b58d11f" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:44.735041 14401 raft_consensus.cc:2468] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:44.735667 14400 raft_consensus.cc:3055] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:44.736040 14400 raft_consensus.cc:740] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:44.736548 14251 leader_election.cc:290] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Requested vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:44.736928 14400 consensus_queue.cc:260] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.737758 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4a68fb8a14f5495fb30dd7dd8b58d11f" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:44.738206 14400 raft_consensus.cc:3060] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.738525 14326 raft_consensus.cc:3060] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.738473 14217 leader_election.cc:304] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: 
I20260709 12:27:44.739679 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90fa812051104a8cba9311995042ec56" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:44.740077 14446 raft_consensus.cc:2804] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:44.740393 14401 raft_consensus.cc:3055] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:44.740670 14446 raft_consensus.cc:697] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Becoming Leader. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Running, Role: LEADER
I20260709 12:27:44.740837 14401 raft_consensus.cc:740] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:44.741778 14325 leader_election.cc:290] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 2 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:44.741626 14446 consensus_queue.cc:237] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.743016 14401 consensus_queue.cc:260] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.744489 14401 raft_consensus.cc:3060] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.747304 14400 raft_consensus.cc:2468] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:44.737687 14327 raft_consensus.cc:2468] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:44.748143 14326 raft_consensus.cc:2468] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:44.737777 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90fa812051104a8cba9311995042ec56" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:44.750003 14251 raft_consensus.cc:3060] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.750774 14217 leader_election.cc:304] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:44.752362 14446 raft_consensus.cc:2804] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:44.753116 14446 raft_consensus.cc:697] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Becoming Leader. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Running, Role: LEADER
I20260709 12:27:44.752435 14147 catalog_manager.cc:5697] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f reported cstate change: term changed from 1 to 2, leader changed from c403bb4886574f1db5b159d81a264617 (127.29.201.194) to bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193). New cstate: current_term: 2 leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:44.754022 14446 consensus_queue.cc:237] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.759140 14401 raft_consensus.cc:2468] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 2.
I20260709 12:27:44.759140 14251 raft_consensus.cc:2468] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 2.
I20260709 12:27:44.760911 14290 leader_election.cc:304] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:44.762046 14517 raft_consensus.cc:2804] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:44.762509 14517 raft_consensus.cc:697] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 2 LEADER]: Becoming Leader. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:44.763384 14517 consensus_queue.cc:237] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.768590 14474 raft_consensus.cc:993] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30: : Instructing follower c403bb4886574f1db5b159d81a264617 to start an election
I20260709 12:27:44.769147 14450 raft_consensus.cc:1081] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Signalling peer c403bb4886574f1db5b159d81a264617 to start an election
I20260709 12:27:44.771005 14327 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "8279557ea37842edb1dc6a1bd65340eb"
dest_uuid: "c403bb4886574f1db5b159d81a264617"
 from {username='slave'} at 127.0.0.1:47618
I20260709 12:27:44.771575 14327 raft_consensus.cc:493] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:44.772017 14327 raft_consensus.cc:3060] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.771669 14146 catalog_manager.cc:5697] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f reported cstate change: term changed from 1 to 2, leader changed from 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195) to bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193). New cstate: current_term: 2 leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:44.771682 14147 catalog_manager.cc:5697] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 reported cstate change: term changed from 1 to 2, leader changed from 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195) to c403bb4886574f1db5b159d81a264617 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "c403bb4886574f1db5b159d81a264617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:44.774683 14326 raft_consensus.cc:1275] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:44.776450 14446 consensus_queue.cc:1048] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:44.782531 14327 raft_consensus.cc:515] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.785816 14327 leader_election.cc:290] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 2 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:44.787606 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8279557ea37842edb1dc6a1bd65340eb" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:44.788506 14401 raft_consensus.cc:3055] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:44.788563 14400 raft_consensus.cc:1275] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:44.789104 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8279557ea37842edb1dc6a1bd65340eb" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:44.789212 14401 raft_consensus.cc:740] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:44.789863 14251 raft_consensus.cc:3060] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.790591 14401 consensus_queue.cc:260] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.792474 14401 raft_consensus.cc:3060] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:44.793835 14446 consensus_queue.cc:1048] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:44.798676 14251 raft_consensus.cc:2468] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 2.
I20260709 12:27:44.800490 14290 leader_election.cc:304] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:44.801455 14517 raft_consensus.cc:2804] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:44.801997 14517 raft_consensus.cc:697] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 2 LEADER]: Becoming Leader. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:44.803185 14517 consensus_queue.cc:237] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:44.804177 14401 raft_consensus.cc:2468] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 2.
I20260709 12:27:44.812009 14147 catalog_manager.cc:5697] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 reported cstate change: term changed from 1 to 2, leader changed from 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195) to c403bb4886574f1db5b159d81a264617 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "c403bb4886574f1db5b159d81a264617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:44.972416 14251 raft_consensus.cc:1275] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Refusing update from remote peer 7310265ac7544fd281403a40bf5d6d30: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:44.973893 14450 consensus_queue.cc:1048] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:44.982491 14327 raft_consensus.cc:1275] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Refusing update from remote peer 7310265ac7544fd281403a40bf5d6d30: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:44.984480 14450 consensus_queue.cc:1048] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:45.026895 14400 raft_consensus.cc:1275] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Refusing update from remote peer c403bb4886574f1db5b159d81a264617: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.028316 14517 consensus_queue.cc:1048] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:45.038024 14251 raft_consensus.cc:1275] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Refusing update from remote peer c403bb4886574f1db5b159d81a264617: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.040370 14447 consensus_queue.cc:1048] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:45.099992 14327 raft_consensus.cc:1275] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.101363 14524 consensus_queue.cc:1048] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:45.111595 14400 raft_consensus.cc:1275] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.113488 14446 consensus_queue.cc:1048] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:45.165021 14327 raft_consensus.cc:1275] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.166551 14446 consensus_queue.cc:1048] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:45.175940 14400 raft_consensus.cc:1275] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.178540 14327 raft_consensus.cc:1275] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.179977 14446 consensus_queue.cc:1048] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:45.181402 14524 consensus_queue.cc:1048] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:45.196823 14400 raft_consensus.cc:1275] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.198514 14446 consensus_queue.cc:1048] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:45.288142 14400 raft_consensus.cc:1275] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Refusing update from remote peer c403bb4886574f1db5b159d81a264617: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.289927 14447 consensus_queue.cc:1048] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:45.299232 14251 raft_consensus.cc:1275] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Refusing update from remote peer c403bb4886574f1db5b159d81a264617: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.300961 14517 consensus_queue.cc:1048] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:45.410672 14400 raft_consensus.cc:1275] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Refusing update from remote peer c403bb4886574f1db5b159d81a264617: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.412099 14517 consensus_queue.cc:1048] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:45.418497 14251 raft_consensus.cc:1275] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Refusing update from remote peer c403bb4886574f1db5b159d81a264617: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:45.420156 14447 consensus_queue.cc:1048] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:47.389449 14146 catalog_manager.cc:2283] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47014:
name: "soft_deleted_table"
schema {
  columns {
    name: "key"
    type: INT32
    is_key: true
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "int_val"
    type: INT32
    is_key: false
    is_nullable: false
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
  columns {
    name: "string_val"
    type: STRING
    is_key: false
    is_nullable: true
    encoding: AUTO_ENCODING
    compression: DEFAULT_COMPRESSION
    cfile_block_size: 0
    immutable: false
  }
}
num_replicas: 3
split_rows_range_bounds {
  rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
  indirect_data: """"
}
partition_schema {
  range_schema {
    columns {
      name: "key"
    }
  }
}
W20260709 12:27:47.392246 14146 catalog_manager.cc:7077] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table soft_deleted_table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260709 12:27:47.446539 14237 tablet_service.cc:1511] Processing CreateTablet for tablet 6a3327f378e64cbda03898ceaa6701a0 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:47.447968 14237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a3327f378e64cbda03898ceaa6701a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.447827 14311 tablet_service.cc:1511] Processing CreateTablet for tablet 6a3327f378e64cbda03898ceaa6701a0 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:47.449256 14311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a3327f378e64cbda03898ceaa6701a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.450605 14239 tablet_service.cc:1511] Processing CreateTablet for tablet 607b3a3e5d56487d815d403fa1373cfc (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:47.452132 14239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607b3a3e5d56487d815d403fa1373cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.457640 14309 tablet_service.cc:1511] Processing CreateTablet for tablet 607b3a3e5d56487d815d403fa1373cfc (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:47.458833 14238 tablet_service.cc:1511] Processing CreateTablet for tablet c8038818270e47f1abba0a9a3dcab848 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:47.459489 14309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607b3a3e5d56487d815d403fa1373cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.460228 14238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8038818270e47f1abba0a9a3dcab848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.457909 14310 tablet_service.cc:1511] Processing CreateTablet for tablet c8038818270e47f1abba0a9a3dcab848 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:47.463454 14310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8038818270e47f1abba0a9a3dcab848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.481178 14235 tablet_service.cc:1511] Processing CreateTablet for tablet d2d0355002344a8086f697db621c2d65 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:47.482252 14313 tablet_service.cc:1511] Processing CreateTablet for tablet d2d0355002344a8086f697db621c2d65 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:47.482860 14235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2d0355002344a8086f697db621c2d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.483747 14313 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2d0355002344a8086f697db621c2d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.484556 14552 tablet_bootstrap.cc:492] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:47.490829 14383 tablet_service.cc:1511] Processing CreateTablet for tablet 6a3327f378e64cbda03898ceaa6701a0 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260709 12:27:47.491371 14388 tablet_service.cc:1511] Processing CreateTablet for tablet 607b3a3e5d56487d815d403fa1373cfc (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260709 12:27:47.492391 14383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a3327f378e64cbda03898ceaa6701a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.493264 14388 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607b3a3e5d56487d815d403fa1373cfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.494498 14384 tablet_service.cc:1511] Processing CreateTablet for tablet c8038818270e47f1abba0a9a3dcab848 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260709 12:27:47.495913 14384 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8038818270e47f1abba0a9a3dcab848. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.510756 14239 tablet_service.cc:1511] Processing CreateTablet for tablet ab7958026c31429697818f80570d4314 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:47.512346 14239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7958026c31429697818f80570d4314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.516852 14314 tablet_service.cc:1511] Processing CreateTablet for tablet ab7958026c31429697818f80570d4314 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:47.518965 14314 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7958026c31429697818f80570d4314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.520548 14552 tablet_bootstrap.cc:654] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.522054 14387 tablet_service.cc:1511] Processing CreateTablet for tablet d2d0355002344a8086f697db621c2d65 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260709 12:27:47.523667 14387 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2d0355002344a8086f697db621c2d65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.540575 14391 tablet_service.cc:1511] Processing CreateTablet for tablet ab7958026c31429697818f80570d4314 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260709 12:27:47.542709 14391 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab7958026c31429697818f80570d4314. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.545912 14312 tablet_service.cc:1511] Processing CreateTablet for tablet 801bc4a961d84dc38e9ead3e902346d1 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:47.547506 14312 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 801bc4a961d84dc38e9ead3e902346d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.559798 14238 tablet_service.cc:1511] Processing CreateTablet for tablet 801bc4a961d84dc38e9ead3e902346d1 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:47.561385 14238 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 801bc4a961d84dc38e9ead3e902346d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.571274 14556 tablet_bootstrap.cc:492] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:47.580268 14389 tablet_service.cc:1511] Processing CreateTablet for tablet 801bc4a961d84dc38e9ead3e902346d1 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260709 12:27:47.581997 14389 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 801bc4a961d84dc38e9ead3e902346d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.583719 14386 tablet_service.cc:1511] Processing CreateTablet for tablet b72d6bd4be824dceb80f1bb341feaf77 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:47.585166 14386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b72d6bd4be824dceb80f1bb341feaf77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.585939 14237 tablet_service.cc:1511] Processing CreateTablet for tablet bb862cfa73bb46f185a2181b6922d403 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:47.587522 14237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb862cfa73bb46f185a2181b6922d403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.599809 14311 tablet_service.cc:1511] Processing CreateTablet for tablet b72d6bd4be824dceb80f1bb341feaf77 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:47.601426 14311 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b72d6bd4be824dceb80f1bb341feaf77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.601059 14317 tablet_service.cc:1511] Processing CreateTablet for tablet 376d2a317bcb4d6299c45dfe6b45f477 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:47.601236 14310 tablet_service.cc:1511] Processing CreateTablet for tablet bb862cfa73bb46f185a2181b6922d403 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:47.602415 14317 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 376d2a317bcb4d6299c45dfe6b45f477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.604632 14310 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb862cfa73bb46f185a2181b6922d403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.605191 14239 tablet_service.cc:1511] Processing CreateTablet for tablet 376d2a317bcb4d6299c45dfe6b45f477 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:47.606588 14239 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 376d2a317bcb4d6299c45dfe6b45f477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.609094 14235 tablet_service.cc:1511] Processing CreateTablet for tablet b72d6bd4be824dceb80f1bb341feaf77 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260709 12:27:47.614238 14235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b72d6bd4be824dceb80f1bb341feaf77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.615295 14554 tablet_bootstrap.cc:492] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:47.617210 14385 tablet_service.cc:1511] Processing CreateTablet for tablet bb862cfa73bb46f185a2181b6922d403 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260709 12:27:47.618744 14385 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb862cfa73bb46f185a2181b6922d403. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.638120 14556 tablet_bootstrap.cc:654] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.643903 14554 tablet_bootstrap.cc:654] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.672094 14386 tablet_service.cc:1511] Processing CreateTablet for tablet 376d2a317bcb4d6299c45dfe6b45f477 (DEFAULT_TABLE table=soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260709 12:27:47.673738 14386 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 376d2a317bcb4d6299c45dfe6b45f477. 1 dirs total, 0 dirs full, 0 dirs failed
I20260709 12:27:47.692806 14552 tablet_bootstrap.cc:492] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:47.693547 14552 ts_tablet_manager.cc:1403] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.209s	user 0.011s	sys 0.011s
I20260709 12:27:47.697289 14552 raft_consensus.cc:359] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.698199 14552 raft_consensus.cc:385] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.698546 14552 raft_consensus.cc:740] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.699611 14554 tablet_bootstrap.cc:492] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:47.700863 14554 ts_tablet_manager.cc:1403] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.086s	user 0.014s	sys 0.008s
I20260709 12:27:47.700425 14552 consensus_queue.cc:260] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.701989 14556 tablet_bootstrap.cc:492] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:47.702616 14556 ts_tablet_manager.cc:1403] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.132s	user 0.008s	sys 0.009s
I20260709 12:27:47.708830 14552 ts_tablet_manager.cc:1434] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.015s	user 0.005s	sys 0.005s
I20260709 12:27:47.704581 14554 raft_consensus.cc:359] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.711336 14554 raft_consensus.cc:385] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.711738 14552 tablet_bootstrap.cc:492] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:47.711733 14554 raft_consensus.cc:740] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.712961 14554 consensus_queue.cc:260] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.706000 14556 raft_consensus.cc:359] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.715121 14554 ts_tablet_manager.cc:1434] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.014s	user 0.008s	sys 0.000s
I20260709 12:27:47.715579 14556 raft_consensus.cc:385] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.716168 14556 raft_consensus.cc:740] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.716503 14554 tablet_bootstrap.cc:492] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:47.717310 14556 consensus_queue.cc:260] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.725633 14556 ts_tablet_manager.cc:1434] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.023s	user 0.005s	sys 0.002s
I20260709 12:27:47.726821 14554 tablet_bootstrap.cc:654] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.730984 14556 tablet_bootstrap.cc:492] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:47.741319 14552 tablet_bootstrap.cc:654] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.742830 14554 tablet_bootstrap.cc:492] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:47.743494 14554 ts_tablet_manager.cc:1403] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.027s	user 0.011s	sys 0.007s
I20260709 12:27:47.745811 14556 tablet_bootstrap.cc:654] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.750880 14554 raft_consensus.cc:359] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.752137 14554 raft_consensus.cc:385] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.752537 14554 raft_consensus.cc:740] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.755779 14554 consensus_queue.cc:260] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.757525 14552 tablet_bootstrap.cc:492] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:47.758133 14554 ts_tablet_manager.cc:1434] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.014s	user 0.007s	sys 0.001s
I20260709 12:27:47.758206 14552 ts_tablet_manager.cc:1403] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.047s	user 0.011s	sys 0.010s
I20260709 12:27:47.760542 14554 tablet_bootstrap.cc:492] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:47.761844 14552 raft_consensus.cc:359] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.762799 14556 tablet_bootstrap.cc:492] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:47.762929 14552 raft_consensus.cc:385] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.763366 14552 raft_consensus.cc:740] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.763486 14556 ts_tablet_manager.cc:1403] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.033s	user 0.020s	sys 0.000s
I20260709 12:27:47.764256 14552 consensus_queue.cc:260] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.771888 14552 ts_tablet_manager.cc:1434] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.013s	user 0.007s	sys 0.004s
I20260709 12:27:47.774183 14557 raft_consensus.cc:493] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.774968 14554 tablet_bootstrap.cc:654] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.774863 14557 raft_consensus.cc:515] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.775283 14552 tablet_bootstrap.cc:492] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:47.775935 14556 raft_consensus.cc:359] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.776764 14556 raft_consensus.cc:385] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.777053 14556 raft_consensus.cc:740] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.777834 14556 consensus_queue.cc:260] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.780555 14556 ts_tablet_manager.cc:1434] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.017s	user 0.011s	sys 0.003s
I20260709 12:27:47.782017 14556 tablet_bootstrap.cc:492] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:47.783336 14552 tablet_bootstrap.cc:654] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.785907 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8038818270e47f1abba0a9a3dcab848" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:47.786963 14251 raft_consensus.cc:2468] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 0.
I20260709 12:27:47.788970 14290 leader_election.cc:304] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:47.777319 14557 leader_election.cc:290] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:47.792603 14557 raft_consensus.cc:2804] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:47.792992 14557 raft_consensus.cc:493] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:47.793349 14557 raft_consensus.cc:3060] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.796365 14554 tablet_bootstrap.cc:492] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:47.796480 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8038818270e47f1abba0a9a3dcab848" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:47.797147 14554 ts_tablet_manager.cc:1403] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.038s	user 0.009s	sys 0.018s
I20260709 12:27:47.797386 14556 tablet_bootstrap.cc:654] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.798019 14551 raft_consensus.cc:493] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.798556 14551 raft_consensus.cc:515] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.800943 14551 leader_election.cc:290] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:47.800746 14554 raft_consensus.cc:359] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.801565 14554 raft_consensus.cc:385] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.801867 14554 raft_consensus.cc:740] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.802790 14554 consensus_queue.cc:260] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.805262 14554 ts_tablet_manager.cc:1434] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.008s	user 0.006s	sys 0.002s
I20260709 12:27:47.806386 14554 tablet_bootstrap.cc:492] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
W20260709 12:27:47.808571 14292 leader_election.cc:343] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.813442 14554 tablet_bootstrap.cc:654] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.818054 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a3327f378e64cbda03898ceaa6701a0" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:47.819118 14327 raft_consensus.cc:2468] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:47.820155 14556 tablet_bootstrap.cc:492] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:47.820813 14366 leader_election.cc:304] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:47.821017 14556 ts_tablet_manager.cc:1403] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.039s	user 0.015s	sys 0.012s
I20260709 12:27:47.821141 14554 tablet_bootstrap.cc:492] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:47.821936 14554 ts_tablet_manager.cc:1403] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.016s	user 0.009s	sys 0.004s
I20260709 12:27:47.823983 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a3327f378e64cbda03898ceaa6701a0" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:47.825289 14554 raft_consensus.cc:359] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.826128 14554 raft_consensus.cc:385] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.825773 14556 raft_consensus.cc:359] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.826496 14554 raft_consensus.cc:740] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.826856 14556 raft_consensus.cc:385] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.827308 14556 raft_consensus.cc:740] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.827342 14554 consensus_queue.cc:260] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.826875 14557 raft_consensus.cc:515] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.828442 14556 consensus_queue.cc:260] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
W20260709 12:27:47.829780 14364 leader_election.cc:343] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.830204 14554 ts_tablet_manager.cc:1434] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:27:47.829037 14551 raft_consensus.cc:2804] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:47.833673 14556 ts_tablet_manager.cc:1434] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.012s	user 0.012s	sys 0.000s
I20260709 12:27:47.834275 14551 raft_consensus.cc:493] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:47.835747 14551 raft_consensus.cc:3060] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.836104 14556 tablet_bootstrap.cc:492] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:47.837321 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8038818270e47f1abba0a9a3dcab848" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:47.838073 14251 raft_consensus.cc:3060] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.845408 14554 tablet_bootstrap.cc:492] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:47.846916 14251 raft_consensus.cc:2468] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 1.
I20260709 12:27:47.850767 14290 leader_election.cc:304] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:47.851213 14557 leader_election.cc:290] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:47.851809 14517 raft_consensus.cc:2804] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:47.852383 14517 raft_consensus.cc:697] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Becoming Leader. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:47.852785 14556 tablet_bootstrap.cc:654] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.853320 14517 consensus_queue.cc:237] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.856505 14554 tablet_bootstrap.cc:654] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.858177 14552 tablet_bootstrap.cc:492] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:47.858876 14552 ts_tablet_manager.cc:1403] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.084s	user 0.019s	sys 0.011s
I20260709 12:27:47.867460 14552 raft_consensus.cc:359] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.868417 14552 raft_consensus.cc:385] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.868018 14551 raft_consensus.cc:515] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.868800 14552 raft_consensus.cc:740] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.869689 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8038818270e47f1abba0a9a3dcab848" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:47.870182 14554 tablet_bootstrap.cc:492] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:47.870591 14551 leader_election.cc:290] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:47.870744 14554 ts_tablet_manager.cc:1403] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.026s	user 0.012s	sys 0.009s
I20260709 12:27:47.871507 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a3327f378e64cbda03898ceaa6701a0" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:47.869860 14552 consensus_queue.cc:260] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.872820 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a3327f378e64cbda03898ceaa6701a0" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:47.873263 14252 raft_consensus.cc:3060] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.873562 14327 raft_consensus.cc:3060] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.874475 14554 raft_consensus.cc:359] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.875269 14554 raft_consensus.cc:385] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.875546 14554 raft_consensus.cc:740] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.876335 14554 consensus_queue.cc:260] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
W20260709 12:27:47.876664 14292 leader_election.cc:343] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.881525 14252 raft_consensus.cc:2468] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:47.884469 14554 ts_tablet_manager.cc:1434] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.013s	user 0.002s	sys 0.004s
I20260709 12:27:47.886173 14554 tablet_bootstrap.cc:492] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:47.886916 14364 leader_election.cc:304] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: 
I20260709 12:27:47.888195 14551 raft_consensus.cc:2804] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:47.890089 14552 ts_tablet_manager.cc:1434] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.031s	user 0.007s	sys 0.001s
I20260709 12:27:47.889550 14149 catalog_manager.cc:5697] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 reported cstate change: term changed from 0 to 1, leader changed from <none> to c403bb4886574f1db5b159d81a264617 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "c403bb4886574f1db5b159d81a264617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:47.891409 14552 tablet_bootstrap.cc:492] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:47.893133 14554 tablet_bootstrap.cc:654] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.895866 14327 raft_consensus.cc:2468] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:47.901079 14552 tablet_bootstrap.cc:654] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.907210 14551 raft_consensus.cc:697] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Leader. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:47.907862 14556 tablet_bootstrap.cc:492] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:47.907883 14554 tablet_bootstrap.cc:492] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:47.908681 14556 ts_tablet_manager.cc:1403] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.073s	user 0.022s	sys 0.009s
I20260709 12:27:47.908685 14554 ts_tablet_manager.cc:1403] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.023s	user 0.014s	sys 0.008s
I20260709 12:27:47.908422 14551 consensus_queue.cc:237] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.909026 14552 tablet_bootstrap.cc:492] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:47.909795 14552 ts_tablet_manager.cc:1403] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.019s	user 0.011s	sys 0.005s
I20260709 12:27:47.912333 14556 raft_consensus.cc:359] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.912523 14554 raft_consensus.cc:359] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.913285 14556 raft_consensus.cc:385] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.913335 14554 raft_consensus.cc:385] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.913743 14556 raft_consensus.cc:740] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.913797 14554 raft_consensus.cc:740] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.914710 14554 consensus_queue.cc:260] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.915019 14556 consensus_queue.cc:260] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.917251 14554 ts_tablet_manager.cc:1434] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:27:47.918334 14556 ts_tablet_manager.cc:1434] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.009s	user 0.005s	sys 0.005s
I20260709 12:27:47.918346 14554 tablet_bootstrap.cc:492] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:47.919622 14556 tablet_bootstrap.cc:492] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:47.925951 14555 raft_consensus.cc:493] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.926400 14554 tablet_bootstrap.cc:654] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.926743 14555 raft_consensus.cc:515] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.927412 14556 tablet_bootstrap.cc:654] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.930593 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607b3a3e5d56487d815d403fa1373cfc" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:47.930814 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607b3a3e5d56487d815d403fa1373cfc" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:47.931488 14252 raft_consensus.cc:2468] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:47.933130 14364 leader_election.cc:304] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: 
I20260709 12:27:47.933307 14555 leader_election.cc:290] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:47.931524 14327 raft_consensus.cc:2468] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:47.936024 14555 raft_consensus.cc:2804] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:47.936132 14517 raft_consensus.cc:493] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.936918 14517 raft_consensus.cc:515] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.936610 14555 raft_consensus.cc:493] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:47.938334 14555 raft_consensus.cc:3060] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.939298 14552 raft_consensus.cc:359] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.939754 14517 leader_election.cc:290] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:47.939944 14554 tablet_bootstrap.cc:492] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:47.940237 14552 raft_consensus.cc:385] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.940567 14554 ts_tablet_manager.cc:1403] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.022s	user 0.015s	sys 0.006s
I20260709 12:27:47.940696 14552 raft_consensus.cc:740] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.941241 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2d0355002344a8086f697db621c2d65" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
W20260709 12:27:47.943171 14292 leader_election.cc:343] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:47.943943 14556 tablet_bootstrap.cc:492] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:47.944790 14556 ts_tablet_manager.cc:1403] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.025s	user 0.011s	sys 0.007s
I20260709 12:27:47.944581 14554 raft_consensus.cc:359] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.945411 14554 raft_consensus.cc:385] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.945729 14554 raft_consensus.cc:740] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.946461 14554 consensus_queue.cc:260] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.941649 14552 consensus_queue.cc:260] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.948864 14554 ts_tablet_manager.cc:1434] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.008s	user 0.005s	sys 0.000s
I20260709 12:27:47.948650 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2d0355002344a8086f697db621c2d65" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:47.949561 14252 raft_consensus.cc:2468] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 0.
I20260709 12:27:47.949958 14554 tablet_bootstrap.cc:492] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:47.950071 14552 ts_tablet_manager.cc:1434] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.040s	user 0.003s	sys 0.004s
I20260709 12:27:47.951119 14290 leader_election.cc:304] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 7310265ac7544fd281403a40bf5d6d30
I20260709 12:27:47.951377 14556 raft_consensus.cc:359] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.952176 14517 raft_consensus.cc:2804] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:47.952169 14556 raft_consensus.cc:385] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.952744 14517 raft_consensus.cc:493] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:47.952780 14556 raft_consensus.cc:740] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.953395 14517 raft_consensus.cc:3060] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.954034 14556 consensus_queue.cc:260] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.957409 14556 ts_tablet_manager.cc:1434] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 12:27:47.957948 14554 tablet_bootstrap.cc:654] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.958487 14556 tablet_bootstrap.cc:492] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:47.960052 14552 tablet_bootstrap.cc:492] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:47.968241 14556 tablet_bootstrap.cc:654] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.970101 14552 tablet_bootstrap.cc:654] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.971756 14146 catalog_manager.cc:5697] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "7310265ac7544fd281403a40bf5d6d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:47.975342 14554 tablet_bootstrap.cc:492] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:47.975957 14554 ts_tablet_manager.cc:1403] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.026s	user 0.009s	sys 0.015s
I20260709 12:27:47.976861 14556 tablet_bootstrap.cc:492] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:47.977411 14556 ts_tablet_manager.cc:1403] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.019s	user 0.011s	sys 0.004s
I20260709 12:27:47.978850 14554 raft_consensus.cc:359] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.979626 14554 raft_consensus.cc:385] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.980190 14554 raft_consensus.cc:740] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.981041 14556 raft_consensus.cc:359] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.981848 14556 raft_consensus.cc:385] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:47.982306 14556 raft_consensus.cc:740] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:47.980924 14554 consensus_queue.cc:260] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.983126 14556 consensus_queue.cc:260] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.985226 14554 ts_tablet_manager.cc:1434] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.009s	user 0.007s	sys 0.000s
I20260709 12:27:47.985440 14556 ts_tablet_manager.cc:1434] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.008s	user 0.003s	sys 0.004s
I20260709 12:27:47.986539 14557 raft_consensus.cc:493] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:47.987125 14557 raft_consensus.cc:515] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.989292 14557 leader_election.cc:290] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:47.989398 14556 tablet_bootstrap.cc:492] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:47.991604 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "801bc4a961d84dc38e9ead3e902346d1" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:47.992435 14401 raft_consensus.cc:2468] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 0.
I20260709 12:27:47.995471 14292 leader_election.cc:304] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:47.996326 14556 tablet_bootstrap.cc:654] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:47.996447 14557 raft_consensus.cc:2804] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:47.996917 14557 raft_consensus.cc:493] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:47.997277 14557 raft_consensus.cc:3060] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:47.996080 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "801bc4a961d84dc38e9ead3e902346d1" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:48.008010 14557 raft_consensus.cc:515] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:47.980858 14517 raft_consensus.cc:515] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.010277 14552 tablet_bootstrap.cc:492] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:48.011036 14552 ts_tablet_manager.cc:1403] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.051s	user 0.019s	sys 0.002s
I20260709 12:27:48.012042 14554 tablet_bootstrap.cc:492] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617: Bootstrap starting.
I20260709 12:27:48.013746 14517 leader_election.cc:290] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.015129 14552 raft_consensus.cc:359] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.018827 14552 raft_consensus.cc:385] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.019235 14552 raft_consensus.cc:740] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.020318 14552 consensus_queue.cc:260] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.021296 14517 raft_consensus.cc:493] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.022449 14517 raft_consensus.cc:515] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.016474 14555 raft_consensus.cc:515] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.023030 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2d0355002344a8086f697db621c2d65" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:48.023756 14552 ts_tablet_manager.cc:1434] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.012s	user 0.009s	sys 0.000s
I20260709 12:27:48.029229 14400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "801bc4a961d84dc38e9ead3e902346d1" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:48.026309 14554 tablet_bootstrap.cc:654] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.033439 14400 raft_consensus.cc:3060] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.034189 14552 tablet_bootstrap.cc:492] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:48.029238 14557 leader_election.cc:290] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.036664 14556 tablet_bootstrap.cc:492] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:48.036470 14517 leader_election.cc:290] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.037251 14556 ts_tablet_manager.cc:1403] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.048s	user 0.023s	sys 0.014s
I20260709 12:27:48.039366 14555 leader_election.cc:290] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
W20260709 12:27:48.033429 14292 leader_election.cc:343] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.040457 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607b3a3e5d56487d815d403fa1373cfc" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:48.041128 14252 raft_consensus.cc:3060] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.042681 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab7958026c31429697818f80570d4314" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:48.043437 14401 raft_consensus.cc:2468] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 0.
I20260709 12:27:48.043835 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "801bc4a961d84dc38e9ead3e902346d1" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:48.044461 14250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2d0355002344a8086f697db621c2d65" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:48.045120 14250 raft_consensus.cc:3060] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.048862 14556 raft_consensus.cc:359] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.049719 14556 raft_consensus.cc:385] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.050021 14556 raft_consensus.cc:740] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.050806 14556 consensus_queue.cc:260] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.051185 14400 raft_consensus.cc:2468] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 1.
I20260709 12:27:48.051337 14292 leader_election.cc:304] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: 
W20260709 12:27:48.043150 14290 leader_election.cc:343] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Illegal state: must be running to vote when last-logged opid is not known
W20260709 12:27:48.053861 14290 leader_election.cc:343] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.053968 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab7958026c31429697818f80570d4314" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:48.054584 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607b3a3e5d56487d815d403fa1373cfc" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:48.056918 14327 raft_consensus.cc:3060] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.054797 14251 raft_consensus.cc:2468] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 0.
I20260709 12:27:48.058995 14292 leader_election.cc:304] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:48.055706 14557 raft_consensus.cc:2804] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.060292 14557 raft_consensus.cc:493] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.060674 14557 raft_consensus.cc:3060] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.063872 14555 raft_consensus.cc:493] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.064430 14555 raft_consensus.cc:515] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.065225 14517 raft_consensus.cc:493] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.067531 14552 tablet_bootstrap.cc:654] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.068631 14556 ts_tablet_manager.cc:1434] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.031s	user 0.004s	sys 0.004s
I20260709 12:27:48.071898 14556 tablet_bootstrap.cc:492] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:48.072417 14251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b72d6bd4be824dceb80f1bb341feaf77" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
W20260709 12:27:48.073982 14364 leader_election.cc:343] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.075611 14327 raft_consensus.cc:2468] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:48.075878 14555 leader_election.cc:290] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:48.070336 14250 raft_consensus.cc:2468] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 1.
I20260709 12:27:48.065836 14517 raft_consensus.cc:515] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.086654 14556 tablet_bootstrap.cc:654] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.088698 14580 raft_consensus.cc:2804] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.102253 14557 raft_consensus.cc:515] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.103874 14552 tablet_bootstrap.cc:492] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:48.104828 14557 leader_election.cc:290] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.107290 14290 leader_election.cc:304] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 7310265ac7544fd281403a40bf5d6d30
I20260709 12:27:48.108425 14552 ts_tablet_manager.cc:1403] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.075s	user 0.006s	sys 0.019s
I20260709 12:27:48.114042 14554 tablet_bootstrap.cc:492] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617: No bootstrap required, opened a new log
I20260709 12:27:48.108915 14557 raft_consensus.cc:2804] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.114652 14554 ts_tablet_manager.cc:1403] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617: Time spent bootstrapping tablet: real 0.103s	user 0.010s	sys 0.007s
I20260709 12:27:48.114717 14557 raft_consensus.cc:697] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Becoming Leader. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:48.115489 14250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab7958026c31429697818f80570d4314" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:48.115893 14557 consensus_queue.cc:237] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.118119 14554 raft_consensus.cc:359] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.118893 14554 raft_consensus.cc:385] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.119169 14554 raft_consensus.cc:740] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.119854 14554 consensus_queue.cc:260] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.113719 14552 raft_consensus.cc:359] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.109525 14366 leader_election.cc:304] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:48.122347 14554 ts_tablet_manager.cc:1434] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617: Time spent starting tablet: real 0.007s	user 0.005s	sys 0.001s
I20260709 12:27:48.123765 14555 raft_consensus.cc:2804] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.124003 14400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ab7958026c31429697818f80570d4314" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:48.124293 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "376d2a317bcb4d6299c45dfe6b45f477" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:48.124468 14555 raft_consensus.cc:697] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Leader. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:48.124763 14400 raft_consensus.cc:3060] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.125520 14555 consensus_queue.cc:237] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
W20260709 12:27:48.127566 14292 leader_election.cc:343] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.128180 14556 tablet_bootstrap.cc:492] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:48.128834 14556 ts_tablet_manager.cc:1403] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.057s	user 0.019s	sys 0.011s
I20260709 12:27:48.116494 14250 raft_consensus.cc:3060] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.133419 14400 raft_consensus.cc:2468] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 1.
I20260709 12:27:48.133467 14556 raft_consensus.cc:359] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.134749 14556 raft_consensus.cc:385] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.135056 14292 leader_election.cc:304] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:48.135118 14556 raft_consensus.cc:740] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.136276 14557 raft_consensus.cc:2804] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.136886 14557 raft_consensus.cc:697] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Becoming Leader. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:48.109859 14517 leader_election.cc:290] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.121956 14552 raft_consensus.cc:385] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.138003 14552 raft_consensus.cc:740] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.117808 14252 raft_consensus.cc:2468] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:48.138263 14557 consensus_queue.cc:237] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.139065 14552 consensus_queue.cc:260] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.140609 14580 raft_consensus.cc:697] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Becoming Leader. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:48.141548 14580 consensus_queue.cc:237] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.141925 14552 ts_tablet_manager.cc:1434] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.032s	user 0.005s	sys 0.000s
I20260709 12:27:48.136559 14556 consensus_queue.cc:260] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.143266 14250 raft_consensus.cc:2468] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 1.
I20260709 12:27:48.145597 14556 ts_tablet_manager.cc:1434] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.016s	user 0.008s	sys 0.000s
I20260709 12:27:48.146826 14556 tablet_bootstrap.cc:492] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30: Bootstrap starting.
I20260709 12:27:48.150981 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b72d6bd4be824dceb80f1bb341feaf77" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:48.152797 14327 raft_consensus.cc:2468] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:48.154335 14556 tablet_bootstrap.cc:654] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.143373 14552 tablet_bootstrap.cc:492] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:48.160566 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "376d2a317bcb4d6299c45dfe6b45f477" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:48.162240 14552 tablet_bootstrap.cc:654] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.146170 14147 catalog_manager.cc:5697] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "7310265ac7544fd281403a40bf5d6d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:48.179131 14552 tablet_bootstrap.cc:492] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:48.179756 14552 ts_tablet_manager.cc:1403] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.037s	user 0.014s	sys 0.003s
I20260709 12:27:48.183063 14366 leader_election.cc:304] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:48.184124 14551 raft_consensus.cc:2804] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.184558 14551 raft_consensus.cc:493] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.185035 14556 tablet_bootstrap.cc:492] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30: No bootstrap required, opened a new log
I20260709 12:27:48.185642 14556 ts_tablet_manager.cc:1403] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30: Time spent bootstrapping tablet: real 0.039s	user 0.007s	sys 0.008s
I20260709 12:27:48.185048 14551 raft_consensus.cc:3060] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.189069 14556 raft_consensus.cc:359] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.189836 14556 raft_consensus.cc:385] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.190204 14556 raft_consensus.cc:740] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.190984 14556 consensus_queue.cc:260] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.193230 14556 ts_tablet_manager.cc:1434] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30: Time spent starting tablet: real 0.007s	user 0.007s	sys 0.000s
I20260709 12:27:48.194388 14552 raft_consensus.cc:359] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.195341 14552 raft_consensus.cc:385] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.195668 14552 raft_consensus.cc:740] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.195647 14149 catalog_manager.cc:5697] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 reported cstate change: term changed from 0 to 1, leader changed from <none> to c403bb4886574f1db5b159d81a264617 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "c403bb4886574f1db5b159d81a264617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.196761 14552 consensus_queue.cc:260] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.199237 14551 raft_consensus.cc:515] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.199353 14552 ts_tablet_manager.cc:1434] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.019s	user 0.009s	sys 0.000s
I20260709 12:27:48.201509 14552 tablet_bootstrap.cc:492] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:48.203284 14551 leader_election.cc:290] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:48.204023 14250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b72d6bd4be824dceb80f1bb341feaf77" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
W20260709 12:27:48.184244 14290 leader_election.cc:343] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.206588 14290 leader_election.cc:304] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c403bb4886574f1db5b159d81a264617; no voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f
W20260709 12:27:48.206933 14364 leader_election.cc:343] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Illegal state: must be running to vote when last-logged opid is not known
I20260709 12:27:48.208096 14580 raft_consensus.cc:2749] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.209007 14552 tablet_bootstrap.cc:654] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.209666 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b72d6bd4be824dceb80f1bb341feaf77" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:48.210315 14327 raft_consensus.cc:3060] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.214665 14552 tablet_bootstrap.cc:492] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:48.215271 14552 ts_tablet_manager.cc:1403] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.014s	user 0.013s	sys 0.000s
I20260709 12:27:48.216832 14327 raft_consensus.cc:2468] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:48.218549 14552 raft_consensus.cc:359] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.219343 14552 raft_consensus.cc:385] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.219677 14552 raft_consensus.cc:740] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.219448 14366 leader_election.cc:304] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:48.220521 14552 consensus_queue.cc:260] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.221062 14555 raft_consensus.cc:2804] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.221858 14555 raft_consensus.cc:697] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Leader. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:48.223011 14552 ts_tablet_manager.cc:1434] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.007s	user 0.006s	sys 0.002s
I20260709 12:27:48.222847 14555 consensus_queue.cc:237] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.224646 14552 tablet_bootstrap.cc:492] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f: Bootstrap starting.
I20260709 12:27:48.233960 14552 tablet_bootstrap.cc:654] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f: Neither blocks nor log segments found. Creating new log.
I20260709 12:27:48.235983 14149 catalog_manager.cc:5697] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 reported cstate change: term changed from 0 to 1, leader changed from <none> to c403bb4886574f1db5b159d81a264617 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "c403bb4886574f1db5b159d81a264617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.238057 14149 catalog_manager.cc:5697] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 reported cstate change: term changed from 0 to 1, leader changed from <none> to c403bb4886574f1db5b159d81a264617 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "c403bb4886574f1db5b159d81a264617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.242542 14552 tablet_bootstrap.cc:492] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f: No bootstrap required, opened a new log
I20260709 12:27:48.243244 14552 ts_tablet_manager.cc:1403] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent bootstrapping tablet: real 0.019s	user 0.008s	sys 0.008s
I20260709 12:27:48.245968 14552 raft_consensus.cc:359] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.246771 14552 raft_consensus.cc:385] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260709 12:27:48.247026 14552 raft_consensus.cc:740] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Initialized, Role: FOLLOWER
I20260709 12:27:48.247711 14552 consensus_queue.cc:260] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.249747 14552 ts_tablet_manager.cc:1434] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f: Time spent starting tablet: real 0.006s	user 0.004s	sys 0.000s
I20260709 12:27:48.252336 14147 catalog_manager.cc:5697] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "7310265ac7544fd281403a40bf5d6d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: HEALTHY } } }
W20260709 12:27:48.294608 14198 auto_rebalancer.cc:288] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260709 12:27:48.305509 14557 raft_consensus.cc:493] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.306193 14557 raft_consensus.cc:515] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.308308 14557 leader_election.cc:290] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.309362 14400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb862cfa73bb46f185a2181b6922d403" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:48.309365 14250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb862cfa73bb46f185a2181b6922d403" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:48.310345 14400 raft_consensus.cc:2468] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 0.
I20260709 12:27:48.310385 14250 raft_consensus.cc:2468] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 0.
I20260709 12:27:48.311866 14290 leader_election.cc:304] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:48.312731 14557 raft_consensus.cc:2804] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.313138 14557 raft_consensus.cc:493] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.313481 14557 raft_consensus.cc:3060] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.319499 14555 raft_consensus.cc:493] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.320345 14555 raft_consensus.cc:515] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.321380 14557 raft_consensus.cc:515] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.322404 14555 leader_election.cc:290] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:48.323396 14557 leader_election.cc:290] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.324119 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8038818270e47f1abba0a9a3dcab848" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:48.324648 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb862cfa73bb46f185a2181b6922d403" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:48.325294 14252 raft_consensus.cc:3060] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.325374 14400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bb862cfa73bb46f185a2181b6922d403" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:48.323289 14250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8038818270e47f1abba0a9a3dcab848" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:48.326084 14400 raft_consensus.cc:3060] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.326421 14250 raft_consensus.cc:2393] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7310265ac7544fd281403a40bf5d6d30 in current term 1: Already voted for candidate c403bb4886574f1db5b159d81a264617 in this term.
I20260709 12:27:48.328475 14364 leader_election.cc:304] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30; no voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617
I20260709 12:27:48.329718 14555 raft_consensus.cc:3060] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.334703 14252 raft_consensus.cc:2468] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 1.
I20260709 12:27:48.336418 14290 leader_election.cc:304] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:48.337411 14557 raft_consensus.cc:2804] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.337466 14400 raft_consensus.cc:2468] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 1.
I20260709 12:27:48.338045 14557 raft_consensus.cc:697] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Becoming Leader. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:48.339427 14555 raft_consensus.cc:2749] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.339509 14557 consensus_queue.cc:237] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.347937 14555 raft_consensus.cc:493] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.348521 14555 raft_consensus.cc:515] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.348707 14145 catalog_manager.cc:5697] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 reported cstate change: term changed from 0 to 1, leader changed from <none> to c403bb4886574f1db5b159d81a264617 (127.29.201.194). New cstate: current_term: 1 leader_uuid: "c403bb4886574f1db5b159d81a264617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.351243 14555 leader_election.cc:290] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:48.352180 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "376d2a317bcb4d6299c45dfe6b45f477" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:48.352360 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "376d2a317bcb4d6299c45dfe6b45f477" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:48.352986 14252 raft_consensus.cc:2468] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:48.353132 14327 raft_consensus.cc:2468] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 0.
I20260709 12:27:48.354121 14364 leader_election.cc:304] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: 
I20260709 12:27:48.355628 14555 raft_consensus.cc:2804] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260709 12:27:48.356022 14555 raft_consensus.cc:493] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260709 12:27:48.356386 14555 raft_consensus.cc:3060] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.363873 14555 raft_consensus.cc:515] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.365715 14555 leader_election.cc:290] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:48.366710 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "376d2a317bcb4d6299c45dfe6b45f477" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:48.366880 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "376d2a317bcb4d6299c45dfe6b45f477" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:48.367352 14252 raft_consensus.cc:3060] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.367506 14327 raft_consensus.cc:3060] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.373307 14252 raft_consensus.cc:2468] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:48.373777 14327 raft_consensus.cc:2468] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 1.
I20260709 12:27:48.374555 14364 leader_election.cc:304] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: 
I20260709 12:27:48.375578 14555 raft_consensus.cc:2804] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader election won for term 1
I20260709 12:27:48.376003 14555 raft_consensus.cc:697] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Leader. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:48.376785 14555 consensus_queue.cc:237] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.385003 14145 catalog_manager.cc:5697] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195). New cstate: current_term: 1 leader_uuid: "7310265ac7544fd281403a40bf5d6d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:48.386096 14555 raft_consensus.cc:493] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.386780 14555 raft_consensus.cc:515] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.388804 14555 leader_election.cc:290] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:48.389786 14252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2d0355002344a8086f697db621c2d65" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" is_pre_election: true
I20260709 12:27:48.389977 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2d0355002344a8086f697db621c2d65" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:48.390681 14252 raft_consensus.cc:2393] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7310265ac7544fd281403a40bf5d6d30 in current term 1: Already voted for candidate c403bb4886574f1db5b159d81a264617 in this term.
I20260709 12:27:48.391785 14364 leader_election.cc:304] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30; no voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617
I20260709 12:27:48.392586 14555 raft_consensus.cc:3060] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.396278 14551 consensus_queue.cc:1048] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:48.399552 14555 raft_consensus.cc:2749] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.413820 14553 consensus_queue.cc:1048] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.439916 14400 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6a3327f378e64cbda03898ceaa6701a0"
dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
mode: GRACEFUL
new_leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:54634
I20260709 12:27:48.440608 14400 raft_consensus.cc:606] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Received request to transfer leadership to TS bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:48.446805 14400 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "607b3a3e5d56487d815d403fa1373cfc"
dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
mode: GRACEFUL
new_leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:54634
I20260709 12:27:48.447398 14400 raft_consensus.cc:606] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Received request to transfer leadership to TS bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:48.453374 14327 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "c8038818270e47f1abba0a9a3dcab848"
dest_uuid: "c403bb4886574f1db5b159d81a264617"
mode: GRACEFUL
new_leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:47642
I20260709 12:27:48.454033 14327 raft_consensus.cc:606] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Received request to transfer leadership to TS bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:48.460335 14327 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "d2d0355002344a8086f697db621c2d65"
dest_uuid: "c403bb4886574f1db5b159d81a264617"
mode: GRACEFUL
new_leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:47642
I20260709 12:27:48.460953 14327 raft_consensus.cc:606] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Received request to transfer leadership to TS bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:48.473203 14400 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "b72d6bd4be824dceb80f1bb341feaf77"
dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
mode: GRACEFUL
new_leader_uuid: "c403bb4886574f1db5b159d81a264617"
 from {username='slave'} at 127.0.0.1:54634
I20260709 12:27:48.473819 14400 raft_consensus.cc:606] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Received request to transfer leadership to TS c403bb4886574f1db5b159d81a264617
I20260709 12:27:48.488359 14580 consensus_queue.cc:1048] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:48.489025 14557 consensus_queue.cc:1048] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:48.499778 14585 raft_consensus.cc:993] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617: : Instructing follower bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:48.500382 14585 raft_consensus.cc:1081] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Signalling peer bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:48.502454 14250 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "c8038818270e47f1abba0a9a3dcab848"
dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:55626
I20260709 12:27:48.504786 14250 raft_consensus.cc:493] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:48.505196 14250 raft_consensus.cc:3060] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.513429 14250 raft_consensus.cc:515] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.516713 14250 leader_election.cc:290] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Requested vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.518543 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8038818270e47f1abba0a9a3dcab848" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:48.518991 14400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c8038818270e47f1abba0a9a3dcab848" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:48.519699 14327 raft_consensus.cc:3055] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:48.519811 14400 raft_consensus.cc:3060] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.520179 14327 raft_consensus.cc:740] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:48.522109 14327 consensus_queue.cc:260] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.523535 14327 raft_consensus.cc:3060] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.525722 14553 consensus_queue.cc:1048] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.531220 14327 raft_consensus.cc:2468] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:48.534900 14217 leader_election.cc:304] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:48.535910 14524 raft_consensus.cc:2804] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:48.536377 14524 raft_consensus.cc:697] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Becoming Leader. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Running, Role: LEADER
I20260709 12:27:48.536545 14400 raft_consensus.cc:2468] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:48.537174 14524 consensus_queue.cc:237] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.542205 14585 consensus_queue.cc:1048] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.549911 14555 consensus_queue.cc:1048] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:48.553145 14149 catalog_manager.cc:5697] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f reported cstate change: term changed from 1 to 2, leader changed from c403bb4886574f1db5b159d81a264617 (127.29.201.194) to bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193). New cstate: current_term: 2 leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.561959 14585 raft_consensus.cc:993] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617: : Instructing follower bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:48.562577 14585 raft_consensus.cc:1081] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Signalling peer bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:48.564733 14555 raft_consensus.cc:993] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30: : Instructing follower bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:48.565313 14555 raft_consensus.cc:1081] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Signalling peer bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:48.567440 14249 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "607b3a3e5d56487d815d403fa1373cfc"
dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:55618
I20260709 12:27:48.568035 14249 raft_consensus.cc:493] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:48.568447 14249 raft_consensus.cc:3060] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.571228 14252 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "d2d0355002344a8086f697db621c2d65"
dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:55626
I20260709 12:27:48.577056 14249 raft_consensus.cc:515] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.577566 14252 raft_consensus.cc:493] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:48.578143 14252 raft_consensus.cc:3060] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.578406 14585 consensus_queue.cc:1048] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.580590 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607b3a3e5d56487d815d403fa1373cfc" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:48.581352 14326 raft_consensus.cc:3060] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.585623 14249 leader_election.cc:290] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Requested vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.586577 14252 raft_consensus.cc:515] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.586908 14401 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "607b3a3e5d56487d815d403fa1373cfc" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:48.588467 14401 raft_consensus.cc:3055] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:48.588809 14401 raft_consensus.cc:740] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:48.589668 14401 consensus_queue.cc:260] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.591758 14401 raft_consensus.cc:3060] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.592904 14252 leader_election.cc:290] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Requested vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.596138 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2d0355002344a8086f697db621c2d65" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:48.596757 14327 raft_consensus.cc:3055] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:48.597085 14327 raft_consensus.cc:740] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:48.597828 14327 consensus_queue.cc:260] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.599561 14327 raft_consensus.cc:3060] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.601405 14400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2d0355002344a8086f697db621c2d65" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:48.602061 14400 raft_consensus.cc:3060] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.602207 14326 raft_consensus.cc:2468] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:48.603523 14217 leader_election.cc:304] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:48.604321 14524 raft_consensus.cc:2804] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:48.604799 14524 raft_consensus.cc:697] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Becoming Leader. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Running, Role: LEADER
I20260709 12:27:48.605731 14524 consensus_queue.cc:237] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.609153 14400 raft_consensus.cc:2468] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:48.612360 14401 raft_consensus.cc:2468] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:48.614511 14217 leader_election.cc:304] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: 
I20260709 12:27:48.616503 14612 raft_consensus.cc:2804] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:48.617074 14612 raft_consensus.cc:697] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Becoming Leader. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Running, Role: LEADER
I20260709 12:27:48.618160 14327 raft_consensus.cc:2468] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:48.618301 14612 consensus_queue.cc:237] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.627331 14145 catalog_manager.cc:5697] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f reported cstate change: term changed from 1 to 2, leader changed from c403bb4886574f1db5b159d81a264617 (127.29.201.194) to bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193). New cstate: current_term: 2 leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.628575 14145 catalog_manager.cc:5697] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f reported cstate change: term changed from 1 to 2, leader changed from 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195) to bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193). New cstate: current_term: 2 leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.632118 14612 raft_consensus.cc:493] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.632838 14612 raft_consensus.cc:515] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.635169 14612 leader_election.cc:290] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.635901 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "801bc4a961d84dc38e9ead3e902346d1" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:48.636754 14400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "801bc4a961d84dc38e9ead3e902346d1" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:48.636850 14580 consensus_queue.cc:1048] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.637696 14400 raft_consensus.cc:2393] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bd3bfb90591a4b8c95c500d57be3d44f in current term 1: Already voted for candidate c403bb4886574f1db5b159d81a264617 in this term.
I20260709 12:27:48.639920 14217 leader_election.cc:304] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f; no voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617
I20260709 12:27:48.641098 14612 raft_consensus.cc:3060] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.649880 14612 raft_consensus.cc:2749] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.653535 14584 consensus_queue.cc:1048] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:48.682199 14553 consensus_queue.cc:1048] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:48.684337 14612 raft_consensus.cc:493] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260709 12:27:48.684968 14612 raft_consensus.cc:515] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.686932 14612 leader_election.cc:290] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.688138 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b72d6bd4be824dceb80f1bb341feaf77" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c403bb4886574f1db5b159d81a264617" is_pre_election: true
I20260709 12:27:48.688331 14399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b72d6bd4be824dceb80f1bb341feaf77" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7310265ac7544fd281403a40bf5d6d30" is_pre_election: true
I20260709 12:27:48.690232 14551 raft_consensus.cc:993] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30: : Instructing follower c403bb4886574f1db5b159d81a264617 to start an election
I20260709 12:27:48.690132 14217 leader_election.cc:304] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f; no voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617
I20260709 12:27:48.690783 14555 raft_consensus.cc:1081] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Signalling peer c403bb4886574f1db5b159d81a264617 to start an election
I20260709 12:27:48.691176 14612 raft_consensus.cc:2749] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260709 12:27:48.693454 14327 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "b72d6bd4be824dceb80f1bb341feaf77"
dest_uuid: "c403bb4886574f1db5b159d81a264617"
 from {username='slave'} at 127.0.0.1:47618
I20260709 12:27:48.694186 14327 raft_consensus.cc:493] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:48.694664 14327 raft_consensus.cc:3060] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.696309 14584 consensus_queue.cc:1048] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.697955 14252 raft_consensus.cc:3060] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [term 0 FOLLOWER]: Advancing to term 1
I20260709 12:27:48.702675 14327 raft_consensus.cc:515] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.703838 14326 raft_consensus.cc:1240] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Rejecting Update request from peer 7310265ac7544fd281403a40bf5d6d30 for earlier term 1. Current term is 2. Ops: []
I20260709 12:27:48.705788 14249 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b72d6bd4be824dceb80f1bb341feaf77" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:48.706055 14553 consensus_queue.cc:1059] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:48.707042 14551 raft_consensus.cc:3055] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:48.707418 14551 raft_consensus.cc:740] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:48.708729 14327 leader_election.cc:290] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 2 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.708380 14551 consensus_queue.cc:260] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.710026 14555 consensus_queue.cc:1048] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:48.710901 14551 raft_consensus.cc:3060] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.712570 14400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b72d6bd4be824dceb80f1bb341feaf77" candidate_uuid: "c403bb4886574f1db5b159d81a264617" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:48.724010 14557 consensus_queue.cc:1048] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.728585 14400 raft_consensus.cc:2468] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c403bb4886574f1db5b159d81a264617 in term 2.
I20260709 12:27:48.731621 14292 leader_election.cc:304] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: bd3bfb90591a4b8c95c500d57be3d44f
I20260709 12:27:48.732919 14583 raft_consensus.cc:2804] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:48.733438 14583 raft_consensus.cc:697] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 2 LEADER]: Becoming Leader. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:48.734326 14583 consensus_queue.cc:237] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.742568 14145 catalog_manager.cc:5697] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 reported cstate change: term changed from 1 to 2, leader changed from 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195) to c403bb4886574f1db5b159d81a264617 (127.29.201.194). New cstate: current_term: 2 leader_uuid: "c403bb4886574f1db5b159d81a264617" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.793995 14553 raft_consensus.cc:993] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30: : Instructing follower bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:48.794521 14551 raft_consensus.cc:1081] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Signalling peer bd3bfb90591a4b8c95c500d57be3d44f to start an election
I20260709 12:27:48.795979 14252 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6a3327f378e64cbda03898ceaa6701a0"
dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
 from {username='slave'} at 127.0.0.1:55618
I20260709 12:27:48.796545 14252 raft_consensus.cc:493] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:48.796869 14252 raft_consensus.cc:3060] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.802142 14252 raft_consensus.cc:515] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.803866 14252 leader_election.cc:290] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Requested vote from peers c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923), 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195:38553)
I20260709 12:27:48.804909 14400 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a3327f378e64cbda03898ceaa6701a0" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
I20260709 12:27:48.804911 14326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6a3327f378e64cbda03898ceaa6701a0" candidate_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:48.805864 14400 raft_consensus.cc:3055] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Stepping down as leader of term 1
I20260709 12:27:48.805956 14326 raft_consensus.cc:3060] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.806253 14400 raft_consensus.cc:740] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:48.807245 14400 consensus_queue.cc:260] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.808233 14400 raft_consensus.cc:3060] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:48.812772 14326 raft_consensus.cc:2468] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:48.813721 14400 raft_consensus.cc:2468] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd3bfb90591a4b8c95c500d57be3d44f in term 2.
I20260709 12:27:48.814355 14217 leader_election.cc:304] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd3bfb90591a4b8c95c500d57be3d44f, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:48.815311 14612 raft_consensus.cc:2804] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Leader election won for term 2
I20260709 12:27:48.815685 14612 raft_consensus.cc:697] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Becoming Leader. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Running, Role: LEADER
I20260709 12:27:48.816573 14612 consensus_queue.cc:237] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:48.824373 14145 catalog_manager.cc:5697] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f reported cstate change: term changed from 1 to 2, leader changed from 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195) to bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193). New cstate: current_term: 2 leader_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: UNKNOWN } } }
I20260709 12:27:48.835409 14583 consensus_queue.cc:1048] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:48.852348 14551 consensus_queue.cc:1048] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:48.854427 14557 consensus_queue.cc:1048] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260709 12:27:48.873618 14551 consensus_queue.cc:1048] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:48.993216 14326 raft_consensus.cc:1275] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:48.994498 14612 consensus_queue.cc:1048] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:49.002542 14399 raft_consensus.cc:1275] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:49.005272 14399 raft_consensus.cc:1275] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:49.005426 14524 consensus_queue.cc:1048] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:49.009909 14524 consensus_queue.cc:1048] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:49.020859 14326 raft_consensus.cc:1275] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:49.022467 14612 consensus_queue.cc:1048] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260709 12:27:49.066798 14326 raft_consensus.cc:1275] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:49.068104 14612 consensus_queue.cc:1048] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:49.078866 14399 raft_consensus.cc:1275] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:49.080555 14612 consensus_queue.cc:1048] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:49.272214 14326 raft_consensus.cc:1275] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:49.273511 14564 consensus_queue.cc:1048] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:49.281018 14400 raft_consensus.cc:1275] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Refusing update from remote peer c403bb4886574f1db5b159d81a264617: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:49.283149 14584 consensus_queue.cc:1048] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:49.286058 14399 raft_consensus.cc:1275] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Refusing update from remote peer bd3bfb90591a4b8c95c500d57be3d44f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:49.287986 14623 consensus_queue.cc:1048] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260709 12:27:49.293859 14249 raft_consensus.cc:3060] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Advancing to term 2
I20260709 12:27:49.303397 14249 raft_consensus.cc:1275] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Refusing update from remote peer c403bb4886574f1db5b159d81a264617: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260709 12:27:49.305217 14584 consensus_queue.cc:1048] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20260709 12:27:51.483765 14145 catalog_manager.cc:2533] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47014:
table { table_name: "soft_deleted_table" } modify_external_catalogs: true reserve_seconds: 3600
I20260709 12:27:51.518537 14145 catalog_manager.cc:3570] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47014:
table { table_name: "soft_deleted_table" } new_extra_configs { key: "kudu.table.disable_compaction" value: "true" }
I20260709 12:27:51.616310 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:51.729326 14664 tablet.cc:1724] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:27:51.819054 14655 log.cc:927] Time spent T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30: Append to log took a long time: real 0.135s	user 0.000s	sys 0.001s
I20260709 12:27:51.830307 14669 tablet.cc:1724] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:51.933037 14674 tablet.cc:1724] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:27:51.937945 14656 log.cc:927] Time spent T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30: Append to log took a long time: real 0.215s	user 0.005s	sys 0.000s
I20260709 12:27:51.943078 14676 tablet.cc:1724] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:51.938540 14664 tablet.cc:1724] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:27:51.964574 14290 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41273, user_credentials={real_user=slave}} blocked reactor thread for 217374us
W20260709 12:27:52.065892 14660 log.cc:927] Time spent T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617: Append to log took a long time: real 0.246s	user 0.004s	sys 0.000s
I20260709 12:27:52.095942 14669 tablet.cc:1724] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.096386 14674 tablet.cc:1724] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.097149 14664 tablet.cc:1724] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.101016 14680 tablet.cc:1724] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.116705 14676 tablet.cc:1724] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.136758 14251 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "6c7f424db5614fef9965b16eaae1864c"
dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
mode: GRACEFUL
new_leader_uuid: "7310265ac7544fd281403a40bf5d6d30"
 from {username='slave'} at 127.0.0.1:55634
I20260709 12:27:52.137439 14251 raft_consensus.cc:606] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Received request to transfer leadership to TS 7310265ac7544fd281403a40bf5d6d30
W20260709 12:27:52.137619 14659 log.cc:927] Time spent T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f: Append to log took a long time: real 0.431s	user 0.002s	sys 0.003s
I20260709 12:27:52.151849 14674 tablet.cc:1724] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.160724 14653 raft_consensus.cc:993] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f: : Instructing follower 7310265ac7544fd281403a40bf5d6d30 to start an election
I20260709 12:27:52.161528 14653 raft_consensus.cc:1081] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Signalling peer 7310265ac7544fd281403a40bf5d6d30 to start an election
I20260709 12:27:52.161376 14676 tablet.cc:1724] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.165345 14400 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "6c7f424db5614fef9965b16eaae1864c"
dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
 from {username='slave'} at 127.0.0.1:54628
I20260709 12:27:52.166011 14400 raft_consensus.cc:493] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:52.166465 14400 raft_consensus.cc:3060] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:27:52.183917 14249 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.592s	user 0.007s	sys 0.019s
W20260709 12:27:52.184429 14249 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.592s	user 0.007s	sys 0.019s
I20260709 12:27:52.186137 14676 tablet.cc:1724] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.219125 14680 tablet.cc:1724] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.224205 14400 raft_consensus.cc:515] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:52.229702 14400 leader_election.cc:290] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 3 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:52.227135 14669 tablet.cc:1724] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.229529 14664 tablet.cc:1724] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.232721 14248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c7f424db5614fef9965b16eaae1864c" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:52.233464 14248 raft_consensus.cc:3055] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:52.233767 14248 raft_consensus.cc:740] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Becoming Follower/Learner. State: Replica: bd3bfb90591a4b8c95c500d57be3d44f, State: Running, Role: LEADER
I20260709 12:27:52.234596 14248 consensus_queue.cc:260] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:52.235801 14248 raft_consensus.cc:3060] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Advancing to term 3
W20260709 12:27:52.241456 14327 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.564s	user 0.013s	sys 0.003s
W20260709 12:27:52.241891 14327 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.565s	user 0.013s	sys 0.003s
I20260709 12:27:52.248549 14692 tablet.cc:1724] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:27:52.232589 14366 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.194:33923, user_credentials={real_user=slave}} blocked reactor thread for 44471.7us
I20260709 12:27:52.244799 14248 raft_consensus.cc:2468] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 3.
I20260709 12:27:52.262467 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6c7f424db5614fef9965b16eaae1864c" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:52.263167 14327 raft_consensus.cc:3060] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:52.304183 14327 raft_consensus.cc:2468] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 3.
I20260709 12:27:52.306770 14366 leader_election.cc:304] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, c403bb4886574f1db5b159d81a264617; no voters: 
I20260709 12:27:52.314790 14693 raft_consensus.cc:2804] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:52.323735 14676 tablet.cc:1724] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.345038 14693 raft_consensus.cc:697] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 3 LEADER]: Becoming Leader. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:52.346808 14685 tablet.cc:1724] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.347180 14696 tablet.cc:1724] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.347616 14693 consensus_queue.cc:237] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:52.347177 14694 tablet.cc:1724] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.411379 14690 tablet.cc:1724] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.422861 14674 tablet.cc:1724] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
W20260709 12:27:52.295871 14364 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.201.193:41273, user_credentials={real_user=slave}} blocked reactor thread for 52964us
I20260709 12:27:52.444315 14692 tablet.cc:1724] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.474213 14664 tablet.cc:1724] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.469254 14669 tablet.cc:1724] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.526669 14145 catalog_manager.cc:5697] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 reported cstate change: term changed from 2 to 3, leader changed from bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193) to 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195). New cstate: current_term: 3 leader_uuid: "7310265ac7544fd281403a40bf5d6d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:52.564009 14676 tablet.cc:1724] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30: Alter schema from (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 0 to (
    10:key INT32 NOT NULL,
    11:int_val INT32 NOT NULL,
    12:string_val STRING NULLABLE,
    PRIMARY KEY (key)
) version 1
I20260709 12:27:52.585292 14327 tablet_service.cc:2072] Received LeaderStepDown RPC: tablet_id: "90fa812051104a8cba9311995042ec56"
dest_uuid: "c403bb4886574f1db5b159d81a264617"
mode: GRACEFUL
new_leader_uuid: "7310265ac7544fd281403a40bf5d6d30"
 from {username='slave'} at 127.0.0.1:47658
I20260709 12:27:52.586175 14327 raft_consensus.cc:606] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 2 LEADER]: Received request to transfer leadership to TS 7310265ac7544fd281403a40bf5d6d30
I20260709 12:27:52.590797 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260709 12:27:52.592554 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:52.609308 14669 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:47614 (request call id 21) took 1031 ms. Trace:
I20260709 12:27:52.609987 14669 rpcz_store.cc:276] 0709 12:27:51.577848 (+     0us) service_pool.cc:167] Inserting onto call queue
0709 12:27:51.577995 (+   147us) service_pool.cc:224] Handling call
0709 12:27:52.609170 (+1031175us) inbound_call.cc:177] Queueing success response
Related trace 'op':
0709 12:27:51.625399 (+     0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0709 12:27:51.639187 (+ 13788us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0709 12:27:51.639226 (+    39us) alter_schema_op.cc:66] Acquired schema lock
0709 12:27:51.639265 (+    39us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0709 12:27:51.639392 (+   127us) alter_schema_op.cc:128] START. Timestamp: 1783600071639321
0709 12:27:51.639414 (+    22us) op_driver.cc:348] REPLICATION: starting
0709 12:27:51.679202 (+ 39788us) log.cc:844] Serialized 204 byte log entry
0709 12:27:52.065153 (+385951us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b4400022b4c. stack: 00007fe8f9cb6343 00007fe8f9cb6240 00007fe8f96451d4 00007fe903383304 00007fe903383299 00007fe8f9b70db1 00007fe8f9c0ec85 00007fe8f9c0f6a2 00007fe8fc9811db 00007fe8fc97caaf 00007fe8fc97086e 00007fe8fc96fbb6 00007fe8fc96f7d8 00007fe8fc979221 00007fe8fc9791d9 00007fe8fc979169
0709 12:27:52.463213 (+398060us) op_driver.cc:464] REPLICATION: finished
0709 12:27:52.468753 (+  5540us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0709 12:27:52.496124 (+ 27371us) log.cc:844] Serialized 5 byte log entry
0709 12:27:52.603011 (+106887us) tablet_metadata.cc:665] Metadata flushed
0709 12:27:52.607784 (+  4773us) log.cc:844] Serialized 14 byte log entry
0709 12:27:52.608937 (+  1153us) alter_schema_op.cc:73] Released schema lock
0709 12:27:52.608972 (+    35us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5032,"prepare.queue_time_us":37267,"prepare.run_cpu_time_us":31897,"prepare.run_wall_time_us":299966,"raft.queue_time_us":37227,"raft.run_cpu_time_us":1326,"raft.run_wall_time_us":1326,"replication_time_us":796145,"spinlock_wait_cycles":2889600,"thread_start_us":253076,"threads_started":3,"wal-append.queue_time_us":119575}]]}
I20260709 12:27:52.634754 14147 catalog_manager.cc:6242] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc: soft_deleted_table [id=8b85bcbaa86549cc8332a0fc52e4e164] alter complete (version 1)
I20260709 12:27:52.787463 14327 raft_consensus.cc:1275] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 3 FOLLOWER]: Refusing update from remote peer 7310265ac7544fd281403a40bf5d6d30: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:52.789075 14693 consensus_queue.cc:1048] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:27:52.801172 14252 raft_consensus.cc:1275] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 3 FOLLOWER]: Refusing update from remote peer 7310265ac7544fd281403a40bf5d6d30: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:52.806945 14697 consensus_queue.cc:1048] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
W20260709 12:27:52.988690 14348 debug-util.cc:398] Leaking SignalData structure 0x7b080034df00 after lost signal to thread 30506
W20260709 12:27:52.990272 14348 debug-util.cc:398] Leaking SignalData structure 0x7b08000cadc0 after lost signal to thread 14183
W20260709 12:27:52.992049 14348 debug-util.cc:398] Leaking SignalData structure 0x7b08000cb260 after lost signal to thread 14276
W20260709 12:27:52.994611 14348 debug-util.cc:398] Leaking SignalData structure 0x7b0800168c40 after lost signal to thread 14425
I20260709 12:27:53.046074 14671 raft_consensus.cc:993] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617: : Instructing follower 7310265ac7544fd281403a40bf5d6d30 to start an election
I20260709 12:27:53.046885 14626 raft_consensus.cc:1081] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 2 LEADER]: Signalling peer 7310265ac7544fd281403a40bf5d6d30 to start an election
I20260709 12:27:53.049060 14398 tablet_service.cc:2044] Received Run Leader Election RPC: tablet_id: "90fa812051104a8cba9311995042ec56"
dest_uuid: "7310265ac7544fd281403a40bf5d6d30"
 from {username='slave'} at 127.0.0.1:54622
I20260709 12:27:53.050225 14398 raft_consensus.cc:493] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260709 12:27:53.050704 14398 raft_consensus.cc:3060] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:53.058774 14398 raft_consensus.cc:515] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:53.061234 14398 leader_election.cc:290] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 3 election: Requested vote from peers bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273), c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923)
I20260709 12:27:53.062454 14327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90fa812051104a8cba9311995042ec56" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "c403bb4886574f1db5b159d81a264617"
I20260709 12:27:53.062517 14248 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "90fa812051104a8cba9311995042ec56" candidate_uuid: "7310265ac7544fd281403a40bf5d6d30" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "bd3bfb90591a4b8c95c500d57be3d44f"
I20260709 12:27:53.063602 14327 raft_consensus.cc:3055] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 2 LEADER]: Stepping down as leader of term 2
I20260709 12:27:53.063668 14248 raft_consensus.cc:3060] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:53.064033 14327 raft_consensus.cc:740] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: c403bb4886574f1db5b159d81a264617, State: Running, Role: LEADER
I20260709 12:27:53.065109 14327 consensus_queue.cc:260] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:53.068068 14327 raft_consensus.cc:3060] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Advancing to term 3
I20260709 12:27:53.074769 14248 raft_consensus.cc:2468] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 3.
I20260709 12:27:53.076303 14364 leader_election.cc:304] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7310265ac7544fd281403a40bf5d6d30, bd3bfb90591a4b8c95c500d57be3d44f; no voters: 
I20260709 12:27:53.077201 14693 raft_consensus.cc:2804] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 3 FOLLOWER]: Leader election won for term 3
I20260709 12:27:53.077719 14693 raft_consensus.cc:697] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 3 LEADER]: Becoming Leader. State: Replica: 7310265ac7544fd281403a40bf5d6d30, State: Running, Role: LEADER
I20260709 12:27:53.078966 14327 raft_consensus.cc:2468] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7310265ac7544fd281403a40bf5d6d30 in term 3.
I20260709 12:27:53.078961 14693 consensus_queue.cc:237] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } }
I20260709 12:27:53.097891 14145 catalog_manager.cc:5697] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 reported cstate change: term changed from 2 to 3, leader changed from c403bb4886574f1db5b159d81a264617 (127.29.201.194) to 7310265ac7544fd281403a40bf5d6d30 (127.29.201.195). New cstate: current_term: 3 leader_uuid: "7310265ac7544fd281403a40bf5d6d30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7310265ac7544fd281403a40bf5d6d30" member_type: VOTER last_known_addr { host: "127.29.201.195" port: 38553 } health_report { overall_health: HEALTHY } } }
I20260709 12:27:53.564500 14252 raft_consensus.cc:1275] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 3 FOLLOWER]: Refusing update from remote peer 7310265ac7544fd281403a40bf5d6d30: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:53.566156 14710 consensus_queue.cc:1048] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd3bfb90591a4b8c95c500d57be3d44f" member_type: VOTER last_known_addr { host: "127.29.201.193" port: 41273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260709 12:27:53.576946 14327 raft_consensus.cc:1275] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 3 FOLLOWER]: Refusing update from remote peer 7310265ac7544fd281403a40bf5d6d30: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260709 12:27:53.579000 14693 consensus_queue.cc:1048] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c403bb4886574f1db5b159d81a264617" member_type: VOTER last_known_addr { host: "127.29.201.194" port: 33923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260709 12:27:53.594403 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:53.611457 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:53.612226 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:54.613441 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:54.619037 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:54.619558 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:55.620693 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:55.626365 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:55.626935 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:56.628017 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:56.633955 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:56.634516 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:57.635982 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:57.645401 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:57.646070 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:58.647419 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:58.655535 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:58.656159 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:27:59.657435 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:27:59.665133 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:27:59.665760 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:00.666992 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:00.672673 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:00.673209 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:01.674302 30503 auto_leader_rebalancer.cc:141] leader rebalance for table test-workload
I20260709 12:28:01.679512 30503 auto_leader_rebalancer.cc:440] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260709 12:28:01.679991 30503 auto_leader_rebalancer.cc:515] All tables' leader rebalancing finished this round
I20260709 12:28:02.749073 30503 tablet_server.cc:179] TabletServer@127.29.201.193:0 shutting down...
I20260709 12:28:02.795615 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:02.796754 30503 tablet_replica.cc:333] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.797897 30503 raft_consensus.cc:2243] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.799007 30503 raft_consensus.cc:2272] T b72d6bd4be824dceb80f1bb341feaf77 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.802066 30503 tablet_replica.cc:333] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.803011 30503 raft_consensus.cc:2243] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.803742 30503 raft_consensus.cc:2272] T 801bc4a961d84dc38e9ead3e902346d1 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.806962 30503 tablet_replica.cc:333] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.807781 30503 raft_consensus.cc:2243] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.808455 30503 raft_consensus.cc:2272] T 376d2a317bcb4d6299c45dfe6b45f477 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.811522 30503 tablet_replica.cc:333] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.812194 30503 raft_consensus.cc:2243] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.813135 30503 raft_consensus.cc:2272] T ab7958026c31429697818f80570d4314 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.817866 30503 tablet_replica.cc:333] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.818886 30503 raft_consensus.cc:2243] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.819762 30503 raft_consensus.cc:2272] T bb862cfa73bb46f185a2181b6922d403 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.824436 30503 tablet_replica.cc:333] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.825758 30503 raft_consensus.cc:2243] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.827931 30503 raft_consensus.cc:2272] T 6a3327f378e64cbda03898ceaa6701a0 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.831764 30503 tablet_replica.cc:333] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.832540 30503 raft_consensus.cc:2243] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.834026 30503 raft_consensus.cc:2272] T c1d4d250a41b4c41b7e0ee5973f1143c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.837566 30503 tablet_replica.cc:333] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.838318 30503 raft_consensus.cc:2243] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.839577 30503 raft_consensus.cc:2272] T c8038818270e47f1abba0a9a3dcab848 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.842909 30503 tablet_replica.cc:333] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.843632 30503 raft_consensus.cc:2243] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.845105 30503 raft_consensus.cc:2272] T d2d0355002344a8086f697db621c2d65 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.848901 30503 tablet_replica.cc:333] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.849553 30503 raft_consensus.cc:2243] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.850133 30503 raft_consensus.cc:2272] T 834228cc45dd4a64bf3883ce6244969f P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.852953 30503 tablet_replica.cc:333] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.853567 30503 raft_consensus.cc:2243] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.854620 30503 raft_consensus.cc:2272] T f71a8ae43eb54833a729dffab676352c P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.857554 30503 tablet_replica.cc:333] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.858098 30503 raft_consensus.cc:2243] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.858579 30503 raft_consensus.cc:2272] T 6c7f424db5614fef9965b16eaae1864c P bd3bfb90591a4b8c95c500d57be3d44f [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.861287 30503 tablet_replica.cc:333] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.861812 30503 raft_consensus.cc:2243] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.862967 30503 raft_consensus.cc:2272] T 607b3a3e5d56487d815d403fa1373cfc P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.867185 30503 tablet_replica.cc:333] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.867795 30503 raft_consensus.cc:2243] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:02.868769 30503 raft_consensus.cc:2272] T 4a68fb8a14f5495fb30dd7dd8b58d11f P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.871377 30503 tablet_replica.cc:333] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.871950 30503 raft_consensus.cc:2243] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.872445 30503 raft_consensus.cc:2272] T ed0b601cccbc41c089d18bd6924693f1 P bd3bfb90591a4b8c95c500d57be3d44f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.875059 30503 tablet_replica.cc:333] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.875627 30503 raft_consensus.cc:2243] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.876209 30503 raft_consensus.cc:2272] T 8279557ea37842edb1dc6a1bd65340eb P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.878948 30503 tablet_replica.cc:333] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.879500 30503 raft_consensus.cc:2243] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.880363 30503 raft_consensus.cc:2272] T 0b256c149e1d418cb673a5372afad5c4 P bd3bfb90591a4b8c95c500d57be3d44f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:02.883008 30503 tablet_replica.cc:333] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f: stopping tablet replica
I20260709 12:28:02.883524 30503 raft_consensus.cc:2243] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:02.884186 30503 raft_consensus.cc:2272] T 90fa812051104a8cba9311995042ec56 P bd3bfb90591a4b8c95c500d57be3d44f [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:02.915151 14290 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 14 similar messages]
W20260709 12:28:02.919425 14290 consensus_peers.cc:597] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 -> Peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Couldn't send request to peer bd3bfb90591a4b8c95c500d57be3d44f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:02.928879 30503 tablet_server.cc:196] TabletServer@127.29.201.193:0 shutdown complete.
I20260709 12:28:02.977567 30503 tablet_server.cc:179] TabletServer@127.29.201.194:0 shutting down...
W20260709 12:28:02.997058 14290 consensus_peers.cc:597] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 -> Peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Couldn't send request to peer bd3bfb90591a4b8c95c500d57be3d44f. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:03.013763 14364 consensus_peers.cc:597] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 -> Peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Couldn't send request to peer bd3bfb90591a4b8c95c500d57be3d44f. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:03.018282 14290 consensus_peers.cc:597] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 -> Peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Couldn't send request to peer bd3bfb90591a4b8c95c500d57be3d44f. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:03.018404 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:03.019498 30503 tablet_replica.cc:333] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.020792 30503 raft_consensus.cc:2243] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 2 LEADER]: Raft consensus shutting down.
W20260709 12:28:03.021108 14290 consensus_peers.cc:597] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 -> Peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Couldn't send request to peer bd3bfb90591a4b8c95c500d57be3d44f. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:03.022198 30503 raft_consensus.cc:2272] T b72d6bd4be824dceb80f1bb341feaf77 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.025115 30503 tablet_replica.cc:333] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.025771 30503 raft_consensus.cc:2243] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.026292 30503 raft_consensus.cc:2272] T 376d2a317bcb4d6299c45dfe6b45f477 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.028957 30503 tablet_replica.cc:333] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.029631 30503 raft_consensus.cc:2243] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:03.030808 30503 raft_consensus.cc:2272] T ab7958026c31429697818f80570d4314 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.033298 30503 tablet_replica.cc:333] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.033994 30503 raft_consensus.cc:2243] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:03.035277 30503 raft_consensus.cc:2272] T 801bc4a961d84dc38e9ead3e902346d1 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.038436 30503 tablet_replica.cc:333] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.039122 30503 raft_consensus.cc:2243] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus shutting down.
W20260709 12:28:03.039469 14364 consensus_peers.cc:597] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 -> Peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Couldn't send request to peer bd3bfb90591a4b8c95c500d57be3d44f. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:03.039790 30503 raft_consensus.cc:2272] T c1d4d250a41b4c41b7e0ee5973f1143c P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.043278 30503 tablet_replica.cc:333] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.044036 30503 raft_consensus.cc:2243] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.044718 30503 raft_consensus.cc:2272] T c8038818270e47f1abba0a9a3dcab848 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:03.045789 14290 consensus_peers.cc:597] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 -> Peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Couldn't send request to peer bd3bfb90591a4b8c95c500d57be3d44f. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:03.047415 30503 tablet_replica.cc:333] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.048132 30503 raft_consensus.cc:2243] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.048723 30503 raft_consensus.cc:2272] T 0b256c149e1d418cb673a5372afad5c4 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.051383 30503 tablet_replica.cc:333] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.052011 30503 raft_consensus.cc:2243] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.052582 30503 raft_consensus.cc:2272] T 4a68fb8a14f5495fb30dd7dd8b58d11f P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.055538 30503 tablet_replica.cc:333] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.056138 30503 raft_consensus.cc:2243] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:03.057205 30503 raft_consensus.cc:2272] T 8279557ea37842edb1dc6a1bd65340eb P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.059828 30503 tablet_replica.cc:333] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.060474 30503 raft_consensus.cc:2243] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.061074 30503 raft_consensus.cc:2272] T 607b3a3e5d56487d815d403fa1373cfc P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.063324 30503 tablet_replica.cc:333] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.063923 30503 raft_consensus.cc:2243] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.064479 30503 raft_consensus.cc:2272] T f71a8ae43eb54833a729dffab676352c P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.066998 30503 tablet_replica.cc:333] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.067616 30503 raft_consensus.cc:2243] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.068159 30503 raft_consensus.cc:2272] T 6c7f424db5614fef9965b16eaae1864c P c403bb4886574f1db5b159d81a264617 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.070453 30503 tablet_replica.cc:333] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.071038 30503 raft_consensus.cc:2243] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:03.072009 30503 raft_consensus.cc:2272] T ed0b601cccbc41c089d18bd6924693f1 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.074544 30503 tablet_replica.cc:333] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.075143 30503 raft_consensus.cc:2243] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:03.076056 30503 raft_consensus.cc:2272] T bb862cfa73bb46f185a2181b6922d403 P c403bb4886574f1db5b159d81a264617 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.078452 30503 tablet_replica.cc:333] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.079085 30503 raft_consensus.cc:2243] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.079707 30503 raft_consensus.cc:2272] T 6a3327f378e64cbda03898ceaa6701a0 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.082167 30503 tablet_replica.cc:333] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.082841 30503 raft_consensus.cc:2243] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.083454 30503 raft_consensus.cc:2272] T d2d0355002344a8086f697db621c2d65 P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.085942 30503 tablet_replica.cc:333] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.086503 30503 raft_consensus.cc:2243] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:03.087464 30503 raft_consensus.cc:2272] T 834228cc45dd4a64bf3883ce6244969f P c403bb4886574f1db5b159d81a264617 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.089964 30503 tablet_replica.cc:333] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617: stopping tablet replica
I20260709 12:28:03.090794 30503 raft_consensus.cc:2243] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.091411 30503 raft_consensus.cc:2272] T 90fa812051104a8cba9311995042ec56 P c403bb4886574f1db5b159d81a264617 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260709 12:28:03.149482 14366 consensus_peers.cc:597] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 -> Peer c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923): Couldn't send request to peer c403bb4886574f1db5b159d81a264617. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:33923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:03.169333 30503 tablet_server.cc:196] TabletServer@127.29.201.194:0 shutdown complete.
I20260709 12:28:03.215864 30503 tablet_server.cc:179] TabletServer@127.29.201.195:0 shutting down...
W20260709 12:28:03.227454 14366 consensus_peers.cc:597] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 -> Peer c403bb4886574f1db5b159d81a264617 (127.29.201.194:33923): Couldn't send request to peer c403bb4886574f1db5b159d81a264617. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.194:33923: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260709 12:28:03.232753 14364 consensus_peers.cc:597] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 -> Peer bd3bfb90591a4b8c95c500d57be3d44f (127.29.201.193:41273): Couldn't send request to peer bd3bfb90591a4b8c95c500d57be3d44f. Status: Network error: Client connection negotiation failed: client connection to 127.29.201.193:41273: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260709 12:28:03.260550 30503 ts_tablet_manager.cc:1507] Shutting down tablet manager...
I20260709 12:28:03.261425 30503 tablet_replica.cc:333] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.262161 30503 raft_consensus.cc:2243] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:03.263188 30503 raft_consensus.cc:2272] T 376d2a317bcb4d6299c45dfe6b45f477 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.265576 30503 tablet_replica.cc:333] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.266234 30503 raft_consensus.cc:2243] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.266903 30503 raft_consensus.cc:2272] T ab7958026c31429697818f80570d4314 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.269243 30503 tablet_replica.cc:333] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.269846 30503 raft_consensus.cc:2243] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.270383 30503 raft_consensus.cc:2272] T b72d6bd4be824dceb80f1bb341feaf77 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.272718 30503 tablet_replica.cc:333] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.273365 30503 raft_consensus.cc:2243] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.273880 30503 raft_consensus.cc:2272] T 801bc4a961d84dc38e9ead3e902346d1 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.276409 30503 tablet_replica.cc:333] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.277004 30503 raft_consensus.cc:2243] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [term 2 LEADER]: Raft consensus shutting down.
I20260709 12:28:03.277832 30503 raft_consensus.cc:2272] T 0b256c149e1d418cb673a5372afad5c4 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.281235 30503 tablet_replica.cc:333] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.282140 30503 raft_consensus.cc:2243] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.282670 30503 raft_consensus.cc:2272] T f71a8ae43eb54833a729dffab676352c P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.285108 30503 tablet_replica.cc:333] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.285671 30503 raft_consensus.cc:2243] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:03.286767 30503 raft_consensus.cc:2272] T 6c7f424db5614fef9965b16eaae1864c P 7310265ac7544fd281403a40bf5d6d30 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.289132 30503 tablet_replica.cc:333] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.289661 30503 raft_consensus.cc:2243] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.290184 30503 raft_consensus.cc:2272] T 607b3a3e5d56487d815d403fa1373cfc P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.292670 30503 tablet_replica.cc:333] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.293246 30503 raft_consensus.cc:2243] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.293709 30503 raft_consensus.cc:2272] T 4a68fb8a14f5495fb30dd7dd8b58d11f P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.296149 30503 tablet_replica.cc:333] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.296710 30503 raft_consensus.cc:2243] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.297132 30503 raft_consensus.cc:2272] T ed0b601cccbc41c089d18bd6924693f1 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.299481 30503 tablet_replica.cc:333] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.300235 30503 raft_consensus.cc:2243] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.300688 30503 raft_consensus.cc:2272] T 8279557ea37842edb1dc6a1bd65340eb P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.302951 30503 tablet_replica.cc:333] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.303484 30503 raft_consensus.cc:2243] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.303972 30503 raft_consensus.cc:2272] T bb862cfa73bb46f185a2181b6922d403 P 7310265ac7544fd281403a40bf5d6d30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.306239 30503 tablet_replica.cc:333] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.306898 30503 raft_consensus.cc:2243] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.307430 30503 raft_consensus.cc:2272] T 6a3327f378e64cbda03898ceaa6701a0 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.309639 30503 tablet_replica.cc:333] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.310158 30503 raft_consensus.cc:2243] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.310663 30503 raft_consensus.cc:2272] T d2d0355002344a8086f697db621c2d65 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.313045 30503 tablet_replica.cc:333] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.313561 30503 raft_consensus.cc:2243] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.314000 30503 raft_consensus.cc:2272] T 834228cc45dd4a64bf3883ce6244969f P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.316457 30503 tablet_replica.cc:333] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.316967 30503 raft_consensus.cc:2243] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 3 LEADER]: Raft consensus shutting down.
I20260709 12:28:03.317871 30503 raft_consensus.cc:2272] T 90fa812051104a8cba9311995042ec56 P 7310265ac7544fd281403a40bf5d6d30 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.320257 30503 tablet_replica.cc:333] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.320735 30503 raft_consensus.cc:2243] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.321241 30503 raft_consensus.cc:2272] T c8038818270e47f1abba0a9a3dcab848 P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.323570 30503 tablet_replica.cc:333] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30: stopping tablet replica
I20260709 12:28:03.324012 30503 raft_consensus.cc:2243] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260709 12:28:03.324460 30503 raft_consensus.cc:2272] T c1d4d250a41b4c41b7e0ee5973f1143c P 7310265ac7544fd281403a40bf5d6d30 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.368922 30503 tablet_server.cc:196] TabletServer@127.29.201.195:0 shutdown complete.
I20260709 12:28:03.409016 30503 master.cc:562] Master@127.29.201.254:37363 shutting down...
I20260709 12:28:03.435256 30503 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [term 1 LEADER]: Raft consensus shutting down.
I20260709 12:28:03.435866 30503 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260709 12:28:03.436213 30503 tablet_replica.cc:333] T 00000000000000000000000000000000 P 981e75aa16d74694a15008175e8137dc: stopping tablet replica
I20260709 12:28:03.458577 30503 master.cc:584] Master@127.29.201.254:37363 shutdown complete.
[       OK ] FilterSoftDeletedTableTest.TestFilterSofteDeletedTable/1 (21459 ms)
[----------] 2 tests from FilterSoftDeletedTableTest (43786 ms total)

[----------] Global test environment tear-down
[==========] 8 tests from 2 test suites ran. (530027 ms total)
[  PASSED  ] 8 tests.
I20260709 12:28:03.520100 30503 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:220: suppressed but not reported on 2250 messages since previous log ~33 seconds ago
I20260709 12:28:03.520316 30503 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:304: suppressed but not reported on 2250 messages since previous log ~33 seconds ago
I20260709 12:28:03.520455 30503 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet_mm_ops.cc:139: suppressed but not reported on 2242 messages since previous log ~33 seconds ago
I20260709 12:28:03.520617 30503 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:600: suppressed but not reported on 9 messages since previous log ~55 seconds ago
I20260709 12:28:03.520781 30503 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/ts_tablet_manager.cc:548: suppressed but not reported on 9 messages since previous log ~55 seconds ago
I20260709 12:28:03.520934 30503 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/master/catalog_manager.cc:4755: suppressed but not reported on 1 messages since previous log ~97 seconds ago
I20260709 12:28:03.521090 30503 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tserver/heartbeater.cc:646: suppressed but not reported on 1 messages since previous log ~138 seconds ago
I20260709 12:28:03.521294 30503 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/rpc/proxy.cc:239: suppressed but not reported on 10 messages since previous log ~0 seconds ago
I20260709 12:28:03.521632 30503 logging.cc:436] LogThrottler /apache/dev/git/kudu/src/kudu/tablet/tablet.cc:2406: suppressed but not reported on 53 messages since previous log ~20 seconds ago
